Systemd

A fleet of servers, ordered as the same part number, imaged identically, configured by the same Ansible role. One of them booted with no network. Nothing in the configuration was wrong. Nothing in the operating system had changed. A storage controller was mounted in the front bay rather than a rear riser slot, which put it on a PCIe root port that enumerates earlier, which shifted every PCI bus number behind it, which renamed four network interfaces, which meant every line of network configuration matching those names quietly applied to nothing at all. ...

I inherited a set of Ubuntu servers that were provisioned outside of our normal provisioning and configuration methods of Foreman and Ansible, talking to Windows DHCP servers that use DHCP Option 81 for dynamic DNS registration. The problem was that some servers were getting their DNS A records registered correctly and some were not. The inconsistency was problematic, and the more I dug into it, the worse it got. This is the story of chasing that inconsistency, finding a workaround, realising the workaround was unnecessary, and ending up with a PR open against systemd upstream to fix a man page that has been inconsistent and ambiguous for years. ...