Posts

A fleet of servers, ordered as the same part number, imaged identically, configured by the same Ansible role. One of them booted with no network. Nothing in the configuration was wrong. Nothing in the operating system had changed. A storage controller was mounted in the front bay rather than a rear riser slot, which put it on a PCIe root port that enumerates earlier, which shifted every PCI bus number behind it, which renamed four network interfaces, which meant every line of network configuration matching those names quietly applied to nothing at all. ...

While investigating a separate network issue at work, we observed unexpected Ethernet link behavior on an embedded ARM system. During boot, the switch port facing it reports link up at 100 Mbps, then down, then up at 100 Mbps again, three times in total, before finally settling at 1 Gbps once the operating system is up. Before proceeding, we needed to determine whether the cycling was related to the issue at hand or independent behavior that could be ruled out. ...

I wanted to measure how much latency a network device adds to a path, precisely enough to characterize a DUT (Device Under Test) that adds less than a microsecond. The instrument is a quad-port Intel i350, which does per-packet hardware timestamping and exposes a PTP hardware clock per port. The DUTs: a Netgear FE104 repeater hub, a Netgear FS105 10/100 switch, and a Netgear GS305 gigabit switch. Results up front: ...

This is not a new problem. There is nothing to patch, because none of it is a defect in the usual sense. It is the documented, intended behavior of a protocol that a surprising amount of infrastructure still runs today, very often underneath home directories. I first ran this audit a few years ago against a large Linux infrastructure running CentOS 7, and recently set out to document it. Reproducing it on a modern Ubuntu 22.04 client took one detour, which I will explain shortly. The result is the same as it has always been: NFSv3 with AUTH_SYS does not authenticate anyone. It takes the client at its word. ...

I work with industrial Linux systems that have strict real-time requirements. The hosts run low-latency kernels and depend on out-of-tree kernel drivers compiled against the running kernel. When any part of that chain breaks, critical functionality is lost. Kernel pinning and out-of-tree driver management are two related problems that need to be solved together. This post covers why, and how. Why the low-latency kernel The generic kernel runs fine on most Ubuntu hosts. The issue is not that it is broken; it is that it does not meet the timing requirements of the software and hardware devices connected to these systems. ...

This is part 2 of the homelab PTP grandmaster series. Part 1 used a Raspberry Pi CM4 with SatPulse. This build uses a NanoPi R5S-LTS and takes a different path to the same destination. The R5S has no standard GPIO header. It has an FPC connector. Getting a GPS module talking to it required a breakout board, a ribbon cable, a soldering iron, and more debugging than expected. The results made it worth it. ...

I work with industrial infrastructure that requires nanosecond-accurate time synchronization. The commercial GPS-disciplined PTP grandmaster clocks that solve this problem run from several thousand dollars on the low end, and considerably more with options and support. I had CM4s and CM4 IO boards already sitting in the lab. A TimeHAT with an OCP M.2 GNSS module would have been the cleaner path at around $400, but that is steep when the goal is learning how PTP actually works, not deploying production infrastructure. This build came in at around $103 in new parts. ...

This post covers setting up SSH public key authentication on AD-joined Linux hosts using SSSD and sss_ssh_authorizedkeys, and two security issues I found in a common SSSD configuration while testing the setup in a lab environment. The pattern stores SSH public keys in Active Directory. SSSD fetches them at login time via the ldap_user_ssh_public_key attribute mapping and hands them to sshd. The relevant snippet in /etc/ssh/sshd_config: AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys %u AuthorizedKeysCommandUser nobody During testing two issues surfaced that are worth documenting: disabled AD accounts could authenticate via SSH key, and revoked keys remained valid for up to 90 minutes after removal from AD. ...

Two NICs, one subnet. Seems fine. It isn’t. I frequently run into this on Linux servers, PTP Grandmaster clocks, telemetry acquisition and analysis servers, industrial controllers and other appliances where Linux is running under the hood whether the vendor advertises it or not. Most of them cover this limitation somewhere in their published documentation. It is worth reading before you plug in the second cable. The symptoms are always the same: traffic arrives on one interface, replies leave on another, sessions drop for no apparent reason, and nothing in the logs explains it. ...

SSH session timeouts looked like a straightforward CMMC Level 2 control. After digging through man pages, source code, bug trackers, and a fair amount of testing, I had a completely different solution than I started with. This is what happened. The Controls Two CMMC Level 2 controls are directly relevant here: AC.L2-3.1.11 (NIST SP 800-171 3.1.11): Automatically terminate user sessions after a defined condition. This means a full termination, not a screen lock, not a network disconnect. The user’s processes must be cleaned up and re-authentication required. For SSH on Linux, the shell must actually exit. ...