Root Was Full. The Disk Wasn't.
A Kubernetes node started evicting pods for DiskPressure. kubelet raises DiskPressure when the filesystem holding /var/lib/kubelet or the container image store runs low, and that filesystem was nearly full. The disk underneath had hundreds of GiB that had never been allocated. Kubernetes was where it surfaced, but a full / or /var breaks plenty of other things first: apt fails mid-upgrade, journald and application logs stop writing, and services that need to write state fail in ways that rarely say “disk full”. ...