Post-mortem: ~10-minute outage of all eom.dev sites (Nov 6, ~22:20–22:30 UTC)

While reworking single sign-on across eom.dev, I had Claude (Anthropic’s AI coding agent) fix the login settings for each service. One step restarted PeerTube. The new PeerTube pod was scheduled onto a node that was already low on disk, and the node evicted pods to free space. One of them was the MetalLB speaker that announces the ingress load-balancer IP on the local network. With nothing announcing that IP, every eom.dev site became unreachable.

Claude spotted the problem when its own checks started failing everywhere at once. Ruling things out one at a time, it showed that the ingress controller was healthy but outside traffic had stopped arriving. It traced that to the evicted MetalLB speaker and the node’s disk pressure, and linked the timing to its PeerTube restart. Its attempt to clean up the evicted pods was blocked by its permission guardrails, which is how I have it set up. By then the disk pressure had cleared and Kubernetes had brought a new speaker up on its own, so service came back without intervention.

Follow-ups: free up disk on that node, and stop heavy workloads from landing on the node that hosts ingress.

Claude is running autonomously, sometimes for several hours at a time. There was definitely an “oh shit” moment when I noticed everything was down and the model was cheerfully saying “now PeerTube is down too”. But the system repaired itself and continued working without any intervention on my part. I was initially disappointed that it caused a major outage, but am actually impressed with how it corrected itself.