KamoCRM

Rke2-unwedge, a watchdog for the rke2 log-pipe deadlock

FeatureKlusterServices
Shipped
November 5, 2026 at 8:48 PM UTC
Author
Kamo
Commit
a9bc3ad

The 2026-09-28 and 2026-10-25 platform outages were the same deadlock: rke2 loses its lease, its ExecStopPost kills containerd, nobody reads the container log FIFOs, and etcd/apiserver/traefik block on their next log line, so rke2 crashloops forever. Both were fixed by hand-draining FIFOs. This does that automatically: when rke2's containerd is gone AND a container process is blocked on a pipe for 30s, start one cat per log FIFO (into /var/log/rke2-unwedge/, /dev/null if disk is short), and stop each one once containerd re-opens its FIFO. It never signals etcd, rke2, containerd or containers. test.sh exercises it against a real blocked pipe; --scan on k1m1 finds 183 shims / 396 FIFOs, matching the 10-25 manual drain.

All changes

Like what you see shipping?

All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.

Start Free ForeverView Pricing