KamoCRM

Reserve host memory, guard etcd's disk at the dm level, uncap CI IO

FixKlusterServices
Shipped
November 17, 2026 at 2:22 AM UTC
Author
Kamo
Commit
06469fe

Root cause of the 2026-11-15..17 control-plane flaps: k1m1 ran out of page cache (min free 33G -> 7G in two weeks, major faults 7/s -> 210/s), so cache misses became reads on the spinning RAID that also holds etcd, etcd stalled and rke2 lost its leader lease ~75 times. - kubelet **************** on both nodes (allocatable 123 -> ~106 GiB), so pod requests can't claim the host's memory or page cache; applied on the nodes with an rke2 restart - io-guard: systemd IO*Max= resolves to the raw disk and never bites for LVM IO; kamo-io-guard writes io.max for the dm devices on k1m1 (containerd, docker, registry; etcd/apiserver untouched). Hand-installed, see README - CI on k3m1: no IO caps (no etcd there; builds sat at the cap with the disk 3% busy). CPU/memory caps stay; per-step watcher keeps cpu/memory only - traefik: 1 replica; rolling updates still surge a second pod to switch

All changes

Like what you see shipping?

All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.

Start Free ForeverView Pricing