- Shipped
- August 25, 2026 at 7:39 PM UTC
- Author
- Kamo
- Commit
- c22edbb
Two faults, one of them mine. MINE FIRST: the `node --test` gate added with the auto-reconnect script pointed at a DIRECTORY, and the runner's node resolves a bare directory as a module to execute — MODULE_NOT_FOUND, exit 1, deploy failed. Both pushes after it (the conditional-restart fix and the canary false-alarm fix) therefore never shipped; runs 9150 and 9152 failed and the cluster stayed on 3342ccd. Naming the test file fixes it. Verified with the exact command CI runs, from the repo root. THE OOM: "an error has occurred" out of nowhere at 19:30 was an OOMKill (exit 137), not a leak and not the deploy restart. `jcmd VM.flags` on the live pod reported MaxHeapSize=32210157568 — a 32 GB heap ceiling inside a 2 GiB container. This is Java 8 on a cgroup v2 host: its container detection never reads /sys/fs/cgroup/memory.max (the limit IS visible there, 2147483648), so it sized itself against the host's 128 GB instead. Eden grew to ~1.0 GB and old gen to ~0.93 GB while the live set was ~23 MB; with no pressure to collect, RSS climbed 821 MiB -> 1963 MiB over four hours and the kernel killed it at the limit. Prometheus shows that curve exactly, and it began BEFORE the canary existed, so the canary is not implicated. -Xmx768m removes the dependence on container detection entirely. MALLOC_ARENA_MAX and ActiveProcessorCount stop glibc and the GC from sizing themselves against the node's 48 CPUs. Raising the limit again was the wrong move on its own: 1Gi->2Gi had already been tried and only changed how long it took. Alerts added for both the approach (>85% of limit) and the event (OOMKilled), because the gateway does not degrade when it runs out of memory — it is killed, and every open desktop goes with it.