- Ya
- 16 Agosti 2026, 01:43 UTC
- Mwandishi
- Kamo
- Ahadi ya
- 4fd9a2a
Nothing was watching memory. MediaService was OOMKilled every few hours for most of a day and EmailService six times, and the way anyone found out was a chat window that had quietly stopped loading messages — a container dying and restarting inside a second reads as a broken feature, not as an outage. Three rules. ContainerOOMKilled fires on the event; ContainerOOMKillLoop separates a one-off from a pod being killed repeatedly, which restarts fast enough to still show 1/1 Running. ContainerMemoryNearLimit is the one that would have caught this days early: >85% of the limit held for 30 minutes. For a JVM on MaxRAMPercentage that is not a busy pod, it is a pod sitting at its heap ceiling with non-heap stacked on top, which is the steady state immediately before the kernel intervenes. A fourth, informational, flags containers with no limit at all — they cannot be OOMKilled by their own cgroup but can take a node down. Verified against real data rather than assumed: the expressions were evaluated on this cluster, and querying the last 12 hours shows reason="OOMKilled" series for exactly kamowsemail (18:33-01:28 UTC) and kamowsmedia (21:28-01:03 UTC) — the two pods that were failing. The rules would have fired critical on both. An alert that silently matches nothing is the same failure as no alert, so that check mattered more than the YAML. Applied by hand; monitoring/ is not a CI apply target.