- Shipped
- 4 Septemba 2026, 19:50 UTC
- Author
- Kamo
- Commit
- ddc2b0f
Deploys replaced the only pod of each service with nothing to catch the requests in flight. Three settings, applied across the fleet: - preStop sleeps 10s before the process sees SIGTERM. Kubernetes removes the pod from its EndpointSlice and signals it at the same moment, and Traefik only learns of the removal by watch — so for a moment it keeps dispatching new requests into a pod that has already begun refusing them. That gap is where the 502s on an otherwise clean rollout came from. - terminationGracePeriodSeconds raised above the preStop sleep, so the hook is not itself SIGKILLed, and in-flight work has room to finish. It is a ceiling, not a wait: an idle pod still exits in about a second. - minReadySeconds 15, so a pod that passes readiness once and then falls over cannot retire the healthy pod it replaced after CI has already called the rollout good. topologySpreadConstraints are added ready for a second replica; they are inert at one. Audited from the live cluster: 63 of 65 deployments in `kamo` ran a single replica, 1 of 65 had a preStop hook, and none had minReadySeconds. The entrypoint already drains correctly on SIGTERM and the grace period is already 660s, but nothing held the pod in rotation for the moment between the EndpointSlice removal and Traefik noticing it — so requests dispatched in that window were refused by a server that had just stopped accepting. terminationGracePeriodSeconds is left at its considered 660.