- Ya
- 4 Septemba 2026, 19:50 UTC
- Mwandishi
- Kamo
- Ahadi ya
- d712ce9
Deploys replaced the only pod of each service with nothing to catch the requests in flight. Three settings, applied across the fleet: - preStop sleeps 10s before the process sees SIGTERM. Kubernetes removes the pod from its EndpointSlice and signals it at the same moment, and Traefik only learns of the removal by watch — so for a moment it keeps dispatching new requests into a pod that has already begun refusing them. That gap is where the 502s on an otherwise clean rollout came from. - terminationGracePeriodSeconds raised above the preStop sleep, so the hook is not itself SIGKILLed, and in-flight work has room to finish. It is a ceiling, not a wait: an idle pod still exits in about a second. - minReadySeconds 15, so a pod that passes readiness once and then falls over cannot retire the healthy pod it replaced after CI has already called the rollout good. topologySpreadConstraints are added ready for a second replica; they are inert at one. Audited from the live cluster: 63 of 65 deployments in `kamo` ran a single replica, 1 of 65 had a preStop hook, and none had minReadySeconds. **************** has set `server.shutdown: graceful` since the beginning, and it has never once been read in production: k8s/deployment.yaml mounts k8s/config.yaml over it as **************** which replaces the file rather than merging with it, and that ConfigMap had no shutdown setting. So APIService — one of the busiest services on the platform — dropped every in-flight request the instant it received SIGTERM, on every deploy. Adds `shutdown: graceful` and a 30s timeout-per-shutdown-phase to the ConfigMap that actually runs.