- Ya
- 4 Septemba 2026, 19:55 UTC
- Mwandishi
- Kamo
- Ahadi ya
- 2909805
Traefik ran a single replica, so every restart of it — a rollout, an OOM, an eviction — took the whole platform's TLS down. 192.168.4.22:443 has no local endpoint while that pod is gone (externalTrafficPolicy: Local), the node falls through to rke2-ingress-nginx's default certificate, and because nearly every kamo router matches on HostRegexp rather than Host, TLS resolves purely by SNI against a TLSStore that is not loaded yet. The result is ERR_CERT_AUTHORITY_INVALID on every org at once for the ~11s it takes to load 190 certificates. - replicas 2. Both stay pinned to k1m1, which is deliberate: the public address NATs there and externalTrafficPolicy: Local means a pod anywhere else can never answer it. This buys pod-level redundancy, not node-level — node-level needs a floating VIP and is separate work. - Traefik's own drain, which a preStop sleep cannot replicate: on SIGTERM it keeps ACCEPTING for requestAcceptGraceTimeout (15s) while /ping starts answering 503, so the readiness probe fails and kube-proxy removes it from the Service while it is still serving. Then up to graceTimeOut (30s) finishing what is open. terminationGracePeriodSeconds 30 -> 60 so the drain is not itself SIGKILLed halfway through. - minReadySeconds 15, and CI now waits on `rollout status` for traefik. Also adds a PodDisruptionBudget directory. The kamo namespace had ZERO PDBs while five other namespaces on this cluster had them, so a drain of a node for a kernel update evicted everything on it simultaneously. README.md records the rule and, importantly, why a single-replica Deployment must never get one — minAvailable: 1 against replicas: 1 permits zero disruptions and wedges the drain forever rather than protecting anything. A `retry` middleware is defined and offered as `kamo-middlewares-retry`, deliberately as a separate chain rather than folded into kamo-middlewares. Retry is not safe everywhere: Traefik does not rewind a request body, so retrying a connection lost partway through a 3 GiB attachment sends a truncated one, and it cannot distinguish a request that never landed from one a backend accepted and died before answering. internal., apps. and api. therefore stay on the plain chain. auto-cert now publishes a defaultCertificate on the TLSStore it already owns, so a handshake whose SNI matches nothing answers with a real platform certificate instead of Traefik's built-in self-signed one. Written there rather than as a static manifest because auto-cert rewrites that store every loop and would have overwritten it.