- Shipped
- August 25, 2026 at 3:34 PM UTC
- Author
- Kamo
- Commit
- 3342ccd
Two gaps, both proven against the live cluster today. FIRST: every existing KamoDesktop alert measures whether a COMPONENT is alive. All four were green during the 2026-08-25 outage — RDP endpoint ready, agent ready, guacamole 1/1, guacd 1/1 — while nobody could connect, because the SSO token's expires was emptying the session's connection list. Liveness cannot see that. So the canary walks the whole path a person walks: SecurityService mints a real link, it is redeemed at the PUBLIC gateway url (exercising DNS, the certificate and Traefik routing rather than bypassing them), the desktop must appear in the connection directory, and a session opened on an EARLIER cycle must still list its connections. That last stage is the point. The outage left fresh logins working and broke only reconnects, so a probe that logs in fresh each time would never have seen it. Verified end to end against the live gateway with a deliberately short window: fresh login green, held session red, exactly the outage signature. It does NOT open a real RDP session. xrdp keys sessions on <user, colour-depth> and colour-depth is pinned so logins REJOIN one desktop — a probe logging in as a real member would attach to their live session and disconnect them every few minutes. Two TCP checks plus the existing endpoint-readiness alert cover that leg without evicting anyone. SECOND, and the real reason a broken desktop could only be found by trying it: Alertmanager's only receiver was one literally named "null", and every alert in the cluster was routed into it. The rules were firing correctly; nothing was listening. Alerts now go to SysAdmin@KamoCRM.com via the cluster's own postfix and to a member's chat thread. Delivered as an AlertmanagerConfig CR because **************** is applied by hand and is not a CI target — routing that exists only after someone re-runs a script is the same class of problem as the rule nobody applied. New alerts cover a broken login, broken reconnects specifically, an expiring certificate, and a STALE canary — a probe that stops reporting must never read as healthy, since silence is the failure mode this whole pipeline removes. The probe ships as a ConfigMap on a stock python image (standard library only), so there is no second image to keep patched for the life of the cluster, and CI gates the deploy on its 17 unit tests: a canary that silently reports success is worse than no canary.