Stop the canary crying wolf after every legitimate restart
session_durable treated a REJECTED token as a durability failure. That is not the outage it guards: a gateway restart, or Guacamole's own inactivity timeout, in...
Stop every deploy from throwing everyone off their desktop
"An error has occurred and this action cannot be completed" after leaving a desktop open was not an idle timeout and not a browser problem. It was CI. deploy-s...
Run a dispatched session in the workspace's common parent
sage's VS Code is a multi-root workspace holding all 63 repositories, so taking the first workspace folder rooted a bug about the login site in aiservice — the ...
Say what a dead KamoDesktop connection actually means
Guacamole's stock CLIENT.ERROR_TUNNEL_204 is "The requested connection does not exist. Please check the connection name and try again." There is no connection n...
Submit the bug report without simulating a keystroke
vscode:// URI handler and sent a synthetic Return. Measured on desktop-1, that cannot work: KDE's screen locker holds an exclusive X input grab whenever the scr...
Cache CORS preflights for two hours, not 100 seconds
`accessControlMaxAge: 100` meant browsers re-ran the preflight roughly once per visit, so every page load on the marketing site paid two extra OPTIONS round-tri...
Stop the edge rate limit from 429ing every cold KamoDesktop load
Opening an assigned Linux instance always landed on Guacamole's generic "An error has occurred and this action cannot be completed" modal, and always worked aft...
Make the HLS playlists immutable, and say what that costs
The playlists were the last entry in Lighthouse's cache-lifetime audit — 300s originally, then an hour, still reported. Everything else in the ladder was alread...
Compress the theme host, which was serving every text asset raw
MinIO returns object bytes exactly as stored — it does not negotiate Content-Encoding — and the two theme IngressRoutes carried `strip-hsts` and `cors` but no c...
Pin and verify the kubectl download instead of falling back to a broken wget
The deploy died at "Install kubectl". Three separate faults lined up: 1. curl aborted mid-stream with (56) "decryption failed or bad record mac" — the corru...
Name channels live/<name>, use Recreate, feed the loop a uniform GOP
Three defects found by exercising the deployed service. 1. The /live/ route 404'd through Traefik. MediaMTX answers the first HLS request with a 302 to `?co...
Move the encode job's decision logic out of the mc container
The plan stage died on `sed: command not found`. The minio/mc image is minimal to the point of having no sed, awk, grep, wget or curl — it can move objects and ...
Upload segments with an explicit glob, not --recursive
mc cp --recursive takes a directory; the segments were being passed as a glob, which is the wrong shape for that flag. Segments and the playlist beside them nee...
Preserve the client IP so the KAMOTEAM export ACL matches
Mounts failed with 'access denied by server'. A packet capture inside the guest showed the connection arriving from 10.42.0.151 — a pod-network address — not th...
Make the workstation fstab rewrite safe
Backs up /etc/fstab, normalises a missing trailing newline (a plain append would otherwise splice onto the last entry), and refuses to install a file that lost ...
Stop xrdp restarts orphaning sessions and causing black screens
systemctl restart xrdp makes xrdp-sesman forget its in-memory session registry while the Xorg processes keep running. The next login cannot find the session so ...
Allow SecurityService egress to the dev machine agent, and apply policies from CI
Cilium egress is default-deny for every endpoint it selects, and toEndpoints:[{}] is scoped to the policy's OWN namespace — so SecurityService could not reach d...
Pin the VM MAC and add RDP readiness/liveness probes
KubeVirt generates a random MAC per VMI when macAddress is absent. The guest's netplan matched on the first-boot MAC, so every VMI recreate silently orphaned th...
Repair the kamouniverse backup, broken since the Yugabyte cutover
Renaming the dead CockroachBackupMissing alert in the previous commit made it fire immediately: no universe backup had succeeded since the 2026-08-09 migration....
Raise CPU limit off the 2-core ceiling
Both replicas sat pinned at 1997m/1999m against a 2-core limit for ~47h. That starved /languages enough that the liveness and readiness probes timed out (443 an...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Stop the 60s entrypoint read timeout capping uploads
respondingTimeouts.readTimeout is the deadline for reading a WHOLE request, body included, and Traefik v3 defaults it to 60s where v2 had no limit. That silentl...
Pin the remaining hostPath workloads to the node holding their data
Same fault as embedding-model, found by auditing every hostPath volume in the repo. A hostPath is node-local and nothing replicates it, so an unpinned pod can b...
Pin to the node its model weights are actually on
The model is a hostPath under /var/lib/embedding-model on k1m1, and nothing replicates it. When k3m1 joined, the new ReplicaSet's pod was scheduled there and ne...
Stop the passdb from splitting a mailbox across two maildirs
password_query returned `email as user`, which overwrote the %Lu-lowercased login name with the row's stored casing. user_query then interpolated %n from the re...
STUN answered on the wrong node, so kamocrm.com:3478 was dead
coturn is hostNetwork with no nodeSelector, so it was scheduled on k3m1 (public 47.181.8.87). Its own --external-ip, and every client's `stun:kamocrm.com:3478`,...
Bridge advertised only private ICE candidates — no meet had media for 35 days
The bridge never completed a single ICE connection in the 35 days this pod has been up (0 successes, 18 terminations). Every conference joined over XMPP fine an...
Liveness probe could never pass — cause of 920 restarts in 35 days
Root cause: the liveness probe was `tcpSocket: port 10000`, but Scylla's REST API binds to api_address 127.0.0.1 (its default) while a tcpSocket probe dials the...
Resync drifted manifests with live before enabling full reconciles
Auditing what a full reconcile would apply turned up two manifests that had drifted far enough from the cluster that re-applying them would have caused an outag...
Stop the avatar-resolver policy blocking every login
This policy took the whole platform's login offline for hours. Cilium enforces egress against the RESOLVED BACKEND, after service DNAT, so the port it matches ...
Master Machine uses VNC (x11vnc), not RDP
KRdp/RDP is incompatible with Guacamole: guacd has no RDP Graphics Pipeline (EGFX) and KRdp is H.264-only, so the connection authenticated but died with GRAPHIC...
Pin CockroachDB to v24.2.4 to avoid v24.3 no-license txn throttle
v24.3 (and 24.2.5+) enforce the new CockroachDB Software License; with no license key the cluster throttles to 5 concurrent open transactions cluster-wide. A v2...
Pin RDP color-depth + reap xrdp sessions so god reuses one desktop
xrdp 0.9.24 keys sessions on <user,bpp> (Policy=Default) and never reaps disconnected ones (KillDisconnected=false, DisconnectedTimeLimit=0). The RDP user is al...
Stop expired/orphaned secrets shadowing live per-host certs
update_tls_store() rebuilt the TLSStore from a plain Secret listing, bucketed only on name prefix and type, with no check that a Certificate still owned the sec...
Explicit priority so the OTA updates route wins over the general theme route
Traefik ranks by rule-string length; the general /public/ theme route is longer, so the expo-updates header injection never ran. priority:100000 forces the win.
Raise guacamole memory 1Gi→2Gi to stop OOM mid-RDP-session
The Tomcat webapp idles near ~800Mi, so a 1Gi ceiling left almost no headroom. Opening an RDP session pushed RSS over 1Gi and the container was OOMKilled (exit ...
Move data dir off MinIO FUSE mount to local disk
rspamd's /var/lib/rspamd was hostPath-backed by the rclone FUSE mount of the MinIO shared-files bucket. Every hyperscan recompile / map refresh rewrote files th...
/dev/shm 2Gi (headless Chrome needs >=2GB) + 6Gi mem limit
Jibri's 10-config init aborts with '/dev/shm must be at least 2GB' when shm is 512Mi, crash-looping the recorder. Bump the tmpfs to 2Gi and the memory limit to ...
Correct Guacamole branding manifest version field
Manifest used guacManifestVersion:1.0; the bundled extensions use guacamoleVersion. The wrong field made Guacamole reject the branding extension ('Unable to rea...
Move guacd to pod network (MTU fix) + unicast WoL
guacd on hostNetwork (k1m1 eno49 MTU 1500) vs guacamole pod on cilium overlay (MTU 1420): RDP connected but large frames were dropped across the boundary -> 'Gu...
Watch the desktop namespace for IngressRoutes
desktop.kamocrm.com IngressRoute lives in the desktop ns, but Traefik's CRD provider only watched **************** — so the route was ignored and requests fell ...
Import VM disk as Filesystem, not Block
CDI's block importer crashlooped: 'blockdev: cannot open /dev/cdi-block-volume: Permission denied' — it can't access the raw Longhorn block device. Filesystem v...
Correct KubeVirt VM cloud-init field + modernize spec
- cloudInitNoCloud userdata secret field is 'secretRef' (not userDataSecretRef); KubeVirt v1.8.4 pruned the wrong field -> 'must have at least one userdatasou...
Proactively provision browser-only aliases (media.*)
The log-scan discovery mechanism can never bootstrap the `media` alias: it is only ever contacted by the browser over a WebSocket/XHR subresource (SockJS /ws/in...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.