Start probing at 15s, not 190 — nothing could open on its first wake
A woken computer is only reachable once its Service has endpoints, and that happens when the readiness probe on 3389 first passes. initialDelaySeconds was 180, ...
Xorg may not start from a remote session, so no computer opened
Guacamole connected, authenticated, and was hung up on: guacd: Security mode: Negotiate (ANY) guacd: Loading keymap "en-us-qwerty" guacd: RDP serve...
Traefik could not see the gateway, and it served at the wrong path
Opening a computer landed on the redirect service's fallback page — "the request reached the fallback page instead of being redirected. Check the Host header." ...
Let auto-cert mirror the gateway certificate into this namespace
Listing computers.kamocrm.com for issuance is only half of it. Traefik requires a TLS secret in the IngressRoute's OWN namespace, so the cert is issued in `kamo...
A computer that has never restarted can never sleep
prepare-sleep writes the swap file and the resume wiring, then reports readiness — and readiness reads `resume=` from /proc/cmdline, which the kernel fixed at b...
Reinstall grub after virt-resize, or the guest boots to a rescue prompt
virt-resize renumbers partitions. Canonical's image is laid out p14 (BIOS boot), p15 (ESP), p16 (/boot) and then p1 (root) physically last, and virt-resize rewr...
The golden disk has to be RAW, or nothing boots and nothing says so
The job wrote the compressed qcow2 the build produces straight into the golden claim. KubeVirt attaches a Filesystem-PVC disk with <driver type='raw'> — always,...
Only the platform reaches the database's and NATS's admin ports
YugabyteDB and NATS run on k1m1's host network, so every pod in every namespace could open every port they listen on. The client ports authenticate (YSQL :5433 ...
NATS refuses anonymous clients — no_auth_user is gone
Second half of 926986d. Every client now presents the kamo_svc login: the Java services through the shared library's NatsConfig (NATS_USERNAME / NATS_PASSWORD f...
Three RBAC grants no provision had ever needed
Nothing had ever been provisioned, so every grant past the namespace was untested. Each of these refused at exactly the point the previous fix unblocked. `patc...
NATS clients authenticate — first half of retiring no_auth_user
NATS runs on the host network of k1m1 and listens on 0.0.0.0, and its config mapped every credential-less connection to the KAMO account (no_auth_user: anon). S...
Only the platform can reach the session Redis
The `kamo` Redis holds every *** session and OTK and has no password. Nothing restricted who could connect to it: from a pod in the `desktop` namespace (where t...
Five reasons the golden image had never actually been built
The build had never been run. Running it found one bug per attempt, and the first three would each have produced a plausible-looking image with something missin...
The three templates that never got committed
service-template.yaml, cloud-init.userdata.yaml and cloud-init.networkdata.yaml. They were written into a second worktree and committed from neither — the earli...
A reply to a sending domain is refused as the recipient, in words that fit NoReply
Campaigns now send Reply-To: NoReply@<their From's domain> by default, and those replies reach postfix-k3m1-inbound. The refusal said "Sender address rejected: ...
Prune unused images on k3m1 too, and stop bulk removals timing out
k3m1 had no prune job, and kubelet image GC waits for 85% disk, so every deploy's SHA-tagged image piled up: 3,368 images (539G), 86 in use. Add a daily k3m1-im...
This relay only sends as kamocrm.com
Anything in mynetworks (every pod, and the LAN) could relay through 47.181.8.84 as any domain, which receivers treat as spoofing and charge to this IP; that is ...
The delivery-events sidecar drops a batch EmailService rejects as invalid
A 400/413/422 means the batch itself will never be accepted; retrying it forever would hold back every result queued behind it. It is logged and dropped. 401/40...
Keep hello.kamocrm.com off this relay, and stop three errors on every start
- A @hello.kamocrm.com sender is refused relay here with a 4xx, so bulk mail can only leave through postfix-bulk; delivery to our own domains is unaffected....
Strip identity headers on myloan.*/api/ before the portal BFF relays them
KamoMLOS's BFF routes under /api/ (the /api/proxy/* passthrough, /api/session/extend, /api/chat/routing, /api/chat/subject-state) copy every client header excep...
Delete identity headers at the edge before header-trusting backends
Add a strip-identity-headers Middleware. A customRequestHeaders value of "" deletes the header, so X-Org-Id, X-Member-Id, X-User-Id, X-God-Mode-Active, X-Member...
Four workers on a node that can hold them, not two on one that cannot
Correcting the previous commit, which was half right and caused a short outage. The memory diagnosis was correct: four workers, each loading its own ~3GB copy ...
Give the translation models room to stay resident, and refuse a scanner
libretranslate ran 4 gunicorn workers against a 12Gi limit, and each worker loads its own ~3GB copy of the argos model set — 11.8GB resident, 98% of the limit, ...
The terminal had no scrollback to scroll, so turn tmux's mouse on
for anything else. I built it against a bare xterm.js writing its own output and never against the real pipeline, which is the whole of the mistake. tmux is a ...
Send To AI must not open a tab in sage's editor
A Send-to-AI hand-off is hosted in a Remote Terminal the operator is already watching in the console. On top of that, the agent also handed the session to VS Co...
Restarting the agent must not kill every member's shells
mid-work today, and would have taken every other member's terminals with it. systemd's default is KillMode=control-group: on stop or restart it SIGKILLs every ...
A budget must not be able to deadlock a node drain
Switches every PodDisruptionBudget from minAvailable: 1 to maxUnavailable: 1. On a two-replica Deployment the two are identical — one pod evictable at a time. ...
Stop running the platform's entire edge on one pod
Traefik ran a single replica, so every restart of it — a rollout, an OOM, an eviction — took the whole platform's TLS down. 192.168.4.22:443 has no local endpoi...
The AI hand-off raced tmux, so nothing was typed and nothing named
Caught by running it on the machine: the terminal opened at the right path, with no command in it and no title. One race, both symptoms. `bootstrap` runs strai...
The terminal list was always empty — tmux escapes control chars
The field separator was a unit separator (0x1f), which is the obvious choice and silently wrong: tmux OCTAL-ESCAPES control characters in `-F` format output, so...
The terminal's IngressRoute was in a directory nothing applies
It was written into securityservice/k8s/ beside the deployment, which reads as the obvious home and is not one: that repo's workflow applies exactly configmap.y...
Drop `su` — it swallowed SIGWINCH, so no terminal ever resized
Caught by running the protocol against the live machine, not by reading it: after a resize control frame, `tput cols` in the shell still reported the size the t...
Add 8777 to the WebRTC endpoint roster
Aiden Perry's extension was created from KamoCRM, which cannot write this file — so it registered over WSS with no DTLS or ICE and its calls would have carried ...
Probe the bridge's real health, not a port that is always open
kamo-meet let one person into a meeting and then ended it the moment a second joined. The bridge had been hard-unhealthy for 41 hours: an in-place container res...
Set the VM's MTU to the pod network's, and let the desktop apply updates
Two separate reasons the "Code" update could not be installed. THE UPDATE COULD NOT BE DOWNLOADED. enp1s0 came up at MTU 1500 while every other workload on thi...
Stop unattended upgrades restarting xrdp under a live session
The black screen was apt. apt-daily-upgrade.service upgraded xrdp at 06:49 UTC on 2026-08-27 and restarted it while a desktop from 2026-08-25 was running. xrdp-...
Keep FreeSWITCH's log readable while it is crash-looping
FreeSWITCH runs with -nc, so mod_logfile is the only record of what it did, and on the container filesystem that record dies with the container — precisely when...
Stop offering 8 unroutable ICE candidates on every call
Callers hear several seconds of one-way audio at the start of a call — the other party says "hi" three or four times before the softphone user hears anything. ...
SIP registration works end to end
An endpoint can now REGISTER from the public internet and Kazoo's API reports it: **************** 200 OK via 47.181.8.87:5060. Four separate faults, each of w...
Zone must be named 'local' to match the node
The config.ini was loading correctly and the AMQP URI was right, but every node logged 'no local zone configured, adding default AMQP' and then retried ********...
Distinct Erlang node names for apps and ecallmgr
One image runs as two nodes. They need different names or they refuse to coexist, and both need the shared cookie that FreeSWITCH's mod_kazoo also uses.
Run the OTP release, not make release
The image now ships a relx release rather than the source tree, so passing 'make release' to the entrypoint failed with 'No rule to make target release'. foreg...
K3m1 is 47.181.8.87 (pbx.k3.kluster.kamocrm.com)
Reverts an incorrect change of mine. I had propagated 42.181.8.87 and edited coturn/deployment.yaml's comment, which was right all along. Verified by DNS: pbx...
Bound the Go heap with GOMEMLIMIT and declare a real memory request
Root cause of the crash-loop was not a leak. The 485Mi steady state that kept tripping the old 512Mi limit was two normal things stacked: ~350Mi Go heap at ...
Raise memory limit to 1Gi and relax liveness probe timeout
Traefik sat at ~485Mi against a 512Mi limit (95%). The cgroup recorded 47836 memory.max hits with oom_kill 0 — never OOM-killed, because enough of the footprint...
Allowlist Telnyx in both fail2ban jails
Adds the Telnyx US signaling addresses (192.76.120.10, 64.16.250.10 — sip.telnyx.com) to ignoreip now that the carrier trunk is provisioned. Also overrides ign...
Make fail2ban actually run — enable Asterisk's security log
fail2ban has never started on this PBX. Its asterisk-security jail watches /var/log/asterisk/security, a file Asterisk never created: FreePBX generates logger_l...
The auto-reconnect reload-looped every page into a blank screen
I shipped this and it took the desktop down: a blank white page and a tab loading forever. Disabled in-cluster immediately; this is the real fix. `ng-switch` s...
The auto-reconnect was watching for a class that never appears
It keyed on **************** That class exists in Guacamole's stylesheet and in none of its templates, so the selector matched nothing: the script loaded, ran, ...
Cap Guacamole's heap, and unbreak the deploy that was blocking it
Two faults, one of them mine. MINE FIRST: the `node --test` gate added with the auto-reconnect script pointed at a DIRECTORY, and the runner's node resolves a ...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.