Correct FPBXCONF heredoc + repoint universe Scylla
- freepbx: the FPBXCONF here-document closing delimiter was indented (8 spaces); bash requires <<WORD delimiters at column 0, so startup.sh died with 'unexpec...
Pin kamo Traefik to k1m1; make auto-cert single-cluster
SSL errors: kamo Traefik is a LoadBalancer with externalIPs 192.168.4.22 (k1m1) + externalTrafficPolicy=Local, so it MUST run on k1m1 — the pod rebalance moved ...
Indent heredoc bodies so the startup.sh configmap is valid YAML
The startup.sh block scalar embedded 5 shell heredocs (rtp.conf, http.conf, an SQL block, WebRTC endpoints, freepbx.conf) whose bodies sat at column 0 — less in...
Move Dovecot maildirs to a local-path PVC (off rclone/MinIO FUSE)
Maildirs were on hostPath /mnt/shared-files/email/mail — the rclone FUSE mount backed by MinIO object storage, which can't safely host a maildir (no atomic rena...
Put MySQL datadir on a local-path PVC instead of rclone/MinIO FUSE
The mail MySQL datadir (and maildirs) were on hostPath /mnt/shared-files, an rclone FUSE mount backed by MinIO object storage, which cannot durably or consisten...
Backup verified working — DB-scoped backup, MinIO region us-west-ca-s-1, EXTERNALIOIMPLICITACCESS grant in bootstrap
Drop WITH revision_history from backup (enterprise-only); core BACKUP is OSS
Loki 6.x SingleBinary needs zeroed scalable targets; crdb exec via localhost (cert SANs)
Refresh TLSStore every loop so newly-issued certs get served
update_tls_store() only ran at startup and on shared-storage imports, so a cert freshly issued by cert-manager (HTTP-01) was Ready with a tls-* secret but never...
Give redirect-only route a backend so Traefik v3 enables it
Traefik v3 disables an IngressRoute router with services: [] when allowEmptyServices is off (as it is here), so the domain-agnostic HTTP->HTTPS redirect router ...
Redirect all HTTP hosts to HTTPS + gate auto-cert to real org domains
Custom org domains (e.g. login.sign.pink) returned Traefik's bare "404 page not found" over plain HTTP: the only port-80 redirect router was hardcoded to *.kamo...
Preserve full FQDN for subdomains deeper than two labels
organize_by_root() was extracting only the first label and the last two labels from each Traefik-logged hostname. For the common two-level case (login.b11capita...
Advertise new public IP 47.181.8.83 for external media
JVB-k1m1 was advertising 75.83.113.11 and coturn was advertising the node's LAN IP via status.hostIP — both broke external WebRTC media now that the public WAN ...
Replace heredoc with grouped echos so YAML literal block parses
The /etc/odbc.ini heredoc body sat at column 1, which dedented out of the surrounding startup.sh: | literal-block scalar and broke `kubectl apply` of the Config...
Follow rclone remounts via HostToContainer mount propagation
Without slave propagation the pod captures the FUSE bind mount once at container start; if rclone-mount.service restarts on the host, the pod keeps pointing at ...
Scale to 2 replicas to absorb sync + runtime traffic
A single replica is the bottleneck for the dictionary sync workflow: each gunicorn worker handles one request then auto-restarts (~5-10s window of connection-re...
Drop unrecognized --workers arg causing crashloop
The libretranslate CLI does not accept --workers, only the gunicorn process inside it does. Passing it caused the pod to fail argparse on boot and CrashLoopBack...
Probe timeout 10s and 2 gunicorn workers
LibreTranslate's /languages probe blocks on the gunicorn worker event loop. With a single worker translating a long chat message, the probe times out at the def...
Rewrite /etc/odbc.ini to point at $MARIADB_HOST on every boot
The escomputers/freepbx:17 image bakes /etc/odbc.ini with Server=db (the docker-compose link target). On this Kubernetes deployment MariaDB lives at \$MARIADB_H...
Configure STUN so browser ICE can complete bidirectionally
Two-party browser-to-browser calls connected at the SIP layer but had no audio either direction. Voicemail (one-way Asterisk → browser) worked because Asterisk'...
Host browser SIP WSS on internal.kamocrm.com (real cert)
Browsers refused the WebSocket connection to **************** with close code 1006 because Traefik was serving its self-signed default cert for that hostname (n...
Make WebRTC config survive pod restart
The previous approach (http_custom.conf with [general] override) didn't actually take effect because Asterisk's INI parser doesn't honor multiple [general] sect...
Enable Asterisk built-in HTTP server on :8089 for WebRTC /ws
A chan_pjsip `protocol=ws` transport rides on top of res_http_websocket, which in turn requires Asterisk's built-in HTTP server to be running. The default FreeP...
Expose port 8088 directly on ClusterIP service
Service only had port 80→8088 mapping. Internal services using **************** (e.g. VOIPService API token requests) were timing out because port 8088 had no l...
Add missing fail2ban-config ConfigMap for k1m1
Deployment references fail2ban-config ConfigMap (jail.local + asterisk-security.conf) but it did not exist, blocking the pod at MountVolume.SetUp and preventing...
Allow /domain/ path prefix on theme IngressRoute for CORS
The theme config at theme.kamocrm.com/domain/... was not matched by the existing route rules (only /public/ and /internal/ were allowed), so Traefik returned a ...
Replace Coqui TTS with Piper TTS (en_US-norman-medium)
Coqui VCTK model has no named speakers — replace with Piper TTS which has the exact norman[medium] voice. Piper binary (~3 MB) and model (~60 MB) are downloaded...
Add local_net for split-DNS / LAN client RTP fix
Phones on the same LAN as k1m1 connect via split DNS (domain resolves to 192.168.4.22 internally). Without local_net, Asterisk still advertises the external dom...
Persist sorcery contact=memory fix and block SIP scanner in startup.sh
- Inject [res_pjsip] contact=memory into sorcery.conf before Asterisk starts; the default sorcery-astdb SQLite backend silently drops all REGISTER contacts - ...
Move hostPath from MinIO (rclone FUSE) to local /var/lib/pbx
MinIO via rclone FUSE does not support POSIX chmod or symlink() — both required by FreePBX's module system. Changed all FreePBX and MariaDB hostPath volumes fro...
Wrap fwconsole/amportal via php for MinIO FUSE hostPath
/var/lib/asterisk is backed by MinIO via rclone FUSE which doesn't support POSIX chmod, so fwconsole scripts can't be made executable on that volume. startup.sh...
Add preStop hook and terminationGracePeriodSeconds
Containers were getting stuck in Terminating because background Asterisk processes kept running after apache2ctl (PID 1) exited. preStop hook sends SIGTERM to A...
Add bootstrap.php require to regenerated /etc/freepbx.conf
The installer writes freepbx.conf with datasource='' and a require_once for bootstrap.php, which config.php depends on to load the FreePBX class. Our regenerate...
Regenerate /etc/freepbx.conf on restart
/etc is ephemeral (not a hostPath), so /etc/freepbx.conf written by the installer is lost on every pod restart. Added else branch to regenerate it from env vars...
Bypass start_asterisk, wait for control socket instead
start_asterisk uses `pidof asterisk` which returns a false positive on pod restarts, causing the wait-for-Asterisk loop to hang. Now startup.sh starts Asterisk ...
Start Asterisk in background, wait for it, then run installer
The entrypoint.sh does not run the FreePBX installer — it only starts services. startup.sh now replicates the entrypoint setup steps, starts Asterisk in backgro...
Let entrypoint handle install, not startup.sh
FreePBX installer (php install -n) requires Asterisk to already be running. Our startup.sh was calling it before Asterisk started, causing "Error communicating ...
Mount /etc/asterisk only, not /etc, to fix PHP simplexml error
Mounting the full /etc as a hostPath volume was corrupting PHP extension loading even after seeding from the image — simplexml_load_file undefined at install ti...
Reduce CPU limits to 2:1 ratio to eliminate overcommitment
Docs 4→1 core, jibri 4→2 cores, libretranslate 2→1 core, bergamot 2→1 core, coturn 1→0.5 core. Saves ~7.5 cores off node limits.
Switch to escomputers/freepbx:17 with correct architecture
tiredofit/freepbx has no FreePBX 17 image (abandoned since 2022). escomputers/freepbx:17 (updated Apr 2026, Asterisk 21 + FreePBX 17) requires a separate MariaD...
Remove nonexistent postfix-lmdb package, install lmdb library instead
postfix-lmdb is not a valid Alpine package; lmdb support is built into the base postfix package. The pod was CrashLoopBackOff due to apk failure.
Fix Docs image registry, add Recreate strategy to coturn/postfix, increase qdrant memory
- Docs: fix wrong image registry (redis.kamo.svc... → **************** add Recreate strategy - coturn: add Recreate strategy to prevent rolling update port conf...
Switch Postfix sasl_passwd from hash to lmdb format
Alpine postfix build does not include hash support — postmap and smtp_sasl_password_maps were silently failing, deferring every outbound relay attempt through s...
Set Traefik externalTrafficPolicy=Local to preserve client IP
With the default "Cluster" policy, kube-proxy SNATs external traffic to the node IP before forwarding to Traefik. That caused Traefik to see the node IP as the ...
Clean up legacy analytics deployment on each CI run
Delete old analytics deployment, service, and ingressroute that conflict with the kamo-analytics deployment managed by its own CI/CD pipeline.
Strip X-Frame-Options header in iframe-compatible middleware
Set X-Frame-Options to empty string to remove the SAMEORIGIN value that was being inherited, allowing CSP frame-ancestors to control iframe embedding instead.
Reset document.title in rebrand() when React overwrites it
React re-applies RSC payload after hydration, resetting the title to "analytics". Add a title check inside the MutationObserver callback so it gets corrected ev...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.
