Note ratestack Forgejo :3000 node registry config requirement
ratestack apps pull from the Forgejo registry 192.168.4.22:3000 (plain HTTP, auth required), which must be added to each node's registries.yaml (mirror -> http ...
Pin kamo Traefik to k1m1; make auto-cert single-cluster
SSL errors: kamo Traefik is a LoadBalancer with externalIPs 192.168.4.22 (k1m1) + externalTrafficPolicy=Local, so it MUST run on k1m1 — the pod rebalance moved ...
Consolidate klusterservices onto single k1m1+k3m1 cluster
- NATS: standalone single pod pinned k1m1 (drop k2m1 + generic cluster variants) - CockroachDB: pin k1m1, single-node --join - MySQL: single instance moved in f...
Bootstrap universe-minio secret + ugc bucket note (§14)
Add step 6 to bootstrap-universe.sh: creates the universe-minio secret (ACCESS_KEY/SECRET_KEY from universe-backup-s3 root creds) and notes that the universe-ug...
Upgrade k1m1 node image v24.2.4 -> v24.3.33
Rolling major upgrade of the k1m1 CRDB StatefulSet node. Note: cluster will not finalize to 24.3 until the k3m1 node is also upgraded (tracked separately).
Drop the decommissioned k2m1 cluster section
k2m1 (10.8.2.1:6443) is decommissioned and unreachable; its steps lived in the same deploy_services job and failed it (missing kubeconfig / no route) even when ...
Indent heredoc bodies so the startup.sh configmap is valid YAML
The startup.sh block scalar embedded 5 shell heredocs (rtp.conf, http.conf, an SQL block, WebRTC endpoints, freepbx.conf) whose bodies sat at column 0 — less in...
Don't let a missing kubectl cache dir kill the whole job
The kubectl-install step runs 'set -eo pipefail' then find **************** -name kubectl ... | head -1 When that cache dir is absent (cleared between runner ...
Upgrade v3.0 -> v3.3 (fix intermittent WebSocket disconnect)
v3.0.0 intermittently completes the WS 101 upgrade then closes the socket before any frame flows — the recurring 'disconnect on load' in the universe game clien...
Move Dovecot maildirs to a local-path PVC (off rclone/MinIO FUSE)
Maildirs were on hostPath /mnt/shared-files/email/mail — the rclone FUSE mount backed by MinIO object storage, which can't safely host a maildir (no atomic rena...
Put MySQL datadir on a local-path PVC instead of rclone/MinIO FUSE
The mail MySQL datadir (and maildirs) were on hostPath /mnt/shared-files, an rclone FUSE mount backed by MinIO object storage, which cannot durably or consisten...
Backup verified working — DB-scoped backup, MinIO region us-west-ca-s-1, EXTERNALIOIMPLICITACCESS grant in bootstrap
Drop WITH revision_history from backup (enterprise-only); core BACKUP is OSS
node-config: document k1m1 forgejo-runner config (--add-host for submodule DNS)
Job containers on k1m1 fall back to public DNS, so the fleet-swept http://k1m1:3000 .gitmodules URLs fail to clone there. Pin k1m1 into job containers via conta...
node-config: document k1m1 docker daemon insecure-registries
The k1m1 forgejo-runner's builds push to the HTTP Forgejo registry; the daemon.json shipped with only the :5000 entries, so every build job landing on k1m1 fail...
K1m1 image registry replacing retired k0m1:5000 (same credentials)
Loki 6.x SingleBinary needs zeroed scalable targets; crdb exec via localhost (cert SANs)
Kamo-universe namespace+quota+valkey, CockroachDB backups, observability stack
- universe/: namespace + ResourceQuota, valkey-universe (AOF, hostPath k1m1), nightly CockroachDB backup CronJob, operator bootstrap script - monitoring/: pin...
Join shared two-node cluster with k3m1 over LAN; account isolation
Retire WireGuard routes (10.8.x.x) for LAN routes (192.168.4.22 <-> 192.168.4.36 :6222) with creds embedded in the route URLs. Rename cluster to kamo-ratestack-...
Add k3m1 node + migrate cluster peering to LAN IPs
- statefulset: advertise/join over 192.168.4.x (k1m1=.22, k3m1=.36); WireGuard dropped (k3m1 wg0 10.8.0.1 collided with k0m1's advertise IP). - init-job + CI ...
Refresh TLSStore every loop so newly-issued certs get served
update_tls_store() only ran at startup and on shared-storage imports, so a cert freshly issued by cert-manager (HTTP-01) was Ready with a tls-* secret but never...
Give redirect-only route a backend so Traefik v3 enables it
Traefik v3 disables an IngressRoute router with services: [] when allowEmptyServices is off (as it is here), so the domain-agnostic HTTP->HTTPS redirect router ...
Redirect all HTTP hosts to HTTPS + gate auto-cert to real org domains
Custom org domains (e.g. login.sign.pink) returned Traefik's bare "404 page not found" over plain HTTP: the only port-80 redirect router was hardcoded to *.kamo...
Pre-seed sign.pink + www.sign.pink in STATIC_DOMAINS
sign.pink is a new apex domain (consumer e-signature brand / white-label Kamo tenant org). Pre-seed apex + www so a Let's Encrypt cert exists before first HTTPS...
Update subdomain map comment KamoLOS → KamoMLOS
The /Next.js/KamoLOS project was renamed to KamoMLOS (Kamo Mortgage Loan Origination System). The KamoLOS name is now used for the new personal-loan service. Th...
node-config: raise k1m1 kubelet max-pods 110 -> 200
Default RKE2 max-pods=110 was being hit, blocking kamolos rolling-update deploys (maxSurge:1,maxUnavailable:0 needs slot to create new pod before killing old; d...
Document myloan alias for KamoLOS
Adds `myloan` to the SUBDOMAINS recognition list and rewrites the comment to make the list's purpose explicit (it's documentary; the dynamic Traefik log scan is...
Preserve full FQDN for subdomains deeper than two labels
organize_by_root() was extracting only the first label and the last two labels from each Traefik-logged hostname. For the common two-level case (login.b11capita...
Manage mail.kamocrm.com TLS end-to-end
Auto-cert now provisions and maintains the Let's Encrypt cert for mail.kamocrm.com (HTTP-01 via Traefik on port 80) and keeps the postfix consumer in sync acros...
Remove decommissioned Janus WebRTC gateway
Janus is no longer in use. Removes the orphaned KlusterServices/janus/ manifests (Janus was host-installed, not cluster-deployed, and is not running on k1m1). ...
Advertise new public IP 47.181.8.83 for external media
JVB-k1m1 was advertising 75.83.113.11 and coturn was advertising the node's LAN IP via status.hostIP — both broke external WebRTC media now that the public WAN ...
Replace heredoc with grouped echos so YAML literal block parses
The /etc/odbc.ini heredoc body sat at column 1, which dedented out of the surrounding startup.sh: | literal-block scalar and broke `kubectl apply` of the Config...
Follow rclone remounts via HostToContainer mount propagation
Without slave propagation the pod captures the FUSE bind mount once at container start; if rclone-mount.service restarts on the host, the pod keeps pointing at ...
Reverse-proxy priceturbo.com apex + wildcard via k1m1 with TLS
k1m1 Traefik now terminates HTTPS for priceturbo.com and any *.priceturbo.com subdomain, forwarding plain HTTP to the upstream Traefik at 192.168.4.36:80 with t...
Scale to 2 replicas to absorb sync + runtime traffic
A single replica is the bottleneck for the dictionary sync workflow: each gunicorn worker handles one request then auto-restarts (~5-10s window of connection-re...
Reverse-proxy rate.loans/stack.loans hosts via k1m1 with TLS
k1m1 Traefik now terminates HTTPS for api.rate.loans, www.rate.loans, and www.stack.loans, forwarding plain HTTP to the upstream Traefik at 192.168.4.36:80. Aut...
Drop unrecognized --workers arg causing crashloop
The libretranslate CLI does not accept --workers, only the gunicorn process inside it does. Passing it caused the pod to fail argparse on boot and CrashLoopBack...
Add pricing@kamocrm.com mailbox and external IMAP NodePort
- New Service mail/dovecot-imap-external (NodePort 30143 -> 143) so IMAP can be reached from outside the cluster on $NODE:30143. - Seed pricing@kamocrm.com in...
Probe timeout 10s and 2 gunicorn workers
LibreTranslate's /languages probe blocks on the gunicorn worker event loop. With a single worker translating a long chat message, the probe times out at the def...
Scope concurrency to build jobs, not Kubernetes deploys
Workflow-level concurrency serialized the whole pipeline (including kubectl rollouts). Move the group to the build job only so deploy jobs from different runs c...
Rewrite /etc/odbc.ini to point at $MARIADB_HOST on every boot
The escomputers/freepbx:17 image bakes /etc/odbc.ini with Server=db (the docker-compose link target). On this Kubernetes deployment MariaDB lives at \$MARIADB_H...
Configure STUN so browser ICE can complete bidirectionally
Two-party browser-to-browser calls connected at the SIP layer but had no audio either direction. Voicemail (one-way Asterisk → browser) worked because Asterisk'...
Seed pbx.k1.kluster.kamocrm.com as a static domain
The browser softphone needs SIP-over-WSS to land on a hostname with a real TLS cert. Add the FreePBX admin / WS hostname to STATIC_DOMAINS so the auto-cert seed...
Host browser SIP WSS on internal.kamocrm.com (real cert)
Browsers refused the WebSocket connection to **************** with close code 1006 because Traefik was serving its self-signed default cert for that hostname (n...
Make WebRTC config survive pod restart
The previous approach (http_custom.conf with [general] override) didn't actually take effect because Asterisk's INI parser doesn't honor multiple [general] sect...
Enable Asterisk built-in HTTP server on :8089 for WebRTC /ws
A chan_pjsip `protocol=ws` transport rides on top of res_http_websocket, which in turn requires Asterisk's built-in HTTP server to be running. The default FreeP...
Add WSS WebRTC transport for browser softphones
Asterisk's chan_pjsip now binds a plain WebSocket transport on 0.0.0.0:8089 and Traefik terminates TLS at **************** (port 443) so kamo-internal can regis...
Expose port 8088 directly on ClusterIP service
Service only had port 80→8088 mapping. Internal services using **************** (e.g. VOIPService API token requests) were timing out because port 8088 had no l...
Add missing fail2ban-config ConfigMap for k1m1
Deployment references fail2ban-config ConfigMap (jail.local + asterisk-security.conf) but it did not exist, blocking the pod at MountVolume.SetUp and preventing...
Like what you see shipping?
Every one of these updates lands in your workspace automatically. Start free and watch it grow week after week.