Raise CPU limit off the 2-core ceiling
Both replicas sat pinned at 1997m/1999m against a 2-core limit for ~47h. That starved /languages enough that the liveness and readiness probes timed out (443 an...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Stop shipping the shared credential in this repo
The mail MySQL password sat in plaintext across fourteen files here — in ConfigMaps that kubectl will hand to anyone who can read them, in two Jobs that CI appl...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Allow YugabyteDB port 5433 in kamowssecurity Cilium egress
A CiliumNetworkPolicy selecting app=kamowssecurity allowed host/remote-node egress on 26257, 4222 and 9000 only. After the cutover the database moved to 5433 an...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Correct the rationale on pre-created schemas
KamoInitializerService does set **************** true, so it creates these schemas itself. The previous comment claimed it did not. The file stays as explicit b...
Stop the 60s entrypoint read timeout capping uploads
respondingTimeouts.readTimeout is the deadline for reading a WHOLE request, body included, and Traefik v3 defaults it to 60s where v2 had no limit. That silentl...
Pin the remaining hostPath workloads to the node holding their data
Same fault as embedding-model, found by auditing every hostPath volume in the repo. A hostPath is node-local and nothing replicates it, so an unpinned pod can b...
Pin to the node its model weights are actually on
The model is a hostPath under /var/lib/embedding-model on k1m1, and nothing replicates it. When k3m1 joined, the new ReplicaSet's pod was scheduled there and ne...
Stop the passdb from splitting a mailbox across two maildirs
password_query returned `email as user`, which overwrote the %Lu-lowercased login name with the row's stored casing. user_query then interpolated %n from the re...
STUN answered on the wrong node, so kamocrm.com:3478 was dead
coturn is hostNetwork with no nodeSelector, so it was scheduled on k3m1 (public 47.181.8.87). Its own --external-ip, and every client's `stun:kamocrm.com:3478`,...
Bridge advertised only private ICE candidates — no meet had media for 35 days
The bridge never completed a single ICE connection in the 35 days this pod has been up (0 successes, 18 terminations). Every conference joined over XMPP fine an...
Liveness probe could never pass — cause of 920 restarts in 35 days
Root cause: the liveness probe was `tcpSocket: port 10000`, but Scylla's REST API binds to api_address 127.0.0.1 (its default) while a tcpSocket probe dials the...
Resync drifted manifests with live before enabling full reconciles
Auditing what a full reconcile would apply turned up two manifests that had drifted far enough from the cluster that re-applying them would have caused an outag...
Stop the avatar-resolver policy blocking every login
This policy took the whole platform's login offline for hours. Cilium enforces egress against the RESOLVED BACKEND, after service DNAT, so the port it matches ...
Mashine ya Master inatumia VNC (x11vnc), sio RDP
KRdp/RDP is incompatible with Guacamole: guacd has no RDP Graphics Pipeline (EGFX) and KRdp is H.264-only, so the connection authenticated but died with GRAPHIC...
Pin CockroachDB to v24.2.4 to avoid v24.3 no-license txn throttle
v24.3 (and 24.2.5+) enforce the new CockroachDB Software License; with no license key the cluster throttles to 5 concurrent open transactions cluster-wide. A v2...
Pin RDP color-depth + reap xrdp sessions so god reuses one desktop
xrdp 0.9.24 keys sessions on <user,bpp> (Policy=Default) and never reaps disconnected ones (KillDisconnected=false, DisconnectedTimeLimit=0). The RDP user is al...
Stop expired/orphaned secrets shadowing live per-host certs
update_tls_store() rebuilt the TLSStore from a plain Secret listing, bucketed only on name prefix and type, with no check that a Certificate still owned the sec...
Explicit priority so the OTA updates route wins over the general theme route
Traefik ranks by rule-string length; the general /public/ theme route is longer, so the expo-updates header injection never ran. priority:100000 forces the win.
Raise guacamole memory 1Gi→2Gi to stop OOM mid-RDP-session
The Tomcat webapp idles near ~800Mi, so a 1Gi ceiling left almost no headroom. Opening an RDP session pushed RSS over 1Gi and the container was OOMKilled (exit ...
Move data dir off MinIO FUSE mount to local disk
rspamd's /var/lib/rspamd was hostPath-backed by the rclone FUSE mount of the MinIO shared-files bucket. Every hyperscan recompile / map refresh rewrote files th...
/dev/shm 2Gi (mahitaji ya kichwa cha Chrome > = 2GB + kikomo cha 6Gi mem)
Jibri's 10-config init mimba na '/dev/shm lazima angalau 2GB' wakati shm ni 512M, vunja rekodi ya rekodi. Weka tmpfs kwa 2Gi na kikomo cha kumbukumbu 6-Kufanya ...
Correct Guacamole branding manifest version field
Ilani ilitumia guacManifestVersion:1.0; upanuzi wa vifungu hutumia guacamoleVersion. Eneo baya lilimfanya Guacamole kukataa upanuzi wa alama ('hauwezi kusoma' y...
Move guacd to pod network (MTU fix) + unicast WoL
guacd on hostNetwork (k1m1 eno49 MTU 1500) vs guacamole pod on cilium overlay (MTU 1420): RDP connected but large frames were dropped across the boundary -> 'Gu...
Angalia eneo la desktop kwa IngressRoutes
desktop.kamocrm.com IngressRoute anaishi katika desktop ns, lakini CRD ya Traefik Picha zote na Wizara ya Mambo ya Ndani ya Nchi. .................................
Uagizaji wa VM disk kama mfumo wa faili, sio kuzuia
CDI's block importer crashlooped: 'blockdev: cannot open /dev/cdi-block-volume: Permission denied' — it can't access the raw Longhorn block device. Filesystem v...
Correct KubeVirt VM cloud-init field + modernize spec
- cloudInitNoCloud userdata secret field is 'secretRef' (not userDataSecretRef); KubeVirt v1.8.4 pruned the wrong field -> 'must have at least one userdatasou...
Proactively provision browser-only aliases (media.*)
The log-scan discovery mechanism can never bootstrap the `media` alias: it is only ever contacted by the browser over a WebSocket/XHR subresource (SockJS /ws/in...
Correct FPBXCONF heredoc + repoint universe Scylla
- freepbx: the FPBXCONF here-document closing delimiter was indented (8 spaces); bash requires <<WORD delimiters at column 0, so startup.sh died with 'unexpec...
Pin kamo Traefik to k1m1; make auto-cert single-cluster
SSL errors: kamo Traefik is a LoadBalancer with externalIPs 192.168.4.22 (k1m1) + externalTrafficPolicy=Local, so it MUST run on k1m1 — the pod rebalance moved ...
Indent heredoc bodies so the startup.sh configmap is valid YAML
The startup.sh block scalar embedded 5 shell heredocs (rtp.conf, http.conf, an SQL block, WebRTC endpoints, freepbx.conf) whose bodies sat at column 0 — less in...
Kuhamia Dovecot maildirs kwa PVC ya ndani-path (off rclone / MinIO FUSE)
Maildirs were on hostPath /mnt/shared-files/email/mail — the rclone FUSE mount backed by MinIO object storage, which can't safely host a maildir (no atomic rena...
Weka datadir ya MySQL kwenye PVC ya mitaa badala ya rclone / MinIO FUSE
The mail MySQL datadir (and maildirs) were on hostPath /mnt/shared-files, an rclone FUSE mount backed by MinIO object storage, which cannot durably or consisten...
Backup verified working — DB-scoped backup, MinIO region us-west-ca-s-1, EXTERNALIOIMPLICITACCESS grant in bootstrap
Drop WITH revision_history from backup (enterprise-only); core BACKUP is OSS
Loki 6.x SingleBinary needs zeroed scalable targets; crdb exec via localhost (cert SANs)
Refresh TLSStore every loop so newly-issued certs get served
update_tls_store() only ran at startup and on shared-storage imports, so a cert freshly issued by cert-manager (HTTP-01) was Ready with a tls-* secret but never...
Give redirect-only route a backend so Traefik v3 enables it
Traefik v3 disables an IngressRoute router with services: [] when allowEmptyServices is off (as it is here), so the domain-agnostic HTTP->HTTPS redirect router ...
Redirect all HTTP hosts to HTTPS + gate auto-cert to real org domains
Custom org domains (e.g. login.sign.pink) returned Traefik's bare "404 page not found" over plain HTTP: the only port-80 redirect router was hardcoded to *.kamo...
Preserve full FQDN for subdomains deeper than two labels
organize_by_root() was extracting only the first label and the last two labels from each Traefik-logged hostname. For the common two-level case (login.b11capita...
Advertise new public IP 47.181.8.83 for external media
JVB-k1m1 was advertising 75.83.113.11 and coturn was advertising the node's LAN IP via status.hostIP — both broke external WebRTC media now that the public WAN ...
Replace heredoc with grouped echos so YAML literal block parses
The /etc/odbc.ini heredoc body sat at column 1, which dedented out of the surrounding startup.sh: | literal-block scalar and broke `kubectl apply` of the Config...
Follow rclone remounts via HostToContainer mount propagation
Without slave propagation the pod captures the FUSE bind mount once at container start; if rclone-mount.service restarts on the host, the pod keeps pointing at ...
Scale to 2 replicas to absorb sync + runtime traffic
A single replica is the bottleneck for the dictionary sync workflow: each gunicorn worker handles one request then auto-restarts (~5-10s window of connection-re...
Drop unrecognized --workers arg causing crashloop
The libretranslate CLI does not accept --workers, only the gunicorn process inside it does. Passing it caused the pod to fail argparse on boot and CrashLoopBack...
Probe timeout 10s and 2 gunicorn workers
LibreTranslate's /languages probe blocks on the gunicorn worker event loop. With a single worker translating a long chat message, the probe times out at the def...
Rewrite /etc/odbc.ini to point at $MARIADB_HOST on every boot
The escomputers/freepbx:17 image bakes /etc/odbc.ini with Server=db (the docker-compose link target). On this Kubernetes deployment MariaDB lives at \$MARIADB_H...
Je, unaona nini kuhusu usafiri?
Kila moja ya hizi updates ardhi katika nafasi yako ya kazi moja kwa moja. Kuanza bure na kuangalia kukua wiki baada ya wiki.