Implementation plan for the CockroachDB to YugabyteDB migration
15 tasks. The two gates sit before any downtime: Task 5 proves the backups actually restore, and Task 10 diffs the KamoInitializer-built schema against the live...
Implementation plan for the CockroachDB to YugabyteDB migration
15 tasks. The two gates sit before any downtime: Task 5 proves the backups actually restore, and Task 10 diffs the KamoInitializer-built schema against the live...
Design for the CockroachDB to YugabyteDB migration
Records the measured current state (3.4 GB, 791 tables, 671 FKs, ~30 consumers), the five design decisions, and the cutover sequence. The two findings that sha...
Stop the 60s entrypoint read timeout capping uploads
respondingTimeouts.readTimeout is the deadline for reading a WHOLE request, body included, and Traefik v3 defaults it to 60s where v2 had no limit. That silentl...
Pin the remaining hostPath workloads to the node holding their data
Same fault as embedding-model, found by auditing every hostPath volume in the repo. A hostPath is node-local and nothing replicates it, so an unpinned pod can b...
Pin to the node its model weights are actually on
The model is a hostPath under /var/lib/embedding-model on k1m1, and nothing replicates it. When k3m1 joined, the new ReplicaSet's pod was scheduled there and ne...
Stop the passdb from splitting a mailbox across two maildirs
password_query returned `email as user`, which overwrote the %Lu-lowercased login name with the row's stored casing. user_query then interpolated %n from the re...
STUN answered on the wrong node, so kamocrm.com:3478 was dead
coturn is hostNetwork with no nodeSelector, so it was scheduled on k3m1 (public 47.181.8.87). Its own --external-ip, and every client's `stun:kamocrm.com:3478`,...
Bridge advertised only private ICE candidates — no meet had media for 35 days
The bridge never completed a single ICE connection in the 35 days this pod has been up (0 successes, 18 terminations). Every conference joined over XMPP fine an...
Liveness probe could never pass — cause of 920 restarts in 35 days
Root cause: the liveness probe was `tcpSocket: port 10000`, but Scylla's REST API binds to api_address 127.0.0.1 (its default) while a tcpSocket probe dials the...
Manage ScyllaDB through deploy-services instead of by hand
scylladb/ held complete manifests but had no deploy step — the comment said it was "deployed MANUALLY after the 16TB drive swap" because the data dir /mnt/backu...
Resync drifted manifests with live before enabling full reconciles
Auditing what a full reconcile would apply turned up two manifests that had drifted far enough from the cluster that re-applying them would have caused an outag...
Rename cluster ratestack-scylla -> kamo-scylla, correct ownership docs
This instance is kamo's: it backs the `kamouniverse` keyspace (read/written by **************** through the scylladb-direct ExternalName) and the `audit` keyspa...
Scale ratestack MySQL to 0 and suspend its backup job
The ratestack estate has been torn down to free cluster resources for kamocrm. The cluster-services MySQL instance serves ratestack only (kamo and mail each run...
Stop the avatar-resolver policy blocking every login
This policy took the whole platform's login offline for hours. Cilium enforces egress against the RESOLVED BACKEND, after service DNAT, so the port it matches ...
Egress containment for the avatar resolver, as a CiliumNetworkPolicy
Inaelezea nini 8f6c672 ilirudi, kwa fomu pekee ambayo inafanya kazi hapa. Jaribio la awali lilishindwa kwa sababu mtandao wa wazi hauwezi kuelezea hili. Topolo...
Drop the avatar-resolver egress policy
Inarekebisha sera iliyoongezwa katika 06db7db na imewekwa katika 24f76ae. Ilichukua Huduma ya usalama chini ya dakika sita katika uzalishaji (pod 0/1, Hikari uh...
Contain egress from the avatar resolver; note the Redis eviction pool
Huduma ya Usalama sasa inaendesha mporaji ambaye URL ya lengo inachaguliwa na wasio waaminifu Chama - rekodi ya BIMI ni maudhui ya DNS yaliyochapishwa na mtu ye...
Mashine ya Master inatumia VNC (x11vnc), sio RDP
KRdp/RDP is incompatible with Guacamole: guacd has no RDP Graphics Pipeline (EGFX) and KRdp is H.264-only, so the connection authenticated but died with GRAPHIC...
Kuongeza "Master Machine" (kamo-dev-1) RDP kifaa kwa Guacamole
Local Kubuntu workstation shares its live Plasma/Wayland session over RDP via KRdp. Adds an idempotent bootstrap SQL connection (192.168.4.46:3389, sage/shared ...
Pin CockroachDB to v24.2.4 to avoid v24.3 no-license txn throttle
v24.3 (and 24.2.5+) enforce the new CockroachDB Software License; with no license key the cluster throttles to 5 concurrent open transactions cluster-wide. A v2...
Pin RDP color-depth + reap xrdp sessions so god reuses one desktop
xrdp 0.9.24 keys sessions on <user,bpp> (Policy=Default) and never reaps disconnected ones (KillDisconnected=false, DisconnectedTimeLimit=0). The RDP user is al...
Stop expired/orphaned secrets shadowing live per-host certs
update_tls_store() rebuilt the TLSStore from a plain Secret listing, bucketed only on name prefix and type, with no check that a Certificate still owned the sec...
Explicit priority so the OTA updates route wins over the general theme route
Traefik ranks by rule-string length; the general /public/ theme route is longer, so the expo-updates header injection never ran. priority:100000 forces the win.
Self-hosted expo-updates headers for Mobile Copilot OTA
expo-protocol-version:1 + expo-sfv-version:0 response headers that the expo-updates client requires and static MinIO cannot emit. Longer path prefix wins over t...
Expose AIService /ws /ai-chat kupitia api. * majeshi kwa wateja wa simu
IngressRoute (priority 100, websecure, ws-upgrade middleware sans compress) routes wss://api.<domain>/ws/ai-chat to kamowsai-service:80 — same pattern as the me...
Raise guacamole memory 1Gi→2Gi to stop OOM mid-RDP-session
The Tomcat webapp idles near ~800Mi, so a 1Gi ceiling left almost no headroom. Opening an RDP session pushed RSS over 1Gi and the container was OOMKilled (exit ...
Weka vipengele tu ambavyo usanidi wake ulibadilika; kamwe usakinishe upya
huduma za kupeleka.yml ilitumia tena kila huduma na waendeshaji wa nguzo zilizosanikishwa (cert-manager, KubeVirt, CDI - na 600s kusubiri) juu ya kila kushiniki...
Move data dir off MinIO FUSE mount to local disk
rspamd's /var/lib/rspamd was hostPath-backed by the rclone FUSE mount of the MinIO shared-files bucket. Every hyperscan recompile / map refresh rewrote files th...
/dev/shm 2Gi (mahitaji ya kichwa cha Chrome > = 2GB + kikomo cha 6Gi mem)
Jibri's 10-config init mimba na '/dev/shm lazima angalau 2GB' wakati shm ni 512M, vunja rekodi ya rekodi. Weka tmpfs kwa 2Gi na kikomo cha kumbukumbu 6-Kufanya ...
Meet-secrets **************** for jibri recording ingest
Lets jibri finalize.sh authenticate its recording upload to MediaService; matches the cluster *** value MediaService validates against.
Make whisper-asr independently scalable (HPA, multi-node)
Remove the single-node pin and switch to a per-pod model cache so pods can schedule on any node, and add an HPA (min 1 / max 4, CPU 75%, deliberate scale-up giv...
Rebrand Guacamole client connection status text to KamoDesktop
Juu ya [TD="width: 456"] [FONT=&](2)[/FONT][FONT=&]Bila kuathiri masharti ya kifungu kidogo (1) 'Kuwasiliana na Guacamole...' skrini (na hali inayohusiana) inas...
Correct Guacamole branding manifest version field
Ilani ilitumia guacManifestVersion:1.0; upanuzi wa vifungu hutumia guacamoleVersion. Eneo baya lilimfanya Guacamole kukataa upanuzi wa alama ('hauwezi kusoma' y...
Moduli ya moderator ya Prosody, Whisper-asr, jibri kurekodi upload
- Meet-prosody: version-control the custom-plugins ConfigMap (captures the existing token_verification no-op override that was created out-of-band) and add ...
Guacamole Json-auth SSO + KamoDesktop rebrand
- JSON_SECRET_KEY (optional) on guacamole from the shared desktop-sso secret enables the bundled guacamole-auth-json extension the god-SSO links validate agai...
Design spec for KamoDesktop god SSO icons + Guacamole rebrand
Icons mbili za Mungu-tu nav (Linux / Windows) katika kamo-internal kwamba kufungua Guacamole katika Huduma ya Usalama (mlango unaotumiwa na Mungu), uliounganish...
Move guacd to pod network (MTU fix) + unicast WoL
guacd on hostNetwork (k1m1 eno49 MTU 1500) vs guacamole pod on cilium overlay (MTU 1420): RDP connected but large frames were dropped across the boundary -> 'Gu...
Angalia eneo la desktop kwa IngressRoutes
desktop.kamocrm.com IngressRoute anaishi katika desktop ns, lakini CRD ya Traefik Picha zote na Wizara ya Mambo ya Ndani ya Nchi. .................................
Uagizaji wa VM disk kama mfumo wa faili, sio kuzuia
CDI's block importer crashlooped: 'blockdev: cannot open /dev/cdi-block-volume: Permission denied' — it can't access the raw Longhorn block device. Filesystem v...
Correct KubeVirt VM cloud-init field + modernize spec
- cloudInitNoCloud userdata secret field is 'secretRef' (not userDataSecretRef); KubeVirt v1.8.4 pruned the wrong field -> 'must have at least one userdatasou...
Harded KubeVirt eneo la mbali + lango la Guacamole
Adds a full remote-desktop stack, GitOps-applied via the deploy workflow: - KubeVirt + CDI installed at deploy time (pinned-stable release URLs) - desktop-1: K...
Design spec for hardened KubeVirt remote desktop + Guacamole gateway
KubeVirt Kubuntu VM (16Gi / 500Gi Longhorn) nyuma ya Apache Guacamole HTML5, Imewekwa kwenye desktop.kamocrm.com kupitia Traefik (TLS + kiwango-limit + IP-ban)....
Proactively provision browser-only aliases (media.*)
The log-scan discovery mechanism can never bootstrap the `media` alias: it is only ever contacted by the browser over a WebSocket/XHR subresource (SockJS /ws/in...
Utoaji SOCIAL MESSAGES JetStream mkondo (social.inbound.>, social.lead.) R1 kwa bomba la ndani la vyombo vya habari vya kijamii
Provision R1 MEDIA JetStream stream via idempotent bootstrap Job
Media/lead-feed events (media.feed.*, media.post.*) had no JetStream stream, so publishes returned '503 No Responders Available For Request' (chat.> is covered ...
node-config(k1m1-runner): record data.forgejo.org IPv4 pin + capacity=6
Documents the live on-node change: data.forgejo.org is IPv6-only and k1m1 has no IPv6 route, so job containers hung ~133s and actions/checkout failed. Pin its I...
Correct FPBXCONF heredoc + repoint universe Scylla
- freepbx: the FPBXCONF here-document closing delimiter was indented (8 spaces); bash requires <<WORD delimiters at column 0, so startup.sh died with 'unexpec...
Je, unaona nini kuhusu usafiri?
Kila moja ya hizi updates ardhi katika nafasi yako ya kazi moja kwa moja. Kuanza bure na kuangalia kukua wiki baada ya wiki.