Size every TEI thread pool to the 1-CPU limit, and let probes outlast a batch
TEI 1.5's ONNX backend starts one inference thread per physical host core (24 on k1m1) and reads no thread setting, so under the pod's 1-CPU quota the CFS throt...
The default user may only PING
SP00 stage R3. Persists the runtime rule for a future start of redis-0: default keeps nopass for the unauthenticated readiness probe and may run PING and nothin...
Deploy Kamo Speech, and pre-add the livekit/ hook so SP13 needs no second full reconcile
Kamo Speech's registry facts and its registration through AIService's internal registry
The last wait needs VERIFIED to hold for 120 s. An attestation itself writes VERIFIED and #36 carries no probe time, so a first VERIFIED read cannot tell SP01's...
Transcribe realtime turns over the pod's own loopback
Speaches 0.9.0-rc.3 posts every realtime turn to /v1/audio/transcriptions. With LOOPBACK_HOST_URL unset it makes that call in-process through httpx's ASGITransp...
A live probe of every Kamo Speech capability, with a four-stream load check
The probe runs inside the pod and prints one JSON document for register.py: a TTS round trip per voice and language (recall >= 0.8), the batch STT row, a realti...
Kamo Speech on k3m1 — pinned Speaches, read-only models, DNS-only egress
Keys and NetworkPolicies on in-cluster inference; Kamo reranker and Kamo Local LLM
TEI embeddings and the new cross-encoder reranker require TEI_API_KEY; bergamot, LibreTranslate and the new Kamo-owned Ollama (qwen3:4b) sit behind the Kamo aut...
Pinned, hash-verified model fetch and a warm-up that holds the pod until every model is resident
The warm-up also drops a model whose load failed: Speaches 0.9.0-rc.3 keeps the entry of a failed load, so the next startupProbe would get 409 and /api/ps would...
Svc_ai publishes ai.registry.changed.> (KamoAI SP01 registry events)
Svc_security publishes kamoai.platform.state, svc_kamoai subscribes to it (KamoAI SP02)
SecurityService announces the platform AI kill switch on kamoai.platform.state and KamoAI Service listens for it. rt.org.<org>.ai-workforce needs no new grant: ...
Reserve 768M for the OS so --memory 3G fits in the 4Gi pod
Seastar keeps max(1.5 GiB, 7%) of the cgroup limit plus ~94 MiB per shard back for the OS and refuses to start when --memory does not fit in what is left. With ...
Scylla to 2 CPU / 4Gi, libretranslate request to 10Gi (32 GiB, 6 CPU released)
Scylla: --smp 2 --memory 3G in a Guaranteed 2 CPU / 4Gi pod (14-day peak 0.66 GiB, 0.05 CPU; the database is 2.93 MB). The init container now has requests == li...
Log in to Redis as mail_rspamd
rspamd reads its password from Secret mail/rspamd-redis-auth, included into local.d/redis.conf. The pod gains fsGroup 11333: the image runs as 11333:11333 and a...
ACL users for the platform, rspamd and operators (default still open)
X keeps counting through a sleep, so every resumed computer went straight back to sleep
A member opening their computer got a black tab. The launch was fine, the wake was fine, the resume was fine — and then the machine powered itself off before th...
A cloud session asks for its task in the terminal
description") unless the account has the interactive cloud backend, which this one does not; given one, it creates the session, prints its task for the cloud se...
Start a coder session in the cloud from a local checkout
repository. The bundle is forced because our remotes are Forgejo, not GitHub, and the point is the checkout on this disk. No --permission-mode (the CLI drops by...
ASCII in finalize.sh's log lines — jibri's logger printed the em dash as ???
Xrdp was told its session backend was itself, so nothing ever rendered
Opening a computer gave a black screen that never became a desktop. The cause is one unanchored sed in this file: sed -i 's/^port=.*/port=3389/' /etc/xrdp/...
Recording had never worked — jicofo never looked for jibri, and jibri never arrived
Four separate faults, each enough on its own to stop every recording: - jicofo had no ENABLE_RECORDING, so the image wrote no `jibri {}` block: it never watc...
Recreate, and probe the XMPP session — a surge left meetings with no focus for 18h
Every jicofo logs in to prosody as the same full JID, focus@auth.meet.Meet/focus. The maxSurge rollout from 9da0998 runs two of them at once, and prosody answer...
Two threads rebuilt the initrd at once and destroyed the machine
A computer provisioned, came up, and panicked on its next boot: Run /init as init process Failed to execute /init (error -2) Kernel panic - not syn...
The restart could land mid-initramfs and leave a computer unbootable
A brand new computer provisioned, came up, reported AWAKE, and ninety seconds later panicked: Kernel panic - not syncing: No working init found. I caused ...
Kamo-internal's session-handle secret
kamo-internal 3e3c3e7a mounts **************** (SESSION_HANDLE_SECRET), which seals the per-tab session handles that replace the *** id in page script. Created ...
A resumed computer went straight back to sleep
Thirty seconds after the first real resume, from the machine's own log: kamo-hc-agent: idle 13202s >= 1800s — going to sleep PM: hibernation: hibernati...
Kamouniverse.com redirect was losing to the www.* catch-all
www.kamouniverse.com was served by the kamocrm.com marketing app instead of redirecting: kamo-marketing-route's HostRegexp(\`^www[.].+\$\`) at priority 24 match...
Add kamouniverse.com as a second regular-mail domain
Adds kamouniverse.com to postfix's virtual_alias_domains with a local-part-preserving regexp so any address on it lands in the identical kamocrm.com mailbox (no...
Forward kamouniverse.com to kamocrm.com
DNS for the apex and www now point at k1m1's public IP. Add a redirectRegex route on both entrypoints (301, path/query preserved, ACME challenge path excluded o...
The login greeter was counting as a member using the computer
rdp_sessions() collected every Xorg process with a display argument. The image installs lightdm and lightdm-gtk-greeter, so lightdm runs its own Xorg on :0 on e...
A stale resume_offset would hibernate into the wrong place
resume_configured() checked that `resume=` and `resume_offset=` were present on the running kernel's command line. Present, not correct — and the difference is ...
Root could not read a member's idle time, so nothing ever slept
The first computer reported its idle time as 13.9, then 29.2, then 14.4 seconds over three minutes while nobody was touching it. That is not a person — it is a ...
The swap file was smaller than the check that measures it
With the restart landing, the first computer came up with resumeConfigured true and stopped on the next precondition — which turns out to be unsatisfiable by co...
A computer could never restart into being able to sleep
The first real Hosted Computer reached AWAKE and then stayed awake forever. Hibernation reported resumeConfigured: false, IdleWatcher only arms when readiness i...
Start probing at 15s, not 190 — nothing could open on its first wake
A woken computer is only reachable once its Service has endpoints, and that happens when the readiness probe on 3389 first passes. initialDelaySeconds was 180, ...
Rename apps DNS alias to apply
auto-cert's SUBDOMAINS list, the availability health-check URL and the middleware comment now match the apps -> apply subdomain rename in kamo-internal and secu...
Xorg may not start from a remote session, so no computer opened
Guacamole connected, authenticated, and was hung up on: guacd: Security mode: Negotiate (ANY) guacd: Loading keymap "en-us-qwerty" guacd: RDP serve...
Traefik could not see the gateway, and it served at the wrong path
Opening a computer landed on the redirect service's fallback page — "the request reached the fallback page instead of being redirected. Check the Host header." ...
An org's own computers host gets a certificate and a route
computers is a standard alias now — every white-label org is told to point one at Kamo on /setup/dns, the way it points meet and sign — so two things had to fol...
Apply the auto-cert mirror Role on deploy
Applied to the cluster by hand when the gateway certificate was set up; this is the same grant in the deploy step, so a rebuilt cluster does not have to redisco...
Let auto-cert mirror the gateway certificate into this namespace
Listing computers.kamocrm.com for issuance is only half of it. Traefik requires a TLS secret in the IngressRoute's OWN namespace, so the cert is issued in `kamo...
Issue and mirror the certificate for computers.kamocrm.com
Opening a Hosted Computer produced a browser tab that appeared and vanished. The launch link is correct and so is the token: SecurityService builds ************...
A computer that has never restarted can never sleep
prepare-sleep writes the swap file and the resume wiring, then reports readiness — and readiness reads `resume=` from /proc/cmdline, which the kernel fixed at b...
Reinstall grub after virt-resize, or the guest boots to a rescue prompt
virt-resize renumbers partitions. Canonical's image is laid out p14 (BIOS boot), p15 (ESP), p16 (/boot) and then p1 (root) physically last, and virt-resize rewr...
The golden disk has to be RAW, or nothing boots and nothing says so
The job wrote the compressed qcow2 the build produces straight into the golden claim. KubeVirt attaches a Filesystem-PVC disk with <driver type='raw'> — always,...
Only the platform reaches the database's and NATS's admin ports
YugabyteDB and NATS run on k1m1's host network, so every pod in every namespace could open every port they listen on. The client ports authenticate (YSQL :5433 ...
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.