Live Change Log

A computer provisioned, came up, and panicked on its next boot: Run /init as init process Failed to execute /init (error -2) Kernel panic - not syn...

A brand new computer provisioned, came up, reported AWAKE, and ninety seconds later panicked: Kernel panic - not syncing: No working init found. I caused ...

kamo-internal 3e3c3e7a mounts **************** (SESSION_HANDLE_SECRET), which seals the per-tab session handles that replace the *** id in page script. Created ...

Thirty seconds after the first real resume, from the machine's own log: kamo-hc-agent: idle 13202s >= 1800s — going to sleep PM: hibernation: hibernati...

www.kamouniverse.com was served by the kamocrm.com marketing app instead of redirecting: kamo-marketing-route's HostRegexp(\`^www[.].+\$\`) at priority 24 match...

Adds kamouniverse.com to postfix's virtual_alias_domains with a local-part-preserving regexp so any address on it lands in the identical kamocrm.com mailbox (no...

DNS for the apex and www now point at k1m1's public IP. Add a redirectRegex route on both entrypoints (301, path/query preserved, ACME challenge path excluded o...

rdp_sessions() collected every Xorg process with a display argument. The image installs lightdm and lightdm-gtk-greeter, so lightdm runs its own Xorg on :0 on e...

resume_configured() checked that `resume=` and `resume_offset=` were present on the running kernel's command line. Present, not correct — and the difference is ...

The first computer reported its idle time as 13.9, then 29.2, then 14.4 seconds over three minutes while nobody was touching it. That is not a person — it is a ...

With the restart landing, the first computer came up with resumeConfigured true and stopped on the next precondition — which turns out to be unsatisfiable by co...

The first real Hosted Computer reached AWAKE and then stayed awake forever. Hibernation reported resumeConfigured: false, IdleWatcher only arms when readiness i...

A woken computer is only reachable once its Service has endpoints, and that happens when the readiness probe on 3389 first passes. initialDelaySeconds was 180, ...

auto-cert's SUBDOMAINS list, the availability health-check URL and the middleware comment now match the apps -> apply subdomain rename in kamo-internal and secu...

Guacamole connected, authenticated, and was hung up on: guacd: Security mode: Negotiate (ANY) guacd: Loading keymap "en-us-qwerty" guacd: RDP serve...

Opening a computer landed on the redirect service's fallback page — "the request reached the fallback page instead of being redirected. Check the Host header." ...

computers is a standard alias now — every white-label org is told to point one at Kamo on /setup/dns, the way it points meet and sign — so two things had to fol...

Applied to the cluster by hand when the gateway certificate was set up; this is the same grant in the deploy step, so a rebuilt cluster does not have to redisco...

Listing computers.kamocrm.com for issuance is only half of it. Traefik requires a TLS secret in the IngressRoute's OWN namespace, so the cert is issued in `kamo...

Opening a Hosted Computer produced a browser tab that appeared and vanished. The launch link is correct and so is the token: SecurityService builds ************...

prepare-sleep writes the swap file and the resume wiring, then reports readiness — and readiness reads `resume=` from /proc/cmdline, which the kernel fixed at b...

virt-resize renumbers partitions. Canonical's image is laid out p14 (BIOS boot), p15 (ESP), p16 (/boot) and then p1 (root) physically last, and virt-resize rewr...

The job wrote the compressed qcow2 the build produces straight into the golden claim. KubeVirt attaches a Filesystem-PVC disk with <driver type='raw'> — always,...

YugabyteDB and NATS run on k1m1's host network, so every pod in every namespace could open every port they listen on. The client ports authenticate (YSQL :5433 ...

Second half of 926986d. Every client now presents the kamo_svc login: the Java services through the shared library's NatsConfig (NATS_USERNAME / NATS_PASSWORD f...

Nothing had ever been provisioned, so every grant past the namespace was untested. Each of these refused at exactly the point the previous fix unblocked. `patc...

NATS runs on the host network of k1m1 and listens on 0.0.0.0, and its config mapped every credential-less connection to the KAMO account (no_auth_user: anon). S...

The `kamo` Redis holds every *** session and OTK and has no password. Nothing restricted who could connect to it: from a pod in the `desktop` namespace (where t...

The build had never been run. Running it found one bug per attempt, and the first three would each have produced a plausible-looking image with something missin...

build-golden-image.sh said "Run by CI (see ../../.forgejo/workflows/)". No such workflow was ever written. It needs libguestfs, qemu-img and virt-customize, non...

service-template.yaml, cloud-init.userdata.yaml and cloud-init.networkdata.yaml. They were written into a second worktree and committed from neither — the earli...

The namespace, the single-replica StorageClass, the clusterwide seal, computeservice's RBAC and the tenant gateway. Guarded on the hosted-computers/ directory c...

A member's own Linux computer, running as a KubeVirt VM in its organization's namespace and opened from the browser. Separate from desktop/ in every way that ma...

The marketing reel was recut (new closing logo shot) and uploaded as public/kamo14.mp4. It gets a new name instead of replacing kamo13 in place because the ladd...

Campaigns now send Reply-To: NoReply@<their From's domain> by default, and those replies reach postfix-k3m1-inbound. The refusal said "Sender address rejected: ...

MinIO withdrew its Docker Hub images, so the backup's upload container sat in ImagePullBackOff from 2026-09-12. With concurrencyPolicy Forbid that one stuck job...

The nightly prune on k1m1 lived only on the host. Its garbage-collect ran with --delete-untagged, which deletes the per-platform manifests an OCI image index re...

The ratestack estate has been scaled to 0 since 2026-08-04, so nothing answers HTTP-01 for stack.loans. Its Certificates expired on 2026-08-02 and their orders ...

It cannot parse this kernel's /proc/fs/xfs/stat ("xpc") and logged an error on every scrape, ~240/h per node. Filesystem usage is unaffected (filesystem collect...

OCRService (web and worker) alters its own ocr_* tables at startup and refuses to start without ownership; after the switch to kamo_app the worker crash-looped ...

Services connected as the superuser kamo. Add an idempotent script that creates kamo_app (no superuser, not a member of kamo so it cannot SET ROLE back) with gr...

A campaign sent through the shared relay has its From at the organization's sending domain as its envelope sender, and that domain is a CNAME to contact.kamocrm...

Connection sizing had no data behind it: nothing scraped the database or the Hikari pools. Add a curated server-level tserver job (the full endpoint is ~3.25M l...

contact.kamocrm.com is the shared org relay's name (47.181.8.86's PTR, MX contact.kamocrm.com), and its MX is 47.181.8.86 → k3m1, where postfix-k3m1-inbound ans...

KamoCRM's own campaign mail keeps leaving through postfix-bulk (47.181.8.87) and regular mail through postfix (47.181.8.84). Every other organization's campaign...

k3m1's third cable is the port the shared bulk relay (postfix-orgs, 47.181.8.86) sends through. DHCP gives it an address and nothing else, so the main table is ...

k3m1 had no prune job, and kubelet image GC waits for 85% disk, so every deploy's SHA-tagged image piled up: 3,368 images (539G), 86 in use. Add a daily k3m1-im...

Gmail counts a campaign message toward its Feedback Loop only when the Feedback-ID header is covered by the sending domain's DKIM signature. local.d/dkim.conf k...

47.181.8.87's reverse DNS is mail.stack.loans and the router forwards its port 25 to k3m1, but nothing there accepted mail, so a provider verifying who runs the...

Spam-complaint feedback-loop reports (ARF, RFC 5965) and abuse reports sent to fbl@ and abuse@hello.kamocrm.com are rewritten to complaints-mail.invalid and han...