Rename apps DNS alias to apply
auto-cert's SUBDOMAINS list, the availability health-check URL and the middleware comment now match the apps -> apply subdomain rename in kamo-internal and secu...
universe backup: pull mc from quay.io, docker.io/minio/mc is gone [skip ci]
MinIO withdrew its Docker Hub images, so the backup's upload container sat in ImagePullBackOff from 2026-09-12. With concurrencyPolicy Forbid that one stuck job...
node-config: registry prune must not garbage-collect with --delete-untagged [skip ci]
The nightly prune on k1m1 lived only on the host. Its garbage-collect ran with --delete-untagged, which deletes the per-platform manifests an OCI image index re...
auto-cert: stop provisioning www.stack.loans
The ratestack estate has been scaled to 0 since 2026-08-04, so nothing answers HTTP-01 for stack.loans. Its Certificates expired on 2026-08-02 and their orders ...
Disable node-exporter's xfs collector [skip ci]
It cannot parse this kernel's /proc/fs/xfs/stat ("xpc") and logged an error on every scrape, ~240/h per node. Filesystem usage is unaffected (filesystem collect...
Kamo_app owns the ocr_* tables [skip ci]
OCRService (web and worker) alters its own ocr_* tables at startup and refuses to start without ownership; after the switch to kamo_app the worker crash-looped ...
Non-superuser kamo_app role for all application workloads [skip ci]
Services connected as the superuser kamo. Add an idempotent script that creates kamo_app (no superuser, not a member of kamo so it cannot SET ROLE back) with gr...
Scrape YugabyteDB tserver/YSQL and SecurityService pool metrics [skip ci]
Connection sizing had no data behind it: nothing scraped the database or the Hikari pools. Add a curated server-level tserver job (the full endpoint is ~3.25M l...
node-config(k3m1-ext): eno1 on DHCP as k3m1-ext, used only by traffic from its own address
k3m1's third cable is the port the shared bulk relay (postfix-orgs, 47.181.8.86) sends through. DHCP gives it an address and nothing else, so the main table is ...
auto-cert: let the junk-cert cleanup actually see the junk it was written for
cleanup_orphan_certificates() listed with '-l auto-cert.kamo/fqdn'. That label is newer than the scanner certs the function exists to remove, so the selector ex...
auto-cert: issue an org's certificates as a batch, not one every 90 seconds
A new organization needs a certificate for all eleven of its hosts, and each one is a separate cert-manager Certificate. The loop waited for each host to become...
Prefer brotli on the theme host, and correct the previous rationale
Two things, one of them a correction to the commit before this. The previous commit justified `encodings: [br, gzip]` with a probe against api.kamocrm.com. Tha...
Prefer brotli, stop compressing what is already compressed
The shared `compress` middleware was `compress: {}`, which is three separate defaults and none of them is the one we want. Traefik picks gzip whenever the clie...
Load with explicit column lists -- positional COPY corrupts silently
COPY t FROM STDIN with no column list maps CSV fields to the table's PHYSICAL column order. The CSVs come from COPY (SELECT *) on CockroachDB, and Hibernate cre...
Load with explicit column lists -- positional COPY corrupts silently
COPY t FROM STDIN with no column list maps CSV fields to the table's PHYSICAL column order. The CSVs come from COPY (SELECT *) on CockroachDB, and Hibernate cre...
Schema parity with CockroachDB -- 714 tables, 8706 columns
The drift diff earned its place. KamoInitializerService builds the schema from JPA entities and produced 599 tables; production carries 714. The 115 difference ...
Schema parity with CockroachDB -- 714 tables, 8706 columns
The drift diff earned its place. KamoInitializerService builds the schema from JPA entities and produced 599 tables; production carries 714. The 115 difference ...
Full CockroachDB->YSQL translator for the universe migrations
All 15 migrations now apply cleanly to YugabyteDB: 111 tables, matching CockroachDB exactly. Eight distinct CockroachDB-only constructs had to be handled, found...
Full CockroachDB->YSQL translator for the universe migrations
All 15 migrations now apply cleanly to YugabyteDB: 111 tables, matching CockroachDB exactly. Eight distinct CockroachDB-only constructs had to be handled, found...
CSV export via COPY TO STDOUT, plus a CSV-aware verifier
Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...
CSV export via COPY TO STDOUT, plus a CSV-aware verifier
Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...
Make every sweep schema-aware; kamo has 714 tables, not 680
Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...
Make every sweep schema-aware; kamo has 714 tables, not 680
Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...
Baseline capture for CockroachDB to YugabyteDB parity checking
Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...
Baseline capture for CockroachDB to YugabyteDB parity checking
Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...
Meet-secrets **************** for jibri recording ingest
Lets jibri finalize.sh authenticate its recording upload to MediaService; matches the cluster *** value MediaService validates against.
Provision R1 MEDIA JetStream stream via idempotent bootstrap Job
Media/lead-feed events (media.feed.*, media.post.*) had no JetStream stream, so publishes returned '503 No Responders Available For Request' (chat.> is covered ...
node-config(k1m1-runner): record data.forgejo.org IPv4 pin + capacity=6
Documents the live on-node change: data.forgejo.org is IPv6-only and k1m1 has no IPv6 route, so job containers hung ~133s and actions/checkout failed. Pin its I...
Consolidate klusterservices onto single k1m1+k3m1 cluster
- NATS: standalone single pod pinned k1m1 (drop k2m1 + generic cluster variants) - CockroachDB: pin k1m1, single-node --join - MySQL: single instance moved in f...
Upgrade v3.0 -> v3.3 (fix intermittent WebSocket disconnect)
v3.0.0 intermittently completes the WS 101 upgrade then closes the socket before any frame flows — the recurring 'disconnect on load' in the universe game clien...
node-config: document k1m1 forgejo-runner config (--add-host for submodule DNS)
Job containers on k1m1 fall back to public DNS, so the fleet-swept http://k1m1:3000 .gitmodules URLs fail to clone there. Pin k1m1 into job containers via conta...
node-config: document k1m1 docker daemon insecure-registries
The k1m1 forgejo-runner's builds push to the HTTP Forgejo registry; the daemon.json shipped with only the :5000 entries, so every build job landing on k1m1 fail...
Join shared two-node cluster with k3m1 over LAN; account isolation
Retire WireGuard routes (10.8.x.x) for LAN routes (192.168.4.22 <-> 192.168.4.36 :6222) with creds embedded in the route URLs. Rename cluster to kamo-ratestack-...
node-config: raise k1m1 kubelet max-pods 110 -> 200
Default RKE2 max-pods=110 was being hit, blocking kamolos rolling-update deploys (maxSurge:1,maxUnavailable:0 needs slot to create new pod before killing old; d...
Update Traefik routes: themes.* assets now only load from k1m1 minio instead of balancing across k0m1, k1m1, and k2m1
Use VPN addresses for k0m1/k2m1 minio-public backends
- Endpoints: k0m1 10.8.0.1, k2m1 10.8.2.1 (VPN); k1m1 stays local - IngressRoute host match: 100.64.x.x -> 10.8.0.1, 10.8.1.1, 10.8.2.1
MinIO secret creation from workflow (MinIO is managed manually)
Update IP addresses from Tailscale (100.64.x.x) to WireGuard VPN (10.8.x.x) for NATS, CockroachDB, and CI/CD workflows
Add complete Kamo logo SVG with all 14 paths
Embed full SVG content directly instead of partial paths
Use CSS-only approach for Kamo logo
Removed broken SVG file mount, using CSS background-image to load external logo directly on #logo element
Fix logo.svg to reference external Kamo logo URL
Use SVG image element to embed external logo instead of complex paths with broken transforms
Replace Roundcube logo.svg with Kamo logo SVG
Mount custom logo.svg to **************** to replace the Roundcube watermark/background logo
Add custom CSS file to override Roundcube watermark logo
- Created custom.css with watermark override - Mount CSS file to /var/www/html/custom.css - Configure Roundcube to load additional_stylesheets
Replace all Roundcube logos with Kamo logo including watermark
Updated skin_logo config to array format to cover: - Main header logo - Small collapsed header logo - Inbox watermark/background logo
Add mail subdomain to auto-cert SUBDOMAINS list
Enables automatic SSL certificate creation for mail.kamocrm.com
Expose Postfix ports 25 and 587 via hostPort for inbound mail
Required for receiving external SMTP connections on K1M1
Configure Roundcube to use Mailgun SMTP directly
Bypassing local Postfix for outgoing mail, connecting directly to smtp.mailgun.org with TLS on port 587
Switch Postfix to Alpine-based deployment with MySQL support
Similar to Dovecot, using Alpine 3.18 with postfix-mysql package installed at runtime for reliable MySQL virtual maps support
Fix postfix command to use apt-get instead of apk
boky/postfix is Debian-based, not Alpine
Add postfix-mysql support for MySQL virtual maps
Installed postfix-mysql package at container startup to enable MySQL dictionary lookups for virtual domains/users/aliases
Like what you see shipping?
All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.