auto-cert: let the junk-cert cleanup actually see the junk it was written for
cleanup_orphan_certificates() listed with '-l auto-cert.kamo/fqdn'. That label is newer than the scanner certs the function exists to remove, so the selector ex...
auto-cert: issue an org's certificates as a batch, not one every 90 seconds
A new organization needs a certificate for all eleven of its hosts, and each one is a separate cert-manager Certificate. The loop waited for each host to become...
Prefer brotli on the theme host, and correct the previous rationale
Two things, one of them a correction to the commit before this. The previous commit justified `encodings: [br, gzip]` with a probe against api.kamocrm.com. Tha...
Prefer brotli, stop compressing what is already compressed
The shared `compress` middleware was `compress: {}`, which is three separate defaults and none of them is the one we want. Traefik picks gzip whenever the clie...
Load with explicit column lists -- positional COPY corrupts silently
COPY t FROM STDIN with no column list maps CSV fields to the table's PHYSICAL column order. The CSVs come from COPY (SELECT *) on CockroachDB, and Hibernate cre...
Load with explicit column lists -- positional COPY corrupts silently
COPY t FROM STDIN with no column list maps CSV fields to the table's PHYSICAL column order. The CSVs come from COPY (SELECT *) on CockroachDB, and Hibernate cre...
Schema parity with CockroachDB -- 714 tables, 8706 columns
The drift diff earned its place. KamoInitializerService builds the schema from JPA entities and produced 599 tables; production carries 714. The 115 difference ...
Schema parity with CockroachDB -- 714 tables, 8706 columns
The drift diff earned its place. KamoInitializerService builds the schema from JPA entities and produced 599 tables; production carries 714. The 115 difference ...
Full CockroachDB->YSQL translator for the universe migrations
All 15 migrations now apply cleanly to YugabyteDB: 111 tables, matching CockroachDB exactly. Eight distinct CockroachDB-only constructs had to be handled, found...
Full CockroachDB->YSQL translator for the universe migrations
All 15 migrations now apply cleanly to YugabyteDB: 111 tables, matching CockroachDB exactly. Eight distinct CockroachDB-only constructs had to be handled, found...
CSV export via COPY TO STDOUT, plus a CSV-aware verifier
Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...
CSV export via COPY TO STDOUT, plus a CSV-aware verifier
Transport is COPY ... TO STDOUT WITH (FORMAT CSV), not `cockroach sql --format=csv`. The latter renders SQL NULL as the literal string "NULL", indistinguishable...
Make every sweep schema-aware; kamo has 714 tables, not 680
Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...
Make every sweep schema-aware; kamo has 714 tables, not 680
Cross-checking SHOW BACKUP (714 objects) against a table_schema='public' inventory (680) exposed 34 tables living in five other schemas: email_metadata, email_p...
Baseline capture for CockroachDB to YugabyteDB parity checking
Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...
Baseline capture for CockroachDB to YugabyteDB parity checking
Two passes: cheap batched count(*), then an order-independent checksum. The checksum sums per-row md5 hashes as integers rather than the obvious md5(string_agg...
Meet-secrets **************** for jibri recording ingest
Lets jibri finalize.sh authenticate its recording upload to MediaService; matches the cluster *** value MediaService validates against.
Provision R1 MEDIA JetStream stream via idempotent bootstrap Job
Media/lead-feed events (media.feed.*, media.post.*) had no JetStream stream, so publishes returned '503 No Responders Available For Request' (chat.> is covered ...
node-config(k1m1-runner): record data.forgejo.org IPv4 pin + capacity=6
Documents the live on-node change: data.forgejo.org is IPv6-only and k1m1 has no IPv6 route, so job containers hung ~133s and actions/checkout failed. Pin its I...
Consolidate klusterservices onto single k1m1+k3m1 cluster
- NATS: standalone single pod pinned k1m1 (drop k2m1 + generic cluster variants) - CockroachDB: pin k1m1, single-node --join - MySQL: single instance moved in f...
Upgrade v3.0 -> v3.3 (fix intermittent WebSocket disconnect)
v3.0.0 intermittently completes the WS 101 upgrade then closes the socket before any frame flows — the recurring 'disconnect on load' in the universe game clien...
node-config: document k1m1 forgejo-runner config (--add-host for submodule DNS)
Job containers on k1m1 fall back to public DNS, so the fleet-swept http://k1m1:3000 .gitmodules URLs fail to clone there. Pin k1m1 into job containers via conta...
node-config: document k1m1 docker daemon insecure-registries
The k1m1 forgejo-runner's builds push to the HTTP Forgejo registry; the daemon.json shipped with only the :5000 entries, so every build job landing on k1m1 fail...
Join shared two-node cluster with k3m1 over LAN; account isolation
Retire WireGuard routes (10.8.x.x) for LAN routes (192.168.4.22 <-> 192.168.4.36 :6222) with creds embedded in the route URLs. Rename cluster to kamo-ratestack-...
node-config: raise k1m1 kubelet max-pods 110 -> 200
Default RKE2 max-pods=110 was being hit, blocking kamolos rolling-update deploys (maxSurge:1,maxUnavailable:0 needs slot to create new pod before killing old; d...
Update Traefik routes: themes.* assets now only load from k1m1 minio instead of balancing across k0m1, k1m1, and k2m1
Use VPN addresses for k0m1/k2m1 minio-public backends
- Endpoints: k0m1 10.8.0.1, k2m1 10.8.2.1 (VPN); k1m1 stays local - IngressRoute host match: 100.64.x.x -> 10.8.0.1, 10.8.1.1, 10.8.2.1
MinIO secret creation from workflow (MinIO is managed manually)
Update IP addresses from Tailscale (100.64.x.x) to WireGuard VPN (10.8.x.x) for NATS, CockroachDB, and CI/CD workflows
Add complete Kamo logo SVG with all 14 paths
Embed full SVG content directly instead of partial paths
Use CSS-only approach for Kamo logo
Removed broken SVG file mount, using CSS background-image to load external logo directly on #logo element
Fix logo.svg to reference external Kamo logo URL
Use SVG image element to embed external logo instead of complex paths with broken transforms
Replace Roundcube logo.svg with Kamo logo SVG
Mount custom logo.svg to **************** to replace the Roundcube watermark/background logo
Add custom CSS file to override Roundcube watermark logo
- Created custom.css with watermark override - Mount CSS file to /var/www/html/custom.css - Configure Roundcube to load additional_stylesheets
Replace all Roundcube logos with Kamo logo including watermark
Updated skin_logo config to array format to cover: - Main header logo - Small collapsed header logo - Inbox watermark/background logo
Add mail subdomain to auto-cert SUBDOMAINS list
Enables automatic SSL certificate creation for mail.kamocrm.com
Expose Postfix ports 25 and 587 via hostPort for inbound mail
Required for receiving external SMTP connections on K1M1
Configure Roundcube to use Mailgun SMTP directly
Bypassing local Postfix for outgoing mail, connecting directly to smtp.mailgun.org with TLS on port 587
Switch Postfix to Alpine-based deployment with MySQL support
Similar to Dovecot, using Alpine 3.18 with postfix-mysql package installed at runtime for reliable MySQL virtual maps support
Fix postfix command to use apt-get instead of apk
boky/postfix is Debian-based, not Alpine
Add postfix-mysql support for MySQL virtual maps
Installed postfix-mysql package at container startup to enable MySQL dictionary lookups for virtual domains/users/aliases
Add auth debugging and ensure MySQL driver packages for Dovecot
- Enable auth_debug, auth_debug_passwords, auth_verbose - Add dovecot-mysql package for proper SQL support - Add mariadb-connector-c for MySQL protocol compatib...
Add MySQL auth fix job for MariaDB client compatibility
Changes kamo user from caching_sha2_password to mysql_native_password Required for Dovecot (Alpine/MariaDB) to connect to MySQL 8
Replace Roundcube logo with Kamo logo
Added skin_logo config pointing to **************** Updated deployment annotation to trigger pod restart
Fix MySQL auth: change kamo user to mysql_native_password
Alpine's MariaDB client doesn't support caching_sha2_password plugin This prevents Dovecot from connecting to MySQL for authentication
Force Roundcube pod restart to pick up ConfigMap change
Added annotation to trigger pod recreation since ConfigMap with subPath doesn't auto-update in running pods
Fix Roundcube redirect loop - disable force_https
Traefik handles TLS termination, so internal traffic is HTTP Setting force_https=true caused Roundcube to redirect to HTTPS internally creating an infinite redi...
Reference kamocrm.com wildcard cert in mail IngressRoute
The certificate secret is in kamo namespace, copied to mail namespace tls.secretName explicitly references wildcard-kamocrm-com
Fix mail IngressRoute to use auto-cert service
Changed tls from certResolver: letsencrypt to tls: {} This allows the auto-cert service to automatically generate certs when mail.kamocrm.com is accessed
Fix Roundcube readiness probe - use TCP socket instead of HTTP
Roundcube redirects / to login page (302), failing HTTP readiness probe TCP socket check on port 80 works correctly
Like what you see shipping?
Every one of these updates lands in your workspace automatically. Start free and watch it grow week after week.