Add LibreTranslate permission init container, scale Bergamot to 0
LibreTranslate was crashing with PermissionError on packages subdir. Added busybox init container to mkdir + chmod 777 the mounted hostPath before the main cont...
JVB_ADVERTISE_IPS must be public IP, not Wireguard IP
Was 10.8.1.1 (Wireguard private) — guests' browsers can't reach this. Changed to 75.83.113.11 (public IP). This is why guests kept getting ICE restarts every ~1...
Disable mod_token_verification via custom plugin override to stop guest kicks
The MUC token_verification module re-validates tokens and demotes guests from PARTICIPANT to VISITOR after ~60 seconds, causing Jicofo to kick them. The module ...
Add guest domain to token_verification_allowlist, increase smacks timeout
Guests authenticate via Meet-anonymous on guest.meet.Meet but the MUC token_verification module was re-checking them and demoting their role from PARTICIPANT to...
Disable Jicofo auth module to prevent 1-minute participant kick
Jicofo's AbstractAuthAuthority has a hardcoded 1-minute auth session lifetime that ignores the config file setting. After 60 seconds, Prosody demotes participan...
Use room names (not full JIDs) for all brewery MUC configs
Both Jicofo and JVB Docker images append @XMPP_INTERNAL_MUC_DOMAIN to the brewery room name automatically. Using full JIDs like **************** caused double-d...
Use TCP probe on port 9090 for JVB health checks instead of HTTP on 8080
JVB's REST API port 8080 conflicts with CockroachDB's admin UI on the same hostNetwork. The JVB_OHTTP_SERVER_PORT env var is not recognized by this JVB image ve...
Resolve JVB crash loop — wrong MUC JID + port 8080 conflict
Two issues causing JVB to crash in a loop for 43+ hours: 1. JVB_BREWERY_MUC was **************** (full JID) but JVB appends XMPP_INTERNAL_MUC_DOMAIN automat...
JVB_BREWERY_MUC should be room name only, not full JID
Was **************** which caused JVB to construct a double-domain JID **************** with an @ in the domain part — crashing JVB with XmppStringprepException...
Add dnsPolicy ClusterFirstWithHostNet to JVB StatefulSets so hostNetwork pods can resolve cluster DNS
Reference *** instead of non-existent turn-secret in Prosody deployment
Enable auto-truncate on embedding model to prevent 413 errors
Add --auto-truncate flag so inputs exceeding the 256 token limit are silently truncated instead of rejected with a validation error.
Load embedding model from local hostPath to prevent HuggingFace download failures
The embedding-model pod was in CrashLoopBackOff because it couldn't download the model from HuggingFace on startup (network/DNS issue). Mount the model from a h...
Auto-cert service not renewing expired SSL certificates
The auto-cert service was importing expired certs from shared storage and skipping Certificate CR creation, so cert-manager couldn't manage renewals. Also fixed...
Fix kubectl not found in pod, add docs alias, proactive SUBDOMAINS
- Mount /usr/local/bin/kubectl from host via hostPath instead of failing apt-get install (kubectl not in default Debian repos) - Add "docs" to SUBDOMAINS list...
Remove forced WebSocket headers from media middleware
Traefik v3 handles WebSocket Upgrade/Connection headers natively. Remove forced headers that broke non-WebSocket API requests.
Add system account to all NATS configs
Add $SYS account with admin credentials to k0m1, k1m1, and k2m1 NATS configs. Enables nats CLI cluster management operations (peer removal, step-down, etc.) for...
Resilient probes so cluster stays up when one node is offline
- Liveness: tcpSocket on 4222 (do not depend on JetStream meta leader) - Readiness: initialDelay 90s, period 15s, failureThreshold 10 so k0m1+k1m1 can form qu...
Skip CockroachDB init job if any pods exist - cluster already initialized
Improve CockroachDB init check to wait for pods and verify cluster state
Prevent CockroachDB init job from running when cluster is already initialized
Update CockroachDB probes, HTTP redirect, and MinIO service endpoints
Add CORS middleware to theme routes to allow cross-origin access from all websites
Add hostPort to Traefik deployment for direct port access
- Use hostPort 443 for websecure entrypoint to bypass NodePort - Allows router to forward port 443 directly to Traefik - Also set hostPort for web (30080) and a...
Set External Traffic Policy to Local for Traefik NodePort service
- Allows proper routing of external traffic from router - Preserves source IP addresses for better logging
Update CockroachDB StatefulSet to use dynamic VPN IP based on node hostname
- Fix advertise-addr to use VPN IP (10.8.1.1 for k1m1, 10.8.2.1 for k2m1) - Update join parameter to use VPN IPs instead of hostnames - Update node certificate ...
Add node certificates to CockroachDB secret creation
- Add node.crt and node.key to cockroachdb directory - Update CI/CD workflow to include node certificates when creating *** secret - Fixes CockroachDB CrashLoop...
Handle BackoffLimitExceeded by deleting failed job and retrying
- When job fails with BackoffLimitExceeded and pods are cleaned up, delete and recreate job - This allows us to get fresh logs from a new pod - Catches 'already...
Handle BackoffLimitExceeded by deleting and retrying init job
- When job fails with BackoffLimitExceeded and pods are cleaned up, delete and recreate job - This allows the job to retry and succeed if cluster is already ini...
Improve CockroachDB init job error handling when pods are cleaned up
- Check previous pod logs if current pods are not found - Test cluster connectivity to verify if already initialized - Better handling of failed jobs with clean...
Resolve Traefik, NATS, and CockroachDB deployment issues
- Remove hostPort from Traefik deployment (conflicts with NodePort service) - Fix NATS deployment to only deploy server-specific statefulsets (nats-k1m1, nats-k...
Improve CockroachDB init job error handling - check pod status and logs after timeout
Remove heredoc whitespace issue - use echo commands and ConfigMap
The heredocs had leading whitespace from YAML indentation causing parse errors - Use echo commands to create dovecot.conf without leading whitespace - Mount aut...
Use driver=sql with sql_driver=mysql and inline connect
Based on official Dovecot 2.4 examples found online: - Named sections: passdb sql { and userdb sql { - driver = sql (not driver = mysql) - sql_driver = mysql (s...
Use driver=mysql (not driver=sql) with inline connect in Dovecot 2.4
Based on official Dovecot 2.4 examples: - Named sections: passdb sql { and userdb sql { - Use driver = mysql (not driver = sql) - Inline connect, default_pass_s...
Dovecot 2.4 requires SQL settings inline, not args parameter
- Dovecot 2.4 removed 'args' parameter for passdb/userdb - SQL connection settings must be specified directly in passdb/userdb blocks - Put driver=mysql, connec...
Disable idle timeout for WebSocket connections
- Added idleTimeout=0 to websecure entrypoint - Prevents Traefik from closing WebSocket connections due to inactivity - Fixes SockJS WebSocket timeout issues
Split port ranges to avoid conflicts
- Coturn relay: 52000-65535 (13,536 ports) - Avoids: Janus RTP (10000-49151) and WireGuard (51820-51821) - No overlap between services
Adjust relay port range to avoid WireGuard/VPN ports
- Changed max-port from 65535 to 51800 to avoid conflicts - Avoids WireGuard on UDP 51820 and other VPN services on 51821 - Provides 2,649 ports (49152-51800) f...
Resolve CrashLoopBackOff by fixing probes and config
- Replace netstat-based health probes with TCP socket checks (netstat not available in coturn/coturn:latest Alpine image) - Remove conflicting no-stdout-log dir...
Add Connection: Upgrade header to WebSocket upgrade middleware
- Add Connection: Upgrade header along with Upgrade: websocket - Both headers are required for proper WebSocket upgrade handshake - This ensures HTTP/2 Extended...
Skip CockroachDB init if job failed (cluster already initialized)
Add webhook wait and sleep to ensure cert-manager is fully ready
Je, unaona nini kuhusu usafiri?
Kila moja ya hizi updates ardhi katika nafasi yako ya kazi moja kwa moja. Kuanza bure na kuangalia kukua wiki baada ya wiki.