- Shipped
- August 13, 2026 at 2:18 AM UTC
- Author
- Kamo
- Commit
- b0d8862
Renaming the dead CockroachBackupMissing alert in the previous commit made it fire immediately: no universe backup had succeeded since the 2026-08-09 migration. Two independent faults, both masked by that alert matching a job name that no longer existed. 1. `mc: command not found`. The CockroachDB job used the server's native BACKUP ... INTO 's3://', so it needed no S3 client in the image. The ysql_dump rewrite kept calling `mc` from the yugabyte image, which does not ship it. Split into an initContainer (yugabyte/ysql_dump) writing to a shared emptyDir and an upload container (minio/mc). initContainer rather than a sidecar so the uploader cannot race the dump and copy a partial file. 2. S3_ENDPOINT in the universe-backup-s3 Secret was http://10.8.1.1:9000 — a WireGuard address. That overlay is dead between k1m1 and k3m1, so the upload blackholed on a TCP timeout. Repointed at **************** the documented rule for reaching host-replicated MinIO from any node. Hardening, because both faults produced a job that looked healthier than it was: the dump writes to .part and renames only on success (a truncated -Fc dump restores silently up to the truncation point), the uploader exits 1 if no dump file exists instead of reporting success with nothing to send, and it runs `mc stat` afterwards to prove the object landed. Verified end to end: 314 KiB dump uploaded, mc stat confirms the object and site replication reports COMPLETED.