- Shipped
- September 7, 2026 at 10:37 PM UTC
- Author
- Kamo
- Commit
- 67b2a62
The mail relay was one of four NATS subscriptions on a plain durable consumer. A plain durable admits exactly ONE subscriber, so at two replicas the second is refused with [SUB-90012] and simply does not listen. The remaining three: MessageIndexer bound on one pod; the other retried forty times and logged "push indexing is disabled for this pod". Indexing worked, by luck, and nothing would have taken it over if that pod had died. SyncEventListener lost a start-up race against the outgoing pod during a rolling deploy — and it has no retry at all, so BOTH pods ended up refused. Nothing consumed daemon.sync.complete; a contact sync would finish and the member's panel sat on "syncing" forever, behind one ERROR line at boot. Verified on the broker: the consumer exists, bound by nobody. Both are WORK — rows written, a search index updated, a phone woken — so both become queue consumers. The deliver group is what turns one durable into a queue the pods share, and what lets a survivor pick the work up. Telling the member is the opposite problem. SyncEventListener is now a queue, so exactly one pod handles the event, and it is very unlikely to be the pod holding that member's WebSocket — convertAndSend's broker is in-heap. Sending from there delivered the status to whoever happened to be connected to the winning pod and to nobody else. So the winner computes the status once and publishes it, and an ephemeral relay on every pod turns it into that pod's own sessions. The wire format the browser sees is unchanged. Two details worth keeping: The sync consumer is RENAMED. The old one still exists on the stream as a plain durable, and a plain durable cannot be joined as a queue. NatsMessageService normally repairs that, but its reconcile looks on this service's own stream and daemon.sync.complete lives on DAEMON_SYNC, not EMAIL_NOTIFICATIONS — so it finds nothing and deletes nothing. Renaming is what gets a correctly shaped consumer without a hand-run purge on the broker. Nothing is lost: DeliverPolicy.New means a fresh consumer starts where the old one stood. The new relay is idempotent and retried whenever a member subscribes. A one-shot @PostConstruct meeting an unavailable broker is exactly how SyncEventListener came to be consuming nothing, and real traffic is a better trigger than a scheduler. An arch ratchet now fails the build if any subscription goes back to the plain form, since with one replica the wrong shape behaves perfectly and only a second pod reveals it.