- Ya
- 4 Septemba 2026, 20:20 UTC
- Mwandishi
- Kamo
- Ahadi ya
- 6ef1e59
MediaService has only ever run a single replica, so a deploy of it was a total chat outage and an OOM was the same. It could not run two, for reasons that were all in the code: 1. THE STOMP BROKER IS IN-HEAP. enableSimpleBroker means convertAndSend reaches only the WebSockets connected to the pod making the call. Thirteen relay controllers already handled this correctly by subscribing to NATS with a plain dispatcher (no queue group, so every pod receives every message) and re-publishing locally. Thirteen other call sites did not — typing indicators, presence, unread badges, reply grants, WebRTC offer/answer/ICE, incoming-chat notifications — and each would have delivered to roughly half its intended audience. StompFanout generalises the pattern those relays already use: publish {destination, payload} on one core NATS subject, every pod relays it into its own broker. Core NATS, not JetStream, because these are live-moment events with no value replayed and no stream covering the subject. Because core NATS also delivers to the publisher, send() does NOT write locally as well — that would deliver twice here. With NATS down it falls back to a local send, which is what the platform did before. 2. THE PER-CONVERSATION CONSUMER WAS AN EXCLUSIVE DURABLE. ChatSessionSubscriptionManager named its JetStream durable after the chat session GUID, and a durable push consumer admits exactly one subscriber. The second pod's bind was refused [SUB-90012] and every member whose socket landed there received nothing — no messages, no read receipts, no member-added events, no error. Now an ephemeral consumer: no name to collide over, one per pod, reaped by the server. The retry this removes invented a fresh durable name on collision. It worked, and it leaked a permanent server-side consumer per collision that nothing ever deleted. 3. FIVE MORE FIXED-NAME DURABLES, needing opposite treatment depending on the job. The conversion and VOIP STOMP relays went through JetStream rather than a core dispatcher, so they had the same exclusivity bug — now ephemeral, since every pod must receive. The chat-email notifier, social inbound consumer and marketing conversion consumer must run exactly ONCE, so they keep their durable and join a deliver group, which also means a survivor picks up the work when a pod dies. Sticky sessions on the media route are required, not optional: these STOMP clients use SockJS, whose xhr-streaming/xhr-polling fallback is several HTTP requests standing in for one connection against session state that lives in one pod's memory. Round-robin breaks it. See ingressroute.yaml. replicas 1 -> 2. 367 tests pass.