KamoCRM

V1.13.10, a room reaper and leak watchdog, metrics and alerts, no roll under live calls

FixKlusterServices
Shipped
October 28, 2026 at 6:34 AM UTC
Author
Kamo
Commit
c873407

livekit-server v1.13.7 kept 202 closed participants of 176 closed rooms alive (their publisher transport's StreamAllocator polled a closed ICE agent every 15 s, 1.5 CPU) until new SIP legs got no media (2026-10-28). Every Python-SDK (single peer connection) participant that closed from 10-25 on leaked, whatever its close reason; Go (livekit-sip) and JS never did. - livekit-server v1.13.10: carries the warp WebRTC deadlock fixes (webrtc-pion #3/#5, ice #4, dtls #1); not v1.13.11 (single-PC data channel changes). prometheus.port, debug_handler on 6790 so a future leak leaves a goroutine profile. - livekit-reaper: every 60 s deletes rooms no live call can be in (empty, phone leg alone 3 min, orphan ear room, older than 6 h, harness rooms after 30 min, no published track 15 min) and removes stuck participants; at most 10 actions a pass, one JSON log line each. Counts leaked participants from livekit-server's log (Loki) and restarts the server only at >= 10 leaked, 0 live rooms, not twice in 6 h, logging the goroutine profile first. - ServiceMonitor for livekit-server, livekit-sip and the reaper; PrometheusRule (leaked participants, rooms/participants high, busy while idle, reaper busy/deferring/down, livekit-sip down); AlertmanagerConfig to SysAdmin@KamoCRM.com. - apply.sh: a deploy that would roll livekit-server or livekit-sip (Recreate, host ports) waits for 0 live rooms (reaper count, 20 min) or fails without touching them.

All changes

Like what you see shipping?

All of it arrives in your workspace on its own. Start on the free plan and read this page again in a month.

Start Free ForeverView Pricing