- Shipped
- September 23, 2026 at 1:54 AM UTC
- Author
- Kamo
- Commit
- d260fad
Quartz here runs 10 in-memory worker threads on one replica. 17 of the 24 jobs had no @DisallowConcurrentExecution, so a run that overran its interval started a second copy beside it — for the per-minute jobs (reminders, recordings, VOIP sync, retail sync, MeridianLink poll, scenery fill, webinar reminders…) that is double processing. Six HTTP clients were `new RestTemplate()` with no connect or read timeout (the Google and Microsoft token refreshers, the SecurityService LOS-apply and pipeline clients, the VOIP global sync and the recording dispatcher), so one hung downstream held its worker forever — and with overlap allowed, took another worker every minute until every job in the daemon had stopped, silently, until a pod restart. - @DisallowConcurrentExecution on every job. - Timeouts on every client: 10 s connect everywhere; read 30 s for token endpoints, 60 s for the SecurityService internal calls, 10 min for the VOIP sync and FFmpeg dispatch (the same bound the other sweep triggers already use). - DaemonJobSafetyTest pins both rules for future jobs (mutation-checked: removing one annotation, or restoring one bare RestTemplate, turns it red). Full suite: 69 tests green.