Web Calls and Phone Calls Disruption
Last updatepostmortemSep 6 · 03:35 UTC
### Summary On September 5, a rare race condition in our custom LLM call path caused call scheduling queues across our fleet to stall. Because the failure also made containers appear underutilized, autoscaling never triggered. Between roughly 10:00 and 16:00 PST the impact grew until most containers were stuck and new calls could not be placed. ### Impact * **Web calls, outbound calls, and inbound calls** were all affected. * Inbound calls were the least affected, since they hold the highest priority in our scheduling queue. * Peak impact occurred around 16:00 PST. ### Root Cause The failure required a specific sequence of events: 1. **Stalled initialization.** If a call ended before its custom LLM websocket was established, the call's initialization never completed. The scheduling queue waited indefinitely for that initialization to finish. 2. **Queue exhaustion.** Each container allows a fixed number of concurrent dequeue operations. Enough stalled initializations consumed all available slots, leaving the container unable to pick up new calls. 3. **Misleading scaling signals.** The call-end path still ran and decremented the container's concurrency counter. As a result, stuck containers reported low concurrency and low resource utilization — so no scale-out was triggered. With every container blocked and autoscaling idle, new calls had nowhere to go. This pattern is only reachable through that exact ordering, which is why it went undetected until it occurred at scale. ### Timeline \(PST\) * 10AM: A customer's custom LLM calls begin dropping before initialization completes. * Afternoon: The customer scales up traffic; the number of stalled initializations grows. * ~16:00: Nearly all containers are blocked on dequeue slots. Autoscaling does not trigger. Call placement is broadly impacted. * After 16:00: Mitigation begins: manual scale-out of containers and clearing of stuck concurrency for the affected customer. Service recovers. ### Prevention 1. \(COMPLETED\): Fix the race condition in custom LLM call initialization 2. \(IN PROGRESS\): Add monitoring and alerting on call scheduling queues to detect stuck states 3. \(IN PROGRESS\): Decouple the scheduling queue from call initialization so a slow or failed init cannot block dequeue 4. \(IN PROGRESS\): Add call scheduling queue pressure as an input to our autoscaling policies ### Closing We apologize for the disruption. We are treating this as an opportunity to harden our scheduling and scaling infrastructure, and we will ship the remaining action items within a few days.
Reported by Retell AI on their status page.