scalr.io Incident History

1. postmortemOct 22 · 18:33 UTC

   Summary

   On October 22, 2026, between 15:45 and 16:35 UTC, some runs failed to start. Affected tasks failed with the error “Cannot provision execution environment”. The issue was caused by a sudden spike in run volume that exceeded a capacity limit in our cluster networking. It resolved on its own once the spike subsided, and the service is operating normally. A fix that removes the underlying cause is rolling out.

   Customer impact

   Between 15:45 and 16:35 UTC, ~2,000 tasks failed across ~1,800 runs, affecting roughly 100 accounts. Affected runs failed with an internal error because the agent worker never picked up the task, so no work was started. Tasks that were already running were not affected.

   If any of your runs failed during this window, please re-run them.

   Timeline

   All times are UTC on October 22, 2026.

   Time	Event
   15:45	Runs begin failing to start as a surge of run activity pushes the platform past a networking capacity limit.
   16:35	The surge subsides, capacity recovers, and new runs start normally.
   After recovery	The cause is identified and a fix begins rolling out.

   Root cause

   Each agent task runs in its own short-lived pod, and each pod was labeled with values unique to that task. Our cluster networking creates a separate network identity for every unique set of labels, so each task created its own identity.

   Those identities are tracked in a table with a fixed size limit. At roughly twice our normal hourly task volume, the number of identities exceeded that limit. Pods that started while the table was full came up without DNS access, so the agent worker could not pick up the task, and it failed after 300 seconds.

   The limit existed before this event. A burst of runs from several customers across many workspaces was what pushed us over it.

   Resolution

   The issue cleared on its own as the burst subsided and unused network identities were cleaned up. In parallel, we began rolling out a release that removes the per-task unique values from task pod labels, so tasks no longer each create their own identity. This reduces identity usage by orders of magnitude, well below the limit.

   Preventing a recurrence

   1. Remove the cause. We are rolling out the label change described above, narrowing the DNS network rule on task pods so it consumes far less capacity, and further hardening our networking configuration, which we will validate outside production first.
   2. Improve monitoring. We are adding alerts on networking capacity so we are warned well before the limit is reached.
   3. Detect failures sooner. We are adding alerts on the rate of internal-error run failures and on the Task was not started by the agent worker within 300s error, so issues like this are caught quickly regardless of cause.

   Our apology

   We are sorry for the disruption this caused your teams and pipelines. Reliable run execution is core to what you depend on us for. If you have questions about how this affected your account, please contact Scalr support.

Reported by scalr.io on their status page.