scalr.io Incident History

**Summary** On July 6, 2026, runs failed to start for approximately 22 minutes, returning the error "Cannot provision execution environment." The root cause was a stuck PVC \(persistent volume claim\) during a migration to new network storage for the Terraform/OpenTofu providers cache. **Impact** Between 16:48 and 16:53 UTC, new runs across affected workspaces failed to schedule with a "Cannot provision execution environment" error. **Timeline** \(all times UTC\) * 16:48 — Issue detected; investigation began * 16:50 — Root cause identified: PVC stuck due to in-use protection finalizer * 16:51 — Mitigation deployed \(providers cache disabled\); monitoring began * 16:53 — Continued monitoring, no further impact observed * 17:10 — Incident resolved **What Happened** During the migration to new network storage for the Terraform/OpenTofu providers cache, the existing PVC was still attached to multiple active workloads. Kubernetes' in-use protection finalizer blocked deletion and recreation of the volume, so new runs could not bind to it and failed to schedule. **Mitigation** We disabled the providers cache to unblock run scheduling while we worked on the underlying volume issue. This restored run scheduling immediately, with only a minor, temporary performance impact from running without a warm provider cache. **Root Cause** The migration runbook did not account for the PVC's in-use protection finalizer when workloads were still actively bound to the volume, which blocked the planned delete-and-recreate step. **Remediation and Prevention** * Immediate: delete the workloads still holding the PVC, recreate the volume, and restore it as the providers cache. * Preventive: update the migration runbook to explicitly drain or cordon any workloads still bound to a PVC before attempting to delete and recreate it, rather than assuming the volume is free.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.