Intermittent API timeouts on create, update and delete requests
Last updateresolvedOct 9 · 15:40 UTC
Summary
On October 8, 2026, between 07:36 and 08:45 UTC, 112 API requests that create, update or delete resources timed out. A fault in our API servers could leave one server process unable to finish these requests until it restarted. The same fault caused occasional timeouts before October 8. We deployed a permanent fix on October 8 at 17:55 UTC.
Your Terraform code and configuration were not the cause.
What you may have seen
- Requests that create, update or delete resources ended with a 504 Gateway Timeout after about 29 seconds. Our other server processes handled requests normally, so most requests succeeded while a few failed. On October 8, read requests were not affected.
- The StackGuardian Terraform provider reported some of these timeouts as a 400 Bad Request, for example "Error reading Role Assignment". Running the plan again worked.
- A request that timed out may still have made its change, because the affected requests stalled after doing their work. If a request timed out for you, check whether the resource was created or updated before you retry it.
Cause
Our API runs as several server processes, each handling many requests at once. After a request changes data, and before it replies, the API writes an audit-log entry. All requests on one process share the connection used for this step, and a lock controls access to it.
A timing fault in a third-party networking library we use could put a request to sleep while it held that lock and never wake it. Our own code triggered the fault: it left some cloud storage connections open, and the automatic cleanup of those connections interrupted the request at the wrong moment. After that, every write request on that process waited for the lock and got no reply until the process was replaced by its routine restart. The process kept answering read requests and health checks, so it appeared healthy and was not replaced sooner.
The 400 Bad Request in Terraform was a separate fault. When the provider retried a timed-out request, it sent the request without its body, and our gateway rejected it as invalid.
What we fixed
Two server fixes have been live since October 8 at 17:55 UTC:
- We upgraded the networking library. We could reproduce the fault on the old version and cannot on the new one.
- The API now closes its cloud storage connections after use, which removes the cleanup that triggered the fault.
Since then, we have not seen an API process get stuck.
The Terraform provider now sends the full request when it retries, so a brief server error is retried correctly instead of showing up as a 400 Bad Request. This fix will ship in the next provider release after v1.12.4.
Preventing a recurrence
- We are adding a time limit to every API request. A request that runs too long will be stopped and its lock released, so one stuck request can no longer hold up others, whatever the cause. This is not live yet.
- Automated tests now check that the API closes its cloud storage connections and that the provider's retry includes the full request.
We apologize for the disruption.
Reported by StackGuardian on their status page.