Arc XP Incident History

1. postmortemAug 31 · 15:15 UTC

   ## Customer Impact On Wednesday, August 27, 2026, beginning at approximately 23:50 UTC, some customers hosted in the US East region were temporarily unable to view or load stories in Composer. During the impact window, affected users who tried to open stories saw them fail to display. Story viewing was fully restored by 00:19 UTC on August 28, approximately 29 minutes later. ## Root Cause During the impact window, an internal configuration issue caused an unexpected surge of internal requests to one of the backend services that Composer relies on to load and display stories. The elevated load briefly exhausted that service's available connections, which cascaded to the services that depend on it and left Composer unable to display stories for affected customers in the US East region. The affected service recovered automatically once the abnormal load subsided, and story viewing returned to normal. Arc XP confirmed the recovery and, as a preventive measure, increased the service's capacity so it can better absorb similar spikes going forward. Arc XP is continuing to investigate the underlying cause of the configuration issue to fully understand how it arose, and is reviewing safeguards, including additional capacity headroom and earlier detection of abnormal request patterns to reduce the likelihood and impact of any recurrence. ## Timeline **All times UTC** | **Time** | **Event** | **SLA Context** | | --- | --- | --- | | Aug 27, 23:50 | Customers in the US East region begin experiencing problems viewing stories in Composer. | Impact start | | Aug 28, 00:02 | Arc XP declares an incident after reports that Composer is not displaying stories. | | | Aug 28, 00:08 | Elevated story-publishing errors are observed, confirming degraded Composer functionality. | | | Aug 28, 00:19 | Story viewing begins to recover for affected customers. | Impact end | | Aug 28, 00:21 | Monitoring shows initial signs of recovery. | | | Aug 28, 00:32 | Investigation identifies an abnormal surge of internal requests overwhelming a backend service as the source of the disruption. | | | Aug 28, 01:07 | Services stabilize and normal operation resumes. | | | Aug 28, 01:19 | Root-cause analysis traces the surge to an internal configuration issue. | | | Aug 28, 01:28 | Incident resolved. | Resolved | ## Arc XP Next Steps * Increased the capacity of the affected backend service so it can better absorb sudden spikes in internal request volume \(already completed\). * Reviewing monitoring and alerting so abnormal request patterns can be detected earlier, before they affect customer-facing functionality. * Evaluating additional safeguards to limit how a single backend service under stress can affect the services that depend on it.

Reported by Arc XP on their status page.