AdBook+ System Performance
Last updatepostmortemOct 14 · 22:15 UTC
Summary of Impact
On September 30, 2026, at 8:17 AM PT, an AdBook deployment of v4.10.01 intended for the sandbox (SBX) environment was unintentionally run against Production. The deployment shut down all environments, and AdBook was unavailable to all customers for 25 minutes. Access was restored at 8:42 AM PT. Some reporting and inventory functions stayed degraded until 2:24 PM PT, when full functionality was restored. No customer data was lost, and no customer action is required.
Summary of events:
- 8.17 am PT - v4.10.01 deployment begins on Production; all environments shut down
- 8.22 am PT - Incident communication protocols begin
- 8.24 am PT - v4.09.03 redeployed; application access restored
- 8.33 am PT - Customers notified of incident via Status page
- 9.30 am PT - DQT (Data Query Tool) reporting restored
- 1.19 pm PT - Inventory availability validation restored
- 2.24 pm PT - Inventory jobs that failed during the incident reprocessed; full functionality restored
Root Cause
While deploying v4.10.01 to SBX, the Engineering team hit deployment failures. During troubleshooting, the deployment pipeline's target environment was unintentionally changed from SBX to Production. Two gaps in tooling and process allowed this:
- The environment selector's clickable area extends well beyond its visible controls, so a click elsewhere on the page can change the target without any visible indication.
- The pipeline has no confirmation step specific to Production, so nothing prompted a final check of the target before services stopped.
The extended recovery after the application was rolled back was a result of.v4.10.01 including database changes that aren't backward compatible with v4.09.03.
Resolution
The Engineering team redeployed v4.09.03 - the current production version - as the fastest and lowest-risk way to restore access. Application access was restored at 8:42 AM PT.
Redeploying the application did not reverse the database changes made by v4.10.01.DQT reporting and inventory availability validation remained affected because database artifacts required by v4.09.03 had been removed or changed during the v4.10.01 deployment. The team restored these objects as they were identified and reprocessed the affected inventory jobs, completing recovery at 2:24 PM PT.
Mitigation and Next Steps
To reduce risk of a similar incident we are adding a mandatory production-intent confirmation gate.
The gate and subsequent confirmation will prominently identify key deployment parameters (such as the target environment, release version), and require explicit acknowledgment before any services are stopped or database changes are applied.
Second-person approval will be added as an additional safeguard, taking staff availability and emergency recovery requirements into account.
Reported by FatTail on their status page.