Between 14:41 UTC and 17:47 UTC on Wednesday, 7 October 2026, customers attempting to access models in us2: Data Center - US West may have experienced intermittent loading delays, network errors, and HTTP 500 errors.
Root Cause
The incident was triggered by an automated maintenance update that resulted in an unexpected network interface reset on an infrastructure host. When the network interface initialized, an unexpected software behavior prevented network traffic from routing normally. While the network interfaces reported a healthy link state to monitoring systems, the driver ceased forwarding network traffic. This silent failure disabled the virtual machines hosted on that server, including primary regional DNS caching and authentication services.
The loss of these primary routing services increased traffic on remaining servers, causing intermittent platform delays and timeouts. Remediation was further slowed because restoring the impacted systems relied on internal services that were temporarily unavailable.
Recovery
To mitigate customer impact and restore platform stability, we immediately mobilized our teams to isolate the source of the network degradation and protect active user sessions. We began by safely redirecting active customer workloads away from the affected infrastructure. In parallel, we implemented emergency traffic rerouting configurations across the platform to bypass impaired network components and route requests strictly through healthy routing systems.
Once traffic pathways were stabilized, we safely relocated the primary network routing services to an alternative, healthy infrastructure. This successfully re-established normal network communication, bringing query response times and system error rates back to baseline levels at 17:17 UTC.
Following the migration, we closely monitored platform traffic, validated model responsiveness, and confirmed that cross-region data imports were completing successfully. Full service was restored by 17:47 UTC.
Corrective & Preventative Actions
To prevent a recurrence and improve platform resilience, we are implementing the following corrective actions:
- To address the trigger for this incident, we have permanently removed the automated update feature that caused the network interface reset.
- We are rolling out an operating system upgrade across infrastructure hosts to permanently resolve the software defect that caused the driver to cease forwarding network traffic despite reporting a healthy link state.
- We are increasing routing capacity and moving key network systems to standalone infrastructure. This will ensure that network routing remains fully independent and resilient during platform events.
We apologize for any impact this issue may have had on your business operations. We are continuously strengthening our systems and procedures to ensure we avoid future disruptions to your business and users.
If you have further questions or concerns, please visit our Support website. We appreciate your patience during this incident and value the trust you place in Anaplan.
