Dates:
Monday, September 28th 2026, 14:19 UTC - 16:52 UTC.
What happened:
Real device tests in US West hung and failed to output test artifacts due to slow uploads.
Why it happened:
A degraded interconnect link suffered high packet loss from optical errors, but BGP stayed up, preventing automatic failover to healthy paths.
How we fixed it:
Manually disabled the degraded port on the edge router, forcing traffic onto healthy alternate interconnects.
What we are doing to prevent it from happening again:
Automating link draining based on CRC error thresholds, improving protocols for isolating flapping vendor links, and hardening upload resilience.