Summary
CloudAMQP lost connectivity with a subset of clusters in Azure west Europe availability zone 2 on October 9th. Single node clusters in AZ2 were most impacted by the incident.
Impact
Customers were unable to access data or mange nodes impacted during the incident.
Timeline of events from Azure.
- 12:25 UTC on 09 October 2026 – We identified this issue via our internal service monitoring alerts.
- 12:29 UTC on 09 October 2026 – Customer impact began, with VM disk failures detected in the affected West Europe storage scale unit.
- 2:30 UTC on 09 October 2026 – The investigation identified loss of heartbeat across multiple storage nodes and fault domains, resulting in the storage scale unit to go in distress mode.
- 15:32 UTC on 09 October 2026 - We took the storage scale unit out of distress mode, isolating the disks on the impacted storage servers to enable recovery of the remaining disks. At this stage, most of the affected customers began seeing recovery.
- 21:00 UTC on 09 October 2026 - Datacenter and Storage engineers continued executing the recovery plan on multiple storage nodes.
- 23:27 UTC on 09 October 2026 – Efforts to recover the affected storage nodes were successful, and work to recover the virtual disks continued. Subsequent recovery actions included node-level recovery and restoration of affected storage resources.
- 03:00 UTC on 10 October 2026 – Service functionality recovered, and customer impact was mitigated.
Root cause
Azure experienced a platform issue resulted in an impact to Azure Virtual Machines storage disks in the West Europe region AZ 2. Impacted customers may have experienced connectivity failures, unexpected restarts, or Virtual Machines entering a failed state in the region.
Resolution
Machine access was restored by Azure in about 3 hours. CloudAMQP was able to restore communication with the impacted nodes. We continued to monitor updates from Azure to ensure nodes were no longer impacted.
