Service Disruption for a subset of customers in ANZ
Last updatepostmortemOct 15 · 04:00 UTC
Issue Summary: At 12:55 PM on 01 October 2026, some customers reported experiencing significant delays in data processing (DP) jobs. This resulted in degraded system performance and a large backlog of submitted actions remaining in a 'pending' or 'submitted' state for several hours.
Root Cause Analysis: The issue was caused by an overload of underlying platform components due to a sudden, very high volume of simultaneous requests, which temporarily exceeded available processing capacity.
Corrective Action: Technology1 observed the delays occurring and the system automatically scaled up the background job processor's capacity to handle the increased load. While the auto-scaling mechanisms successfully provisioned the necessary resources, the sheer volume of the accumulated backlog required significant time to clear. The system continuously processed the jobs at this expanded capacity until all pending actions were resolved and the queue returned to normal levels.
Preventative Measures
- Optimize auto-scaling thresholds and improve monitoring to react even faster to unexpected spikes in demand.
- Implement additional flow controls to manage high processing volumes and ensure system settings remain within safe operating limits.
Reported by TechnologyOne on their status page.