Exalate Incident History

1. postmortemOct 25 · 12:22 UTC

   Incident: JiraCloud Nodes Experiencing Instability - July 16, 2026

   Duration: 12:28 CET – 16:57 CET (approximately 4.5 hours). Full stability confirmed at 18:18 CET.

   Impact: Some Jira Cloud integration nodes experienced intermittent instability during this window. Typically ~10 nodes were affected simultaneously. Issue synchronisation on unaffected nodes continued to run normally. No data was lost.

   Summary: On July 16, 2026, some Jira Cloud integration nodes on the Exalate Cloud platform experienced intermittent restarts and timeouts. The root cause was a resource constraint on one of the underlying hosting servers. A coincidental software update initially directed investigation towards the application layer, but cross-environment analysis confirmed that nodes on other infrastructure were unaffected — narrowing the issue to a single hosting server.

   Timeline:

   • 12:28 CET - Issue detected, investigation initiated
   • 13:03 CET - Stabilisation measures applied, rolling restart initiated
   • 16:57 CET - All nodes back to normal operation
   • 18:18 CET - Full stability confirmed

   Root Cause: A hosting server reached its resource ceiling due to concentrated workload during a routine software update cycle. This caused intermittent resource starvation for nodes running on that server.

   Resolution: Workload was redistributed across the hosting infrastructure to balance resource utilisation. All affected nodes recovered with no data loss.

   Preventive Measures:

   • Proactive alerting thresholds for infrastructure resource utilisation have been implemented
   • Workload distribution across hosting servers has been rebalanced to prevent concentration

   We apologize for the disruption. We remain committed to the reliability of the Exalate platform.

Reported by Exalate on their status page.

1. postmortemOct 25 · 10:27 UTC

   Incident: Exalate Cloud service disruption - 4 May 2026

   Duration: 10:45 – 22:10 CEST (approximately 11 hours 25 minutes, intermittent)

   Impact: Several hundred Exalate Cloud nodes on one of our European clusters became repeatedly unreachable during the working day. Synchronisations on affected nodes did not run during these windows and resumed automatically each time service returned. Other clusters were unaffected. No data was lost, and no customer action was required.

   Summary: On 4 May 2026, the database serving these nodes lost its standby replica, leaving one instance carrying all traffic. Its storage shared a physical device with another workload, and under load the resulting contention slowed disk writes enough for our automated health check to time out. The platform treated the timeout as a failure and restarted a database that was busy rather than broken, disconnecting the nodes on that cluster each time.

   Timeline:

   • 10:45 CEST — Nodes reported unreachable, investigation initiated
   • 15:24 CEST — Incident declared and published to our status page
   • 22:10 CEST — Standby replica restored, cluster confirmed healthy

   Root Cause: Storage contention on a shared physical device caused health-check timeouts that triggered automated restarts. Rebuilding the standby replica amplified the same contention, as streaming a copy of the database drew disk throughput away from the instance still serving customers.

   Resolution: Storage was moved to a dedicated device, additional CPU allocated, and disk throughput raised on the primary while deliberately limited on the replica so the rebuild drew data more gently.

   Preventive measures: We are separating database storage from other workloads across this environment, retuning health-check thresholds so a database under heavy but healthy load is not restarted, parallelising the node recovery path to shorten restoration, and adding host-level monitoring so contention alerts before it reaches customers.

   We apologise for the disruption. We remain committed to the reliability of the Exalate platform.

Reported by Exalate on their status page.

1. postmortemOct 25 · 10:25 UTC

   Incident: Exalate Cloud service disruption - 2 April 2026

   Duration: 16:30 – 21:25 CEST (4 hours 55 minutes)

   Impact: Approximately 60 Exalate Cloud nodes hosted on our Belgian cluster were unavailable. Synchronisations on those nodes did not run for the duration of the outage and resumed automatically once the nodes were restored. No data was lost.

   Summary: During a scheduled rotation of internal Kubernetes certificates, every worker node in the cluster restarted its agent at the same time. Each worker then simultaneously asked the storage API to re-verify its disks, and the volume of simultaneous requests overwhelmed that API. With the storage API unresponsive, affected nodes could not attach their disks and remained offline.

   Resolution: Traffic to the storage API was isolated and the component restarted to clear the backlog. Nodes were then returned to service in a controlled sequence, one worker at a time, until all nodes were online at 21:25 CEST.

   Preventive measures: The certificate rotation procedure is being changed so that node restarts are paced and batched rather than executed simultaneously, and this pacing is being built into the tooling rather than applied manually. In addition, we are working with our infrastructure partner on protective rate limiting for the storage API, on a change to the storage driver so that nodes verify their disks locally instead of calling the API, and on monitoring that alerts on API saturation before it affects customers.

   We apologise for the disruption. We remain committed to the reliability of the Exalate platform.

Reported by Exalate on their status page.