TechnologyOne Incident History

1. postmortemSep 15 · 01:15 UTC

   Incident Summary

   Between 31 August 2026 and 2 September 2026, some customers experienced intermittent service disruptions, including slow application responsiveness, connection timeouts, and periods of Ci unavailability. The issue was caused by internal DNS resolution failures within an underlying CiA cluster, which affected communication between application services and backend caching components. Service stability has been restored and all affected environments are operating normally.

   Root Cause

   A significant increase in internal DNS requests related to Redis caching placed unexpected demand on the cluster's DNS service (CoreDNS). This resulted in memory exhaustion within CoreDNS, preventing it from scaling effectively to meet demand. As a result, some application services experienced delayed or failed DNS lookups, leading to connection timeouts and degraded application performance.

   Corrective Actions Taken

   To restore service and stabilise the platform, we:

   • Increased CoreDNS capacity by scaling the service.
   • Rebalanced workloads by moving affected environments to alternative CiA clusters.
   • Restarted impacted DNS services and closely monitored DNS performance until normal operation was confirmed.

   Preventative Actions

   • Optimising Redis connection management and DNS resolution behaviour to reduce DNS query volumes.
   • Reviewing and refining CoreDNS autoscaling policies, resource limits, and capacity thresholds.
   • Assessing long-term cluster capacity and workload distribution strategies.
   • Enhancing monitoring and alerting to identify and respond to DNS-related issues before customer impact occurs.

Reported by TechnologyOne on their status page.

1. ResolvedSep 2 · 04:02 UTC

   Following an extended period of stable performance and continued monitoring, this incident has now been resolved. Error rates have remained at expected levels, and no further customer impact has been observed. Our teams will continue to review the incident and implement any identified improvements as part of our standard post-incident process. We appreciate your patience and understanding while we worked to restore and validate service stability.

Reported by TechnologyOne on their status page.

1. postmortemSep 15 · 01:14 UTC

   Incident Summary

   Between 31 August 2026 and 2 September 2026, some customers experienced intermittent service disruptions, including slow application responsiveness, connection timeouts, and periods of Ci unavailability. The issue was caused by internal DNS resolution failures within an underlying CiA cluster, which affected communication between application services and backend caching components. Service stability has been restored and all affected environments are operating normally.

   Root Cause

   A significant increase in internal DNS requests related to Redis caching placed unexpected demand on the cluster's DNS service (CoreDNS). This resulted in memory exhaustion within CoreDNS, preventing it from scaling effectively to meet demand. As a result, some application services experienced delayed or failed DNS lookups, leading to connection timeouts and degraded application performance.

   Corrective Actions Taken

   To restore service and stabilise the platform, we:

   • Increased CoreDNS capacity by scaling the service.
   • Rebalanced workloads by moving affected environments to alternative CiA clusters.
   • Restarted impacted DNS services and closely monitored DNS performance until normal operation was confirmed.

   Preventative Actions

   • Optimising Redis connection management and DNS resolution behaviour to reduce DNS query volumes.
   • Reviewing and refining CoreDNS autoscaling policies, resource limits, and capacity thresholds.
   • Assessing long-term cluster capacity and workload distribution strategies.
   • Enhancing monitoring and alerting to identify and respond to DNS-related issues before customer impact occurs.

Reported by TechnologyOne on their status page.

1. ResolvedAug 31 · 05:43 UTC

   After considerable monitoring, this incident is now resolved. We will undertake a Post Incident Review and findings will be posted here on completion. We apologise for how you and your business may have been affected by this incident. Please contact the TechnologyOne Support team if you continue to experience issues.

Reported by TechnologyOne on their status page.