We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident.
Detection and Impact:
On October 10th at 12:50 AM PT, Okta’s monitoring systems alerted on-call teams to a database server issue on EMEA Cell EU1.
During this time, administrators and end users may have encountered Read Only Mode (See KB: https://support.okta.com/help/s/article/What-is-Oktas-Readonly-Mode ) during this period, as well as errors related to write operations and certain authentication flows.
Root Cause Summary:
The root cause of the disruption was an unexpected crash of the primary data store supporting the impacted cell, which automatically resulted in Okta application servers transitioning to Read Only mode to ensure data integrity.
Remediation Steps:
Immediately upon receiving alerts, incident response processes were activated. On-call incident responders analyzed the database service failure and took actions to failover to a healthy primary and restore service functionality.
Preventative Actions:
In order to prevent and mitigate similar incidents in the future, Okta has currently implemented the following:
- Improved alerting for faster impact analysis
- Conducted a full scan of all database error log files to ensure this issue had not happened previously
- Fixed a bug related to authentication failures during Read Only Mode related to Device Assurance checks
Additionally, Okta is reviewing operating procedures to recover the service faster and is currently working with external infrastructure vendors to investigate the underlying cause of the database crash.
Duration (# of minutes): 35