Clerk.io Incident History

1. maintenanceSep 4 · 21:00 UTC

   In response to the incident yesterday (2026-09-02 - https://status.clerk.io/incident/1046330) we will be performing software and configuration upgrades on all of our databases. The maintenance window is 3 hours but is expected to be shorter. During this window the system database will be placed in read-only mode, meaning that all your products are still available, but updates to designs, merchandising and similar will be paused. Importers and analysis jobs will also be unable to be started in this window. The read-only window is only expected to last part of the window and we endeavor to minimize the time-frame where services are impacted. We apologize for the inconvenience. We make these upgrades to maintain the best possible experience for you and your customers.

Reported by Clerk.io on their status page.

1. resolvedSep 2 · 09:46 UTC

   # Preliminary incident report: API availability **Date:** 2026-09-02 T 07:05 CEST **Duration:** Approximately 10 minutes **Impact:** Some API requests and background operations were delayed or failed for a subset of stores. This also had a knock-on effect on the API availability for other stores as workers got tied up, until automatic scaling added new servers. ## What happened One of our database servers stopped processing queries after an internal MariaDB/InnoDB synchronization operation became stuck. The database remained online at the network level but could not complete normal work. After its internal safety timeout was reached, MariaDB terminated the stalled process and recovered automatically during restart. Service returned to normal once database recovery completed. ## Cause The incident was caused by an internal database metadata lock that failed to make progress. The likely trigger was an unusual interaction between a high-volume background table rebuild and database checkpoint processing. This was not caused by increased customer traffic or an individual customer's actions. The same background processing normally completes without affecting availability. The precise database thread responsible could not be identified because detailed crash diagnostics were not enabled at the time. ## Data integrity MariaDB completed its normal crash-recovery process successfully. Interrupted transactions were rolled back automatically, and we found no evidence of database corruption. No customer action is required. ## Improvements We have increased the database log capacity to provide more processing headroom during large background operations. We are also preparing the following changes: - Remove unnecessary database metadata operations from bulk imports. - Upgrade MariaDB to the latest maintenance release. - Collect detailed diagnostics if a similar database failure occurs again. - Reduce the time MariaDB waits before recovering from a confirmed internal stall. - Improve monitoring and recovery automation for database operations that appear online but cannot process real queries. - Extend failure handling for background processing as well as customer-facing API requests. These changes are intended both to reduce the likelihood of recurrence and to restore service more quickly if another internal database stall occurs.

Reported by Clerk.io on their status page.

1. resolvedAug 27 · 14:30 UTC

   The issue has been resolved, and the API is fully operational again. From 16:11 to 16:22 CEST, v2/search, v2/recommendations, and the dashboard were unavailable due to a database disruption. We are currently reviewing the incident and implementing measures to prevent such incidents from happening in the future. We apologize for the inconvenience.

Reported by Clerk.io on their status page.