Supabase Incident History

Dashboard and Management API are seeing elevated error rates.Existing projects are unaffected, and are serving traffic normally.

We are seeing longer-than-usual wait times for project creation or configuration change events.

We observed intermittent connection issues to our Supavisor cluster in ap-northeast-1 from 16:17 - 16:42 UTC.This issue has been resolved.

Users with third-party auth configured may see auth failures when making storage API requests. The team has identified the issue and is working on a solution.

Some users attempting to restore backups may see failures in the restore process. All backup data is safe and accounted for, and we are working on a fix for the restore process.

Some users attempting to restore backups may see a restore to a more recent point than they selected. All backup data is safe and accounted for, and we are working on a fix for the restore process.

The Database Health Report, and any other custom reports that refer to the resource utilization (e.g. CPU, memory) on the database will be briefly unavailable. The maintenance is expected to take less than 30 minutes.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

We are currently experiencing some issues in our logging layer, which will result in delayed logs for some features. The team is currently investigating the cause and is adding additional capacity to compensate.

This incident has been resolved.

We seen elevated error rates due to increased requests and added extra nodes in eu-west-2, us-west-1 and ap-southeast-2 to cope with this increase, we are monitoring to ensure this is adequate.

We are currently investigating this issue.

This incident has been resolved.

The fix has been rolled out. Block rates at the API layer have returned to normal levels, and things are looking stable now. API requests to your supabase projects should now be working as expected.

With the assistance of our API gateway partner, we have identified the configurations resulting in the Blocking and are working to update them.

Our team has reached out to our API gateway partners and are continuing to investigate the periodic Block errors that folks are receiving across all API endpoints.

We are receiving reports of user API requests receiving API gateway errors when making Auth and other API calls. The team is currently looking into the reason and for mitigating steps.

This incident has been resolved.

We increased available resources in us-east-1, which has reduced error rates to normal levels. All regions currently looking stable. Due to the volatility here, we'll be keeping an eye on things for a while to be sure.

US-East-1 has begun showing elevated error rates again. The team is currently looking into these errors now.

The fix for the secondary issues has been deployed, and things are looking stable again. We will continue to monitor for any further issues.

The fix is still being deployed across the fleet. We hope to have edge functions behaving normally again soon.

We have identified a second issue, and the team is rolling out a fix shortly.If you are seeing 503s and "worker boot error" messages in your edge function logs, this fix will address those situations.

Unfortunately, while it immediately appeared that the fix we deployed was working, error rates across all regions have begun to climb again. The team is continuing to investigate.

The fix has been deployed. Error rates across all regions have returned to normal levels. We'll keep an eye on things to ensure they remain stable.

The issue has been identified and a fix is being implemented.

Elevated error rates have expanded to all regions, globally.

Users with functions firing in eu-central-1 and eu-west-1, and us-east-1 may see increased error rates. You can, if you like, choose a region to execute functions in to avoid these regions by passing a header. More info on that here: https://supabase.com/docs/guides/functions/regional-invocation

This incident has been resolved.

We are continuing to monitor for any further issues.

The team have successfully deployed a fix and we are monitoring.

The team have identified that a recent deploy is affecting a small subset of requests to our Management API, we are working on a fix.Data APIs and Postgres APIs for existing projects remain unaffected. Your projects should continue to work as normal.

Data APIs and Postgres APIs for existing projects are unaffected.

We have rolled out improvements to the restart process which lets us pre-reserve capacity before a restart/compute resize operation, with this improvement we are able to abort the restart process if there isn't sufficient capacity available, hence avoiding downtime.This does not resolve the ongoing capacity issue present in eu-west-3b, but it will negate any type of restart or compute resize-related outage which might be encountered and inform users if this didn't succeed due to a capacity-related issue.This additionally allows the platform to act more dynamically, allowing these operations to either execute successfully or fail gracefully, based on how AWS capacity fluctuates, and reducing the need for us to institute blanket guards at an availability zone or regional level.All project operations have been restored in previously affected availability zones:- ap-south-1a- eu-central-1a- eu-west-3c- eu-west-2bAll project operations except database version upgrades have been restored in the following availability-zone:- eu-west-3b

We have rolled out improvements to the restart process which lets us pre-reserve capacity before a restart/compute resize operation, with this improvement we are able to abort the restart process if there isn't sufficient capacity available, hence avoiding downtime.This does not resolve the ongoing capacity issue present in eu-west-3b, but it will negate any type of restart or compute resize-related outage which might be encountered and inform users if this didn't succeed due to a capacity-related issue.This additionally allows the platform to act more dynamically, allowing these operations to either execute successfully or fail gracefully, based on how AWS capacity fluctuates, and reducing the need for us to institute blanket guards at an availability zone or regional level.All project operations have been restored in previously affected availability zones:- ap-south-1a- eu-central-1a- eu-west-3c- eu-west-2b- eu-west-3b

Project restarts, as well as compute and database version upgrade operations have been disabled in the following availability zones:- eu-west-3bWe are continuing to work with our cloud provider to in order to address compute capacity issues.All project operations have been restored in previously affected availability zones:- ap-south-1a- eu-central-1a- eu-west-3c- eu-west-2b

Project restarts, compute operations and database version upgrades have been disabled in eu-west-2b and eu-west-3b availability zones.Project restarts, as well as compute and database version upgrade operations have been re-enabled in the previously impacted ap-south-1a, eu-central-1a, and eu-west-3c availability zones. These operations are currently disabled in the still-impacted eu-west-3b and eu-west-2b availability zones.

We are continuing to work with our cloud provider to in order to address compute capacity issues in the eu-west-3b availability zone.Project restarts, as well as compute and database version upgrade operations have been re-enabled in the previously impacted ap-south-1a, eu-central-1a, and eu-west-3c availability zones. These operations are currently disabled in the still-impacted eu-west-3b availability zone.Project creation has been re-enabled in the eu-west-3 region.

We are continuing to work on a fix for this issue.

We have identified an issue with our cloud provider regarding insufficient compute capacity in the following availability zones:- ap-south-1a- eu-central-1a- eu-west-3b- eu-west-3cWe have disabled the ability to restart projects, compute or database version upgrades within these availability zones.Project creation has been disabled for the eu-west-3 region.

We are currently investigating this issue.