Cerebrium Incident History

The issue has been resolved and all regions are fully operational again.

Inference requests to all regions are 4xx'ing

Maintenance has been completed and the region is fully restored. Expect a few slower cold starts while caches warm up.

Maintenance in the us-east1-a region is complete. Our upstream provider is still restoring access to a small number of remaining hypervisors, expected to be fully recovered by 17:45 UTC, at which point (and after verification) access to the region will be restored. Customers on our global router remain unaffected (Insofar as inference is concerned). Traffic has been failing over to other regions and providers throughout, no action required. Builds continue to remain affected If you have workloads pinned directly to us-east1-a that are still unavailable, they should recover automatically. You can also pin to a different provider or region in the meantime.

Planned maintenance is affecting new builds in the US from starting. We expect this to be resolved shortly. Inference and other services are not affected.

Maintenance ended

One of our upstream infrastructure providers (Crusoe) will be conducting planned network maintenance affecting the us-east-1 region. Most customers will not be affected. If you're on our global router, your workloads will automatically fail over to a different region and/or provider; no action is required. If you're running workloads pinned directly to Crusoe us-east-1, you may experience intermittent unavailability during the maintenance window. To avoid disruption, you can pin your application to a different provider or region for the duration, or take advantage of our global routing to handle failover automatically. All services in other regions will remain fully operational.

Planned maintenance is affecting new builds in the US from starting. We expect this to be resolved shortly. Inference and other services are not affected.

Services are operating normally again

Some issues persist with that provider

Underlying provider has resolved the issue

1 provider is down in the US East region. Existing apps should shift to available providers and regions. New deployments of L40s will be effected until the issue is resolved.

The issue has been patched upstream and Cerebrium health checks have been succeeding for the past 45 minutes. Marking as resolved.

The issue is on going and we are still communicating with the upstream provider.

The upstream provider has implemented a mitigation. Services should be restored, however we are still tracking this issue and will revert with any updates.

1 provider in US East 1 is having networking issues. Traffic should automatically reroute to others but service is degraded