IONOS Cloud Status

MongoDB Playground and Business Edition clusters are currently not available for provisioning in de/fra/2 (Frankfurt East). This is due to a capacity limitation in that location that prevents these cluster types from being created reliably, and it is expected to persist for the time being. We will update this status page as soon as this limitation is lifted. As an alternative, you can: - Provision your Playground or Business Edition cluster in another available location, or - Use MongoDB Enterprise Edition, which remains available in Frankfurt East.

We are closing the incident as no further error spikes were recorded. We will share the trigger of this incident once the analysis has been conducted.

Error rates have significantly decreased. We are monitoring the situation further and will resolve the incident if the situation stays as is.

The mitigation efforts have lead to first improvements. We have downgraded the incident severity. However, we still see elevated error rates and are working towards resolving the remaining issues.

Our Object Storage Team has identified an issue connected with gateways in the affected region and is currently mitigating the problem. Customers may see connectivity issues and timeouts. We will keep you updated on the mitigation efforts.

We are currently investigating increased error rate for S3 in eu-central-3

The team found the issue in a bad network setting / routing. This was fixed, so the service is back again. It happened 12:45 - 13:20 UTC today, but now all endpoints are available. We are monitoring the situation, and we will provide an analysis once the investigation was finished.

Currently we face major availability problems in our Object Storage service in Frankfurt. You might encounter degraded access to your buckets and objects in de (eu-central-1). The team is investigating the issue and trying to fix it as soon as possible.

The AI Model Hub is currently experiencing high demand. As a shared infrastructure environment, this temporary surge in utilization may result in increased latency and varied performance across the platform. Current Impacts - Increased Latency (TTFT): High GPU utilization may cause delayed responses (Time to First Token) when initiating model requests. We are continuously managing platform resources to maximize availability across all concurrent workloads. - Llama 405B Constraints: Due to its scale and high compute requirements, the Llama 405B model is particularly sensitive to traffic spikes, leading to potential timeouts or higher latency—we recommend switching to alternative models for time-critical workloads requiring faster response times. - Collections: Collections can also be influenced negatively by high demand and utilization. Please see the notes on new deployments in our documentation: https://docs.ionos.com/cloud/ai/ai-model-hub/how-tos/document-collections Recommendations: - To mitigate temporarily performance or availability issues, we encourage our customers and partners to implement (exponential backoff) retry mechanisms in their projects and pipelines. Usage peaks that lead in timeouts and degraded performance are usually transient. - We kindly ask our customers to not create additional Support Tickets for questions and reports related to the performance of models in the Model Hub. Our Product and Tech Teams are aware and are actively monitoring and working on improving the service to meet demand. - We ask Llama 405B users that face issues to consider a switch to GTP-OSS-120B instead. In most real world use cases, the model is more cost efficient and delivers a more robust performance Outlook The following measures are currently underway to improve the performance of the AI Model Hub: - Various (model specific) optimizations are rolled out on a regular basis - Improved cross-GPU load balancing (ETA July) - Adjustments to error codes (replacing HTTP 5xx with HTTP 429) to allow clients to better understand and react to situations

Currently, there is an increased processing time for provisioning orders that are initiated via Data Center Designer or API. Occasionally, connections may be lost in the direction of the Data Center Designer. Availability and accessibility of your virtual data center resources will remain unaffected. We will inform you as soon as the functionality has been restored.

We are currently experiencing significant load, which is affecting the response times of our Modelhub. Our team is working on removing performance bottlenecks.

We are currently investigating increased latency and error count in the AI Model Hub