Object Storage - Increased latency in eu-central-1
Last updatepostmortemSep 8 · 12:25 UTC
# Root Cause Analysis ## What happened? Starting approximately 19:00 UTC on 24 August 2026, customers accessing S3 Object Storage in the Frankfurt \(FRA4\) data center experienced elevated latency across all operations - uploads, downloads, metadata requests, and deletions. Intermittent HTTP 503 Service Unavailable and 404 Not Found errors were observed on object read requests. The impact was measurable for all customer which had buckets in the both affected datacenters of the region, with some customers experiencing severe degradation depending on their bucket configuration and access patterns. The incident remained in progress with high priority from 25 August through 3 September 2026. The latency issue was mitigated at approximately 13:40 UTC on 3 September 2026. ## How was this possible? \(Root Cause\) **Primary cause - Software bug in Quality of Service \(QoS\) subsystem** IONOS S3 Object Storage in the Frankfurt region is using a distributed object storage system. A QoS feature of this service - implemented via a redis-qos service - applies rate limiting to S3 requests at the cluster level. A bug in the S3 service leads to not well distributed queries to the Redis-QOS service \(which is served by multiple server for high availability\) this caused the Redis-QOS process to reach and sustain 100% CPU utilization, progressively slowing down all S3 request processing at the cluster level. This affected every request passing through the affected nodes, irrespective of the operation type or the specific bucket accessed. This is an internal defect within the software used. The bug caused the S3 service to consume all available resources before load reached request levels that would normally trigger throttling, meaning the degradation occurred continuously rather than only under peak conditions. In close cooperation with the software vendor, we disabled the QoS rate-limiting function on 3 September, which fully resolved the latency issue. S3 is currently operating without some QoS features while a permanent fix is prepared by the vendor. ## What are we doing to prevent recurrence? **Already completed:** * QoS disabled in the S3 service - fully resolved the latency issue. \(DONE\) * Requesting permanent solution from the software vendor. \(INPROGRESS\) **Short-term - ETA: within 2 weeks:** * Permanent fix for the Cloudian QoS bug: IONOS Cloud is in active coordination with the vendor to obtain and deploy a fix for the redis-qos defect. Once the fix is validated, lost QoS features will be re-enabled. * Database partition monitoring: We are implementing monitoring that alerts on partition size growth before any individual partition approaches a problematic threshold. This will allow our team to identify and address bucket layout issues proactively. **Mid-term - ETA: 1 to 3 months:** * QoS architecture review: Following the permanent QoS fix, we will review the architectural isolation of the QoS service together with the vendor to ensure that a future resource contention event in the rate-limiting layer cannot propagate to the request path at the same scale. * Monitoring and alerting improvements: We are extending cluster-level monitoring to surface redis-qos CPU saturation and database compaction backlog as first-class incident signals, with automated escalation before customer-visible latency develops. ## Closing remarks An incident of this duration in a core infrastructure service is not acceptable. The high-latency period persisted for nine days, during which customer workloads depending on S3 in the Frankfurt region were degraded. Multiple optimisations and mitigation strategies were implemented during the course of the incident, but could only improve the situation for individual buckets and only to a certain extent. Detecting the underlying QoS bug and developing a mitigation required coordination with the vendor’s engineering team. While the latency issue is mitigated, we remain in close contact with the vendor. The engineering work to deliver a permanent QoS fix, reduce database partition pressure, and prevent recurrence is in progress. We are also working closely with our technology partner to understand delays in the analysis of the root cause of this incident. We will conduct a joint post mortem to identify areas where collaboration during incidents can be improved. We recognise the impact this incident caused to your operations. We believe that the listed measures will help us prevent similar error patterns and speed up analysis and recovery for software related issues in the future. We thank you for your patience during the incident.
Reported by IONOS Cloud on their status page.