Tideways Incident History

## Summary Elevated write latency on an OpenSearch node caused a backlog in our data collection queues, leading to memory shortages and a short gap in incoming data \(13:30–13:45 UTC\) as well as delayed processing of measurement and trace data afterward. ## Timeline * **~10:00 UTC** — OpenSearch node write latency slowly starts increasing * **12:15 UTC —** Issue internally detected, investigation started * **13:30–13:45 UTC** — Data collection gap \(no data ingested\) * **13:45 UTC** — Informing our customers about known issues \(queue backlog, memory shortages, possible UI errors\) * **14:15 UTC** — Root cause identified: elevated write latency on an OpenSearch node; engaged hosting provider * **15:51 UTC** — Measurement data normalized; trace data still delayed ~30–45 min * **16:10 UTC** — Trace queue still catching up * **18:11 UTC** — All queues fully caught up, incident resolved ## Root Cause An OpenSearch node experienced elevated write latency, causing data collection queues to back up and exhaust available memory, which blocked further ingestion for a short window. ## Impact * ~15 minute total data gap across customers \(13:30–13:45 UTC\) * Delayed \(not lost\) measurement and trace data for several hours afterward * Possible transient UI errors during the incident window ## Resolution Worked with the hosting provider to resolve the underlying OpenSearch node latency issue; queues drained and fully caught up by 18:11 UTC. Additionally, increased the number of nodes in the OpenSearch cluster, massively reducing the risk of another write-performance degradation causing similar issues in the future. ## Follow-ups * Optimizing write operations in the ingest pipeline to reduce the impact of similar performance degradations * Improving alerting configuration to catch these issues earlier

All queues have fully caught up, no delays remain.

The trace queue still needs some time to fully catch up. We'll keep monitoring the situation and will update you as it progresses.

Measurement data has caught up and is now up to date. Trace data is still delayed by roughly 30-45 minutes. We're continuing to work on increasing processing throughput, but it may take some time before traces fully catch up as well.

We've identified the root cause: elevated write latency on one of our OpenSearch nodes is causing the processing backlog. We're working with our hosting provider to resolve it and will post updates as we have them.

We are currently investigating an issue regarding our data collection queues, leading to memory shortages. This might also involve error message in our UI.