Tideways Incident History

1. resolvedOct 21 · 14:10 UTC

   Updates went smooth, everything looks good on our end. We will keep monitoring our systems closely for a few more hours. Thank you for your patience and best of luck with your updates.

Reported by Tideways on their status page.

1. ResolvedOct 21 · 14:10 UTC

   Updates went smooth, everything looks good on our end. We will keep monitoring our systems closely for a few more hours.Thank you for your patience and best of luck with your updates.

Reported by Tideways on their status page.

1. resolvedOct 21 · 01:51 UTC

   Recovered on its own for now.

Reported by Tideways on their status page.

1. ResolvedOct 21 · 01:51 UTC

   Recovered on its own for now.

Reported by Tideways on their status page.

1. resolvedOct 20 · 16:19 UTC

   All queues have caught up. Data is up-to-date and everything is operational.

Reported by Tideways on their status page.

1. ResolvedOct 20 · 16:19 UTC

   All queues have caught up. Data is up-to-date and everything is operational.

Reported by Tideways on their status page.

1. postmortemOct 16 · 14:53 UTC

   Summary

   A maintenance query to delete old entries in a large table was not adequately optimized and required a lot of load. This brought the primary database server under heavy load. Cancelling the query lead to a rollback that also took a while for the system to recover finally.

   Customers experienced a dataloss of incoming data between 13:42 and 14:28 UTC time on October 16th, 2026.

   We are very sorry for the inconvenience and you can be assured that we are following up on this incident to make sure a similar mistake does not happen again and is otherwise softened by better automatic recovery and retry mechanisms.

   Timeline

   • 13:30 UTC - Maintenance query to large table was started.
   • 13:42 UTC - Database server load increased to a level that made UI/Ingest unavailable.
   • 13:49 UTC - query was killed and started to roll back.
   • 14:28 UTC - system recovered and load normalized.

   Root Cause

   An unoptimized maintenance SQL query to cleanup a large table caused high load.

   Impact

   • ~45 minute total data gap across customers (13:42–14:28 UTC)
   • Possible transient UI errors during the incident window

   Follow-ups

   • Current defined best-practices from Ops team were not followed for the maintenance query, so work is being done on reviewing them in the team and potentially surfacing them during code-reviews automatically.
   • We are investigating how the retry mechanism of daemons from customer machines worked and if we can optimize and improve it for the future.

Reported by Tideways on their status page.

1. ResolvedOct 16 · 14:33 UTC

   Database is responsive again.

Reported by Tideways on their status page.

1. resolvedOct 9 · 16:15 UTC

   Trace processing speeds are back to normal. The backlog is currently processed and will be fully caught within the next 30-60 minutes.

Reported by Tideways on their status page.

1. postmortemJul 13 · 13:35 UTC

   Summary

   Elevated write latency on an OpenSearch node caused a backlog in our data collection queues, leading to memory shortages and a short gap in incoming data (13:30–13:45 UTC) as well as delayed processing of measurement and trace data afterward.

   Timeline

   • ~10:00 UTC — OpenSearch node write latency slowly starts increasing
   • 12:15 UTC — Issue internally detected, investigation started
   • 13:30–13:45 UTC — Data collection gap (no data ingested)
   • 13:45 UTC — Informing our customers about known issues (queue backlog, memory shortages, possible UI errors)
   • 14:15 UTC — Root cause identified: elevated write latency on an OpenSearch node; engaged hosting provider
   • 15:51 UTC — Measurement data normalized; trace data still delayed ~30–45 min
   • 16:10 UTC — Trace queue still catching up
   • 18:11 UTC — All queues fully caught up, incident resolved

   Root Cause

   An OpenSearch node experienced elevated write latency, causing data collection queues to back up and exhaust available memory, which blocked further ingestion for a short window.

   Impact

   • ~15 minute total data gap across customers (13:30–13:45 UTC)
   • Delayed (not lost) measurement and trace data for several hours afterward
   • Possible transient UI errors during the incident window

   Resolution

   Worked with the hosting provider to resolve the underlying OpenSearch node latency issue; queues drained and fully caught up by 18:11 UTC.

   Additionally, increased the number of nodes in the OpenSearch cluster, massively reducing the risk of another write-performance degradation causing similar issues in the future.

   Follow-ups

   • Optimizing write operations in the ingest pipeline to reduce the impact of similar performance degradations
   • Improving alerting configuration to catch these issues earlier

Reported by Tideways on their status page.