Mixpanel Incident History

1. postmortemSep 8 · 18:25 UTC

   Mixpanel RCA: Temporary Query Service Disruption for US Projects August 26, 2026

   Summary

   Between approximately 2:02 PM and 3:19 PM PT on August 26, 2026, projects in the US region experienced failures loading reports and running queries through the Mixpanel UI and Query API. During this window, some queries in the region failed or returned errors. No customer data was lost, and data ingestion was not affected. All events continued to be collected and stored normally throughout the incident; once query service was restored, all reports reflected complete, accurate data with no customer action required.

   The cause was identified as a recently deployed internal tool for diagnostic query replay, a capability our engineers use to re-run copies of past queries to debug performance, which unexpectedly wrote large amounts of data to the disks of our query servers, consuming storage capacity the servers need to operate. When those disks filled, the affected servers took themselves out of service. Service was restored, the impacted servers were brought back online, and the internal tooling was disabled. The remediations below add storage safeguards to remove internal tooling's ability to consume resources on query servers, and are designed to prevent this class of failure from happening again in the future.

   What happened

   Mixpanel's query engine runs on a fleet of servers that each use a set of local storage volumes to cache the data needed to answer queries quickly. Separately, our engineers use diagnostic query replay tooling, a capability our engineers use to reproduce and debug query performance.

   While applying the diagnostic query replay tooling to a large, complex query, a software bug caused the query replay captures to fan out across the entire fleet instead of remaining confined to a single server. Additionally, it caused far more data than intended to be saved without timely eviction on a single server. Two factors then widened the impact of the issue:

   • A single full storage volume took a server fully out of service. Each server treats its cache as unhealthy if any one of its storage volumes crosses a usage threshold, even when all other volumes are healthy. The replay data was written to one specific volume on every server, so servers across the region failed their health checks nearly simultaneously.
   • Cleanup limits did not account for data size. The safeguard limiting replay data on disk counted items at the application level rather than bytes at the filesystem level, so a small number of unexpectedly large captures passed the check while consuming most of the volume’s capacity.

   Together, these allowed a single debugging workflow that normally has a negligible footprint to interrupt production query serving across the US region.

   Timeline (Pacific Time, August 26, 2026)

   • 2:00 PM — First oversized diagnostic replay capture was written; storage volumes began to reach their capacity and query success rate began to drop shortly after.
   • 2:17 PM — Automated alerting paged the on-call engineer; investigation began immediately and additional engineers were engaged.
   • 2:39 PM — Status page incident posted; in-app banner displayed at 2:40 PM.
   • 2:53 PM — Root cause identified; recovery efforts began on the first affected server group.
   • 3:19 PM — Query service restored for the vast majority of traffic; the final server group fully recovered at 3:27 PM.
   • 3:45 PM — Incident resolved after a stable observation period. The diagnostic replay tooling that triggered the issue was fully disabled the same evening.

   Root cause

   1. A software bug in our query replay tooling caused unbounded writes to production storage. A recently deployed capability for reproducing queries mishandled a particular class of complex query, causing captures to spread to every query server in the region and to write far more data to the affected volumes than the design assumed.
   2. Replay files consumed disk capacity that query serving depends on. The replay tooling wrote its files to the query servers' local disks, so runaway replay data exhausted storage capacity the servers need to answer queries.
   3. Safeguards only partially accounted for the behavior. The cleanup policy for diagnostic replay data limited the number of items on disk, but not their total size, so it did not engage. Alerting on storage volumes flagged the growth but was not escalated as critical on a per-server basis, which delayed detection until query failures began.

   What we're changing

   The end state we are building toward: internal diagnostic query replay data stored in dedicated object storage, placing no load on production query servers.

   Already deployed:

   • Disabled the internal tooling that caused the incident, and remediated the underlying issue so diagnostic captures are confined to a single server and the specific query class is handled correctly.
   • Documented the targeted recovery procedure used during the incident, i.e., clearing only the affected storage volume rather than restarting full server groups, in our operational runbooks, shortening recovery time if any volume’s capacity runs out in the future.

   In progress:

   • Filesystem-level, size-based limits on diagnostic replay data, capping total bytes on disk rather than item counts, so oversized captures are rejected or evicted before they can affect the volume’s capacity.
   • Stricter storage alerting, escalating per-server volume saturation as critical before it can affect query health checks.
   • Moving query replay data to dedicated object storage, so it consumes no resources on production query servers.

   Common questions

   • Was any data lost? No. Data ingestion was unaffected throughout the incident: events continued to be collected, queued, and stored normally. Only the ability to query was interrupted. Once service was restored, all reports reflected complete data.
   • Were saved reports, dashboards, or project settings affected? No. The incident affected query execution only. Nothing stored in your project changed.
   • Why did it affect multiple US projects at once? The oversized diagnostic replay data written to every query server's disks at nearly the same time, and each server removes itself from service when any one volume fills. Preventing internal tooling from consuming query-server storage is a core part of our remediation work.
   • How is this prevented going forward? The tooling issue is resolved and the tooling remains disabled until size-based limits are in place. Storage alerting is being tightened so saturation is caught before it affects query serving. Structurally, we are moving diagnostic data off production query servers entirely, so internal debugging data will consume no storage on the servers responsible for queries processing.

   We apologize for the disruption and for the time reports were unavailable. Please reach out through your account team or support with any questions.

Reported by Mixpanel on their status page.

1. resolvedAug 20 · 03:05 UTC

   This incident has been resolved.

Reported by Mixpanel on their status page.