Advanced Aggregations — Degraded Performance
Last updatepostmortemSep 8 · 22:48 UTC
### Summary On September 4–5, 2026 \(UTC\), we experienced an issue affecting **Advanced Aggregations** **in which some customers saw inaccurate aggregation results because recent card transaction data may have been missing**. We fixed **the issue, and backfilled the data**. Advanced Aggregations results are expected to be correct again after **2026-09-05 03:50 UTC**. ### What happened A subset of card transaction records failed to be written into one of our tables because the column used to store the payment method id could not support the size of newly generated card IDs. As a result, those specific transactions were missing from the aggregated dataset, which could lead to incorrect results for customers ingesting this type of data. An impact assessment is available upon request. No data was permanently lost: failed writes were routed to a durable deadletter queue for later reprocessing. After mitigation was deployed, we replayed the queued events and completed a backfill so that aggregations reflected the correct underlying transaction data. Incident started at 2026-09-04 01:02 UTC and resolved at 2026-09-05 03:50 UTC. ### Why it happened The root cause was a **data type mismatch**: the `cards` table `id` values grew beyond the maximum value supported by a **4-byte integer**. `id` field has 8-byte integer so there was no error storing cards. However, `card_id` field in the Advanced Aggregations schema was 4-byte, causing inserts to fail for transactions related to newer cards. To remediate safely at scale, we: * Added new **8-byte integer \(**`bigint`\) columns for the affected identifiers and updated write logic to use them. * Introduced a **compatibility read path** so reads return the correct identifier whether data is stored in the old or new columns, along with new indexes to preserve query performance. ### What we are doing about it * We’re reviewing improvements to monitor schema limit risks earlier and prevent similar issues in the future. * We’ll enhance monitoring on deadletter queue so we can detect issue like this sooner.
Reported by Sardine AI on their status page.