Sardine AI Incident History

1. postmortemOct 9 · 14:29 UTC

   Summary

   On 1 October 2026, from approximately 14:20 to 15:45 UTC, our EU environment experienced increased latency and timeouts on the /v1/customers, /v1/issuing/risks, and /v1/feedbacks endpoints. We added service capacity to end the main customer-impact period and subsequently deployed a code change that resolved the underlying cause.

   Customer impact

   Customers using the affected EU endpoints experienced slow responses, errors, and timeouts.

   The verified affected endpoints were POST /v1/customers, POST /v1/issuing/risks, and POST /v1/feedbacks. Other measured endpoints did not show comparable latency increases. The US environment and other environments were not affected.

   What happened and why

   Under peak EU traffic, a service supporting these endpoints allocated memory faster than it reclaimed it. Part of this work involved unnecessary database reads and writes to calculate a crypto-recipient feature even when the request contained no crypto recipient address. The service's memory management was also not configured to account for its operating memory limit.

   As instances reached that limit, they became unhealthy and restarted. Their traffic moved to the remaining instances, increasing the load and causing further restarts. The EU environment had insufficient spare capacity, and our automatic scaling settings did not add capacity quickly enough to contain the failure.

   Our monitoring did not give us a clear, sustained signal of the underlying problem. Some latency alerts cleared on their own and were not thoroughly investigated, and we lacked dedicated alerts for this service's memory pressure and repeated restarts. A customer report escalation brought the disruption to our attention. We also mistook a temporary improvement for recovery while requests were still affected.

   Resolution

   At 15:33 UTC, we increased service capacity in the EU environment. Tail latency across all three affected endpoints returned to its usual level at approximately 15:42 UTC, and service restarts had stopped by approximately 15:45 UTC, ending the main customer-impact period. This capacity increase was a mitigation rather than the root-cause fix.

   At 17:59 UTC, we deployed a code change that skips unnecessary crypto-recipient database operations when no recipient address is present. Post-deployment measurements showed a 79% reduction in memory use, a 40% reduction in CPU time, and a 67% reduction in garbage-collection activity. This change eliminated more than 99% of the unnecessary database reads and writes and resolved the underlying memory-allocation cycle.

   Timeline (UTC)

   All events below occurred on 1 October 2026.

   • 14:20: Customer-impact window began.
   • 14:39: Our on-call engineer received and acknowledged a latency alert for this incident.
   • 14:44: A customer reported the disruption to us.
   • 15:03: We declared an incident and began a coordinated response.
   • 15:33: We increased service capacity.
   • 15:42: Tail latency across the affected endpoints returned to its usual level.
   • 15:45: The customer-impact window ended; service restarts had stopped and the expanded capacity was ready.
   • 17:59: We deployed the root-cause code fix to skip unnecessary database operations.

   Prevention and action items

   • We have added capacity in the EU and raised the memory available to the service.
   • We have changed the service so it manages memory against its real limit.
   • We are correcting the automatic scaling trigger so it adds capacity before instances fail.
   • We are adding alerts for memory pressure, repeated restarts and out-of-memory terminations, and per-client latency so we can detect this type of issue ourselves and notify customers first.
   • We deployed the root-cause code fix that skips unnecessary database operations when no crypto-recipient address is present.
   • We are continuing work to reduce memory use per request for high-volume business workloads.

Reported by Sardine AI on their status page.

1. postmortemSep 29 · 21:43 UTC

   Summary

   On September 27–28, 2026, Sardine experienced two periods of elevated latency and request timeouts in our Australian production region. Requests to /v1/issuing/risks , /v1/customers, and /v1/feedbacks were affected from 12:16–22:00 UTC on September 27 and again from 23:20 UTC on September 27 until 00:23 UTC on September 28.

   A separate high-volume data-processing workload overloaded a shared regional database. An unbounded historical aggregation initially consumed most of the database capacity. As records accumulated, stale database statistics caused another frequently used transaction lookup to select an inefficient query plan. Query timeouts then recycled database connections, repeatedly reintroducing that plan and amplifying the load.

   The incident affected API availability, transaction persistence, and the risk level evaluation.

   Timeline

   All times UTC.

   • September 27, 12:16 — A high-volume data-processing workload begins. Database CPU reaches capacity and /v1/issuing/risks timeouts start.
   • 14:20 — We receive the first customer report.
   • 14:29 — The primary support on-call acknowledges a page, but the issue is not escalated effectively.
   • 18:02 — We send the first substantive customer impact update.
   • 18:05 — The need for engineering escalation is identified.
   • 21:06 — Sardine declares an incident.
   • 21:38 — Active engineering investigation and remediation begin.
   • 22:03 — A database resize and primary switchover restore customer-facing service.
   • 22:12 — The high-volume workload is paused and database utilization returns to normal.
   • 22:40 — The first event is marked resolved.
   • 22:49 — Database CPU monitors are added for all production regions.
   • 23:20 — The workload resumes and triggers a second degradation.
   • 23:41 — The new database monitor pages the on-call team and is acknowledged within 13 seconds.
   • September 28 00:14 — A second database resize and primary switchover resolve the inefficient transaction lookup.
   • 00:23 — Optional aggregation work is disabled and the remaining customer-facing timeouts end.
   • 00:26 onward — Database statistics are refreshed and an optimized composite index is deployed.

   Root Cause

   The incident resulted from the interaction of four conditions:

   1. Unbounded aggregation: A synchronous query scanned the full transaction history for given condition. This consumed most database capacity when the high-volume workload began.
   2. Inefficient transaction lookup: As new records accumulated, database statistics did not refresh quickly enough. The query planner selected an index that scanned a customer’s growing transaction history rather than performing a targeted lookup.
   3. Connection recycling: Timed-out queries caused database connections to close. Replacement connections repeatedly began with the same inefficient query plan, creating a feedback loop of expensive scans, timeouts, and reconnects.

   Contributing Factors

   • Our API monitoring excluded the relevant client-timeout responses, and we did not alert on query cancellations or sustained database-connection churn. Database CPU monitor was only configured for non-pagerduty alert
   • The customer detected the first event before our monitoring did. Delayed engineering escalation extended the response time.
   • Our first recovery addressed the immediate pressure but did not prevent the workload from triggering a second event when it resumed.

   Resolution and Recovery

   For the first event, we resized the database and switched to the new primary, then paused the high-volume workload. For the second event, we again resized and switched the primary, then disabled optional aggregation to remove the remaining load.

   After service recovery, we:

   • Refreshed database statistics.
   • Deployed a composite index that gives both new and established connections an efficient transaction-lookup path.
   • Added database CPU monitors for every production region.
   • Removed optional, unused aggregation from the /v1/customers path
   • Kept the Australian production database at a higher baseline capacity while we complete the permanent query fixes and capacity review.

   Corrective Actions

   We are taking the following actions:

   Database and application resilience

   • Replace the affected transaction lookup with an atomic, consistently indexed operation.
   • Skip unnecessary historical aggregation and bound any required fallback query.
   • Evaluate a read pool so issuing-risk reads do not compete with customer-event writes on the primary.
   • Review steady-state database capacity after the query fixes are complete.

   Monitoring and response

   • Expand monitoring for client timeouts and database metrics
   • Improve incident routing so customer reports, support tickets, and on-call escalation converge quickly.

Reported by Sardine AI on their status page.