Outcomet Incident History

1. resolvedOct 29 · 09:19 UTC

   Resolved: access to the Outcomet app has recovered. A signed-in user confirmed that both Companies and People load again, and backend API checks have returned to normal. We are continuing the root-cause review; this update does not imply a change to contact data.

Reported by Outcomet on their status page.

1. resolvedOct 21 · 23:28 UTC

   Retrospective incident report — October 21, 2026. This incident is resolved. This entry was added after recovery; its publication timestamp is not the outage start time. Impact Between October 10 and October 21, 2026, the production database experienced sustained high CPU utilization (~87%) and elevated transaction rollback rates, causing intermittent request timeouts and temporary realtime connection drops in the Outcomet app (including Delivery and Signals views). No data loss occurred. Cause Four database RPC functions returned PostgreSQL SQLSTATE 40001 (serialization_failure) when rejecting stale optimistic-concurrency version checks. Because PostgREST treats SQLSTATE 40001 as a transient transaction-isolation conflict, two pooled API connections entered an unbounded automatic retry loop on deterministic version mismatches, continuously re-executing and rolling back the failed transactions in the background. Recovery The two stuck database backend sessions were terminated, immediately dropping the transaction rollback rate to zero and restoring normal database CPU utilization. All affected database RPC functions were updated to return SQLSTATE 23505 (HTTP 409 Conflict) for optimistic concurrency conflicts so the API layer returns conflict responses immediately without retrying. Timeline (UTC) • Oct 10, 17:15 — First pooled API session entered a persistent retry loop after an optimistic concurrency conflict. • Oct 11, 19:16 — Second pooled API session entered a persistent retry loop on a second record. • Oct 21, 22:34 — Both runaway backend sessions were identified and terminated; transaction rollbacks dropped immediately to zero. • Oct 21, 22:39 — All four RPC functions were hot-patched in production and staging to use SQLSTATE 23505 instead of 40001. • Oct 21, 23:12 — Permanent database migration, client conflict mapping, and automated regression guards passed CI and deployed to production. Prevention and verification Optimistic concurrency checks across database RPCs now exclusively use non-retryable conflict error codes (SQLSTATE 23505 / HTTP 409) rather than class-40 transaction rollback codes, enforced by automated migration regression tests in CI. We are sorry for the disruption.

Reported by Outcomet on their status page.

1. resolvedOct 13 · 19:52 UTC

   Retrospective incident report — October 13, 2026. This incident is resolved. This entry was added after recovery; its publication timestamp is not the outage start time. Impact Users experienced errors and timeouts when loading Contacts, Campaigns, Problems and Signals in the Outcomet app. The total number of affected accounts and requests, and the exact outage duration, were not measured. Cause A frontend release was published before four required corrective database migrations had reached production. The resulting mismatch left obsolete database access policies and query paths active, causing read requests to fail or time out. Recovery The corrective migrations were applied successfully. The obsolete policies were removed, sampled authenticated reads succeeded, and the inspected post-recovery database logs contained no statement timeouts. No evidence of data loss was identified during the response. Timeline (UTC, October 13) • 13:49:52 — Frontend release published. • 13:55:25 — Corrective database migrations completed. • From 13:56 — Recovery checks confirmed successful sampled reads and no statement timeouts in the inspected log window. • 19:01:26 — The first release using the new deployment gate was published successfully, after its database check completed and the same commit was promoted. The approximately 5 minutes 33 seconds between frontend publication and migration completion is the confirmed release mismatch interval, not a measurement of the full outage duration. Prevention and verification Production frontend publication now depends on successful CI, successful database migrations and deployment of any changed Edge Functions. Only then is the same commit promoted for frontend publication. Failed checks block promotion. The first automatic release through this gate was verified successfully in production; the deployment-gate remediation is complete. We are sorry for the disruption. Quantifying the full historical impact remains a follow-up.

Reported by Outcomet on their status page.