Fini AI Incident History

1. resolvedOct 20 · 16:30 UTC

   20 minutes of increased latency or missed replies due to server lockup. Fix has been pushed

Reported by Fini AI on their status page.

1. postmortemOct 14 · 23:46 UTC

   Summary

   On 14th October, our shared background-processing infrastructure reached its configured memory limit. This caused some requests that depended on background job creation or processing to fail or experience delays.

   Customer impact

   During the incident, some integration requests could not be successfully queued for processing and returned errors. Background workers also encountered memory-related errors.

   The full impact window is from 21:10 UTC - 22:50 UTC.

   Root cause

   Our background-processing system uses Redis to store queued jobs and recent processing history. A concentrated workload generated multiple jobs containing unusually large, repeated copies of conversation history. Completed and failed jobs retained these payloads for diagnostic purposes.

   At the same time, historical document-processing events were retaining large result payloads separately from their job records. This created substantial baseline memory usage.

   The combination exhausted Redis’s configured memory limit. Redis then rejected operations that required additional memory, affecting other workloads sharing the instance.

   Our retention controls limited the number of jobs and events, but did not sufficiently constrain their total memory footprint. Shared capacity also allowed one workload’s memory consumption to affect others.

   Mitigation

   The Redis memory limit was increased, and subsequent inspection confirmed available memory headroom. This addressed the immediate capacity constraint.

   We are separately validating processing recovery and reconciling requests whose queue submission failed or could not be confirmed.

   Corrective and preventive actions

   The following changes are planned to address the underlying causes:

   • Reduce queued payloads by storing references instead of repeated conversation histories.
   • Reduce retained job and event-history data, including large document-processing results.
   • Introduce stronger workload isolation and per-customer resource controls.
   • Improve alerts for memory growth, oversized payloads, and queue submission failures.
   • Reconcile affected requests and reprocess them where appropriate, with duplicate-processing safeguards.

   Our investigation supports rapid accumulation of oversized retained data as the immediate cause; the source of the repeated workload is fully recovered and functiona

Reported by Fini AI on their status page.

1. resolvedSep 29 · 15:01 UTC

   This incident has been resolved.

Reported by Fini AI on their status page.

1. ResolvedSep 29 · 15:01 UTC

   This incident has been resolved.

Reported by Fini AI on their status page.