Lucidworks Platform Incident History

1. postmortemAug 18 · 00:18 UTC

   Summary

   On August 7, 2026, Lucidworks was alerted to several application credentials being invalidated across Connected Search, Lucidworks AI, and Lucidworks Platform, and declared a Sev1 incident. Lucidworks found that a regular cleanup job had begun erroneously designating active integrations for deletion starting on August 5, 2026, resulting in credentials that were actively in use being incorrectly invalidated and recreated. Lucidworks later determined that this same issue also invalidated embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking. We disabled the cleanup job on August 7 to stop further disruptions, rotated credentials for confirmed impacted customers, and deployed a permanent fix on August 12. No further customer-facing errors have been detected since the fix was deployed.

   Root Cause

   A recent code change to the backend service that manages customer integrations with a third-party authentication and authorization provider altered how that service checked for outdated integrations across a customer's full set of applications. As a result, the service could only see a partial view of each customer's active integrations rather than the complete list. Due to this, it incorrectly treated valid, active integrations as no longer in use and recreated them, generating new backend credentials in the process. Recreating an integration this way had a secondary effect: it also invalidated the embed tokens used by customer-embedded Agent Studio widgets, Analytics signal collection Beacons, and usage metrics tracking, in addition to the backend Platform credentials. This behavior occurred in irregular, clustered bursts rather than on a steady, predictable schedule, which made the resulting customer impact appear intermittent and difficult to correlate to a single cause.

   Lucidworks determined the root cause by correlating the timing of the incorrect credential and embed token changes with a recent code change to the affected service, and confirmed the diagnosis by tracing how that service determined which integrations were still active. Lucidworks confirmed the fix by deploying a corrected version of the service and validating that credential and embed token regeneration had stopped.

   Lucidworks Actions

   Lucidworks has taken and will take the following actions as a result of this incident:

   • Disabled the automated integration-reconciliation process to immediately stop further credential and embed token disruptions
   • Deployed a corrected version of the affected service and re-enabled the reconciliation process after validating the fix
   • Rotated third-party authentication and authorization credentials for confirmed impacted customer environments
   • We will audit and add consistent customer, application, and request tracking to logs across affected services to reduce future diagnosis time
   • We will evaluate an underlying third-party library defect identified during this investigation and determine whether an upgrade can remove the need for its current workaround
   • We will review how integration recreation is handled to reduce the blast radius of future defects, so that a single error cannot invalidate embed tokens for unrelated functionality
   • We will review our proactive customer outreach process to ensure affected customers are notified consistently, regardless of whether a customer contacts Support directly themselves

   Recommended Client Actions

   Lucidworks previously recommended that affected customers verify that any embedded Agent Studio widgets, Analytics signal collection Beacons, or usage metrics tracking are functioning normally following this incident, and contact Lucidworks Support if authentication errors persist. Lucidworks communicated this recommendation directly to affected customers prior to publication of this report.

Reported by Lucidworks Platform on their status page.