Assembled Incident History

This incident has been resolved.

All affected metrics have been restored and are now up-to-date. We will continue to monitor.

We've identified the issue in our metric computation job and have released a fix. Metrics will be restored over the next few hours. We will continue to provide updates as metrics are restored.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.

We have pushed a fix and agent state syncs have recovered for the affected accounts.

Assembled is experiencing an issue with delayed Salesforce agent state syncs for some customers. We have identified a cause and are implementing a fix.

Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.

There is some staleness in agent and ticket-based metrics after March 29 1:00am UTC as they are being recomputed using data from the upgraded database. We have verified that the metric computation jobs are running, and will continue to monitor them as they make progress.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

As a follow-up to the database upgrade, the team is performing additional validation to ensure sustained system stability. There will be a small window of performance degradation during the next hour, which should last no longer than 3-4 minutes. There will be no data loss during this period. We will provide an update once this additional validation step has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Assembled will be undergoing a database upgrade and may see degraded performance from 12:00 AM to 2:00 AM on Sunday, March 30 (Eastern Time). There will be no data loss during this period.

All users have been switched onto a fallback model provider which should resolve all requests. Our engineering team is looking into longer term remediations for automatic fallbacks.

We're currently investigating an issue where Assist related requests are failing or taking a long time to load.

The issue affecting Assist-related products has now been completely resolved.Following our model provider change, we've successfully restored functionality across all impacted services. Our engineering team is looking into longer term solutions that will allow us to change model provides more quickly and seamlessly in the future.

Most features should be working again as of 11:18am PT including Draft Reply, Compose from Notes, and other copilot features after we changed model providers.We're still updating our models for some workflows and Assist for Chat features.

We're seeing elevated error rates on Assist related products. We've identified the root cause and are mitigating the issue.

This is now resolvedThe following functionality has now been restored: viewing the Realtime Analysis page, seeing agent states from the Assembled API, and adding an ID for people via the API. We can confirm that no agent state data has been lost. From 0111 UTC to 1402 UTC, the aforementioned functionality was not available.

Agent state data is now syncing as expected via our API and no agent state data was lost as a result of this issue. We are continuing to monitor and confirming root cause.

Assembled is investigating failures to retrieve agent state data via our API beginning yesterday around 1:00 AM UTC.

Assembled experienced a delay when syncing Zendesk agent states due to a temporary issue with Zendesk's API. The issue has now been resolved.

We're currently investigating issues with Zendesk Chat agent states not syncing into Assembled beginning around 3/19 3:45am EST. As of 12:30pm EST, Zendesk Chat agent states are now appearing correctly in Assembled. Our team is continuing to investigate the root cause and is monitoring for recurrences.

Between 4:27pm ET and 4:32pm ET, an infrastructural issue with one of our database nodes resulted some requests to the database failing altogether. Agent states exclusions were applied from 4:28 PM ET to 4:31 PM ET.