Assembled

Is Assembled Down Right Now? Check if there is a current outage ongoing.

Assembled is currently Operational

Last checked from Assembled's official status page

Historical record of incidents for Assembled

Report: "Public chat widget not loading"

Last update
resolved

Issue preventing interactions with the chat widget.

Report: "Voice AI failing back to human agent because of ElevenLab outage"

Last update
identified

The issue has been identified and a fix is being implemented.

Report: "Voice AI failure from ElevenLabs"

Last update
resolved

Voice calls that used our ElevenLabs model failed due to ElevenLabs Failure

Report: "Voice failure from ElevenLabs"

Last update
resolved

Voice calls that used our ElevenLabs model failed because ElevenLab issue

Report: "Draft replies and compose from notes are experiencing problems"

Last update
resolved

From 6:24pm PT to 6:41pm PT, draft replies and compose from notes on the Assist product were unable to retrieve documents, which caused both operations to fail. The team has identified the underlying root cause and rolled back to a correct state and will be performing a full postmortem.

identified

We've identified an issue that is causing the draft replies and compose from notes to not work correctly.

Report: "Draft replies and compose from notes are experiencing problems"

Last update
Identified

We've identified an issue that is causing the draft replies and compose from notes to not work correctly.

Report: "Voice call errors"

Last update
resolved

We identified the root cause as service limits with a downstream service provider. We've fully resolved the issue and voice call performance has been restored. For calls that were routed to us during the period of degraded performance, any errors were successfully routed to the fallback destination in each customer's IVR.

monitoring

We've implemented a fix and Assist Voice call traffic is now restored for all customers.

identified

We've identified the issue and are implementing a fix.

investigating

We are currently investigating an issue with some Assist Voice calls dropping.

Report: "Voice call errors"

Last update
Monitoring

We've implemented a fix and Assist Voice call traffic is now restored for all customers.

Identified

We've identified the issue and are implementing a fix.

Investigating

We are currently investigating an issue with some Assist Voice calls dropping.

Report: "Voice calls dropping"

Last update
Monitoring

We've implemented a fix and Assist Voice call traffic is now restored for all customers.

Identified

We've identified the issue and are implementing a fix.

Investigating

We are currently investigating an issue with some Assist Voice calls dropping.

Report: "Realtime overview data failing to load"

Last update
resolved

We've resolved this issue and confirmed that functionality has been restored. We identified the root cause as a recent code change with unintended impacts to our Realtime Overview page.

monitoring

We've identified the issue and have started to restore data on the Realtime Overview page. We're continuing to confirm the root cause of this issue and monitor for reocurrences.

investigating

We're continuing to actively investigate an issue with data on our Realtime Overview page. We appreciate your patience!

investigating

We are currently investigating an issue with data on our Realtime Overview page failing to load.

Report: "Realtime overview data failing to load"

Last update
Investigating

We are currently investigating an issue with data on our Realtime Overview page failing to load.

Report: "Some data missing from the Realtime overview page"

Last update
resolved

The incident has been resolved. The team will continuing to monitor the implemented fix over the next two days to ensure stability.

monitoring

A fix has been implemented, and we're monitoring the results.

identified

For most customers who are experiencing missing certain queues from the realtime overview, the queues have now re-appeared.

identified

The issue has been identified and a fix is being implemented.

investigating

We believe we've found the cause of missing realtime overview data, and we're implementing mitigation strategies. Missing queues are gradually re-appearing as the problem resolves.

investigating

We are investigating an issue where certain queues are not visible on the realtime overview page for some customers.

Report: "Some data missing from the Realtime overview page"

Last update
Investigating

We are investigating an issue where certain queues are not visible on the realtime overview page for some customers.

Report: "Some Assist activities inaccessible"

Last update
resolved

We've confirmed that Assist is fully operational. We identified the root cause as a recent code change that unintentionally affected draft reply and dashboard access in specific cases. We've identified and fixed the issue to restore full functionality.

monitoring

We've implemented a fix and all Assist activities are now accessible. We are continuing to monitor and investigate root cause.

identified

The issue has been identified and a fix is being implemented.

investigating

Some Assist activities are currently inaccessible for some users. We are currently investigating this issue. There is no detected impact to Assist automations.

Report: "Some Assist activities inaccessible"

Last update
Resolved

We've confirmed that Assist is fully operational. We identified the root cause as a recent code change that unintentionally affected draft reply and dashboard access in specific cases. We've identified and fixed the issue to restore full functionality.

Monitoring

We've implemented a fix and all Assist activities are now accessible. We are continuing to monitor and investigate root cause.

Identified

The issue has been identified and a fix is being implemented.

Investigating

Some Assist activities are currently inaccessible for some users. We are currently investigating this issue. There is no detected impact to Assist automations.

Report: "Some Assist activities inaccessible in the dashboard"

Last update
Investigating

Some Assist activities including automations and channel configurations are currently inaccessible for some users. We are currently investigating this issue. There is no detected impact to Assist automations.

Report: "Company Settings page inaccessible for some users"

Last update
Update

We're currently investigating an issue where certain users are unable to access the Company Settings page. This appears to be affecting multiple user roles, including Standard users, Basic users, and Team Leads. Our engineering team is actively working to identify the root cause and resolve the issue as quickly as possible.

Investigating

We are currently investigating this issue.

Report: "[Resolved] Missing and incorrect agent and ticket based metrics"

Last update
resolved

This incident has been resolved.

monitoring

All affected metrics have been restored and are now up-to-date. We will continue to monitor.

monitoring

We've identified the issue in our metric computation job and have released a fix. Metrics will be restored over the next few hours. We will continue to provide updates as metrics are restored.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.

Report: "[Resolved] Missing and incorrect agent and ticket based metrics"

Last update
Resolved

This incident has been resolved.

Update

All affected metrics have been restored and are now up-to-date. We will continue to monitor.

Monitoring

We've identified the issue in our metric computation job and have released a fix. Metrics will be restored over the next few hours. We will continue to provide updates as metrics are restored.

Update

We are continuing to investigate this issue.

Update

We are continuing to investigate this issue.

Investigating

Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.

Report: "[Resolved] Delayed Salesforce Agent Syncs"

Last update
resolved

We have pushed a fix and agent state syncs have recovered for the affected accounts.

identified

Assembled is experiencing an issue with delayed Salesforce agent state syncs for some customers. We have identified a cause and are implementing a fix.

Report: "[Resolved] Delayed Salesforce Agent Syncs"

Last update
Resolved

We have pushed a fix and agent state syncs have recovered for the affected accounts.

Identified

Assembled is experiencing an issue with delayed Salesforce agent state syncs for some customers. We have identified a cause and are implementing a fix.

Report: "Missing and incorrect agent and ticket based metrics"

Last update
Investigating

Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.

Report: "Staleness in agent and ticket-based metrics"

Last update
resolved

Agent and ticket-based metrics have been refreshed as of March 30th 3:50 PM UTC. We will continue to monitor the stability and performance of our systems.

monitoring

There is some staleness in agent and ticket-based metrics after March 29 1:00am UTC as they are being recomputed using data from the upgraded database. We have verified that the metric computation jobs are running, and will continue to monitor them as they make progress.

Report: "Staleness in agent and ticket-based metrics"

Last update
Monitoring

There is some staleness in agent and ticket-based metrics after March 29 1:00am UTC as they are being recomputed using data from the upgraded database. We have verified that the metric computation jobs are running, and will continue to monitor them as they make progress.

Report: "Database upgrade validation"

Last update
In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

As a follow-up to the database upgrade, the team is performing additional validation to ensure sustained system stability. There will be a small window of performance degradation during the next hour, which should last no longer than 3-4 minutes. There will be no data loss during this period. We will provide an update once this additional validation step has been completed.

Report: "Database upgrade"

Last update
In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

Assembled will be undergoing a database upgrade and may see degraded performance from 12:00 AM to 2:00 AM on Sunday, March 30 (Eastern Time). There will be no data loss during this period.

Report: "Degraded performance on Assist requests"

Last update
resolved

All users have been switched onto a fallback model provider which should resolve all requests. Our engineering team is looking into longer term remediations for automatic fallbacks.

identified

We're currently investigating an issue where Assist related requests are failing or taking a long time to load.

Report: "Degraded performance on Assist requests"

Last update
Resolved

All users have been switched onto a fallback model provider which should resolve all requests. Our engineering team is looking into longer term remediations for automatic fallbacks.

Identified

We're currently investigating an issue where Assist related requests are failing or taking a long time to load.

Report: "Elevated errors on Assist across Draft reply, Compose from notes, and other copilot responses"

Last update
resolved

The issue affecting Assist-related products has now been completely resolved. Following our model provider change, we've successfully restored functionality across all impacted services. Our engineering team is looking into longer term solutions that will allow us to change model provides more quickly and seamlessly in the future.

monitoring

Most features should be working again as of 11:18am PT including Draft Reply, Compose from Notes, and other copilot features after we changed model providers. We're still updating our models for some workflows and Assist for Chat features.

monitoring

We're seeing elevated error rates on Assist related products. We've identified the root cause and are mitigating the issue.

Report: "Elevated errors on Assist across Draft reply, Compose from notes, and other copilot responses"

Last update
Resolved

The issue affecting Assist-related products has now been completely resolved.Following our model provider change, we've successfully restored functionality across all impacted services. Our engineering team is looking into longer term solutions that will allow us to change model provides more quickly and seamlessly in the future.

Update

Most features should be working again as of 11:18am PT including Draft Reply, Compose from Notes, and other copilot features after we changed model providers.We're still updating our models for some workflows and Assist for Chat features.

Monitoring

We're seeing elevated error rates on Assist related products. We've identified the root cause and are mitigating the issue.

Report: "[Resolved] Temporarily degraded API performance"

Last update
resolved

This is now resolved The following functionality has now been restored: viewing the Realtime Analysis page, seeing agent states from the Assembled API, and adding an ID for people via the API. We can confirm that no agent state data has been lost. From 0111 UTC to 1402 UTC, the aforementioned functionality was not available.

monitoring

Agent state data is now syncing as expected via our API and no agent state data was lost as a result of this issue. We are continuing to monitor and confirming root cause.

investigating

Assembled is investigating failures to retrieve agent state data via our API beginning yesterday around 1:00 AM UTC.

Report: "[Resolved] Temporarily degraded API performance"

Last update
Resolved

This is now resolvedThe following functionality has now been restored: viewing the Realtime Analysis page, seeing agent states from the Assembled API, and adding an ID for people via the API. We can confirm that no agent state data has been lost. From 0111 UTC to 1402 UTC, the aforementioned functionality was not available.

Monitoring

Agent state data is now syncing as expected via our API and no agent state data was lost as a result of this issue. We are continuing to monitor and confirming root cause.

Investigating

Assembled is investigating failures to retrieve agent state data via our API beginning yesterday around 1:00 AM UTC.

Report: "[Resolved] Zendesk Chat Agent States Not Syncing"

Last update
resolved

Assembled experienced a delay when syncing Zendesk agent states due to a temporary issue with Zendesk's API. The issue has now been resolved.

monitoring

We're currently investigating issues with Zendesk Chat agent states not syncing into Assembled beginning around 3/19 3:45am EST. As of 12:30pm EST, Zendesk Chat agent states are now appearing correctly in Assembled. Our team is continuing to investigate the root cause and is monitoring for recurrences.

Report: "[Resolved] Zendesk Chat Agent States Not Syncing"

Last update
Resolved

Assembled experienced a delay when syncing Zendesk agent states due to a temporary issue with Zendesk's API. The issue has now been resolved.

Monitoring

We're currently investigating issues with Zendesk Chat agent states not syncing into Assembled beginning around 3/19 3:45am EST. As of 12:30pm EST, Zendesk Chat agent states are now appearing correctly in Assembled. Our team is continuing to investigate the root cause and is monitoring for recurrences.

Report: "[Resolved] Failing Database Requests"

Last update
resolved

Between 4:27pm ET and 4:32pm ET, an infrastructural issue with one of our database nodes resulted some requests to the database failing altogether. Agent states exclusions were applied from 4:28 PM ET to 4:31 PM ET.

Report: "[Resolved] Failing Database Requests"

Last update
Resolved

Between 4:27pm ET and 4:32pm ET, an infrastructural issue with one of our database nodes resulted some requests to the database failing altogether. Agent states exclusions were applied from 4:28 PM ET to 4:31 PM ET.

Report: "[Assist] Assist Dashboard not Accessible on Reload"

Last update
resolved

From 11:23 AM Pacific time to 12:29 PM Pacific time, users were experiencing failures to load Assist. We've resolved the issue and apologize for the inconvenience.

identified

We are currently investigating an issue where the Assist dashboard is not accessible on reload. From 11:23 AM Pacific Time, users were experiencing failures to load Assist after loading the application for the first time. We have found a fix and are working to deploy.

Report: "[Resolved, now monitoring] Temporarily degraded performance"

Last update
resolved

We’ve fully resolved this issue and observed no issues with the platform’s performance during the monitoring period. We are finalizing our analysis of the root cause and are committed to ensuring our platform’s pages and reports respond swiftly.

monitoring

We’re pleased to inform you that the recent performance degradations have been resolved. We continue to actively monitor system performance to ensure stability. The performance degradations occurred from 12:44pm - 13:27pm Eastern Standard Time. If you experience any further issues, please contact our support team at support@assembledhq.com.

monitoring

Some users may still be experiencing slow load times. We are continuing to investigate and resolve the issue. We appreciate your patience.

monitoring

Some users may still be experiencing slow load times. We are continuing to investigate and resolve the issue. We appreciate your patience.

monitoring

Some users may still be experiencing slow load times. We are continuing to investigate and resolve the issue.

monitoring

Some users may still experience slow load times. We are continuing to investigate and resolve the issue.

monitoring

Assembled load times are now back to normal. We appreciate your patience here. We experienced slow loading times from 12:44 PM Eastern to 12:54 PM Eastern.

investigating

Assembled is experiencing degraded performance, which may cause slow load times.

Report: "Slowdown in email delivery"

Last update
resolved

After closely monitoring our email system for the past 24 hours, we're able to confirm it's now been functioning normally for the past few hours. All messages are being delivered at standard speeds, and no emails were lost during the recent slowdown. Our engineering team will continue to keep a close eye on the system to ensure everything runs smoothly going forward.

monitoring

Our email sending system is experiencing slower than normal delivery times. Our team is working to return to full email delivery over the next 24-48 hours. What to expect during this time: - Emails may take longer to reach recipients - No emails will be lost during this period - All messages will be delivered, just with some delay

Report: "Elevated Errors for Email Notifications"

Last update
resolved

We experienced an elevated level of errors in our email notifications from 1:43 PT to 3:45 PT. The issue has been resolved.

Report: "Draft replies, Compose from notes, and Question and Answer functionality not working"

Last update
resolved

From 6:07am PT until 6:29am PT, Assembled Assist experienced an outage affecting multiple platform features. During this period, the following functionalities were unavailable: Draft replies Compose from notes Question and Answer functionality Assist for Chat Our engineering team was alerted and identified issues with a third-party service that helps retrieve documents. All services have been fully restored as of 6:29am PT. We're very sorry for any inconvenience this may have caused and appreciate your patience while we worked to resolve this issue. If you continue to experience any problems, please contact our support team.

Report: "High error rates on Assist Copilot, API, and Automations"

Last update
resolved

We've been continuing to monitor error rates and can confirm that the issue has been fully resolved. The timeline for the final outage was from 15:20 PT to 15:34 PT.

investigating

We are currently experiencing intermittent delays and increased error rates in our response generation. Our team is actively monitoring the situation and working to minimize disruptions.

Report: "Assist admin platform is experiencing high latency"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating this issue.

Report: "Assembled is experiencing increased latencies"

Last update
resolved

Assembled was experiencing degraded performance, which was causing an elevated incidence of slow requests and page loads. The issue has been resolved.

investigating

Assembled is continuing to experience degraded performance, which is causing an elevated incidence of slow requests and page loads.

investigating

We're currently investigating issues with Assembled. We’re working to fix the problem as quickly as we can.

Report: "Slow replies on Assist"

Last update
resolved

The issue has been resolved now. Latency is back to regular range.

monitoring

A fix has been implemented and we are monitoring.

identified

We have identified the issue and working on rolling out a fix.

investigating

We are currently investigating this issue.

Report: "Delay in our integration syncs pulling live data"

Last update
resolved

We identified an issue that caused live integration updates to be delayed between 1:19 and 1:37 PST. This would have been reflected on the real-time analysis page. The issue has been resolved, and we’re making sure all syncs are fully caught up.

Report: "Slow replies on Assist"

Last update
resolved

From 1:55pm PT to 2:48pm PT, Assist requests experienced very high latencies and were very slow to load. Impact: Users experienced significantly elevated response times when making Assist requests (such as Draft reply, and Question and Answer) during this period. Details: The latency issues have been fully resolved and service has returned to normal. Our engineering team is conducting a thorough investigation to identify the root cause and implement preventive measures to avoid similar incidents in the future.

Report: "[Past Incident] Assembled App was not accessible"

Last update
resolved

Assembled experienced downtime on January 2, 2025, from 13:57:50 - 14:04:50 Eastern Time / 18:53:50 - 19:04:50 UTC. An infrastructure change impacted all requests to app.assembledhq.com. Please note that the API and Assist services were not affected during this period.

Report: "Assist - Admin Configuration Page was not loading"

Last update
resolved

On the Assist Product, admins were not able to access the Configuration Page for 12 minutes. This did not effect automations or the Zendesk App and should have only affected anyone logging into cal.assembledhq.com

Report: "[Assist] High error rates on Assist Copilot, API, and Automations"

Last update
resolved

We've been continuing to monitor error rates and can confirm that the issue has been fully resolved. The timeline for the final outage was from 3:50pm PT to 4:16pm PT. We're looking into remediation items that will allow us to switch providers more quickly for similar future outages.

monitoring

We continue to see error rates decline. We're continuing to monitor error rates and request latency.

identified

We've switched over to different service providers and are starting to see error rates decline. We're continuing to monitor error rates and request latency.

investigating

We are currently experiencing intermittent delays and increased error rates in our response generation. Our team is actively monitoring the situation and working to minimize disruptions. This issue is related to a latency problem with one of our core service providers, which is temporarily impacting our system’s performance. We're actively switching over to another core service provider to reduce error rates.

Report: "Assembled was not reachable"

Last update
resolved

Downtime information: The downtime was caused by issues with read replica connections. Assembled was down from 9:02-9:46 AM Pacific / 5:02–5:46 PM UTC

monitoring

Ticket syncs are now functioning as expected, and Assembled remains fully accessible. We are continuing to monitor the situation to ensure ongoing stability. Thank you for your patience and support.

monitoring

Assembled is now accessible but our tickets syncs are currently not functioning as intended. We are working to address this as well.

monitoring

We are continuing to monitor for any further issues.

monitoring

We have begun propagating the change to bring services back to a functioning state, including Assembled. Our team is actively monitoring the situation to ensure stability and full restoration. Thank you for your patience and understanding as we work through this.

identified

As of now, we are still deploying changes to bring Assembled back to a functioning state. Thank you for your continued understanding and patience as our teams work to resolve these issues.

identified

We are still deploying changes to bring Assembled back to a functioning state. Thank you for your continued understanding and patience as our teams work to resolve these issues.

identified

We are still developing understanding of the cause of the issue but are deploying a change to revert to the previous functioning state.

identified

We have identified the cause of this issue and are working to deploy a fix. We appreciate your patience as we continue to fix this.

investigating

We are continuing to investigate this issue.

investigating

We are currently experiencing a full outage affecting all of our services. Our team is actively investigating the issue and working to resolve it as quickly as possible. We will provide updates here every 30 minutes or as new information becomes available.

Report: "High error rates on Assist Copilot, API, and Automations"

Last update
resolved

The issue causing delays and increased error rates has been resolved. After switching to a different service provider, performance has stabilized, and error rates have significantly decreased. We'll continue to monitor for performance issues over the next few hours.

monitoring

We've switched over to different service providers and are starting to see error rates decline. We're continuing to monitor error rates and request latency.

monitoring

We are continuing to monitor for any further issues.

monitoring

We are currently experiencing intermittent delays and increased error rates in our response generation. Our team is actively monitoring the situation and working to minimize disruptions. This issue is related to a latency problem with one of our core service providers, which is temporarily impacting our system’s performance. We're actively switching over to another core service provider to reduce error rates.

Report: "Degraded syncs for various platform integrations"

Last update
resolved

From 2024-10-28 10:23 PM UTC until 2024-10-28 10:34 PM UTC, we experienced behavior that led to degraded syncs for the following platform integrations, Zendesk, Zendesk Chat, Zendesk Talk, Zoom Contact Center, Kustomer, and Salesforce. This affected the realtime analysis page as well as reporting on ticket metrics. We've backfilled historical ticket data for this period so that all ticket metrics should be back up to date and added an agent state exclusion for this time period for all affected customers to ensure that adherence is not affected. In addition, we are actively exploring and implementing remediations to prevent further degradations.

Report: "Ticket lifecycle sync issue"

Last update
resolved

This incident has been resolved.

monitoring

We have resolved the issue that was affecting a small subset of customers using our case lifecycle data. From 8:02 AM - 8:20 AM and 8:54 AM - 9:24 AM PST, case lifecycle syncs were not syncing data. The fix has been successfully implemented, and we are continuing to monitor the situation to ensure stability. The root cause has been identified and addressed. Thank you for your patience. (Resolved at 9:56 AM PT)

monitoring

We are investigating an issue affecting a small subset of customers using our case lifecycles data. We've identified the issue and are monitoring an initial fix while confirming root cause.

Report: "Autosend delivery failure"

Last update
resolved

A change was deployed earlier today at 10:52am PT which inadvertently prevented a significant number of autosends from sending for a few hours. We've rolled out a fix, so Assist should now be working as expected as of 4:14pm PT. This did not affect predrafted or autofilled replies which continued to operate normally -- this was isolated to emails that had been identified for automation. We're adding tests to prevent this from happening again and looking deeper into our processes to see what other safeguards we can add. Thank you for your patience and understanding.

Report: "Intermittent issues with Assist related features"

Last update
resolved

From 08:39AM to 08:56AM PDT, there was an outage with one of our AI vendors that has since been resolved. Impacted flows would vary across most of Assist's Zendesk App features. All services are now functioning as expected.

Report: "Data on the Realtime overview is delayed"

Last update
resolved

This incident has been fully resolved and the realtime metrics are up to date.

monitoring

We've identified the problem and have implemented an update. We're slowly recomputing Realtime metrics for customers to ensure that the Realtime dashboard is back up to date, but it may take some time for all metrics to recover.

investigating

We're currently investigating delays on metrics for the Realtime overview, which are not currently updating.

Report: "Assembled experienced a service outage"

Last update
resolved

This incident has been resolved. The application was inaccessible from 00:01 UTC to 00:21 UTC.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Service has been restored and we're currently investigating the root cause.

investigating

We are continuing to investigate this issue.

investigating

Assembled is experiencing a major service outage. Broad parts of the application are unreachable. We are currently investigating and apologize for the disruption.

Report: "Intermittent issue with Google Calendar updates"

Last update
resolved

We have resolved the issue and all Google Calendar syncs are up-to-date.

monitoring

We have identified a fix for this issue and are currently monitoring to ensure that event changes are correctly synced to impacted users.

investigating

We are currently investigating an issue with event updates in Assembled not consistently being sent to individual agents' linked Google Calendars. We appreciate your patience and will provide further updates as soon as possible.

Report: "Assembled experienced an outage due to an unusually high volume of requests"

Last update
resolved

This incident has been resolved and users should now be able to access the platform again.

monitoring

We are actively monitoring an issue causing Assembled to be down for users due to an unusually high volume of requests. Our team is working on a solution to resolve this promptly and effectively. We apologize for any inconvenience and appreciate your understanding as we work to resolve this issue and improve our systems.

Report: "Some users of Assembled experienced intermittent connectivity issues"

Last update
resolved

From 14:23:40 PST to 14:24:20 PST and 14:53:00 PST to 14:54:00 PST, some users of Assembled experienced intermittent connectivity issues. The root cause was unexpected database load. All services are now functioning as expected.

Report: "Managers and Administrators scheduling issue with Schedule Change Requests turned on"

Last update
resolved

We're delighted to inform you that the issue with the schedule change request workflow is no longer persisting. Our team has successfully resolved the issue, and all functionalities are now operating as expected. Thank you for your patience and understanding throughout this process. Should you encounter any further issues or have any questions, please don't hesitate to reach out to us. Outage time: 6:49 AM Pacific to 8:58 AM Pacific / 1349 UTC to 1558 UTC

monitoring

Our team is currently monitoring the effects of the deployed fix to determine if the issue with the schedule change request workflow persists. We are closely observing the system to ensure that all functionalities are operating as expected. Your patience during this monitoring phase is greatly appreciated. We will provide further updates as soon as we have more information. Thank you for your continued patience.

identified

Our team has produced a fix for the schedule change request workflow issue, and this solution is currently being deployed. We appreciate your patience as we work to resolve this issue. Further updates will be provided once the deployment is complete. Thank you for your understanding.

identified

We are currently investigating an issue where managers and administrators cant edit schedules when schedule change requests are enabled. Our team is actively working on a fix for this issue. We apologize for any inconvenience this may cause. Please stay tuned for further updates. Thank you for your patience.

Report: "Search results and Draft Reply are unavailable in Assembled Assist"

Last update
resolved

We've fallen back to another search provider. All queries and "Draft reply" functionality should be working as expected again.

identified

We've identified an issue with one of our service providers for our search system. We're identifying workarounds to use a different search system and working with them to remediate the outage.

investigating

We're currently investigating a problem with Assembled Assist where users aren't able to retrieve results for "Draft reply"

Report: "Realtime Overview and Forecasted vs Actual Report were unavailable"

Last update
resolved

We experienced an issue impacting the availability of the Realtime Overview and the Forecasted vs Actual Report. The issue stemmed from a recent code change, which has since been identified and resolved by reverting the code alteration. The disruption occurred between 10:07 AM and 10:25 AM Pacific Time / 17:07 - 17:25 UTC. All services are now functioning as expected. We apologize for any inconvenience caused and appreciate your patience.

Report: "Issue with logging in with SAML SSO"

Last update
resolved

SSO SAML logins are now working. We had an issue from 9:11 am PT to 10:54 am that would prevent users from logging in via SSO SAML. All other login methods were unaffected including Google SSO and Okta SSO.

investigating

We are currently investigating an issue with logging in using SSO SAML that may be affecting some customers

Report: "Degraded syncs for Zendesk integration users"

Last update
resolved

We have resolved the issue and Zendesk integrations are functioning normally.

investigating

We are experiencing degraded syncs for Zendesk integration users and are currently investigating. We apologize for the inconvenience.

Report: "Degraded syncs for certain Salesforce and Zendesk integration users"

Last update
resolved

This incident has been resolved. From 2024-05-04 12:35pm UTC until 2024-05-04 4:35pm UTC, we had a recurrence of the prior incident spanning 2024-05-02 11:41pm UTC until 2024-05-03 1:11 UTC that led to degraded syncs for certain Salesforce and Zendesk integration users. As before, this affected the realtime analysis page as well as reporting on ticket metrics. We've taken the same steps to resolution. We've backfilled historical ticket data for this period so that all ticket metrics should be back up to date and added an agent state exclusion for this time period for all affected customers to ensure that adherence is not affected. In addition, we are actively exploring and implementing remediations to prevent further degradations.

monitoring

Assembled is experiencing degraded syncs for certain Salesforce and Zendesk integration users, causing missing data. We've identified the root cause, implemented a resolution, and are monitoring. We apologize for the disruption.

Report: "Degraded syncs for certain Salesforce and Zendesk integration users"

Last update
resolved

This incident has been resolved. From 2024-05-02 11:41pm UTC until 2024-05-03 1:11 UTC, a proxy server for our Salesforce and Zendesk syncs had a configuration change and stopped pulling in data. This affected the realtime analysis page as well as reporting on ticket metrics. We identified the issue and resolved it -- all syncs are now functional. In addition, we've backfilled historical ticket data for this period so that all ticket metrics should be back up to date. We have added an agent state exclusion for this time period for all affected customers to ensure that adherence is not affected. Thank you for your patience during this incident, our engineering team will be conducting an incident review to identify remediations to prevent similar outages in the future.

identified

We have restored select sync functionality for Salesforce and Zendesk syncs. We are continuing to work on restoring remaining functionality.

identified

Assembled is experiencing degraded syncs for certain Salesforce and Zendesk integration users. We've identified a root cause and are working on resolving the issue. We apologize for the inconvenience.

Report: "Assembled services are unavailable"

Last update
resolved

Assembled was unavailable from 8:06pm PT to 8:14pm PT. We've identified and resolved the issue which was caused by an unexpected change in our infrastructure. We've remediated the issues and all systems should be running normally.

Report: "Manual tracking in the Zendesk App is unavailable"

Last update
resolved

From 3:10pm PT to 4:22pm PT, the Zendesk Application was unavailable for manual time tracking (agents would see a loading spinner). Automatic agent state tracking and reporting were unaffected and users would still receive agent states from automatic time tracking but were unable to end agent states manually. We've identified the root cause of the issue as a misconfiguration and have taken steps to remediate this for future cases.

Report: "Users with certain internet providers have intermittent connectivity issues with Assembled"

Last update
resolved

We've isolated the problem to a connectivity issue facing Xfinity / Comcast users in the western part of the United States. This issue has been resolved for all users who were reporting this issue.

monitoring

We're continuing the monitor the situation. Some users are not able to access the Assembled site. We believe this is due to a routing / DNS issue with certain internet providers (namely Xfinity / Comcast) in certain geographic locations. We're continuing to monitor the situation.

investigating

We're investigating issues that are causing customers to be unable to access Assembled

Report: "Empty reopens data"

Last update
resolved

Due to an unintended consequence of a recent update, specifically a change that was deployed on February 13th, the process for backfilling contact stats inadvertently deleted reopen data. We have identified the root cause of this issue and have taken immediate action by reverting the problematic commit. Our team has also initiated a comprehensive backfill of contact stats for all customers to restore the missing data. The issue began on February 13th and was fully resolved by February 24th. A full backfill of the impacted data has been initiated, and we anticipate that all reopens data will be completely updated and restored to its accurate state shortly.

Report: "Database Reboot"

Last update
resolved

AWS applied a system patch and rebooted our main database. This caused a complete Assembled outage between 9:44 PM - 9:45 PM PT. Impact: There was a data delay immediately after the outage until 12:45 AM PT. All data has been restored except agent states for many customers whose data syncs were disrupted between 9:44 PM - 12:45 PM PT. We are adding an exclusion for agent states in the disrupted time period.