Historical record of incidents for Assembled
Report: "Public chat widget not loading"
Last updateIssue preventing interactions with the chat widget.
Report: "Voice AI failing back to human agent because of ElevenLab outage"
Last updateThe issue has been identified and a fix is being implemented.
Report: "Voice AI failure from ElevenLabs"
Last updateVoice calls that used our ElevenLabs model failed due to ElevenLabs Failure
Report: "Voice failure from ElevenLabs"
Last updateVoice calls that used our ElevenLabs model failed because ElevenLab issue
Report: "Draft replies and compose from notes are experiencing problems"
Last updateFrom 6:24pm PT to 6:41pm PT, draft replies and compose from notes on the Assist product were unable to retrieve documents, which caused both operations to fail. The team has identified the underlying root cause and rolled back to a correct state and will be performing a full postmortem.
We've identified an issue that is causing the draft replies and compose from notes to not work correctly.
Report: "Draft replies and compose from notes are experiencing problems"
Last updateWe've identified an issue that is causing the draft replies and compose from notes to not work correctly.
Report: "Voice call errors"
Last updateWe identified the root cause as service limits with a downstream service provider. We've fully resolved the issue and voice call performance has been restored. For calls that were routed to us during the period of degraded performance, any errors were successfully routed to the fallback destination in each customer's IVR.
We've implemented a fix and Assist Voice call traffic is now restored for all customers.
We've identified the issue and are implementing a fix.
We are currently investigating an issue with some Assist Voice calls dropping.
Report: "Voice call errors"
Last updateWe've implemented a fix and Assist Voice call traffic is now restored for all customers.
We've identified the issue and are implementing a fix.
We are currently investigating an issue with some Assist Voice calls dropping.
Report: "Voice calls dropping"
Last updateWe've implemented a fix and Assist Voice call traffic is now restored for all customers.
We've identified the issue and are implementing a fix.
We are currently investigating an issue with some Assist Voice calls dropping.
Report: "Realtime overview data failing to load"
Last updateWe've resolved this issue and confirmed that functionality has been restored. We identified the root cause as a recent code change with unintended impacts to our Realtime Overview page.
We've identified the issue and have started to restore data on the Realtime Overview page. We're continuing to confirm the root cause of this issue and monitor for reocurrences.
We're continuing to actively investigate an issue with data on our Realtime Overview page. We appreciate your patience!
We are currently investigating an issue with data on our Realtime Overview page failing to load.
Report: "Realtime overview data failing to load"
Last updateWe are currently investigating an issue with data on our Realtime Overview page failing to load.
Report: "Some data missing from the Realtime overview page"
Last updateThe incident has been resolved. The team will continuing to monitor the implemented fix over the next two days to ensure stability.
A fix has been implemented, and we're monitoring the results.
For most customers who are experiencing missing certain queues from the realtime overview, the queues have now re-appeared.
The issue has been identified and a fix is being implemented.
We believe we've found the cause of missing realtime overview data, and we're implementing mitigation strategies. Missing queues are gradually re-appearing as the problem resolves.
We are investigating an issue where certain queues are not visible on the realtime overview page for some customers.
Report: "Some data missing from the Realtime overview page"
Last updateWe are investigating an issue where certain queues are not visible on the realtime overview page for some customers.
Report: "Some Assist activities inaccessible"
Last updateWe've confirmed that Assist is fully operational. We identified the root cause as a recent code change that unintentionally affected draft reply and dashboard access in specific cases. We've identified and fixed the issue to restore full functionality.
We've implemented a fix and all Assist activities are now accessible. We are continuing to monitor and investigate root cause.
The issue has been identified and a fix is being implemented.
Some Assist activities are currently inaccessible for some users. We are currently investigating this issue. There is no detected impact to Assist automations.
Report: "Some Assist activities inaccessible"
Last updateWe've confirmed that Assist is fully operational. We identified the root cause as a recent code change that unintentionally affected draft reply and dashboard access in specific cases. We've identified and fixed the issue to restore full functionality.
We've implemented a fix and all Assist activities are now accessible. We are continuing to monitor and investigate root cause.
The issue has been identified and a fix is being implemented.
Some Assist activities are currently inaccessible for some users. We are currently investigating this issue. There is no detected impact to Assist automations.
Report: "Some Assist activities inaccessible in the dashboard"
Last updateSome Assist activities including automations and channel configurations are currently inaccessible for some users. We are currently investigating this issue. There is no detected impact to Assist automations.
Report: "Company Settings page inaccessible for some users"
Last updateWe're currently investigating an issue where certain users are unable to access the Company Settings page. This appears to be affecting multiple user roles, including Standard users, Basic users, and Team Leads. Our engineering team is actively working to identify the root cause and resolve the issue as quickly as possible.
We are currently investigating this issue.
Report: "[Resolved] Missing and incorrect agent and ticket based metrics"
Last updateThis incident has been resolved.
All affected metrics have been restored and are now up-to-date. We will continue to monitor.
We've identified the issue in our metric computation job and have released a fix. Metrics will be restored over the next few hours. We will continue to provide updates as metrics are restored.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.
Report: "[Resolved] Missing and incorrect agent and ticket based metrics"
Last updateThis incident has been resolved.
All affected metrics have been restored and are now up-to-date. We will continue to monitor.
We've identified the issue in our metric computation job and have released a fix. Metrics will be restored over the next few hours. We will continue to provide updates as metrics are restored.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Some metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.
Report: "[Resolved] Delayed Salesforce Agent Syncs"
Last updateWe have pushed a fix and agent state syncs have recovered for the affected accounts.
Assembled is experiencing an issue with delayed Salesforce agent state syncs for some customers. We have identified a cause and are implementing a fix.
Report: "[Resolved] Delayed Salesforce Agent Syncs"
Last updateWe have pushed a fix and agent state syncs have recovered for the affected accounts.
Assembled is experiencing an issue with delayed Salesforce agent state syncs for some customers. We have identified a cause and are implementing a fix.
Report: "Missing and incorrect agent and ticket based metrics"
Last updateSome metrics including adherence metrics are currently incorrect or missing for customers after April 1st at 7:00AM ET. We are currently investigating this issue and working on a fix.
Report: "Staleness in agent and ticket-based metrics"
Last updateAgent and ticket-based metrics have been refreshed as of March 30th 3:50 PM UTC. We will continue to monitor the stability and performance of our systems.
There is some staleness in agent and ticket-based metrics after March 29 1:00am UTC as they are being recomputed using data from the upgraded database. We have verified that the metric computation jobs are running, and will continue to monitor them as they make progress.
Report: "Staleness in agent and ticket-based metrics"
Last updateThere is some staleness in agent and ticket-based metrics after March 29 1:00am UTC as they are being recomputed using data from the upgraded database. We have verified that the metric computation jobs are running, and will continue to monitor them as they make progress.
Report: "Database upgrade validation"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
As a follow-up to the database upgrade, the team is performing additional validation to ensure sustained system stability. There will be a small window of performance degradation during the next hour, which should last no longer than 3-4 minutes. There will be no data loss during this period. We will provide an update once this additional validation step has been completed.
Report: "Database upgrade"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
Assembled will be undergoing a database upgrade and may see degraded performance from 12:00 AM to 2:00 AM on Sunday, March 30 (Eastern Time). There will be no data loss during this period.
Report: "Degraded performance on Assist requests"
Last updateAll users have been switched onto a fallback model provider which should resolve all requests. Our engineering team is looking into longer term remediations for automatic fallbacks.
We're currently investigating an issue where Assist related requests are failing or taking a long time to load.
Report: "Degraded performance on Assist requests"
Last updateAll users have been switched onto a fallback model provider which should resolve all requests. Our engineering team is looking into longer term remediations for automatic fallbacks.
We're currently investigating an issue where Assist related requests are failing or taking a long time to load.
Report: "Elevated errors on Assist across Draft reply, Compose from notes, and other copilot responses"
Last updateThe issue affecting Assist-related products has now been completely resolved. Following our model provider change, we've successfully restored functionality across all impacted services. Our engineering team is looking into longer term solutions that will allow us to change model provides more quickly and seamlessly in the future.
Most features should be working again as of 11:18am PT including Draft Reply, Compose from Notes, and other copilot features after we changed model providers. We're still updating our models for some workflows and Assist for Chat features.
We're seeing elevated error rates on Assist related products. We've identified the root cause and are mitigating the issue.
Report: "Elevated errors on Assist across Draft reply, Compose from notes, and other copilot responses"
Last updateThe issue affecting Assist-related products has now been completely resolved.Following our model provider change, we've successfully restored functionality across all impacted services. Our engineering team is looking into longer term solutions that will allow us to change model provides more quickly and seamlessly in the future.
Most features should be working again as of 11:18am PT including Draft Reply, Compose from Notes, and other copilot features after we changed model providers.We're still updating our models for some workflows and Assist for Chat features.
We're seeing elevated error rates on Assist related products. We've identified the root cause and are mitigating the issue.
Report: "[Resolved] Temporarily degraded API performance"
Last updateThis is now resolved The following functionality has now been restored: viewing the Realtime Analysis page, seeing agent states from the Assembled API, and adding an ID for people via the API. We can confirm that no agent state data has been lost. From 0111 UTC to 1402 UTC, the aforementioned functionality was not available.
Agent state data is now syncing as expected via our API and no agent state data was lost as a result of this issue. We are continuing to monitor and confirming root cause.
Assembled is investigating failures to retrieve agent state data via our API beginning yesterday around 1:00 AM UTC.
Report: "[Resolved] Temporarily degraded API performance"
Last updateThis is now resolvedThe following functionality has now been restored: viewing the Realtime Analysis page, seeing agent states from the Assembled API, and adding an ID for people via the API. We can confirm that no agent state data has been lost. From 0111 UTC to 1402 UTC, the aforementioned functionality was not available.
Agent state data is now syncing as expected via our API and no agent state data was lost as a result of this issue. We are continuing to monitor and confirming root cause.
Assembled is investigating failures to retrieve agent state data via our API beginning yesterday around 1:00 AM UTC.
Report: "[Resolved] Zendesk Chat Agent States Not Syncing"
Last updateAssembled experienced a delay when syncing Zendesk agent states due to a temporary issue with Zendesk's API. The issue has now been resolved.
We're currently investigating issues with Zendesk Chat agent states not syncing into Assembled beginning around 3/19 3:45am EST. As of 12:30pm EST, Zendesk Chat agent states are now appearing correctly in Assembled. Our team is continuing to investigate the root cause and is monitoring for recurrences.
Report: "[Resolved] Zendesk Chat Agent States Not Syncing"
Last updateAssembled experienced a delay when syncing Zendesk agent states due to a temporary issue with Zendesk's API. The issue has now been resolved.
We're currently investigating issues with Zendesk Chat agent states not syncing into Assembled beginning around 3/19 3:45am EST. As of 12:30pm EST, Zendesk Chat agent states are now appearing correctly in Assembled. Our team is continuing to investigate the root cause and is monitoring for recurrences.
Report: "[Resolved] Failing Database Requests"
Last updateBetween 4:27pm ET and 4:32pm ET, an infrastructural issue with one of our database nodes resulted some requests to the database failing altogether. Agent states exclusions were applied from 4:28 PM ET to 4:31 PM ET.
Report: "[Resolved] Failing Database Requests"
Last updateBetween 4:27pm ET and 4:32pm ET, an infrastructural issue with one of our database nodes resulted some requests to the database failing altogether. Agent states exclusions were applied from 4:28 PM ET to 4:31 PM ET.
Report: "[Assist] Assist Dashboard not Accessible on Reload"
Last updateFrom 11:23 AM Pacific time to 12:29 PM Pacific time, users were experiencing failures to load Assist. We've resolved the issue and apologize for the inconvenience.
We are currently investigating an issue where the Assist dashboard is not accessible on reload. From 11:23 AM Pacific Time, users were experiencing failures to load Assist after loading the application for the first time. We have found a fix and are working to deploy.
Report: "[Resolved, now monitoring] Temporarily degraded performance"
Last updateWe’ve fully resolved this issue and observed no issues with the platform’s performance during the monitoring period. We are finalizing our analysis of the root cause and are committed to ensuring our platform’s pages and reports respond swiftly.
We’re pleased to inform you that the recent performance degradations have been resolved. We continue to actively monitor system performance to ensure stability. The performance degradations occurred from 12:44pm - 13:27pm Eastern Standard Time. If you experience any further issues, please contact our support team at support@assembledhq.com.
Some users may still be experiencing slow load times. We are continuing to investigate and resolve the issue. We appreciate your patience.
Some users may still be experiencing slow load times. We are continuing to investigate and resolve the issue. We appreciate your patience.
Some users may still be experiencing slow load times. We are continuing to investigate and resolve the issue.
Some users may still experience slow load times. We are continuing to investigate and resolve the issue.
Assembled load times are now back to normal. We appreciate your patience here. We experienced slow loading times from 12:44 PM Eastern to 12:54 PM Eastern.
Assembled is experiencing degraded performance, which may cause slow load times.
Report: "Slowdown in email delivery"
Last updateAfter closely monitoring our email system for the past 24 hours, we're able to confirm it's now been functioning normally for the past few hours. All messages are being delivered at standard speeds, and no emails were lost during the recent slowdown. Our engineering team will continue to keep a close eye on the system to ensure everything runs smoothly going forward.
Our email sending system is experiencing slower than normal delivery times. Our team is working to return to full email delivery over the next 24-48 hours. What to expect during this time: - Emails may take longer to reach recipients - No emails will be lost during this period - All messages will be delivered, just with some delay
Report: "Elevated Errors for Email Notifications"
Last updateWe experienced an elevated level of errors in our email notifications from 1:43 PT to 3:45 PT. The issue has been resolved.
Report: "Draft replies, Compose from notes, and Question and Answer functionality not working"
Last updateFrom 6:07am PT until 6:29am PT, Assembled Assist experienced an outage affecting multiple platform features. During this period, the following functionalities were unavailable: Draft replies Compose from notes Question and Answer functionality Assist for Chat Our engineering team was alerted and identified issues with a third-party service that helps retrieve documents. All services have been fully restored as of 6:29am PT. We're very sorry for any inconvenience this may have caused and appreciate your patience while we worked to resolve this issue. If you continue to experience any problems, please contact our support team.
Report: "High error rates on Assist Copilot, API, and Automations"
Last updateWe've been continuing to monitor error rates and can confirm that the issue has been fully resolved. The timeline for the final outage was from 15:20 PT to 15:34 PT.
We are currently experiencing intermittent delays and increased error rates in our response generation. Our team is actively monitoring the situation and working to minimize disruptions.
Report: "Assist admin platform is experiencing high latency"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Assembled is experiencing increased latencies"
Last updateAssembled was experiencing degraded performance, which was causing an elevated incidence of slow requests and page loads. The issue has been resolved.
Assembled is continuing to experience degraded performance, which is causing an elevated incidence of slow requests and page loads.
We're currently investigating issues with Assembled. We’re working to fix the problem as quickly as we can.
Report: "Slow replies on Assist"
Last updateThe issue has been resolved now. Latency is back to regular range.
A fix has been implemented and we are monitoring.
We have identified the issue and working on rolling out a fix.
We are currently investigating this issue.
Report: "Delay in our integration syncs pulling live data"
Last updateWe identified an issue that caused live integration updates to be delayed between 1:19 and 1:37 PST. This would have been reflected on the real-time analysis page. The issue has been resolved, and we’re making sure all syncs are fully caught up.
Report: "Slow replies on Assist"
Last updateFrom 1:55pm PT to 2:48pm PT, Assist requests experienced very high latencies and were very slow to load. Impact: Users experienced significantly elevated response times when making Assist requests (such as Draft reply, and Question and Answer) during this period. Details: The latency issues have been fully resolved and service has returned to normal. Our engineering team is conducting a thorough investigation to identify the root cause and implement preventive measures to avoid similar incidents in the future.
Report: "[Past Incident] Assembled App was not accessible"
Last updateAssembled experienced downtime on January 2, 2025, from 13:57:50 - 14:04:50 Eastern Time / 18:53:50 - 19:04:50 UTC. An infrastructure change impacted all requests to app.assembledhq.com. Please note that the API and Assist services were not affected during this period.
Report: "Assist - Admin Configuration Page was not loading"
Last updateOn the Assist Product, admins were not able to access the Configuration Page for 12 minutes. This did not effect automations or the Zendesk App and should have only affected anyone logging into cal.assembledhq.com
Report: "[Assist] High error rates on Assist Copilot, API, and Automations"
Last updateWe've been continuing to monitor error rates and can confirm that the issue has been fully resolved. The timeline for the final outage was from 3:50pm PT to 4:16pm PT. We're looking into remediation items that will allow us to switch providers more quickly for similar future outages.
We continue to see error rates decline. We're continuing to monitor error rates and request latency.
We've switched over to different service providers and are starting to see error rates decline. We're continuing to monitor error rates and request latency.
We are currently experiencing intermittent delays and increased error rates in our response generation. Our team is actively monitoring the situation and working to minimize disruptions. This issue is related to a latency problem with one of our core service providers, which is temporarily impacting our system’s performance. We're actively switching over to another core service provider to reduce error rates.
Report: "Assembled was not reachable"
Last updateDowntime information: The downtime was caused by issues with read replica connections. Assembled was down from 9:02-9:46 AM Pacific / 5:02–5:46 PM UTC
Ticket syncs are now functioning as expected, and Assembled remains fully accessible. We are continuing to monitor the situation to ensure ongoing stability. Thank you for your patience and support.
Assembled is now accessible but our tickets syncs are currently not functioning as intended. We are working to address this as well.
We are continuing to monitor for any further issues.
We have begun propagating the change to bring services back to a functioning state, including Assembled. Our team is actively monitoring the situation to ensure stability and full restoration. Thank you for your patience and understanding as we work through this.
As of now, we are still deploying changes to bring Assembled back to a functioning state. Thank you for your continued understanding and patience as our teams work to resolve these issues.
We are still deploying changes to bring Assembled back to a functioning state. Thank you for your continued understanding and patience as our teams work to resolve these issues.
We are still developing understanding of the cause of the issue but are deploying a change to revert to the previous functioning state.
We have identified the cause of this issue and are working to deploy a fix. We appreciate your patience as we continue to fix this.
We are continuing to investigate this issue.
We are currently experiencing a full outage affecting all of our services. Our team is actively investigating the issue and working to resolve it as quickly as possible. We will provide updates here every 30 minutes or as new information becomes available.
Report: "High error rates on Assist Copilot, API, and Automations"
Last updateThe issue causing delays and increased error rates has been resolved. After switching to a different service provider, performance has stabilized, and error rates have significantly decreased. We'll continue to monitor for performance issues over the next few hours.
We've switched over to different service providers and are starting to see error rates decline. We're continuing to monitor error rates and request latency.
We are continuing to monitor for any further issues.
We are currently experiencing intermittent delays and increased error rates in our response generation. Our team is actively monitoring the situation and working to minimize disruptions. This issue is related to a latency problem with one of our core service providers, which is temporarily impacting our system’s performance. We're actively switching over to another core service provider to reduce error rates.
Report: "Degraded syncs for various platform integrations"
Last updateFrom 2024-10-28 10:23 PM UTC until 2024-10-28 10:34 PM UTC, we experienced behavior that led to degraded syncs for the following platform integrations, Zendesk, Zendesk Chat, Zendesk Talk, Zoom Contact Center, Kustomer, and Salesforce. This affected the realtime analysis page as well as reporting on ticket metrics. We've backfilled historical ticket data for this period so that all ticket metrics should be back up to date and added an agent state exclusion for this time period for all affected customers to ensure that adherence is not affected. In addition, we are actively exploring and implementing remediations to prevent further degradations.
Report: "Ticket lifecycle sync issue"
Last updateThis incident has been resolved.
We have resolved the issue that was affecting a small subset of customers using our case lifecycle data. From 8:02 AM - 8:20 AM and 8:54 AM - 9:24 AM PST, case lifecycle syncs were not syncing data. The fix has been successfully implemented, and we are continuing to monitor the situation to ensure stability. The root cause has been identified and addressed. Thank you for your patience. (Resolved at 9:56 AM PT)
We are investigating an issue affecting a small subset of customers using our case lifecycles data. We've identified the issue and are monitoring an initial fix while confirming root cause.
Report: "Autosend delivery failure"
Last updateA change was deployed earlier today at 10:52am PT which inadvertently prevented a significant number of autosends from sending for a few hours. We've rolled out a fix, so Assist should now be working as expected as of 4:14pm PT. This did not affect predrafted or autofilled replies which continued to operate normally -- this was isolated to emails that had been identified for automation. We're adding tests to prevent this from happening again and looking deeper into our processes to see what other safeguards we can add. Thank you for your patience and understanding.
Report: "Intermittent issues with Assist related features"
Last updateFrom 08:39AM to 08:56AM PDT, there was an outage with one of our AI vendors that has since been resolved. Impacted flows would vary across most of Assist's Zendesk App features. All services are now functioning as expected.
Report: "Data on the Realtime overview is delayed"
Last updateThis incident has been fully resolved and the realtime metrics are up to date.
We've identified the problem and have implemented an update. We're slowly recomputing Realtime metrics for customers to ensure that the Realtime dashboard is back up to date, but it may take some time for all metrics to recover.
We're currently investigating delays on metrics for the Realtime overview, which are not currently updating.
Report: "Assembled experienced a service outage"
Last updateThis incident has been resolved. The application was inaccessible from 00:01 UTC to 00:21 UTC.
A fix has been implemented and we are monitoring the results.
Service has been restored and we're currently investigating the root cause.
We are continuing to investigate this issue.
Assembled is experiencing a major service outage. Broad parts of the application are unreachable. We are currently investigating and apologize for the disruption.
Report: "Intermittent issue with Google Calendar updates"
Last updateWe have resolved the issue and all Google Calendar syncs are up-to-date.
We have identified a fix for this issue and are currently monitoring to ensure that event changes are correctly synced to impacted users.
We are currently investigating an issue with event updates in Assembled not consistently being sent to individual agents' linked Google Calendars. We appreciate your patience and will provide further updates as soon as possible.
Report: "Assembled experienced an outage due to an unusually high volume of requests"
Last updateThis incident has been resolved and users should now be able to access the platform again.
We are actively monitoring an issue causing Assembled to be down for users due to an unusually high volume of requests. Our team is working on a solution to resolve this promptly and effectively. We apologize for any inconvenience and appreciate your understanding as we work to resolve this issue and improve our systems.
Report: "Some users of Assembled experienced intermittent connectivity issues"
Last updateFrom 14:23:40 PST to 14:24:20 PST and 14:53:00 PST to 14:54:00 PST, some users of Assembled experienced intermittent connectivity issues. The root cause was unexpected database load. All services are now functioning as expected.
Report: "Managers and Administrators scheduling issue with Schedule Change Requests turned on"
Last updateWe're delighted to inform you that the issue with the schedule change request workflow is no longer persisting. Our team has successfully resolved the issue, and all functionalities are now operating as expected. Thank you for your patience and understanding throughout this process. Should you encounter any further issues or have any questions, please don't hesitate to reach out to us. Outage time: 6:49 AM Pacific to 8:58 AM Pacific / 1349 UTC to 1558 UTC
Our team is currently monitoring the effects of the deployed fix to determine if the issue with the schedule change request workflow persists. We are closely observing the system to ensure that all functionalities are operating as expected. Your patience during this monitoring phase is greatly appreciated. We will provide further updates as soon as we have more information. Thank you for your continued patience.
Our team has produced a fix for the schedule change request workflow issue, and this solution is currently being deployed. We appreciate your patience as we work to resolve this issue. Further updates will be provided once the deployment is complete. Thank you for your understanding.
We are currently investigating an issue where managers and administrators cant edit schedules when schedule change requests are enabled. Our team is actively working on a fix for this issue. We apologize for any inconvenience this may cause. Please stay tuned for further updates. Thank you for your patience.
Report: "Search results and Draft Reply are unavailable in Assembled Assist"
Last updateWe've fallen back to another search provider. All queries and "Draft reply" functionality should be working as expected again.
We've identified an issue with one of our service providers for our search system. We're identifying workarounds to use a different search system and working with them to remediate the outage.
We're currently investigating a problem with Assembled Assist where users aren't able to retrieve results for "Draft reply"
Report: "Realtime Overview and Forecasted vs Actual Report were unavailable"
Last updateWe experienced an issue impacting the availability of the Realtime Overview and the Forecasted vs Actual Report. The issue stemmed from a recent code change, which has since been identified and resolved by reverting the code alteration. The disruption occurred between 10:07 AM and 10:25 AM Pacific Time / 17:07 - 17:25 UTC. All services are now functioning as expected. We apologize for any inconvenience caused and appreciate your patience.
Report: "Issue with logging in with SAML SSO"
Last updateSSO SAML logins are now working. We had an issue from 9:11 am PT to 10:54 am that would prevent users from logging in via SSO SAML. All other login methods were unaffected including Google SSO and Okta SSO.
We are currently investigating an issue with logging in using SSO SAML that may be affecting some customers
Report: "Degraded syncs for Zendesk integration users"
Last updateWe have resolved the issue and Zendesk integrations are functioning normally.
We are experiencing degraded syncs for Zendesk integration users and are currently investigating. We apologize for the inconvenience.
Report: "Degraded syncs for certain Salesforce and Zendesk integration users"
Last updateThis incident has been resolved. From 2024-05-04 12:35pm UTC until 2024-05-04 4:35pm UTC, we had a recurrence of the prior incident spanning 2024-05-02 11:41pm UTC until 2024-05-03 1:11 UTC that led to degraded syncs for certain Salesforce and Zendesk integration users. As before, this affected the realtime analysis page as well as reporting on ticket metrics. We've taken the same steps to resolution. We've backfilled historical ticket data for this period so that all ticket metrics should be back up to date and added an agent state exclusion for this time period for all affected customers to ensure that adherence is not affected. In addition, we are actively exploring and implementing remediations to prevent further degradations.
Assembled is experiencing degraded syncs for certain Salesforce and Zendesk integration users, causing missing data. We've identified the root cause, implemented a resolution, and are monitoring. We apologize for the disruption.
Report: "Degraded syncs for certain Salesforce and Zendesk integration users"
Last updateThis incident has been resolved. From 2024-05-02 11:41pm UTC until 2024-05-03 1:11 UTC, a proxy server for our Salesforce and Zendesk syncs had a configuration change and stopped pulling in data. This affected the realtime analysis page as well as reporting on ticket metrics. We identified the issue and resolved it -- all syncs are now functional. In addition, we've backfilled historical ticket data for this period so that all ticket metrics should be back up to date. We have added an agent state exclusion for this time period for all affected customers to ensure that adherence is not affected. Thank you for your patience during this incident, our engineering team will be conducting an incident review to identify remediations to prevent similar outages in the future.
We have restored select sync functionality for Salesforce and Zendesk syncs. We are continuing to work on restoring remaining functionality.
Assembled is experiencing degraded syncs for certain Salesforce and Zendesk integration users. We've identified a root cause and are working on resolving the issue. We apologize for the inconvenience.
Report: "Assembled services are unavailable"
Last updateAssembled was unavailable from 8:06pm PT to 8:14pm PT. We've identified and resolved the issue which was caused by an unexpected change in our infrastructure. We've remediated the issues and all systems should be running normally.
Report: "Manual tracking in the Zendesk App is unavailable"
Last updateFrom 3:10pm PT to 4:22pm PT, the Zendesk Application was unavailable for manual time tracking (agents would see a loading spinner). Automatic agent state tracking and reporting were unaffected and users would still receive agent states from automatic time tracking but were unable to end agent states manually. We've identified the root cause of the issue as a misconfiguration and have taken steps to remediate this for future cases.
Report: "Users with certain internet providers have intermittent connectivity issues with Assembled"
Last updateWe've isolated the problem to a connectivity issue facing Xfinity / Comcast users in the western part of the United States. This issue has been resolved for all users who were reporting this issue.
We're continuing the monitor the situation. Some users are not able to access the Assembled site. We believe this is due to a routing / DNS issue with certain internet providers (namely Xfinity / Comcast) in certain geographic locations. We're continuing to monitor the situation.
We're investigating issues that are causing customers to be unable to access Assembled
Report: "Empty reopens data"
Last updateDue to an unintended consequence of a recent update, specifically a change that was deployed on February 13th, the process for backfilling contact stats inadvertently deleted reopen data. We have identified the root cause of this issue and have taken immediate action by reverting the problematic commit. Our team has also initiated a comprehensive backfill of contact stats for all customers to restore the missing data. The issue began on February 13th and was fully resolved by February 24th. A full backfill of the impacted data has been initiated, and we anticipate that all reopens data will be completely updated and restored to its accurate state shortly.
Report: "Database Reboot"
Last updateAWS applied a system patch and rebooted our main database. This caused a complete Assembled outage between 9:44 PM - 9:45 PM PT. Impact: There was a data delay immediately after the outage until 12:45 AM PT. All data has been restored except agent states for many customers whose data syncs were disrupted between 9:44 PM - 12:45 PM PT. We are adding an exclusion for agent states in the disrupted time period.