Historical record of incidents for Ada
Report: "Voice Channel Unavailable in some ai agents"
Last updateWe've identified the cause of the issue and are working with our vendor to resolve the issue. For more details check here https://status.elevenlabs.io/
Issue: Some ai agents are not able to service callers over the voice channel. Who it affects: ai agents who are using non-SSML variants of voices for their ai agents will be affected. We recommend changing your voice configurations to use an SSML variant to avoid this behaviour while we investigate the root cause.
Report: "Increased Latency in Ai Agent Performance"
Last updateIssue: Some Ada bots are experiencing a longer than typical time to load initial conversation. In some cases conversations may fail to load to users. Who it affects: newly opened conversations may be impacted by this.
Report: "Increased Latency in Ai Agent Performance"
Last updateWe've observed that some declarative bots may experience increased latency or failure to load as well.
Issue: Some Ai Agent are experiencing a longer than typical time to load initial conversation. In some cases conversations may fail to load to users. Who it affects: Currently the issue has been seen to affect generative ai agents.
Report: "Investigating issues with Ada Chat services in EU"
Last updateSome customers are experiencing issues with our chat services. Chatters interacting with AI Agents on the EU cluster may experience slow or no response. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Knowledge Base Articles Sync Currently Not Syncing"
Last updateA number of knowledge articles from external sources (ex. Zendesk and Salesforce) are not being synced correctly. We have identified the cause of the issue, and are working on the best way to restore these articles.
Report: "Database Upgrade"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Our Database will be undergoing an update, we do not expect any service disruptions during this time.This update only affects chatbots and ai agents in the US cluster.This can be identified by the format of the address used to access the Ada dashboard (https://[bot-handle].ada.support).
Report: "Bot Latency across all channels for the US Cluster"
Last updateLatency was observed, and end users were presented with the fallback message when attempting to interact with bot.
Report: "Investigating issues with Ada Voice service"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results. The issue is currently limited to utilization of Boosted Terms. All other related issues are resolved at this time.
Some customers are experiencing issues with our Voice services, which are affecting chatters interacting with Ada Voice bots. We’re aware of the issue, and our team is actively working to get everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Latency issues with Ada Chat have been identified"
Last updateThis incident has been resolved.
Services are now stable and we are continuing to monitor.
We are continuing to work on a fix for this issue.
Our team has identified a latency issue with our third party service providers impacting all clusters. Apologies for this inconvenience. We will continue to keep you up to date every 30 minutes.
Report: "Increased Latency in Ai agents"
Last updateInitial changes are available. Some longer term improvements are also scheduled to improve overall performance and these will be added as general improvements to the Product.
A fix is available and we'll be monitoring results. Some users may take longer to see improvements as the fix is propagated.
The issue has been identified and a fix is being implemented.
Issue: Some AI agents are experiencing an increase in their response time to user messages. Who it affects: We're still investigating the full impact. While not all ai agents are affected we have confirmed multiple agents are experiencing an increase in response time.
Report: "Some users are unable to share files with agents over chat"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Issue: Chatters speaking to agents on Zendesk Messaging are unable to share files over chat. Who it affects: ai agents and bots that are using the Zendesk Messaging integration may be affected by this issue.
Report: "Early rate limit enforcement on Ada API's"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Issue: Some clients are seeing unexpected rate limiting on calls to Ada's API's Who it affects: Clients or sessions making calls to our Email API and Data API may occasionally receive a 429 response.
Report: "API Errors/Connection Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are investigating an issue with an increased error rate on our Data Export API service version 1.x. Our engineering team is investigating.
Report: "Decreased performance in multilingual ai agent"
Last updateThis incident has been resolved.
A fix has been implemented and we are currently monitoring the results.
We have identified the issue and are working to deploy a fix.
Issue: Some multilingual ai agents are seeing a decrease in response performance (matching content to questions). Who it affects: the issue is seen in ai agents that offer chats in multiple languages but only reference knowledge articles in English. Ai agents with multiple language knowledge sources are unaffected.
Report: "Topics tab missing from Ada dashboard (Generative and Declarative bot) for customers"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Investigating issues with Ada Chat services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some customers are experiencing issues with our chat services. This is affecting some chatters interacting with Ada bots over Social Channels. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "New email channel activations failing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified with one of our vendors and they have been notified of the issue.
Issue: Clients trying to activate the email channel on their ai agent may be met with an error. Who it affects: this only affects new activations. Existing ai agent email configurations remain unaffected.
Report: "Degraded performance in ai agents knowledge matching"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Issue: some ai agents are seeing a decrease in their ability to match questions to existing knowledge articles. Who it affects: we're working on determining full impact and initial review shows that not all ai agents are affected or affected to the same degree.
Report: "Coaching issue"
Last updateWe had a coaching issue from 8:45pm-9:25pm EST. This incident has been resolved.
Report: "Investigating issues with Analytics"
Last updateThis incident has been resolved.
We are continuing to monitor the data backfill.
A fix has been implemented and we are monitoring the results. Data backfill is expected throughout the day for any missing information. ETA 6-7 hours from the time of this update.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Some customers are experiencing issues with Analytics. This is affecting the Reports View in the Ada dashboard. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Delay in message delivery Sunco-related handoffs/escalations"
Last updateThis incident has been resolved.
Sunshine Conversations is reported to have returned to normal delivery times. We will continue to monitor the status of the incident.
Issue: Some agent messages sent by agents may be delayed. Who it affects: We've identified that the issue is related to a service disruption with Sunshine Conversations. Ai agents and chatbots using the Sunshine Conversations integration may see a delay in the delivery of some messages.
Report: "Chat service unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Issue: some users may see the fallback message when opening a chat window. Who it affects: users who have loaded the latest version of the chat embed. Users who have existing conversations will not be affected.
Report: "Investigating message latency with Ada Chat services"
Last updateThis incident has been resolved.
A fix has been implemented by the third party service provider and we are monitoring the results.
We are continuing to work on a solution for this issue.
The issue has been identified with a third party service and they have been notified.
We are continuing to investigate this issue.
Some customers are experiencing messaging latencies with our chat services. This is affecting some chatters interacting with Ada bots. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Voice channel outage in US cluster from 01:11 - 01: 30 am EST"
Last updateThis incident has been resolved.
There was a time window from 1:11am EST to 1:30 am EST where voice was unavailable. The voice channel has been returned to normal and our team is monitoring the performance.
Report: "Investigating issues with Ada Chat services"
Last updateThis incident has been resolved.
The issue with the Ada Dashboard has been resolved and Dashboard users should see things functioning normally again. Our team will continue to monitor the fix. We’re sorry for the impact of this issue and appreciate your understanding.
Our team has identified the problem with Ada Voice and we're implementing a fix for the issue. Thank you for your patience as we work on a resolution. Apologies for this inconvenience. We will continue to keep you up to date every 30 minutes.
Some customers are experiencing issues with our chat services. This is affecting some chatters interacting with Ada bots in regards to knowledge retrieval. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Investigating issues with Ada Dashboard"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Some customers are experiencing issues with the Ada Dashboard. This is affecting Ada Dashboard users only and affects the ARR report. Chatters interacting with your bot are unaffected. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Voice Channel Missing Messages"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Issue: Ai Agents connected to the voice channel sometimes are unable to present an answer to the caller. Who it affects: The issue is seen in ai agents (ie. generative) that are connected to the Voice channel.
Report: "Bot Dashboard Inaccessible"
Last updateThe incident has been resolved.
We have identified the cause of the rate limits and have implemented appropriate changes to prevent recurrence. Additionally, we have set up monitoring to provide early warnings in case of any regression.
Some endpoints for the Ada Dashboard were hitting rate limits, causing overall slowness and, in some cases, loading attempts to time out.
Users are seeing a “Service Unavailable” message when attempting to log in to the bot. End users however are still be able to access and interact with the bot.
Report: "Investigating issues with Coaching on Ada Dashboard"
Last updateThis incident has been resolved.
Our team is currently investigating and implementing a fix for the issue with the Ada Dashboard. Thank you for your patience as we work on a resolution. Apologies for this inconvenience. We will continue to keep you up to date every 30 minutes.
Some customers are experiencing issues with the Ada Dashboard. This is affecting the Coaching feature on the Ada Dashboard only. Chatters interacting with your bot are unaffected. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Chat bot window not leading to a chat session in some bots"
Last updateThis incident has been resolved.
A fix has been pushed and bot windows will open to a conversation as expected.
The issue has been identified and a fix is being implemented.
Issue: Some chatbot windows are opening to an unexpected page when the bot is first opened. Who it affects: chatbots not hosted in the US cluster (to determine this you will see a specific value between the bot handle and "ada.support" in the address you use to log into the dashboard eg bot-handle.[cluster].ada.support).
Report: "Investigating issues with Analytics"
Last updateThis incident has been resolved.
The issue with Analytics has been resolved and Dashboard users should see things functioning normally again. Our team will continue to monitor the fix. We’re sorry for the impact of this issue and appreciate your understanding.
Our team has identified the problem with Analytics and we're implementing a fix for the issue. Thank you for your patience as we work on a resolution. Apologies for this inconvenience. We will continue to keep you up to date every 30 minutes.
We are continuing to investigate this issue.
Some customers are experiencing issues with AR Estimation and Topics Generation. This is affecting Ada Dashboard users in the form of delays for these features. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Investigating issues with Ada Chat services"
Last updateThe issue with Ada Chat has been resolved and chatters should see things functioning normally again at 1:15PM EST. Our team will continue to monitor the fix. We’re sorry for the impact of this issue and appreciate your understanding.
A fix has been implemented and we are monitoring the results.
Some customers are experiencing issues with our chat services. This is affecting some chatters interacting with Ada AI Agents in the form of latency while knowledge is being accessed. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Investigating issues with Ada Chat services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Some customers are experiencing issues with our chat services. This is affecting some chatters interacting with Ada bots in the form of increased bot message latency. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Increased handoff rates"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Issue: Elevated handoff rates for Generative AI Agents Who it affects: Chatters interacting with the AI agent may notice handoffs for prompts that the AI agent would be able to handle under normal circumstances.
Report: "Knowledge Article Update Latency"
Last updateThis incident has been resolved.
Backfill of affected articles has been completed.
A fix has been implemented for the current updates. Changes to articles moving forward will reflect in the ai agent knowledge. We're working on backfilling any updates that have not been captured during the incident.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Issue: Some articles in AI agents may be delayed in reflecting the latest updates from knowledge sources. Who it affects: Chatters that receive a generated answer that references an affected article. The response may not contain the latest updates made to the referenced articles.
Report: "Conversation Latency"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate the issue. Based on our logs we have not seen any spikes latency since 3:31 pm EST. No other spikes have been detected at this point and no other impact to end users.
Issue: We're seeing an increase in the latency of conversations loading. Who is affected: Chatters opening chat for the first time may experience a slightly longer load time until they can see the initial greeting/content.
Report: "Ada App Blocks Partial Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Issue: some blocks built on Ada's Integrations service (eg. Answer Utilities, Fetch Chatter MetaData, etc.) are not working as expected. Who it affects: Chatters who are interacting with answers where these blocks are used will not be able to proceed past this block
Report: "Investigating issues with Email Action App on EU and Maple Clusters"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Some customers are experiencing issues with the Email Action App. This may be affecting chatters who are interacting with your bot and attempting to handoff using the Email Action App on EU and Maple clusters. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Conversation Topics and Resolution Estimation Latency"
Last updateThis incident has been resolved.
New conversations are getting Topics assigned delayed. Applying AR estimation to conversations is delayed.
Report: "Dashboard Performance Degradation"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Issue: Ada dashboards fail to load occasionally for backend users. Who it affects: this issue affects backend users like admins who access the Ada bot dashboard for content changes and reports. Chatters are not affected by this
Report: "Delay in Conversation View Refresh"
Last updateThis incident has been resolved.
We have allocated additional resources to address the issue, reducing the delay in conversation view updates to five minutes. Our team is actively working to restore the data pipeline to its original state.
We are currently experiencing an increased volume of conversations, which is causing congestion in our data pipeline and resulting in delays in updating the conversation view. Please note that this does not affect chat functionality, and end users should not encounter any issues while interacting with the bot.
Report: "Dashboard Update Delay Affecting Certain Pages"
Last updateThis incident has been resolved.
We are actively working on a more permanent fix for the issue while ensuring that traffic continues to flow without interruption. Currently, there is still a delay of approximately one minute for the most recent conversations to appear in the Conversation View. We are committed to restoring the original five-second update time as soon as the pipeline upgrades take effect.
We have identified the issue affecting the Conversation View in the data pipeline. The root cause of the delay was a spike in traffic, and measures have been taken to address it. For now, users may notice that the Conversation View takes approximately one minute to update, instead of the usual five seconds for recent conversations.
Certain pages such as the conversation view are taking time to refresh and show the most recent data
Report: "Intermittent salesforce handoff impact"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some customers are experiencing intermittent issue with Salesforce handoff. We’re aware of the issue and have identified the root cause. A fixed will be implemented soon.
Report: "Oracle Service Cloud "Failed to send agent message" Failures"
Last updateThe issue has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified as some API requests being sent to terminated pods - the team is currently working on a fix.
We are continuing to investigate the issue. A definitive root cause has not yet been identified.
We are currently investigating a spike in chatters unable to see agent messages for clients using Oracle Service Cloud.
Report: "Conversation View Fails to Display Recent Conversations"
Last updateThe conversation view has been updated to the most recent conversations.
Conversation data is still being processed to catch up to the most recent activity. Backend users will notice a delay in the availability of conversation transcripts found in the Conversations View. We are continuing to monitor the situation until all the data is available in the dashboard.
The issue has been resolved, and we are actively monitoring the situation to ensure that the data displayed on the dashboard is fully caught up.
We can confirm that the data is now coming through gradually. However, we are continuing to address the root cause to ensure a consistent flow and prevent any further issues.
A third-party service within our data pipeline has been identified as the cause of the issue. We are currently investigating the root cause and working to restore capacity as quickly as possible.
The root cause of the issue has been identified, and we are actively working to resolve it as quickly as possible.
We are currently investigating this issue.
Report: "Investigating issues with Automated Resolution Analytics"
Last updateThis incident has been marked resolved.
We are continuing to investigate this issue.
Some customers are experiencing issues with Automated Resolution Analytics. This is affecting Ada Dashboard users. We’re aware of the issue and our team is actively working on getting everything back to normal. We're sorry for the inconvenience this causes. We will continue to share status updates every 30 minutes.
Report: "Degraded Performance in Reasoning Engine"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A change has been pushed that allows the Reasoning Engine to behave as normal. While we're keeping the status of the incident as "Identified" the end user experience has returned to normal.
We are continuing to investigate this issue.
Issues: We have identified a small bump in errors with our Reasoning Engine. A few chatters may receive an unexpected response from Generative bots that could lead to an escalation. Who it affects: users who interact with Generative (AI) bots.
Report: "Voice channel outage"
Last updateIssue: An outage with the Voice channel between 10:50am to 11:15am ET resulted in calls to voice channel bots being routed to the fallback number. A fix has been implemented and calls are back to routing as normal.
Report: "Proactive SMS campaigns not delivered due to Twilio outage"
Last updateProactive campaigns were not triggered due to third party provider incident.
Report: "Newly created Actions are not appearing under Action section"
Last updateThis incident has been resolved.
The issue has been resolved.
All the existing actions are working fine. We are investigating the issue now.
Report: "New sunshine conversation integrations blocked"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Issue: Clients connecting bots to a Sunshine Conversations (ie. Sunco) app are blocked from completing the connection. Who it affects: This issue only affects new setups. Existing integrations are not affected by this.
Report: "Token Vault Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented. We have further identified that this issue only impacts answers/processes that have been updated since since 11am ET.
We are continuing to investigate this issue.
Issue: values stored in the token vault are not accessible. Who it affects: Answers and Processes using sensitive tokens from the token vault are not able to access the value stored. These processes will fail if the token is essential.
Report: "Greetings Failing to Display for Generative Clients"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Title: Greetings Failing to Display for Generative Clients Issue: Greetings are failing to display when a chatter opens a generative bot, resulting in a blank chat greeting. Who it affects: This is a Generative-Only issue.
Report: "Ai Agent outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for the remaining features.
We have pushed a fix to restore functionality back to the AI Agents. Chatter questions are now answered normally. Some generative features may still experience latency.
The issue has been identified and a fix is being implemented.
Issue: Ai Agents are currently unable to generate answers to chatter questions. Who is affected: clients using our Ai Agent/Generative bots. Declarative bots are unaffected by this.
Report: "Product outage"
Last updateAt approximately 10:21 am to 10:25 am EST users would have experienced an outage in the Ada product and services. The team was notified of the issue through logging alerts and shortly after able to return functionality to the bot.
Report: "New Email activations do not complete"
Last updateThis issue has been resolved by our vendor and email activations are now successful.
Our vendor is continuing to work on the issue. For more details see here(https://status.sendgrid.com/incidents/dhql5fztjsny)
The issue has been identified with one of our vendors and they have been contacted for a fix.
Issue: new email addresses cannot be connected to Ada bots. Who it affects: Any clients attempting to set up the email channel in their bot for the first time. Existing email bots are not affected by this issue.