Historical record of incidents for verloop
Report: "Verloop bots on India / US region are down."
Last updateThis incident has been resolved.
We are working on the issue and will update next steps in few mins.
Report: "MENA Region BOTS and Agent Portal is down."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are seeing network related issues on the MENA region of the Verloop. Team is working on it on priority.
Report: "Global Whatsapp messaging issues."
Last updateThis incident has been resolved.
The issue has been resolved and we are monitoring the recovery of GRAPH API's of whatsapp.
We are seeing increased number of errors from META API for sending messages and getting updates from META. We are closely monitoring the situation and will update soon on the developments.
Report: "Degraded performance on chat and voice calls."
Last updateThis issue has been resolved.
We are getting the services failover to back up instances.
We are working on the issue with the DB on Verloop side, we are trying to restore services as of now.
Report: "Whatsapp business management api's failing (EditTemplate / Fetchall flows / create template / fetch template etc)"
Last updateThe issue has been resolved.
We are seeing errors on the whatsapp business management. This does not impact sending messages or existing outreaches. The impact is on fetching / creating new templates or flows. Team is investigating the issue and will update on progress.
Report: "Whatsapp messaging is affected."
Last updateThis incident has been resolved.
This was a partial outage for some customers.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
All the Whatsapp messaging on Verloop GCP platform is affected due to issue on the META API. We are working with META team closely to resolve the issue.
Report: "BOTs are not responding to customer chat"
Last updateAll the channels are back to normal. We will be sharing a detailed RCA in sometime.
We have pushed a fix for the issue and bots are responding fine, we are currently monitoring.
This is only affecting customers in specific region, this would not impact Middle east customers.
Verloop bots are not responding to customer chats. We are working on the issue on high priority. We know the root cause of the issue to be the queuing service.
Report: "Middle east customers might face some intermittent issues due to global network outage of MS Azure."
Last updateThe issues from Azure side are resolved, Verloop services are back up. We are constantly monitoring.
We are closely working with microsoft team. Last update from Microsoft: Current Status: Automatic monitors detected a thermal anomaly. As a proactive measure, resources were powered down, causing a drop in availability. This manifested in connectivity failures for Virtual Machines for a subset of customers. We have successfully restored cooling systems, and temperatures have returned to normal operating thresholds. We have commenced a structured power- up sequence for previously powered- down compute and storage resources and are actively monitoring as we work through this extended process.
We are still working with MS Azure team on getting our internal services back up and running. As the UAE region DB services are impacted the DR and Primary both DB are not reachable. We are working closely with Azure team to get the issue resolved.
Azure network issue still seems to be having issues. We are still seeing issues with the internal services including in the DR zone are not reachable. We are trying other options to restart the Verloop chatbot services. We appreciate your patience.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Due to Global issues with the Microsoft Azure network services (https://azure.status.microsoft/en-gb/status), Verloop customers of Middle east region might face intermittent issues. We are closely monitoring the situation and will inform on further action soon.
Report: "Bot responses are slow or non responsive"
Last updateThe issue has been identified and resolved. We will reporting the RCA on this in coming days.
We are seeing bot responses being slow or non responsive in some cases. Team is working on resolving the issue.
Report: "Issue with Whatsapp bots and Reporting"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Google DNS changes on the networking level has made some of the critical services of Verloop being inaccissible. Team is working to resolve the issue.
Report: "Experiencing delays and higher error rates for BOTS in MENA region."
Last updateBOTS Are now responding to customers. Issue has been resolved. Detailed RCA to follow.
We are seeing higher error rates in MENA region which is affecting BOT responses to customers. We are investigating the issue.
Report: "Degraded performance on chat assignment and latencies in messages."
Last updateThe DB load has come down and we are now seeing services are functioning normally.
We are currently investigating the issue.
Report: "Chats on all channels are showing degraded performance for UAE customers."
Last updateIssue has been resolved. We are working on updating the root cause here.
We have identified the issue, we are working on fix.
We are investigating the issue. This issue is limited to UAE region customers only.
Report: "Realtime dashboard widgets taking longer time to load or failing to load for the Asia region customers."
Last updateThe incident has been resolved.
We are currently investigating the issue.
Report: "Whatsapp Business messages has a issue."
Last updateUpdate from WhatsApp The fix for the major disruption impacting sending messages on WhatsApp Cloud API has been deployed and this issue is now mitigated as of 3rd Aug 2023 11:50 PM PDT. Thank you for your patience.
WhatsApp Cloud API is experiencing major disruption impacting sending messages. This issue started at 3rd Aug 2023 6:30 PM PDT. WhatsApp engineering teams have identified the root cause of the issue. Whatsapp engineering team is currently deploying the fix and the expected deployment time is in 6 hours. We will provide another update within 4 hours or sooner if additional information is available. This impacts all the regions for the WhatsApp message delivery.
Report: "Chat bots are not responding to the chats."
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating the issue with the bots not responding to chats on GCP. This does not affect customers hosted on Azure.
Report: "Whatsapp bots not responding."
Last updateThe WOXY service, which is responsible for connecting to WhatsApp, encountered an issue where it became unresponsive and unable to establish a connection. This problem arose due to a network glitch that occurred within our subnet, disrupting the normal functioning of the service. Consequently, the WOXY service entered a crashloop backoff state within our Kubernetes cluster. In order to resolve this issue, we took appropriate action and initiated a restart of the WOXY service. By doing so, we aimed to bring the service back online and restore its normal functionality. This restart was necessary to rectify the underlying problem caused by the network glitch and ensure that the WOXY service could once again establish a stable connection to WhatsApp.
Report: "This is an example incident"
Last updateWhen your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.
Empathize with those affected and let them know everything is operating as normal.
As you continue to work through the incident, update your customers frequently.
Let your users know once a fix is in place, and keep communication clear and precise.