Historical record of incidents for tawk.to
Report: "Failure to deliver push notification for mobile application"
Last updateIssue resolved
We have identify the cause of the failure. Currently working on fixing the issue.
Report: "Delays in email deliveries to @gmail.com mailboxes"
Last updateWe're no longer seeing general issues delivering to gmail based domains. If you are still seeing issues, please contact us in live chat or at support@tawk.to
General deliveries to gmail are back to normal. We continue to monitor the situation and work with a handful of affected property owners who still are seeing issues.
Through continued efforts of our engineers, we have narrowed down the issue to a handful of affected sender domains and we're continuing to work with affected property owners on solution on a case-by-case basi
We are starting to slowly roll out new sender domain for transactional emails as we continue work on improving our sender reputation.
Our engineers are working to establish a new sender domain for transactional emails such as tickets. By doing so we are looking to separate sender reputations for notifications and transactional emails.
Our engineers have updated handling of DMARC records to help improve and separate reputation of individual property specific sender domains.
We're still experiencing heavy rate-limiting when sending to GMail based domains and GMail itself. Our engineering team is in contact with our google account managers looking for potential solutions. In the mean time we're restricting deliveries of chat transcript emails to @gmail.com domains to prioritize ticket deliveries
We've deployed additional outbound email servers to increase sending capacity
Our team has identified source of large volume of unsolicited emails being generated that caused queue buildup. Offending senders have been suspended indefinitely and pending emails form those senders has been dropped from outbound email queue.
We've identified issue in our email infrastructure that is causing deliveries to @gmail.com domains to be delayed
Report: "AI Assistant Outage"
Last updateWe're pleased to report that the restoration of the Vector Database is now complete. Full access to all knowledge sources should now be restored.
There were unexpected issues during the 'Website Knowledge Source' restoration process, causing a delay. However, the dedicated team has efficiently addressed and resolved these challenges. The restoration process is now back on track, and it is expected to be completed within approximately one hour.
We wish to update you on our ongoing efforts to restore the 'Website Knowledge Source' in our Vector Database. At present, we have successfully restored close to 30% of the overall 'website knowledge', and we are fully committed to achieving complete restoration as promptly as possible.
We are pleased to report significant progress in our restoration efforts. The AI response functionality has been fully restored, and the 'KB (Knowledge Base) knowledge source' is now operational again. Our team is currently focused on the restoration of the 'website knowledge source'.
The restoration process for both the 'KB (Knowledge Base) knowledge source' and 'website knowledge' is currently underway. We are making good progress and are optimistic about a swift completion. Additionally, we are in the process of re-enabling the AI Assistant to provide its full range of services.
We are pleased to announce that the 'document knowledge source' has been successfully restored in our Vector Database. Our current focus is on restoring the 'KB (Knowledge Base) knowledge source' and the 'website knowledge'. Our team is working tirelessly to complete these tasks and fully reinstate all services.
We have successfully restored the 'plain text knowledge source' in our Vector Database. Our team is continuing to work diligently on restoring the 'document knowledge source'. We are committed to resolving this remaining issue as quickly as possible and will provide a further update soon.
We are actively working on restoring the 'document knowledge source' in our Vector Database. Our team is dedicated to resolving this as swiftly as possible. We will provide a further update soon.
We have successfully restored the 'shortcut knowledge source' index in our Vector Database. We are now working on restoring the remaining knowledge sources. Progress is being made, and we anticipate a full resolution soon.
The initial fix attempted for the Vector Database issue was unsuccessful. We are now implementing a secondary plan, which involves re-indexing to a new cluster. This approach is aimed at resolving the problem more effectively.
We have identified the root cause of the issue with our Vector Database. Our team is now actively working on applying the necessary changes to fix the problem.
Currently addressing an issue with our Vector Database in collaboration with the vendor. We are working diligently to resolve this as quickly as possible and apologize for any inconvenience caused.
The issue has been identified as originating from our Vector DB vendors. A support ticket has been raised with the vendor, and we are currently awaiting their response.
Upon thorough investigation, the root cause has been traced to our Vector DB, which is currently not responding to queries as expected. This is directly impacting the functionality of the AI Assistant addon, leading to the observed increase in API errors.
We have identified a critical issue affecting the AI Assistant addon, marked by an elevated rate of API errors. This incident was initially detected around 1420 UTC today.
Report: "Widget Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
At 18:28 UTC, we noticed elevated error rates on backend services, which was caused by one of our databases going down. We have restored the database, services are recovering. We will continue monitoring the issue.
Report: "Degraded conversation service"
Last updateBetween 26th September 2022 2330 UTC to 27th September 2022 0830 UTC, we experienced degraded conversation service due to issues in one of our database clusters. This caused delays in : 1. Chats being processed to history 2. Sending out offline message emails 3. Sending out ticketing emails The database issue was resolved at 27th September 2022 0730 UTC. The ticketing issue was resolved at 27th September 2022 0830 UTC. The conversation history processing delay was resolved at 27th September 2022 1020 UTC.
Report: "Live chat Outage"
Last updateBetween 22nd September 2020 1350 UTC to 22nd September 2020 1410 UTC. We were experiencing major outage on live chat widget due operation latency issue in one of our databases. This caused widget stop appearing on websites and incoming chats could not be answered. The issue is resolved and live chat was fully restored at 22nd September 2020 1411 UTC
Report: "Widget maintenance"
Last updateThis incident has been resolved.
Widget is currently unavailable due to maintenance. It will be back online in an hour.
Report: "Degraded service"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've noticed an issue with one of our auto-scale groups and are applying a fix
Report: "Widget outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We've found an issue and are resolving it.
We've applied a fix and will continue to monitor the database
We've identified an issue with one of our databases and are applying a fix
We are currently investigating this issue.
Report: "Degraded widget load time"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Widget outage"
Last updateThis incident has been resolved.
The issue is resolved and live chat was fully restored
We were experiencing a major outage on the live chat widget due to a DDOS attack on the widget API server We are currently investigating this issue.
Report: "Live chat widget outage"
Last updateBetween 2nd June 2021 0835 UTC to 2nd June 2021 0920 UTC. We were experiencing a major outage on the live chat widget due to a DDOS attack on the widget API server The issue is resolved and live chat was fully restored on 2nd June 2021 0920 UTC
Report: "Degraded widget performance"
Last updateThis incident has been resolved.
We've identified an issue with widgets loading and a fix is under way.
Report: "Shipping major updates to widget"
Last updateThis incident has been resolved.
We will be undergoing scheduled maintenance during this time. The widget will be down during this period. Expected restoration of service at 11.30 AM UTC
Report: "Degraded service"
Last updateThis incident has been resolved.
We've implemented a fix and restored functionality. We will continue to monitor the issue.
We've identified the issue and are working on a fix. We will provide an update by 19:00 UTC
We noticed that something went wrong. Will provide update at 19:00 UTC
Report: "Degraded service"
Last updateWe identified the issue with one of our database clusters handling visitors. The issue has been resolved and additional capacity has been added to aid in clearing of the backlog. We're continuing to monitor the database.
We've identified the issue and have partially restored service. Some widgets may still experience issues loading. We will provide an update by Tuesday, 2021-03-23 15:30 UTC
We noticed that something has gone wrong We will provide an update by Tuesday, 2021-03-23 15:00 UTC
Report: "Duplicate ticket notification"
Last updateWe have fixed the issue. Ticket processing is restored.
We notice that recent changes to ticket processing introduce new bug which causes duplicated ticket notification email to be sent. We have temporarily disabled processing of tickets and sending of ticket notification email. We will provide an update by Wednesday, 2021-03-17 14:00 UTC
Report: "Live chat widget failed to load"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues. We will provide an update by Tuesday, 2021-03-16 10:30 UTC
We are continuing to monitor for any further issues. We will provide an update by Tuesday, 2021-03-16 10:30 UTC
We will continue monitor to DDOS mitigation solution. We will provide an update by Tuesday, 2021-03-16 09:30 UTC
We are experiencing Distributed Denial of Service. We have taken steps to mitigate the DDOS attack. During this period widget service might experience degraded performance.
We are currently investigating ongoing issue of widget failing to load. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-03-16 09:00 UTC
Report: "Degraded widget performance"
Last updateThis incident has been resolved.
We are currently investigating this issue.
We are currently investigating
Report: "Service management grid fail-over"
Last updateDue to a localized issue in our service management grid at 20:47 UTC a large scale systems fail-over occurred. Recovery was reached at 20:49 UTC. Degraded performance was observed till 20:55 while systems fully failed over
Report: "Degraded widget load time"
Last updateMinor incident occurring in one of auto-scaled instance groups resulted in degraded widget performance from 14:24 UTC to 14:35 UTC
Report: "Degraded widget load time"
Last updateMinor incident on one of auto-scaled instance groups lead to degraded widget performance from 18:23 UTC to 18:36 UTC
Report: "Degraded Live Chat"
Last updateBetween 22nd September 2020 1604 UTC to 22nd September 2020 1634 UTC. We were experiencing degraded service due misconfigured database from previous outage. We have identified and fixed the issue. The issue is resolved and live chat was fully restored at 22nd September 2020 1635 UTC