Historical record of incidents for Tonkean
Report: "We are experiencing an unexpected load"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "We are experiencing an unexpected load"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "System Slowness Across Multiple Services"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing slowness across several services in our system. Our engineering team is actively investigating the root cause and working to restore optimal performance as quickly as possible. We apologize for the inconvenience and will provide updates as we learn more.
Report: "Investigating major outage due to database failure."
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Slowness in Tonkean App and executing workflows"
Last updateThis incident has been resolved.
Tonkean App and Workflows should be back and running as expected. We're monitoring the system to make sure everything is operational.
We are continuing to investigate this issue.
We are investigating slowness experienced in the Tonkean app and in executing workflows.
Report: "Unexpected Infrastructure Issue"
Last updateThis incident has been resolved.
We experienced a brief database disruption due to an unexpected update. The issue was identified and resolved within approximately 5 minutes. Systems are now fully operational.
Report: "Infrastructure issue"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Insert and update of external data into Tonkean is limited. Querying external data in Tonkean platform is limited. We are investigating.
Report: "We are having infrastructure issues"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Uploading files to Google Drive errors out"
Last updateSeems like the issue is currently resolved. confirmed by other users in the issuetracker as well. will keep monitoring.
We have identified an outage with Google Drive that impacts our Google application and does not allow it to make "upload" API calls due to some sort of protection on the Google side. This is a global Google Drive issue that happened in September and now resurfaced. It has been reported, acknowledged, and reopened here: https://issuetracker.google.com/issues/369788650?pli=1 We are tracking this and looking for ways to resolve it.
Report: "Uploading files to Google Drive errors out"
Last update**Impact** The incident affected our ability to upload files to Google Drive via the API. During the outage, Google Drive’s API calls for uploading files were blocked due to an issue on Google’s end. No other core functionalities of our application were impacted. **Timeline of Events** * **Sep 26th, 10:00 AM PST – Incident Identified** We detected an issue preventing file uploads to Google Drive using the API. The root cause appeared to be related to a protection or outage on Google’s side. Multiple high-priority support cases were raised with Google, and the issue was escalated for resolution. _Data Sources: Webhooks, Data Collection & Actions_ * **Sep 27th, 5:00 AM PST – Incident Resolved** Google resolved the underlying issue temporarily, and upload API calls started functioning normally. * **Sep 28th, 11:30 PM PST – Issue Resurfaces** The same issue occurred again, impacting the ability to upload files via the Google Drive API. We reopened the case with Google, further escalating the issue. * **Sep 30th, 7:00 AM PST – Incident Resolved Again** Google resolved the issue once more, and file uploads via the API returned to normal functionality. _Data Sources: Webhooks, Data Collection & Actions_ **Root Cause Analysis** The outage was caused by a global Google Drive issue, as referenced in this [Google Issue Tracker](https://issuetracker.google.com/issues/369788650?pli=1). The protection or failure in Google’s system affected multiple products globally, including our integration with Google Drive for file uploads. **Resolution and Mitigation** * Google resolved the underlying issue both times, restoring full functionality to the Google Drive Upload API. * No action was required on our side once Google fixed the problem. * We will continue to monitor the situation and re-open the incident if the problem reoccurs. **Action Items** * Continue communication with Google to understand potential long-term fixes. * Implement the capability to configure a server for tunneling traffic through different regions. * Add capability to route Google Drive API calls through a different region using a tunneling method if the issue occurs again, as a temporary solution to mitigate potential regional service disruptions. **Lessons Learned** * **Monitoring**: Relying on external service providers \(like Google\) requires robust monitoring and communication channels with those providers to promptly escalate and track issues. * **Communication**: Swift communication with Google and prompt escalation of support cases helped minimize confusion and ensured a timely resolution. **Next Steps** * Add capability to configure tunneling of traffic through different regions. * Ensure that API calls for Google Drive can be rerouted through a different region in the event of a similar outage. * Continue to monitor Google Drive API performance and follow up with Google for detailed RCA from their side if applicable.
The issue is currently resolved and upload API calls are working as expected. This seems to be a global Google Drive outage that impacted other products and customers as seen in this issue: https://issuetracker.google.com/issues/369788650?pli=1 We've opened multiple support cases with Google and escalated the issue. We will re-open this incident if the problem appears again
We have identified an outage with Google Drive that impacts our Google application and does not allow it to make "upload" API calls due to some sort of protection on the Google side. We've opened a high-priority support case with Google and have escalated this issue as much as we can. We're waiting to get a response from Google promptly.
Report: "Email delivery delayed for some customers"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Investigating slowness in our database"
Last updateThe incident has been resolved.
We are currently investigating this issue.
Report: "Users are unable to access Tonkean"
Last updateThis incident has been resolved.
A fix has been implemented and we are currently monitoring the results.
We are currently investigating the issue.
Report: "Investigating an issue that the system temporary stopped executing actions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Investigating slowness in our database"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We're experiencing slowness in the system"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Duplicate execution of actions"
Last updateActions were executed multiple times instead of once (started at 2:40 PST)
Report: "Infrastructure outage"
Last updateThis incident has been resolved.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Module execution is delayed"
Last updateThis incident has been resolved.
We identified delayed in module action executions, trigger matching, matched entities and accessing interfaces.
Report: "Intermittent Slowness in the Tonkean App"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We're seeing intermittent slowness in accessing the Tonkean app, through interfaces, forms, workspace apps and module editor. We're investigating the root cause. It should not affect any data coming through Tonkean.
Report: "The Tonkean app is not loading, and incoming webhooks are not being handled"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
The Tonkean app is not loading, and incoming webhooks into Tonkean are not being handled.
Report: "Big delays in modules execution and issues receiving data from external systems"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The outage in Tonkean is caused by an on-going outage in Elastic Cloud services, noted in these two incidents - https://status.elastic.co/incidents/xqngb41mglj8 https://status.elastic.co/incidents/p1dtb3mdlt5d Elastic Cloud are working on a fix, and we will update as we know more details on the incidents.
We are continuing to work on a fix for this issue.
We're experiencing an outage in module processing and receiving of external data into Tonkean.
Report: "Investigating unexpected services crash"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "Slow responses and timeouts when loading Business Reports"
Last updateFound the issue in the database and resolved the issue. Started at 04:40 PDT
Report: "Investigating reports that the website is unavailable"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Some business reports are not loading"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Slowdown in loading the Tonkean App, and workflow actions execution, for EU regions only"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "We're experiencing delay in running actions"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We're experience slowness in one of our databases"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Delay in execution of module actions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Tonkean is investigating reports of degraded performance"
Last updateThis incident has been resolved.
Tonkean is currently investigating reports of degraded performance. Further updates will be provided, as more information becomes available.
Report: "Slowness in execution of module actions"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Delay of up to 60 minutes in action execution"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Delay of up to 50 minutes in execution of module actions"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Up to 25 minutes delay in execution of module actions and populating fields"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Delay in execution of module actions"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Delay in execution of module actions"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Slowness in execution of module actions"
Last updateThis incident has been resolved.
We are seeing a delay of 5-10 minutes in responding to events and executing module actions. We're investigating the root cause.
Report: "Action execution slowness"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases and background services"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Report: "Action execution slowness"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Slowness in modules execution"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases and background services"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases and background services"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases and background services"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases and background services"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases and background services"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We're experiencing slowness in some of our background services"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We are experiencing slowness in our databases"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "We're experiencing slowness in processing events"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Delay in action execution and history display"
Last updateThis incident has been resolved.
We are currently investigating this issue.