Historical record of incidents for Torq
Report: "System Outage"
Last updateWe are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently experiencing an issue with our cloud provider that is affecting system performance and availability across both our US and EU regions.
Report: "Partial UI Outage"
Last updateA fix has been implemented and we are monitoring the results.
We are currently experiencing a partial UI outage affecting the Case Management section. Users may encounter issues viewing or interacting with certain case-related elements through the interface. This issue does not impact underlying workflows, case creation, or backend functionality. Our team is actively investigating the root cause and working to restore full functionality as quickly as possible. We will provide an update as soon as more information becomes available. Thank you for your patience.
Report: "Some features are not displayed correctly in the UI"
Last updateThe issue has been resolved
Our third-party vendor has implemented a fix for the issue. We are seeing improvements and most services are operating as expected. We continue to monitor the situation carefully to ensure full stability
We've identified the issue affecting the display of certain features and the accessibility of some functionalities within the platform. The root cause has been traced to a third-party application. Our team is actively working with the vendor to implement a fix. We’ll continue to provide updates as progress is made.
We are currently investigating an issue where certain features are not being displayed correctly in the UI. Some users may also experience limited access to specific functionalities within the platform. The system is fully functional in the backend. Our engineering team is actively working to identify the root cause and restore full service as quickly as possible. We will provide updates as more information becomes available. We apologize for the inconvenience and appreciate your patience.
Report: "Some features are not displayed correctly in the UI"
Last updateThe issue has been resolved
Our third-party vendor has implemented a fix for the issue.We are seeing improvements and most services are operating as expected.We continue to monitor the situation carefully to ensure full stability
We've identified the issue affecting the display of certain features and the accessibility of some functionalities within the platform. The root cause has been traced to a third-party application.Our team is actively working with the vendor to implement a fix. We’ll continue to provide updates as progress is made.
We are currently investigating an issue where certain features are not being displayed correctly in the UI. Some users may also experience limited access to specific functionalities within the platform. The system is fully functional in the backend.Our engineering team is actively working to identify the root cause and restore full service as quickly as possible. We will provide updates as more information becomes available.We apologize for the inconvenience and appreciate your patience.
Report: "Maintenance Window (US) - Data Infrastructure Upgrade"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
As we are happy to see a growing adoption of use for Workspace Table Variables across multiple use-cases, we would like to perform a proactive upgrade of the underlying database(s) infrastructure in order to improve performance, scalability and reliability of the automations dependent on this data.We have developed and tested a dedicated mechanism for this operational window that pauses the execution of processes dependent on Workspace Table Variables for 2 5-minute-long intervals, resuming these processes without any loss or otherwise impact at the interval end.The procedure has been thoroughly tested and already executed on multiple environments and is not expected to introduce impact beyond the communicated one.
Report: "[US] - Elevated Error Rates for Workflow Steps Utilizing Tables"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified the root cause of the elevated error rates affecting Table steps in the US region. Our team is actively working on developing and testing a fix for the underlying issue and plans to deploy it as soon as possible.
We are currently investigating elevated error rates impacting workflow steps that interact with Variables. Our engineering team is actively working to identify the root cause. We will provide an update as soon as more information is available.
Report: "[EU] Delays and Failures in Accelerated Step Executions"
Last updateThe issue has been resolved.
Mitigation measures have been implemented, and step executions are now succeeding. Users may still experience intermittent delays. We are actively monitoring the system to ensure stability and are continuing to work toward a full resolution.
We have implemented measures to mitigate the failures, and step executions are no longer failing. However, users may still experience delays. We continue to monitor the situation and are working toward full resolution.
We are currently experiencing delays and intermittent failures in accelerated step executions due to an issue with the underlying Google Cloud Platform (GCP) infrastructure. Our team is actively working with Google to resolve the problem and restore full functionality as soon as possible. We will provide updates as we learn more. Thank you for your patience.
Report: "All Systems Operational"
Last updateThis incident has been resolved.
We’ve resolved the issues affecting activity log delays and event attachments. The system is now fully operational. Our team will continue to monitor closely to ensure ongoing stability and performance. Thank you for your patience throughout the incident.
We’ve resolved the issues affecting activity log delays and event attachments. The system is now fully operational. Our team will continue to monitor closely to ensure ongoing stability and performance. Thank you for your patience throughout the incident.
We’re pleased to report that the issue with attaching events to cases has been fully resolved and is now operating normally. We are continuing to close the gap on the delay in publishing activity logs and are seeing steady improvements. Our team remains focused on ensuring full recovery. We will provide further updates as needed. Thank you for your continued patience.
We’ve made significant progress in resolving the delay in publishing activity logs and are actively closing the gap. Additionally, we’ve reduced the number of failures related to attaching events to cases. We continue to monitor the situation closely to ensure full recovery and system stability. Thank you for your continued patience.
We are currently experiencing a delay in publishing the activity logs for executed workflows. Additionally, there is an issue preventing events from being attached to cases. Workflows execution and case creation are not impacted in any way. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. We will provide an update as soon as more information becomes available. Thank you for your patience.
Report: "[EU] - Resolved - Service Degradation Impacting User Operations"
Last updateRESOLVED: We recently experienced an infrastructure issue that caused operation failures for some users in the EU region. Our team has successfully identified and fixed the underlying problem. The incident affected services between 21:00-21:55 UTC.
Report: "[US] - Access to Cases for Users and Automations is Slow with Periodic Failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "[US] elevated error rate for accelerated steps"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We're seeing an improvement in error rate, we're monitoring.
We are currently investigating this issue.
Report: "[US] slowness non-accelerated step executions"
Last updateThis incident has been resolved.
non-accelerated steps are operational, we are continuously monitoring the fix.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue, accelerated steps are working and we're seeing slowness in step executions running on non-accelerated infra.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "slowness and failures in step executions"
Last updateThis incident has been resolved.
Issue was resolved by GCP We are continuously monitoring
The issue related to an incident on Google Cloud PubSub For GCP incident details, please refer to: https://status.cloud.google.com/incidents/ghMho2Gka33Exr9UNavz We see that the activity is back to normal levels for some time now, we are continuously monitoring and keeping in close contact with Google Cloud Platform support.
We investigate slowness and failures in step executions.
Report: "[US] Elevated errors on none accelerated steps"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented
Report: "[US+EU] - Errors in step executions for Microsoft 365 Delegated Access steps."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Executions using Microsoft 365 Delegated Access integration are failing on the token is expired
Report: "[US] - Slowness in viewing execution logs"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Slowness in viewing execution logs, single steps results and manual executions results in US.
Report: "[US] Elevated error rate for torq public api"
Last updateThis incident has been resolved.
a fix has been implemented and we are monitoring the results
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "[EU] - Slowness in viewing execution logs"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Slowness in viewing execution logs, single steps results and manual executions results in EU.
Report: "[US] elevated error rate for accelerated steps"
Last updateIncreased latency and error rates observed on Google App Engine and Google Cloud Functions resulted in elevated error rate for accelerated steps https://status.cloud.google.com/incidents/1yphfNLPHEnwJcWqwxbu
Report: "EU elevated error rate for non-accelerated steps"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issues with execution of non-accelerated steps is related to a problem with an underlying infrastructure, owned by the Google Cloud Platform operations.
We are currently investigating this issue.
Report: "US non-accelerated steps intermittent failures"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issues with execution of non-accelerated steps is related to a problem with an underlying infrastructure, owned by the Google Cloud Platform operations. New infrastructure for steps execution is created in parallel to Google Cloud Platform operations working on rectifying the original issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Increased steps failure rate US"
Last updateOur team noticed an increase in the steps failure rates between 06:19 ET and 06:23 ET. The issue has been resolved, with the root cause being removed. Investigation into procedural updates will be completed.
Report: "Partial outage on Case Management. Creation and update of certain cases during the impact period (a number of minutes) was delayed, and access to Cases Dashboard was partially unavailable."
Last updateThis incident has been resolved
Report: "(EU) Partial outage with existing workflows (accessing and running)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented
Report: "Occasional errors in step executions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're investigating an increase in failures in workflows executions.
Report: "Service Degradation Impacting User Operations (US)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results
We are experiencing an issue with our infrastructure impacting some users by making certain operations slower than usual. Our team is currently engaged in rectifying the issue and returning the system to normal operation for all users.
Report: "Issue with Global Variables in US environment"
Last updateVariables are being restored, reach out support if you still having issues
We implemented a fix and working towards restoring the affected variables
We are working to resolve this issue, please contact support if you have urgent need.
We are experiencing an issue with missing data on some of the Global Variables in the US environments. The issue is caused by an underlying infrastructure problem that took place earlier today at the Google Cloud Platform. The issue may impact the execution of workflows relying on the information stored in the Global Variables. We are working on restoring all the data and are confident in our ability to perform this with zero loss.
Report: "Intermittent error with the Torq Console"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are looking at networking issues inside our cluster, this issue only impacting the US
We are currently investigating this issue.
Report: "We investigate slowness in step executions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Occasional errors in step executions"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently working on fixing this issue The error shown is: "The step was not executed. Contact Torq support for additional information."
Report: "We are investigating issues with the Torq platform"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "We investigate slowness in workflow and single step executions"
Last updateThis incident has been resolved.
We see things coming back to normal performance
We are currently investigating this issue.
Report: "Intermittent login errors to Torq console"
Last updateThis issue is now resolved
Report: "Degraded performance for workflow execution"
Last updateThis incident has been resolved.
We see an improvement, executions are running well now
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Issue with google based logins"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Insight page is partially unavailable"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Torq Learning Center is not available"
Last updateThis incident has been resolved.
We're aware that the Torq Learning Center is not available and are working to fix it quickly. If you need immediate access to documentation, contact us, and we'll get you exactly what you need.
Report: "GCP networking issue may cause intermittent network timeouts for some of the steps"
Last updateWe deployed a fix that overcomes the GCP networking issue, steps should work properly now. For any other help, please reach support@torq.io
The following GCP issue: https://status.cloud.google.com/incidents/sZVuKPuYgMQiB7AzmAnL#RP1d9aZLNFZEJmTBk8e1 may cause some of the steps to timeout before reaching their destination. We identified the issue and working to provide a transparent workaround to overcome that. In the meantime, as a workaround it's possible to disable accelerated execution to overcome that by going to "Edit YAML" and change "disable: false" to "disable: true" under the executor config, save and re-publish your workflow. Please don't hesitate to reach support@torq.io for any help needed.
Report: "Delay in workflow executions and workflow page is not available"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Partial outage with Single Step and Autocomplete"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating issues with Single Step and Autocomplete
Report: "Issue with workflow execution"
Last updateThis incident has been resolved.
Our fix seems to resolve the issue, we continue to monitor
We are seeing issue with workflow execution - investigating and will continue to update
Report: "Scheduled integration and IMAP not triggering"
Last updateIncident is now resolved, for any issue please reach our support.
A fix has been implemented and we are monitoring the results.
We identified the root cause and working on the fix
We are currently investigating this issue.
Report: "Partial workflow execution failures"
Last updateWe detected an issue that may cause executions to fail between 12:52 UTC to 12:58 UTC. This issue is now resolved.
Report: "Partial issue for workflows with Wait and Nested Workflows"
Last updateIssue resolved, workflows with wait and nested workflows calls may remain in 'on-hold' status
Report: "Intermittent errors in Torq management ui"
Last updateThis incident has been resolved.
Everything seems to be stable as Google Cloud Platform starting to recover their services. For more details, see: https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCMqV
We continue to work on routing traffic to healthy zones
We are working to route relevant workloads to healthy zones
Our underlying compute services provider, Google Cloud Platform, is experiencing connectivity issues in certain regions in the US. We do not see any impact on the execution of Torq workflows, but see sporadic issues impacting working sessions on our Management UI. Retrying a failed operation from the UI will correct the issue. We keep monitoring the impact and are re-routing the requests to “healthy” regions where possible.
Report: "Partial degraded performance with step execution times"
Last updateThis incident has been resolved.
We are monitoring the system status to make sure workflow execution performance is back to normal.
Our cloud provider notified us that they found and fixed the issue on their side. We are monitoring the status on our side and will start restoring workflow execution performance gradually.
We are continuing to work on a fix for this issue.
Our cloud provider confirmed this incident is related to a latency issue they are having with their compute services. We continue to work with them towards a fix. Workflow and step execution continue to be operational with a slight impact on workflows and step execution times.
We are continuing to investigate this issue.
We are continuing to investigate this issue with our cloud provider.
We are continuing to investigate this issue. all workflows and steps should execute successfully with a slight impact on execution times.
We are currently investigating this issue.
Report: "Slowdowns and stability issues in execution of some workflows"
Last updateHad a partial impact on workflow execution performance and stability, the vast majority of all executed workflows operated properly.
Report: "Workflow notifications aren't delivered"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Slowdowns and stability issues in execution of some workflows"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
A number of system components have been scaled up to provide better performance. We are monitoring the impact.
We are currently investigating the issue. We see that the impact on the overall execution is currently partial, i.e., vast majority of all executed workflows operate properly.
Report: "Editing workflows, adding integrations is impacted. Executing workflows via triggers may be delayed."
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Due to problems with Google Firebase, logging in to an Administration Portal is currently impossible. Users that are logged-in can continue working safely for the length of the session (default - 8 hours)"
Last updateThe underlying problem was resolved by Fastly. All functionality is back to normal. Here is a reference to the underlying incident: https://status.fastly.com/incidents/wkgnl07cyy75
We are continuing to investigate the issue. It is regional, potentially related to an outage at a Fastly CDN service. Currently, it seems that accounts that are related to Google Firebase (and, potentially, other) services related to Fastly's Frankfurt site may experience the outage.
The issue has been identified and a fix is being implemented.