Historical record of incidents for Todyl
Report: "Office 365 Access Issues"
Last updateA backend update unintentionally altered routing behavior for a subset of partners. This change impacted conditional access to Microsoft 365 when Todyl’s IP ranges were explicitly defined in access policies. The update was quickly reverted, and a permanent code fix has been implemented to prevent this issue from recurring.
Report: "SIEM/EDR: Intermittent Agent Offline Reports"
Last updateThis incident has been resolved.
We are aware of an issue where SIEM/EDR agents are intermittently reporting as offline in the dashboard. We are actively rolling out a fix for this issue. The only impact is a potential delay in the application of new SIEM/EDR policies to agents. Agents remain fully connected and are continuing to enforce existing policies as expected. This issue does not affect SASE functionality.
Report: "SIEM/EDR: Intermittent Agent Offline Reports"
Last updateThis incident has been resolved.
We are aware of an issue where SIEM/EDR agents are intermittently reporting as offline in the dashboard.We are actively rolling out a fix for this issue.The only impact is a potential delay in the application of new SIEM/EDR policies to agents.Agents remain fully connected and are continuing to enforce existing policies as expected. This issue does not affect SASE functionality.
Report: "SIEM Ingestion Delays"
Last updateThe issue has been resolved and our systems are operating as expected.
We are currently experiencing delays in log ingestion for our SIEM platform. This issue is affecting the timeliness of data being searched and displayed in dashboards.
Report: "SIEM Ingestion Delays"
Last updateThe issue has been resolved and our systems are operating as expected.
We are currently experiencing delays in log ingestion for our SIEM platform. This issue is affecting the timeliness of data being searched and displayed in dashboards.
Report: "SIEM Search and Dashboard Performance Issues"
Last updateThis incident has been resolved.
We have made changes to address the earlier performance issues affecting SIEM search and dashboards. The system is now functioning as expected. We are continuing to monitor to ensure full stability.
We are currently investigating an issue causing slow performance and intermittent timeouts in SIEM search and dashboard views. Important: This issue does not impact the underlying protection or detection capabilities. All detections are continuing to run on time, and systems remain fully protected. We will provide updates as we work toward resolution. Thank you for your patience.
Report: "SIEM/EDR: Intermittent Agent Offline Reports"
Last updateThis incident has been resolved.
A fix has been deployed, and the system is now operating normally. We are continuing to monitor to ensure stability and full resolution.
We are actively rolling out a fix for this issue. The only impact is a potential delay in the application of new SIEM/EDR policies to agents. Agents remain fully connected and are continuing to enforce existing policies as expected. This issue does not affect SASE functionality.
We are currently investigating an issue where SIEM/EDR agents are intermittently reporting as offline in the dashboard. Important: Despite these status reports, agents remain fully protected and continue to enforce policies as expected. This does not impact SASE. Our team is actively working to identify and resolve the root cause. We will provide updates as we learn more.
Report: "Network Connectivity Issues - Houston Region"
Last updateThis issue is resolved.
Our Houston (HOU) Point of Presence is now back online and operating normally. We have returned it to rotation, and all systems are functioning as expected. We will continue to monitor the environment to ensure ongoing stability.
Our Houston (HOU) Point of Presence is currently offline. We have proactively removed the PoP from rotation while we work with the datacenter provider to restore service. Most customers will not experience any disruption. However, customers who have configured only a single tunnel to the HOU PoP may experience connectivity issues until service is restored. We will reintegrate the PoP into rotation once the datacenter is back online. Updates will be posted here as they become available.
Our Houston (HOU) Point of Presence is currently offline. We have proactively removed the PoP from rotation while we work with the datacenter provider to restore service. Most customers will not experience any disruption. However, customers who have configured only a single tunnel to the HOU PoP may experience connectivity issues until service is restored. We will reintegrate the PoP into rotation once the datacenter is back online. Updates will be posted here as they become available.
We are currently investigating this issue.
Report: "Delays in SIEM Ingestion for Portal"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Unexpected Session Termination"
Last updateThe issue has been resolved, and all systems are now operating normally.
A fix has been identified and deployed. We believe the issue is resolved, but we will continue to monitor the situation.
A fix has been implemented and we are monitoring the results.
We are currently investigating elevated error rates across all pages in the portal. Users may experience unexpected logouts when interacting with certain areas of the Management Portal.
Report: "Delays in SIEM Log Ingestion"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "SIEM Processing Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue causing delays in the processing of new events within SIEM. A fix is currently being deployed to restore normal functionality. During this time, some recent events may not be immediately searchable. We will provide updates as the fix progresses and services return to normal.
Report: "Misconfiguration in macOS 15.4 Agents"
Last updateThis issue has been resolved, and systems are now operating as expected.
We have identified an issue affecting a small percentage of agents running on macOS version 15.4, resulting in a misconfiguration. Our team is actively working to deploy a solution to remediate the issue.
We are currently investigating a potential misconfiguration affecting a subset of agents running on macOS version 15.4. Impacted agents may experience degraded performance, or unexpected behavior. Our team is actively reviewing logs and configuration data to assess the scope and identify the root cause. At this time, the issue appears to be isolated to macOS 15.4 environments and SIEM features, we do not expect SASE connectivity or performance to be impacted.
Report: "SIEM Processing Delays"
Last updateThis issue has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified an issue causing delays in the processing of new events within SIEM. A fix is currently being deployed to restore normal functionality. During this time, some recent events may not be immediately searchable. We will provide updates as the fix progresses and services return to normal.
Report: "SIEM Processing Delay"
Last updateThe system continues to operate normally, and the issue has been fully resolved.
The fix has been successfully deployed, and the system is now operating normally. We will continue to monitor for any further issues.
We have identified an issue causing delays in processing new events for SIEM. A fix is currently being deployed to restore normal operations.
Report: "SIEM Processing Delays"
Last updateThe system continues to operate normally, and the issue has been fully resolved.
The fix has been successfully deployed, and the system is now operating normally. We will continue to monitor for any further issues.
The fix has been successfully deployed, and the system is now operating normally. We will continue to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We have identified an issue causing delays in processing new events for SIEM. A fix is currently being deployed to restore normal operations.
Report: "Resolved: SASE Connectivity -- AKL"
Last updateTimeframe: 19:45 UTC – 20:20 UTC (3.6.25) Some users experienced delays in establishing new SASE connections and higher-than-normal latency in the South Pacific region.
Report: "IPSec Tunnels"
Last updateThis incident has been resolved.
The fix has been successfully deployed, and all IPSec tunnels are fully operational. We will continue to monitor the situation.
We have identified the cause are in the process of rolling out a fix. During this time IPSec tunnels for DFW may have intermittent connectivity.
IPSec tunnels configured for the DFW PoP are currently down. We are investigating the cause. Agent connections to the PoP are operational.
Report: "Degraded Performance: SIEM Search Taking Longer Than Expected"
Last updateThis issue has been resolved. All systems are operational.
A fix has been deployed, and SIEM search performance has been restored. We will continue to monitor the issue.
We have identified an issue where SIEM searches looking back more than 7 days are taking longer than normal to return results. We are working on deploying a fix for this issue.
Report: "Delays in SIEM ingestion"
Last updateData ingestion systems are operating as expected.
We are currently experiencing small delays in the ingestion of SIEM data. The cause has been identified and fixed. We are monitoring the situation while the ingestion of SIEM data continues to catch up.
Report: "Todyl Community - Issues Logging In"
Last updateThis incident has been resolved.
A fix has been applied by the provider, and all systems are fully operational. We will continue to monitor this issue.
We have identified an issue with an upstream service provider. They are aware of the issue and we are working to resolve it.
We are currently investigating issues logging in to Todyl's Community.
Report: "Elevated Errors on SGN"
Last updateFrom 7:30 p.m MST to 9:30 p.m. MST, we saw elevated connection errors on some devices that led to agents being unable to connect to the SGN network. As of 9:01 p.m. MST we remediated the issue and have seen devices reconnecting. It may take a few minutes for devices to reconnect, please reach out to support if you continue to see issues.
Report: "Intermittent slowness with the Portal"
Last updateAfter sufficient monitoring we are moving this issue to resolved.
The issue with Portal slowness and timeouts has been resolved. It may take some time for SGN logs to catch up in SIEM. We will continue to monitor the situation.
We are currently investigating an issue where some operations in the portal are intermittently slow or timeout.
Report: "Delays in connecting to AMS"
Last updateDelays with new SASE connections to the AMS PoP has been resolved.
Report: "Network connectivity Issues - Los Angeles Region"
Last updateThis incident has been resolved.
Network connectivity issues should have been resolved.
We are currently investigating network connectivity issues impacting the Los Angeles region. Another update will be provided as soon as possible.
Report: "Intermittent issue with agents connecting to SGN"
Last updateThis issue has been resolved.
The fix has been implemented, and the issue has been resolved. We will continue to monitor this issue.
The problem has been identified and a fix is being deployed globally.
We are currently investigating an intermittent connectivity issue for agents attempting to connect to SGN.
Report: "Delays with SIEM search"
Last updateThe issue has been resolved. The system is fully operational.
The fix has been deployed. Some users may continue to experience intermittent errors with search while the system returns to steady state.
We have identified an issue impacting SIEM search. A fix is being deployed.
Report: "Issues accessing the customer portal"
Last updateThis incident has been resolved.
The fix has remediated the issue, we are continue to monitor to ensure complete remediation.
The issue has been identified and a fix has been implemented.
We are experiencing an outage of the customer portal. We are currently investigating the issue.
Report: "SIEM Detection Rule Delays"
Last updateThe issue has been resolved. The system is fully operational.
The fix has been successfully deployed and this issue is resolved. We will continue to monitor.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being deployed.
We are investigating an issue where users are unable to see new cases.
Report: "SIEM Ingestion Delays - Ubiquiti"
Last updateThis incident has been resolved.
All Ubiquiti event ingestion is now current. We will continue to monitor all SIEM ingestion prior to marking this incident resolved.
All SIEM events are current except for the Ubiquiti backlog. We will be processing these events throughout the afternoon and closely monitoring for any negative impacts on ingestion of other event types. Another update will be provided as soon as possible.
We have determined that delays for Ubiquiti integration events were leading to ingestion delays for other event types. We have temporarily disabled Ubiquiti ingestion to allow all other event types to ingest at normal rates. The team will continue to work on a fix for Ubiquiti integration ingestion delays. Another update will be provided as soon as possible.
The team is investigating delays in ingestion for a subset of SIEM events. All data is intact and we will provide another update as soon as possible.
Report: "Intermittent 502s with SIEM search"
Last updateThe incident has been resolved. The system is fully operational.
The fix has been deployed and the system is fully operational. We will continue to monitor the situation.
We have identified the issue at hand and started deploying the fix.
We are making progress on isolating a root cause. This issue is still under investigation. We will provide a further update as soon as possible.
We are continuing to investigate the issue. Another update will be provided as soon as possible.
We are currently investigating an issue where SIEM search intermittently returns a 502.
Report: "Intermittent delays with SIEM search"
Last updateAfter monitoring we are confident the issue is resolved and the system is fully operational.
After investigation it appears the issue has resolved itself. We will continue to monitor.
We are currently investigating an intermittent issue where the loading SIEM search results is delayed.
Report: "SIEM Data Loading Delays"
Last updateThe system has continued to perform well throughout our monitoring period. The issue has been fully addressed, and all services are operating normally.
We have been monitoring the impact of our solution and the system has returned to normal operation. We will continue to monitor.
We have identified the potential cause and are working on a solution to resolve it.
We are currently investigating intermittent loading delays of SIEM data within the management portal. All data is intact, this is impacting visualization only. A further update will be provided as soon as possible.
Report: "Users may experience delays when making updates to portal"
Last updateThe issue is resolved. The portal is fully operational and there are no ingestion delays.
The fix has been deployed and validated. The system is fully operational. We will continue to monitor this situation.
We have identified the cause and the fix is being deployed. As a result of this users may see ingestion delays for new events from Office365 and Azure over the next few minutes.
We are currently investigating an issue where users may experience a delay in being able to view the updates that were made to portal.
Report: "Possible slow login to portal with 2FA"
Last updateA fix has been deployed and the issue has been resolved. Logging in to portal with 2FA is back to normal.
We have identified the problem and we expect the fix to be applied in the next 30 minutes.
We are currently seeing an issue where there is a longer than normal delay between logging into the portal and being prompted for your 2FA credentials. This only impacts the login.
Report: "Delays in Agents connecting to SGN"
Last updateThis incident has been resolved.
The fix has been applied globally, and all agents should see normal performance. We are continuing to monitor this closely. If an agent still experiences delays in connecting to SGN, please reconnect it from the management portal, or the device itself.
We are currently experiencing an issue with SGN connectivity. A subset of Agents are taking longer to connect. The problem is understood and a fix is being applied.
Report: "Delays in Agents connecting to SGN"
Last updateThis incident has been resolved.
The fix has been applied globally, and all agents should see normal performance. We are continuing to monitor this closely. If an agent still experiences delays in connecting to SGN, please reconnect it from the management portal, or the device itself.
We have remediated the issue for the majority of agents. Some agents may still intermittently experience this issue. We are continuing to apply the fix globally.
We are seeing connectivity being restored for the subset of Agents - any agents that remain impacted should be reconnected via the Management Portal - https://portal.todyl.com/device/list
We are currently experiencing an issue with SGN connectivity. A subset of Agents are taking longer to connect. The problem is understood and a fix is being applied.
Report: "Intermittent issue with agents connecting to SGN"
Last updateThe issue has been resolved. If any of your devices are still experiencing connectivity issues, please reconnect the agent or reboot the device.
The fix has been applied to all SGN PoPs.
We are experiencing an issue on 1 of the 3 protocols that is used to connect to SGN. The problem is understood, and a fix is being applied. This will primarily impact agents installed on Linux, Windows Server 2012, Windows 8 and older.
Report: "SIEM Ingestion & Alert Triggering Delays"
Last updateThis incident has been resolved.
The team is continuing to investigate delays in ingestion.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
The team is investigating delays in alert triggering. An update will be available in 30 minutes, if not sooner.
Report: "SIEM Ingestion Delay - Subset of Data and Agents"
Last updateThis incident has been resolved.
85% of the subset (5%) of events have now ingested successfully along with all new events which indicates that our fix is working as intended. We will mark this issue resolved once all events are ingested.
A fix has been implemented and we are monitoring the results.
The issue has been identified and the team is working on a fix. Another update will be provided as soon as it become available.
We are currently investigating a SIEM ingestion delay that is impacting 5% of SIEM data for a subset of agents (less than 30%). Another update will be provided as soon as it becomes available.
Report: "Network Performance Issues - Atlanta Region"
Last updateThis incident has been resolved.
The issue with the degraded nodes has been identified and a fix is being implemented. The degraded nodes remain out of the connection pool so they will not accept new agent connections.
An additional degraded node have been identified and removed from the connection pool. Please reconnect any remaining impacted agents in the Atlanta region to establish a connection to a new node. Another update will be provided as soon as possible.
A degraded node has been removed from the connection pool. Please reconnect any impacted agents in the Atlanta region to establish a connection to a new node. Another update will be provided as soon as possible.
We are currently investigating network performance issues impacting the Atlanta region. Another update will be provided as soon as possible.
Report: "Todyl.com - Intermittent Errors"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
The team is investigating an issue with intermittent errors while loading resources at https://www.todyl.com The management portal is still available at https://portal.todyl.com/session/login An update will be provided as soon as it becomes available.
Report: "Cloud to Cloud Integration Ingestion Failures"
Last updateThis incident has been resolved.
A fix has been implemented we are monitoring the results. We expect some data loss between 11:30 p.m. UTC and 12:20 a.m. UTC.
The issue has been identified and a fix is being implemented.
Cloud to Cloud data is currently failing to be ingested, this will impact detections and the ability to search on these events through the portal.
Report: "Milan Region Power Failures"
Last updateThis incident has been resolved.
The team is monitoring power failures impacting availability of the Milan region. The SGN Connect agent will connect to the closest, fastest region. We will provide another update as soon as it become available.
Report: "Performance Degradation - New York & Chicago"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring results. We advise reconnecting any impacted agents at this time.
The issue has been identified and we are implementing a fix.
The team is investigating intermittent degraded performance in the New York and Chicago regions.
Report: "Todyl Management Portal Delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
In the process of implementing a fix, the team has identified other inconsistencies that are under investigation.
The team is continuing to work on a fix for this issue.
The team has identified an issue that is causing intermittent delays in loading resources from the Todyl management portal and is working on implementing a fix.
Report: "AT&T Outages Impacting Connectivity To Dallas Region"
Last updateConnection paths between certain AT&T locations and our Dallas Region appear to have been restored.
We are continuing to monitor AT&T outages and it is still advised to connect any impacted agents to another region via connection policy in the Todyl management portal.
We are currently monitoring ISP (AT&T) outages in the Dallas area. This has led to a subset of devices experiencing performance and connectivity issues to the Dallas region. At this time, we advise utilizing connection policy to connect impacted agents to another region.
Report: "Agent Connection Investigation - Seattle Region"
Last updateThis incident is resolved. The title has been updated to reflect that only the Seattle region was impacted.
Agent connectivity is operating as expected in both regions. We are continuing to monitor stability.
We have identified a temporary network outage that impacted the Seattle region. At this time agent connections to the Seattle region are recovering. Connectivity to Vancouver is still being investigated.
We are investigating agent connectivity degradation that is believed to impact the Seattle and Vancouver regions. As the scope of impact becomes more clear we will provide an additional update.
Report: "DNS Resolution Failures - Washington DC"
Last updateThis incident has been resolved.
A fix has been implemented with the service provider. The Washington DC region has been placed back in the connection pool and we are monitoring results.
We have identified an upstream failure and are working with the appropriate service provider to implement a fix.
We are continuing to investigate this issue. Another update will be provided as soon as possible.
As the team investigates, DC has been removed from the connection pool. Please reconnect any agents still connected to the Washington DC region to move to the nearest, fastest region that is not impacted.
We are currently investigating DNS resolution failures for the Washington DC region. This impacts agents connected to the region as well as static IP egress from that region.
Report: "SGN Outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Agents and SIEM Search Pages Not Loading in Management Portal"
Last updateThis incident has been resolved. All SIEM events impacted during the event are visible in the management portal.
A fix has been implemented so both the agents and SIEM search pages are now accessible. This incident will remain in monitoring status as we validate that any SIEM events impacted by this event ingest properly to be visible within the management portal.
The issue has been identified and a fix is being implemented.
While investigating, we have determined that this impacts the SIEM search page as well. The incident has been escalated from degraded performance to partial outage. We will provide another update as soon as it becomes available.
We are currently investigating an issue with loading the agents page within the Todyl management portal.
Report: "Connection Delays for 2012 Servers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results. A subset of devices will connect shortly.
The issue has been identified and a fix is being implemented.
The team is investigating a delay in connection times for a subset of 2012 R2 servers. An update will be posted within the hour.
Report: "SGN Auckland Reachability"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
We are currently still waiting on an update from the Auckland Datacenter about resolution to this issue. An update will be posted when we have received one.
We have confirmed that there is an outage in the Auckland Datacenter and we are working towards a resolution with our providers.
We are currently investigating connection issues in the Auckland Region