Is ThreatX Down Right Now? Check if there is a current outage ongoing.

The incident has been resolved and all service are operating normally.

We are currently investigating an issue with API that is also impacting access to Ctrlx. Sensors remain unaffected.

This incident is now resolved.

ThreatX Engineering is continuing to investigate this issue. At this time we believe the problem is limited to the ThreatX Dashbaord. Hosted WAF operations are not affected. ThreatX Engineering is actively investigating this issue and will post an update as more information becomes available.

ThreatX Engineering is investigating a backend service outage. Most backend services appear to be affected. Hosted WAF operations are not affected. ThreatX Engineering is actively investigating this issue and will post an update as more information becomes available.

ThreatX Engineering believes the problem has been addressed. All customers should now be able to login without delays through all SSO providers.

ThreatX Engineering has observed the connection issues appear to be subsiding. At this time we are no longer observing client timeouts for any customer attempting to log in to the ThreatX Dashboard. We will continue to monitor and troubleshoot for slow login symptoms.

ThreatX Engineering is continuing to investigate this issue. We believe we have narrowed down the problem to customers using Microsoft Azure SSO to log in to the ThreatX Dashboard. All other functionality including application protection appears to be unaffected at this time. ThreatX Engineering is actively investigating this issue and will provide updates as more information becomes available.

ThreatX Engineering has identified degraded performance and intermittent timeouts affecting the ThreatX Dashboard page. The problem appears to be an intermittent issue at this time. ThreatX Engineering is actively investigating this issue and will provide updates as more information becomes available.

This incident has been resolved.

All ThreatX services are operating within normal parameters. This incident is now resolved.

ThreatX Engineering is continuing to monitor closely as our hosting provider continues to restore connectivity to affected services. Some processing delays may continue to exist as affected systems recover. We will update the status page once all services are operating within normal parameters.

Our hosting provider, IBM, has begun restoring services that were impacting ThreatX. As systems come back online, some processing delays may occur. ThreatX Engineering is monitoring closely and will update the status page once all services are operating within normal parameters.

We are currently experiencing an outage impacting our main platform and API services due to an issue with our hosting provider, IBM. Our team is actively working with IBM to resolve the problem and restore service as quickly as possible. All sensors and protection remain fully operational. We will provide updates as soon as we have more information.

Issue resolved, all looks green across the board.

Services are operating normally and we are continuing to monitor.

ThreatX Engineering has detected intermittent degraded performance affecting ThreatX hosted services (Dashboard, API, Gateway, and Risk Analytics Engine). We are currently investigating the issue. At this time we have no reason to believe protection of customer sites (ie Hosted WAF instances) is affected in any way.

Incident has been resolved, all services are operating normally.

ThreatX Engineering has identified and removed a faulty service instance. Performance now appears to be within nominal levels. ThreatX Engineering is monitoring the system for any potential additional delays.

ThreatX Engineering is actively investigating an issue in which sensor updates are delayed in taking effect on customer sensors. At this time we believe updates do eventually reach customer sensors after an atypical delay of up to several minutes. This issue appears to be intermittent for affected customers. Existing security protections and site configurations already in place are not affected by this issue. ThreatX Engineering will provide updates as we learn more.

ThreatX Operations has confirmed this incident is no longer active.

ThreatX Operations has identified the issue as originating in a cloud services provider and affecting a small subset of ThreatX customers. ThreatX SOC is contacting affected customers directly to coordinate responses.

ThreatX Security Operations has detected what appears to be a performance degradation affecting hosted sensors in certain regions in Canada. Affected customers may experience intermittent degraded performance. ThreatX Operations is actively investigating this incident to determine the extent of the impact and what corrective action can be taken.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently experiencing issues with a 3rd party vendor that is resulting in delays and intermittent outages to threatx services. We are actively troubleshooting and have reached out to our vendor for support. Please note that active sensors are still functioning/protecting, however new sensors will fail to initialize. We will update this incident as more information becomes available.

This incident has been resolved.

ThreatX Engineering is currently investigating a partial outage in customer Analytics Events data. We have confirmed some analytics events data for some customers are either delayed or experiencing "gaps" in reporting. We do not yet have confirmation of which customers are affected and for which event types and time periods. We will update this status page with more information as it becomes available. Update: To clarify, ThreatX protection remains in effect for all customers. This incident applies only to reported analytics event data.

After an additional period of monitoring, ThreatX Engineering is satisfied that all affected customers have now resumed normal risk processing operations. This incident is now resolved.

ThreatX Engineering has identified the likely source of the problem and has implemented a fix. Processing rates appear to be returning to normal. ThreatX Engineering is monitoring closely.

ThreatX Engineering is investigating an issue in which a subset of customer traffic is not being properly processed for risk-based blocking. This outage affects a subset of customers and does not affect other forms of blocking such as request-based blocking or access-list-based blocking. ThreatX Engineering is actively investigating and will post an update as soon as more information is known.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and ThreatX Engineering is monitoring.

ThreatX Engineering has identified a possible cause of the outage and is working on restoring the security rules.

A subset of security rules are not being enforced by ThreatX sensors due to an unknown failure in the security rule distribution mechanism. ThreatX Engineering is actively investigating.

ThreatX Engineering has confirmed the mitigating change was executed successfully and normal access to the ThreatX Dashboard and Provisioning API has been restored.

ThreatX Engineering has implemented a change to address the outage and is monitoring system availability.

ThreatX Engineering has identified a likely cause of the outage and is currently working on a solution.

We are continuing to investigate this issue.

Users of the ThreatX Dashboard have reported being unable to log in. ThreatX Engineering is actively investigating this issue.

This incident has been resolved.

We are aware of an issue with domains resolving to txproxy.com. We are currently investigating and hope to have the issue resolved shortly. This impacts any sensor traffic flow routed through txproxy.com.

INAP has now fully restored their NYJ data center. All services are operating normally. We will continue to monitor and re-issue a status alert if new issues arise.

Latest update from INAP: Update - The Secaucus data center remains powered down at this time, per the fire marshal. Evocative onsite management met with the fire marshal and electrical inspectors to review the current state of the facility. The fire marshal has reviewed the current level of progress on the cleaning and requested additional remediation actions around the fire system components. The follow up inspection from the fire marshal is 9:00 AM EDT, Wednesday. The Evocative team is bringing on additional personnel and vendors to comply completely with the additional requests by this deadline. In preparation for being able to allow clients onsite, the fire marshal has stated that Evocative must perform a full test of the fire/life safety systems. This must be performed after utility power has been restored and fire system components replaced. Vendors have already been scheduled and are standing by to complete this work tomorrow. If there are no significant changes in the timeline, the earliest INAP will be allowed back into the site to power up servers will be later in the day on Wednesday. We anticipate the next update of significance will be provided by Evocative Wednesday morning, and will keep you updated as information becomes available. Jul 11, 2023 - 16:04 CDT

Latest update from INAP: Update - Dear Valued Customer, Evocative has worked with their remediation vendor through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and aim to have the cleaning completed by mid-day. There is currently a 2PM EDT meeting scheduled, during which Evocative staff and local township officials will perform a walk-through of the site. Evocative will need approvals from the Mayor, the fire marshal, and the local utility before the site can be re-energized. Depending on their inspections and approvals, Evocative staff will then be able to access the site and initiate the sequence to bring all data center systems back online. Evocative will be providing us a status update after this meeting takes place. While we do not have an ETA for when power will be fully restored, the re-energizing process can take 4-5 hours after it is initiated. Regards, INAP Cloud Support Jul 11, 2023 - 11:35 CDT

Latest update from INAP: Update - Dear Valued Customer, As we reported earlier, the incident is related to a UPS system in the Secaucus data center owned and operated by Evocative. There was a fire in the UPS room, which was extinguished, but the incident resulted in a shutdown of all power to the facility and the involvement of the local fire department. This is the reason that redundant power systems did not keep any of the servers up and operational. The fire marshal is requiring extensive cleaning of the affected systems and areas before the site can be re-energized. That work is in progress. We were informed that Evocative will need to get an OK from the Mayor, the fire marshal, and the local utility to re-energize the site; depending on their inspection and approval, they can enter the site and begin to sequence bringing all data center systems back online. There is currently no ETA, but we will continue to provide updates. Once that is done, we have our teams standing by both locally and within our engineering and support organizations to ensure that all servers and services are brought back online as swiftly as possible. Regards, INAP Cloud Support Jul 11, 2023 - 08:06 CDT

Further updates from INAP below. It is looking likely that there will be no resolution before 08:00 EDT tomorrow. Update - Our provider has shared further details about the incident and current status. A small electrical failure occurred with one of the redundant UPS devices which began smoking and led to a small fire in the UPS room. The fire department was dispatched and the fire extinguished quickly. The fire department subsequently cut power to the datacenter and disabled the generators while they and the utility company inspected the electrical system. The fire marshal is requiring extensive cleaning of the UPS devices and rooms before the site can be reenergized. There is a vendor onsite currently who will be performing the cleanup. The next update from our provider will be at 8:00 AM EDT unless there are significant changes overnight. We will continue to pass along information as we receive it. Jul 10, 2023 - 22:21 CDT

See below for an additional update from INAP. Update - Dear Valued Customer, We have spoken with our provider and received additional details and a tentative ETA. There was a problem within the UPS system that required the site to be fully powered down. Vendors are onsite and working to resolve the issue. The current estimate is that power should be restored within the next several hours. We will continue to work with our provider and post regular updates. Regards, INAP Cloud Support Jul 10, 2023 - 18:59 CDT

From INAP Update - Dear Valued Customer, We are seeing intermittent power availability issues within this datacenter and are working diligently with the provider on a resolution. Regards, INAP Cloud Support Jul 10, 2023 - 18:02 CDT

INAP is experiencing network connectivity issues in their NYJ data centers which may be impacting ThreatX hosted sensors in that region. Please note that this is only impacting INAP sensors in NYJ, AWS sensors are not impacted. We will update this page as more information becomes available.

This incident is now resolved. More information from service provider: 1:06 PM PST Between 11:34 AM and 12:51 PM PST, customers experienced Internet connectivity issues for some networks to and from the US-EAST-2 Region. Connectivity between instances within the Region, in between Regions, and Direct Connect connectivity were not impacted by this issue. The issue has been resolved and connectivity has been fully restored. More information available at https://health.aws.amazon.com/health/status

The partial outage appears to have been resolved by the computing resource provider in question. ThreatX Engineering is continuing to monitor the situation.

ThreatX Engineering has confirmed the incident is due to an Internet Connectivity problem reported by AWS, affecting their US-EAST-2 Region. More information available at https://health.aws.amazon.com/health/status. ThreatX Engineering is continuing to investigate the situation and will post updates as we learn more.

ThreatX Engineering is investigating a partial outage affecting customers deployed in the tx-us-east-2a availability zone. This outage prevents users in certain areas in the US Midwest from connecting to ThreatX sensors and using the ThreatX Dashboard. ThreatX Engineering is actively investigating the incident and will provide updates when available.

ThreatX Engineering has concluded their investigation and has determined the system is now stabilized.

The period of performance degradation has abated. ThreatX Engineering is continuing to monitor and investigate.

ThreatX Engineering is investigating degraded performance in the ThreatX Dashboard web application. We will post updates as we learn more about the extent of the impact.

This incident has been resolved with the release of ThreatX Dashboard 4.55.1.

ThreatX Engineering has identified a regression in the ThreatX Dashboard web application preventing a subset of SSO tenant accounts from logging in. ThreatX Engineering is planning on resolving the issue by executing a release of the ThreatX Dashboard on October 27, 2022, most likely in the hours between 16:30 and 20:30 UTC. Users may see intermittent page load errors during the time of release, which should be resolved by reloading the page.

ThreatX Engineering has identified the cause of the outage as a misconfiguration and has resolved the incident. Customers should once again be able to log in and access the ThreatX Dashboard web application as normal. At this time we do not have any reason to believe that ThreatX Protection services were negatively impacted in any way in the course of this incident.

We are continuing to investigate this issue.

ThreatX Engineering has received reports that the ThreatX Dashboard web application is inaccessible and timing out during attempts to login. Engineering has verified the reports and is investigating the problem to determine cause and next steps. At this time we do not have any reason to believe that ThreatX Protection services are negatively impacted in any way.

ThreatX engineering has concluded its investigation into a reported issue involving degraded performance on the Dashboard page. We have determined the problems are caused by a bug in the Dashboard page in which risk reported for entities is out of date. After investigation, we have concluded that this bug does not represent any wider operational issue affecting the ThreatX platform. ThreatX product management will prioritize the Dashboard entity risk bug for a fix in a timely manner. Please note this bug is limited to the Dashboard page and does not negatively affect risk-based blocking. As a workaround, users can view the most up to date risk for an entity on the Entity Details page.

ThreatX engineering is investigating an issue in which the Dashboard page is suffering from degraded performance. Preliminary investigations appear to show recent data displayed in the Dashboard is missing or incomplete. As a workaround, customers may use the Explorer or Live View pages until the incident is resolved. This issue in the Dashboard page does not appear to have any impact on ThreatX protection services.

AWS has published an update indicating the operational issue in their us-east-2 region is now fully resolved. Operations continue to be stable on the ThreatX platform and for all customer infrastructure operated by ThreatX. This incident now appears to be resolved.

ThreatX technical staff have determined that all customer infrastructure and platform infrastructure appear to be operating under normal parameters. The AWS Health Dashboard still indicates ongoing recovery efforts. We are continuing to monitor the situation and will update this status page with any further developments.

AWS has posted an update to their AWS Health Dashboard with more details on the problem affecting a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region. AWS reports they are in the process of rectifying the situation. ThreatX on call staff are continuing to investigate and monitor the situation.

ThreatX on call staff are investigating a partial outage in the US-east-2 availability zone. This outage potentially affects customers with sensors deployed in the AWS us-east-2 availability zone, as well as customers with ThreatX accounts in the ThreatX us-east-2a availability zone. ThreatX staff is actively investigating and will update with more information as it becomes available.

ThreatX engineering believes this incident is now resolved.

The ThreatX Dashboard web application and services supporting the web application are now available. ThreatX engineering is continuing to monitor the situation.

The service powering the ThreatX Dashboard web application is currently unavailable. This incident is most likely related to the ongoing maintenance being performed but was not explicitly planned as part of the maintenance effort. ThreatX engineering is investigating the outage. At this time we do not believe any customer traffic is affected. We will update as soon as we know more details.

The ThreatX Dashboard is now available. ThreatX engineering is monitoring the situation to ensure continued access.

The service powering the ThreatX Dashboard web application is currently unavailable. ThreatX engineering is investigating the outage. At this time we do not believe any customer traffic is affected. We will update as soon as we know more details.

This incident has been resolved.

The fix has been deployed to all environments and we are monitoring the results.

We found a resolution and are deploying it to all sensors.

We discovered an issue with our hosting provider DNS services that are preventing some sensors from forwarding traffic. Our engineers are investigating the issue.

This incident has been resolved. We are continuing to monitor for issues with DNS services.

Upstream DNS services are recovering and we are monitoring the situation.

We are currently investigating issues with SSO authentication that appear to be caused by an outage with DNS services. ThreatX engineers are investigating the issue and working with upstream providers to resolve the problem.

ThreatX sensors hosted in INAP Dallas, San Jose, and Amsterdam datacenters experienced connectivity issues from 12:04pm to 12:16pm MST. Customers with sensors in theses datacenters may have experienced connectivity issues. Everything is back to normal operations as of 12:16pm MST.

The issue was related to an external network provider (Centurylink) and appears to be resolved at this time.

We are currently investigating this issue.

This incident has been resolved.

All systems normal and operating at expectation

Intermittent packet loss has started to improve. Not yet all clear

This incident has been resolved.

ThreatX Customers, We are currently investigating intermittent performance degradation when attempting to utilize our web UI. We will update this status page with any relevant updates as well as when performance is back to 100%. Best, ThreatX Team

ThreatX Customers, This incident has been resolved by the hosting provider and operations will continue as normal with traffic being routed through sensors in SJC. Thank you, ThreatX Team

ThreatX Customers, Our hosting provider is currently experiencing intermittent connectivity issues affecting our sensors hosted in SJC, which will cause application traffic to route through redundant WAF sensors hosted in other locations until the issue is resolved. We are currently working with our provider in the region to resolve this issue and will send a follow-up notification when the issue has been resolved. The below message is from our provider: "Networking engineering saw a second impact to connectivity at our SJC2 datacenter. The problem has been identified and a corrective measure is being implemented urgently to prevent further occurrences. As soon as the corrective measure is in place, a notification will be sent." As soon as an update is provider from our provider, this notification will be updated as well. Thank you, ThreatX Team

ThreatX Customers, We have resolved the issue with hosted sensors in Dallas, and connectivity has been successfully established again. We are fully dedicated to providing certain availability and uptime for our customer web applications. As a result, ThreatX deploys a redundant sensor infrastructure across various provider networks as well as physical locations so that when an incident like this does occur, whether it's related to provider networking or the physical location itself, it remains isolated and does not negatively impact availability for customer applications. Thank you, ThreatX Team

ThreatX Customers, We are currently investigating intermittent connectivity issues affecting our sensors hosted in Dallas, which will cause application traffic to route through redundant WAF sensors, hosted in other locations, until the issue is resolved. We are currently working with our provider in the region to resolve this issue and will send a follow-up notification when the issue has been resolved. Thank you, ThreatX Team

ThreatX Customers, We had a minor incident with a status check that led to approximately 2 minutes of unavailability for the web interface while the services were being rebooted. This incident had no impact on WAF operations or rule enforcement. This has been resolved and will continue to be monitored.

The incident has been resolved and performance has been verified to be 100% again.

ThreatX Customers, We are currently performing unplanned database maintenance that is causing intermittent service degradation to the ThreatX Dashboard as well as the Provisioning API. This maintenance does not impact WAF operations, and rules are still being enforced as expected.

ThreatX Customers, We are currently performing unplanned database maintenance that is causing intermittent service degradation to the WebUI (CtrlX) as well as the Provisioning API. This maintenance does not impact WAF operations, and rules are still being enforced as expected.

We experienced degraded performance with the analytics engine, resulting in intermittent loss of event information. The issue was due to a bad pre-processor container. Customer production flows were not affected, rules were still being enforced as usual.

Microsoft has resolved the Azure issue. Services have returned to normal.

We are watching an ongoing condition with Azure served origin servers https://status.azure.com/en-us/status that is effecting some sites returning 504 timeouts getting content.

Connectivity to INAP NYJ datacenter restored by the provider. Sensors are back in the rotation.

INAP NYJ datacenter lost connectivity. Customer traffic failed over to other US regions.