NinjaOne Trust Incident History

This incident has been resolved.

Mitigation efforts have shown positive results and we are seeing agents reporting status correctly at this time. We are continuing to monitor the incident.

To help mitigate some agents still showing offline we are restarting our frontend web systems. Some users may see a slight increase in response time during this process.

During the AWS Outage, Ninja Agents attempted and failed to reconnect to our backend services. This automatically triggers a backoff/retry mechanism in our agents that is designed to prevent traffic flooding or thundering herd behavior during significant outages. Agents are slowly reconnecting, but will randomize their reconnection attempts until successful. Additionally the service that was responsible for processing device status was struggling with the volume of agent status changes. We have scaled that capacity out and are seeing healthy metrics. Over the last hour we have seen our connected devices increase by over 150k devices, but recovery will likely be slow while the agents attempt to gracefully reestablish service

We are continuing to monitor for any further issues.

We have identified a broader connectivity issue within our cloud provider's infrastructure that is causing intermittent service disruptions for some customers. Recovery efforts are ongoing and service stability has improved, though some impact may persist while restoration work continues.

We have identified a broader connectivity issue within our cloud provider's infrastructure that is causing intermittent service disruptions for some customers, and we are actively monitoring recovery efforts.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have noted that this behavior is also seen in the US2 region. We are continuing to investigate the issue.

We are aware of intermittent issues with connecting to devices via Ninja Remote. We are currently investigating this issue

The issue causing automations, scripts, and scheduled tasks to remain queued on NMS delegates has been resolved for affected customers in the EU region following a successful targeted hotfix, which we have confirmed through testing. To fully address this across all regions, we will be rolling out the same fix via scheduled maintenance plans next week. Each affected region will receive advance notice of its specific maintenance window through this status page. No action is required from customers in the meantime. We appreciate your patience and apologize for any disruption this may have caused.

Our testing indicates the issue is remediated within EU. We are assessing the scope of customers affected in our other regions so we can hotfix as necessary to minimize impact to all customers. More updates to come as we progress through those remediation steps.

We have successfully deployed the hotfix to the affected customers in EU. We are validating the fix is working as expected.

We are preparing to deploy a targeted hotfix for the customers who are currently affected by this issue in our EU region. There will be a brief disruption of API and Web portal access for a subset of customers while we push the change.

We have identified that the issue is only affecting NMS delegates. To identify NMS delegates navigate to the Devices page, select the More Filters button in the top bar and select NMS Delegate.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Some customers may experience automations, scripts, and scheduled tasks remaining in a queued state and not executing. Affected devices will otherwise appear online and healthy. We have identified the cause and a fix is in progress. Further updates will follow as deployment timing is confirmed.

This incident has been resolved.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

The issue has been identified and working on the remediation

We are currently investigating this issue.

We will be undergoing scheduled maintenance.

We will be undergoing scheduled maintenance.

We are continuing to investigate this issue.