Historical record of incidents for AlertOps
Report: "Inbound SMS ACK/Assign/Close failing"
Last updateThis incident has been resolved.
Service is functioning normally now.
Report: "SMS ACK/Assign/Close inbound"
Last updateThe services is back to normal
Report: "AlertOps Delivery Thread Cancellation Issue"
Last updateOn June 4, AlertOps experienced a system-wide issue causing alert escalations to persist even after alerts were assigned or closed. As a result, users continued to receive escalation notifications despite taking action on those alerts. This issue was caused by a code regression that disrupted the logic responsible for halting escalation threads when alerts are updated with a resolved or assigned state. We sincerely apologize for the inconvenience and appreciate your patience as we worked to resolve the issue.
Report: "AlertOps US VOIP provider outage"
Last updateAs of 8:33 the issue has been resolved. US outbound VOIP (voice) notifications are operating normally.
Our US VOIP service provider is down. One of their data centers is down. They are working on Failover backup. We do not have an ETA. This affects only US outbound voice notifications, inbound calls (live call routing) are not impacted by this outage. All other notification channels are functioning normally (SMS, email, Microsoft Teams, Slack, mobile push notifications), ,
Report: "AlertOps US VOIP provider outage"
Last updateOur US VOIP service provider is down. One of their data centers is down. They are working on Failover backup. We do not have an ETA. This affects outbound US VOIP notifications.
Report: "Inbound web API intermittent processing"
Last updateThis incident has been resolved.
Beginning at 11:44 AM CST some inbound requests to the inbound web API received 500 errors
Report: "Intermittent SMS Notification Failures"
Last updateThere were Intermittent SMS Notification Failures due to service provider issues between 12:00 am 1:00 am
Report: "AlertOps Android mobile app intermittent push notification delivery issues"
Last updateFor Android users only. Please go to the Android Playstore and download the Beta version of AlertOps.
Users of the AlertOps Android mobile app may be experiencing intermittent push notification delivery issues. We are currently investigating the issue.
Report: "Inbound Processing Issue"
Last updateBetween 2:28 pm CST and 2:35 pm CST, Our monitoring systems detected an issue with few inbound requests failing with 500 response code. This was due to an unexpected failure on one of our servers. Our service provider identified and automatically remediated the issue.
Report: "Inbound API processing issue"
Last updateAt 1:30 PM CT this incident was resolved .
We are continuing to investigate this issue.
We are currently investigating an issue with processing of inbound API messages. Most incoming messages may not be processed during this time. Some inoming messages are processed. This affects both incoming email and incoming web API messages.
Report: "Inbound email API not processing incoming emails"
Last updateWe experienced an email API inbound outage from 6:30 to 10:00 AM November 30. During this time AlertOps incoming email API was unable to receive and process any incoming emails. No other services were impacted by this outage.
Report: "AlertOps web app intermittent connectivity issues"
Last updateThis incident has been resolved.
Inbound integrations and web API may also be impacted.
At 11:15 the AlertOps web app began to experience intermittent connectivy issues due to a Microsoft Azure issue. This affects the web application only. Microsoft is currently looking into the issue. We will continue to monitor the situation.
Report: "Inbound errors and high latency"
Last updateBetween 6/13/22 11:00 pm CST and 6/14/22 00:45 am CST, we identified an issue with receiving inbound requests. A fix has been implemented, and service is back to normal.
Report: "Delays in workflow module"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Intermittent delays and failures with outgoing SMS messages"
Last updateThis incident has been resolved.
The downstream provider has mitigated the issue and they are now monitoring the recovery.
We are continuing to investigate this issue.
AlertOps is currently experiencing Intermittent delays and failures with some outgoing SMS messages. Customer may not receive some SMS messages or may get delayed delivery of others, while SMS messages are still going through as well. All other outbound messaging services (voice, email, push, chat) are operational and not impacted by this issue. We are monitoring our downstream provider as they are working on the issue.
Report: "Delays in delivery of outbound messages"
Last updateThis incident has been resolved.
We are currently experience delays in delivering outbound integrations messages. Though the messages are going through, customers may see delays in receiving communications messages from AlertOps. We are investigating this issue with our downstream service provider.
Report: "Delay in inbound email receiving"
Last updateThis incident has been resolved.
Our upstream service provider is working to mitigate this issue. Customers may experience delays in receiving emails in their AlertOps inbound email API.
Report: "Intermittent outages"
Last updateIntermittent outages were identified with the AlertOps services beginning at 10:20 AM. The issues were resolved at 10:40.
Intermittent outages with AlertOps services
Report: "Intermittent issues with our AlertOps sites"
Last updateThis incident has been resolved.
(from Microsoft Azure) Starting at approximately 21:30 UTC on 01 Apr 2021, a subset of customers may experience intermittent issues accessing Microsoft services, including Azure, Dynamics, and Xbox Live. (from Microsoft Azure) : Engineering teams have been engaged and are currently investigating a potential underlying DNS issue. The next update will be provided in 60 minutes or as events warrant. This message was last updated at 22:12 UTC on 01 April 2021 (from AlertOps) 22:39 UTC The Microsoft Azure DNS issue has been corrected. All AlertOps services are now operational
We are currently investigating this issue.
Report: "Inbound email processing slow"
Last updateThe issue has been resolved. Services are normal.
We are continuing to monitor the situation and awaiting resolution from our email service provider
We are currently experiencing intermittent slowness in our inbound email processing service. Customers may see delays in incoming alerts from emails. We have identified the cause as being our email processing service provider.
Report: "Inbound email processing slow"
Last updateThe incident has been resolved.
We are currently experiencing intermittent slowness in our inbound email processing service. Customer may see delays in incoming alerts from emails. We will post updates as we continue to investigate.
Report: "AlertOps Service Degradation"
Last updateThe incident has been resolved by our hosting provider. All systems are operational. Read the official status page summary here: https://azure.microsoft.com/en-us/status/history/
The hosting service has upgraded the status from Error to Warning. A fix has been applied. Most services are showing recovery.
Situation has improved.
AlertOps services are currently experiencing a performance degradation. We have been notified by our cloud hosting service that we may be experiencing system wide slowness.
Report: "Apple push notification intermittent delivery"
Last updateOur engineers quickly identified the cause and at 11:00 P.M. UTC implemented an update to resolve the issue. -AlertOps
We are continuing to investigate this issue.
Today, March 12th at 9:00 P.M. UTC we identified an issue related to intermittent delays with Apple® push notification delivery. Our engineers are actively investigating the situation -AlertOps
Report: "Incoming email alert outage"
Last updateAt 3:45 pm we began receiving inbound emails from our email provider. After 15 minutes the logs were completely cleared. At 4:10 pm our email provider announced that the outage had been resolved.
This incident has been resolved.
We are continuing to monitor for any further issues.
Beginning at 3 AM UTC February 4th, 2019, our incoming email provider began experiencing an outage. During this outage AlertOps did not receive any incoming email alerts from our provider to our Inbound Integration Email API. This has no affect on any other processing, including the Inbound Web API. We will continue to monitor the situation with our provider and will provide updates as soon as possible.
Report: "Slow response times"
Last updateWe are continuing to work on a fix for this issue.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue
Report: "AlertOps web site down"
Last updateAll systems operational as of.3:10 pm CST.
All systems are up now.
Latest update from Microsoft: Warning Network Infrastructure - North Central US Last updated 8 minutes ago An alert for Network Infrastructure in North Central US is being investigated. More information will be provided as it is known. Information Alert - North Central US Last updated 13 minutes ago Engineers are investigating alerts in North Central US. Additional information will be provided as it becomes available.
AlertOps web site went down due to an issue with Microsoft SQL Azure network infrastructure. Microsoft is investigating.
Report: "Intermittent slowness on web site"
Last updateThe AlertOps web site experienced intermittent slowness this morning between 6 am and 8 am CST.
Report: "Web app errors"
Last updateThe issue has been resolved. The web app session is now functioning normally.
We are currently experiencing errors in our web app with the Session caching service. This issue is only impacting users on the web site. Mobile users are not impacted. Inbound and Outbound Services are not impacted, so incoming and outgoing alerts are functioning normally.
Report: "AlertOps web app degraded performance"
Last updateThe AlertOps service provider hosting service has resolved the degraded issue. AlertOps web app performance returned to normal at 8:02 AM CST.
Due to an issue with the AlertOps hosting service provider, the AlertOps web app began experiencing degraded performance at 6:48 AM CST. Inbound API services and outbound notifications were not impacted.