Historical record of incidents for Abnormal Security
Report: "Google Service Disruption Impacting Abnormal Email Security Services"
Last updateStarting around 17:50 UTC on June 12th, Abnormal experienced service disruptions affecting Inbound Email Security, AI Security Mailbox, and Graymail services for Google Workspace customers in both US and EU regions due to a widespread Google outage. As of the latest update, Google services are showing signs of recovery. Abnormal's engineering team continues to monitor the situation closely to ensure full restoration of all services. During this period, some email security functions may experience intermittent delays. Abnormal will provide another update once all services have been confirmed fully operational or if there are any changes to recovery status.
Report: "Inbound Email Security and Email Productivity – (EU)"
Last updateAs of 00:53 UTC, our engineering team has successfully resolved the issue impacting Inbound Email Security and Email Productivity in the EU. Emails affected during the incident window have been reprocessed and remediated. If you experience any further issues, please reach out to our support team at support@abnormalsecurity.com
Starting at 23:50 UTC on April 17th, Abnormal began experiencing service degradation affecting Inbound Email Security and Email Productivity. EU customers may experience delayed email remediation. Our engineering team is actively working to mitigate the issue and restore full functionality. We will provide the next update within the next hour or sooner as more information becomes available.
Report: "Inbound Email Security Message Remediation Delays – (EU)"
Last updateAs of 20:25 UTC, the backlog of messages from the incident has been fully processed. Message remediation for Inbound Email Security impacting EU customers has fully recovered, and the issue is now resolved. If you experience any further issues, please don’t hesitate to reach out to our customer support team at support@abnormalsecurity.com. We appreciate your patience throughout this incident.
As of 18:10 UTC, we’ve begun to see recovery in message remediation for Inbound Email Security impacting EU customers. Our team is now focused on processing the backlog of messages accumulated during the incident to ensure full restoration of service. Thank you for your continued patience as we work to resolve this.
Starting around 16:00 UTC, Abnormal began experiencing message remediation delays affecting Inbound Email Security for our EU customers due to backend system errors. Our engineering team is actively investigating and working to restore full functionality. We’ll provide the next update within 60 minutes or sooner as more information becomes available.
Report: "Microsoft Outage causing Intermittent Remediation issues."
Last updateAbnormal services have fully recovered from the Microsoft outage, and the backlog of messages from the incident duration has been successfully processed. We appreciate your patience as Microsoft worked to restore its services. If you have any questions or continue to experience issues, please reach out to our support team at support@abnormalsecurity.com.
Abnormal services are recovering as we continue to monitor progress. We are actively working through the backlog of messages from the incident duration to ensure all pending items are processed. We appreciate your patience as Microsoft worked to restore its services. If you have any questions or continue to experience issues, please contact our support team at support@abnormalsecurity.com.
Microsoft has confirmed that a platform upgrade is causing an availability issue with Azure Event Hubs Premium, impacting Abnormal services. Microsoft has engaged additional engineers to investigate and implement a fix. We are starting to see a partial event Hub recovery, and continue to actively work with Microsoft engineering and support teams to fully resolve the issue as quickly as possible. We will provide updates as more information becomes available. Thank you for your patience.
Starting at 18:20 UTC on March 5, 2025, Abnormal began experiencing intermittent remediation issues across our products due to a widespread Microsoft outage. Microsoft has acknowledged the issue and is actively working to restore service. As a result, you may notice delays or disruptions in remediation functionality. We understand the importance of timely email security and are closely monitoring the situation while assessing any possible mitigations to minimize impact. We appreciate your patience and will provide updates as soon as we have more information. If you have any questions or need further assistance, please reach out to our support team.
Report: "US Portal Service Disruption"
Last updateBetween 20:45 UTC and 21:05 UTC, some US customers may have experienced issues logging into the Abnormal Portal and slowness when navigating the platform. Our engineering team quickly identified and addressed the issue, and normal service has now been restored. We sincerely appreciate your patience while we worked to resolve this disruption. If you continue to experience any issues or need further assistance, please reach out to our support team at support@abnormalsecurity.com.
Report: "Microsoft Outage Causing Elevated Latency and Delayed Email Remediation"
Last updateThe Microsoft service disruption that began at 20:50 UTC (12:50 PM PST) has been fully resolved as of 21:45 UTC (1:45 PM PST). Abnormal Security’s email remediation processing times have stabilized and are now operating at normal levels. Engineers are reprocessing messages that were impacted during the incident duration. Throughout the incident, all other Abnormal Security services, including the Portal, remained fully operational. Only email remediation latency was impacted. If you experience any further issues pertaining to this incident, please contact us at support@abnormalsecurity.com.
Microsoft is currently experiencing a widespread service disruption, which, starting at 20:50 UTC (12:50 PM PST), has impacted Abnormal Security’s email remediation processing times. This Microsoft outage is affecting remediation latency for customers across multiple regions. We are beginning to observe signs of recovery as Microsoft works to restore its services. Abnormal engineers continue to closely monitor the situation and are assessing potential mitigations to minimize the impact. All other Abnormal Security services, including the Portal, remain fully operational. Only email remediation latency is impacted. The next update will be provided in 1 hour or as soon as we receive further information from Microsoft regarding service restoration.
Report: "Incident with URL Wrapping Feature Activation"
Last updateOn January 9, 2025, between 09:00 UTC and 16:15 UTC, Abnormal's URL Wrapping feature was unintentionally enabled. This feature is part of ongoing efforts to improve detection efficacy against malicious links. The incident impacted a limited subset of suspicious email messages (Less than 0.1% of total messages processed). We have identified and reverted all impacted messages. There was no disruption to customer business operations or detection efficacy during this time, and employee access to links remained unaffected.
Report: "Account Takeover (ATO) service disruption"
Last updateBetween 14:30 UTC and 17:00 UTC on January 7th, customers experienced empty Account TakeOver (ATO) list views due to a UI-only issue. Detections remained fully operational and were not impacted. A recent deployment was rolled back, and functionality was restored. For any further questions, please reach out to support@abnormalsecurity.com
Report: "Service Disruption impacting US and EU customers due to Microsoft 365 API Latency"
Last updateMicrosoft services have remained stable. Processing for Inbound Email Security, AI Security Mailbox, and Graymail have remained at baseline levels since 22:45 UTC. A postmortem report will be available to share with customers once Microsoft has provided us with details pertaining to this incident. This incident is resolved and no further updates will be provided. If you experience any further issues pertaining to this incident, please contact us at support@abnormalsecurity.com.
Microsoft reports that they are seeing service recovery on their end. Engineers have observed that processing times for Inbound Email Security, AI Security Mailbox, and Graymail services are slowly recovering. Our engineering teams continue to actively monitor for full service restoration. The next update will be provided when services are fully restored or more information is provided by Microsoft.
Microsoft reports experiencing delays in their recovery efforts, which continues to impact Abnormal Security’s processing times for Inbound Email Security, AI Security Mailbox, and Graymail services. Our engineering teams continue to actively investigating optimizations to mitigate the impact of the Microsoft 365 latency. The next update will be provided when we receive further information from Microsoft about their API performance restoration.
Microsoft 365 is experiencing elevated API latency, which beginning at 07:30 UTC has impacted Abnormal Security’s processing times for Inbound Email Security, AI Security Mailbox, and Graymail services. This Microsoft platform slowdown is affecting remediation times for both EU and US customers. Our engineering teams are actively implementing optimizations to mitigate the impact of the Microsoft 365 latency in the US region. EU systems are already showing improvement as Microsoft’s services recover in that region. The next update will be provided in 1 hour or when we receive further information from Microsoft about their API performance restoration.
Report: "Intermittent M365 Graph API Issues Impacting Remediation"
Last updateAs of June 5, 2023 at 10:50pm UTC (3:50pm PT), the intermittent remediation delays have been resolved and customers should no longer experience any further issues with delays. Customers may still see a backlog of messages that experienced the delays being processed. We apologize for any inconveniences that this issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Starting on June 5, 2023 at 2:15pm UTC (7:15am PT), customers may have experienced intermittent delays with email processing and remediation due to a Microsoft Graph API service issue which Microsoft had acknowledged and the initial issue was resolved as of June 5, 4:00pm UTC (9:00am PT). The M365 link regarding the incident is https://twitter.com/MSFT365Status/status/1665734492122742790 As of 7:30pm UTC (12:30pm PT) there have been reports of a recurrence of this issue. Our Engineering team is working with Microsoft for ETA on resolution of their Graph API service. We will provide updates as soon as there is anything significant to report. We apologize for any inconveniences that this issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Dashboard loading issues"
Last updateAs of July 10, 2023 at 10:15 pm UTC (3:15 pm PT), the Abnormal Security Dashboard loading issues have been resolved. Some customers may continue to experience some intermediate delays as the processes stabilize. We apologize for any inconvenience that this issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
As we continue to investigate the issue of the Abnormal Dashboards intermittently taking extended times to load, we have seen better success with limiting the date range for the dashboards. Please be assured we are actively working to resolve the dashboard issues as quickly as possible. Please also be assured that despite this issue, there is no impact on the detection or remediation services we provide. Our systems are still effectively monitoring and responding to all potential threats. We appreciate your patience and understanding. We will keep you informed of our progress and provide timely updates.
Our team at Abnormal Engineering is actively investigating the root cause of the issue affecting the Abnormal Security Graymail Dashboard. We have implemented several fixes that have already improved functionality for some customers. Please be reassured that despite this issue, there is no impact on the detection or remediation services we provide. Our systems are still effectively monitoring and responding to potential threats. We appreciate your patience and understanding. We will keep you informed of our progress and provide timely updates. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
We have identified an issue where the Abnormal Security Graymail Dashboard is not showing data for some customers and will timeout upon loading. A small subset of customers may also experience an issue with the audit logs not loading in the portal. Our team has identified the issue and we are working to complete and test a potential fix. There is no impact to detection or remediation services. We will provide another update as soon as we have one. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Potential Abuse Mailbox delays for Gsuite customers"
Last updateAs of 12:28 am UTC ( 5:48 pm PST), this incident has been resolved and will reprocess all emails submitted to the Abuse Mailbox during the incident. Users do not need to resubmit the affected emails. We sincerely apologize for any inconvenience caused by this issue. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
On July 10, 2023 starting at 2:25 pm UTC ( 7:25 am PST), Google Workspace customers may experience intermittent delays with the Abuse Mailbox reported messages being analyzed. We are working on diagnosing the issue, and will provide an update as soon as possible. This is only affecting Google Workspace customers who use the “Report Phish” option for submission to Abuse Mailbox. All other methods (forwarding to mailbox or mailing list) are operating as normal. Microsoft 365 integrated tenants are not affected by this issue. We apologize for any inconvenience that this issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Abnormal Portal and Remediation Degradation"
Last updateAs of July 13, 2023, at 7:48 PM UTC (12:48 PM PDT), the reprocessing of impacted remediation actions was completed and customers should no longer experience any further issues. We apologize for any inconvenience that this issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
The reprocessing of impacted remediation actions is still progressing. We will provide an update once reprocessing is completed.
The reprocessing of impacted remediation actions is still progressing. We will provide an update once reprocessing is completed.
The reprocessing of impacted remediation actions is still progressing. We will provide an update once reprocessing is completed.
The reprocessing of impacted remediation actions is still progressing. We will provide an update once reprocessing is completed.
The reprocessing of impacted remediation actions is still progressing. We will provide an update once reprocessing is completed.
We have started reprocessing the impacted remediation actions during the incident. We will provide an update once reprocessing is completed.
On July 12, 2023 starting at 5:10 pm UTC (10:10 am PT), some customers may have experienced intermittent access to the portal and with email remediation. The underlying issue was identified and resolved as of 5:46 pm UTC (10:46 am PT). All products were fully operational except for the Abnormal Portal and Remediation. Inbound Email Security - Fully Operational Email Productivity - Fully Operational Abuse Mailbox - Fully Operational Account Takeover - Fully Operational Secure Posture Management - Fully Operational We apologize for any inconvenience that this incident has created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "EMEA Data Center Degraded Performance"
Last updateAs of July 18, 2023, at 11:00 PM UTC, the reprocessing of impacted remediation actions was completed and customers should no longer experience any further issues. We apologize for any inconvenience that this issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details
The reprocessing of impacted remediation actions is still progressing. We will provide an update once reprocessing is completed.
On July 18, starting at 12:50 UTC and lasting 30 minutes, performance was degraded in the EU Data Center. Customers would have seen message detection and remediation issues during the affected time frame. As of 13:20 UTC, the incident was resolved and all new messages are being remediated and judged properly. We have started reprocessing the impacted remediation actions during the incident. We will provide an update once reprocessing is completed. Please open a Support Case or contact Abnormal Support (support@abnormalsecurity.com) for additional details.
Report: "Abnormal Portal and Remediation Degradation"
Last updateOn August 23, 2023, starting at 4:20 AM PT, customers may have experienced intermittent access to the Threat Log and Abuse Mailbox Dashboards and with email remediation. The reprocessing of impacted remediation actions is in progress. We will provide an update once reprocessing is completed. As of 7:50 AM PT services were fully operational. Inbound Email Security - Fully Operational Email Productivity - Fully Operational Abuse Mailbox - Fully Operational Account Takeover - Fully Operational Secure Posture Management - Fully Operational We apologize for any inconvenience that this has caused you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Abuse Mailbox Campaign Processing Delay"
Last updateAs of 12:51 pm UTC ( 5:51 am PT), the delay in processing Abuse Mailbox submissions has been resolved. New submissions should be processed as expected going forward. Abuse Mailbox - Fully Operational Inbound Email Security - Fully Operational Email Productivity - Fully Operational Account Takeover - Fully Operational Secure Posture Management - Fully Operational We sincerely apologize for any inconvenience caused by this issue. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Abnormal Security customers with the Abuse Mailbox feature may experience a delay in processing new submissions. Our engineering team is working to determine the cause and resolve it as quickly as possible. Abuse Mailbox - Degraded Inbound Email Security - Fully Operational Email Productivity - Fully Operational Account Takeover - Fully Operational Secure Posture Management - Fully Operational We apologize for any inconvenience that this has caused you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "ATO Services Degradation"
Last updateAs of September 8 2023 at 3:25 PM PDT The degraded functionality of ATO signals due to Microsoft 365 Graph API issues has been resolved. Customers should no longer be experiencing delays with the ATO events. We apologize for any inconvenience the issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
As of 3 pm PDT, the mitigations that were put in place have improved the latency in ingesting the events and the ATO detection metrics have returned to a ‘normal’ level,. Customers should no longer be experiencing delays with the ATO events. We are continuing to work with Microsoft to determine the next steps. We are monitoring the progress and will update again once we have additional information from Microsoft.
We are continuing to work with Microsoft to determine the next steps. The mitigations that were put in place have improved the latency in ingesting the events. We are monitoring the progress.
We are actively working with Microsoft to determine the next steps. We have put a mitigation plan in place to help increase the number of events being ingested.
As of September, 8 2023 at 7:00 AM PDT Abnormal began experiencing degraded functionality of ATO signals due to Microsoft 365 Graph API issues. The Abnormal engineering team continues to investigate the situation. We will continue to share updates as soon as we learn more.
Report: "Increased amount of False Positives"
Last updateMessage restore has been completed and data ingested in the Threat Log is no longer delayed. All services are fully operational.
Due to this incident, we are experiencing delays in ingesting updated data in the threat log for the impacted and new messages since September 12, 2023 at 3:20 PM PDT. There is no impact to detection or remediation services. Messages impacted earlier and the Threat log will be restored by 11:59 PM PDT, September 12, 2023
Restoring messages is underway and will be completed by 11:59 PM PDT, September 12, 2023. This is an automated process conducted by Abnormal Engineering; thus, no customer action is required.
On September 12, 2023, at approximately 7:24 AM PDT Abnormal made a back-end change that resulted in undesirable message judgments, including an increased number of False Positives and also ATO False positives for approximately 0.0001% of messages/events. This change was reversed at 08:01 AM PDT on Sep 12, 2023. We are in the process of correcting errant judgments and restoring messages to end-user inboxes.
Report: "ATO Case Creation Service was down"
Last updateThe backlog of ATO case creation has been resolved, and the ATO case creation process is back to normal operation. We apologize for any inconvenience the issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Starting September 19, 2023, 11:00 am PDT we stopped creating new ATO cases. This was resolved and on September 20, 2023, 12:00 pm PDT we started creating ATO cases again, starting with the backlog (oldest cases first). We will update when the backlog has been resolved or 2 hrs whichever is first.
Report: "High rate of Spam False Positives"
Last updateThe process for restoring messages has been completed as of 9:54 PM PDT, September 26, 2023. We apologize for any inconvenience the issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
The process for restoring messages has exceeded our initial estimated completion time and is still currently underway. This is an automated process conducted by Abnormal Engineering, and thus no customer action is required. We apologize for any inconvenience the issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Less than 0.02% of all messages were incorrectly remediated due to an attempted improvement for Spam efficacy between 9/26/2023 11:46 PT and 9/26/2023 12:00 PT. This has since been corrected. All incorrectly remediated messages are being restored into the correct folders. The process for restoring messages is underway and should be completed by 4:00 PM PDT, September 26, 2023. This is an automated process conducted by Abnormal Engineering, and thus no customer action is required. We apologize for any inconvenience the issue created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Starting September 26, 2023, at 11:46 am PDT a detection control update was accidentally launched, causing spam False Positives and high persistence rates. This was reverted on September 26, 2023, at 12:00 pm PDT. We are currently reprocessing the affected messages during this time. We will update in 1 hour or a major update.
Report: "Degradation of Inbound Email Protection"
Last updateThe impacted remediation actions were reprocessed as of 10:00 AM PT 09/30. Incident Summary: Starting September 29, 2023, at 07:06 am PDT we experienced an issue affecting our Inbound Email Protection service causing False Negatives and affecting Abuse Mailbox remediations. Affected Services: Inbound Email Protection & Abuse Mailbox Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT Resolution Time: September 29, 2023, 10:49 am PDT Questions?: If you're experiencing any issues following the resolution, please get in touch with us immediately at support@abnormalsecurity.com.
Major Incident Update The reprocessing of impacted remediation actions is still progressing. Next Update: Upon completion of the replay of messages. Incident Summary: Earlier today we had an issue affecting our Inbound Email Protection service causing False Negatives and affecting Abuse Mailbox remediations. Affected Services: Inbound Email Protection & Abuse Mailbox Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT Resolution Time: September 29, 2023, 10:49 am PDT Questions?: For more details or immediate concerns, contact us at support@abnormalsecurity.com.
Major Incident Update The reprocessing of impacted remediation actions is still progressing. Next Update: Upon completion of the replay of messages. Incident Summary: Earlier today we had an issue affecting our Inbound Email Protection service causing False Negatives and affecting Abuse Mailbox remediations. Affected Services: Inbound Email Protection & Abuse Mailbox Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT Resolution Time: September 29, 2023, 10:49 am PDT Questions?: For more details or immediate concerns, contact us at support@abnormalsecurity.com.
Major Incident Update The reprocessing of impacted remediation actions is still progressing. Next Update: 2 hours Incident Summary: Earlier today we had an issue affecting our Inbound Email Protection service causing False Negatives and affecting Abuse Mailbox remediations. Affected Services: Inbound Email Protection & Abuse Mailbox Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT Resolution Time: September 29, 2023, 10:49 am PDT Questions?: For more details or immediate concerns, contact us at support@abnormalsecurity.com.
Incident Summary: We are currently investigating an issue that is affecting our Inbound Email Protection service causing False Negatives and also affecting Abuse Mailbox remediations. Affected Services: Inbound Email Protection Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT Latest Developments: We have identified the root cause and a fix has been applied. We have started the reprocessing of messages during the affected time. Resolution Time: September 29, 2023, 10:49 am PDT Updated Workarounds: N/A Next Update: Upon completion of the replay of messages. Questions?: For more details or immediate concerns, contact us at support@abnormalsecurity.com.
Incident Summary: We are currently investigating an issue that is affecting our Inbound Email Protection service causing False Negatives. Affected Services: Inbound Email Protection Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT Latest Developments: We have identified the root cause and are working to identify a fix. Revised Estimated Resolution Time: TBD Updated Workarounds: N/A Next Update: 10:30 PDT Questions?: For more details or immediate concerns, contact us at support@abnormalsecurity.com.
Incident Summary: We are currently investigating an issue that is affecting our Inbound Email Protection service causing False Negatives. Affected Services: Inbound Email Protection Severity Level: Major Incident Start Time: September 29, 2023, 07:06 am PDT What We're Doing: We are currently investigating to determine the root cause of this issue. Estimated Resolution Time: TBD Current Workarounds: N/A Next Update: 9:30 am PDT Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Email Productivity Remediation Delay (Graymail)"
Last updateStarting at 09:00 am PDT on October 10, 2023, Abnormal Security was made aware of a possible delay in remediation for Email Productivity (EPR / Graymail). Some of the Abnormal EPR customers may have experienced a delay in the remediation of Graymail. A fix was identified and applied at 1 pm PDT on October 10, 2023, with remediation times returning to normal by 2 pm PDT. On Oct 11, 2023, an additional occurrence of the remediation delay was reported. This was identified as an edge case, and a fix was put in place at 9:30 am PDT on October 11, 2023, to address this edge case. As of 9:45 am PDT, all Graymail remediation times were back to a normal rate. We apologize for the delay in posting this update, as we worked to resolve the issue and determine the breadth of impact.
Report: "Degradation of Inbound Email Protection"
Last updateAs of 9:15 PDT, message replay and reprocessing has completed and all false positive judgements have been rectified.
Starting at 18:00 UTC to 18:20 UTC on October 12, 2023, customers may have experienced a small amount of False Positives for inbound traffic. Affected Services: Inbound Email Protection & Abuse Mailbox Severity Level: Major Incident Start Time: October 12, 2023, 18:00 UTC What We're Doing: The issue has been identified and corrected. We are currently in the process of identifying messages for reprocessing. Estimated Resolution Time: October 12, 2023, 18:20 UTC Next Update: Upon completion of the replay of identified messages Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Threat Log Message Details Unavailable"
Last updateAs of October 19, 2023, 07:01 UTC the issue has been resolved and all Threat Log message details are now visible.
Starting at approximately 21:00 UTC on October 18, 2023, our database which powers message details in the Portal Threat Log Details page began experiencing data staleness issues resulting in <content unavailable> being displayed. There is no impact to remediation, only to visibility to message contents within the Portal. Messages older than approximately October 19, 2023 02:00 UTC should be visible, however newer messages will not be visible until the data staleness issues have been resolved. Affected Services: Abnormal Portal / Threat Log Severity Level: Major Incident Start Time: October 18, 2023, 21:00 UTC What We're Doing: The root cause has been identified and we are actively working to remediate the data staleness issues Next Update: October 19, 2023, 09:30 UTC or upon resolution of the data staleness issues Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Abnormal Support Portal access"
Last updateThe Abnormal Support Portal is now accessible as expected. Incident Summary: The Abnormal Security Support Portal was not accessible from within the Abnormal product. Affected Services: Customer Support Portal Severity Level: Minor Incident Start Time: 11/2/2023 09:20 PDT (16:20 UTC) What We're Doing: This has been resolved. Estimated Resolution Time: 11/3/2023 08:20 PDT (15:20 UTC) Current Workarounds: Customers can email support@abnormalsecurity.com with any issues, questions, or concerns or call the support hotline at 866-466-9321 Next Update: N/A Questions?: If you continue to experience any issues accessing the Support Portal, please reach out via support@abnormalsecurity.com.
Incident Summary: We are currently investigating the issue with our customer support portal provider to determine the root cause. Affected Services: Customer Support Portal Severity Level: Minor Incident Start Time: 11/2/2023 09:20 PDT (16:20 UTC) What We're Doing: Working with our customer support portal provider to determine the root cause. Estimated Resolution Time: TBD Current Workarounds: Customers can email support@abnormalsecurity.com with any issues, questions or concerns or call the support hotline at 866-466-9321 Next Update: 5 pm PT ( Questions?: Please contact us at support@abnormalsecurity.com.
We are currently investigating an issue with access to the Abnormal Support Portal. When attempting to access the Support Portal from within the Abnormal product, and selecting the option to sign in using Microsoft SSO, the user is experiencing a looping condition, where they are not able to gain access to the Support Portal. Note: This only impacts access to the Support Portal, and has no effect on any of the Abnormal Products. In the interim, please contact the Abnormal Security Support group by email at support@abnormalsecurity.com
Report: "Degradation of Inbound Email Protection, Graymail, and Abuse Mailbox"
Last updateIncident Summary: Starting at 13:54 UTC, customers may have experienced a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search&Respond Severity Level: Minor Incident Start Time: November 7, 2023, 13:54 UTC What We're Doing: All message processing has returned to normal. The reprocessing of affected messages during the incident was completed by 23:00 UTC. Estimated Resolution Time: November 7, 2023 19:16 UTC Current Workarounds: Customers can email support@abnormalsecurity.com with any issues, questions, or concerns or call the support hotline at 866-466-9321 Next Update: This issue has been resolved. Questions?: Please contact us at support@abnormalsecurity.com.
Incident Summary: Starting at 13:54 UTC, customers may have experienced a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Affected Services: Inbound Email Protection, Graymail, & Abuse Mailbox Severity Level: Minor Incident Start Time: November 7, 2023, 13:54 UTC What We're Doing: A fix has been applied, and we are now processing the backlog of messages affected during the incident timeline. Some customers may continue to experience a delay in message processing, as we work through the backlog of messages. Estimated Resolution Time: TBD Current Workarounds: Customers can email support@abnormalsecurity.com with any issues, questions, or concerns or call the support hotline at 866-466-9321 Next Update: 1 hour Questions?: Please contact us at support@abnormalsecurity.com.
Starting at 13:54 UTC, customers may have experienced a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Affected Services: Inbound Email Protection, Graymail, & Abuse Mailbox Severity Level: Major Incident Start Time: November 7, 2023, 13:54 UTC What We're Doing: The issue has been identified, and we are working to resolve it as soon as possible. We are currently in the process of identifying messages for reprocessing. Estimated Resolution Time: TBD Next Update: 1 Hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Degradation of Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond"
Last updateStarting at 14:15 UTC, customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Also, Search & Respond may be impacted as well. Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. Severity Level: Major Incident Start Time: November 14, 2023, 14:15 UTC What We're Doing: 17:30 PM UTC: All live message processing is recovered and customers should no longer be experiencing message processing delays. This issue has been resolved, and we are now processing messages in real-time. Our R&D team is working on the COE. Our current analysis of the root cause is as follows: -- Our Live Scoring capability leverages several auxiliary services. One of these is the DomainAge service which inspects the age of the domain from which the email originates and employs it as a signal/attribute in judging whether the email is an attack. -- Dependency on one of the auxiliary services is disrupting the throughput of the Live Scoring service which caused a backlog, potentially the DomainAge service. -- This resulted in the delay of 25% of messages -- The issue has been now resolved, and we are processing messages in real-time. Estimated Resolution Time: 17:30 PM UTC Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com
Starting at 14:15 UTC, customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Also, Search & Respond may be impacted as well. Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. Severity Level: Major Incident Start Time: November 14, 2023, 14:15 UTC What We're Doing: We are currently investigating the issue, to determine the Root Cause and identify a fix. Estimated Resolution Time: TBD Next Update: 1 Hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Live Scoring Remediation Delay"
Last updateStarting at 13:00 UTC, customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Also, Search & Respond may be impacted as well. Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. Severity Level: Major Incident Start Time: November 20, 2023, 13:00 UTC Resolved: Live processing has recovered and is healthy going forward for all customers. Estimated Resolution Time: November 20, 2023, 17:02 UTC Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Starting at 13:00 UTC, customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Also, Search & Respond may be impacted as well. Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. Severity Level: Major Incident Start Time: November 20, 2023, 13:00 UTC What We're Doing: We are close to returning to real-time processing and out of the incident. With the backlog reprocessed to delayed remediation during the incident period Estimated Resolution Time: TBD Next Update: 1 Hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Starting at 13:00 UTC, customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Also, Search & Respond may be impacted as well. Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. Severity Level: Major Incident Start Time: November 20, 2023, 13:00 UTC What We're Doing: We are currently investigating the issue, to determine the Root Cause and identify a fix. Estimated Resolution Time: TBD Next Update: 1 Hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "2023-12-07 Remediation delayed for IES and AMB"
Last updateTLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline. Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 09:00 UTC: Processing for messages that were delayed during the incident timeline from 00:00 - 02:15 UTC is complete. The incident is now fully resolved. 02:15 UTC: Live processing for message remediation has recovered for all customers going forward. 01:55 UTC Other required backend services required to ensure message remediation success are healthy. The team is increasing traffic volume to the relevant backend systems to regular message processing volume. 01:45 UTC Database used to ensure message remediation success is healthy. The team is restoring regular message processing traffic volume to the relevant backend systems. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline. Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 02:15 UTC: Live processing for message remediation has recovered for all customers going forward. 01:55 UTC Other required backend services required to ensure message remediation success are healthy. The team is increasing traffic volume to the relevant backend systems to regular message processing volume. 01:45 UTC Database used to ensure message remediation success is healthy. The team is restoring regular message processing traffic volume to the relevant backend systems. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB What We're Doing: The remediation actions that were delayed are currently being processed Estimated Resolution Time: December 08, 2023, 02:15 UTC Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline. Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 01:45 UTC Database used to ensure message remediation success is healthy. The team is restoring regular message processing traffic volume to the relevant backend systems. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB What We're Doing: The team is restoring regular message processing traffic volume to the relevant backend systems. Estimated Resolution Time: TBD Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
TLDR: Due to a large spike in detected attacker activity, a subset of our backend systems are experiencing extremely high load, which is resulting in delayed message movement starting at 00:00 UTC on December 08, 2024, impacting IES and AMB. Messages are being judged correctly, but movement to the appropriate remediation folder is delayed during the incident timeline. Affected Services: Inbound Email Protection and Abuse Mailbox Severity Level: Major Incident Start Time: December 08, 2023, 00:00 UTC Timeline: 01:10 UTC: Current status - investigation of solution for delayed remediation. 00:30 UTC: Identified the source of high attacker activity 00:00 UTC: Message remediation starts being delayed for IES and AMB What We're Doing: We are currently investigating the appropriate solution to solve the degradation. Estimated Resolution Time: TBD Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "2024-01-03 Remediation delayed for IES, AMB, and EPR"
Last updateTLDR: IES, AMB, and EPR experienced delayed remediation by several minutes during a ~1h incident period starting at 15:40 UTC on 1/3/24. What Happened The live scoring system system experienced instability between 15:40 and 16:05 UTC. This led to delayed remediation for attack and graymail messages from 15:40 to 16:45 UTC by several minutes. All systems are stable now and all messages are processing at normal processing times as of 16:45 UTC. All messages delayed during the period of instability have been remediated as of 16:45 UTC. No customer action is needed. Root Cause Excessive load on a database outside of the intended live scoring flow led to live message remediation delay. We are investigating the root of the excessive load and will ensure we address the scale aspect of the systems concerned. Follow Ups Root cause investigation for excessive database load.
Report: "Abnormal Portal Degradation"
Last updateOn Jan 17, 2024, at 16:28 UTC, we experienced a degradation of the Abnormal Portal. Customers may have experienced intermittent access issues when trying to access the Portal. There was no impact on email remediation during this incident. We apologize for any inconvenience that this outage created for you. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
On 1/17/2024 at 16:30 UTC, we are experiencing a degradation of the Abnormal Portal. Customers may experience intermittent access issues when trying to access the Portal. There is no impact on email remediation during this incident. We apologize for any inconvenience that this outage created for you. We are working to resolve this issue as quickly as possible. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Abnormal Portal Degradation - Dashboard Data is Stale"
Last updateA fix for the dashboard data has been rolled out. All customer dashboards should reflect the correct data as of 1:00 pm EST on Jan 29, 2024. Please, don't hesitate to get in touch with Abnormal Support (support@abnormalsecurity.com) for any additional details.
We have identified and verified a fix and will be deploying this fix by approximately 2 pm EST. Once the fix is deployed, dashboards will show correct data for all time ranges A fix to ensure correct data is being returned from our SOAR APIs and data in Sentinel and Cortex should be correct at this time. Don't hesitate to get in touch with Abnormal Support (support@abnormalsecurity.com) for any additional details.
We are currently investigating an issue with the Abnormal Security Dashboard. The data presented is not updating as expected. This only affects the data displayed on the Dashboard and does NOT affect message detection or remediation. All other services are working properly. We are actively working to identify and resolve this issue and will provide an update in the next hour. Don't hesitate to get in touch with Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Remediation delayed for Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond"
Last updateThis incident has been fully resolved. On March 5, 2024, starting at 16:00 UTC (8 am PT), customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Search & Respond may also be impacted as well. The root cause is a Google Workspace incident outage for Gmail. https://www.google.com/appsstatus/dashboard/incidents/shD5VvSGTxETw1YLbsCt - Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. - Severity Level: Major - Incident Start Time: March 5, 2024, starting at 16:00 UTC (8 am PT) - What We're Doing: Live processing is recovered for all customers from 17:00 UTC. - Recovery - 17:00 UTC: Live processing is recovered for all customers going forward from this time. - 18:30 UTC: Messages affected during the incident timeline (16:00-17:00 UTC) have been processed. - 20:50 UTC: Processing has recovered for POV customers. - What Happened: Google Workspace experienced an incident outage for Gmail starting at 15:25 UTC. Our processing clusters currently process M365+GWorkspace traffic together; as a result, the latency on Google Workspace is causing M365 processing to also be affected. Questions? For any immediate concerns, please contact us at support@abnormalsecurity.com.
On March 5, 2024, starting at 16:00 UTC (8 am PT), customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Search & Respond may also be impacted as well. The root cause is a Google Workspace incident outage for Gmail. https://www.google.com/appsstatus/dashboard/incidents/shD5VvSGTxETw1YLbsCt - Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. - Severity Level: Major - Incident Start Time: March 5, 2024, starting at 16:00 UTC (8 am PT) - What We're Doing: Live processing is recovered for all customers from 17:00 UTC. - Recovery - 17:00 UTC: Live processing is recovered for all customers going forward from this time. - 18:30 UTC: Messages affected during the incident timeline (16:00-17:00 UTC) have been processed. - (In progress) Processing is being recovered for POV customers with an ETA of recovery of 20:00 UTC. - What Happened: Google Workspace experienced an incident outage for Gmail starting at 15:25 UTC. Our processing clusters currently process M365+GWorkspace traffic together; as a result, the latency on Google Workspace is causing M365 processing to also be affected.
On March 5, 2024, starting at 16:00 UTC (8 am PT), customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Search & Respond may also be impacted as well. - Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. - Severity Level: Major - Incident Start Time: March 5, 2024, starting at 16:00 UTC (8 am PT) - What We're Doing: Live processing is recovered for all customers from 17:00 UTC. - Recovery: Messages affected during the incident timeline (16:00-17:00 UTC) are currently being processed as part of recovery steps, with an ETA for recovery at 19:00 UTC, and no customer action is required. - Estimated Resolution Time: March 5, 2024, 19:00 UTC - Next Update: 2 hours
On March 5, 2024, starting at 16:00 UTC (8 am PT), customers may have started experiencing a delay in message remediation for Inbound Email Protection, Graymail remediation, and Abuse Mailbox processing. Search & Respond may also be impacted as well. - Affected Services: Inbound Email Protection, Graymail, Abuse Mailbox, and Search & Respond. - Severity Level: Major - Incident Start Time: March 5, 2024, starting at 16:00 UTC (8 am PT) - What We're Doing: We are currently investigating the issue, to determine the Root Cause and identify a fix. - Estimated Resolution Time: TBD - Next Update: 1 Hour Questions? For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Abuse Mailbox Degradation (EU Customers Only)"
Last updateOn March 29, at 00:00 UTC, Abnormal Security stopped processing Abuse Mailbox reported messages for a subset of EU customers. AMB processing is fully recovered as of 4/2/24 19:00 UTC. Abuse mailbox reports from during this incident period will be automatically reprocessed, and no customer action is required. Microsoft has a planned deprecation for Outlook REST v2.0 endpoints by 03/31/2024 and requires all clients to migrate to the Graph API by that time frame. During the mandatory migration of our mail notification processing to Graph API, the team identified that the Graph API does not return a necessary value in its responses compared to the Outlook REST API. This caused Abuse Mailbox processing to come to a halt for a subset of EU customers between 3/29/24 00:00 UTC and 4/2/24 19:00 UTC Affected Services: Abuse Mailbox Severity Level: Major Incident Start Time: March 29, at 00:00 UTC Recovery: We deployed a fix at 4/2/24 19:00 UTC and confirmed that abuse mailbox processing is recovered for all affected customers What We're Doing: Abuse mailbox reports from during this incident period will be automatically reprocessed, and no customer action is required. Resolution Time: April 2, 2024 19:00 UTC Next Update: No further updates. Questions? For any immediate concerns, don't hesitate to contact us at support@abnormalsecurity.com.
On March 29, at 00:00 UTC, Abnormal Security stopped processing Abuse Mailbox reported messages for a subset of EU customers. Those impacted have been notified. The issue is related to the mandatory Microsoft Graph API migration, where the Graph API did not return a necessary value in its responses, which caused Abuse Mailbox processing to come to a halt. Affected Services: Abuse Mailbox Severity Level: Major Incident Start Time: March 29, at 00:00 UTC What We're Doing: We're testing a fix and expect to deploy it in the next hour. Workaround: We've been running manual recovery since 15:00 UTC on 04/02/24. Recovery: Once the fix is rolled out, we will be reprocessing all affected Abuse Mailbox messages from the incident period Estimated Resolution Time: TBD Next Update: 1 Hour Questions? For any immediate concerns, don't hesitate to contact us at support@abnormalsecurity.com.
Report: "Degradation of Inbound Email Protection"
Last updateAs of 11:26 PM UTC, the remediation of impacted Inbound Email Security messages from the incident period was completed. Overview: On April 3, at 4:51 PM UTC, customers started experiencing delayed message remediation for approximately 1% of Inbound Email Protection messages. The issue has been identified and corrected. We are currently in the process of remediating impacted messages. Affected Services: Inbound Email Security Severity Level: Major Incident Start Time: April 3, at 4:51 PM UTC What We're Doing: The issue has been identified and corrected. Recovery: As of 11:22 PM UTC, the remediation of impacted Inbound Email Security messages from the incident period was completed. Resolution Time: April 3, at 9:05 PM UTC
The remediation of impacted Inbound Email Security messages from the incident period is still ongoing. Next Update: 1 Hour Questions? For any immediate concerns, don't hesitate to contact us at support@abnormalsecurity.com.
On April 3, at 4:51 PM UTC, customers started experiencing delayed message remediation for approximately 1% of Inbound Email Protection messages. The issue has been identified and corrected. We are currently in the process of remediating impacted messages. Affected Services: Inbound Email Security Severity Level: Major Incident Start Time: April 3, at 4:51 PM UTC What We're Doing: The issue has been identified and corrected. Recovery: We will be remediating impacted Inbound Email Security messages from the incident period. Resolution Time: April 3, at 9:05 PM UTC Next Update: 1 Hour Questions? For any immediate concerns, don't hesitate to contact us at support@abnormalsecurity.com.
Report: "Email Account Takeover False Positive Case Creation"
Last updateOn April 5, at 11:30 AM UTC, customers with Account Takeover Protection may have seen 1-2 High Confidence False Positives due to an errant detector which over-weighted the Microsoft Risk Event. The cases created could have remediated the impacted user when Auto-Remediation for High Cases is enabled. The issue has been identified and corrected. Affected Services: Email Account Takeover Severity Level: Major Incident Start Time: April 5, at 11:30 AM UTC What We're Doing: The issue has been identified and corrected. Recovery: For customers who received more than 3 email notifications for these cases, those cases have been removed from the Portal. Customer's who only received 1 notification the cases remain visible and can be resolved as not an attack. Resolution Time: April 5, 9:48 PM UTC Next Update: Resolved Questions? For any immediate concerns, don't hesitate to contact us at support@abnormalsecurity.com.
Report: "Remediation delayed for Inbound Email Protection, Graymail, and Abuse Mailbox."
Last updateStarting at 9:20 a.m. PT, all messages affected by the incident were processed, and the incident is now fully resolved. - Affected Services: Inbound Email Protection, Graymail, and Abuse Mailbox. - Severity Level: Major - Incident Start Time: May 1, 2024, starting at 13:00 UTC - What We're Doing: The Root Cause has been identified and resolved. Messages affected during the incident are being processed. - Estimated Resolution Time: May 1, 2024, starting at 9:20 am PT - Next Update: The incident is resolved Questions? If you have any immediate concerns, please get in touch with us at support@abnormalsecurity.com.
Starting at 9 am PT, Inbound Email Protection, Graymail, and Abuse Mailbox services were restored and are now running as usual. Messages affected during the incident are being processed. - Affected Services: Inbound Email Protection, Graymail, and Abuse Mailbox. - Severity Level: Major - Incident Start Time: May 1, 2024, starting at 13:00 UTC - What We're Doing: The Root Cause has been identified and resolved. Messages affected during the incident are being processed. - Estimated Resolution Time: May 1, 2024, starting at 9 am PT - Next Update: 30 minutes Questions? If you have any immediate concerns, please get in touch with us at support@abnormalsecurity.com.
On May 1, 2024, starting at 13:00 UTC, Abnormal experienced a delay in remediation for Inbound Email Protection, Graymail, and Abuse Mailbox. Mailboxes affected during this period will have messages automatically processed, and no customer action is required. - Affected Services: Inbound Email Protection, Graymail, and Abuse Mailbox. - Severity Level: Major - Incident Start Time: May 1, 2024, starting at 13:00 UTC - What We're Doing: The Root Cause has been identified, and we are working to resolve it as soon as possible. - Estimated Resolution Time: TBD - Next Update: 1 Hour Questions? For any immediate concerns, please get in touch with us at support@abnormalsecurity.com.
Report: "Email Details Unavailable in Threat Log"
Last updateThe issue has been resolved as of May 2024, 22:23 UTC, and all Threat Log message details are now visible. There was no impact on email detection and remediation during this incident.
Starting at approximately 19:00 UTC on May 14, 2024, our database, which powers message details in the Portal Threat Log Details page, began experiencing an unexpected increase in load, resulting in email content unavailable being displayed There is no impact on email detection and remediation. Messages older than approximately May 14, 2024, 19:00 UTC should be visible. However, newer messages after May 14, 2024, 19:00 UTC will not be visible until the data staleness issues have been resolved. Affected Services: Abnormal Portal / Threat Log Severity Level: Major Incident Start Time: May 14, 2024, 19:00 UTC What We're Doing: Our Engineering team has identified the root cause and is implementing a solution. We expect the database performance to improve in the next hour. Next Update: May 14, 2024, 22:30 UTC Questions?: For any immediate concerns, please get in touch with us at support@abnormalsecurity.com.
Report: "Abnormals Threat Log and SOAR API Latency"
Last updateThis incident has been resolved.
The issue causing increased load on our database, which impacted the Abnormal Portal Threat Log and SOAR API endpoints, has been resolved. Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints Severity Level: Major Incident Start Time: May 31, 2024, 10:00 UTC Resolution Timeline: May 31, 2024, 21:00 UTC Current Status: The affected services are now fully operational. We have verified that the database load has normalized, and all systems are functioning as expected. Next Steps: Our team will continue to monitor the system closely to ensure stability. We are also conducting a post-incident review to prevent similar issues in the future. Questions?: For any further concerns, please contact us at support@abnormalsecurity.com. Thank you for your patience and understanding throughout this incident.
Our team continues to work on resolving the increased load on our Abnormal Portal / Threat Log and SOAR API endpoints. While we have made progress, the issue remains unresolved. Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints Severity Level: Major Incident Start Time: May 31, 2024, 10:00 UTC Current Status: Our Engineering team is actively implementing solutions to mitigate the increased load. We are committed to resolving this issue as quickly as possible and will keep you updated on our progress. Next Update: In one hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Incident Update: Our team continues to work on resolving the increased load on our database caused by the recent Azure Sentinel integration change. Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints Severity Level: Major Incident Start Time: May 31, 2024, 10:00 UTC Current Status: Our Engineering team is actively working on implementing a solution. Unfortunately, the issue is still not fully resolved. We are making progress and will provide further updates as soon as possible. Next Update: In one hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
On May 31, 2024, at approximately 10:00 UTC, a change was released to the Azure Sentinel integration for all customers using the Abnormal Security. This change altered how Sentinel queries our API by implementing a different date range filter, resulting in queries over a larger time range. Consequently, this led to an increased load on the database powering all threat information in both the SOAR API and the Portal, causing higher latency in the Threat Log and errors in Azure Sentinel calls due to a broken date range filter. There is no impact on Email Detection and Remediation. Affected Services: Abnormal Portal / Threat Log, SOAR API endpoints Severity Level: Major Incident Start Time: May 31, 2024, 10:00 UTC What We're Doing: Our Engineering team has identified the root cause and is implementing a solution. We expect the database performance to improve in the next hour. Next Update: May 31, 2024, 18:45 UTC Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Remediation delay for Inbound Email Protection, Abuse Mailbox and Graymail"
Last updateIncident Resolved: On June 18, 2024, at approximately 14:45 UTC, Abnormal experienced a delay in remediation for Inbound Email Protection, Abuse Mailbox, and Graymail. The root cause was a system delay in handling large attacker activity. Recovery is now completed, and all systems are fully operational. Affected Services: Inbound Email Protection, Abuse Mailbox, and Graymail Severity Level: Major Incident Start Time: June 18, 2024, 14:45 UTC Resolution Timeline: 14:45 UTC: Live remediation starts to be delayed for Email Protection, Abuse Mailbox, and Graymail. 15:48 UTC: Recovery is in progress, and the live remediation delay starting to normalize. 16:15 UTC: Live remediation is fully recovered. Next Steps: Our team will monitor the system closely to ensure stability. We are also conducting a post-incident review to prevent similar issues in the future. Questions?: For any further concerns, please contact us at support@abnormalsecurity.com.
What Happened: On June 18, 2024, at approximately 14:45 UTC, Abnormal experienced a delay in remediation for Inbound Email Protection, Abuse Mailbox, and Graymail. Affected Services: Inbound Email Protection, Abuse Mailbox, and Graymail Severity Level: Major Incident Start Time: June 18, 2024, 14:45 UTC What We're Doing: Our Engineering team is actively working to address the increased latency. Next Update: In 1 hour or less. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Abnormal Portal Degradation"
Last updateAfter extensive investigation, Abnormal Security was unable to identify any underlying issues, such as heavy load, backend failures, or third-party outages, that caused the degradation in Abnormal Portal responsiveness. We believe the slower-than-expected load times experienced during this incident were isolated to specific clients or browsers, and the issue resolved itself without intervention from Abnormal Security. We will continue to monitor the situation and provide further updates if necessary. For any further concerns, please contact us at [support@abnormalsecurity.com](mailto:support@abnormalsecurity.com). Thank you for your patience and understanding.
Incident Update: We are pleased to inform you that the degradation of the Abnormal Portal has been resolved. The issue was traced back to the client's browser, which was causing the problem. There was no impact on email remediation during this incident. Affected Services: Abnormal Portal | Threat Log | Search & Response Severity Level: Minor Incident Start Time: June 25, 2024, 10:00 UTC Resolution Timeline: June 25, 2024, 11:39 UTC Next Steps: Our team will monitor the system closely to ensure stability. If you continue to experience issues, please contact your CS partner for assistance. Questions?: For any further concerns, please contact us at support@abnormalsecurity.com.
Incident Update: Our team continues to work on resolving the degradation of the Abnormal Portal. Some customers may experience intermittent access; for others, the portal is slow on the Threat Log and Search and Response. There is no impact on email remediation during this incident. Affected Services: Abnormal Portal / Threat Log / Search & Response Severity Level: Minor Incident Start Time: June 25, 2024, 10:00 UTC Next Update: 1 hour Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
On 6/25/2024 at 10:00 UTC, we are experiencing a degradation of the Abnormal Portal. Some customers may experience intermittent access issues when trying to access the portal; for others, the portal is slow on the threat log and search and response. There is no impact on email remediation during this incident. We apologize for any inconvenience that this outage created for you. We are working to resolve this issue as quickly as possible. Please contact Abnormal Support (support@abnormalsecurity.com) for any additional details.
Report: "Remediation Delay - Inbound Email Security and Email Productivity"
Last updateIncident Summary: On July 10, 2024, at approximately 14:00 UTC, Abnormal experienced a delay in remediation for Inbound Email Protection and Email Productivity. What Happened: The root cause was a system delay in handling large attacker activity. Recovery is now completed, and all systems are fully operational. Affected Services: Inbound Email Security, Email Productivity Severity Level: Major Incident Start Time: July 10, 2024, 14:00 UTC Incident Resolved Time: July 10, 2024, 15:30 UTC Timeline: Identified: 14:00 UTC - Live remediation starts to be delayed for Inbound Email Protection and Email Productivity. Recovery: 15:10 UTC - Recovery is in progress, and the live remediation delay starting to normalize. Resolved: 15:30 UTC - Live remediation is fully recovered. What We're Doing: Our team will monitor the system closely to ensure stability. We are also conducting a post-incident review to prevent similar issues in the future. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com
Report: "Remediation delay for Inbound Email Protection, Abuse Mailbox and Email Productivity"
Last updateIncident Resolved: As of 17:00 UTC, the remediation delay issue for Inbound Email Protection, Abuse Mailbox, and Email Productivity was resolved. Affected Services: Inbound Email Protection, Abuse Mailbox, and Graymail Severity Level: Major Incident Start Time: July 16, 2024, 15:00 UTC Incident Resolution Time: July 16, 2024, 17:00 UTC What We're Doing: our team will monitor the system closely to ensure stability. We are also conducting a post-incident review to prevent similar issues in the future. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com
Incident Update: Remediation delays have been reduced to 1 minute or less for Inbound Email Protection, Abuse Mailbox, and Email Productivity. Affected Services: Inbound Email Protection, Abuse Mailbox, and Productivity Severity Level: Major Incident Start Time: July 16, 2024, 15:00 UTC What We're Doing: The team continues to resolve the issue, and an update on the resolution status will be shared at the next planned update time. Next Update: In 1 hour or less. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
What Happened: On July 16, 2024, at approximately 15:00 UTC, Abnormal experienced a delay in remediation for Inbound Email Protection, Abuse Mailbox, and Graymail. Affected Services: Inbound Email Protection, Abuse Mailbox, and Graymail Severity Level: Major Incident Start Time: July 16, 2024, 15:00 UTC What We're Doing: The team has identified a fix and is actively resolving the issue. An update on the resolution status will be shared at the next planned update time. Next Update: In 1 hour or less. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Abnormal Security Unaffected by CrowdStrike and Microsoft Service Disruptions - July 19, 2024"
Last updateAbnormal Security is aware of the recent outage affecting CrowdStrike and Microsoft services and is closely monitoring it. At this time all systems are healthy and fully operational, including Inbound Email Security and our integrations with CrowdStrike. We will keep you informed if there are any changes. We value your trust and satisfaction and are committed to maintaining the highest standards of service. If you have any concerns or require further information, please don't hesitate to contact Support.
Report: "Inbound Email Protection, Email Productivity, and AI Security Mailbox"
Last updateOn July 23, 2024, Abnormal Security experienced a brief delay in remediation for Inbound Email Protection, Email Productivity, and AI Security Mailbox. These delays occurred during the periods of 14:05 UTC to 14:35 UTC and 15:05 UTC to 15:35 UTC. We want to assure you that all affected mailboxes during this time had their messages automatically processed, and no action is required from our customers. Affected Services: - Email Protection - Email Productivity - Abuse Mailbox Severity Level: Major Incident Timelines: Incident Period 1: July 23, 2024, from 14:05 UTC to 14:35 UTC Incident Period 2: July 23, 2024, from 15:05 UTC to 15:35 UTC Actions Taken: We have implemented a fix to prevent further delays and will conduct a thorough review during our postmortem investigation. Rest assured, all affected mailboxes had their messages automatically processed. Estimated Resolution Time: Resolved; all affected mailboxes had their messages automatically processed. For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Email Productivity Remediation Delay (Graymail)"
Last updateThe incident has been resolved as of 2024-07-31 02:45 UTC, Email Productivity, Account Takeover and SIEM/SOAR are now fully functional. Affected Services: Email Productivity Account Takeover SIEM/SOAR Integrations Severity Level: Major Incident Start Time: July 30, 2024, 22:10 UTC Incident Resolution Time: July 31, 2024, 02:45 UTC What We're Doing: Our team will monitor the system closely to ensure stability. We are also conducting a post-incident review to prevent similar issues in the future. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com
Starting at 22:10 UTC on July 30, 2024, Abnormal Security was made aware of a possible delay in remediation for Email Productivity (EPR / Graymail) due to an ongoing operational issue with our hosting provider AWS. Abnormal is continuing to assess and resolve any impact on additional services. Inbound Email Security is unaffected by the ongoing AWS issue. Update: This AWS outage is also impacting: Account Takeover: Some customers may experience a delay in lateral phishing detection. SIEM/SOAR Integrations: Some customers may experience increased time-out requests Affected Services: Email Productivity Account Takeover SIEM/SOAR Integrations Severity Level: Major Incident Start Time: July 30, 2024, 22:10 UTC What We're Doing: The team continues to resolve the issue, and an update on the resolution status will be shared at the next planned update time. Next Update: In 1 hour or less. Questions? For any immediate concerns, please contact us at support@abnormalsecurity.com.
Starting at 22:10 UTC on July 30, 2024, Abnormal Security was made aware of a possible delay in remediation for Email Productivity (EPR / Graymail) due to an ongoing operational issue with our hosting provider AWS. Abnormal is continuing to assess and resolve any impact on additional services. Inbound Email Security is unaffected by the ongoing AWS issue. Affected Services: Email Productivity Severity Level: Major Incident Start Time: July 30, 2024, 22:10 UTC What We're Doing: The team continues to resolve the issue, and an update on the resolution status will be shared at the next planned update time. Next Update: In 1 hour or less. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Remediation Delay - Inbound Email Security"
Last updateWhat Happened: On September 10, 2024, at 16:00 UTC, Abnormal and its vendor partner started experiencing unknown issues, leading to elevated remediation times for Inbound Email Protection. This incident affected Microsoft and Google customers, causing delays in email remediation. Affected Services: Inbound Email Protection Severity Level: Major Incident Start Time: September 10, 2024, at 16:00 UTC Resolution: The incident was fully resolved as of September 10, 2024, at 18:45 UTC (11:45 AM PST). Remediation times have returned to normal levels. Questions? For any immediate concerns, please contact us at support@abnormalsecurity.com.
What Happened: On 9/10 at 16:00 UTC, Abnormal and its vendor partner started experiencing unknown issues, leading to elevated remediation times for Inbound Email Protection. This incident affects Microsoft and Google customers, who can expect email remediation delays. Affected Services: Inbound Email Protection Severity Level: Major Incident Start Time: Sep 10, 2024, at 16:00 UTC What We're Doing: Our Engineering team is actively working to address the increased latency. Next Update: In 1 hour or less. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Inbound Email Security - Google Customers Only"
Last updateWe have successfully resumed processing all GSuite email traffic in both EU and US environments. This issue was isolated to GSuite services and did not affect email delivery for our customers. It stemmed from a broader Google disruption, impacting GSuite integrations for their customers globally. Throughout the incident, Abnormal Security services remained fully operational, ensuring ongoing protection for our customers. Update Time: 2024-09-13 5:15 PM UTC Next Update: No further updates. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
What Happened: On 9/13 at 3:08 PM UTC, Gmail's API unexpectedly stopped connecting, causing problems with Inbound Email Security for Google Workspace customers only. Affected Services: Inbound Email Security Severity Level: Major Incident Start Time: Sep 13, 2024, at 3:08 PM UTC What We're Doing: We have contacted Google Support for further information. Next Update: In 1 hour or less. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "EMEA Data Center Degraded Performance"
Last updateWhat Happened: On September 16, 2024, at around 5:00 pm UTC, Abnormal experienced a degradation in performance in the EU (Azure) Data Centre. During the affected timeframe, a partial customer base would have seen email detection and remediation issues, resulting in a delay in scanning. Affected Services: Inbound Email Security Severity Level: Major Geography: EMEA-based customers (partial customer base - Azure Data Centre) Incident Start Time: September 16, 2024, 5:00 PM UTC Incident Resolution Time: September 17, 2024, 1:00 PM UTC What We're Doing: We completed reprocessing all impacted emails. A full Root Cause Analysis will be completed and available upon request by contacting Support. Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
What Happened: On September 16, 2024, at around 5:00 pm UTC, Abnormal experienced a degradation in performance in the EU (Azure) Data Centre. During the affected timeframe, partial customer base would have seen message detection and remediation issues, resulting in a delay in scanning. Affected Services: Inbound Email Security Severity Level: Major Geography: EMEA-based customers (partial customer base - Azure Data Centre) Incident Start Time: September 16, 2024, 5:00 PM UTC Incident Resolution Time: September 17, 2024, 1:00 PM UTC What We're Doing: Our engineering team has resolved the issue, and emails are now being remediated as usual. We have started reprocessing all impacted emails. We will provide an update once reprocessing is completed. A full Root Cause Analysis will be completed and available upon request by contacting Support. Next Update: We will provide an update once we can share an ETA on processing completion time Questions?: For any immediate concerns, please contact us at support@abnormalsecurity.com.
Report: "Inbound Email Security - Attack remediation delays to processing time"
Last updateBetween 19:10 and 20:57 UTC on October 7th, Abnormal experienced delayed processing of attack remediation which impacted all US data center customers. Graymail was not impacted as part of this incident. The emails not analyzed during the incident time have been fully processed and services have been recovered as of 20:57 UTC. If you experience any further issues pertaining to this incident, please contact us at support@abnormalsecurity.com.
Functionality to remediation processing has recovered as of 20:57 UTC. Engineers identified a large Database query, which was terminated. Processing is now back within SLA. There is a backlog which is expected to take approximately 2 hours to complete. The next update will be provided when the backlog has finished processing.
We are continuing to investigate this issue.
Starting at 19:10 UTC on October 7, 2024, Abnormal's attack remediation began to experience delays to processing time for both Microsoft 365 and Google Workspace customers. Graymail remediation is not affected. This is impacting all customers in the US data center. The next update will be posted in 1 hour or when further information is available.
Report: "Email remediation issues for Inbound Email Security"
Last updateBetween 16:34 UTC and 21:51 UTC on November 5th, 2024, email Remediation for Inbound Email Security customers in North America was impacted. Email remediation has recovered and is now processing in real time. If you experience any further issues pertaining to this incident, please contact us at support@abnormalsecurity.com.