Historical record of incidents for Beyond Identity
Report: "Partial Outage"
Last updateThis incident has been resolved. Customers will receive an RCA via email.
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented
Report: "Partial Outage"
Last updateThis incident has been resolved. Customers will receive an RCA via email.
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented
Report: "Android Product Authenticator having trouble registering new passkeys on specific tenant configurations"
Last updatePlease update your Android Beyond Identity Apps to the Latest version, and it will resolve the issue.
We are continuing to investigate this issue.
We are investigating reports of the newest version of the Android Product authenticator having trouble registering new passkeys with specific tenant configurations.
Report: "Degraded performance"
Last updateSome customers may have experienced degraded authentication service during a scheduled release at 4:30 p.m. CST. This has been fully resolved.
Report: "Events not rendering correctly in console"
Last updateThis incident has been resolved.
Incident Resolved
Report: "Partial API outage. Increased latency and failed requests on a subset of tenants."
Last updateThis incident has been resolved.
This issue has been identified, and a fix is being implemented.
Report: "Secure Access Platform - Role Based Access Control Scopes not functioning properly"
Last updateThis incident has been resolved.
Secure Access Platform - Role-Based Access Control Scopes not functioning correctly.
Report: "Secure Access and Secure Workforce Platform - Admin Console Events Latency Identified"
Last updateThis incident has been resolved.
Issue with event latency in the Admin Console, impacting SIEM integrations as well
Report: "Secure Access Minor Outage"
Last updateCustomers on the Secure Access Platform experienced increased latency and failed authentications from 2025-03-10 16:10 UTC to 2025-03-10 16:24 related to a release of cloud services.
Report: "Events not loading correctly in administrator console"
Last updateWe are currently investigating this issue.
Report: "Layered authentication login flows experiencing issues"
Last updateThis incident has been resolved.
Customers that are in early access of our layered authentication capability, which allows for second device biometric step up when primary device biometrics are unavailable, may experience authentication failures.
Report: "Issue with CrowdStrike API Integration"
Last updateThis incident has been resolved.
The issue affecting our integration with the CrowdStrike API has been resolved. The service is now fully operational. We are actively monitoring the system to ensure its stability and performance. Thank you for your patience and understanding.
We have identified an issue impacting our integration with the CrowdStrike API. This may affect the functionality of related services. Our team has confirmed that the issue originates from the CrowdStrike side, and we are actively monitoring the situation. We will provide updates here as more information becomes available. Thank you for your understanding.
Report: "EU Admin console won't load content"
Last updateThe issue with the EU Admin Console has been resolved. All systems are now operational.
All services are functioning as expected. Our team is continuing to monitor the situation and investigate the underlying cause to prevent potential recurrence.
We are aware that the issue with the EU Admin Console has recurred. The Admin Console in the EU region is again not loading content. Authentication and other services remain unaffected and continue to operate normally. Our team is actively investigating the issue to identify the cause and implement a permanent solution. We will provide further updates as soon as we have more information.
The issue with the EU Admin Console has been resolved. A fix has been successfully implemented. All services, including the Admin Console in the EU region, are back up and functioning as expected. We are actively monitoring our services to ensure they remain fully operational.
The EU Admin Console is currently experiencing an issue and is not loading any content. This issue is limited to the Admin Console in the EU region. Authentication services and other functionalities remain unaffected and are operating normally. We have identified the problem and working on the implementation of the fix. We will provide updates as soon as more information becomes available. Thank you for your patience.
Report: "Apple Updates - iOS 18 and macOS"
Last updateVersion 2.100.3 Endpoint Release Resolved: Authenticator unable to communicate with Safari Version 18 and above.* Authenticator unable to communicate on iOS 18.* *iCloud private relay must be disabled for Beyond Identity to function correctly.
We are continuing to work on a fix for this issue.
Apple has made changes to their Safari browser and embedded web viewer on iOS 18 and macOS 15 that block the primary mechanism by which the Beyond Identity authenticator communicates with Safari on those platforms. Authentication will still be successful, but users will see additional prompts when logging in. On iOS 18 we recommend enabling our new “Safari Beyond Identity Extension” for the best end user experience. Guide Here: https://support.beyondidentity.com/hc/en-us/articles/22996465952151-Enable-the-Safari-extension-on-iOS-devices On macOS 15 Sequoia our team is in the final stages of releasing authenticator version 2.100.3, which addresses these changes. We will update when this is available.
Report: "Microsoft Azure Experiencing Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Beyond Identity is monitoring service degradation with the following Microsoft services: - Microsoft 365 admin center - Intune - Entra - Power Platform During this period, certain integrations may experience failures. - Intune integrations - Windows Desktop Login Enrollments We will continue to update this page.
Report: "Data export and event processing experiencing a delay."
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Degraded Service"
Last updateProblem Between the hours of 2:30PM EST to 6:00PM EST on Friday December 15th 2023 a subset of users were unable to connect to app.byndid.com Resolution This issue was reported resolved at approximately 6:30PM EST
Report: "Users located in the Pune region of India experienced degradation of service"
Last update**Problem:** Between the hours of 7AM to 9AM CST on December 4th, 2023 several users located in the Pune region of India experienced degradation of service. Several users reported slow authentication speeds and several others reported incomplete authentications due to timeouts. **Cause:** A backbone local internet service provider reported service outages during this period. Beyond Identity believes this to have caused a local and isolated degradation of service which resulted in several authentication failures. Source: [https://istheservicedown.in/problems/vodafone-idea-india](https://istheservicedown.in/problems/vodafone-idea-india) **Resolution:** The issue was resolved with no intervention after a period of approximately two hours.
Users located in the Pune region of India experienced degradation of service
Report: "Windows Platform Authenticator unable to login to Microsoft Office products on Windows"
Last update# INC-232 Unable to authenticate to Microsoft apps - postmortem report # Executive Summary ## Summary Beyond Identity released endpoint version 2.88.0 on September 21st, 2023, at 11:18 PM. The Windows Platform Authenticator contained a code change that didn’t pass the correct redirection to the Microsoft webview window, breaking authentication to native Microsoft Office apps on Windows. A hotfix 2.88.1 was released on September 22nd, 2023, at 07:20 PM CDT for Windows platform to fix the issue. ## Customer Impact All customers using Windows who upgraded to the 2.88.0 version were impacted. Updating to the fixed version will fix the issue immediately. # Leadup Beyond Identity released 2.88.0 endpoints on September 21st, 2023, at 11:18 PM CDT. # Fault On Windows, authenticating to native Microsoft Office Applications was hanging in the webview window after successful authentication. # Detection A customer support ticket was opened on September 22nd, 2023, at 11:23 AM CDT. # Root causes * An unintentional change in the Microsoft webview window behavior on the Windows platform. * The code review didn’t catch the change as the impacted line was not changed directly. * Microsoft caches the login, and the QA team didn’t catch the issue. * A faulty code was released. # Mitigation and resolution * A bug was identified and fixed. * Hotfix 2.88.1 for Windows was released. # Lessons learned * \[BIT-1643\] We need to add a unit test that tests whether the content passed to this specific webview is actually a URL. * \[BIT-1644\] The end-to-end testing needs to include a case that verifies that we do a valid redirect back to the application at the end. * \[BIT-1645\] The QA team will add a pre-requisite for automated UI testing to clear potentially cached Microsoft sign-in.
A hotfix for Windows platforms has been released. The 2.88.1 version will fix the issue. Release notes can be found at https://support.beyondidentity.com/hc/en-us/articles/17749590174871-Version-2-88-1-Endpoint-Release-Notes. The postmortem will be published once the engineering team has performed it.
The issue has been identified, and a hotfix will be released. We are tracking this issue with bug tracker BIT-1638.
We have identified an issue with the release 2.88.0 on the Windows platform when signing in to Microsoft Office applications. The web apps work just fine, but the locally installed Office apps have issues signing in with a new user. Our engineering team is investigating.
Report: "Windows Release 2.84.1 Unable to Launch"
Last update# Executive Summary ## Summary On August 7th, 2023, at 4:06 PM CDT, Beyond Identity released a hotfix to address the Windows Desktop Login issue. The build pipeline encountered an issue, and we released a broken build. Once upgraded to the broken build 2.84.1, the Windows Platform Authenticator didn’t launch due to a missing reference. This prevented users from using the product entirely. The initial after-release to production testing detected the issue, and we reverted the release at 5:14 PM CDT. A service bulletin was released on August 8th at 11:44 AM to address the issue. The impact continued to the users who had received the upgrade dialog but had not reacted to it. If they chose to upgrade even after we had reverted the release, it could still install the broken version. If the user chose to dismiss the dialog, they were not impacted. The broken binaries were removed from storage services on August 8th at 1:06 PM CDT preventing the installation of the broken version if the dialog was still open. Once the hotfix 2.84.2 was released on August 8th at 10:02 AM CDT, the fix was to upgrade to that version. Before the fixed version was released, the fix was to uninstall and re-install the previous version. The service bulletin document was updated accordingly. ## Impacted Customers This issue impacted all customers using Windows Platform Authenticator \(not Windows Desktop Login version\) without using version control. ## External Links * [Service Bulletin #230 - Windows Release 2.84.1 Issue – Beyond Identity](https://support.beyondidentity.com/hc/en-us/articles/16602800489367-Service-Bulletin-230-Windows-Release-2-84-1-Issue) # Incident Details ## Leadup Beyond Identity released a hotfix 2.84.1 to address the Windows Desktop Login enrollment issue, which also required a change in the Windows Platform Authenticator — both WDL and Windows PA releases were broken. ## Fault Windows Platform Authenticator did not launch at all. The users were unable to use the product at all. ## Detection The support team validated the WDL fix as a post-release check and detected that the Platform Authenticator didn’t launch after upgrading. ## Root causes * Build pipeline errored with dependencies. * Rust dependency generation error caused the build pipeline to pull the wrong library version. * The manifest and downloads page was updated automatically after the binaries were released as a part of the hotfix release flow. * When we identified the issue, reverting the release did not include removing the broken binaries. * A broken build got released. ## Mitigation and resolution * The release was reverted from the release manifest. * The download page was reverted. * Hotfix 2.84.2 was released. * Broken binaries were removed from production. * A service bulletin was released. ## Lessons learned * The hotfix release process needs to have a stop-gap between publishing binaries and releasing the manifest and download page update. \([SRE-1721](https://beyondidentity.atlassian.net/browse/SRE-1721)\) * Reverting the release needs to prevent access to the reverted release-related binaries. \([SRE-1720](https://beyondidentity.atlassian.net/browse/SRE-1720)\) * Identified a new test case for the QA team to perform in the pre-release environment. \([BIT-1566](https://beyondidentity.atlassian.net/browse/BIT-1566)\) * The statuspage needs to be updated to have more detailed services listed. \([SUPO-820](https://beyondidentity.atlassian.net/browse/SUPO-820)\) ## Timeline 2023-08-07 4:06 PM CDT - Build pipeline completed the release and updated manifest and download page 2023-08-07 4:16 PM CDT - An issue with the build was identified 2023-08-07 4:20 PM CDT - The engineering team started investigating the issue 2023-08-07 4:58 PM CDT - The issue was reproduced, and the build was identified as defective 2023-08-07 5:05 PM CDT - The site reliability engineering team dispatched to revert the release 2023-08-07 5:14 PM CDT - Release manifest reverted 2023-08-07 5:32 PM CDT - Downloads page reverted 2023-08-07 7:27 PM CDT - Root cause identified and fix implemented 2023-08-07 9:47 PM CDT - The QA team started the release testing process 2023-08-08 10:02 AM CDT - Hotfix 2.84.2 was released 2023-08-08 11:44 AM CDT - Service bulletin released ## Metrics Time to Detection - 10 minutes Time to SME investigation - 4 minutes Time to Repair - 54 minutes Time to Recovery - 17h 56 minutes
The Windows Platform Authenticator release 2.84.1 was broken. After upgrading to that version, the authenticator was unable to launch.
Report: "Authentication issues"
Last update### Summary A third-party service provider's partial outage affected some customer networks accessing the cloud services. Our services were impacted from 12/5/2022, 14:00 CST, to 12/5/2022, 14:55 CST. The estimated partial outage lasted 55 minutes. The SRE team reacted promptly and identified that it was a third-party issue. The issue was resolved while preparing to fail over to another data center. ### Customer Impact During the partial outage, some customers were unable to authenticate. ### Mitigation and resolution While testing internally, some team members could authenticate, and some did not. Due to the intermittent failures, Beyond Identity investigated and assessed the impact to minimize the recovery impact on the user base. The SRE team started to prepare for failing over to another data center. By the time the preparations were complete, the service provider had restored the service.
The incident has been resolved. The postmortem will be provided in a timely manner.
The service has been recovered. The site reliability engineering team will continue to monitor.
We have identified a third-party service provider-related internet connection issue causing a partial outage. The engineering team is performing activities to recover.
We are continuing to investigate this issue.
We have identified issues with authentication. We are investigating the issue.
Report: "Windows update errors with Downloaded update hash mismatch"
Last updateThe issue with the Windows authenticator update has been fixed.
We have identified an issue with a Windows authenticator application update. The engineering team is working on the fix. The only impact is that the update will fail with error message "Downloaded update hash mismatch."
Report: "Incident with service provider"
Last updateThis incident has been resolved.
The third-party service provider has recovered, and we have verified that our services are functional again. Our engineering team will continue monitoring the situation.
We are experiencing a third-party service interruption. Their services are being recovered, and we are investigating the impact on our side. We have identified that the Beyond Identity services are working intermittently.
We are currently investigating this issue.
Report: "Location specific issues accessing services"
Last update**Executive Summary** The Beyond Identity team added additional detection rules to our WAF. These changes led to one customer being unable to authenticate or access [app.byndid.com](http://app.byndid.com/). Beyond Identity was alerted to the problem, and mitigation was promptly implemented. Only one customer was experiencing issues. **Root Cause** * The security team added additional WAF rules to block a collection of disclosed vulnerabilities. These changes identified traffic coming in from a single customer as malicious/anomalous. * The root cause was traced to a single WAF rule blocking a specific CVE.
We have verified that the issue is resolved.
The site reliability engineering team has made an update to the configuration that was causing an issue. We keep monitoring to verify that this is fixing the problem.
Our site reliability team is investigating the issue.
Report: "Third party service interruption"
Last updateThe DNS service provider experienced an outage and has published a detailed report [https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/](https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/) The incident started at 1:27 AM CDT on June 21st 2022. Beyond Identity's services were impacted for 53 minutes and experienced intermittent failures. The services recovered as the DNS service provider implemented a fix.
The DNS service provider had an outage that impacted our services.
Report: "Intermittent authentication failures"
Last update# Summary Intermittent authentication failures occurred between June 7th, 2022, at 1:06 pm \(CDT\) and 1:38 pm \(CDT\). # Cause A specific user action in a specific sequence caused the directory service to crash. The directory service recovered itself immediately. During the few seconds it took for the directory service to come back online, authentications failed. # Mitigation The site reliability engineering team instantly identified the root cause and prevented the sequence from being executed. The software engineering team implemented a permanent fix to the directory service to address the specific use case.
This incident has been resolved.
We are continuing to monitor for any further issues.
We identified an issue with the authentication service. The issue was fixed within minutes. The engineering team is monitoring the situation.
Report: "Intermittent DNS resolution issues"
Last updateA third-party DNS service provider had intermittent name resolution failures causing our cloud services routing to fail. The issue was resolved when the third party fixed their problems.
A third-party service provider experienced intermittent DNS resolution issues for certain specific endpoints.
Report: "Unplanned Configuration Change"
Last update### **Leadup** The SRE team executed a planned deployment on the evening of Tuesday, Oct 18. This event triggered a security policy that forced a key rotation. The key rotation process failed and went undetected. However, the services were left in a consistent state. ### **Fault** The initial planned key rotation failed and went undetected during the deployment. This set the stage for an unplanned key rotation triggered by the cloud provider over 24 hours later. This key rotation process that was triggered only completed a single step in the full key protection process defined by the security policy. This left the system in an inconsistent state. **Detection** An automated alert immediately notified the SRE team once the applications failed to access their persistent storage as they had old credentials. **Root causes** The failure of the key rotation security policy did not alert the SRE team that the policy had not been completed successfully. If this had happened, the issue could have been completely averted. The mechanism for alerting was misconfigured and, as such, did not trigger. Also, an issue with the cloud provider's automatic key rotation mechanism caused a portion of the key rotation security policy to trigger and leave the system in an inconsistent state. ### **Mitigation and resolution** The team logged into the service provider and restarted the database services, which allowed the key rotation to succeed. This restored all of the services. The key rotation security policy trigger has been reconfigured to be executed manually during release operations.
An unplanned configuration change was triggered by an unrelated event that caused an interprocess communication to fail.
Report: "Intermittent authentication falures"
Last updateAfter our 3rd party cloud infrastructure partner had recovered from their incident, Beyond Identity migrated its services back to their standard configuration. During this migration, there was an intermittent interruption in the authentication service. The interruption lasted 30 minutes.
Report: "Cloud Services Experiencing Delays"
Last updateThis incident has been resolved, and the authentication latency is back at normal latency.
We have mitigated the issue, and our services are working normally. We will continue to monitor the situation.
We're experiencing some delays in our cloud services caused by our vendor having issues with their infrastructure. The delays may cause longer than usual authentications or needing to retry authentication.
Report: "DNS-related partial outage"
Last updateThe DNS-records have been verified being propagated, and the incident is closed.
The DNS-records have been added to alternate servers fixing the issue. All systems are operational, and our site reliability engineering team continues to monitor the situation.
We have identified an issue in our DNS-records. The problem is already fixed; however, the propagation of DNS-records will take some time. A new update will be provided in a timely manner.
Report: "Authentication service failed to restart"
Last update**Leadup** While performing a routine deployment of a new version to production, an anomaly occurred, causing the authentication service to fail to start up. **Fault** A database migration failed, causing subsequent database migration tasks to not execute. **Detection** QA immediately ran a test after the deployment to log out and log back into production using Beyond Identity Authenticator, and they observed the issue. **Root cause** Failed database migration left the database in a state not expected by the cloud services. **Mitigation and resolution** The Site Reliability Team escalated to the cloud engineering team, who identified the root cause and prepared a manual fix to the content in the database. After implementing the data repair, the SRE team re-executed the routine deployment successfully, restoring all services back to their normal state.
While performing a routine deployment of a new version to the production, an anomaly occurred, causing the authentication service to fail to start up.
Report: "Intermittent issues using Admin Portal"
Last update## Leadup Automated monitoring detected a crash in the API microservice. ## Detection The engineering team started investigating and identified an issue with messages coming from another microservice. ## Root Cause A poorly formed message between two microservices caused the recipient service to crash. Automation restored the crashed service, but as it started re-processing the queue a new crash occurred. This caused intermittent issues accessing and operating Admin Portal. ## Mitigation and Resolution The engineering team isolated the poorly formed message type and added them to be ignored to prevent further crashing restoring the service to a fully operational state. The development team had already identified this potential issue and implemented fixes in the code which will allow the system to be resilient for poorly formed messages. As this is not possible to be caused via user interaction, the permanent fix will be deployed in the next release to prevent similar situations from occurring in the future.
The incident has been resolved. Preventive actions have been implemented. A permanent fix will be deployed in the next release.
The issue has been identified and the engineering team is working towards a stable fix. The Admin Portal is currently functioning normally.
We are currently investigating an issue with Admin Portal. There are some intermittent login issues.
Report: "Windows application 2.10.0 missing DLL"
Last update# Lead up Customer reported via user feedback that they experienced an error when updating the software. Support engaged to investigate with help from a developer. While investigating the issue we discovered an unrelated missing DLL issue. This was discovered by examination of the user’s application logs which displayed an exception while trying to check for updates. # Mitigation and Resolution 1. The the 2.10.0 release installer was replaced with 2.9.0 release installer to prevent further installations of impacted version. 2. A fix was created for the missing DLL and the fixed version was released as 2.10.1. # Preventive Actions QA team is going to add testing for upgrading to newer version as part of their process to ensure that automatic updates feature is not broken.
This incident has been resolved.
The Windows installer 2.10.0 has been reverted in the production. New installations are not impacted by the issue.
We are removing the 2.10.0 Windows installer from production.
We are continuing to investigate this issue.
We have created an installer package for Windows users that seems to have a missing DLL file. Our engineering team is investigating the issue.