Is HYPR Services Down Right Now? Check if there is a current outage ongoing.

2 Upstreams service providers which HYPR Relies upon, or which individual HYPR customers may be integrated with are currently reporting issues impacting their services. HYPR has observed intermittent issues for end users of our service as a result. We will continue to monitor all three issues as they develop. https://status.firebase.google.com/incidents/YUxWS9naU43zfSvzA1i1 https://www.cloudflarestatus.com/

Resolution Microsoft has identified this problem as a bug introduced in the January 2025 Windows Update, and has fixed the problem in the February 2025 Windows Update. Windows Server 2019 February 11, 2025 — KB5052000 (OS Build 17763.6893) https://support.microsoft.com/en-us/topic/february-11-2025-kb5052000-os-build-17763-6893-2197b079-9a41-4e07-95c5-9ebce5fe20ec Windows Server 2022 February 11, 2025 — KB5051979 (OS Build 20348.3207) https://support.microsoft.com/en-gb/topic/february-11-2025-kb5051979-os-build-20348-3207-890f2739-1188-4367-bf10-4377a72db8ec These updates were published by Microsoft on February 11, 2025 (“patch Tuesday”), and will be deployed on servers configured to automatically install Windows Updates. If automatic updates are not enabled, customers can manually force the server to install the latest Windows Update via the GUI. (It can probably also be done via Group Policy, Intune, PowerShell script, etc. – these methods are left as an exercise for the reader.) Special considerations for Windows Server VM’s running in Microsoft Azure Windows Server running in an Azure VM gets different updates compared to Windows Server running on-prem or with a different provider (e.g. AWS). Windows Server on Azure gets a standard update once every quarter, and then two monthly hotpatch updates that don't require a reboot. So they only reboot once per quarter, rather than every month. The problem is that the February 2025 update on Azure is of the hotpatch variety, and it doesn't include a fix for the certificate authentication problem. Running Windows Update, either automatically or manually, will not fix the certificate problem. Microsoft has confirmed that a “standard” cumulative update package can be installed on Azure VM’s, and it won’t disrupt future hotpatch updates. Unfortunately, the standard package has to be downloaded from the Microsoft Update Catalog web site. The process isn’t too painful, but it’s not as simple as clicking one button. Windows Server 2019 Go to this Windows Update Catalog page: https://www.catalog.update.microsoft.com/Search.aspx?q=KB5052000 Click the Download button for “2025-02 Cumulative Update for Windows Server 2019 for x64-based Systems (KB5052000)”. A secondary window will pop up. Click on the link to download the update package file: windows10.0-kb5052000-x64_4d5c653ed24d769894ed1a2855d1c59fa70135af.msu Open the downloaded MSU file, either from the browser, or using Windows Explorer. This will start the Windows Update Standalone Installer utility. When prompted, confirm that you want to install the update. Like all Windows Update processes, it may take a long time, and the progress bar may be misleading! When the update has finished installing, reboot the server. Windows Server 2022 Go to this Windows Update Catalog page: https://www.catalog.update.microsoft.com/Search.aspx?q=KB5051979 Click the Download button for “2025-02 Cumulative Update for Microsoft server operating system, version 22H2 for x64-based Systems (KB5051979)” or “2025-02 Cumulative Update for Microsoft server operating system version 21H2 for x64-based Systems (KB5051979)” The updates are identical. It doesn’t matter which one you choose. A secondary window will pop up. Click on the link to download the update package file: windows10.0-kb5051979-x64_8b40fef9cafae506c09275c96e3aa883a9e3ae39.msu Open the downloaded MSU file, either from the browser, or using Windows Explorer. This will start the Windows Update Standalone Installer utility. When prompted, confirm that you want to install the update. Like all Windows Update processes, it may take a long time, and the progress bar may be misleading! When the update has finished installing, reboot the server.

Microsoft has conducted a thorough analysis of the problem reported to them by HYPR with the latest Cumulative Updates for Windows Server. They have determined that there was an issue introduced that causes problems for certain types of Authentication Certificates. Microsoft has provided the below information which will allow you to consume the Cumulative update while 'turning off' the specific single update responsible for this error, allowing Cert auth to work properly until they release a complete patch/fix for this issue. Info provided to us by Microsoft below: Problem Statement ================= After Applying the January 14, 2025, Cumulative Update for Windows Server 2022, or Windows Server 2019 on Domain Controllers January 14, 2025—KB5049983 (OS Build 20348.3091) - Microsoft Support https://support.microsoft.com/en-us/topic/january-14-2025-kb5049983-os-build-20348-3091-789bf923-7777-419d-9c3a-23f7c814930f January 14, 2025—KB5050008 (OS Build 17763.6775) - Microsoft Support https://support.microsoft.com/en-us/topic/january-14-2025-kb5050008-os-build-17763-6775-9a174725-a7ea-4e37-a6f8-e86f7c4d3f31 Windows Server Domain Controllers may fail to map a Client Certificate offered for Windows Logon if the Certificate Subject is "Empty". This happens even though the Certificate contains other properties as documented by Microsoft. Certificate Requirements and Enumeration | Microsoft Learn https://learn.microsoft.com/en-us/windows/security/identity-protection/smart-cards/smart-card-certificate-requirements-and-enumeration Root Cause ========== The root cause is determined to be defect included in the January 14, 2025 update for Windows Server 2022 and Windows Server 2019 that is checking the Certificate for the Subject information to be not blank. The failure return prevents the Domain Controller from finding a mapping for the Certificate within the Active Directory. Remediation/Recommendation ========================= To provide immediate temporary relief without uninstalling the January 14, 2025 Cumulative Update, Microsoft has published the Known Issue Rollback (KIR) Package for the respective Operating Systems for each Active Directory Services Domain Controller using the Microsoft documentation. Use Group Policy to deploy a Known Issue Rollback - Windows Client | Microsoft Learn https://learn.microsoft.com/en-us/troubleshoot/windows-client/group-policy/use-group-policy-to-deploy-known-issue-rollback Apply KIR to a single device using Group Policy. To use Group Policy to apply a KIR to a single device, follow these steps: 1.Download the KIR policy definition MSI file to the Domain Controller. Important: Make sure that the operating system that is listed in the .msi file name matches the operating system of the device that you want to update. Windows Server 2022 https://download.microsoft.com/download/f3c22ae2-4a00-4954-aa98-fe558e85dbc8/Windows%20Server%202022%20KB5044281%20250131_090543%20Feature%20Preview.msi Windows Server 2019 https://download.microsoft.com/download/7648edb5-d483-4ea4-9c10-69a6d58fb4bc/Windows%2010%201809%20and%20Windows%20Server%202019%20KB5044277%20250131_11501%20Feature%20Preview.msi 2.Run the .msi file on the Domain Controller. This action installs the KIR policy definition in the Administrative Template. 3.Open the Local Group Policy Editor. To do this, select Start, and then enter gpedit.msc. 4.Select Local Computer Policy > Computer Configuration > Administrative Templates > KB ####### Issue XXX Rollback > Windows 10, version YYMM. Note: In this step, ####### is the KB article number of the update that caused the problem. XXX is the issue number, and YYMM is the Windows 10 version number. 5.Right-click the policy, and then select Edit > Disabled > OK. 6.Restart the Computer. Investigation ============= Microsoft has determined the root cause of the incident. Microsoft has provided mitigations for immediate temporary relief without uninstalling the cumulative security update that has been verified by Microsoft and HYPR. Microsoft will continue to investigate a permanent solution for the incident in a future update. Additional Information / Discussion =================================== Use Group Policy to deploy a Known Issue Rollback - Windows Client | Microsoft Learn https://learn.microsoft.com/en-us/troubleshoot/windows-client/group-policy/use-group-policy-to-deploy-known-issue-rollback

As we have continued to investigate this with customers we have determined that this issue is encountered across multiple versions of Windows Server by specific KB updates all pushed around the same time: server 2019: KB5050008 server 2022: KB5049983 Depending on your version of Windows Server you will need avoid or uninstall the associated Windows Updates to restore Certificate Authentication functionality if you have been impacted. The changes in these windows updates appear to cause a problem with Smart Card Authentication Certificates. We are currently engaged with Microsoft regarding a potential fix. edit: Removed Server 2016 reference. Microsoft has determined this version was not impacted.

We’ve discovered a compatibility issue between Windows update KB5050008 (OS Build 17763.6775) and the HYPR Desktop Client when this Windows update is installed on domain controllers. This update seems to cause a problem with certificate authentication and can disrupt authentication and registration of HYPR for workstations. Please do not install this update on domain controllers, until HYPR is able to investigate further and advise on a solution; if already installed, contact HYPR Support immediately.

We have recceived Confirmation that the issue with Firebase has been resolved.

One of our subprocessors (Google Firebase) has notified us of a performance impacting issue with their Dynamic Link generation system. Internal testing has not revealed any issue with the HYPR Platform however if you do experience issues relating to Dyanmic links please follow this thread for up to date information: https://status.firebase.google.com/incidents/Q51i82jx6yV7p2Fid6Km

HYPR has completed rolling the fix out to all tenants. After observing the log data for the past hour we can confirm that the error rate for this issue has now come down to 0 and all hosted services should be operating normally again.

HYPR is currently in the process of rolling out this fix to all cloud environments. We are seeing overall reduced error rates as the change propogates to each system. Please check back for further updates as we work to get all systems remediated.

HYPR has identified the root cause of this issue and is in the process of rolling out a fix. Systems hosted on our cloud will be remediated automatically over the course of the next few minutes.

HYPR is currently investigating a partial outage of services impacting some cloud and on premise hosted customers. This issue relates to a certificate chain issue tied to licensing and impacts FIDO Registration (WFA and Web). FIDO2 registration and FIDO/FIDO2 authentication are not impacted. We will update this page with additional information as we progress toward a resolution.

This incident has been resolved.

We are not seeing any customer impact, we are monitoring closely

We are currently investigating this issue.

This incident has been resolved.

Issue has been resolved. All systems are operating within normal parameters.

We are continuing to monitor for any further issues.

A fix has been implemented by Firebase and we are monitoring the results.

Customer with Push notifications enabled currently may be affected. QR code authentication is not affected.

This incident has been resolved.

We are currently in the process of rolling out the fix to our customers that are affected.

The issue has been identified and we are working on rolling out a fix to all our cloud hosted environments.

HYPR is experiencing an issue that may impact your ability to register users. We are actively investigating the issue and will provide periodic updates as we work to resolve the issue as quickly as possible

This incident has been resolved, and HYPR will be continuing to monitor all dependent services for any further issues.

AWS is still working on services that are impacted and we will continue to monitor through out the night.

HYPR has recovered from initial issues caused by upstream provider AWS. We will continue to monitor for further issues throughout the rest of the day/night.

We are continuing to monitor for any further issues.

We are continuing to monitor the situation with AWS. At this time, there are still service disruptions impacting our systems ability to handle certain authentication requests. Presently Workstation and Web Login should be operating normally, however logins to the Control Center may continue to be impacted.

We are continuing to monitor the situation with AWS. At this time, there are still service disruptions impacting our systems ability to handle certain authentication requests. Presently Workstation and Web Login should be operating normally, however logins to the Control Center may continue to be impacted.

We are continuing to monitor the situation with AWS. At this time, there are still service disruptions impacting our systems ability to handle certain authentication requests.

Our Upstream Hosting Provider(AWS) is currently experiencing an issue impacting their API services in the US-East-1 Region. As a result some of our HYPR Zero customers may be impacted, and experience errors when authenticating. We are tracking this issue closely and will provide further updates once we know more, and/or the issue is resolved