Is Keeper Down Right Now? Check if there is a current outage ongoing.

We are investigating login errors in the EU data center.

### **Incident Summary** The Keeper DevOps team was alerted to API errors in the US region. Upon investigation, we identified a high number of database deadlocks affecting performance. To mitigate the immediate impact, we restarted the affected writer instance. ### **Root Cause** Historically, similar issues were linked to specific queries impacting MySQL 8.x updates on the RDS Aurora platform. In this case, we discovered that a utility designed to monitor deadlocks and capture diagnostic data was inadvertently contributing to the problem. A known bug in the MySQL minor version used in production caused our data-gathering query—intended to analyze deadlocks—to trigger additional locking, exacerbating the issue. ### **Resolution & Mitigation** * We promptly disabled the problematic monitoring utility and applied database parameter optimizations in collaboration with the AWS team to enhance performance under peak load. * These changes were fully implemented within 24 hours, and no further excessive row locks have been observed. * Our DBA team is actively working with backend engineers to further optimize write operations for improved performance. * A backend code update with additional optimizations is scheduled for release the week of March 17. We appreciate your patience and will continue to monitor and enhance system performance to prevent similar issues in the future.

Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.

### **Incident Summary** In the US, EU, and AU regions, users experienced vault errors when attempting to download a file attachment immediately after adding it to a record. This issue affected only newly uploaded attachments in these three data center regions. The Gov, JP, and CA regions were not impacted. ### **Root Cause** During our investigation, we identified a discrepancy in how file attachment triggers were implemented across different regions. A recent Terraform change inadvertently removed the triggers responsible for moving file attachments in the US, EU, and AU regions, preventing newly uploaded files from being immediately available for download. ### **Resolution** We corrected the issue by restoring the missing triggers and applying the necessary changes. Additionally, any affected files were successfully moved to their intended locations. ### **Impact & Mitigation** * No data loss occurred. * All impacted files were successfully resolved by our corrections. * We corrected the issue to prevent similar discrepancies in future infrastructure changes. We apologize for any inconvenience and appreciate your patience while we resolved this issue.

The issue has been resolved. Additional information is in the postmortem report.

Some users are receiving errors when downloading file attachments. We are working on resolution.

The issue has been resolved

We are continuing to investigate this issue.

We are investigating login errors and our team is currently working on the issue.

We identified a backend API and resulting database query which triggered a high database load and affected production services. We are publishing a permanent fix on Monday, Feb 24.

The issue has been resolved and root cause determined. We will update the postmortem on this issue shortly.

We are still working on the issue which is causing login errors in the US region. Please use Keeper's "work offline" while we are working to resolve the issue.

The issue has been resolved.

We are currently investigating login errors in the US region.

At 9:21AM PST we received notifications of API errors coming from our backend systems. The issue resolved itself without any required action at 9:31AM PST. During that 10 minute period, there was a database lock in the primary writer instance causing new login sessions to fail. Recently, Keeper upgraded all production RDS Aurora databases to MySQL 8.x. This updated database engine is sensitive to one particular SQL query that was causing large table scans to occur. The code changes and QA process was already under way when this issue occurred. The backend code change which addresses this issue passed QA at 12:48PM PST and we released the fix to prevent this issue from occurring again \(Release Jira ticket REL-4633\). We’re keeping a close eye and will issue additional updates as necessary.

We have resolved the issue and plan to deploy a backend update shortly.

We are aware of a high number of errors in the US region. The issue will be fixed shortly.

The SMS delivery issue with T-mobile has been resolved.

We are currently experiencing delivery issues with T-Mobile SMS 2FA messages. While AWS works to resolve the issue, SMS-based 2FA codes will be temporarily re-routed to email for affected users.

On Tues, Sept 24 at 6:06AM PDT, the DevOps Engineering team started receiving notifications regarding API slowness and HTTP 504 errors coming from the Keeper AWS infrastructure. After an investigation, it was determined that our NGINX instances were throwing the errors due to an overload of network traffic. It was then determined that the network traffic increase was caused by an update to the Keeper Desktop application which was published on Monday, the day before. The DevOps team scaled the AWS infrastructure several times over the period of 60 minutes to address the issue. The issue was fully stabilized and resolved by 7:20AM PDT. After a full analysis of the product update process, the engineering team has decided to change the update mechanism of the Keeper Desktop application to utilize a new distributed content delivery network which does not impact the other Keeper AWS infrastructure. This change will be implemented over the next 2 weeks and in preparation of the next Keeper Desktop application update.

This incident has been resolved.

We are continuing to monitor for any further issues.

A resolution has been rolled out - the team is monitoring system performance.

Root cause has been identified and we are actively working on a resolution.

We are currently investigating reports of slow server response times and request timeouts

This issue has been resolved.

We are continuing to work on a fix for this issue.

We have identified an issue with Keeper Secrets Manager Java SDK that is throwing API errors. A solution is underway and we'll be releasing an update on the Keeper backend API shortly. No changes are required on the SDK. This also affects Keeper Connection Manager integrations with KSM.

SMS delivery to T-mobile customers appears to be resolved.

SMS codes to T-mobile subscribers appears to be delayed or having delivery issues. Codes have been temporarily routed to email until the issue has been resolved. Please check your email for any 2FA codes.

The issue has been resolved. The issue was caused by a backend software update which had the effect of throwing an error during some Okta SSO login transactions. Our DevOps team quickly reverted the change and our engineers will be addressing the bug in a follow-up release.

We're aware of Okta SSO login errors. We have identified the issue and it will be resolved within a few minutes.

From 2:45PM to 3:00PM PST, several of the Keeper application API instances in the US region were responding with HTTP 500 errors due to an internal application error. The Keeper DevOps team identified the error and restarted those affected instances. The software engineering team has been investigating the root cause and will be addressing in an upcoming backend release.

The issue has been resolved

The Keeper DevOps team is aware of vault API errors and we're currently working with AWS to address the issue. The team will have resolution shortly.

The issue has been resolved. An underlying AWS database error resulted in a failover event which was handled successfully after a few minutes. We are working with the AWS team to identify the root cause.

We have identified the issue with AWS and the issue is being resolved in a few minutes.

We are investigating login errors in the US production region due to AWS connectivity issues.

A spike in API errors was detected by our team at 6:24 PM PST due to an AWS ElastiCache error in the AU \(Sydney\) region. The affected resource was restarted and the errors cleared at 6:43 PM PST. Our engineering team will investigate root cause and address any findings.

Vault login issues in the AU region have been resolved.

We are investigating vault login errors in the AU region.

AWS RDS cluster instances were automatically rebooted at approximately 09:27 Central (04:27 UTC). We are investigating the issue with the AWS team and will update our statuspage with the findings.

The issue has been resolved. We have reverted the backend update in GovCloud and services are restored.

We are aware of vault login errors in the US_GOV region. Keeper DevOps team is investigating the issue related to a recent backend app update.

The Keeper engineering team has been researching the cause for recent Duo Push issues. Our investigation has resulted in the identification of a couple issues. The description is below: 1. The Duo Push services also have a direct dependency on the Keeper Push services which utilize websocket communication from the Keeper backend to the client device. 2. We determined that scaling additional instances of our push services was necessary to meet the growing usage of Duo and general push features across the Keeper end-user client apps 3. Scaling the backend instances immediately resolved the latency and the intermittent issues encountered by users. 4. In addition, we met with the Duo team, revisited our Duo integration code and we decided to rewrite some portions of the integration to reduce the number of dependencies and asynchronous processes. This is planned for release by December 20, 2023.

Duo Push issues in the US region have been resolved. It may be necessary to reload the web vault page or restart the web browser.

We are investigating intermittent Duo Push issues in the US region. Hard refreshing the browser window appears to mitigate the issue.

The FIDO2 / Yubikey login issues have been resolved. The engineering team is still investigating root cause and we'll update the case notes shortly.

We are investigating FIDO2 / Yubikey login issues in the US data center.

The SCIM issue with Azure has been resolved.

Today's backend release introduced errors with SCIM provisioning via Azure. A fix has been created and it is going through the deployment process right now. ETA 4:35PM PST.

At 12:51AM PST, an AWS Aurora database failover occurred. The Keeper application servers properly recovered after 60 seconds. From 1:08AM PST to 1:16AM PST, a high number of login errors occurred from a subsequent and related AWS Elasticache failure. After further investigation, the team will update this event postmortem.

Vault login errors have been resolved.

We are currently investigating vault login errors.

GovCloud Vault Login issues have been resolved. We will update this status event with additional details shortly.

We are aware of vault login issues in the GovCloud region. The issue has been identified and it will be resolved momentarily.

Amazon AWS has resolved their SMS delivery issue to Verizon customers.

SMS delivery to Verizon is currently affected by an AWS issue. We have temporarily routed SMS 2FA delivery to customer email while AWS researches the issue on their side. Please check your email to receive your SMS 2FA codes. We will update the status page when the issue has been resolved.

SMS delivery to Verizon is currently affected by an AWS issue. We have temporarily routed SMS 2FA delivery to customer email while AWS researches the issue on their side. Please check your email to receive your SMS 2FA codes. We will update the status page when the issue has been resolved.

We are investigating SMS 2FA delivery in the US data center

Our team received notification of API errors at 10:20AM PST. Between 10:20AM and 10:50AM the errors were intermittent and causing vault login errors in the US data center. The DevOps team resolved the issue fully at 10:53AM PST. After resolution of the issue, the engineering team researched the cause and found that a bug in the desktop app and also a related bug down the chain of processes in the backend API caused some client devices to generate a high number of full-sync API calls. We have identified the root cause and a patch is being applied to both Desktop App and Backend API components over the next 24 hours.

Vault login issues in the US has been resolved. We will update this statuspage event with additional details.

We are investigating vault login issues in the US data center.

The GovCloud DevOps team uses automated infrastructure deployment systems via Terraform to manage all environments. A change was made to subnet VPC routing which inadvertently caused connectivity issues between certain backend services and a specific Redshift cluster. This issue affected admins on the Keeper Admin Console who have ARAM permissions applied to their role policies. Delegated admins without reporting permissions were not affected. Vault login and other client-side vault applications were not affected. Recent improvements to the Vault and Admin Console backend APIs masked the issue for the automated detection systems which would have immediately flagged the error and alerted the DevOps team. We have resolved the infrastructure route and updated the monitoring systems to take this into account and will continue to monitor for any future issues.

The issue has been resolved. Console login to GovCloud for users with ARAM reporting administrative functions has been restored.

We are continuing to work on a fix for this issue.

We have identified an issue affecting users on the Admin Console in the GovCloud region who have administrative rights over Advanced Reporting & Alerts module (ARAM).

Version 16.6.2 has been published by Mozilla and it is available for update from the Firefox add-on store https://addons.mozilla.org/en-US/firefox/addon/keeper-password-manager/?src=search

We have published version 16.6.2 to the Mozilla store and await review. This update resolves all Firefox related issues.

If you encounter any problems with Firefox extension 16.6.1 after restarting your browser, please uninstall and reinstall the extension. Another update 16.6.2 is being created to address this issue.

A redshift cluster in the AWS region was unhealthy due to high CPU load from a recent backend API change. DevOps was alerted to the issue and worked with the engineering team to revert the change. Engineering will redeploy the update once the software changes have been implemented, and in addition a change is being made to prevent any redshift related issues to affect vault logins in the future \(KA-5630\). Additional scaling is planned on redshift \(SYS-2595\).

The issue has been resolved.

We are aware of slow vault login issues in the US data center, and the issue will be resolved in approx. 20 minutes.

Mozilla has released version 16.6.1. Visit the Firefox Add-ons screen and check for updates to install.

We have pushed a new version to Mozilla (16.6.1) which is pending review to address autofill issues on Firefox.

We are aware of login and filling errors with the KeeperFill browser extension on Firefox. This issue is randomly occurring during initialization of the extension. Restarting Firefox when this occurs may temporarily resolve the issue. We have pushed a new version to Mozilla (16.6.1) which is pending review. Mozilla should release the update within 24 hours.

The login issue has been resolved. Please force-reload the web vault if you are still experiencing any issue. Download Keeper Desktop version 16.10.5 from our website or update from the app store when this version has been published in the next 24 hours.

We are releasing a Web Vault and Desktop App update to version 16.10.5 that resolves a login error for customers with email domains containing certain characters such as a dash. This update will be published by approximately 12PM PST. Pushing to the app stores for Desktop version may take up to 24 hours. After the update goes live, we will send another notification.

For 9 minutes from 2:46PM PST to 2:55 PM PST, a high number of login errors occurred as a result of a scheduled backend API software update. The issue resolved itself after the update was completed. The development team is researching the underlying cause.

Login issues have been resolved.

We are aware of login issues in the US data center. We are working towards resolution.

Amazon has resolved the issue.

AWS acknowledged that they are having some SMS delivery problems. We have routed SMS delivery to email temporarily.

Amazon AWS is having an SMS delivery outage. They are working on it. We have temporarily routed SMS codes to email. Please check your email to receive the code.

We are investigating SMS delivery issues in the US Data Center.

Issues related to record field changes is resolved in production, however new updates are still being published over the next 24 hours to take into account the upcoming feature updates planned for next week.

Several updates will be published later today on end-user applications to address a few production issues related to a new record field type change. - Keeper Connection Manager 2.15.1 - Android 16.6.70 - iOS 16.8.3 - Keeper Secrets Manager SDK for Java The issues affect the creation and editing of new records in certain scenarios. We will post updates when the fixes have been published. Customers affected will need to update to the new versions.

Some SSO logins on the GovCloud environment were generating a 403 error. An overly aggressive WAF rule in the Amazon Shield system blocked the requests. The DevOps team applied the necessary WAF rule policy changes to resolve the issue.

We are aware of 403 errors that were occurring when logging in through GovCloud with SSO. The issue has been resolved. An overly aggressive WAF rule in the Amazon Shield system blocked the requests.

At 3:45PM PST the DevOps team was notified of a high number of API errors. Upon investigation we found the cause of the errors in 2 of the application servers. These application servers were subsequently removed from the pool. The issue was resolved between 4:06PM and 4:15PM PST. After further investigation, the root cause was found by the backend engineering team, and all of the application servers have been updated.

This issue has been resolved.

We have identified the cause of some vault login errors in the US data center. The Keeper DevOps team is resolving the issue in a few minutes.

Between 8:40AM PST and 9AM PST there were latency issues and some timeouts during the login and sync process between Keeper client applications and backend APIs in the US data center. The engineering team identified the cause of the issue related to certain database transactions. The issue was resolved and affected services recovered after a few minutes. The engineering team also identified the root cause and a change ticket has been created which will be implemented at 7PM PST tonight.

Vault login issues have been resolved.

We are aware of slow vault login issues in the US data center. The issue has been identified and the team is working on resolution.

The issue has been resolved.

We have identified an issue in the US Data Center causing Duo Push to fail periodically. This will be resolved within 15 minutes.

Keeper’s backend has a system in place to control which end-user application versions are allowed to authenticate. During our publishing process of version 16.4.14 we mistakenly blocked version 16.4.13 application during one of the 10-minute update cycles. We have corrected the error in the process which caused the issue.

We are aware that some users are receiving a prompt to update the browser extension upon logging in. This has been resolved and will automatically clear out in a few minutes.

This incident has been resolved.

Keeper DevOps is performing system maintenance for approx. 10 minutes in the US Data Center.

Consumer customers in the US data center may have received an error on the Web Vault when logged in for a few minutes. This was resolved and due to system maintenance that took a few minutes longer than expected.

Keeper’s push services were not communicating properly to the Duo Push servers for a brief period due to an error being generated in the Keeper Push application servers. We have opened a ticket with our DevOps team to add additional alerts to ensure that we are notified in case of a similar issue in the future.

Duo Push was not responding to users in the EU data center for a brief period. The issue has been resolved.

On February 3, 2023 at approximately 12:20 pm PST, the Keeper website encountered a surge in network traffic having over 100 million requests per minute coming from a global distributed IP network. Keeper's DevOps team worked closely with the AWS Shield team to successfully block it. We implemented mitigations in the AWS Shield environment to protect against this in the future. Service was restored at 2:08 pm PST.  This outage impacted users on our web-based apps and cloud APIs. Keeper's native applications, including the Web Vault, Desktop App, iOS, Android, Windows, Linux and offline mode were not affected. To learn more about offline mode, see: [https://docs.keeper.io/user-guides/vault-offline-access](https://docs.keeper.io/user-guides/vault-offline-access) We greatly value your understanding, patience and support. If you have any questions please contact us at: [https://www.keepersecurity.com/support.html](https://www.keepersecurity.com/support.html)

We resolved a network issue that temporarily affected vault logins. Native app and off-line vault access was not affected. Thank you for your patience.

We have resolved the network issue and vault logins are now recovering.

Our DevOps team is currently working with Amazon AWS team to resolve a network issue affecting user logins. We will continue to post updates on the status page here.

We are continuing to work on a fix for this issue.

We have identified the source of the surge in traffic causing network issues and we are working towards resolution.

We are continuing to investigate this issue.

We are experiencing a network issue, the Keeper Security DevOps team is investigating.

We are currently investigating vault login issues in the US

The issue has been resolved.

We are aware of login issues on the Web Vault for consumer accounts. This will be resolved shortly.

Push issues are resolved

We are investigating an issue with Push Services in the US data center. Some real-time syncing features may be affected on the Web Vault and Desktop App.

AWS has confirmed resolution

AWS is having SMS delivery issues to users on the T-Mobile network. We have routed SMS 2FA codes to email while this issue persists.

SMS delivery appears to be resolved.

There appears to be a high failure rate of AT&T SMS delivery. Amazon AWS is investigating the issue. We have temporarily routed 2FA messages to email.

This is an important reminder that tomorrow (Aug 17, 2022) Keeper's Cloud SSO Certificate used for signing SAML requests is expiring and a new cert is available. Enterprise customers using SSO Connect Cloud may be affected. Please read the page below for more details: https://docs.keeper.io/release-notes/enterprise/other/cloud-sso-certificate-renewal-2022

Keeper’s new SSO certificate has been live since Aug 8, however a change made in the SAML request to send the new cert inadvertently caused some identity provider configurations to reject the request. Updating the SP Cert on the identity provider would have quickly resolved this for all affected customers, but we went ahead and reverted the change on our side. As a reminder, the old SSO Cloud certificate will expire on Aug 17th which will affect some identity providers. We recommend that customers using SSO Connect Cloud proactively update the SP Cert on the identity provider before the 17th per the instructions below: [https://docs.keeper.io/release-notes/enterprise/other/sso-certificate-renewal-2022](https://docs.keeper.io/release-notes/enterprise/other/sso-certificate-renewal-2022)

The SSO issue has been resolved for those affected. We'll post information in the postmortem page.

A change was pushed live which inadvertently affected several customers who are using certain IdP configurations. The change is being reverted within 15 minutes.

We are continuing to work on a fix for this issue.

We are aware of an SSO login issue for some customers which will be resolved shortly.

We had a momentarily high load on the vault due to the periodic clearing of old / expired sessions. We have corrected the process.

The issue has been resolved.

We are investigating vault login errors.

Further regional transient internet latency has not been observed.

Cause determined to be transient regional internet latency. Team will continue to monitor.

We are currently investigating reports of latency in the EU data region.

Keeper maintains multiple Chrome extensions on the Google Chrome Store. One of the reasons for this is to stage our rollouts to different sized populations of users. Between the 2nd and 3rd stage of deployment, a bug was introduced due to a packaging issue which caused Autofill to stop working for that population. Once the issue was discovered, the Keeper browser extension team immediately identified and resolved the issue. The release was posted to the Google Chrome store within 1 hour of notification. Google Chrome Store’s policy requires a member of their app review team to manually approve all submissions. Google approved the release at 10:47PM PST. As a result of this issue, the engineering management and QA team have made an improvement to our release process to prevent this type of issue from occurring in the future. If you have any questions please contact us at [https://www.keepersecurity.com/support.html](https://www.keepersecurity.com/support.html)

The Chrome browser extension has been published by Google. All users will automatically receive the update. If you installed the alternate Keeper extension, you may continue using it. Keeper maintains multiple browser extensions on the Chrome Store and they are always updated to the latest version.

We are aware of an Autofill issue with Chrome extension 16.4.4. Edge, Safari and Firefox are not affected. We have published a fix to the Google Chrome store and we're currently waiting for them to approve the update. We expect the fix to go live within 24 hours. For a temporary workaround, please follow the steps below: 1. Go to Window > Extensions and remove the current extension 2. Install the version from the link below: https://chrome.google.com/webstore/detail/kbedblbpfmeicfpadihimgombbafaeeh 3. "Pin" the new extension to make the icon visible Important: Ensure only one Keeper extension is installed at a time. For customers who roll out with group policy, you can push the extension listed above.

We are aware of an Autofill issue with Chrome extension 16.4.4. A fix has been posted awaiting approval from the Google Chrome store.