Is JumpCloud Down Right Now? Check if there is a current outage ongoing.

We're currently investigating reports of issues with JumpCloud's RADIUS-as-a-Service. We are investigating the cause of the issues.

We are currently investigating intermittent reports of Administrators being unable to submit support cases via the JumpCloud Admin Portal. Support cases can still be submitted by emailing support@jumpcloud.com. We are investigating the cause of the issues currently, and will update the status event once resolved.

We are currently investigating an issue with System Insights displaying incorrect Local User Information in the Admin Console. We will report back as soon as we have an update. We apologize for any disruption this may cause. Thank you,

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're currently investigating an issue with JumpCloud's Console where Administrators may be unable to save changes for some device records. We are investigating the cause of the issues currently, and will update the status event once resolved.

We're currently investigating an issue with JumpCloud's Console where Administrators are unable to save changes for device records. We are investigating the cause of the issues currently, and will update the status event once resolved.

 # **Incident Report** ‌ **Date**: May 12, 2025 **Date of Incident:** May 2, 2025 **Description**: RCA for ADI User Login Errors ‌ **Summary:** On May 2nd, at approximately 07:00 UTC, a change was deployed to JumpCloud’s Active Directory Integration causing a warning message to appear and preventing successful logins for some users across a subset of organizations.  During that time the workaround for users was to change their password, which then allowed a successful login. ‌ **Root Cause:** The change introduced was part of an effort to modernize external user identification in Active Directory, which inadvertently caused an error in handling some restricted attributes. This resulted in the incorrect transformation of these fields, setting the password status of some external user accounts to "pending”.  This event disrupted user access and necessitated a password update for the affected users until JumpCloud successfully rectified their account status. ‌ How was this missed in your testing?  Unfortunately, the change affected a subset of users that had a unique combination of values set on their user record.  Specifically, the logic was only applied to users that had an empty value on their restricted fields.  The testing that was performed did not contain a user in that state, incorrectly passing the test. ‌ **Corrective Actions / Risk Mitigation:** 1. Revert that code version - DONE 2. Modify all incorrect user entries in our backend systems - DONE 3. Increased testing scenarios verifying all user states and backward compatibility - IN PROGRESS

This incident has been resolved.

We have completed implementation of our remediation plan and do not anticipate any further impact. We will continue to monitor for any issues.

We are continuing with our implementation of our remediation plan. We anticipate several more hours to complete. We will provide another update in two hours.

We have identified a fix for this issue. We have validated our remediation plan and are implementing it now. The plan will take several hours to complete. We will provide another update in one hour.

We are in the process of validating our remediation plan. We will communicate it as soon as it is ready. We will share another update in one hour.

We have reverted the update which caused the initial impact. There should be no new occurrences. We are creating a remediation plan at this time and will share it as soon as it is ready. We will share another update in one hour.

We have identified the cause of the issue and are refining the fix to be implemented. We will share another update in one hour.

We have identified the cause of the issue and a fix is being implemented. We will share another update in one hour.

We are continuing to investigate reports of user accounts being changed to pending status for tenants integrated with Active Directory Bridge. We will share another update in one hour.

We are investigating reports of user accounts being changed to pending status for tenants integrated with Active Directory Bridge.

 # Incident Report **Date**: May 5, 2025 **Date of Incident:** Apr 30, 2025 **Description**: RCA for User Portal / SSO Failed Authentication ‌ **Summary:** On April 30th at 12:35 UTC our monitors detected a significant increase in errors across our User Console API endpoints. The increased errors manifested as failed authentication for new attempts to the User Portal, SSO and other services.  Existing connections to applications continued uninterrupted. At 12:41 UTC a formal incident was declared and multiple teams were paged. The issue was resolved at 12:58 UTC ‌ **Root Cause:** A state transition anomaly occurred during the credential rotation process for a database within our authentication infrastructure, resulting in connection failures by dependent services to that layer. Consequently, end-users experienced authentication errors when attempting to establish new sessions with specific applications. ‌ The authentication service team was able to identify a failure with secondary credentials during the rotation and quickly failed back to valid secrets. ‌ ‌ **Corrective Actions / Risk Mitigation:** 1. Fail back to primary secrets - DONE 2. Increased automation for secrets rotation at this layer - IN PROGRESS

This incident has been resolved.

We have identified an issue that was affecting authentication and SSO and have implemented a fix. Services have recovered and we are monitoring.

This incident has been resolved.

We have identified an issue that was affecting authentication and SSO and have implemented a fix. Services have recovered and we are monitoring.

We are investigating reports of user accounts being changed to pending status for tenants integrated with Active Directory Bridge.

We have identified an issue that was affecting authentication and SSO and have implemented a fix. Services have recovered and we are monitoring.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with JumpCloud Reports failing to download. We will update the status as we learn more.

We are currently investigating an issue with JumpCloud Reports failing to download. We will update the status as we learn more.

 **Date**: Apr 23, 2025 **Date of Incident:** Apr 18, 2025 **Description**: RCA for ADI Outage ‌ **Summary:** On April 18th at 1:27 PM MT JumpCloud experienced an issue affecting our Active Directory Integration \(ADI\) agents. Some customers using ADI sync and Delegated Auth experienced connectivity issues, resulting in failed directory synchronizations and authentication attempts. New ADI installs would have failed during the incident window, and ADI agents attempting to restart or reconnect during the incident window would have failed. ‌ **Root Cause:** The incident was caused by the expiration of a certificate used for mutual TLS \(mTLS\) authentication between ADI agents and our infrastructure. This certificate is essential for securing the communication channel between customer environments and JumpCloud services. ‌ Why wasn’t this caught before the certificate expired? In mid-2024 we migrated our global proxy infrastructure from EC2 into Kubernetes. During this transition, monitoring for the proxies that handle mTLS were mistakenly deleted and not recreated on the new infrastructure. ‌ Why did it take so long to recover? There are two reasons recovery took longer than it should have. The first reason is the missing monitoring above meant this went unnoticed by our operations teams once the cert expired. The second is our teams did not see any instability or outage signals in our internal telemetry, which led them to believe this was customer environment specific. This was because existing connections were un-impacted. ‌ **Corrective Actions / Risk Mitigation:** ‌ 1. Enhancing our certificate monitoring systems to provide earlier notifications of upcoming expirations - DONE 2. Improving our internal escalation procedures to reduce time-to-resolution - DONE 3. Implement additional automated testing to detect potential authentication issues - PLANNING 4. Creating more comprehensive documentation for critical operational procedures - WIP

This issue has been resolved and service has now returned to normal.

We are currently in the process of implementing a fix for this issue.

We have identified an issue affecting the Active Directory Integration Sync and Import services, including Active Directory Delegated Authentication. This issue affects users, user updates, and password changes are not syncing between JumpCloud and Active directory, as well as delegated authentication to Active Directory. In the meantime, if your users are unable to authenticate because you have delegated authentication enabled for your users, please review the following article for steps on disabling delegated authentication. https://jumpcloud.com/support/adi-use-ad-delegated-authentication#disabling-delegated-authentication-for-users

The Active Directory Integration Service was found encountering errors and we are investigating the issue.

This issue has been resolved and service has now returned to normal.

We are currently in the process of implementing a fix for this issue.

We have identified an issue affecting the Active Directory Integration Sync and Import services, including Active Directory Delegated Authentication.This issue affects users, user updates, and password changes are not syncing between JumpCloud and Active directory, as well as delegated authentication to Active Directory.In the meantime, if your users are unable to authenticate because you have delegated authentication enabled for your users, please review the following article for steps on disabling delegated authentication.https://jumpcloud.com/support/adi-use-ad-delegated-authentication#disabling-delegated-authentication-for-users

The Active Directory Integration Service was found encountering errors and we are investigating the issue.

The Active Directory Integration Service was found encountering errors and we are investigating the issue.

This incident has been resolved.

We believe we have identified the issue that was impacting the distribution of the Password Manager enrollment emails. We are currently working on implementing a fix.

We are currently investigating an issue involving the distribution of Password Manager enrollment emails. We are investigating the cause of the issue currently, and will update the status event once resolved.

We are currently investigating an issue with the Password Manager notifications part of our platform.

We are currently investigating an issue with the Password Manager notifications part of our platform.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently observing elevated error rates in the management of iOS devices. This may manifest for administrators as missing MDM tabs, impaired policy functionality, and failures in executing MDM commands.

We are currently observing elevated error rates in the management of iOS devices. This may manifest for administrators as missing MDM tabs, impaired policy functionality, and failures in executing MDM commands.

This incident has been resolved.

The delays with policy deployment have returned to normal and we are continuing to monitor queues.

There continues to be a delay in processing the application of new and modified device policies to macOS, Windows, and Linux systems. During this time existing policies applied to devices will continue to function as expected but application of new policies and policy changes will be impacted.

We are continuing to investigate reports of JumpCloud Device Policies failing to apply on macOS, Linux and Windows devices.

We are continuing to investigate reports of JumpCloud Device Policies failing to apply on macOS, Windows, and Linux operating systems.

We are continuing to investigate reports of JumpCloud Device Policies failing to apply on macOS, Linux and Windows devices.

We are currently investigating reports of JumpCloud Device Policies failing to apply on macOS, Windows, and Linux operating systems.

We are currently investigating reports of JumpCloud Device Policies failing to apply on macOS and Windows devices.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We're currently investigating reports of degraded performance or intermittent connection issues with JumpCloud's Remote Assist Agent. We are investigating the cause of the issues currently, and will update the status event once resolved.

This incident has been resolved.

We are currently experiencing issues with the JumpCloud Premium Support Chat function. Customers are able to submit tickets via the Admin Console. Instructions for submitting tickets via the Admin Console can be found here: https://jumpcloud.com/support/contact-jumpcloud-support#submitting-a-support-ticket-from-the-admin-portal

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with higher than normal propagation times for devices. We will update this issue as we know more.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being evaluated.

We're investigating reports of degraded performance with JumpCloud's Directory Insights service. We are investigating the cause of the issues currently, and will update the status once resolved.

This incident has been resolved.

We are currently monitoring RADIUS authentication failures in the US West region. The service is operational at this time.

 **Date**: Jan 17, 2025 **Date of Incident:** Jan 16, 2025 **Description**: RCA for Apple MDM Un-enrollment ‌To start, we’d like to acknowledge and apologize for the impact this incident had. We pride ourselves in operating with excellence and have increased efforts to minimize impact when things go wrong. We missed the mark here. ‌ **Summary:** On January 16th at 11:38 AM MT JumpCloud deployed a new version of Apple MDM.  At 11:47 AM JumpCloud detected macOS devices un-enrolling from Apple MDM and an investigation began.  At 12:03 PM a formal incident was declared and our incident management team came online to coordinate multiple teams in recovery efforts.  At 12:14 PM MT a feature flag was disabled, stopping the queuing of any further un-enrollments \(more on this later\).  At 12:15, the new code was completely rolled back, and at 12:32 JumpCloud deleted the command queue containing any further un-enrollment directives.  At this point, the teams worked to find the quickest solutions to provide customers for safely re-enrolling systems, and a comms plan for affected customers making them aware of the issue. ‌ **Root Cause:** First, let's start with some context on how JumpCloud uses feature flags. Our practice is to deliver code to production through small changes. This reduces the risks inherent with large changes and allows us to quickly identify what specific change could be responsible for erroneous behavior.  Feature flags are essentially if-statements in the code that determine which path to follow and execute.  When a flag is “on” new code is executed, and when the flag is “off” the code is skipped.  Our teams use these often with our deployments, providing the ability to turn features on or off based on certain attributes \(like organization id\) without modifying the source code.  This is where things went wrong. ‌ The actual change deployed was in an effort to introduce Apple Declarative Device Management \(DDM\) support, which lets devices apply configurations independently based on certain criteria.  For an existing device we match on a number of factors, one of which being the unique device identifier \(UDID\) generated at enrollment. This is a security measure to prevent impersonation or a fake device.  Unfortunately, with this new API the UDID did not match for some macOS devices, causing device un-enrollment.  This was not caught in our pre-production environments because the intended state for the feature flag controlling this change was off. In the pre-production environments our testing passed because this code path was not active. ‌ Why was the feature flag on?  The teams use a rollout status with each flag, along with other identifiers for that code block.  What we missed was a validation step ensuring the feature flag was in the expected state before releasing the code to production. ‌ What are you doing to ensure this doesn’t happen again?  With every incident, we perform a thorough post incident review to address gaps in many areas, including process and testing.  We take these very seriously and discuss all incidents on a bi-weekly cadence with the entire engineering organization.  These reports then get rolled up to our executive staff.  This incident clearly exposed a gap in validation and we have changes in flight immediately to address that.  We’ve also stopped any future deployments until this gap \(and any others we find with this investigation\) is closed and approved by our SRE team. ‌ ‌ **Corrective Actions / Risk Mitigation:** 1. Flip the feature flag to off for this code block, and revert the code - DONE 2. Stop production deploys for this code base  - DONE 3. Review and harden the validation steps for feature flags - IN PROGRESS 4. Modify our automation to test code with feature flags in both positions - IN PROGRESS

This incident has been resolved. To enroll devices again, customers can follow the below steps: ADE-enrolled devices If you have devices configured for ADE there are two options to restart the ADE enrollment; Administrator Privileges Required* see: https://jumpcloud.com/support/set-admin-sudo-permissions Setup a command to execute `sudo profiles renew -type enrollment` Or Have your users execute `sudo profiles renew -type enrollment` in a terminal window In both scenarios the user will be prompted to complete the re-enrollment of the device. This action will require the user to have administrator privileges on the device, at that time. Non-ADE enrolled devices For devices that are not configured for ADE enrollment it is advisable to apply the JumpCloud MDM Enrollment policy (https://jumpcloud.com/support/create-a-mac-mdm-enrollment-policy). Note, this still requires the user to have administrator privileges to complete the MDM enrollment when they complete the enrollment process. Admins may temporarily elevate standard users to admin users by following the steps laid out in this article: https://jumpcloud.com/support/set-admin-sudo-permissions. Additional Support Articles for Remediation https://jumpcloud.com/support/re-enroll-apple-devices-into-mdm-and-preserve-device-record https://jumpcloud.com/support/set-admin-sudo-permissions https://jumpcloud.com/support/create-a-mac-mdm-enrollment-policy

We've identified the issue and implemented a fix. Users that were un-enrolled from Apple MDM may need to re-enroll their device. More information on re-enrolling Apple devices can be found below: https://jumpcloud.com/support/re-enroll-apple-devices-into-mdm-and-preserve-device-record

We are currently investigating an issue with agent installs and Apple MDM enrollments. We will update this issue as we know more.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're currently investigating reports of intermittent authentication issues with JumpCloud's MFA for LDAP in Europe. We are investigating the cause of the issues currently, and will update the status event once resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with Windows MDM failing to download the agent. We will update this issue as we know more.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We've identified the issue and are working on a fix. Customers experiencing this issue can re-try the registration attempt and the request should eventually be successful.

We are investigating increased errors with a service responsible for managing our token service which may cause issues for some users with registering JumpCloud Go. We will update this issue as we know more.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

The issue has been identified and a fix is being implemented.

We're currently investigating reports of degraded performance or intermittent synchronization issues with JumpCloud's Directory Services. We are investigating the cause of the issues currently, and will update the status event once resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue with the deployment of policies to macOS, Windows and Linux devices.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with propagation delays with Directory Insights events. We will update this issue as we know more.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're currently investigating reports of degraded performance or intermittent synchronization issues with JumpCloud's Directory Services. We are investigating the cause of the issues currently, and will update the status event once resolved.

This issue has been resolved and our Premium Chat Support is fully functional at this time.

A fix has been implemented and we are seeing recovery across our Premium Chat support platform.

We are currently experiencing issues with our Premium Support Chat tool. This is preventing us from replying to chat requests. We are working with the vendor to get this resolved as quickly as possible. Customers can still reach JumpCloud support via Premium Phone Support or by submitting a case via email or through the admin portal. Details on how to contact support can be found here; https://jumpcloud.com/support/contact-jumpcloud-support

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're investigating reports of degraded performance with JumpCloud's Insights reporting. The cause of the issue has been identified, and this status message will be updated once resolved.

This incident has been resolved.

We have identified and are working on an incident where a small subset of customers may experience errors sending MDM commands to devices, or retrieving results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We're currently investigating reports of degraded performance or intermittent connection issues on jumpcloud.com. We are investigating the cause of the issues currently, and will update the status event once resolved. Note that the JumpCloud Admin Console & User Portal are operational.

This issue has been resolved and our Premium Chat Support is fully functional at this time.

A fix has been implemented and we are seeing recovery across our Premium Chat support platform.

We are currently experiencing issues with our Premium Support Chat tool. This is preventing us from replying to chat requests. We are working with the vendor to get this resolved as quickly as possible. Customers can still reach JumpCloud support via Premium Phone Support or by submitting a case via email or through the admin portal. Details on how to contact support can be found here; https://jumpcloud.com/support/contact-jumpcloud-support

This incident has been resolved.

A fix has been implemented and we are monitoring the results. RADIUS server configuration propagation should catch up shortly.

The issue has been identified and a fix is being implemented.

We are investigating an issue with propagation delays specific to configuration changes with RADIUS servers. RADIUS authentication is not impacted.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently experiencing an increase in error rates when modifying and saving an existing software application. The creation of new applications appear unaffected at this time. We will update this incident as we know more.

This incident has been resolved.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with increased error rates for Password Manager. Password Manager may not be loading correctly in the Admin Portal, and end users may not be able to sync passwords or save new items. We will update this issue as soon as we have more information.

This incident has been resolved.

We've released a new version (v2.0.12.1) of the Password Manager Chrome Extension which should download shortly. If it doesn't happen automatically, customers can prompt updates by going to chrome://extensions, hitting "Developer Mode", and hitting the "Update" button.

The latest Chrome update may be conflicting with the JumpCloud Password Manager Chrome Extension on certain websites. If you are experiencing performance issues with Chrome, you may need to disable the Password Manager Chrome Extension, use Chrome in incognito mode, or use another browser until we are able to provide a solution for this issue. We are working on a fix and will provide an update as soon as possible.

The incident has been resolved. It's possible users may experience slower load times of the Admin Portal temporarily while the team investigates other opportunities around the change Chrome version 129 has introduced.

A fix has been implemented and we are monitoring the results.

We have identified the changes in Chrome to cause the issue with loading the Admin Portal and are working on implementing a fix for this.

We are currently investigating an issue with the latest Chrome release of version 129 causing failed logins to the Admin Portal. Firefox, Safari, and other browsers including Mobile App for Admin work successfully at this time. We will update this issue as we know more.

An intermittent connectivity issue with JumpCloud's Console Service for administrator and user login at https://console.jumpcloud.com has been addressed by our CDN vendor. All operations have returned to normal, and the issue is currently considered resolved.

CloudFlare, our CDN vendor, has applied a fix and the Admin console is fully operational now. We continue to monitor the situation.

We're currently seeing degraded performance or intermittent connectivity issues with JumpCloud's Console Service for administrator and user login at https://console.jumpcloud.com/. Degradation of performance of their CDN at the vendor has been identified to be the cause and we are waiting on updates from the vendor.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently seeing an increase in error rates on policies. Some customers may see an error "Unable to fetch policies!"

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are investigating an issue with propagation delays between the ADI agent and JumpCloud. We will update this issue when we have more information.

### **JumpCloud Incident Report** **Date**: 2024-08-23 **Date of Incident**: 2024-08-20 **Description**: RCA for JumpCloud Push Notifications ‌ **Summary:** On August 20th, at 3:46 AM Mountain Time, the APNs certificate attached to the AWS SNS Platform Application for Apple expired, causing Apple devices to fail to receive or acknowledge push notifications. Additionally, new device registrations began failing. Already registered devices could authenticate using the TOTP code in the JumpCloud Protect app. The on-call team was paged, and the engineer started investigating. An internal incident was called at 4:31 AM, and responders from different teams began triaging MFA failures. The team identified the errors from SNS and escalated for additional support at 4:53 AM. At 5:25 AM, the team attempted a manual rotation of the certificate in SNS, but due to access restrictions, IaC configuration was needed. This discovery and limitation delayed the overall recovery window. By 5:59 AM, the Notification Service had uptaken the new APNs certificate, and a rolling restart was initiated. The APNs certificate was updated in the AWS SNS Platform Application by 6:10 AM, and push notifications began working again. ‌ **Root Cause:** The root cause of the incident was the expiration of the APNs certificate. This led to the failure of Apple devices to receive or acknowledge push notifications and the failure of new device registrations. ‌ **Corrective and Preventative Actions:** **Immediate Corrective Actions:** * Manual rotation of the expired APNs certificate in the JumpCloud Notification Service. * Update of the APNs certificate in AWS SNS Platform Application. **Preventative Actions:** * Implement monitoring to alert the team well in advance of certificate expiration dates for SNS Platform Applications. * Review all critical certificates to ensure monitoring coverage. * Automate the certificate renewal process where possible to reduce the risk of human error. * Conduct training sessions for the team on handling certificate expirations and renewals.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're currently investigating reports of degraded performance or intermittent connectivity issues with JumpCloud's PUSH Services on iOS.

This incident has been resolved and this function available for use.

Creation of new Microsoft store applications is currently unavailable due to deprecation of service from Microsoft. Existing Microsoft store applications are unaffected and will continue to install/update. Further, existing JumpCloud managed Microsoft store applications may be edited and/or device associations changes and will continue to work as expected. For new Microsoft store applications, our team is actively working to support an alternative service and expect to have this available ASAP. Thank you for your patience as we work to restore this functionality.

Microsoft is currently experiencing issues with an endpoint used to install new applications using the Microsoft store app. We are working to regain access, and will update this issue as we know more.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some customers may experience provisioning or enrollment issues and see a generic "identity provider error". We've identified this issue and are in the process of implementing a fix. We will update this issue as we know more.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Launching Remote Assist in incognito mode or clearing the browser cache may be a temporary workaround for some customers.

We are currently investigating an issue with Remote Assist connections

A new version of the Admin app is available for both iOS and Android.

A new version of the Android Admin application is available for download which should solve this issue. The iOS version should be available for download shortly.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue with the JumpCloud iOS and Android Admin application.

**Date**: 2024-06-21 **Date of Incident**: 2024-06-05 **Description**: RCA for Google Apps Directory Dispatch Delay ‌ **Summary:** At approximately 20:30 MDT on 2024-06-05, JumpCloud received a large ingress of messages in a Google Apps FIFO queue.  By 21:40 MDT on 2024-06-05 this exceeded 20,000 messages causing processing delays for some customers.  At 21:24 on 2024-06-05 an alert fired for this delay, but was incorrectly labeled as a non-production issue, causing it to be misrouted.  At 07:15 MDT on 2024-06-06 the root cause was identified and a diversion queue was constructed where those messages were moved for processing.  The processing delay lasted until approximately 11:30 MDT on 2024-06-06. ‌ **Root Cause:** A contiguous block of messages larger than 50,000 from a single organization was sent to the main Google Apps FIFO queue.  Normally this would initiate a diversion queue construct prior to 20,000 messages, but was missed due to an incorrect alerting change during work in the Google Apps application layer. This FIFO queue is partitioned by organization ID, and once the limit of 20,000 messages is reached, only one message at a time can be processed regardless of the number of available worker threads. ‌ #### **Contributing Factors** * **Monitoring Gaps**: Lack of effective monitoring and alerting for this queue and processing latency. * **Scaling Issues**: Workers were scaled vertically but not timely enough to handle the sudden spike in queue size. * **Delay Building Diversion Queue**: Unforeseen issues requiring manual changes extending the time to completion for the diversion queue. #### **Immediate Actions Taken** 1. **Scaling Up**: Increased the number of worker instances to process the backlog of messages. 2. **Configuration Adjustments**: Reviewed and adjusted the FIFO queue and worker settings for optimal performance. 3. **Diversion Queue**: Diversion queue created for influx of messages. #### **Long-term Corrective Actions** 1. **Automate Message Diversion**: Review and optimize diversion queue policies ensuring rapid queue build. 2. **Monitoring and Alerts**: Review and increase comprehensive monitoring and alerting mechanisms for early detection of queue build-ups and processing delays. 3. **Documentation and Training**: Document the incident and update operational runbooks.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We have identified the issue and are working on a solution.

We have received reports that synchronization of user information between JumpCloud and Google Workspace is not occurring as expected. We are investigating these reports.

This incident has been resolved.

The issue has been identified and a fix implemented. We are continuing to monitor this issue.

We are continuing to investigate this issue and work towards a resolution.

The issue has been identified and a fix has been implemented.

We are investigating an issue with our cloud directory integration service.

This incident has been resolved.

We're currently investigating edge cases of failed logins on devices. We are investigating the cause of the issues currently, and will update the status event once resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing a delay in reporting the results of policies that have been applied to devices. This is under investigation.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are seeing an increase in error rates on JumpCloud's User Portal which may be causing login issues for some users. We will update this incident as we know more.

This issue has been resolved and our Premium Chat Support is fully functional at this time.

We are continuing to monitor for any further issues.

A fix has been implemented and we are seeing recovery across our Premium Chat support platform.

We have identified an issue with our Premium Chat support platform. Responses in chat will be slow. Customers are encouraged to submit tickets through the administration console. Details on alternate methods can be found in this Help Center article: https://jumpcloud.com/support/contact-jumpcloud-support

This issue has been fully resolved. We do not expect any additional interrupts in service.

Customers can still reach JumpCloud support via Premium Phone Support or by submitting a case via email or through the admin portal. Details on how to contact support can be found here; https://jumpcloud.com/support/contact-jumpcloud-support

We are currently experiencing issues with our Premium Support Chat tool. This is preventing us from replying to chat requests. We are working with the vendor to get this resolved as quickly as possible.

This incident has been resolved.

A fix has been implemented and we are monitoring results.

We are currently investigating an issue with the Password Manager module in the Admin Console.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the issue and a fix is being evaluated.

We continue to investigate the cause of the issue currently and will update the status event once resolved.

We are currently investigating an issue with the Admin console UI when pulling up existing user and device groups.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating increased error rates and failures to load the Admin Portal. We will update this incident as we know more.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are currently experiencing increased error rates with Password Manager backend systems. We will update this issue as we know more.