Historical record of incidents for CyberQP
Report: "System Wide Outage due to Google/GCP"
Last updateDue to the outage with Google GCP Services, many functions of our solution are currently inaccessible. We are closely monitoring the Google GCP Outage and will advise of the status as the GCP Services come back online.
Report: "Short Maintenance - Certificate Update"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
On Saturday May 31st, at 9 AM PST, CyberQP will be performing a quick maintenance update to change the certificate of our web interface.This should take less than 45 minutes to perform, and we'll advise when it has been completed.
Report: "JIT Enable Error - All Directory Sources"
Last updateThe issue affecting JIT account enablement (AD/Local/EntraID) via the Admin Dashboard and Passwordless authentication has been successfully resolved. All functionality is now fully restored. Please reach out if you encounter any further issues.
We’ve identified an issue from a recent release affecting JIT account enablement (AD/Local/EntraID) via the Admin Dashboard and Passwordless authentication. JIT account creation is unaffected, but enabling pre-existing JIT accounts is not working as expected. Our team is actively working on a fix, and we expect full functionality to be restored shortly.
Report: "Notification emails not delivered"
Last updateWe see the queue of backlogged emails has cleared. Please reply to this email if you are still seeing significant delays on email delivery.
We have started to see the queue of backlogged emails processing now. You should start to see emails arriving in your designated inbox. We will continue to monitor and advise when the entire backlog has been processed.
We have received reports regarding outbound emails from our solution being queued in our email delivery system. After thorough investigation, we have identified the cause of the issue in collaboration with our email delivery vendor. As a result, you may experience a bulk delivery of emails or receive notifications that were previously sent out. We apologize for any inconvenience this may cause and want to assure you that we are actively monitoring the situation. We will provide an update once the queue has been cleared. Thank you for your understanding and patience.
Report: "EU Admin Dashboard - Cannot login"
Last updateWith no further reports of login issues, this incident is being marked as Resolved.
We have isolated and resolved the cause of the login challenge. EU Logins now are working. Please reply to this email if you are still having challenges or open a ticket by sending an email to support@cyberqp.com
We have begun investigating the possible cause. This is only affecting the EU Admin Dashboard at this time.
Report: "FALSE POSITIVE - Agent update 5.3.6.0 detected as malware by Webroot - FALSE POSITIVE"
Last updateWebroot support indicates that they have whitelisted the Agent update for 5.3.6.0 - we are hopeful their support team will be willing to work with us for future Agent updates so that this doesn't happen again. Agent updates, that failed, may require the Agent to be reinstalled. You can simply install over top of the existing installation via your RMM, scripting process, or manually via the GUI.
We are aware that Agent version 5.3.6.0 update is being flagged as potentially malicious by Webroot. PLEASE NOTE: We can confirm this is a FALSE POSITIVE. If you are impacted, kindly submit this to Webroot as a FALSE POSITIVE. Our security and development teams are actively investigating the issue and working to address why this update was flagged. We’ll share updates as soon as more information becomes available.
A fix has been implemented and we are monitoring the results.
We are aware that Agent version 5.3.6.0 update is being flagged as potentially malicious by Webroot. PLEASE NOTE: We can confirm this is a FALSE POSITIVE. If you are impacted, kindly submit this to Webroot as a FALSE POSITIVE. Our security and development teams are actively investigating the issue and working to address why this update was flagged. We’ll share updates as soon as more information becomes available.
Report: "Dashboard Outage - Error when Accessing Customers"
Last updateThe resolution of the challenge for access to the Vault based Tenants has been resolved. The development team was able to quickly identify and remediate this challenge. Please do a full refresh of the Dashboard (CTRL - F5 on Chrome based browsers) and access your customers. If you are still seeing this error message, please reply to this email and a ticket will be created for you.
We are continuing to investigate this issue.
We are continuing to investigate this challenge. We have isolated this to just Tenant that use our Vault. All other Customers with ITGlue or Hudu integrations do not seem to be affected. This is the top priority for our team to get resolved.
We are seeing reports of an error message banner appearing when attempting to access the Customer List on the Dashboard. "Invalid `prisma.$executeRaw()` invocation: Authentication failed against database server at ..........etc." Our team has already begun the investigation and we'll update you ASAP on the resolution/fix.
Report: "Just in Time Creation - Error 3 INVALID_ARGUMENT"
Last updateDue to positive feedback from our MSP Partners, this incident will be closed. If you are still seeing any challenge related to this, please reply to this email and a ticket will be created for you.
A solution has been successfully implemented to resolve the issue preventing the creation of Just-in-Time (JIT) accounts. This fix has been deployed to the production environment, and active monitoring is in place to ensure continued performance and stability. Further updates will be provided as needed.
We have identified the cause of this challenge and the Development team is testing in our QA environment. Once we confirm this and release it to the public we'll send another update.
We are currently investigating an Error message that appears when creating new Just in Time (JIT) accounts. We will continue to update this incident with our progress on resolution.
Report: "Just-in-time Account Creation Issue"
Last updateThe team found the cause of the challenge and resolved it. The creation or enabling of a JIT account should be working for any affected MSP Partner now. Please respond to this email if you are still seeing this challenge in your environment.
Our team has identified an issue impacting the creation of Just-in-time Accounts from the web dashboard, desktop app, and QTech mobile app. We are currently working on this challenge with high priority.
Report: "Intermittent Web Dashboard Slowness"
Last update* **What happened?** * The API that powers our Dashboard experienced high latency in a few endpoints which caused corresponding pages to have high response times. * **Why did it happened?** * Over the past few weeks, we've had an influx of notifications from the agents that have slowed down the service that processes these notifications. The Dashboard is dependent on this service for communication to the agents which is why it slowed down. * **How we fixed it?** * Since then, we have pushed a few fixes to decouple the dashboard page loads from that notification service so that the dashboard does not rely on it and therefore loads quicker. We also pushed an optimization to the notification processing where we offload the heavy processing to a separate, more scalable and load balanced service so that notifications can be processed more efficiently. * **What are long term plans to make sure it doesn't happen again?** * We've added additional monitoring and alerting to critical components that will notify us when performance is degraded. We're continuing to improve the communication between our agent and our cloud services, including optimizing queries to our database and queuing heavy workloads to be processed timely and efficiently. We're improving our incident response process, so we are quicker and more responsive during these incidents.
After continued monitoring shows a return to normal operations for the Accounts related pages, this incident is resolved. We are aware of slowness on the Agents page, and our team will continue to release fixes to address this in the coming days. If you continue to receive error banners, please purge the Cache and Cookies for the Dashboard site, and relaunch the Browser and log in again. We've heard from a number of users that this resolved the error. A Post Mortem will be posted tomorrow for this incident. Please reply to this email if you are still experiencing slow responses on the Dashboard.
The CyberQP team worked overnight to implement some integral changes to the Dashboard loading process. These changes, coupled with additional changes that we will be implementing over the next 24 hours, should return the Dashboard to normal functionality. A postmortem will be posted with the final resolution of this incident. If you are still seeing slowness when loading the Accounts lists for your customers, taking longer than 30 seconds to load, please advise us via a support ticket (reply to this email to automatically create the ticket).
After a weekend of good results, Tuesday has resurfaced this challenge. Our team is working on this as quickly as possible and we'll continue to update everyone as we have more to share. We have seen early morning (6am to noon EST) seeing much better response times, however as the day progresses, the slowness returns.
We have made a significant changes to resolve this. We will be making additional changes and continue to monitor this over the weekend. Additional updates will be made Monday/Tuesday as we continue to improve the responsiveness of the Dashboard.
We have seen a return of the Dashboard slowness this afternoon and are continuing to monitor and work on a permanent resolution. We will continue to update the page as we have more details.
Our team has investigated the causes and will be monitoring today.
Our team is seeing intermittent slowness in the web dashboard (admin.getquickpass.com). We are actively investigating and will provide further updates as more information is made available.
Report: "Web Dashboard Slowness"
Last updateWith no additional reports of Dashboard slowness, this incident is marked as resolved. If you are still seeing Dashboard responsiveness challenges, please submit a ticket to support@cyberqp.com
Our team will continue to monitor this over the next few hours. Normal operational speed has resumed.
Our team is seeing intermittent slowness in the web dashboard (admin.getquickpass.com). We are actively investigating and will provide further updates as more information is made available.
Report: "Intermittent Web Dashboard Slowness"
Last updateAfter continuous monitoring, we are no longer seeing dashboard slowness. Pages are responsive and load in an appropriate amount of time. Please let us know via email response if you are still experiencing any challenges accessing the content of the Dashboard.
Our team continues to monitor the improved performance of the Dashboard. At this time the speed appears to have returned to normal, and we will continue to monitor the performance over the next few hours.
Our team is seeing intermittent slowness in the web dashboard (admin.getquickpass.com). We are actively investigating and will provide further updates as more information is made available.
Report: "Dashboard Slowness Navigating Accounts Inside Customer"
Last updateOn May 1st, we deployed a standard maintenance release that included a change to unify where we pull data from in our backend services. On May 2nd, during peak usage hours, we received reports of certain pages loading too slowly. At the same time, we noticed abnormal load on one of our services causing it to take a long time to respond to requests. We began troubleshooting and determined it was related to the unifying code we deployed and decided to revert the change. We're taking steps to mitigate this in the future by adding additional monitoring and alerting as well as adding steps in our design and test processes to better understand and forecast performance impacts.
Dashboard, ConnectWise POD, and AutoTask Insights are all responsive and performing as expected. We will continue to monitor. If you are still having challenges with Dashboard speed, or functionality, please respond or create a new ticket to support@cyberqp.com.
A fix has been put in place and dashboard response times are coming back to normal. We will continue to monitor the situation to ensure that the dashboard stays responsive. Another update will be posted when the work is completed.
We are aware of dashboard slowness and increased load times when navigating to accounts once clicked into a Customer. Our team is actively investigating and will update as more information is made available.
Report: "Web Dashboard Slowness"
Last updateDashboard speed and functionality has returned to normal. If you are still having challenges with a slow loading dashboard, please make sure your Customer's agents have updated to 4.1.3.0 and if you are still experiencing slowness, reach out to support@cyberqp.com.
We are continuing to monitor the Dashboard slowness. The Dashboard is not back to 100% performance but is functioning much better than previously. We will continue to make improvements today and will update when it is operating at normal performance.
As agent begin to update to the Hotfix release 4.1.1.1, and the Agents come back online, the Dashboard slowness will be reduced. We will continue to monitor this throughout the rest of the day and overnight.
Additional resources have been added to resolve this challenge. Some of the Dashboard slowness appears to be easing as the load is redistributed. We will continue to monitor for additional improvements and update further.
We are currently investigating reports of Web Dashboard slowness. We thank you for your patience and understanding as our team continues with our investigation. Updates will be provided as more information becomes available.
Report: "Agents Offline - Service running"
Last updateThe vast majority of Agents have updated and we're seeing consistent results with connections. If you are still having challenges with the Agent updates, please reach out to support@cyberqp.com .
Our team worked late into the night last night and has released Agent 4.1.3 as a resolution to the challenge seen yesterday. The majority of Customer agents have updated overnight but if an agent is still stuck at a previous version, please update manually. How to get the Agent Update Auto Update: No action required. The agent checks for new updates every two hours. Force Agent Update: Via RMM or other Management Tool force the Update Executable from this Path: "C:\Program Files\Quickpass Software\Quickpass Server Agent\AgentUpdater.exe" - This will need to be ran with Administrative privileges. Scripting/Direct Download Link: The agent download link is now updated and downloads via your script
We are pleased to inform you that a Fixed Agent has been released as a Hotfix. Agent Version 4.1.1.1 has been released. How to get the Agent Update Auto Update: No action required. The agent checks for new updates every two hours. Force Agent Update: Via RMM or other Management Tool force the Update Executable from this Path: "C:\Program Files\Quickpass Software\Quickpass Server Agent\AgentUpdater.exe" - This will need to be ran with Administrative privileges. Scripting/Direct Download Link: The agent download link is now updated and downloads via your script Post-Update Verification: After the update, there will be a period of time that the Agent still appears as offline. The Agent is designed to reconnect to the CyberQP infrastructure on a regular basis. As the queue of Agents that were on the previous version clears, the Agents will begin to appear as Online. Should you experience any further challenges, please do not hesitate to reach out to support@cyberqp.com .Thank you for your patience while we worked on a resolution and we do apologize for any inconvenience this may have caused. Regards, CyberQP Support
We have identified the issue and our team is currently working on a hotfix. We will send another update when the hotfix will be released today.
We are currently investigating an issue with Agents that have updated to 4.1.0.0 overnight. We're seeing behaviour where the Agent service is running on the system, however it is not communicating with the back end. Symptoms include: 1. Agent Showing offline on the Dashboard 2. Service on Server is running. 3. Event Viewer -> Applications and Services Logs -> Quickpass Events -> Log entries indicating that the Service is attempting to connect to server "waiting for authentication" or repeated "Disconnect from Server" and then attempts to connect again. As soon as we have an update from our internal teams we will update you with more information.
Report: "ConnectWise POD - 502 Error"
Last updateWith no further incidents reported this has been marked Resolved.
We have implemented a change to resolve this challenge. The POD appears to be loading from all tests ran. Please advise via a support ticket if you are still having challenges.
CyberQP is aware of a challenge some customers are seeing when loading the ConnectWise Manage POD. A 502 Error may be displayed or the POD may just "spin" attempting to load the page. Our team is investigating the cause and we will continue to update you here.
Report: "Agent Update Creating Login Issues for Partners"
Last updateWith no further reports of the Technician Login appearing on systems, we are moving this to resolved. If you do have any further challenge, please reach out to Support.
We are pleased to inform you that a Fixed Agent has been released as a Hotfix. How to get the Agent Update Auto Update: No action required. The agent checks for new updates every two hours. Force Agent Update: Navigate to Customer Name -> Agent -> Select the Checkbox for all Agents -> Select the Force Agent Update button. Scripting/Direct Download Link: The agent download link is now updated and downloads via your script Post-Update Verification After the agent has been updated, the “Technician Login” option should no longer present on the main Windows Login page. Should you experience any further challenges, please do not hesitate to reach out to support@cyberqp.com .Thank you for your patience while we worked on a resolution and we do apologize for any inconvenience this may have caused. Regards, CyberQP Support
The cause of the Technician Login appearing on the page has been isolated and an updated Agent is being built now to address this. An update to the Status page will be published once the new Agent has been released so that you can force the update.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
CyberQP is aware that a recent update to our agent is creating login issues for partners. We apologize for any disruptions, and have paused this agent release. CyberQP is preparing another update to remediate these issues for immediate release. If your users are having trouble logging in, please use the following PowerShell script to remove the Technician Sign-In tile. If you continue to experience login access problems, please contact CyberQP Support for assistance. End Users can select the "Other User" or their Account option on the left side of the sign in screen. Script: # Define the registry key path and key name $registryKeyPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Providers" $keyName = "{67D6B25B-3419-4C60-A4B5-A7CE535AD300}" # Check if the registry key exists if (Test-Path "$registryKeyPath\$keyName") { # Delete the registry key Remove-Item -Path "$registryKeyPath\$keyName" -Force Write-Host "Registry key '$keyName' deleted successfully from '$registryKeyPath'." } else { Write-Host "Registry key '$keyName' does not exist in '$registryKeyPath'." } Once again, we apologize for the inconvenience, and we’ll continue to update you as we continue to address these issues. Sincerely, The CyberQP Team
Report: "ITGlue Maintenance NA & EU Feb 3 8AM to 9AM PST"
Last updateITGlue's Scheduled Maintenance for North America and Europe may have affected rotations of accounts. Be aware that rotations scheduled between the hours of 8 AM PST and 9 AM PST on Feb 3rd, 2024 may have failed. Please schedule a manual rotation of affected accounts, or wait until the next scheduled rotation to ensure that password values are updated.
Report: "Error: CyberQP Technician Vault - Cannot read Properties of undefined "Access_Denied""
Last updateNo further reports of this have been submitted overnight.
We have released a fix for this challenge. We will continue to monitor the situation. Please report any additional challenges with viewing passwords in the CyberQP Technician Vault to support@cyberqp.com
We have identified the cause of this challenge and will be releasing a fix for it shortly.
We are investigating the cause of this error for Engineer and HelpDesk Login Roles (Technicians) within the Quickpass Technician Vault (formerly Password Vault). We will update with additional details as we have updates from our Development team.
Report: "Login to Dashboard - Error: Failed to Authenticate Token"
Last updateAfter monitoring, this challenge seems to be resolved. Logins to the Dashboard should be fully functional again. If you experience any challenges logging in, please report them to support@cyberqp.com.
The challenge seems to be resolved. We're investigating the root cause to prevent this from happening again.
We are aware of a challenge with logging into the Dashboard. Our team is working on the resolution now. Existing sessions with the Dashboard should continue to operate.
Report: "Performance Challenges"
Last updateFunctionality has been restored.
A fix has been implemented. We are aware of an ongoing challenge with re-enabling Azure Just in Time (JIT) accounts and we are continuing to diagnose. Source Sync, and Automatic import configuration should be operational now.
We have identified the cause of this challenge and are working to resolve this as quickly as possible. A further update will be provided when the fix has been put into place and we begin monitoring the resolution.
CyberQP is aware of some performance challenges with the Dashboard that may be presenting with multiple symptoms including error messages when Sync'ing with the Account Source, setting up new Automatic imports, and JIT Account activation. We are investigating the root cause. As soon as we know more about the cause we will update the incident.
Report: "Agent update to 3.6.2.0 Challenges"
Last updateAn Agent update will be released to resolve this challenge tonight. Please see this KB for resolution steps if your Customer was affected by the 3.6.2.0 agent update failure. https://support.getquickpass.com/hc/en-us/articles/19563292930327-Agent-3-6-2-0-Recall-Notice-Resolution-Steps The new Agent update version will be 3.6.5.0, which also includes some new functionality.
CyberQP is aware of some challenges with Customer Agents updating to the latest release 3.6.2.0. We have recalled this agent from the Automatic import process. We will be releasing a fix for this agent version very shortly. If you are having challenges with systems that have failed to update to 3.6.2.0 please reach out to support@cyberqp.com to get a ticket logged.
Report: "ITGlue has Notified of Challenge for the Australian Hosted Environment"
Last updateITGlue indicates this outage should be resolved.
IT Glue is continuing to monitor status for Australia and is seeing some improvement.
Logging in and Performance issues - IT Glue - Australia New incident: Investigating We are investigating performance and logging-in issues that may prevent access to IT Glue for some of our partners in our Australian Datacenter. You may see 403 Forbidden Error pages on certain pages in the app. We will provide further details here as soon as we can. Region: Australia Applications: Web App, Mobile App, Chrome Extension
Report: "QP Dashboard Degraded Performance"
Last updateWe have seen the QP dashboard performance return to normal. If you experience excessive slowness please contact support@cyberqp.com.
We are seeing improvement in the QP Dashboard performance. We will continue to monitor throughout the day.
We are currently investigating an issue where the QP Dashboard is displaying degraded performance. More information will be made available as the investigation continues.
Report: "ITGlue has a Scheduled Maintenance for the NA Hosted Environment"
Last updateAlthough no online updates were given by ITGlue the current status of their page indicates that their maintenance on the 16th of September was completed.
ITGlue has scheduled Maintenance for the Customers hosted in their North American Environment. Upcoming scheduled maintenance notice We will be undergoing planned database maintenance on September 16th, 2023, between 4:00 am - 5:00 am Eastern Time on our North American datacenter. We expect IT Glue and MyGlue to be unavailable for 30 mins during this time in the NA region. Be aware that Accounts scheduled for rotation during this Maintenance window will not execute. We will advise when ITGlue returns to normal operations.
Report: "Password rotation errors - Unable to get IT Glue password: Error On Integrations Service - connect ECONNREFUSED 10.52.2.149:8090"
Last updateThis has now been resolved and all ITGlue Rotations should continue as they previously did. If your Customer was affected by this incident, please ensure you rotate the passwords or wait for the next scheduled rotation.
We have resolved the cause of this integration error. We recommend manually rotating or resetting the passwords from the CyberQP Dashboard to ensure that the ITGLUE information is updated to a current password. We will continue to monitor the system.
We have seen a number of ITGlue Rotation errors with the message "Unable to get IT Glue password: Error On Integrations Service - connect ECONNREFUSED 10.52.2.149:8090" over the weekend and this morning. We are investigating the cause. We will update you when we know more about the cause of this error.
Report: "ITGlue North America - Outage"
Last updateITGlue is confident this issue has been resolved.
ITGlue has advised that they believe this challenge is resolved and they are working to monitor the situation.
ITGlue has advised of an outage that may affect Quickpass Account rotations. "You may see error pages on certain pages in the app. Users will be getting a 502 Bad Gateway error when accessing the application. " We will update as soon as ITGlue advises they are back online.
Report: "ConnectWise Manage - Cloud and Self Hosted Outage"
Last updateThe ConnectWise Manage outage seems to be resolved. Tickets should display the POD and Matching should be returned to normal operations.
ConnectWise Manage is currently reporting an outage. It is affecting Quickpass Customers that may be attempting to match customers, create or match ConnectWise Manage Contacts, or using the Quickpass POD in ConnectWise Manage Service Tickets. We will continue to monitor CW's outage status and update when we know more.
Report: "General Dashboard Slowness"
Last updateBased upon current performance and reports from customers this has been resolved. Please report any challenges with the Dashboard loading the list of Customers or List of Accounts inside the customer to support@getquickpass.com
Dashboard loading times have improved but a full investigation will be ongoing. Additional updates will be provided as the investigation continues.
Our team is continuing to investigate the root cause of this. Additional updates will be supplied as we are closer to a resolution.
Tenants may experience general slowness when loading the Customer list page or Accounts list for a Customer. A refresh of the page may load the information. We are investigating the cause of the slow down and will update the Status page as we have additional details.
Report: "ITGlue Customer Matching - Filter Results not displaying"
Last updateBased upon testing and customer reports this has been resolved.
The Hotfix has been released. We will continue to monitor the situation.
An additional Hotfix will be released 5/18/2023 at 5PM PST to resolve this. We will continue to monitor with affected Tenants.
Our team is continuing to investigate the root cause of this. Additional updates will be supplied as we are closer to a resolution.
We are investigating a Challenge with matching Customers from the ITGlue Integration Page. We have identified the cause and are working on releasing a Hot Fix to resolve this.
Report: "ITGlue has a Advised of a Performance Issue for North America Tenants"
Last updateITGlue indicates that this should be fully resolved. Functionality with Quickpass should return to normal operations.
A fix has been implemented and ITGlue is currently monitoring results. Users should be able to access the app again without errors.
ITGlue is investigating performance issues that may prevent access to IT Glue for some of our partners. You may see error pages on certain pages in the app. We will provide further details here as soon as we can. Region: North America Applications: Web App, Mobile App, Chrome Extension
Report: "ITGlue has a Scheduled Maintenance for the EU Hosted Environment"
Last updateITGlue advises that the European Host maintenance has been completed. Rotations scheduled to occur after this point should work as desired.
The issue has been identified and a fix is being implemented.
ITGlue has scheduled Maintenance for the Customers hosted in their EU Environment. April 18th, 2023 between 8:00 pm - 10:00 pm GMT Be aware that Accounts scheduled for rotation during this Maintenance window will not execute. We will advise when ITGlue returns to normal operations.
Report: "ITGlue has a Scheduled Maintenance for the NA Hosted Environment"
Last updateITGlue indicates that the maintenance has been completed. All updates scheduled from this point on should rotate as desired.
ITGlue has scheduled Maintenance for the Customers hosted in their North American Environment. April 18th, 2023 between 5:00 am - 7:30 am PDT Be aware that Accounts scheduled for rotation during this Maintenance window will not execute. We will advise when ITGlue returns to normal operations.
Report: "ITGlue has a Scheduled Maintenance for the Australian Hosted Environment"
Last updateITGlue advises that the Australian Hosted Environment should be fully operational now. Any scheduled rotation occurring after this point should proceed.
The Maintenance has started.
ITGlue has scheduled Maintenance for the Customers hosted in their Australian Environment. April 17th, 2023 between 11:00 pm - April 18th, 2023 1:00 am AEST Be aware that Accounts scheduled for rotation during this Maintenance window will not execute. We will advise when ITGlue returns to normal operations.
Report: "ITGlue has a Scheduled Maintenance for the NA Hosted Environment"
Last updateITGlue reports the maintenance has been completed.
We are continuing to investigate this issue.
ITGlue has scheduled Maintenance for the Customers hosted in their North American Environment. April 13th, 2023 between 7:00 pm - 9:00 pm PDT Be aware that Accounts scheduled for rotation during this Maintenance window may not execute. We will advise when ITGlue returns to normal operations.
Report: "ITGlue EU Host - Scheduled Maintenance"
Last updateMaintenance complete - ITGlue EU Hosts. The scheduled maintenance has been completed. Any rotations scheduled after this should completed properly.
ITGlue Hosts in Europe are undergoing maintenance 8:00 pm GMT - 10:00 pm GMT. Quickpass accounts scheduled for rotation during this maintenance will fail.
Report: "Australian Hosted ITGlue Maintenance"
Last updateThis incident has been resolved.
ITGlue Reports that the Australian host Maintenance is now complete. Any scheduled rotations from now on should work as expected.
ITGlue will be starting a Maintenance window for their Australian Hosted Environment, March 31st, 2023, 2:00 am - 4:00 am AEDT. Password rotations scheduled during this maintenance window may not run as scheduled.
Report: "IT Glue Outage"
Last updateIT Glue is now back to normal service. Thank you for your attention.
Please note: IT Glue API outage is currently causing issues with password rotation, as currently IT Glue is in read-only mode. We're monitoring this and will post updates as they are presented.
Report: "Minor Agent Disconnection"
Last updateThis incident has been resolved. After investigating, we've discovered that the issue was caused by the password rotation retry mechanism, after an offline agent came back online. We have temporarily disabled this feature while we work on an update to ensure the retry mechanism does not cause issues with agent connections.
Thank you for your patience - we've resolved the agent connectivity issue. We have identified an issue causing the problem and have deployed a hotfix. We are continuing to monitor to ensure the hotfix has resolved the issue.
We're still investigating this issue. Thank you for your patience.
We are currently investigating intermittent disconnections affecting some customers. Our team is on it and investigating the root cause currently. Apologies in the inconvenience this is causing your workflow.
Report: "Minor Performance Issues"
Last updateThank you for your patience. We've resolved the issue and the dashboard is back to normal.
We are continuing to investigate this issue. Stay tuned as we post regular updates.
Our dashboard is experiencing minor slowdown during normal usage. Our team is on it and investigating the root cause currently. Apologies in the inconvenience this is causing your workflow.
Report: "Minor Performance Issues"
Last updateWe’ve pinpointed the root cause of the performance issue. We’ve added additional infrastructure resources, and after implementing these changes, all North America clients were back to normal performance. Apologies for the inconvenience this has caused.
Thank you for your patience. A fix has been implemented and we're back to normal!
We've identified the root cause, and a fix is in progress.
We are continuing to investigate this issue. More updates to come.
We are continuing to investigate this issue.
Our dashboard is experiencing minor slowdown upon login and usage. Our team is on it and investigating the root cause. Apologies in the inconvenience this is causing your workflow.
Report: "Minor outage - Database slowdown"
Last updateAn access issue with the Quickpass MongoDB database caused a slowdown with the platform. The incident was reported at 6:54 am PST. The platform team found the root cause, and a fix was deployed to production at 7:54 am PST. Some partners may have reported a slowdown in usage of the platform. Internal platform team is developing an action plan to mitigate this type of outage in the future.
Report: "IT Glue Outage"
Last updateIT Glue API outage caused issue with password rotations and matching with IT Glue objects.