Historical record of incidents for DrChrono
Report: "Electronic Prescription Generation Issue"
Last updateWe are currently investigating an issue affecting the ability to generate refill prescription data in the e-Rx ordering screen in DrChrono. Our team is actively working with engineering to identify the root cause and resolve the problem as quickly as possible. Prescriptions are still being sent successfully, but you may need to re-enter the medication information in the e-Rx ordering screen. We understand the urgency of this functionality and will provide updates here as we learn more. Thank you for your patience.
Report: "Appointment Reminders Issue"
Last updateWe have successfully transmitted all email appointment reminders originally scheduled for delivery from 05/23/2025-05/26/2025 for appointments with dates of service 05/28/2025 and later. It was determined that we are unable to re-send SMS reminders from this time period. This issue is now resolved and our team confirms that all reminders are transmitting without issue since 05/27/2025.
We have successfully deployed a fix for the reminder transmission issue. Our team was also able to successfully send out all reminders for future appointments that were originally scheduled to be delivered 05/27/2025. All future appointment reminders will transmit without issue. We are still working on identifying appointment reminders for future appointments originally scheduled for delivery 05/23/2025-05/26/2025. We will continue to provide updates on the previous reminders here.
Our team has identified an issue that has prevented SMS, Email, and Phone Call appointment reminders scheduled to be sent from 05/23/2025-05/27/2025 from being delivered successfully. Email invitations to telehealth appointments are not affected. We are implementing a fix to ensure future appointment reminders will be delivered successfully and exploring the possibility of re-sending failed reminders scheduled to send from 05/23/2025-05/27/2025 for future appointment dates. We will continue to provide updates here.
The appointment reminder issue has been identified and a fix is being implemented. We will provide updates as they are available.
Our team is investigating an issue with appointment reminders. We will provide updates as they are available.
Report: "Appointment Reminders Issue"
Last updateWe have successfully transmitted all email appointment reminders originally scheduled for delivery from 05/23/2025-05/26/2025 for appointments with dates of service 05/28/2025 and later. It was determined that we are unable to re-send SMS reminders from this time period. This issue is now resolved and our team confirms that all reminders are transmitting without issue since 05/27/2025.
We have successfully deployed a fix for the reminder transmission issue. Our team was also able to successfully send out all reminders for future appointments that were originally scheduled to be delivered 05/27/2025. All future appointment reminders will transmit without issue. We are still working on identifying appointment reminders for future appointments originally scheduled for delivery 05/23/2025-05/26/2025. We will continue to provide updates on the previous reminders here.
Our team has identified an issue that has prevented SMS, Email, and Phone Call appointment reminders scheduled to be sent from 05/23/2025-05/27/2025 from being delivered successfully. Email invitations to telehealth appointments are not affected. We are implementing a fix to ensure future appointment reminders will be delivered successfully and exploring the possibility of re-sending failed reminders scheduled to send from 05/23/2025-05/27/2025 for future appointment dates. We will continue to provide updates here.
The appointment reminder issue has been identified and a fix is being implemented. We will provide updates as they are available.
Our team is investigating an issue with appointment reminders. We will provide updates as they are available.
Report: "Billing Tab Loading Issues- DrChrono"
Last updateThe issue with the billing tab has been resolved. We are sorry for any issues this may have caused.
Users who are not currently utilizing our Support Chat functionality are experiencing loading issues with items in the Billing menu. Our team is working on resolving this issue urgently. Please reach out to our Support Team if you are experiencing this issue.
We are aware that some users are experiencing issues with the billing tab not loading properly. Our team is actively investigating the situation and working closely with our engineering team to resolve this as quickly as possible. We will continue to update this page with updates.
Report: "OnPatient Login Issues with DrChrono"
Last updateThis incident has been resolved.
We are pleased to inform you that a fix has been implemented affecting users logging into OnPatient via DrChrono. Our team is continuing to monitor the situation closely to ensure everything remains stable.
We are currently aware of issues affecting users attempting to log into OnPatient via DrChrono. Our team is actively investigating the situation and working to resolve it as quickly as possible. We appreciate your patience and understanding as we work on resolving this. We will provide updates as soon as we have more information.
Report: "DrChrono Intermittent Issues"
Last updateWe have implemented a fix for the outage and the issue is resolved. We are sorry for the inconvenience. Please reach out to support for further questions.
We are continuing to investigate this issue.
DrChrono is experiencing a disruption with faxing and sending lab orders. We are investigating the issue and will continue to update this page. Please reach out to support for further questions.
Report: "OnPatient"
Last updateThis incident has been resolved.
A fix for this issue has been implemented and our team is monitoring the results. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding.
Our team is investigating reports of an issue with OnPatient, which may prevent a patient from logging into OnPatient or joining a telehealth visit. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding.
Report: "DrChrono Documents"
Last updateThis incident has been resolved.
Identified: We have identified an issue with documents displayed in the patient chart. We have identified a fix and are working towards resolving this issue.
Monitoring: We have identified an issue with documents displayed in the patient chart. We have implemented a fix and are monitoring the results.
Report: "OnPatient"
Last updateThis incident has been resolved.
A fix for this issue has been implemented and our team is monitoring the results.
Our team is continuing to investigate reports of an issue with OnPatient, which may prevent a patient from logging into OnPatient or joining a telehealth visit. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding.
Our team is investigating reports of an issue with OnPatient, which may prevent a patient from logging into OnPatient or joining a telehealth visit. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding.
Report: "Third Party Vendor: ID.me"
Last updateThis incident has been resolved.
ID.me, our third-party vendor, has resolved their incident. Thank you for your patience and understanding.
ID.me, our third-party vendor, has implemented a fix and is continuing to monitor their system. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding. Please follow their status page for more information: https://status.id.me/
ID.me, our third-party vendor, is continuing to work on a fix to address their ongoing platform degradation. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding. Please follow their status page for more information: https://status.id.me/
ID.me, our third-party vendor, has identified an issue within their system and is actively working on deploying a fix. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding. Please follow their status page for more information: https://status.id.me/
ID.me, our third-party vendor, has identified an issue within their system and is actively working on deploying a fix. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding. Please follow their status page for more information: https://status.id.me/
Report: "Third Party Vendor: Updox"
Last updateUpdox, our third-party vendor, has resolved their incident. Thank you for your patience. Updox Status Page: https://status.updox.com/
Updox, our third-party vendor, has implemented a fix for the disruption in service, and the issue is resolved. All systems are operational at this time. They will continue to monitor. If there are no additional occurrences, the status of this incident will be moved to resolved. Thank you for your patience. Updox Status Page: https://status.updox.com/
Updox, our third-party vendor, faxing, has reported a service disruption that may affect fax confirmation pages, uploading (drag and drop), and virtual printing. We will continue to provide updates as we have more information. Thank you for your patience. Updox Status Page: https://status.updox.com/
Updox, our third-party vendor, faxing, has reported a service disruption that may affect fax confirmation pages, uploading (drag and drop), and virtual printing. We will continue to provide updates as we have more information. Thank you for your patience. Updox Status Page: https://status.updox.com/
Report: "Third Party Vendor: Updox"
Last updateUpdox, our third-party vendor, has been restored. Please note, that data will temporarily be limited to the past 30 days. The system will continue to process historical data throughout the night reaching full restoration by tomorrow morning. Inbound faxes were queued throughout the day and will be processed overnight. Please see their Status Page for more information: https://status.updox.com/
Updox, our third-party vendor has taken their application offline at 4:30 PM (Eastern) to perform emergency maintenance. The application will be down for approximately 1 hour. Once the application is restored we will post an update. Updox Status Page: https://status.updox.com/
Updox, our third-party vendor, has identified the root cause of the issue and is actively working to resolve it. We will continue to provide updates as we have more information. Thank you for your patience. Updox Status Page: https://status.updox.com/
Updox, our third-party vendor, has identified the root cause of the issue and is actively working to resolve it. We will continue to provide updates as we have more information. Thank you for your patience. Updox Status Page: https://status.updox.com/
Updox, our third-party vendor, faxing, has reported a service disruption that may affect fax confirmation pages, uploading (drag and drop), and virtual printing. We will continue to provide updates as we have more information. Thank you for your patience. Updox Status Page: https://status.updox.com/
Report: "Third Party Vendor: Quest Diagnostics"
Last updateQuest, our third-party vendor, has resolved the issue that may have impacted lab ordering and results transmission. If you are still experiencing any lingering effects, please don't hesitate to reach out to Support at support@drchrono.com for assistance.
Quest, our third-party vendor, is still investigating their system issues that may impact lab ordering and results transmission. We will continue to use this page to monitor the status. In the meantime, missing lab results can be accessed through the Quest Quanum portal: https://auth2.questdiagnostics.com/cas/login?service=https%3A%2F%2Fphysician.quanum.questdiagnostics.com%2Fhcp-server-web%2Flogin%2Fcas
Quest, our third-party vendor, has reported system issues that may impact lab ordering and results transmission. We will continue to use this page to monitor the status. In the meantime, missing lab results can be accessed through the Quest Quanum portal: https://auth2.questdiagnostics.com/cas/login?service=https%3A%2F%2Fphysician.quanum.questdiagnostics.com%2Fhcp-server-web%2Flogin%2Fcas
Report: "System Degradation"
Last updateWe have worked with our cloud provider to resolve the intermittent performance degradation issues experienced over the last few days. We are now moving this to solved. We are sorry for any issues this may have caused.
We have worked with our cloud provider to resolve the intermittent performance degradation issues experienced over the last few days. We will continue monitoring for another hour before marking this incident fully resolved.
We are aware that some users may be experiencing short, transient periods of system degradation every few hours that began yesterday. Our team has worked with our hosting provider to implement some changes overnight and we continue to partner with them to make additional changes to continue to improve performance.
We will be performing an emergency maintenance tonight at 11:30 PM EST. The Dr Chrono application will be offline for 5-10 min.
We are aware that some users may be experiencing short, transient periods of system degradation every few hours that began yesterday. Our team has worked with our hosting provider to implement some changes overnight and we continue to partner with them to make additional changes to continue to improve performance.
We are aware that some users may be experiencing short, transient periods of system degradation every few hours that began yesterday. Our team has worked with our hosting provider to implement some changes overnight and we continue to partner with them to make additional changes to continue to improve performance.
Report: "Third Party Vendor- Square"
Last updateThe issue with the payment processor, Square, has been resolved. Thank you for your patience and understanding.
Our team has implemented a fix for this issue and our teams are monitoring the results. We will provide another update once we have confirmed this was successful in resolving the issue.
We are aware of the issues you are experiencing with the payment processor, Square. We are working with our internal teams on this and will continue to use this page to provide updates.
Report: "Slowness and/or Errors"
Last updateWe have seen no additional occurrences of this issue. We are transitioning this issue from monitoring to resolved.
We continue to monitor the issue within our third party hosting provider that caused errors and intermittent slowness. This issue lasted between 11:45am ET and 12:45pm ET. We have seen no recurrence since this time frame. Our system health, response times and experience continue to be normal. We are working with our hosting provider to understand the root cause and permanent remediation. We are sorry for any inconveniences this has caused.
We have implemented a fix for the slowness and/or error issue within DrChrono. We will continue to monitor along with our third-party hosting provider. We will continue to provide updates every 45 minutes. Thank you for your patience.
Our team continues to investigate the slowness and/or errors within DrChrono. We are working on a resolution with our third-party hosting provider. We will continue to provide updates every 45 minutes. Thank you for your patience.
Our team continues to investigate the slowness and/or errors within DrChrono. We are working on a resolution with our third-party hosting provider. We will continue to provide updates every 45 minutes. Thank you for your patience.
Our team continues to investigate the slowness and/or errors within DrChrono. We are working on a resolution with our third-party hosting provider. We will continue to provide updates every 45 minutes. Thank you for your patience.
DrChrono has identified that some users may experience slowness and/or errors within the application. We are investigating the issue and will post updates every 45 minutes.
Report: "System Slowness"
Last updateOur team has successfully deployed a resolution, and response times have returned to normal. The issue is resolved, and all affected services are fully operational. If you experience any further issues, please contact our support team.
We have completed deployment of the additional optimization. Our response times have returned to normal. We will continue to monitor our site performance for the next 60 minutes, and we will resolve this issue once we confirm stability is persisting.
Our team has identified an optimization we can make on our side that will help mitigate the slightly elevated response times a subset of users may be experiencing relating to our third party host's affected database. We are completing testing on this change and will deploy it once tested.
We have determined this slowness has been caused by an issue with our third party hosting service provider. Our team is coordinating with them to restore site stability fully and we will update again once we have confirmation the site performance has stabilized.
Our team has identified the issue and has deployed temporary mitigations. We believe the experience for most users has improved to normal or only slightly elevated response times. We are working with our hosting provider to resolve the identified root cause and to bring resolution to remaining issues that a subset of users are experiencing. We will continue to update this status page.
DrChrono has identified that some users may be experiencing system slowness with slightly elevated response times. We are investigating the issue and will post updates every 45 minutes.
Report: "Clearinghouse Vendor- Change Healthcare Cyber Security Issue"
Last updateWe wanted to provide you with an update regarding our Everhealth clearinghouse partner, eProvider Solutions (ePS). They have informed us that they are closely monitoring Change Healthcare’s (CHC) ongoing efforts to restore connections with payers for claims and Electronic Remittance Advice (ERAs) that were exclusively connected to CHC. We're pleased to report that ePS has successfully reconnected 99% of the impacted payers for claims, with only 1% remaining. They are working diligently to restore all impacted ERAs as well. CHC data review is in its late stages as explained on https://www.unitedhealthgroup.com/ns/health-data-breach.html. Change Healthcare has begun mailing written notices via U.S. mail to affected people identified in the ongoing data review for whom Change Healthcare has a sufficient address. Please note, Change Healthcare may not have sufficient addresses for all affected individuals. The mailing process began on July 29 and will continue on a rolling basis. As CHC continues to work with industry experts to analyze the data involved in this cyberattack, they are offering immediate support and enhanced protections. For more information and resources, visit the CHC dedicated website: CHC Health Data Breach Information. Thank you for your continued partnership and cooperation during these challenging times.
Dear Valued Client, Change Healthcare is still experiencing effects from a cyber security issue that occurred on February 21, 2024, which prompted them to disconnect its systems from customers and partners. Recently we were notified that Change Healthcare will not restore its connection with DrChrono. At this time, we are transitioning our customers over to our new clearinghouse eProvider Solutions (EPS). While doing this migration, you will be able to submit to all payers who do not have a direct connection with Change Healthcare. This will also allow you to start receiving Electronic Remittance Agreements (ERAs) once the enrollments are completed. We are committed to keeping you informed throughout this process. We have put together a support article to help answer any questions you may have. If you have more questions, please contact support and reference EPS migration. Thank you for your continued trust and partnership as we navigate this challenge together. We remain fully committed to delivering the highest service and support to you, our valued clients. Sincerely, The DrChrono by EverHealth Team
It's important to understand that when Change Healthcare (CHC) announces they are "beginning to process claims” or have “reestablished connections” this does not mean that all claim platforms are available, previously services are fully available or connection to the previous set of payer exists. Please be assured we are actively collaborating with CHC and other industry partners to restore all services and connections as soon as they are available. To this end, we are in constant communication with the CHC team as they discuss their plans for reconnection. At present, we are awaiting an updated timeline and further instructions from CHC to initiate the reconnection process to their various platforms and payers that DrChrono uses. We understand that waiting for CHC to become available is frustrating. Therefore, if you wish to migrate over to our EverHealth Clearinghouse services, please email support with a subject line of MIGRATION TO EPS to assist you in your migration. Statement Update: DrChrono has been working on the backend to reestablish connection to mail your patient statements. We are happy to announce a temporary solution has been put in place to support your patient statement processing. To understand the patient statement solution, click here to view our article: https://support.drchrono.com/hc/en-us/articles/25180407552027-Patient-Statements-through-Optum
Dear Valued Client, We understand that staying updated on industry changes from Change Healthcare (CHC) can be challenging given their wide array of platforms and services. In addition to being one of the largest conglomerates of clearinghouses nationwide, CHC offers vital daily healthcare applications ranging from provider enrollment and patient statements to electronic prescriptions and claim adjudication. Many industry applications rely on several of CHC services, largely due to exclusive contracts CHC holds with many payers. Consequently, the impact on our customers varies depending on regional factors and each facility’s specific payer mix. This exclusivity means everyone must rely upon CHC for these specific payers and there are no alternative channels available. It's important to understand that when CHC announces they are "beginning to process claims” or have “reestablished connections” this does not mean that all claim platforms or previously available services are up and running. Please be assured that we are actively collaborating with CHC and other industry partners to restore all services and connections. Currently, we are on daily calls with the CHC team as they discuss their plans for reconnection. At this time, we are awaiting an updated timeline and further instructions from CHC to initiate the reconnection process to their platforms and products as they become available. We appreciate your ongoing patience and understanding during this period. We will continue to provide regular updates on potential impacts, along with the progress being made towards a full restoration. Sincerely, DrChrono Support
As we work to navigate the recovery from the Change Healthcare Services cyberattack and its impact on the Healthcare Ecosystem, we want to express our cautious optimism regarding the recovery process. Today, Change Healthcare published an update via a press release stating that critical system recovery is underway. Change Healthcare indicated that they expect to begin testing and reestablishing connectivity to their claim network and software starting the week of March 18th. While the restoration work continues, Change Healthcare encourages trading partners and providers to continue using the established workarounds. Since the initial communication of the Cyberattack DrChrono remains focused on maintaining the highest standards of service and security while minimizing the impact of this attack on your practice by creating alternate claim submission options. We will continue to act as your advocate and work on alternate solutions until our team has high confidence in the success of Change Healthcare's restored functionality. Once we are confident that Change Healthcare can properly support your practice’s needs, we will partner with you on the plan to move you back over to Change Healthcare as your clearinghouse. Attached is a link to the Change Healthcare press release. https://www.unitedhealthgroup.com/newsroom/2024/2024-03-07-uhg-update-change-healthcare-cyberattack.html If you would like to enquire about work around solutions, please submit a ticket that states "work around clearinghouse" to support@drchrono.com and we will assist you the best we can.
As we work to navigate the recovery from the Change Healthcare Services cyberattack and its impact on the Healthcare Ecosystem, we want to express our cautious optimism regarding the recovery process. Today, Change Healthcare published an update via a press release stating that critical system recovery is underway. Change Healthcare indicated that they expect to begin testing and reestablishing connectivity to their claim network and software starting the week of March 18th. While the restoration work continues, Change Healthcare encourages trading partners and providers to continue using the established workarounds. Since the initial communication of the Cyberattack DrChrono remains focused on maintaining the highest standards of service and security while minimizing the impact of this attack on your practice by creating alternate claim submission options. We will continue to act as your advocate and work on alternate solutions until our team has high confidence in the success of Change Healthcare's restored functionality. Once we are confident that Change Healthcare can properly support your practice’s needs, we will partner with you on the plan to move you back over to Change Healthcare as your clearinghouse. Attached is a link to the Change Healthcare press release. https://www.unitedhealthgroup.com/newsroom/2024/2024-03-07-uhg-update-change-healthcare-cyberattack.html Please reach out to support with further questions.
While DrChrono is not facing any issues at this time, our 3rd party vendor, Change Healthcare is having a cyber security issue which may result in issues with Eligibility, ERAs, Claim Submissions, and sending Electronic Prescriptions to certain pharmacies. The disruption is expected to last throughout the day. We will continue to update you as we have more information. Please reach out to support with further questions and workarounds.
While DrChrono is not facing any issues at this time, our 3rd party vendor, Change Healthcare is having a cyber security issue which may result in issues with Eligibility, ERAs, Claim Submissions, and sending Electronic Prescriptions to certain pharmacies. The disruption is expected to last throughout the day. We will continue to update you as we have more information. Please reach out to support with further questions.
While DrChrono is not facing any issues at this time, our 3rd party vendor, Change Healthcare is having a cyber security issue which may result in issues with Eligibility, ERAs, Claim Submissions, and sending Electronic Prescriptions to certain pharmacies. The disruption is expected to last throughout the day. We will continue to update you as we have more information.
While DrChrono is not facing any issues at this time our 3rd party vendor Change Healthcare, is experiencing a full outage which may result in issues with eligibility, prescriptions, ERAs, Patient Statements and Claims. We will continue to update you as we have more information. Change Healthcare has a status page that can be followed here: https://status.changehealthcare.com/
While DrChrono is not facing any issues at this time our 3rd party vendor Change Healthcare, is experiencing a full outage which may result in issues with eligibility, prescriptions, and ERAs. We will continue to update you as we have more information.
Report: "Vendor Issue: Quest Diagnostics"
Last updateWe have completed the restoration of the previous compendium and Quest lab ordering should be restored for affected users.
We have determined that there is an issue with the compendium files that we are receiving from Quest for certain directory locations. We are coordinating with Quest to correct this issue. We are also working to restore the previous compendium set for the problematic locations to bring relief until Quest corrects the files.
Our team is investigating reports of test codes not populating on Quest lab orders. We believe this may be a result of an error with file transmission from certain Quest directory locations to DrChrono. This issue is affecting only a subset of users who are attempting to order labs through Quest via DrChrono. We will continue to use this page to provide updates.
Report: "DrChrono System Slowness"
Last updateAs of 12am ET, our hosting provider resolved the issues experienced for several services within our primary region. As of 8:30 ET this morning, we verified their fix and removed workarounds that were temporarily put in place. After a period of monitoring, we can confirm users are experiencing normal response times. We are transitioning this from monitoring to considered resolved.
We have fully deployed our workaround to circumvent issues being experienced by a regional outage for several services within our hosting provider. Users should be experiencing normal response times. For users that are not, please contact support. We are continuing to monitor in case additional changes are necessary.
We have partially deployed the identified workaround to circumvent issues being experienced by a regional outage for several services within our hosting provider. Via internal testing, we can confirm that most users should be able to access the system and experience normal response times. We are working to finalize the deployment of the workaround and will update accordingly.
We have identified a workaround and are working to deploy the change to mitigate the errors we are experiencing due to our hosting provider. We will provide another update in 60 minutes.
We have identified the issue to be related to ongoing API issues with our hosting provider. We are attempting to identify a workaround while they resolve their issue in parallel. We will continue to update this page.
We are continuing to investigate this issue.
DrChrono has identified that some users may experience slowness within the application. We are investigating the issue and will post updates every 60 minutes.
Report: "Clinical Template Generative Text on Preview Note"
Last updateOur team has not identified any further issues with generative text from smart switch fields since the release of the change on Thursday night. We are transitioning from the monitoring phase to considering this issue as resolved.
We have deployed the fix for the issues related to generative text from smart switch fields and its inclusion in previewed notes and expect users to see resolution. We will continue to monitor tonight and tomorrow through a period of normal utilization of the platform during core hours of usage.
Testing of our correction in our non-production environment has proven successful. Our team is focusing on preparing for the deployment to our production environment. Estimated time to complete the production deployment is 60-90 minutes.
Testing of our correction in our non-production environment is progressing positively. We will be Sharing an expected resolution time in our next post.
We have developed a plan to correct this issue and we are testing our potential fix in a non-production environment. We will continue to provide updates as we progress through testing and communicate the expected resolution time as soon as that is available.
We are continuing work to implement a solution and will continue to provide updates as we have them. We appreciate your patience while we work to resolve this issue.
We have identified two issues that users may be experiencing. Issue #1: Generative text from Smart Switch fields and any child fields tied to the Smart Switch does not render in the clinical note preview. Issue #2: Generative text from switch fields that have been turned off persist in the clinical note preview.Affected users may also experience billing codes tied to these switch fields that persist in the clinical note preview when the switch is disabled. Our team is working diligently to make changes related to these issues, test thoroughly and release to end users. We will provide additional updates as progress is made.
We are investigating user reports of changes made in the clinical charting template not generating the expected text on the clinical note preview. Our team understands the impact this poses to affected users and we are approaching this with the highest priority. We will provide an additional update as soon as possible.
Report: "Vendor Outage Quest Diagnostics"
Last updateQuest has resolved their issues. We are sorry for any issues this may have caused.
Quest Diagnostics has confirmed they are experiencing issues on their end. Our team is working with them to restore functionality. This will affect your ability to send and receive orders. We will use this page to provide updates.
Report: "CrowdStrike Global Outage and Support"
Last updateMicrosoft systems third-party security vendor, Crowdstrike has communicated that the global issue is resolved. We will continue to monitor. https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
There is an ongoing widespread and global issue affecting Microsoft systems triggered by a third-party security vendor. This situation could affect certain system availability related to our vendors and trading partners. We are actively monitoring the global issue and will provide updates as available.
There is an ongoing widespread and global issue affecting Microsoft systems triggered by a third-party security vendor. This situation could affect certain system availability related to our vendors and trading partners. We are actively monitoring the global issue and will provide updates as available.
Report: "Vendor Outage Quest Diagnostics"
Last updateWe have been notified by Quest Diagnostics that this issue has been resolved. We are sorry for any issues this may have caused you.
We have been made aware from our third party vendor, Quest that they are having system issues. This will effect lab orders, as well as receiving results. We will continue to use this page to monitor the status.
Report: "Partner ID.me Outage"
Last updateWe have been made aware that the issues with ID.me have been resolved. We are sorry for any issues this may have caused.
Our vendor, ID.me, has implemented a fix and is currently monitoring the results. The ability to send prescriptions may still be affected as ID.me services are restored. You can monitor their progress at https://status.id.me/. We will continue to monitor this until ID.me is fully restored.
We have been notified by our vendor ID.me that they are currently experiencing a service outage. This will effect your ability to send prescriptions. You can follow their status page here: https://status.id.me/. We will continue to monitor this page with updates.
Report: "OnPatient Issues"
Last updateOnPatient issues have been resolved. We are sorry for any issues this may have caused.
We have implemented a fix. Please note that it may take 15-30 minutes for the fix to take effect for some users. We will continue to monitor the situation until it is fully resolved. We will continue to update this status page.
Our team is aware of an issue with OnPatient. We are working diligently to resolve this issue. We will continue to use this page to update the status of our progress.
Report: "DrChrono Payments"
Last updateDrChrono has resolved the issue around payments. We are sorry for any issues this may have caused. Please reach out to support with any questions.
We are aware that some users may be experiencing an issue with their DrChrono Payments accounts appearing unlinked within DrChrono in the account settings. The back-end integrity of the connection between impacted users' accounts and DrChrono Payments is not affected, however, users may only see the "Enroll with DrChrono Payments Now" button in the Patient Payments set-up screen. This may impact the ability to successfully run patient payments for DrChrono Payments users. Our team is investigating and we will update you as soon as we have further information
Report: "Quest Lab Results"
Last updateThe DrChrono team has resolved issues with Quest Labs. We are sorry for any inconveniences this caused. If you have any further questions please reach out to support.
The issue has been identified and a fix is being implemented.
We are currently experiencing an issue importing lab results from Quest Labs. Our engineering team is aware of the issue and is actively working towards a resolution in partnership with Quest Labs. Lab result delivery may be delayed until the issue is resolved.
Report: "DrChrono Data Not Loading"
Last updateThe incident causing a page to partially render and or look blank has been resolved by our teams. We are sorry for any issues this may have caused.
A subset of users are experiencing issues in the webapp, which we believe are related to the browser version, that may cause a page to partially render and or look blank. If you are experiencing this situation please submit a support ticket below and a member of support will be in contact. The system is otherwise operating normally. If you experience this issue with Google Chrome, as a temporary workaround please access the DrChrono webapp using another browser including Firefox, Safari, and Microsoft Edge. We are investigating the issue and will post updates every 2 hours.
DrChrono is experiencing a disruption in service. Users may be experiencing data not loading in multiple parts of the platform. We recommend using a different browser (Firefox, Safari, or Edge) as this issue is mainly affecting Google Chrome. We are investigating the issue and will post updates every 2 hours.
DrChrono is experiencing a disruption in service. Users may be experiencing data not loading in multiple parts of the platform. We recommend using a different browser (Firefox, Safari, or Edge) as this issue is mainly affecting Google Chrome. We are investigating the issue and will post updates every 60 minutes.
DrChrono is experiencing a disruption in service. Users may be experiencing data not loading in multiple parts of the platform. We recommend using a different browser (Firefox, Safari, or Edge) as this issue is mainly affecting Google Chrome. We are investigating the issue and will post updates every 60 minutes.
Our team continues to investigate the service interruption with data not loading and we will continue to provide updates every 60 minutes. Thank you for your patience.
DrChrono is experiencing a disruption in service. Users may be experiencing data not loading in multiple parts of the platform. We recommend using a different browser (Firefox, Safari, or Edge) as this issue is mainly affecting Google Chrome. We are investigating the issue and will post updates every 60 minutes.
Report: "Third Party Vendor-Updox"
Last updateUpdox has notified us that the issue has been resolved. Please reach out to support if you still see issues with faxing.
Our 3rd party vendor, Updox is experiencing a disruption in service. Users may see slowness and errors when attempting to fax or when receiving faxes. We will continue to use this page to update you. We are sorry for any issues this may cause.
Report: "Vendor-Quest Labs"
Last updateQuest has resolved their outage. We are sorry for any issues this may have caused.
We are continuing to investigate this issue.
Quest, our lab vendor, is currently experiencing an outage that may affect lab results and lab orders. We will continue to use this page to monitor as we learn more from Quest. We apologize for any issues this may cause.
Report: "DrC Slowness"
Last updateOur team has resolved the issues related to system slowness. Please reach out to support if you continue to experience issues. We are sorry for the inconvenience this may have caused.
Our engineers have pushed a resolution and are monitoring the system at this time. We will move the status of this incident to resolve in the next 30 minutes.
DrChrono is monitoring the system slowness.
DrChrono users may be experiencing slowness. We are working with our engineering team on a resolution. This page will be updated every hour for updates.
Report: "Appointment Reminder Issue"
Last updateOur team believes they have fixed the issue with appointment reminders. We are sorry for any inconveniences this has caused. Please reach out to support if you continue to experience issues.
Our team has delivered a fix overnight for newly created appointments and is finishing repairing previously created appointments. This final step is anticipated to complete later today. We will continue to use this page to update you.
The appointment reminders issue has been identified and engineering is working on a fix. We will continue to update this page.
We have received several reports of appointment reminders not being sent. Our engineering team is currently investigating this issue. We will continue to use this page to update you of the status.
Report: "EHR iOS Application"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our team is currently investigating an issue updating the status of an appointment or utilizing appointment filters via the EHR iOS application. At this time, some users may receive a 500 error or a blinking spinning wheel when attempting to update appointment status or use appointment filters. Please note that the status can be updated successfully via the website as a workaround.
Report: "Reports of Workflow Errors"
Last updateDrChrono believes the issues have been resolved. Please reach out to support if you are still having issues.
We are investigating reports of workflow errors throughout the system. We will continue to update this status page every 45 min. We are sorry for any issues this may be causing.
Report: "DrChrono system slowness"
Last updateAt 2:15 PM ET on December 18th, a system hosted by our cloud provider became degraded and intermittently unavailable. This caused the capacity we had to serve requests to degrade. We had to pause application traffic and place a maintenance page to allow the system to recover. At 4:45 PM ET, we lifted the maintenance page, and the system returned to health. We saw traffic levels and background processing return to normal levels. The failing system in question is the same system that resulted in other degradations and outages in the last 90 days. In every case, the outage was not caused by a recent DrChrono release but by hardware failure occurring on an ad-hoc basis. Since the October 18th outage, the team has been working as quickly as possible to patch the system to prevent recurrence and, in parallel, to replace this system entirely. In late November, we fixed the system in place with the understanding and expectation that it would remove the memory leak that is the cause of degradation via assurances provided by our cloud provider. Unfortunately, the memory leak remains in our running version, as experienced on December 18th. We recognize it is unacceptable for this system to disrupt you and your patients. Fundamentally, with assurances from our cloud provider, we believed the patch would provide relief and time for the workstream to replace the existing system to complete their project without an additional event occurring. The patch did not offer that relief, and the team is working to further expedite the replacement workstream. The code for using the new system backing the background task processor was released to production on December 14th in beta mode for us to roll out and test slowly. The full rollout of the replacement is currently slated to be complete in the first half of January. The team is working quickly, including over the holidays, to expedite this further. We will communicate once this has been completed. Please know that this is our team’s number one priority. Fundamentally, we are replacing a vital piece of DrChrono's foundation. Significant testing is required to ensure you and your patients are supported appropriately. In the meantime, while the DrChrono application is supported by the system that we will replace, the infrastructure team has provisioned a spare cluster in case the existing cluster fails again. A contributing factor to these downtimes' length is that new cluster provisioning takes roughly one hour. Running a spare will help us move to a new cluster to restore service faster than waiting on server restarts or new cluster provisioning. The team is adding such an approach to their playbooks for all systems that support the DrChrono application. **Corrective actions related to this event include:** * Further expediting the system replacement workstream – **In progress** * Updating Maintenance Page wording to make it more straightforward that it was _unplanned_ or emergency maintenance – **In progress** * Developing a playbook to maintain the spare cluster and procedure to enable the light cluster – **Complete as of December 19th** * Researching and implementing further current system upgrades if thought to further reduce the occurrence rate of memory leak issues - **In progress** * Investigate and publish a plan with timelines regarding an “offline viewer” that can be used by DrChrono customers in the event of a degradation or outage to make schedule and clinical information available in a read-only mode – **In progress with a plan published in the coming weeks.** We architected our solution based on a cloud provider-managed service for our background task processors. This solution has served us well for many years. While cloud provider solutions are generally highly resilient and highly available, it is no longer the case for this service. Ultimately, your and your patients' experience is our responsibility and is taken very seriously. We are aggressively working towards delivering a solution that will provide consistent service. In addition, as the last corrective action above notes, we consider it an action item to provide our customers with an “offline viewer” to lessen the disruption and enable continued and safe care. We will share more on those plans in the coming weeks. We appreciate your patience as we roll out these changes and find more stable ground.
This incident has been resolved.
A fix has been implemented and we are monitoring the results. A formal RCA will be made available via our status page in the coming days.
We have made progress resolving the root issue and will have an ETA or full resolution soon.
The issue has been identified and we are currently working to restore service as soon as possible and will continue to post updates here.
We are continuing to investigate this issue and are currently down for unplanned maintenance. We will continue to provide updates via our status page.
We are currently investigating reports of intermittent slowness in the platform. We apologize for this inconvenience. Updates will be provided here as we have additional information to share.
Report: "DrChrono Slowness"
Last updateBetween 3:25 PM EST and 5:28 PM EST, we experienced an extreme traffic flood via our public API that caused backend message broker resource pressure resulting in degraded performance. The team worked quickly to identify and limit the source of the traffic flood. Once the flood was halted the team then worked to relieve broker resource pressure before reinstating general access to the system. We are currently evaluating improvements we can make to fortify our throttling capability to prevent this issue from reoccurring in the future.
A fix has been implemented and we are continuing to monitor the situation. This status page will continue to remain open and updates provided.
Engineering has identified the issue and is working to provide a solution. This status page will be updated again in the next 45 min. We are sorry for any inconveniences this has caused.
DrChrono is currently investigating system slowness. We will continue to use this page for updates. We are sorry for any inconveniences this may cause.
Report: "DrChrono System Performance"
Last update**RCA for DrChrono application outage on 18-October-2023** **Issue Start Date/Time**: 18-Oct-2023 at 10:50 AM ET **Issue Resolution Date/Time**: 18-Oct-2023 at 6:30 PM ET **Issue Summary:** Customers experienced 503 errors and slow performance while logging into DrChrono web application from 10:50 AM ET on Oct 18th, 2023. The service was restored fully by 6:30 PM ET on Oct 18th, 2023. **How were customers impacted?** Customers could not log into or utilize the DrChrono web application, mobile application or public APIs. **Root Cause:** At 10:50 AM ET two systems hosted by our cloud provider became degraded and intermittently unavailable. Due to the interplay of the two systems, customers had an often severely degraded experience in the window mentioned above. This was not caused by a change made by the DrChrono team but was a point in time event due to apparent hardware failure. The first system that failed backs our audit log storage and prevented any action that would have been written to the audit log. This was determined to be caused by disk pressure and likely a bad disk in the cloud provider’s environment as latency for writes was very high, but the volume of writes was normal. The pressure created from this degradation caused the second system, our background task processor, to fail to complete tasks and build up work to a degree that deadlocked the background task processor. Of note, the number of tasks was within expected bounds and should _not_ have caused this degradation but did nonetheless – and has been confirmed by our cloud provider to be a bug in the version they provide to us. To ensure writes are processed for all incoming requests and provide a high level of data integrity, all requests ensure that the background task processor is available. With the background task processor in an intermittent degraded state, all requests that were unable to reach the background task processor returned a 503 error. To that end, both systems needed repair to return to healthy service. Both systems were configured for high availability and had automatic failover. The systems’ attempts at automatically fixing the issue and cloud provider dashboards reporting as healthy masked the true issues for some time. Additionally, the background task processor is configured to add more instances to complete tasks upon degradation or building queues. In this instance, that made the performance of the background task processor counterintuitively _worse_ due to the bug mentioned above. Due to masked root causes, significant investigation was performed by the DrChrono _and_ cloud provider support teams. It took several hours to pinpoint the true root causes and resolutions necessary. Once found, resolutions were performed. **Resolution\(s\):** At roughly 4:00pm ET, the audit log datastore was re provisioned with more CPU, RAM and storage – not to provide more resources but to force the instance shifting from failed to healthy hardware within the cloud provider. Within 15 minutes, the new instance was up, running and healthy. It was keeping up with the same volume of writes that was failing just previously. Unfortunately, the background task processor was yet in a failed state due to the bug mentioned above and pressure created from the audit log datastore while it was intermittently out of service. With page loads checking on background task processor health, DrChrono still had issues serving customer requests. At roughly 6:15pm ET, it was determined that _reducing_ the number of servers performing tasks for the background task processor would put less pressure on the bug causing deadlocks. Within 10 minutes of reducing the number of servers, the background task processor recovered, and service returned to normal. We did receive reports of slowness for roughly 10 minutes, but we believe via the data collected that this was the system continuing to recover. Based on our own testing, monitoring and data we consider DrChrono to have returned to healthy service at 6:30pm ET. Throughout the path of investigation and resolution, we noted many opportunities to implement process, changes and enhanced monitoring to the system to prevent this from occurring in the future. **Mitigation steps planned/taken:** 1. Review and implement configuration to set number of servers for the background task processor to an ideal level to avoid the bug experienced. This has been completed as of 10/18. 2. Replace the existing background task processor system with another system that handles our volumes of tasks more performantly and does not have defects that affect our workloads. This is in progress and is our highest priority open item. We expect this to be completed in the next 30 days and have temporary relief implemented via item #1 above. 3. Introduce a buffer for audit log writes so that if audit log storage is unavailable operations can continue and no data loss occurs. This is in progress, and we expect this to be completed in the next 60 days. 4. Improve alerts for issues experienced above and other issues that may occur in these secondary systems within the DrChrono infrastructure stack. This is in progress, and we expect this to be completed in the next 30 days.
DrChrono has resolved all issues. We apologize for any inconveniences this may have caused.
DrChrono is continuing to monitor system system performance today.
We believe system performance has improved. We are no longer seeing 503 errors, however you may still be experiencing system slowness. We are continuing to investigate to bring a full resolution.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 60 minutes.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 60 minutes.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 60 minutes.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 60 minutes.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 60 minutes.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 60 minutes.
We are continuing to investigate the issues with slowness and intermittent outages. We are sorry for the issues this is causing. We will update this status page again in 45 minutes.
DrChrono is currently experiencing slowness or intermittent outages. Our engineering team is looking into this issue. We are sorry for any inconveniences this has caused. We will continue to update this page.
Report: "DrChrono System Issues Custom Data"
Last updateThe issue with custom data fields has now been fixed by the DrChrono engineering team. We are sorry for any inconveniences this may have caused you. Please reach out to support if you feel the issues have not been resolved for you.
We are working with DrChrono Engineering on the continued issues around custom data. We will continue to update this status page for updates. We are sorry for any issues this has caused.
The issue with custom data fields has now been fixed by the DrChrono engineering team. We are sorry for any inconveniences this may have caused you. Please reach out to support if you feel the issues have not been resolved for you. We will keep this status page set to monitoring to make sure this is completely resolved.
The issue has been identified and DrChrono is releasing a fix into production now. This fix will take around 30-45 minutes to see the effected changes. Once this has been restored the status page will be set to monitoring.
We are continuing to investigate this issue.
DrChrono is actively investigating issues around custom data. This may be effecting multiple parts of the system. We will continue to update this status page.
Report: "DrChrono Performance Issues"
Last update**Issue Summary** Customers experienced delays with items such as lab orders and referrals showing up in the message center between 9 AM EST and 6:45 PM EST. Between 3:30 PM EST and 6:45 PM EST, customers also experienced intermittent 503 errors while attempting to log into the system as well as slow/sluggish behavior while using the web application. During the latter incident window, 5% of all requests resulted in 503 errors while a total of 22% of requests failed to complete successfully \(some due to user canceling/refreshing\). **How were customers impacted?** Customers experience delays in receiving items in the message center as well as general application slowness and at times, trouble logging in. **Root Cause** It was determined that a production hotfix deployed on Friday evening at 9 PM EST resulted in a flood of messages that overwhelmed our background processing capabilities for some activities in the application. As a result, our Amazon AWS-hosted queuing infrastructure tipped over under resource pressure from the increased load and took longer than expected to failover and recover. However, this flood was not experienced until hitting Monday traffic levels. **Resolution** The hotfix code was reverted and the message backlog was drained. **Mitigation steps planned/ taken** * Update background processes that share the same processing queue to have dedicated queues. * Evaluate and improve the scalability and resiliency properties of our queue infrastructure. * Identify and remove unnecessary coupling of the web app to the message broker. * Improve rollback/revert efficiency * Improve performance testing in lower environments to better simulate production workloads for message queueing
This incident is resolved. Thank you for your patience as we continued to monitor system performance over the last day. A Root Cause Analysis (RCA) will be available via our status page soon.
DrChrono believes all issues have been resolved with intermittent service outages, slow loading, reports not arriving to the message center, referrals not loading, lab order failure, and other related issues. If you are still trying these actions and not seeing the expected results please reach out to support.
A fix has been implemented and we are monitoring the results.
Our engineering team continues to investigate today's ongoing performance issues including reports from some users that they are not able to log in to DrChrono. Thank you for your continued patience. We apologize for this inconvenience.
We are continuing to investigate this issue with high priority. Additional updates will continue to be posted on our status page.
We have received customer feedback that these issues have not been completely mitigated. Our engineering team is continuing to investigate.
A fix has been implemented and we are monitoring the results.
Our engineering team is continuing to work to implement a fix to resolve the performance issues. Thank you for your patience during this time.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
DrChrono is currently experiencing intermittent service outages, slow loading, reports not arriving to the message center, referrals not loading, lab order failure, and other related issues. Please be aware that we take this very seriously and our engineering team is actively investigating and working to resolve this situation as quickly as possible. We will provide another update via this status page as soon as possible.
Report: "Issues with clinical notes, faxing (Document-generating services) and Something Went Wrong screen"
Last updateWe've received an update that as of June 13th 3:37PM PDT, AWS services have been restored. DrChrono impacted functionalities should now be back to normal. Our team will be preparing an RCA and will be available via this status page.
We've received information that affected AWS services are seeing improvement in their error rates and are working towards full recovery. You may see improvements in DrChrono related to previously reported errors on accessing and saving clinical notes including faxing as a result. If you continue to experience issues, please reach out to our Support team by submitting a ticket. Thank you for your patience.
We have received information that Amazon Web Services (AWS) services in the US-EAST are encountering significant issues due to AWS multiple service failure. We have identified that this impacts document-generating functions such as viewing, locking, downloading or saving clinical notes as well as sending out faxes. Our team is working on ways to assess the impact of this and create mitigation steps. We will post additional updates as more information becomes available.
Report: "Not able to submit Quest lab orders"
Last updateThis incident has been resolved.
Quest lab orders appear to be working again, though no engineering changes were implemented by DrChrono. There appears to have been some type of connectivity disruption between DrChrono and Quest that has resolved itself.
DrChrono is currently investigating a service disruption with sending lab orders to Quest Diagnostics. Please be aware that our engineering team is prioritizing the investigation of this issue and working to resolve this situation as quickly as possible. We will provide another update via this status page as soon as possible.
Report: "Unable to join telehealth video visit from the web"
Last updateThis incident has been resolved.
The issue has been identified as stemming from a third party service our platform utilizes. We are reaching out to them in order to continue efforts to mitigate the problem. We will continue to post updates here as we have additional information.
We are currently investigating reports from our users that they are unable to join telehealth video visit calls from the DrChrono web application. As a workaround, video visits can be joined from the EHR iOS application. We will provide an update as soon as we have additional information.
Report: "Issue with Square: Disruption in Payment Acceptance"
Last updateMessage from Square: We’re no longer seeing elevated levels of payment declines and card processing activity has returned to normal. Our team will continue to work with our partners on uncovering the underlying cause of the earlier disruption and putting in place safeguards to help prevent issues of this nature from recurring in the future. We understand how important it is for all of our services to be running for your business, thank you for your patience with us as we monitored the situation.
Update from Square: While we are seeing less elevated level of declines with most card carriers, we are continuing to see reports of declines when using Discover. If you continue to experience declines, please advise your customers to use an alternative card or form of payment. We appreciate your patience as we continue to work on resolving this disruption. We will continue monitoring the situation and provide updates as we have them.
We received information from Square that they are currently encountering a system issue related to accepting payments. To get direct updates about this incident, please visit this link: https://www.issquareup.com/incidents/v952qqw12x1c We will continue to cross-post updates as we receive them.
Report: "Authentication issues with select partners connecting via API"
Last updateOur team has confirmed the the issue has been resolved. If you continue to experience issues, please reach out directly to the respective partner for further guidance on re-authenticating based on their individual workflow.
The team has identified the issue and are testing a fix to be implemented.
We've received communication that some customers are experiencing authentication issues or "API error 500" when trying to connect with select partners. Our team is currently investigating the issue and will provide further updates once they are available. We apologize for the inconvenience.
Report: "Task and message counts temporarily unavailable"
Last updateOn 11/4, the task and message count numbers were temporarily deactivated as a means to mitigate the overall system performance issues by re-allocating resources to increase capacity resources and help site speeds. The full RCA is accessible in the postmortem for the overall slowness issue. [Click here to view the page](https://status.drchrono.com/incidents/gw0f7l9xx3gn).
Our team has confirmed that the active tasks and message counts should now be displayed correctly. System speed and functionality should also be operating normally. Please feel free to reach out to the Support team should you continue to experience issues and we will be happy to assist you.
Our engineering team is increasing capacity for vital system functions by blocking task counts and messaging counts. This will be temporary. You will not be able to see the active task/message counts but will still be able to access your tasks and messages.
Our engineering team is increasing capacity for vital system functions by blocking task counts and messaging counts. This will be temporary. You will not be able to see the active task/message counts but will still be able to access your tasks and messages.
Report: "General system slowness on the DrChrono platform"
Last updateOn 11/3/ users encountered difficulties accessing the website and other system functionalities. ## Timeline All times are PST Timezone. | **Date/Time** | **Activity** | | --- | --- | | 2022-11-02 22:00:00 | An Amazon Web Services system failure causes the primary database to be restarted. The DrChrono EHR platform becomes inaccessible. | | 2022-11-02 22:15:00 | Amazon Web Services automatically restarts the affected server, but a metadata loss causes other servers to stop being able to serve production database traffic as well. | | 2022-11-02 22:15:00 | An engineer on the DevOps team comes online after being notified of the issue from automated alarming systems and starts investigating. | | 2022-11-02 22:25:00 | The root cause of the issue is identified and the engineer reroutes traffic to preserve system availability. The DrChrono EHR platform is now accessible, but running at reduced capacity due to the loss of extra servers. | | 2022-11-02 23:30:00 | After the first attempts to recover the affected servers directly fail, the DevOps engineer initiates a backup and restore process to prepare capacity for the upcoming day traffic for one of the affected servers. | | 2022-11-03 01:30:00 | The rescue process does not succeed for the additional servers and it's determined that they will also need to be restored from backups. The process is initiated. | | 2022-11-03 05:44:00 | Status page was created to notify customers about “General system slowness on the DrChrono platform”. | | 2022-11-03 8:30:00 | One of the replica servers finalizes phase 1/3 of backup restore; the next steps are engaged immediately. | | 2022-11-03 11:45:00 | One of the replica servers completes phase 3/3. Additional replica server restoration processes are initiated. | | 2022-11-03 14:15:00 | Additional replica servers' process restoration fails. | | 2022-11-03 16:27:00 | Additional replica servers' process restoration was attempted but also fails. | | 2022-11-03 17:45:00 | A backup process from the primary DB started. | | 2022-11-03 20:55:00 | One additional replica server finishes process restoration. | | 2022-11-03 21:15:00 | Spun up 3 more database replicas as a backup solution. | | 2022-11-03 22:30:00 | Added an additional 3 database replicas. | | 2022-11-04 00:05:00 | 6 new database replicas are installed and configured. | | 2022-11-04 01:40:00 | The backup from the primary database is finalized. | | 2022-11-04 01:45:00 | The primary backup snapshot with fast restore option is started. | | 2022-11-04 02:10:00 | The standby server started phase 1/3 of backup restore from the primary database. | | 2022-11-04 04:00:00 | Restore process is completed. | | 2022-11-04 07:00:00 | Status page was updated to resolved. | | 2022-11-04 07:07:00 | One of the replica servers is removed. After reviewing the server response, the team found it was not at full capacity and will need a complete restoration. | | 2022-11-04 08:10:00 | Tasks and message count display were blocked increasing the capacity of responses from the third server and primary. | | 2022-11-04 08:42:00 | A status page was created to notify customers about the “Task and message counts are temporarily unavailable”. | | 2022-11-04 08:42:00 | A replica server finalizes phase 2/3 of backup restore; the final phase is engaged. | | 2022-11-05 17:00:00 | Replica restoration process and shutdown of other servers are completed. All DB servers were monitored for the next hours. | | 2022-11-06 09:50:00 | Announced the system returned to full capacity and operational. | | 2022-11-07 06:33:00 | Status page “Task and message counts are temporarily unavailable“ was resolved. | ## Contributing Factor\(s\) Sentry alerts contain information to identify the current issue. ## Stabilization Steps A configuration change was made in the database to match the previous behavior. ## Impact Users experienced overall system slowness affecting major functionalities such as errors copying notes, locking notes, saving information, and the likes. ## Corrective Actions The Ops team has fixed the issue in the database table and will roll out updates and connections to the database gradually. Outlining the exact process for any database configuration changes and mitigation/fallback procedures will prevent a similar situation from happening again.
Our team has confirmed that system issues related to site slowness have been resolved. We will be continuing maintenance and server optimization over the weekend. An RCA will be published on this site once available. There may be instances where viewing saved information is not immediately present, or encounter issues with saving data. This is a temporary effect of our systems refreshing and should resolve on its own within the day. If the issue persists, please reach out to our support team so we can further troubleshoot with you.
Our team has applied measures to correct the slowness issues customers have encountered today. Our server speeds are recovering and are expected to be restored to normal performance levels within the next few hours.
We are beginning to see improvements in site performance. Our engineering team is continuing efforts to restore performance fully. Thank you for your continued patience throughout this process.
We are continuing to work on a fix for this issue.
Our team has identified an issue on the system that may impact the overall performance across the DrChrono platform. This may become more apparent during peak hours of the day. We are working towards getting this resolved and will post updates as they become available.
Report: "Labcorp results temporarily suspended"
Last updateAll affected Labcorp result files received between October 28th and November 8th have been reprocessed and are now available in the DrChrono platform. As mentioned yesterday, new results received from late November 8th onward are processing as normal. Thank you for your patience while we worked to resolve this issue.
We are continuing to reprocess all affected Labcorp result files received between October 28th and November 8th to make these results available in the DrChrono platform. New results received from late November 8th onward have been reenabled.
The root cause of the issue has been identified. A change has been put into place by Labcorp that will allow us to properly digest and route result files. We are working now to reprocess all affected result files received since October 28th after which customers will begin to see these Labcorp results in the DrChrono platform.
We are continuing to investigate this issue.
We’ve encountered an issue that potentially impacts some Labcorp customers where patient test results appear to have multiple values of the same test or multiple test results appearing on a patient’s chart. We are urgently working with Labcorp to get this resolved immediately. Test results will be temporarily suspended from being delivered to DrChrono until the problem is fully resolved to prevent more errors from being passed through. As an additional remediation step, all affected lab results since October 28th have been removed from DrChrono. Results will be reprocessed after completion of the investigation with Labcorp. Information will not be lost during the suspension and will resume transmission once the connection is re-established.
Report: "Connectivity Issues Between DrChrono and LabCorp"
Last updateThis issue has been resolved. Please note that our system is currently working through the backlog of orders and results that were queued for processing. It may take several hours for this backlog to be cleared this evening. In the meantime, new orders that are placed will expectedly populate a "Timeout" notice on the screen but will be added to the queue of orders that is currently processing. There is no need to try sending the order again. Thank you for your patience during this investigation and resolution.
A fix has been implemented and we are monitoring the results.
We are encountering connectivity issues between DrChrono and LabCorp as of earlier today. This impacts both the incoming and outgoing transmission of data, which can be seen as time+out errors when attempting to send lab orders. Our team is currently working on getting the connection restored and will provide you with updates once we have more to share. Please know that your information remains in the database and will resume transmission once the connection is back online.
Report: "Unable to Copy Previous Note"
Last updateThis incident has been resolved. Thank you for your patience while our team investigated and fixed the issue. A root cause analysis will be available in the coming weeks attached to this status page incident.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating reports of the inability to use the Copy Previous Note feature. This does not appear to be affecting all users of the platform. Our team is looking into this with high priority. We will continue to post updates via this status page. Thank you for your patience.
Report: "Quest lab ordering service outage"
Last update# Root Cause Analysis: 08/30/2022, Quest Diagnostics ## Quest Service Disruption ## Summary The system was unable to process an update to the Quest compendium because of a 500 error from the Quest side. There is a problem on our side, however, where if anything goes wrong during downloading or parsing, any subsequent codes won’t be there because they are removed. ## Timeline \(EST, 24-hour clock\) All times are EST. | **Date/Time** | **Activity** | | --- | --- | | 2022-08-30 08:32 | Started receiving initial reports of issues with Quest test orders. Investigation started. | | 2022-08-30 09:00 | Support team reached out to Quest to see if there were any issues on the Quest side. | | 2022-08-30 09:30 | Informed by Quest that they did not have any issues on the Quest side. | | 2022-08-30 10:22 | Status Page created. | | 2022-08-30 10:28 | Support team pulled a ticket from a previous occurrence from March 2019. | | 2022-08-30 10:32 | Problematic scheduled job identified. | | 2022-08-30 10:34 | Specific error from scheduled job identified. | | 2022-08-30 12:36 | Restarted scheduled job. | | 2022-08-30 12:46 | Status Page updated to identified | | 2022-08-30 12:56 | Verified that issue is resolved | | 2022-08-30 13:00 | Status Page updated to monitoring | | 2022-08-30 14:17 | Status Page resolved. | ### Contributing Factor\(s\) The management command deletes all the rows from the table so if something goes wrong nothing is left. ## Impact Impact was widespread to all customers utilizing Quest for test orders. This requires development effort from the engineering team. ## Corrective Actions Management command ran again to fix the problem.
DrChrono is pleased to report that our engineering team has implemented a resolution to address the service outage related to Quest lab ordering. We want to thank our users for their patience and support as we worked to address this matter as quickly as possible.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue and will provide additional updates via this status page. Thank you for your patience.
DrChrono is currently experiencing a service outage related to placing Quest lab orders. Please be aware that our engineering team is prioritizing the investigation of this issue and working to resolve this situation as quickly as possible. We will provide another update via this status page as soon as possible.
Report: "Site Wide Outage"
Last updateThis incident has been resolved. Thank you for your patience while our team worked to identify and solve the issue. A root cause analysis will be available via the status page incident within the coming days.
A fix has been implemented and we are monitoring the results.
DrChrono is currently experiencing a major service outage. Please be aware that we take this very seriously and our engineering team is actively investigating and working to resolve this situation as quickly as possible. We will provide another update via this status page as soon as possible.