Is Doxy.me Inc. Down Right Now? Check if there is a current outage ongoing.

**Summary of event** On April 28, our monitoring system detected a sharp slowdown in platform performance, preventing some users from accessing doxy.me. This disruption lasted for approximately 35 minutes before service was restored by increasing system capacity. **Business impacts** During the incident, users were unable to access doxy.me for telehealth sessions, leading to delayed or missed appointments and interruptions in patient care. **Root cause** A recent system update unintentionally lowered the number of active servers. Because this deployment occurred after our daily resource allocations, the platform did not have sufficient capacity to handle the level of traffic, resulting in degraded performance. **How the issue was resolved** Once the root cause was identified, our engineering team increased the number of active servers. This promptly reduced system load and restored access for users. **Preventative next steps** * Update deployment procedures to appropriately size server capacity at the time updates are launched * Improve intra-day automated resource allocations and real-time monitoring

The system issue preventing patient check-in and provider account access has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.

The system is returning to normal. We are continuing to monitor the situation.

We're currently investigating a reported system outage that's impacting patient check-in and provider account access.

The issue causing the app outage has been fixed and the system is now fully operational. All apps are functioning normally. A post-mortem report will be provided in the coming days.

Monitoring - App functionality has been restored. We’ll continue to monitor the situation.

We've identified the cause of the outage. Our team is working on a fix, and we’ll share updates as we make progress.

We're actively investigating an outage affecting Apps, such as File Transfer and Whiteboard.

We recently encountered an issue that prevented patients from appearing in their provider's waiting room queue. This issue has now been resolved, and patient check-in is functioning as expected. A post-mortem report will be provided in the coming days.

A fix has been implemented and we are monitoring the results.

System is returning back to normal. We are continuing to monitor the situation.

We're currently investigating a reported system outage that's impacting patient check-in and provider account access.

The system issue preventing patient check-in and provider account access has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.

The system is returning to normal. We are continuing to monitor the situation.

We are currently investigating this issue.

**Summary of event**: On October 24, patients who successfully checked in did not appear in the Waiting Room queue due to an incident from our real-time partner, PubNub. PubNub provides the technology that enables instant updates to [doxy.me](http://doxy.me) when a patient has joined a provider's Waiting Room and allows us to start a video call. This led to an outage on [doxy.me](http://doxy.me) approximately between October 24, 7:15 AM ET - 10:00 AM ET and October 24, 8:15 PM ET - October 25, 6:00 AM ET. **Business impacts:** Although providers were able to log in, their Waiting Room queue remained empty, preventing a session from starting between the patient and provider. **Root cause:** Our real-time partner, PubNub, encountered an unusual surge in traffic from sources other than [doxy.me](http://doxy.me), that temporarily overwhelmed their system, preventing it from responding quickly enough. As a result, PubNub’s communications to [doxy.me](http://doxy.me) were interrupted, causing patients to not appear in the Waiting Room queue.  **How was the issue resolved:** PubNub blocked the source of surging traffic and made changes to handle the increased demand. Once PubNub made this update, [doxy.me](http://doxy.me) was fully operational again. [Doxy.me](http://Doxy.me) did not make any changes.  **Preventative next steps:** [Doxy.me](http://Doxy.me) will collaborate with PubNub to ensure PubNub can manage traffic surges from its other customers and maintain scalable infrastructure, minimizing any potential impact on [doxy.me](http://doxy.me) services and customers.

We recently encountered an issue that prevented patients from appearing in their provider’s waiting room queue. This issue has now been resolved, and patient check-in is functioning as expected. We will provide a post-mortem report in the coming days.

System is returning back to normal. We are continuing to monitor the situation.

We’re aware that patients are not appearing in their provider’s waiting room queue, and we are actively investigating the issue. We will provide updates as soon as we have more information. Thank you for your patience.

**Summary of event**: On October 19, [doxy.me](http://doxy.me) encountered an issue that prevented patients from checking in on [doxy.me](http://doxy.me) 2.0. This outage resulted in 8 hours and 28 minutes of unplanned downtime between 8:01 AM ET and 4:30 PM ET. **Business impacts:** Patients were unable to complete the check-in process. After entering their name and clicking the Check-In button, they remained stuck on the check-in screen instead of joining their provider’s waiting room. While providers were able to log in during this incident and receive patient check-in notifications, patients didn’t appear in their waiting room queue. This prevented a provider from starting a video call with the patient. **Root cause:** Security improvements implemented in our real-time messaging system and [doxy.me](http://doxy.me) 2.0 platform led to an issue that arose approximately 48 hours after the release. The continuous release cycle of [doxy.me](http://doxy.me) prevented the issue from appearing in our non-production environments used for pre-release testing. **How was the issue resolved:** Engineering identified real-time messaging as the root cause of the outage and updated accordingly while maintaining the security enhancements implemented.   **Preventative next steps:** We’re enhancing our testing framework to include additional scenarios that will help identify unexpected real-time issues.

We recently encountered an issue that prevented patients from checking in on doxy.me 2.0. This issue has now been resolved, and patient check-in is functioning as expected. We will provide a post-mortem report in the coming days.

Resolved - The issue affecting patient check-ins and starting doxy.me appointments has been resolved and will continue to be monitored to ensure stable performance

Our real-time service provider is seeing improvements with the patient check-in and appointment start issues, and we will continue to closely monitor the recovery process

Our real-time service provider has identified the issue and is working on a fix. We will continue to monitor and work towards restoring our services.

We are currently investigating this issue of patients not being able to check in and start call. We will update this status as soon as we have more information

This incident has been resolved.

The issue has been identified and a fix is being created. We are continuing to monitor the situation and have seen normal site functionality return.

We are continuing to investigate this issue.

We are aware of an outage currently affecting doxy.me. Our team is currently investigating at this time. We will post an update as we know more.

Between 12:37 PM and 1:13 AM EST, our partner AWS experienced elevated error rates for request serviced by the CloudFront Origin Shield and Regional Edge Cache in the US-EAST-1 region. The issue has been resolved and service is operating normally.

We are continuing to investigate this issue.

We are currently investigating this issue.

**Summary of event:** On May 25th, [Doxy.me](http://doxy.me/) experienced a database issue that resulted in 27 minutes of unplanned downtime from 2:03pm to 2:30pm ET. **Business Impacts:** Sessions in progress were not impacted. The incident resulted in providers being unable to log in for their sessions. In addition, patients who were not already in the queue were unable to check in to their providers waiting room. **Root cause:** A data import file was processed internally. While it was being processed, a series of long-running secondary server processes were triggered. Unfortunately, these processes built up in number, consuming additional server resources until the servers were backed up. **How was the issue resolved:** The file was able to finish processing and the system self recovered. **Preventative next steps:** We are optimizing the the long-running processes which will help prevent this scenario from happening again. In the short term, we are no longer running large data imports during peak business hours.

Doxy.me services continue to be stable. We are considering this incident resolved and will provide a post mortem within the few business days.

System levels have remained stable overnight. We are continuing to monitor the system performance and will provide another update by 12pm ET.

We are seeing our system stabilizing. We are continuing to look into the incident and are going to continue to monitor the situation.

We are aware of an outage currently affecting doxy.me. This appears to affect sign-in, call connecting, and various other system services. Our team is currently investigating at this time. We will post an update as we know more.

**Summary of event:** On January 10th, an update caused out-of-date browsers to not load [doxy.me](http://doxy.me/). **Business Impacts:** Patients or providers using the impacted browsers were unable to access [doxy.me](http://doxy.me/). **Root cause:** One of the software packages used in our code was updated on January 10th, 2023 at 08:20 AM UTC. The updated version was incompatible with and no longer supported older browser versions, including any versions older than the following: * Chrome: 93 \(released August 31st, 2021\) * Firefox: 92 \(released September 7th, 2021\) * Safari: 15.4 \(released March 15th, 2022\) * Edge:  93 \(released August 3rd, 2021\) **How was the issue resolved:** [Doxy.me](http://doxy.me/) released a fix that reverted the update to the software packages on January 11th, at 8:03 AM UTC. **Preventative next steps:** We will be updating our internal system requirements and our supported browser policies. Once approved, we will increase our communication efforts with our customers to make it easier for them to recognize when they are using an out-of-date browser and how to update it. We will also shore up our testing coverage to better detect these problems in advance. Thank you for your patience.

We have deployed a fix restoring functionality to the out of date browsers. We recommend keeping your devices and browsers up to date in general, but do apologize for any disruption this incident may have caused. We will share the post mortem within the next few business days.

We have determined the root cause for this incident and are currently testing a fix. At this time, we expect the fix to be deployed early tomorrow, between 3:00am and 7:00am EST (between 08:00 and 12:00 UTC).

We are currently investigating an issue where older versions of web browsers (Safari, Chrome, Firefox, Edge) are not allowing doxy.me to load. Updating your browser to the latest version will allow doxy.me to load. Browsers able to load doxy.me: Safari 15.4 and newer Chrome 93 and newer Firefox 92 and newer Edge 93 and newer If you or your client is impacted at this time, our <a href="https://help.doxy.me/en/articles/2350628-update-your-web-browser">help article</a> will help guide you to update.

We have received confirmation that the email services have been restored. We are closing this incident as resolved.

Our email vendor is currently working to resolve connection issues which may impact doxy.me email invites sent to patients and patient check-in email notifications sent to Providers. We are monitoring the situation and will provide updates as we learn more.

**Summary of event:** The [doxy.me](http://doxy.me/) marketing site was not successfully rendering on December 5th 2022 from 6:45AM to 8:06AM EST. Individuals trying to access the home page were presented with improperly formatted images and links, making it appear non-functional. **Business Impacts:** [Doxy.me](http://doxy.me/) services were not ultimately impacted. Providers may not have been able to log in via the [doxy.me](http://doxy.me/) homepage, but [doxy.me/sign-in](http://doxy.me/sign-in) and Clinic sign-in pages were working as expected. Patient check-in pages were not impacted. **Root cause:** There was an error with our marketing website hosting vendor which cascaded down to visible web pages. **How was the issue resolved:** The marketing site was restored to a checkpoint from the previous day. Shortly thereafter the website displayed correctly. **Preventative next steps:** The sequence of events leading up to the error has been identified and is being prevented in the future through an adjustment to policies and controls relating to this vendor.

This incident has been resolved. We will post an RCA within the next few business days.

The doxy.me marketing page was not rendering correctly. Upon going to doxy.me, the site would load with incorrect formatting and was presenting a 500 error message. Any providers trying to login to doxy.me should use doxy.me/sign-in or use their clinic sign-in page to access their accounts.

**Summary of event:** On June 13th between 8:22pm and 11:30pm, the [doxy.me](http://doxy.me/) homepage was unresponsive. During this time, our services were still up and operational. Patients were able to check in using their providers room links and providers were able to log in via [doxy.me/sign-in](http://doxy.me/sign-in). **Business Impacts:** Providers wanting to create an account or sign in from the homepage were unable to do so. **Root cause:** The company hosting our homepage blacklisted traffic from our IP Addresses due to abnormal activity. **How was the issue resolved:** The hosting company whitelisted our IP Addresses. **Preventative next steps:** The hosting company is reviewing configurations to help prevent this from happening again in the future.

This incident has been resolved. We will provide a post mortem in the next few days.

We have identified that our website content host is blocking traffic while trying to access doxy.me. We have worked with them to get our IP's whitelisted. The doxy.me homepage is accessible once again. We are continuing to monitor this situation.

The doxy.me homepage is currently down and showing a "504 ERROR The request could not be satisfied" screen. This is only impacting the doxy.me homepage. Free and Professional providers are still able to log in by using https://doxy.me/sign-in. Clinic providers should continue using clinic landing page to log in as a provider. <a href=" https://help.doxy.me/en/articles/3808532-clinic-login">Click here</a> for more information with clinic provider logins. Patients should not be impacted as they would check into the provider's room directly.

**Summary of event:** On 5/17/22 from 3:32 to 10:26 AM ET, some Providers were unable to log in to their accounts. The reCAPTCHA that is in place to prove that users are not robots was preventing the logins. **Business Impacts:** A limited number of providers were unable to host sessions with their clients. **Root cause:** A configuration within the reCAPTCHA integration was incorrect. **How was the issue resolved:** We identified the incorrect setting and applied an update to correct the misconfiguration. **Preventative next steps:** Scenarios identified are being added to our QA testing processes, both manual and automated going forward.

The reports from login issues have ceased. Our support team has confirmed providers reporting the problems have regained access to their accounts. We will provide an RCA in the coming days.

A fix has been implemented and we are seeing successful reports of the accounts being able to be accessed. We are continuing to monitor this incident to ensure it is fully resolved.

We are seeing an issue where providers are unable to sign in to their accounts. After resetting passwords and checking reCaptcha, we are still unable to successfully log in. Our developers are actively researching this issue.

The iOS 15.4 update appears to have resolved the errors preventing calls from being made. We are considering this incident resolved.

Apple did not include the fix for the iCloud Relay issue in the iOS 15.3 update as we anticipated. We now expect the fix should be included in the iOS 15.4 update expected in the first half of March.

Doxy.me developers were able to confirm the 'firewall errors' on iOS and MacOS devices are being generated by Apple's 'iCloud Relay' update. This is a complex VPN-like system designed to obscure your IP address and encrypt your traffic over the internet. We were able to determine that the code with the networking changes to support this new feature is enabled, even if the feature itself is not. Apple has developed a fix for this which is expected to be rolled out with iOS 15.3. We do not know when 15.3 is expected. We have also applied an update to change how iOS devices are being routed through the connection process that does not conflict with the Apple update to mitigate this problem until Apple's fix has been made public. We are seeing more than 80% reduction in the errors being logged since our fix was deployed.

We have identified a setting that is enabled by default on updated iOS devices that is preventing doxy.me calls from connecting. To correct this, please disable the 'WebRTC PlatformUDP Sockets' setting. On iPhone or iPad: 1. Open up the device’s settings 2. Navigate to Safari > Advanced > Experimental Features 3. Disable WebRTC Platform UDP Sockets. 4. Close Safari and open it again. 5. Log back into doxy.me

We have implemented our vendor's fix for this incident and are no longer seeing related user reports.

We have rolled back updates from our third party vendor to reduce the occurrences of this issue. We are pending another update from the vendor as we continue to work together to resolve this problem. Please continue to use the workarounds listed.

We are continuing to see reports with Safari’s media stream being destroyed during calls and we continue to investigate the issue with our media vendor. We recommend using either Chrome or Firefox as we are no longer receiving reports of this problem with these browsers. We see fewer issues being reported after updating the experimental feature setting within Safari. If you’d prefer to continue to use Safari, please update the WebRTC Platform UDP Socket setting: 1. On iPhone or iPad, open the device’s settings. 2. Select Safari > Advanced > Experimental Features. 3. Disable WebRTC Platform UDP Sockets (switch toggle from Green to Gray). 4. Close Safari and open it again. 5. Return to doxy.me.

We have delivered another update that should resolve the remaining problems with the media publishers being destroyed on Safari during a call. Our teams will continue to monitor the situation until issues are confirmed to be resolved.

We deployed a fix that is resolving the issue in Chrome and Firefox. We are continuing to look into the publishing issue with Safari browsers. If you use Safari as your default browser, please continue to use Chrome for now.

We have identified an issue with Firefox failing to send the audio and video from the device after the first few minutes of a call. The individual using Firefox can still see and hear the other participant(s) on the call during this error. The person using Firefox can no longer be seen or heard. Temporary Workaround: Switch to another supported browser (Chrome, Safari, Edge) for the time being.

Postmortem included on 3/17/22’s [Increased Latency and Response Times](https://status.doxy.me/incidents/f545th0zlx5j) incident.

Yesterday, we restored the previous version of doxy.me to remedy the issues with the site not loading. We are marking this incident resolved. We will continue to investigate the root cause of these issues in the update we are trying to apply. We do not have an estimated timeline on a completed root cause analysis at this time, but we will share a post mortem as soon as we can.

Access to Doxy.me was restored. We continue to monitor the situation and will post any updates on this status page.

We are continuing to investigate this issue.

We are currently investigating an issue of the Doxy.me site not loading for 10 minutes, we are monitoring the situation and will post updates here.

Postmortem included on 3/17/22’s [Increased Latency and Response Times](https://status.doxy.me/incidents/f545th0zlx5j) incident.

This incident has not reoccured. We are considering this resolved and will provide a root cause analysis by the end of the week.

We implemented a fix this morning for the cause of yesterday's slowness. We are no longer seeing any abnormal behavior. We will continue to monitor this issue until we are sure it has been fully resolved.

We are no longer seeing slowness in the sign-in process and page load times have returned to normal. However, we are continuing to look into the cause for future prevention.

We are investigating reports of providers being unable to sign in via doxy.me/sign-in. We are seeing increased slowness in the sign-in process and slower page load times than usual. We are working to get this resolved.

**Summary of event:** [Doxy.me](http://Doxy.me) experienced a severe increase in API response time, preventing login attempts and causing overall system slowness. **Business Impacts:** [Doxy.me](http://Doxy.me) was intermittently unresponsive from 2:10 PM until roughly 2:23 PM EDT on Thursday, March 17th. **Root cause:** The latency spikes which occurred three times over the past few weeks were the result of a sub-processor answering our requests too slowly. It was responding so slowly that systems kicked in and rebooted everything before logs could be sent. Confusing the issue, we noticed this each time acutely following deployments, leading us to question whether the problem was in our new code, leading to rollbacks and introspection. Finally, we discovered a service call that was taking up to 5 minutes, and which was called frequently enough in our busy periods that it almost immediately caused the server to be non-functional and rebooted. **How was this resolved:** We identified and removed the system calls that were occasionally slow enough to take down our service. **Preventative next steps:** In addition, we are reassessing any unnecessary API calls that are being made and the length of time we wait for a response so we do not continue to run into long running requests that tie up our server resources. We are also working on improving our system health tracking so we can identify and address anomalies more efficiently.

The fix has been implemented. Our servers are returning to normal capacity, response times are within expected limits, and functionality has been restored.

We have identified an issue and are implementing a fix.

We are currently investigating this issue.

Apple has resolved the issue with the low audio bug by changing their configurations. We are considering this resolved.

Based on Apple's suggestion, we have applied an update that forces an audio gain to impacted Apple devices to reduce the impact of this bug on our providers. Apple has separately developed a fix that is expected to be deployed with iOS 15.4. Apple is researching an additional scenario with 15.4 Beta where, when a participant mutes their microphone and then unmutes it, the low audio is experienced again.

We are continuing to monitor for any further issues.

In a continuing effort to mitigate the low audio bug, we have found that following these steps helps resolve the issue during a call. When the patient says audio is low: 1. Copy this link and put it in your chat box: https://ccrma.stanford.edu/~jos/mp3/pno-cs.mp3 2. Ask the patient to click the link. When they click it they will hear a piano sound AND the volume will increase. 3. Ask them to close that tab and go back to the tab with doxy.me in it. 4. If the camera stops (this may happen occasionally) ask them to simply turn the camera off and on again to bring back the video. 5. The audio level should be back to normal now.

Apple has started rolling out iOS 15.2 which they claim has an improvement (not a permanent fix) for the Low Audio bug. They state that the audio is now coming out of both the ear piece and loudspeaker, so the audio volume should be louder after the update. Please note this is not the long term fix. Please check your device for this update. If you have a headset or AirPods that you can use, we recommend using those if you continue to experience problems with the low audio.

We are aware of a bug introduced by the deployment of Safari 15 that is impacting all WebRTC systems. Many Safari 15 users may experience very low or hard to hear audio from their clients while on calls with their clients or providers. If you experience this issue during a call, please try another internet browser such as Google Chrome or Firefox. We are working internally and with our vendors to see if there is anything we can do to help mitigate the issue. Our Vendor has confirmed that they have filed a bug with Apple. We are also anticipating the issues will be resolved in an upcoming update by Apple. We do not have any information from Apple regarding an expected fix date. We have identified a couple workarounds that is having some success with increasing the volume during sessions: Workaround #1 1) Start a call 2) Open iOS settings > Accessibility > Hearing devices 3) Hearing Aid Compatibility> Toggle On/Off. The end result of leaving the setting on or off does not matter for the doxy.me call, just flip the setting. Note: Workaround 1 only impacts the current call. These steps would need to be repeated for your following calls. Workaround #2 1) Open a new browser tab in that browser (DO NOT close the doxy tab) 2) Pull up http://youtube.com/ in the new tab 3) Select anything and play it for a few seconds 4) Close the tab with youtube in it and go back to Doxy tab 5) The Volume should be remain at the normal higher volume

We are aware of a bug introduced by the deployment of Safari 15 that is impacting all WebRTC systems. Many Safari 15 users may experience very low or hard to hear audio from their clients while on calls with their clients or providers. If you experience this issue during a call, please try another internet browser such as Google Chrome or Firefox. We are working internally and with our vendors to see if there is anything we can do to help mitigate the issue. Our Vendor has confirmed that they have filed a bug with Apple. We are also anticipating the issues will be resolved in an upcoming update by Apple. We do not have any information from Apple regarding an expected fix date. We have identified a workaround that is having some success with increasing the volume during sessions: 1) Open a new browser tab in that browser (DO NOT close the doxy tab) 2) Pull up http://youtube.com/ in the new tab 3) Select anything and play it for a few seconds 4) Close the tab with youtube in it and go back to Doxy tab 5) The Volume should be remain at the normal higher volume

**Summary of event:** [Doxy.me](http://Doxy.me) experienced a severe increase in API response time. **Business Impacts:** [Doxy.me](http://Doxy.me) was unresponsive from 10:01 AM until roughly 10:20 AM. **How was the issue resolved:** We rolled back an update that was pushed out this morning. Removing the update from our production servers allowed our servers to recover and return to normal capacity. **Root cause:** One of the updates we rolled out on 3/10/2022 caused a problem with our servers writing to the cache. An error occurred as we were writing the data to the cache, which led to the servers failing.  As servers failed, response times increased and the system became unresponsive. **Preventative next steps:** We have prepared a fix to the problematic code and will be redeploying the update with the fix next week. In addition, we are improving our testing on our staging environments to help expose these problems before they are introduced to our providers and patients.

The fix we implemented has corrected the problem. We are marking this incident resolved. We will be sharing a post mortem in the coming days.

A fix has been implemented and we are continuing to monitor the situation.

The issue has been identified, and we are currently working on a fix.

We are continuing to investigate this issue.

We are currently investigating the issue.

This incident has been resolved.

Google moved the deployment from Chrome 98 (February) to Chrome 97 (January). If you are still experiencing this issue, please ensure your browser has been updated to the latest version which would contain the fix.

Google Chrome has confirmed the fix for this issue will be deployed with Version 98. In the meantime, the workaround will still be needed: Android 12 users are able to fix this with these steps: 1) Go to chrome://flags 2) under "Search Flags", enter "hardware" 3) Disable the "WebRTC Hardware Video Encoding" Setting 4) Check back in and restart the call.

We have identified an issue where the video from Android 12 devices appears as scrambled lines instead of a clear picture. Audio is not impacted and comes through normally. Android 12 users are able to fix this with these steps: 1) Go to chrome://flags 2) under "Search Flags", enter "hardware" 3) Disable the "WebRTC Hardware Video Encoding" Setting 4) Check back in and restart the call.

**Summary of Event:** An update was applied to help resolve the [iOS 15 Firewall Errors](https://status.doxy.me/incidents/6rd8z716mdfr) that prevent calls from connecting. **Business Impacts:** Providers were unable to access their accounts from 12:00pm EST until 12:17pm EST on Friday, January 7, 2022, due to this update. **Short-term Resolution:** Once the login problem was identified, the update was reverted. **Root Cause:** The update caused the API to not be reachable which resulted in [doxy.me](http://doxy.me) not loading in order to successfully log in or check in. **Future Prevention:** We have added this scenario to our planning and testing processes to prevent the same issue from occurring in the future. We are investigating another approach to resolve the initial issue of iOS 15 Firewall Errors.

This incident has been resolved. Our team is working on the post-mortem which will be posted next week.

We are seeing that our API is continuing to stabilize. We are looking into what caused this spike while we continue to monitor the incident.

We are looking into the reports of clients being unable to log into doxy.me. We will provide updates within the next 30 minutes.

This incident has been resolved.

A fix has been implemented and we are monitoring the results for the next 30 minutes. Patients should be able to check-in as usual.

We have identified an issue where patients are not able to check-in. A fix is in progress, and this is expected to be resolved in the next 5 minutes.

The issue was closed by our vendor on Dec 17, 2021 at 14:27 UTC.

Doxy.me’s 3rd party video provider reported a failure with a server in the Eastern United States. They have listed these problems as resolved. We are seeing our services returning to normal. We are currently looking into reports and data on our end and everything now seems stable.

AWS reported internet connectivity issues in the US West Regions including N. California and Oregon. They have listed these problems as resolved. We are seeing our services returning to normal.

We are currently looking into reports of providers appearing offline and calls not connecting. We believe we are being impacted by an AWS problem that is impacting numerous websites and services.

The issue has been resolved. If the issue has been affecting you, refreshing the page in the browser should be enough. Some users may need to additionally clear their browser cache. If you continue to experience issues, please let us know at support@doxy.me.

A workaround has been identified to get the video image unfrozen: 1) If the video display is frozen on Safari 15, try clicking the Gear Icon for Call Settings. 2) When the call settings open, click the “X” to close out the Call Settings. 3) When it closes the screen should no longer be frozen.

We are continuing to monitor for any further issues.

We are aware of a bug introduced by the deployment of Safari 15 that is impacting all WebRTC systems. Many Safari 15 users may experience frozen video images while on calls with their clients or providers. If you experience frozen images during a call, please try another internet browser such as Google Chrome or Firefox. We are working internally to see if there is anything we can do to help mitigate the issue. We are also anticipating the issues will be resolved in an upcoming update by Apple. We do not have any information from Apple regarding an expected fix date.

* **Summary of event:** Doxy.me experienced connectivity issues due to a 3rd party’s servers going down. * **Business Impacts:** Providers were unable to log in to doxy.me. Patients were unable to check in to their provider's queue. * **How was the issue resolved:** We deployed a software update which prevented our system from making the call to the 3rd party provider. * **Root cause:** Our 3rd party provider was refusing connections at that time.  The part of our system that was trying to connect to this 3rd party and get an answer was not able to recover from the refused connection. * **Preventative next steps:** We are continuing to test and implement updates to our code that will better handle future scenarios when our vendors have server problems.

This incident has been resolved. We will provide a port mortem in the next few days.

A fix has been implemented and we are continuing to monitor the results.

The root cause of the login and access issues this morning has been identified. We are implementing a fix with a vendor who's services went down this morning and are continuing to monitor this problem.

We are investigating an issue with intermittent login issues. We will provide an update in the next 30 minutes.

* **Summary of event:** Users trying to access [doxy.me](http://doxy.me) were experiencing ‘Server Problem’ error pages. * **Business Impacts:** Providers were unable to log in to start calls. Patients were not able to check in to see their providers. * **Root cause:** An update deployed last Thursday \(6/3/21\) contained an incorrect change to database indexes. Under the load of users today, the pods began to restart frequently. This created additional strain on the servers. * **How the issue was resolved:** We added temporary resources to help stabilize the working environment. * **Preventative next steps:** The incorrect change to the database indexes will be rolled back and taken out of our production codebase so users won’t be affected.

This incident has been resolved. We will be providing a post mortem in the next 2 business days.

Doxy.me pages are now loading, and we are now having successful calls internally. If you are still experiencing the server error page, please do a hard refresh or open a new browser to resume using doxy.me. To do a hard refresh, try one of the following based on your device: For Mac users: - Hold Shift and click the reload button on your browser - Press the Command, Shift, and R keys at the same time For Windows users: - Hold down Ctrl and click the Reload button. - Hold down Ctrl and click F5. - Hold down Ctrl + Shift and click R. We are continuing to research for the root cause of the server errors this morning.

We are currently investigating this issue.

The vendor's latency has returned to their normal range. This incident has been resolved.

We are continuing to monitor for any further problems.

We are continuing to monitor this issue. The vendor has implemented a fix, and their latency statistics are showing improvement. Internally, we increased our server capacity to help mitigate the strain caused by the increased latency. We are seeing the number of errors being logged vastly reducing as the latency numbers improve.

One of our vendors is reporting a latency spike which is causing the server errors we are experiencing. They are still investigating the root cause of their spike.

We are looking into reports of several errors trying to send invites as well as checking into doxy.me.

The ability to send texts has been resolved with our partner. We will continue to monitor and follow up with more details if needed.

We are continuing to investigate this issue. We will update as soon as we know more.

Our Text/SMS partner has reported a potential issue with sending text/sms messages this morning. Their engineering team has been alerted and is actively investigating. We will update as soon as we have more information.

AWS has resolved the issue with the network load balancers and is showing their service is operating normally. We are no longer seeing latency spikes. This issue is now resolved.

The vendor is continuing to monitor their system and has updated their status back to fully operational. AWS is continuing to roll out their updates to resolve the issue with their network load balancers. Our logged errors have been drastically reduced thanks to these efforts. We will continue to monitor this incident into tomorrow.

The vendor has implemented a fix to route traffic around the failure in their system as a temporary fix. In addition, this incident appears to have been magnified by an incident at Amazon Web Services (AWS). AWS is currently rolling out a fix on their side to address problems with their Network Load Balancers. We are continuing to monitor this incident and are seeing a significant reduction in the errors being logged since these efforts have been announced.

We are continuing to monitor this incident. The vendor is continuing to work on fixes to bring everything back to an operational status.

We are continuing to monitor this incident. The vendor is continuing to work on fixes to bring everything back to an operational status.

We are continuing to monitor this incident and are still seeing intermittent errors due to latency. Overall, the system continues to show a much quicker response time than it did earlier in the day. The vendor is continuing to work on fixes to bring everything back to an operational status.

We are continuing to monitor this incident. The vendor is continuing to work on fixes to bring everything back to an operational status.

The vendor is only seeing a few remaining service errors being logged on their side. They are continuing to address the elevated latency issues while we continue to monitor our API calls.

We are continuing to monitor the incident. We are still seeing improvements as the vendor's system continues to recover. We will provide another update in 20 minutes.

The vendor is reporting that their main functionality has been restored. We are still monitoring the API calls as they are continuing to recover from today's event. These generally take more time to return to normal. We will provide another update in 20 minutes.

We are continuing to monitor the latency issues with our check-in vendor as their statistics are trending in the right direction. Internally, we are intermittently able to check in and have calls, but we are also still seeing the server errors. We do not have an ETA for a fix to share at this time. We will provide another update in 20 minutes.

We are continuing to investigate this issue. We are seeing patients are now able to check in, but are not showing up for the provider queue.

We are currently investigating latency spikes that are preventing patients from checking in.

The vendor who provides check-in notifications had a series of latency spikes that ultimately led to timeout errors. This prevented patients from checking in as the data could not be sent and received within the required window. The vendor implemented a fix at 11:34 AM EST which resulted in their system resolving the latency issues. [[Doxy.me](http://Doxy.me)](http://doxy.me) noticed immediate positive results as the check-in functionality began to work as expected. We did see a few intermittent issues continue as their system caught up from the spikes. They marked their incident resolved on Feb 09, 2021 at 1:17PM EST.

We have continued to monitor this issue over the last few hours. We are marking it resolved at this time as functionality has been restored.

The system has recovered and patients are able to check in at this time. We are continuing to monitor the incident.

We are currently looking into an issue where patients are unable to check in.

**Summary of event**: [Doxy.me](http://Doxy.me) website was failing to load, providing users an "Oops" error message or white screen. **Business Impacts**: Providers and Patients were unable to access the site to be able to have their appointments. **Root cause**: On Wednesday, December 16th around 4pm and Thursday around 11am EST, we had an increase of web traffic that overloaded our servers. Our systems did not add additional servers in time to process the traffic. **How was the issue resolved**: Additional servers were added to production to assist in sharing the web traffic. **Preventative next steps**: We have added more servers, are improving our monitoring, and have reduced internal latency which also increases network capacity.

The incident has been resolved. We will be providing a post mortem in the next couple days.

Users are able to log in again. We are still seeing intermittent system slowness and are continuing to research the issue.

We are currently looking into this issue.

**Summary of event**: [Doxy.me](http://Doxy.me) website was failing to load, providing users an "Oops" error message or white screen. **Business Impacts**: Providers and Patients were unable to access the site to be able to have their appointments. **Root cause**: On Wednesday, December 16th around 4pm and Thursday around 11am EST, we had an increase of web traffic that overloaded our servers. Our systems did not add additional servers in time to process the traffic. **How was the issue resolved**: Additional servers were added to production to assist in sharing the web traffic. **Preventative next steps**: We have added more servers, are improving our monitoring, and have reduced internal latency which also increases network capacity.

This issue has been resolved. we will provide a Post Mortem in the coming days.

A fix has been implemented. We are continuing to monitor the situation and will provide a Post Mortem in the coming days.

This is a large system issue and we are continuing to look for a resolution. The page is loading very slowly, and will load after two minutes. We do not have an ETA at this time.

We are currently investigating an issue with Doxy.me's webpage not loading.

Apple has deployed an update to iOS and MacOS, upgrading to version 14.3 and 11.1 respectively on December 14th, 2020. This update has resolved the audio bug introduced by the recent updates to Safari. We recommend you update your browser to the latest version to ensure you are not impacted by the audio bug. You can do that by following these instructions for iPhone and iPad. iOS Devices: https://support.apple.com/en-us/HT204204 MacOS: https://support.apple.com/en-us/HT201541

We recently identified an issue in Safari version 14.0.1 that is causing Static/clicking noises for the first minute or so, and then it goes to normal. It is not just us impacted, but all webRTC applications. We are waiting for Apple to release version 14.3 that fixes the issue. Here are the best known workarounds: 1) Start chat with the patient and tell them the audio just needs 30-60 seconds to resolved. 2) Ask if they have another device at their disposal and use one of below browsers: For Windows Laptop/Desk: Chrome, Edge, Firefox (No Internet Explorer) For Mac Laptop/iMac: Chrome, Safari, Firefox For iPhone/iPad: Safari Only at this time (This may have brief audio clicking) For Android Mobile/Tablet: Chrome, Edge, Samsung, Firefox (No I.E.)

**Summary of event**: From 12:45 EST to 13:10 EST on 12/01/2020, [doxy.me](http://doxy.me) customers experienced server errors, inability to log in, or load the page. **Root cause**: There was an internal service that temporarily needed 70ms instead of 1ms for its requests due to a failover.  While the difference between 70 ms and 1 ms may seem small, this internal service receives a large number of requests in the middle of the day which caused a memory shortage. **How was the issue resolved**: We routed all traffic to another set of servers. We moved the traffic back and increased server capacity and memory to accommodate the additional requests on 12/02/2020. **Preventative next steps**: In the next several weeks, we will fail back to the internal service that had the issue and the load will be reduced. The event allowed us to refine our scaling capabilities to respond better to service or server failovers.

The incident has been resolved.

Doxy.me is back up at this time. We are continuing to monitor the situation.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating the issue. Text and email alerts are still being delivered, but the in-browser notifications are not consistently displaying.

This incident has been resolved.

A fix for the Firefox Audio and Video issue has been deployed. We are continuing to monitor the results.

Our real time communications vendor is testing a fix that would resolve the audio and video issues when using new versions of Firefox.

We have identified the underlying cause for these issues, which are related to the browser update released on 8/25. Our third-party provider has acknowledged the issue and is working on a resolution.

We are continuing to investigate this issue.

We are continuing to investigate this issue caused by some of the changes introduced with the Firefox 80.0 update. There has been a temporary workaround identified: 1) You can open your Firefox browser and search "about:config" in the url. Then Select Accept the Risk and Continue button on page 2) From there you can find and update by typing in "media.peerconnection.video.use_rtx" then click the arrow toggle to switch the setting to false to resolve this issue. We are continuing to look into a permanent fix. Google Chrome and Microsoft Edge are also viable options if you are unable to change these settings on your browser.

We have identified an issue with audio and video not working when using Firefox 80.0. Please use Google Chrome while this is being resolved.

* **Summary of event**: On Thursday October 8th, [doxy.me](http://doxy.me) experienced roughly 90 minutes of client downtime. * **Business Impacts**: [[Doxy.me](http://Doxy.me)](http://doxy.me) waiting room pages failed to load. Users saw a blank screen, and were ultimately redirected to a “clinic does not exist” error message. * **How was the issue resolved**: We stopped and restarted the [doxy.me](http://doxy.me) services. This solved the immediate issue and service was restored. * **Root cause**: Connections to one of our vendors were staying open for too long without timing out. This used up the available connections until our service was no longer able to respond. * **Preventative next steps**: A solution is in progress and will be included in a future release.

We are marking this issue as resolved and we will post a full root cause analysis in the coming day.

A fix has been implemented and the doxy.me application is now becoming accessible. We are continuing to monitor closely.

We still do not have the underlying cause identified. The doxy.me application is down and neither providers nor patients can access it. Our engineering team is working on this. We will continue to post updates frequently, but we still do not have an estimated time to resolution.

We are continuing to investigate this issue. There is no time estimate for recovery at this moment.

We are continuing to investigate this issue.

We have identified an issue with loading a blank page at this time and are working on a resolution, we will post updates here.

This incident has been resolved.

A planned deployment is taking longer than usual. Ability to login and access Doxy.me is affected.

This issue is resolved. A Root Cause Analysis will be posted after some further investigation.

A fix has been implemented and we are monitoring the results.

The system is not letting patients check in on professional accounts who have enabled a room passcode. We've identified this issue and are implementing a fix. A workaround in the meantime is to disable the room passcode.

We have confirmation that the DNS lookup failure issues from a provider were confirmed resolved around 7h30 EST. Customers should not be facing more issues derived from those. Please reach out to support@doxy.me if you have any further questions. This incident has been resolved. For further details please see: https://status.tokbox.com/

Update - The issue in the US east coast area may impact calls and they may need to be tried several times. We are monitoring the situation and will update promptly. If you have any questions please reach out to us at support@doxy.me , you can also see updates here: https://status.tokbox.com/

We are currently experiencing some degradation of service in the US east coast area. We are monitoring the situation and will update consequently

Our network test that runs in the background would fail when iPads would “request desktop website” while on a mobile device. We had to catch this particular case when iPads were using the “desktop” version of the website. In the future we will ensure this “desktop” website on mobile goes through regression testing.

We've been monitoring this problem after our fix and it is now resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We are continuing to monitor for any further issues.

Cloudflare has implemented a fix and they have provided this update: "This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some potions of the network to not be available. We believe we have addressed the root cause and monitoring systems for stability now." Doxy.me will continue to monitor the situation for any lingering issues with any of our service providers.

We have been notified from an upstream service provider that the issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating an issue impacting access to doxy.me at this time.

We are not seeing any more errors related to this issue and are marking it as resolved. Although we will be making an RCA in the coming days when we have more information to share.

We noticed there were delays to our backend server cluster so we added additional capacity and it seems to have resolved the problems. We are continuing to monitor while we look into the root cause and will update here accordingly.

We are getting reports of providers getting blank pages and problems with being able to start or stay on call. We are investigating and will update with more information shortly.

We rolled back the particular changed that caused the issue. Please refresh your page if you are still experiencing this problem.

The issue has been identified and a fix is being implemented.

We are investigating an issue where a person using the Chrome browser, cannot hear a person using the Firefox browser. We will update here in the next hour with any updates as to the root cause. In the meantime, if both people use Firefox, or both people use Chrome they are able to hear each other fine.

From Vonage: ### Public PM. IM-539 - 2020-05-27 - DNS Misconfiguration **What Happened** Starting on May 27th, OpenTok clients using a given set of API servers received private IP addresses in the STUN, TURN/UDP and TURN/TCP server configuration, for some of the servers in EAST and WEST coast regions making those protocols effectively unusable by clients on those servers. Only TURN/TLS was accessible for those cases, as clients receive the server domain names in that case, instead of IP addresses. Clients using those API servers had to resort to TURN/TLS, which takes longer to establish, making more probable a timeout during negotiation, and requires more CPU resources on the servers. Both factors resulted in an increased error rate with code 1554 on clients using a given set of API servers. An increase in the CPU load of TURN servers was detected on May 29th, a temporary fix was deployed on May 29th, while the root cause was being investigated. This reduced the error rates to values close to normal ones. Independently a faulty TURN server involved in the investigation resulted in a spike in errors on June 6th-8th. The faulty server was removed on June 8th, bringing error rates back to close to normal levels. Root cause investigation progressed, final remediation was put in place June 12th. Failure rates have been normal since June 8th. **Root Cause** Root cause was a DNS misconfiguration after several API servers were moved to a different cloud provider, starting on May 27th. As a result of that suboptimal configuration, API servers resolved the domain names for the same region TURN servers as private IP addresses. Clients using those API servers, and in the specific region of the affected TURN servers could not use STUN, TURN/UDP or TURN/TCP, having to resort to TURN/TLS. **Impact** Clients connected to several API servers in the US east and west coast, and using TURN in those regions respectively for proper connectivity, had an increase in error rate for “1554” errors, related to timeouts in media connection establishment during May 27th-28th and June 6th-8th. Once the connection was established, there was no further impact for those clients. Clients using relayed \(point to point\) sessions saw a higher relative impact as those scenarios are more likely to require STUN/TURN. Clients not using TURN were not affected. Clients connected to distinct API servers from the affected TURN servers region were not affected. Clients in other regions were not affected. **Preventive actions** Update server migration processes with additional configuration and testing steps to enhance validation of DNS configuration and TURN scenarios. Increase monitoring and tune alerting thresholds for client failure rates. Review decision criteria to raise incidents and perform Incident Management training across teams.

We have received the Root Cause Analysis from Vonage on this topic and are going to mark it as resolved. We will post the RCA immediately after this post.

Starting around June 1st, doxy.me customers on restricted networks (behind firewalls), began seeing elevated failure rates when trying to connect to patients, mostly on mobile devices and/or networks. One of our service providers, Vonage, told us today they implemented a fix and we should start seeing these errors decrease. We will continue to monitor and post updates here as they come. I would like to take a moment to address the lack of communication on this topic. As is evident from our discussion board ( https://discuss.doxy.me/t/all-my-providers-having-issues/30480/50 ) lots of people were seeing these failures and we did not post anything on our status page. I would like to assure our customers that we believe in transparency and good communication. In this regard, I would say we failed. Going forward, we are going to be making a few changes with how we confirm systemic issues and how we communicate these to our customers. I'm troubled by the amount of frustration this caused our customers and their patients; the slow reaction to identify the issue; and the poor communication. We are working very closely with our vendor (Vonage in this case), to be better informed and updated with important information so we can communicate that clearly. We are also going to start offering IP Whitelisting in the coming weeks to better ensure traffic on restricted networks is properly handled. If you are interested in IP Whitelisting please reach out to sales@doxy.me or your Customer Success Manager for more information or to request early access. We will be posting updates and a Root Cause Analysis here once we get them.

We have transitioned to using the new shortcode and so far it looks like things are going well with this functionality (the outage earlier today was the result of a different change and RCA to follow later today). We are going to mark this as resolved. For any questions or comments don't hesitate to reach out to us here: https://discuss.doxy.me/t/consolidating-sms-notification-numbers-to-us-shortcode/31472

Tomorrow, June 16th, 2020 we will be replacing our US long codes and toll-free numbers with a shortcode. Specifically, with the shortcode: 675-79 This number is the number providers will get notified from that a patient checked in, and if a provider chooses to invite their patient via SMS. There are a few instances when a long code will still be used: * The receiving number cannot receive from shortcodes - it will fall back to a long code * If the receiving number is outside the USA, it will fall back to a local shortcode or US long code The standard START/STOP/HELP phrases allow you to control messages you get from this number. Once we deploy and all looks well, we will make another update here to resolve the issue. Please reach out to support, or post here for any questions or comments: https://discuss.doxy.me/t/consolidating-sms-notification-numbers-to-us-shortcode/31472

I just received a call from a representative from T-mobile and he was incredibly helpful. He said they would implement a short term fix until we can be migrated to a dedicated short code. I can’t thank them and you all enough for making this happen. He said it might work for AT&T and Verizon as well. I tested a number from both and it went through. I will update you as we make progress on a longer term solution.

Because of the rapid onboarding of hundreds of thousands of people, the carriers marked our SMS invites spam and are not being delivered by carriers. This is out of our control but we are working directly with our vendor to see if there is any way we can get the spam lifted during this national emergency. We understand this is impacting how you invite patients. There are still ways to get it to them: https://help.doxy.me/en/articles/95870-how-to-invite-a-patient-to-your-room You could also use a service like Google Voice to send these out in the meantime: https://support.google.com/voice/answer/115116?co=GENIE.Platform%3DDesktop&hl=en Simply copy the SMS template from Doxy.me and then send it to your patient via Google Voice. You can use your own phone to text them if you don't mind them having your phone number.