Historical record of incidents for Hypercare
Report: "[US and Canada Regions] Partial Service Interruption Due to Third-Party Outage"
Last updateWe are currently experiencing a partial service disruption caused by an ongoing outage with Doppler, one of our third-party service providers. As a result, the following Hypercare features are impacted: - Escalation notifications related to code team activations (note: code team activations remain fully operational) - Schedule change email notifications - Schedule and Equity report downloads - Everbridge file generation We are actively monitoring the situation and will provide updates as Doppler resolves the issue. You can view the status of the Doppler incident here: https://www.dopplerstatus.com/incidents/8jtzf7nrl9tv
Report: "Unable to verify on PIN code page/ biometrics [iOS App Only - CA and US Region]"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The iOS fix has been submitted and approved by Apple yesterday at 1:17am ET. The release is available on App Store for iOS. Users are highly recommended to download the latest version.
We have identified the issue. The team is working on creating a fix.
We are currently investigating reports of users being unable to bypass the PIN code page on the mobile app after entering the correct PIN code or biometrics. The issue is currently being investigated. Users are advised to continue accessing Hypercare via a web browser and/or click "Forget Pin?", continue to log out, and then log into the app again.
Report: "Notification delays [CA Region Only]"
Last updateThis incident has been resolved.
Hypercare message notifications experienced delays in delivery between 11:56 am to 12:13 pm EST. Users may have experienced a notification delay of up to 15 minutes during this time period. The team has confirmed normal operations have resumed. We are continuing to monitor the notification service.
Normal operations of message notifications have resumed. The team is actively investigating the root cause.
We are currently investigating reports of Hypercare message notifications experiencing delays in delivery. Our team is actively investigating the issue. This is only affecting users in Canada.
Report: "Notification delays [CA Region Only]"
Last update**What Happened?** At 18:50 EST, our monitoring system flagged a potential issue with the notification service in Hypercare. An immediate investigation was launched by the team. Testing concluded the issue was isolated to users based in Canada. **Impact** The notification service interruption affected several critical functionalities within Hypercare, including: delivery of Hypercare message notifications within the platform, outgoing pages and SMS sent through Hypercare and integrations with some third-party services, such as Everbridge. The investigation indicated that messages sent between 18:34 and 19:05 EST experienced prolonged delays of up to 20 minutes in notification delivery. **Resolution and Next Steps** The issue was resolved at 19:05 EST. Our team is continuing to monitor the notification service and conducting further investigation on the root cause.
This incident has been resolved.
We are continuing to monitor for any further issues.
Delivery of notifications have resumed normal operations. Our team is continuing the investigation and conducting additional testing.
We are currently investigating reports of Hypercare message notifications experiencing delays in delivery. Our team is actively investigating the issue. This is only affecting users in Canada.
Report: "[CA Region only] Unable to Initiate Hypercare Login"
Last update### **What Happened?** An emergency system update was carried out in the Canadian region starting at 22:15 \(EDT\) on December 19, 2024. An issue was detected at 23:22, where the data migration part of the system update resulted in authentication services to become temporarily unavailable. ### **Impact** Any services requiring authentication were temporarily unavailable. This impacted the following Hypercare services: new logins, virtual paging and integrations. For users who were already logged into Hypercare, there were no interruptions to services. ### **Resolution and Next Steps** We have identified key steps to prevent a recurrence: 1. **Additional testing workflows and simulation:** Numerous checkpoints, test cases, and forced stop points were added for both simulation testing and deployment procedures to identify and mitigate any potential for service interruptions. 2. **Enhanced Monitoring:** new automated monitoring deployed to simulation environments to provide more granular information on migration performance . We sincerely apologize for the disruption and remain dedicated to ensuring a reliable, high-performing platform for all users.
This incident is now resolved.
The issue has now been resolved. Authentication, integrations, and our virtual pager is behaving as normal. Our team is currently monitoring.
Our team currently has identified the issue and is currently identifying the best path for resolution. Our team has also identified features that depend on authentication could be affected including virtual pagers and integrations that depend on such.
There is interruptions on Hypercare new login sessions in CA Region. Users who are logged in already will stay logged in. We're currently investigating the issue.
Report: "[US Region only] Interruption to all Hypercare services"
Last update**Overview:** This incident was caused by a technical constraint related to database query limits and a recent release aimed at improving the search feature on Messaging and Contacts. **What Happened:** On December 11, 2024, at approximately 11:00 a.m. ET, the platform experienced a surge in concurrent searches across a large user base. This triggered the database's fixed data limit for query execution, leading to search processes pooling and eventually crashing. The issue was observed in the United States, where higher user loads exceeded expectations. Canadian operations were unaffected. **Impact:** From 11:10 a.m. to 11:50 a.m. ET, Hypercare Messaging and Virtual Paging services were non-operational, and on-call schedules loaded slowly. **Root Cause:** The root cause was the interaction between the database’s inherent query size limit and the recent enhancement to our search functionality on Messaging and Contacts. Under high concurrent usage, query sizes exceeded the database's fixed threshold, resulting in the experienced performance issues. **Resolution:** The search enhancement release has been rolled back to stabilize the system and restore full functionality. All services are now operational, and additional safeguards have been implemented to prevent similar issues in the future. We sincerely apologize for the disruption and remain dedicated to ensuring a reliable, high-performing platform for all users.
The issue is resolved. All services operational now.
Services are recovering. Messaging and Virtual Paging services should be operational. We're closely monitoring the issue.
The issue has been identified and a resolution is being worked on. Users can log into Hypercare and the On-Call schedules are loading very slowly. Messaging and Virtual Paging services are currently still non-operational.
We are currently investigating reports of all Hypercare services being unaccessible in the US region. Our Engineering team is currently working on identifying the root cause.
Report: "[CA & US Region] Users being logged out of Hypercare app"
Last updateUpdates to our iOS app on November 20, 2024 have addressed the issue of Users being logged out from the Hypercare app. Users are being encouraged to ensure their iOS app is updated to version 18.3.16 or later.
Minor enhancements were made to our iOS app on November 20, 2024 to address potential causes of Users being logged out from the app. Reports cases of logs outs have decreased and Users are being encouraged to ensure their iOS app is updated to version 18.3.16. Our team will continue to monitor the issue and report further updates as they become available.
Minor enhancements have been made to our iOS app. Please ensure Users on an iPhone update their app to version 18.3.16, which is now available in the App Store.
We are currently investigating reports of users being routinely logged out of their Hypercare app on Android and iOS devices, impacting their ability to receive app push notifications. Reports have been received from users in both CA and US region. Our Engineering team is investigating potential root causes for this unexpected behaviour on some user devices. Users are being encouraged to turn on Safeguard Notifications in Hypercare should they wish to receive a back-up notifications. https://support.hypercare.com/hc/en-us/articles/5376178050445-Safeguard-Notifications-FAQ
Report: "[CA & US Region] Android only - Inconsistent Notifications"
Last updateWe've been monitoring the notifications and confirmed that the incident has been resolved.
Android notifications seem stable. We are continuing to monitor the behaviour of Android notifications.
The fix has been deployed and we're currently monitoring the Android notifications behaviour.
The issue has been identified and engineering is deploying a fix.
We have received reports on Android users not receiving notifications. The engineering team is currently investigating.
A fix for the root cause of the issue has been implemented. It's backwards compatible and users do not need to update the Android app. We're currently doing more testing and monitoring the performance to ensure notifications are performing as expected.
Our team has identified the issue and shipped out the fix on the server side. Unfortunately, we will also need to release a new version of the Android app to fully resolve the issue. Our team is aiming to release the new version of Android by September 6th to account for sufficient time for testing and quality assurance purposes.
The potential cause for inconsistent push notifications on Android device has been identified, and the fix will need another 2 days to be implemented. In the meantime, Android users are advised to set up the safeguard notifications (FAQ: https://support.hypercare.com/hc/en-us/articles/5376178050445-Safeguard-Notifications-FAQ) or revert to manual escalation if necessary.
We have received reports of inconsistent notifications on Android devices. We're currently investigating the issue.
Report: "[CA & US Region] Delays in notification delivery"
Last updateThe incident has been resolved.
A fix has been implemented and notification delivery services should be recovering.
We're observing delays in notification delivery. We're currently investigating this issue.
Report: "[US Region Only] Interruption to all Hypercare Services"
Last updateThis incident has been resolved.
A fix has been implemented. The team will continue monitoring the platform to ensure all services have stabilized.
Hypercare services continue to be interrupted in the US region due to ongoing maintenance work. The Engineering Team has identified the issue and a fix is underway.
Report: "[US and Canada Regions] Virtual Paging Service Interruption"
Last updateThe issue has been resolved
The issue has been identified and a fix is being implemented.
We are currently investigating an issue related to our virtual paging services in both Canada and US regions. All other Hypercare services, such as Messaging, On-Call and Team Activations, remain operational.
Report: "[CA & US Region] Login flow is not functioning as expected"
Last updateThis incident caused an unplanned interruption to organizations that use the Single Sign-On \(SSO\) service to log into Hypercare via web browsers. This incident occurred during routine maintenance on our web client platform, which was intended to improve overall performance. **What Happened:** On July 30, 2024 at approximately 4 p.m. EST, our team deployed an update to the web client platform. This update, designed to enhance performance, had been thoroughly tested in our Staging environment without any issues. However, upon release to the Production environment, an unexpected outcome caused an interruption to our SSO service. **Impact:** As a result of this interruption, Users experienced difficulties with new Login attempts via SSO on all web browsers from 4 p.m. to 4:50 p.m. EST. SSO Login attempts on our mobile applications and users already logged into the platform were not impacted. **Root Cause:** The root cause of the issue was an unforeseen interaction between the updated web client platform and the Production environment’s SSO configuration. This interaction did not present during our Staging environment tests, leading to the unexpected service disruption. We understand the inconvenience this caused and sincerely apologize for any disruption to your operations.
The fix is deployed and users with SSO should be able to login now.
There is an issue with logging in through single-sign-on (SSO) for new sessions. Users who are already logged in are not impacted. Issue has been identified and the fix is deploying.
Report: "[CA & US Region] Admin Portal - User invitation flow is not functioning as expected"
Last updateThis incident has been resolved.
We have identified issues with user invitation flow on admin portal. Organization administrators may run into issues when inviting new users or re-inviting users to organization. Product team is working on a fix.
Report: "[CA & US Region] Android login flow is not functioning as expected"
Last updateThe fix is tested and released on Google Playstore. Users are strongly advised to update their app to the latest version 14.1.9 to ensure they can login to Hypercare.
The Android app version is currently under testing and review. In the meantime, if users are logged out from the account and not logged into any mobile device, they should receive SMS for every message they receive if they have a mobile number on profile. Users can log into web to check messages until the issue is fixed. Users can also add their phone number to their profile following this instruction: https://support.hypercare.com/hc/en-us/articles/24953957201037-How-to-Add-an-Email-Address-and-Mobile-Number-to-Your-Profile
The issue has been identified and a fix is under testing.
Issue is isolated to Android only.
Users might encounter issue when logging in. We are currently investigating the issue.
Report: "[Web App Only] Overnight Shifts from June 30th Not Showing Accurately"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
[Web App Only] The on-call schedule for overnight shifts within the client web application may incorrectly display "No one is currently on call," despite the possibility that someone may indeed be on call during this period. Recommendation: In the meantime, we suggest using the Hypercare mobile app to access the on-call schedule. Alternatively, for users opting to use the web app, navigate to the "Schedule" tab located to the left of "Messages," select the appropriate department, switch the month to June 2024, and review the overnight shift schedule for June 30th.
Report: "[US and Canada Regions] Virtual Page tied to On-Call roles - Service Interruption"
Last updateThis incident has been resolved
A fix has been implemented and virtual paging services that are tied to on-call schedule roles have been restored. The Hypercare Engineering team will continue to monitor the fix to ensure paging services have stabilized.
We have identified an issue that have impacted virtual paging services that are tied to on-call schedule roles. Virtual paging services that are tied directly to individual user account(s) are not impacted. External callers attempting to dial the virtual pager number may encounter a timeout error, resulting them in not being able to leave a callback number and/or an associated voicemail. External callers can try to call again until they are able to successfully try again. The Hypercare Engineering team is continuing to investigate the root cause of the issue and will provide further updates as soon as they become available.
Report: "[CA & US Region] Login flow is not functioning as expected"
Last updateThe issue has been resolved.
The issue has been identified and we're working on a fix.
Login flow is not functioning as expected. The engineering team is investigating on this issue.
Report: "[US & CA Region Only] Delay in Sending and Receiving Messages"
Last updateAfter monitoring, we have not seen any delays or issues. We are now fully closing this incident.
The engineering team continues to investigate delay and monitor response time. Service is operational.
[CA Region only] Deployment in CA region and might experience a temporary service downtime. The service is now recovered.
We're observing longer processing time in sending and receiving messages in US and CA region. The team is currently investigating the issue.
Report: "Viewing on-call might not load for some sites on web"
Last updateOur team has identified the issue and resolved the issue.
Our team has identified one organization where one of their sites are not viewable on the web platform due to a security configuration. The organization has been notified. We are currently identifying whether this is a issue that affects more than that organization or limited to only one site. There is no issues viewing on iOS or Android.
Report: "[US Region only] Reports of higher latency"
Last updateThe team has completed their monitoring of the slowness experienced across Hypercare services on May 29, 2024. No new issues arose during the monitoring period after the incident.
Slowness experienced across Hypercare services have stabilized and the root cause has been identified. The team will continue monitoring responsiveness of Hypercare services. Intermittent slowdowns may continue to occur as the team continues to conduct tests and make enhancements as part of the resolution.
The team is continuing to investigate reports of slowdown in services for US-based users.
We are currently noticing that certain actions can take quite a long time. We are currently investigating the root cause.
We have received reports of higher latency to perform actions on the U.S. server. We are currently investigating to see the extent of the problems.
Report: "CA Region Only - Notifications delivery interruption"
Last updateThe incident has been fully resolved.
The issue has been identified and notification services have been restored.
Internal alerts identified interruptions to message delivery in the Hypercare across all platforms (web and mobile apps). We are currently investigating and will provide an update shortly. This issue is only affecting Users in Canada.
Report: "[CA and US Region] Hypercare platform interrupted service"
Last updateThis incident has been resolved.
US Region services have been recovered. We're closely monitoring the platform.
Canadian region has been recovered. The team is still recovering the US region.
We're currently experiencing a downtime in CA and US region. We're currently investigating the issue.
Report: "[CA Region Only] Notifications delay in CA region"
Last updateThis incident has been resolved.
A fix has been deployed and it's operational. We're actively monitoring the issue.
We have observed delay in notifications. Engineering team is investigating in this.
Report: "[CA Region Only] Hypercare platform being monitored for interrupted service"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The Engineering team has identified the cause of the 4:06pm EST downtime and a fix is being deployed.
The service has been restored at 4:17pm and is operational now. Engineers are investigating the cause.
Engineers are currently investigating the issue.
The issue has been identified and the Engineering team will continue monitoring the platform.
Internal logs indicated a six minute downtime occurred at 1:26 pm EST and was restored at 1:32 pm EST today for Users in the Canadian Region only. At this time, Hypercare is now fully operational. Our Engineering team is currently investigating the cause of the downtime.
Report: "[US Region only] Reports of Message and Notifications Delay"
Last updateThe team has identified the issue for the message delay and it has been fixed.
We have received reports that message and notifications on U.S. systems are taking longer than expected to process. We are currently investigating the reason and understanding the full impact.
Report: "[US Region Only] Intermittent service interruption"
Last updateThe incident has been fully resolved.
Hypercare's Engineering team will be conducting performance testing between 9:45 p.m. to 11 p.m EST to continue monitoring efforts of the intermittent service interruptions experienced earlier today. Degraded performance may be experienced during this time. For urgent inquiries, please call our Support Team at 1 (877)-379-3533.
The platform is operational and we're closely monitoring the performance in US region. For urgent matters, please call our support at +1 (877)-379-3533.
The degraded performance does not impact individual users. We are still investigating the possible causes.
Unfortunately, we have not fully recovered our service yet. We are still investigating other possible causes as our other fixes have only partially recovered the service but not a full recovery yet.
Users may be able to perform actions as our systems are recovering, we will continue to update until a full recovery has been completed. Some actions performed may still fail until the full recovery is completed.
The issue has now been identified with our engineering team and they are currently attempting to recover the service.
We have received reports that actions on U.S. systems are taking longer than expected. We are currently investigating the reason and understanding the full impact.
Report: "[US Region Only] Reconnecting banner appearing in Web platform"
Last updateThe issue was identified and resolved. All systems are fully operational.
We are investigating reports of a "Reconnecting" banner appearing on the Web platform of Hypercare in the US region only. All systems are operational on web and mobile platforms. This is only impacting customers in the US.
Report: "[CA Region Only] Reports of intermittent interruption and slowdown to service being monitored across all platforms"
Last updateThis is incident is resolved. Our Engineering Team continued monitoring on March 5 after 15:46 when the root issue was identified and performance returned and continued at full operations with no concerns.
The issue was identified with our Authorization service. Our Engineering team will continue monitoring the platform to ensure optimized performance.
We are currently investigating reports of intermittent interruption and slowdowns to service on Hypercare across all Web and mobile client and admin platforms. Performance and functionality of Hypercare has resumed and our Engineering team is actively investigating to understand the cause of the slowdowns. This is currently only affecting users in Canada.
Report: "[US Region Only] Downtime reported at 4:19 PM EST and being monitored"
Last updateOur Engineering Team has monitored the enhancement made to our Authorization service in the US and this incident is now resolved.
US Region Only: Hypercare's internal alerts captured a 10 minute downtime starting at 4:19 PM EST. All platforms are now fully operational. Our Engineering Team has identified the issue and is monitoring our systems to ensure everything is fully operational.
Report: "[US and CA Region] Hypercare performance being monitored across all platforms (Web and Mobile Apps)"
Last updateA downtime was reported between 4:51 PM and 4:57 PM EST today in our Canada Region and 4:54 PM and 5:04 PM EST today in our US Region across all User and Admin platforms. All systems are now fully functional and our Engineering team is investigating the cause of the downtime. They will continue monitoring the platform to ensure optimal performance.
Report: "[CA and US Region] On-Call Schedule Email Notifications"
Last updateEmail notifications are now fully functional.
Our Engineering team has been informed of an influx of Hypercare email notifications being sent to users regarding changes to on-call schedules. Email notifications were temporarily turned off in September 2023 due to issues identified with the feature. The team was implementing a fix to the feature today that unexpectedly notified all users of emails notifications that were in the backlog. This has since been stopped and turned off. We apologize for the inconvenience and confusion this has caused. The team will continue to monitor and investigate the issue and thank you for your patience during this incident.
Report: "[US Region Only] - Issue with notifications of messages on Web platform"
Last updateMessage notifications on the Web platform have been fixed and are now fully operational.
We are currently investigating an issue with Web notifications not working as expected. iOS and Android message notifications are operational. Our Engineering team has identified the issue and are working on a fix.
Report: "[Canadian Region] - System slowdown"
Last updateNo more slow downs have occurred. Closing the issue as our suspected issue seems to the be culprit.
We have seemed to identified the issue. We will continue to monitor to ensure it's stable.
Our engineering team received a system downtime notification from our monitoring system at 11:41:46 am for our API being down for the Canadian region. The system was recovered by itself by 11:47:46 AM. We are currently investigating why the system slowed down, to ensure this does not happen again.
Report: "CA and US Region - On-Call shifts and roles for Dec 31 not showing in the iOS app"
Last updateThis incident has been resolved an the iOS app is showing the on-call shifts for Dec 31.
An issue has been identified with on-call shifts and roles not showing in the iOS app. The Android app and Web Browser version of accessing the on-call schedule are functioning as expected without interruption.
Report: "[CA and US Region] Admin Portal - Department and Site User list not loading"
Last updateThis issue is now resolved. This was due to a recent update, which the engineering team has now resolved the issue on the update. We will continue to monitor to ensure this update does not have anymore side effects.
We are currently investigating an issue with the user list in departments and sites not loading. Users lists in an Organization are loading with no issues and departments can continue scheduling users into on-call schedulers in the Admin Portal. Our Engineering team is currently investigating the issue to identify the root cause.
Report: "[Canadian Region] - Reports of slowdown on admin portal"
Last updateOur team has reverted the change and everything should behave normally now. We will be shipping this feature out in the future once we identify the performance issue associated with this new feature.
We launched a new feature to allow our customers to sync up their internal directory with Hypercare including custom labels. However, the new feature is causing some performance slowdowns of admin portal with large organizations over 2K members potentially not loading. We are currently investigating and we may revert the feature update.
Report: "[Canada] Notification of real-time connection issues"
Last updateThis issue has now been resolved. This is due to the server restarting, which is now resolved. We are investigating why this occurred to ensure there is not a deeper issue.
Our team has noticed elevated errors and a disconnection banner. We are investigating this issue. We have not gotten any support calls on this.
Report: "[CA and US Region] Virtual pagers connected to on-call schedule roles not working"
Last updateThe incident has been resolved. The virtual pagers tied to on-call schedule roles are now fully operational.
The issue has been identified and a fix is being implemented. Virtual pagers for individual users are not impacted by this issue.
We are currently investigating reports of Hypercare virtual pagers that are tied to on-call scheduled roles are not functioning as expected. This is impacting both Canadian and US Region customers. Our Engineering team is actively investigating the issue. The next update is expected in 30 minutes.
Report: "Reports of on-call and contacts not loading"
Last updateThis issue was due to a recent hot fix to resolve another issue which inadvertently caused this issue of loading contact information from either the on-call page or contacts page.
At 18:40 eastern time, we got a support call that indicated on-call was not loading when clicking on the contacts. It was verified by us that it was also happening to us on the web platform and mobile. Our engineering team was escalated as P0 incident, and a recent change that was pushed out at 18:23 eastern time was reverted. The issue was resolved by 18:44 due to reverting a recent change.
Report: "[All Regions] - Shifts that span into next month"
Last updateIssue was successfully resolved Oct 30 with shifts on the last day of the month validated.
Our engineering team is planning on resolving issues with schedules Oct onwards that miss the last shift of the day in the rare event it's not showing up by Oct 20th prior to the end of the month in order to not have any missing shifts.
We are continuing to work on a fix for this issue.
We have identified that some schedules are missing the last shift of the month if it spans into the next month. We have attributed this issue due to a migration problem. The data is not lost, however our system is not displaying it properly. Our team is actively restoring the last shift of each month from October onwards as a priority and will be restoring the last shift of each month for past schedules in the near future. We are happy to provide past schedules with all shifts displayed properly in an Excel spreadsheet if needed. Please email support@hypercare.com to request the schedule you want. Thank you for your understanding and we will keep you updated with any progress.
Report: "[Canadian Region] - Reports on notifications not being sent"
Last updateThe resolution deployed has been monitored and no further issues were observed. The notification issues on General messages are now fully resolved.
A fix has been implemented and we will continue to monitor for any further issues. Notifications for General messages across web and mobile platforms should be functioning as expected.
Notifications of Priority messages (sent with Urgent and STAT) are not impacted by this issue and are notifying Users as expected. Only General messages are impacted. Our Engineering team is working on a resolution.
The issue has been identified and is confirmed to only be affecting the Canadian Region. Our Engineering team is actively working on a resolution.
We have received reports that users are not receiving or receiving delayed notifications of Hypercare messages. We are currently investigating this issue. We will be updating this every 15 minutes.
Report: "[U.S. Region] Login Issue"
Last updateA fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We have received reports that users are currently experiencing issues logging in affecting our US region. Our team is investigating and we will be providing an update every 15 minutes.
Report: "[All Regions] On-call paging issue"
Last updateThe fix deployed is functioning as expected and the issue has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue related to on-call paging not being operational for users located outside of Eastern Daylight Time (EDT). Our team is investigating and we will be providing an update every 30 minutes.
Report: "[All Regions] Issues with syncing personal schedules to external calendars (Google, Outlook, Apple)"
Last updateThere has been no further issues and users who has reported inability to sync their schedules to their preferred calendar of choice has confirmed it is working.
We have resolved the issue as of 13:16 by enabling the feature after resolving the performance issues. If you notice your calendar is not syncing up, you may need to do a one time removal of the calendar and re-import using the iCal link. We are still monitoring this feature to ensure there are no further issues.
We are some reports with syncing to personal calendars (Google, Outlook, Apple) from Hypercare Personal Calendars ical sync link. We have identified the issue and we are hoping to resolve the issue. We will continue to update this status page as the situation changes. During this time, personal schedules on Hypercare will not be able to be synced to calendar of choice externally to Hypercare.
Report: "[All Regions] Elevated errors on removing users from admin portal"
Last updateThis issue is now resolved. Administrators should be able to remove users. Upon investigation, we realized that this has to do with a recent feature where we automatically remove corporate emails when the user is removed. Unfortunately this introduced a bug where users with corporate emails could not get removed.
We have received a report that an admin could not remove users from the admin portal. We have been able to reproduce the issue, however this is only affecting a limited amount of users that cannot be removed. Our product team is currently investigating the root cause. In the meantime, a workaround identified by our support team involves removing the users' email address which allows the admin to remove the users from the organization.
Report: "[All Regions] Inability to export the schedules into Excel Format"
Last updateThis issue is now fully resolved. Administrators can download and export the entire schedule. We apologize for this issue happening in the first place.
The issue has now been identified and a fix is currently underway.
Currently, Hypercare schedules cannot be exported into Excel format and it currently shows up blank. This has been reported recently by several users. This is affecting two main features of Hypercare: 1. Exporting schedules from administrator portal into Excel format 2. Getting email notifications on schedule updates and changes as it attaches an Excel sheet This issue is now escalated to engineering and identified to be a high priority item. We will continue to update everyone on a routine 30 minute basis.
Report: "[All Regions] - Issue with application loading and messages sent delay intermittently"
Last updateThe Engineering Team continued monitoring the performance of Hypercare today and it has returned back to being operational after identifying the causes of the intermittent delays. We will be temporarily disabling notifications of on-call schedule changes and the conflicting & warning shifts feature when building on-call schedules in the Admin Portal until further notice. The return of these features will be communicated to Admin Users once available. Thank you for your patience during this incident and we apologize for any inconveniences caused.
After the fix that was deployed yesterday, the performance and functionality of Hypercare has returned to being operational. Some features still remain temporarily unavailable. We will be temporarily disabling notifications of on-call schedule changes and the conflicting & warning shifts feature when building on-call schedules in the Admin Portal. The team will continue monitoring the performance and functionality of Hypercare with these temporary change. We sincerely apologize for the inconvenience.
The Engineering Team has identified the causes of the intermittent delays and interruptions to accessing the Hypercare Client and Admin platforms. We will be temporarily disabling the conflicting & warning shifts feature when building on-call schedules in the Admin Portal. The team will continue monitoring the performance and functionality of Hypercare with this temporary change. We sincerely apologize for the inconvenience.
A possible cause of the intermittent delays on accessing Hypercare has been identified and our Engineering Team is working on a resolution. The team will continue to monitor the platform. We apologize for the inconvenience this has caused.
The Engineering Team is continuing to investigate this issue and the root cause is being identified. The next update is expected in the next hour and we apologize for the inconvenience.
The Engineering Team is continuing to investigate this issue and the root cause is being identified. The next update is expected in 30 minutes and we apologize for the inconvenience.
The Engineering Team is continuing to investigate this issue and the root cause is being identified. The next update is expected in 30 minutes and we apologize for the inconvenience.
We are receiving reports that Hypercare is not loading and/or messages sent are being delayed. Interruptions are intermittent and our team is currently investigating. The next update is expected in 30 minutes.
Report: "[Canadian Region] Issue code team activations feature"
Last updateThis incident has been resolved and code team activations are now fully operational. We apologize for the interruption this incident caused and our team will be reaching out with more communications on a debrief of the incident.
We are continuing to monitor for any further issues.
A fix has been implemented and our team will continue monitoring that code team activations are fully operational.
The issue has been identified and the Engineering team is working on a fix. An outage remains on Code Team Activations.
We are currently investigating reports of code team activations currently not being functional. Our Engineering Team is actively investigating the issue. This issue is only impacting users in Canada. All other functionality in the on-call schedules remain operational.
Report: "[North America] Delivery issues with SMS"
Last updateAs of 10:44 PM EST, Twilio has confirmed all services to be completely operational.
We continue to monitor this situation. Twilio has not given an all clear on this issue however based on our audit trail, no SMS has failed to be delivered from Hypercare system through Twilio since 20:40 EST. We will continue to monitor.
Several of our users have identified that SMS delivery via Hypercare has not been delivered. Upon investigation Twilio, our third party provider, are having some issues with notification delivery. It seems like the issue was identified by Twilio from 20:15. From our side, failures started occurring prior at 20:04 but no SMS failures since 20:40 with 100% delivery of SMS since then. From our side, we will continue to flag this component as partial outage until Twilio confirms their system has no more issues. This does not affect any usage of paging via Hypercare to physical pagers, nor virtual pagers. This is only limited to SMS (sending a non-Hypercare message) via Hypercare to a user's cellphone.
Report: "Canadian Region - Successful Maintenance"
Last updateWe have been monitoring, with no user complaints and issues. Successful deployment.
The upgrade has been performed successfully with no issues. However some users, you may see a "reconnecting" banner within the application since this was a DNS change. DNS changes can take up to 24 to 48 hours, but it could be shorter or longer depending on various factors. During this time, you should not be expected to miss any notifications however if you are on a screen within the application while it is reconnecting you may have to refresh the screen to see updated changes.
Report: "Hypercare Service Disruption on Web App - Mobile not affected"
Last updateUpon performing security updates, the Hypercare web app began to malfunction. The Hypercare team has isolated the issue and has deployed the fix. If you are still experiencing an issue and have not already logged a case, please contact Support.
There’s a Hypercare service disruption. During a service disruption, end users are reporting that the web app isn't sending or receiving messages. Currently Hypercare Mobile is not affected by this disruption. We’re investigating the issue and an update will be provided in 30 minutes or sooner if we have more information.