Historical record of incidents for ChartHop
Report: "ChartHop is slow to load"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Users report that ChartHop is slow to load, sitting on the "loading bunny" screen for an extended time.
Report: "ChartHop is slow to load"
Last updateUsers report that ChartHop is slow to load, sitting on the "loading bunny" screen for an extended time.
Report: "ChartHop app is not loading for some users"
Last updateThis incident has been resolved.
The update has been fully deployed, and we are monitoring for any remaining impacts.
An update is being deployed to address the issue and resolve service for the remaining impacted users.
Some ChartHop users are unable to use the ChartHop application at this time and getting either a blank page or a failure to load. We have identified the issue and working to restore service.
Report: "User interface unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have been made aware that some customers are unable to open the ChartHop user interface. We have identified the issue and a fix is being deploy. ChartHOP API and integrations is unaffected.
Report: "Reports slow"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating a performance degradation for some dashboards and reports. There are no known impacts outside of dashboards and reports.
Report: "Severe Performance Degradation"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We're experiencing intermittent but severe performance degradation across the platform.
Report: "Degraded performance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
ChartHop Engineering is investigating degraded performance in the ChartHop API that is impacting the user interface. A solution has already been identified and is in process of testing and deployment to our fleet.
Report: "Compensation Reviews are not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified some additional issues impacting the preview and testing of Compensation Reviews. Additional work is in progress.
We are continuing to monitor for any further issues.
A fix has been deployed and we are monitoring the situation.
The Compensation Review API endpoints and UI elements are currently erroring out. The team has investigated and identified the cause and is working to deploy a fix.
Report: "ChartHop Dashboard occasional failures"
Last updateNo new CloudFront errors have been reported.
CloudFront errors appear to have stabilized and we are not seeing continued errors accessing the ChartHop Dashboard at this time.
The ChartHop Dashboard is occasionally failing to load with various error messages from AWS' CloudFront service. This appears to be an AWS issue as being reported by multiple services on Twitter and other sources. ChartHop has reached out to AWS for assistance and status.
Report: "Reports service experiencing delays"
Last updateThis incident has been resolved.
Reports are now working again.
We are currently experiencing slowness when generating reports. We have identified the issue and are working to resolve this.
Report: "Band pages are not loading"
Last updateWe have resolved the issue, and Bands pages are now rendering.
We have identified the issue, and a fix is being rolled out.
We are continuing to investigate this issue.
Currently, our Band pages are not loading. The team is investigating and will update you shortly.
Report: "API failures in all regions (US, Europe, and India)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Engineering has identified the root cause of the performance degradation and a fix is in process.
We are seeing degraded performance on our API servers, leading to occasional errors seen in the User Interface. Engineering is investigating.
Report: "Degraded performance for report generation."
Last updateThis incident has been resolved.
We have implemented a fix and are monitoring the system.
We are experiencing slowness in report generation. The investigation is ongoing, and we hope to have this resolved shortly.
Report: "UI failing to load"
Last update# Summary An upgrade to a third-party component contained an unexpected additional header value. The header value was not accepted by our API servers leading to a [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) error. # Details One of our third-party javascript components introduced support for the [baggage](https://www.w3.org/TR/baggage/) HTTP header. The header was not accepted by our cross-origin allowed headers policy and the Access Control headers were not returned to prevent potential security issues. Browsers rejected the API request from the UI due to the lack of Access Control headers. # Impact The UI was inaccessible to all customers during this outage. API requests were still working and data was still ingested into the system \(e.g. Payroll, ATS, and Equity.\)
UI failing to render for all customers
Report: "Degraded Performance"
Last updateSummary On Thursday, January 13th, 2022 between 12:43 PM EST to 6:16 PM EST, several areas of the ChartHop platform was experiencing significant slowdowns due to a high volume of traffic in the database. Impact The impact was slowness and inability to use the platform its full functionality for approximately 5.5 hours. Resolution ChartHop’s DevOps teams increased the size of the server to process the backup traffic which restored the full functionality of the platform. Steps of prevention Install more indexes that will prevent future traffic issues on our servers. Additionally, introduce roadmap items on the data model to reduce reliance on specific servers to further improve performance.