Historical record of incidents for Root
Report: "Policy notes"
Last updateUnable to create or view policy notes
Report: "Policy notes"
Last updateUnable to create or view policy notes
Report: "Document creation delay"
Last updateThis incident has now been resolved. Customers may have experienced up to one hour's delay in receiving policy and terms documents.
We are currently investigation the issue.
Report: "Product module code timeouts"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Product module code functions timing out
Report: "South African Multi-tenant degraded performance"
Last updateThe issue has been resolved.
Maintenance Update Action: The affected indexes have been rebuilt. We are thoroughly testing the system to confirm the improved post-maintenance performance.
Maintenance Update Action: We are currently performing maintenance to optimize our system performance. Our team is actively rebuilding the affected indexes. This action addresses the root cause of the performance degradation issue.
South African Multi-tenant is stable and we are continuing to monitor the performance
South African Multi-tenant is stable and we are continuing to monitor the performance
South African Multi-tenant is stable and we are continuing to monitor the performance
Incident: Slow API Response Times for South African Multi-tenant Cause: The core issue was identified as a performance degradation in our database after a recent version upgrade. Action: We have added additional resources to maintain an acceptable level of system performance for the day and have planned after-hours maintenance to re-build the affected indexes.
Incident: Slow API Response Times for South African Multi-tenant Issue: Our South African Multi-tenant service is currently experiencing slow API response times. This may lead to delays in data retrieval and other dependent operations for our users, we are currently investigating the issue
Report: "Majour outage"
Last updateThis incident has been resolved.
The issue has been addressed and we are continuing to monitor the issue
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "Dashboard outage"
Last updateThis incident has been resolved
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Platform emergency maintenance"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Performing emergency database upgrades
Report: "Major outage in our API"
Last updateThis incident has been resolved.
We've resolved the issue and all systems are operational again. We are still monitoring
We are continuing to investigate this issue.
We have observed to a network connection issue affecting our workers. We're investigating
Report: "Elevated API Errors"
Last updateThis incident has been resolved.
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "Elevated API Errors"
Last updateThis incident has been resolved.
This incident has been resolved.
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "Additional platform optimisation"
Last updateThe scheduled maintenance is completed, and the Root platform is fully operational.
The scheduled platform maintenance is taking longer than anticipated. The Root platform is currently still unavailable. We are working on restoring normal operations as soon as possible.
Report: "Peach Payments CDV Temporarily Offline"
Last updatePeach Payments CDV (Check Digit Verification) for bank account number checks (used when issuing debit-order policies from the dashboard) temporarily went offline from 17:11, when we first detected it, to 17:30 when it came back online. This has been resolved and we are closely monitoring the situation.
Report: "2FA SMS messages delayed"
Last updateUser logins prevented by delayed 2FA SMS messages. Due to a backlog in queue infrastructure SMS OTPs were severely delayed.
Report: "Cloudflare experiencing network issues"
Last updateThis incident has been resolved and reported stable by Cloudflare: https://www.cloudflarestatus.com/incidents/46z55mdhg0t5
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
Cloudflare has been experiencing network issues (https://www.cloudflarestatus.com/incidents/46z55mdhg0t5) which affects certain clients behind their proxy. We've mitigated this issue and are actively monitoring it.
Report: "Degraded performance"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The issue has been temporarily resolved and the platform is stable. We're closely monitoring performance whilst we're implementing a permanent solution. For the time being, certain search functionality has been disabled. We'll keep updating progress here.
We're experiencing degraded API performance. We've identified the issue and are busy implementing a fix. We'll keep updating progress here.
Report: "Delayed sending of emails and SMS'"
Last updateWe're experiencing delayed sending of emails and SMS' through the API. We've identified and resolved the issue.
Report: "Slow response times when adding debit order payment methods during policy issuing"
Last updateThis incident has been resolved.
A payment service provider of Root is currently experiencing slow response times for check digit verification of bank account details. This is affecting the Root Dashboard, and is causing slow response times and/or timeouts when attempting to add a debit order payment method during policy issuing. We are continuing to monitor the situation, and will provide an update once the response times have returned to normal.
Report: "Slow response times when adding debit order payment methods during policy issuing"
Last updateThis incident has been resolved.
A payment service provider of Root is currently experiencing slow response times for check digit verification of bank account details. This is affecting the Root Dashboard, and is causing slow response times and/or timeouts when attempting to add a debit order payment method during policy issuing. We are continuing to monitor the situation, and will provide an update once the response times have returned to normal.
Report: "Issue with fetching policies"
Last updateA permanent fix has been tested and deployed, and the underlying issue is now resolved.
We have identified a permanent fix, which is currently in development and testing. We will keep you posted as we progress.
We have deployed a temporary fix and are monitoring it. The API and dashboard are functioning normally again. If you’re experiencing any issues, please contact us at support@root.co.za We’ll keep you posted here as we progress.
We're experiencing issues on the "fetch policies" endpoint. This is affecting both API integrations and dashboard users. We're aware of the problem and busy working on a fix for the problem. We will keep you posted as we progress.
Report: "Possible issues accessing Root services"
Last updateThis incident has been resolved.
All Root services look to be operating normally. We will continue to monitor the situation.
One of our upstream providers, Cloudflare, is experiencing network-related issues. Customers may have issues accessing the Root platform during this time. We are continuing to monitor the situation.
Report: "Delayed notifications"
Last updateWe experienced a delay in the sending of emails and SMS' on the Root platform. The cause was identified and resolved, and notification sending has returned to normal speeds.
Report: "Degreaded API performance"
Last updateWe experienced degraded performance on the Root API between 14:15 and 14:45. The issue was identified and resolved.
Report: "Peach Payments CDV Temporarily Offline"
Last updatePeach Payments have resolved the issue, and CDV checks on Root have returned to normal operation.
Peach Payments are experiencing system issues, which is impacting CDV (Check Digit Verification) on Root for bank account number checks. This is used when issuing policies from the dashboard with debit order payment methods. We are working with Peach to resolve the issue, and will continue to monitor the situation.
Report: "Request timeouts when creating applications"
Last updateThe team have identified and resolved the underlying issue.
The Root Engineering team have been alerted to an increased number of request timeouts when creating applications on the Root API. We are currently investigating the cause.
Report: "Peach Payments CDV Temporarily Offline"
Last updatePeach Payments have resolved the issue, and CDV checks on Root have returned to normal operation.
Peach Payments are experiencing system issues, which is impacting CDV (Check Digit Verification) on Root for bank account number checks. This is used when issuing policies from the dashboard with debit order payment methods. We are working with Peach to resolve the issue, and will continue to monitor the situation.
Report: "Root platform partially unavailable"
Last updateDuring a routine deployment of new functionality at 18:01 on Tuesday, 23 February 2021, some functionality (including user logins) became unavailable on the Root platform for roughly 20 minutes. The platform returned to normal operation at 18:20. Upon investigation, the team identified that database changes being made as part of the deployment were unexpectedly slowing down or blocking some functionality across the platform. The team have since identified the underlying cause, and are in the process of rectifying this to ensure the issue does not reoccur. Furthermore, the team also noted that Root's public status page did not reflect this incident in a timely manner. We have since identified the cause of this, and are currently altering our processes to ensure clients are proactively notified of such issues in future.
Report: "Delayed Data Exports"
Last updateThe issue has been resolved and data exports are now running as expected.
An issue has been identified on the Platform which has caused some Data Exports to be delayed. We have since put a fix in place and are now monitoring the Platform to ensure that Data Exports are running as expected.
Report: "Delayed SMSes"
Last updateThis incident has been resolved.
We are investigating an issue causing delays in sending SMSes from the Platform. This includes delays in sending OTP login SMSes which is preventing some users from accessing the Dashboard. We are currently urgently investigating this issue.
Report: "Delayed data exports"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
An issue has been identified which is causing some data exports to be delayed. Our engineers are currently working on a fix for this. If you urgently require access to an export, please reach out to support@root.co.za, and we'll work with you to extract the required data.
Report: "Delayed Data Exports"
Last updateAn issue has been identified on the Platform which has caused some Data Exports to be delayed. We are urgently investigating and monitoring the issue.
Report: "API responsiveness degraded"
Last updateThe incident has been identified and resolved. The API & Web responsiveness have returned to normal
We are experiencing technical difficulties resulting in very slow responsiveness on the API and web dashboard. We are currently investigating.
Report: "Delayed Data Exports"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
An issue has been identified on the Platform which has caused some Data Exports to be delayed. We are urgently investigating and monitoring the issue.
Report: "Degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Degraded performance on the Dashboard"
Last updateAn issue was identified and subsequently resolved on the Dashboard whereby in rare cases, some pages would not fully load or operate as expected. This issue mainly affected claim pages. The issue has now been resolved.
Report: "Degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are urgently investigating an issue whereby policy documents and email notification attachments are failing to generate.
Report: "Unable to view PDF documents on the Root Dashboard"
Last updateThis incident has been resolved.
The issue has been identified, and a fix is currently being worked on.
We are currently investigating an issue preventing users from viewing PDF documents on the Root Dashboard. We will keep you updated on our investigations here.
Report: "Unable to view PDF documents on the Root Dashboard"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue preventing users from viewing PDF documents on the Root Dashboard. We will keep you updated on our investigations here.
Report: "Account verification API is offline"
Last updateThis incident has been resolved.
The 3rd party API service used to process bank account verification checks is currently offline due to a technical malfunction. We are engaging with our service provider to urgently resolve the issue and restore the verification service.
Report: "Increased response times and error rates"
Last updateThis incident has been resolved.
A fix has been implemented, which we are currently monitoring. Response times have returned to normal.
We have identified an issue causing an increase in response times and error rates across the Root platform. We are currently working on a fix.
Report: "Sandbox and scheduled-function degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified two related issues where: 1. Some product modules are experiencing issues on sandbox policies. 2. Sandbox and production scheduled-functions (product module code) have not been able to execute since the previous night. A fix for the issue has been identified and are being deployed to production.
Report: "Root dashboard and website unavailable"
Last updateThis incident has been resolved.
A fix has been implemented by our upstream provider and our website and dashboard seem to be online again. We are continuing to closely monitor from our side.
Our upstream provider has identified the cause of the issue and is now working on resolving the issue.
One of our upstream hosting providers is currently experiencing issues that have caused our website and dashboard to be temporarily unavailable. We are continuing to investigate and monitor.
Our dashboard and website are currently unavailable. We are urgently investigating this issue.
Report: "API unresponsive"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating
Report: "Root API performance issues"
Last updateThis incident has been resolved.
The fix is being reviewed and tested. We will update with further progress when available.
The issue has been identified and a fix is being implemented.
The Root Engineering team are investigating intermittent performance issues across Root's API
Report: "Document generation"
Last updateThis incident has been resolved.
Due to issues with an upstream service provider, our document generation service is currently down. This will cause delayed notifications being sent to policyholders.
Report: "Dashboard, unable to access new workbench page"
Last updateNo further issues have been reported or seen. Issue is resolved.
Issue relating to access to settings were resolved - this was caused by updates to required permissions. Issues relating to login / OTP's are confirmed to be functioning again. We are actively monitoring to ensure systems are fully operational.
A new dashboard feature was deployed that included the new 'workbench' page. Several users reported issues ranging from not being able to access settings or receive OTPs. The deployment was rolled back. Remaining issues are being investigated.
Report: "Bank account verification offline"
Last updateWe can confirm that the 3rd party service is fully operational. We will continue to engage with our provider to make sure we get notified of issues or downtime in a timeous manner.
We are seeing successful responses from our 3rd party API. We will continue to monitor.
The 3rd party API service used to perform bank account verification checks, is currently not responding. We are engaging with our service provider to urgently resolve the issue and restore the verification service. This could affect the ability to capture banking details on our platform.
Report: "AWS Cape Town outage affecting dashboard and API"
Last updateAll AWS services are operational again and our dashboard & API are back to normal.
AWS services are starting to resolve themselves and our dashboard is back up. The API seems to be mostly operational as well, we are still monitoring closely to confirm that all the related services are back up and running.
AWS has confirmed that they are experiencing elevated API errors and latency in the AF-SOUTH-1 region, which impacts services used by our dashboard, API & database.
Currently seems to have only affected our web dashboard - the API seems to be operational. We are monitoring closely.
Our service provider AWS is experiencing issues. We are monitoring closely.
Report: "Data exports are not being generated"
Last updateThis incident has been resolved.
Backlogs have been cleared, and exports are being monitored.
The issue has been identified and fixed. We are currently working through a backlog of data exports.
We are currently investigating this issue
Report: "Root Platform downtime"
Last updateThe Root Platform experienced downtime on 2022-10-28 from 15:31 until around 15:50 SAST, due to high levels of database activity. The platform is currently fully operational, and the engineering team is investigating the root cause of the incident.