Root

Is Root Down Right Now? Check if there is a current outage ongoing.

Root is currently Operational

Last checked from Root's official status page

Historical record of incidents for Root

Report: "Policy notes"

Last update
resolved

Unable to create or view policy notes

Report: "Policy notes"

Last update
Resolved

Unable to create or view policy notes

Report: "Document creation delay"

Last update
resolved

This incident has now been resolved. Customers may have experienced up to one hour's delay in receiving policy and terms documents.

investigating

We are currently investigation the issue.

Report: "Product module code timeouts"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

Product module code functions timing out

Report: "South African Multi-tenant degraded performance"

Last update
resolved

The issue has been resolved.

investigating

Maintenance Update Action: The affected indexes have been rebuilt. We are thoroughly testing the system to confirm the improved post-maintenance performance.

investigating

Maintenance Update Action: We are currently performing maintenance to optimize our system performance. Our team is actively rebuilding the affected indexes. This action addresses the root cause of the performance degradation issue.

investigating

South African Multi-tenant is stable and we are continuing to monitor the performance

investigating

South African Multi-tenant is stable and we are continuing to monitor the performance

investigating

South African Multi-tenant is stable and we are continuing to monitor the performance

investigating

Incident: Slow API Response Times for South African Multi-tenant Cause: The core issue was identified as a performance degradation in our database after a recent version upgrade. Action: We have added additional resources to maintain an acceptable level of system performance for the day and have planned after-hours maintenance to re-build the affected indexes.

investigating

Incident: Slow API Response Times for South African Multi-tenant Issue: Our South African Multi-tenant service is currently experiencing slow API response times. This may lead to delays in data retrieval and other dependent operations for our users, we are currently investigating the issue

Report: "Majour outage"

Last update
resolved

This incident has been resolved.

investigating

The issue has been addressed and we are continuing to monitor the issue

investigating

We're experiencing an elevated level of API errors and are currently looking into the issue.

Report: "Dashboard outage"

Last update
resolved

This incident has been resolved

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Platform emergency maintenance"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

Performing emergency database upgrades

Report: "Major outage in our API"

Last update
resolved

This incident has been resolved.

monitoring

We've resolved the issue and all systems are operational again. We are still monitoring

investigating

We are continuing to investigate this issue.

investigating

We have observed to a network connection issue affecting our workers. We're investigating

Report: "Elevated API Errors"

Last update
resolved

This incident has been resolved.

investigating

We're experiencing an elevated level of API errors and are currently looking into the issue.

Report: "Elevated API Errors"

Last update
postmortem

This incident has been resolved.

resolved

This incident has been resolved.

investigating

We're experiencing an elevated level of API errors and are currently looking into the issue.

Report: "Additional platform optimisation"

Last update
resolved

The scheduled maintenance is completed, and the Root platform is fully operational.

investigating

The scheduled platform maintenance is taking longer than anticipated. The Root platform is currently still unavailable. We are working on restoring normal operations as soon as possible.

Report: "Peach Payments CDV Temporarily Offline"

Last update
resolved

Peach Payments CDV (Check Digit Verification) for bank account number checks (used when issuing debit-order policies from the dashboard) temporarily went offline from 17:11, when we first detected it, to 17:30 when it came back online. This has been resolved and we are closely monitoring the situation.

Report: "2FA SMS messages delayed"

Last update
resolved

User logins prevented by delayed 2FA SMS messages. Due to a backlog in queue infrastructure SMS OTPs were severely delayed.

Report: "Cloudflare experiencing network issues"

Last update
resolved

This incident has been resolved and reported stable by Cloudflare: https://www.cloudflarestatus.com/incidents/46z55mdhg0t5

monitoring

We are continuing to monitor for any further issues.

monitoring

We are continuing to monitor for any further issues.

monitoring

Cloudflare has been experiencing network issues (https://www.cloudflarestatus.com/incidents/46z55mdhg0t5) which affects certain clients behind their proxy. We've mitigated this issue and are actively monitoring it.

Report: "Degraded performance"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

The issue has been temporarily resolved and the platform is stable. We're closely monitoring performance whilst we're implementing a permanent solution. For the time being, certain search functionality has been disabled. We'll keep updating progress here.

identified

We're experiencing degraded API performance. We've identified the issue and are busy implementing a fix. We'll keep updating progress here.

Report: "Delayed sending of emails and SMS'"

Last update
resolved

We're experiencing delayed sending of emails and SMS' through the API. We've identified and resolved the issue.

Report: "Slow response times when adding debit order payment methods during policy issuing"

Last update
resolved

This incident has been resolved.

monitoring

A payment service provider of Root is currently experiencing slow response times for check digit verification of bank account details. This is affecting the Root Dashboard, and is causing slow response times and/or timeouts when attempting to add a debit order payment method during policy issuing. We are continuing to monitor the situation, and will provide an update once the response times have returned to normal.

Report: "Slow response times when adding debit order payment methods during policy issuing"

Last update
resolved

This incident has been resolved.

monitoring

A payment service provider of Root is currently experiencing slow response times for check digit verification of bank account details. This is affecting the Root Dashboard, and is causing slow response times and/or timeouts when attempting to add a debit order payment method during policy issuing. We are continuing to monitor the situation, and will provide an update once the response times have returned to normal.

Report: "Issue with fetching policies"

Last update
resolved

A permanent fix has been tested and deployed, and the underlying issue is now resolved.

monitoring

We have identified a permanent fix, which is currently in development and testing. We will keep you posted as we progress.

monitoring

We have deployed a temporary fix and are monitoring it. The API and dashboard are functioning normally again. If you’re experiencing any issues, please contact us at support@root.co.za We’ll keep you posted here as we progress.

identified

We're experiencing issues on the "fetch policies" endpoint. This is affecting both API integrations and dashboard users. We're aware of the problem and busy working on a fix for the problem. We will keep you posted as we progress.

Report: "Possible issues accessing Root services"

Last update
resolved

This incident has been resolved.

monitoring

All Root services look to be operating normally. We will continue to monitor the situation.

monitoring

One of our upstream providers, Cloudflare, is experiencing network-related issues. Customers may have issues accessing the Root platform during this time. We are continuing to monitor the situation.

Report: "Delayed notifications"

Last update
resolved

We experienced a delay in the sending of emails and SMS' on the Root platform. The cause was identified and resolved, and notification sending has returned to normal speeds.

Report: "Degreaded API performance"

Last update
resolved

We experienced degraded performance on the Root API between 14:15 and 14:45. The issue was identified and resolved.

Report: "Peach Payments CDV Temporarily Offline"

Last update
resolved

Peach Payments have resolved the issue, and CDV checks on Root have returned to normal operation.

monitoring

Peach Payments are experiencing system issues, which is impacting CDV (Check Digit Verification) on Root for bank account number checks. This is used when issuing policies from the dashboard with debit order payment methods. We are working with Peach to resolve the issue, and will continue to monitor the situation.

Report: "Request timeouts when creating applications"

Last update
resolved

The team have identified and resolved the underlying issue.

investigating

The Root Engineering team have been alerted to an increased number of request timeouts when creating applications on the Root API. We are currently investigating the cause.

Report: "Peach Payments CDV Temporarily Offline"

Last update
resolved

Peach Payments have resolved the issue, and CDV checks on Root have returned to normal operation.

monitoring

Peach Payments are experiencing system issues, which is impacting CDV (Check Digit Verification) on Root for bank account number checks. This is used when issuing policies from the dashboard with debit order payment methods. We are working with Peach to resolve the issue, and will continue to monitor the situation.

Report: "Root platform partially unavailable"

Last update
resolved

During a routine deployment of new functionality at 18:01 on Tuesday, 23 February 2021, some functionality (including user logins) became unavailable on the Root platform for roughly 20 minutes. The platform returned to normal operation at 18:20. Upon investigation, the team identified that database changes being made as part of the deployment were unexpectedly slowing down or blocking some functionality across the platform. The team have since identified the underlying cause, and are in the process of rectifying this to ensure the issue does not reoccur. Furthermore, the team also noted that Root's public status page did not reflect this incident in a timely manner. We have since identified the cause of this, and are currently altering our processes to ensure clients are proactively notified of such issues in future.

Report: "Delayed Data Exports"

Last update
resolved

The issue has been resolved and data exports are now running as expected.

monitoring

An issue has been identified on the Platform which has caused some Data Exports to be delayed. We have since put a fix in place and are now monitoring the Platform to ensure that Data Exports are running as expected.

Report: "Delayed SMSes"

Last update
resolved

This incident has been resolved.

investigating

We are investigating an issue causing delays in sending SMSes from the Platform. This includes delays in sending OTP login SMSes which is preventing some users from accessing the Dashboard. We are currently urgently investigating this issue.

Report: "Delayed data exports"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

An issue has been identified which is causing some data exports to be delayed. Our engineers are currently working on a fix for this. If you urgently require access to an export, please reach out to support@root.co.za, and we'll work with you to extract the required data.

Report: "Delayed Data Exports"

Last update
resolved

An issue has been identified on the Platform which has caused some Data Exports to be delayed. We are urgently investigating and monitoring the issue.

Report: "API responsiveness degraded"

Last update
resolved

The incident has been identified and resolved. The API & Web responsiveness have returned to normal

investigating

We are experiencing technical difficulties resulting in very slow responsiveness on the API and web dashboard. We are currently investigating.

Report: "Delayed Data Exports"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

An issue has been identified on the Platform which has caused some Data Exports to be delayed. We are urgently investigating and monitoring the issue.

Report: "Degraded performance"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Degraded performance on the Dashboard"

Last update
resolved

An issue was identified and subsequently resolved on the Dashboard whereby in rare cases, some pages would not fully load or operate as expected. This issue mainly affected claim pages. The issue has now been resolved.

Report: "Degraded performance"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are urgently investigating an issue whereby policy documents and email notification attachments are failing to generate.

Report: "Unable to view PDF documents on the Root Dashboard"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified, and a fix is currently being worked on.

investigating

We are currently investigating an issue preventing users from viewing PDF documents on the Root Dashboard. We will keep you updated on our investigations here.

Report: "Unable to view PDF documents on the Root Dashboard"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating an issue preventing users from viewing PDF documents on the Root Dashboard. We will keep you updated on our investigations here.

Report: "Account verification API is offline"

Last update
resolved

This incident has been resolved.

identified

The 3rd party API service used to process bank account verification checks is currently offline due to a technical malfunction. We are engaging with our service provider to urgently resolve the issue and restore the verification service.

Report: "Increased response times and error rates"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented, which we are currently monitoring. Response times have returned to normal.

identified

We have identified an issue causing an increase in response times and error rates across the Root platform. We are currently working on a fix.

Report: "Sandbox and scheduled-function degraded performance"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We've identified two related issues where: 1. Some product modules are experiencing issues on sandbox policies. 2. Sandbox and production scheduled-functions (product module code) have not been able to execute since the previous night. A fix for the issue has been identified and are being deployed to production.

Report: "Root dashboard and website unavailable"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented by our upstream provider and our website and dashboard seem to be online again. We are continuing to closely monitor from our side.

identified

Our upstream provider has identified the cause of the issue and is now working on resolving the issue.

identified

One of our upstream hosting providers is currently experiencing issues that have caused our website and dashboard to be temporarily unavailable. We are continuing to investigate and monitor.

investigating

Our dashboard and website are currently unavailable. We are urgently investigating this issue.

Report: "API unresponsive"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating

Report: "Root API performance issues"

Last update
resolved

This incident has been resolved.

monitoring

The fix is being reviewed and tested. We will update with further progress when available.

identified

The issue has been identified and a fix is being implemented.

investigating

The Root Engineering team are investigating intermittent performance issues across Root's API

Report: "Document generation"

Last update
resolved

This incident has been resolved.

monitoring

Due to issues with an upstream service provider, our document generation service is currently down. This will cause delayed notifications being sent to policyholders.

Report: "Dashboard, unable to access new workbench page"

Last update
resolved

No further issues have been reported or seen. Issue is resolved.

monitoring

Issue relating to access to settings were resolved - this was caused by updates to required permissions. Issues relating to login / OTP's are confirmed to be functioning again. We are actively monitoring to ensure systems are fully operational.

investigating

A new dashboard feature was deployed that included the new 'workbench' page. Several users reported issues ranging from not being able to access settings or receive OTPs. The deployment was rolled back. Remaining issues are being investigated.

Report: "Bank account verification offline"

Last update
resolved

We can confirm that the 3rd party service is fully operational. We will continue to engage with our provider to make sure we get notified of issues or downtime in a timeous manner.

monitoring

We are seeing successful responses from our 3rd party API. We will continue to monitor.

investigating

The 3rd party API service used to perform bank account verification checks, is currently not responding. We are engaging with our service provider to urgently resolve the issue and restore the verification service. This could affect the ability to capture banking details on our platform.

Report: "AWS Cape Town outage affecting dashboard and API"

Last update
resolved

All AWS services are operational again and our dashboard & API are back to normal.

monitoring

AWS services are starting to resolve themselves and our dashboard is back up. The API seems to be mostly operational as well, we are still monitoring closely to confirm that all the related services are back up and running.

monitoring

AWS has confirmed that they are experiencing elevated API errors and latency in the AF-SOUTH-1 region, which impacts services used by our dashboard, API & database.

investigating

Currently seems to have only affected our web dashboard - the API seems to be operational. We are monitoring closely.

investigating

Our service provider AWS is experiencing issues. We are monitoring closely.

Report: "Data exports are not being generated"

Last update
resolved

This incident has been resolved.

monitoring

Backlogs have been cleared, and exports are being monitored.

identified

The issue has been identified and fixed. We are currently working through a backlog of data exports.

investigating

We are currently investigating this issue

Report: "Root Platform downtime"

Last update
resolved

The Root Platform experienced downtime on 2022-10-28 from 15:31 until around 15:50 SAST, due to high levels of database activity. The platform is currently fully operational, and the engineering team is investigating the root cause of the incident.