TestRail Cloud

Is TestRail Cloud Down Right Now? Discover if there is an ongoing service outage.

TestRail Cloud is currently Operational

Last checked from TestRail Cloud's official status page

Historical record of incidents for TestRail Cloud

Report: "Degraded Performance"

Last update
resolved

This incident has been resolved.

monitoring

We're pleased to report significant performance improvements across our platform, with decreased error rates and API rate limits restored to their normal levels. Our team continues to closely monitor the system to ensure ongoing stability and reliability. Thank you for your patience and support.

monitoring

Some customers are experiencing degraded performance. Our engineers are investigating.

monitoring

The incident has been resolved, and all customers should now have access to their instances. Please note that we have temporarily increased the API rate limit restrictions, and reduced the background task frequency while we monitor the system. We appreciate your patience during this time, and are committed to maintaining service stability.

identified

Some customers may experience maintenance windows longer than 15 minutes as our team continues to work to resolve the incident.

identified

Our team is actively working to resolve the incident. Customers may see their instances in maintenance mode intermittently, but for no longer than 15 minutes.

investigating

Some customers may be experiencing degraded performance. Our engineers are investigating.

Report: "Degraded Performance"

Last update
resolved

This incident has been resolved.

identified

Our engineers have identified the potential source and implemented a fix.

investigating

Some customers may be experiencing degraded performance. Our engineers are investigating.

Report: "Degraded Performance for Some Customers"

Last update
resolved

We successfully resolved the issue, and normal service has been restored.

monitoring

We're continuing to monitor for any further issues while the team is working to a full resolution.

monitoring

We identified degraded performance that may affect some customers in connection with the 14-Feb partial outage. Our team is actively investigating to ensure a complete resolution.

Report: "Partial outage due to an overrunning maintenance task"

Last update
resolved

The incident has been resolved. We are continuing to monitor for errors.

monitoring

Monitoring continues while the rate of errors continues to reduce sharply.

monitoring

We are continuing to monitor this issue.

monitoring

Customers should be seeing improvements. We will continue to monitor whilst services return to normal.

identified

We're currently experiencing a partial Cloud outage which may result in error messages relating to failing API calls or attachment related nodes. We're working on the issue right now and expect to have a resolution shortly.

Report: "Degraded Cloud Attachments Service"

Last update
resolved

Our DevOps team has applied a fix for this issue and after a period of monitoring, has determined that users should no longer be experiencing issues related to the TestRail attachments service. We appreciate your patience while we dealt with this matter. If you're still experiencing errors that result in a "Call to a member function prepare()" message, please reach out to our support team and we'll be happy to assist.

monitoring

Our team is stabilising our hosted infrastructure after applying the necessary fixes. We are continuing to monitor the situation closely.

identified

We are continuing to work on a fix for this issue which is affecting a small number of customers. Thanks for your patience. Our team is working hard to complete this activity as quickly as possible and we appreciate your bearing with us while we do so.

identified

We're still working to fix this issue as fast as we can. Thanks for your patience while we do so. We'll share updates as progress is made.

identified

The issue has been identified and a fix is being implemented.

investigating

The TestRail Cloud Attachments service is currently experiencing performance issues affecting a subset of attachments functionality for a small number of TestRail Cloud users. Our DevOps team is deploying a fix and we expect to resolve this matter promptly.

Report: "Some TestRail instances are currently in maintenance mode"

Last update
resolved

All systems are operational.

monitoring

The database has been recovered and all affected TestRail cloud instances should now be back online. We are continuing to monitor the database for any further issues.

identified

The cause of the database issue has been identified and our engineers are working to solve the problem. In the meantime, we are also building a new database from a backup with a view towards switching affected customers to the new database.

investigating

One of our databases is not responding correctly and as a result, TestRail instances reliant on it are currently in maintenance mode. Our DevOps team is investigating the problem.

Report: "Status page launched!"

Last update
resolved

No incident here—just excited to announce our new Statuspage is live!

Report: "Investigating some reports of slow performance"

Last update
resolved

Our DevOps team quickly identified the culprit behind performance degradation and was able to resolve the issue quickly.

Report: "TestRail CSS & JS assets are loading slowly or not at all for some instances"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating this issue.

Report: "Some instances are currently showing maintenance mode."

Last update
resolved

We are working directly with affected customers and/or customer instances and will be restoring data to them over the next ~2 weeks. We will contact affected customers directly to confirm when this has been done. Since instances are functioning correctly at this time, we are treating this incident as resolved and no further updates will be made here.

monitoring

We have successfully recovered data that was added to affected instances between September 28, 2021, 1:11:11 AM UTC and September 28, 2021, 12:12PM UTC. We will reach out to provide more details about how we will be restoring recovered data to affected TestRail instances once we have more information. Please note that you are still able to use your instance at this time, as we will be working to merge your current data with the restored data.

monitoring

There are no new updates at this time. We are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. Please note that you are still able to use your instance at this time, as we will be working to merge the current data after the restore has completed. We will post another update on October 4, 2021, at 15:00 UTC, unless we receive more information before that.

monitoring

There are no new updates at this time. We are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. We will post another update at or before October 1, 2021, at 15:00 UTC, unless we receive more information before that.

monitoring

All Cloud systems are operational, but we are continuing to work with Amazon to recover and restore data added to affected instances after the backup timestamp, September 28, 2021, 1:11:11 AM UTC. We will post another update at or before September 30, 2021, (today) at 22:30 UTC, unless we receive more information before that.

monitoring

Per our earlier update, all affected instances should now be accessible. It’s possible that affected instances may still be experiencing challenges with lost data however. This is due to the fact that the instances have been restored from a backup prior to the crash. The backup used has the following timestamp: September 28, 2021, 1:11:11 AM UTC. What this means: Data added to your instance after the timestamp but before the outage may not have been fully restored to your instance. We are continuing to work with AWS to recover and restore data added to your instance after the backup timestamp, but this process is expected to take a little while. We will update this status once we have completed our work with AWS to restore all of your data. We appreciate your patience and apologise for any inconvenience caused in the meantime.

monitoring

All affected instances are now functioning. Data has been restored to the affected instances from a backup prior to the crash: September 28, 2021, 1:11:11 AM UTC. We are working to restore any changes after this time to the affected instances, and will continue to provide updates on progress as we make it.

identified

We’re continuing to work with AWS Support and we will keep you updated as the situation progresses.

identified

We are continuing to implement the fix identified earlier today, and service has been restored to a subset of affected instances. We will provide an update as soon as we have more information.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Degraded attachment performance"

Last update
resolved

Users of TestRail were seeing ‘Cannot achieve consistency level LOCAL_ONE’ and ‘Request timed out’ errors. The issue was related to overloaded Cassandra nodes, but has now been resolved.

Report: "Instances in Maintenance Mode"

Last update
resolved

Multiple Instances were stuck in maintenance mode. We have quickly identified and resolved the issue after receiving the reports.

Report: "Investigating some reports of degraded performance"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are actively investigating an issue that may be causing slow performance or issues accessing certain TestRail Cloud instances.

Report: "Gateway errors when accessing TestRail"

Last update
resolved

This incident has been resolved.

monitoring

We've identified the cause of the gateway errors and implemented a fix. Service to TestRail Cloud has been fully restored and all instances are showing as online and operational. We will continue to monitor closely to ensure there are no other issues. If you are still receiving any 502 or 504 gateway errors, please email us at contact@gurock.com so that we can take a closer look at your specific instance.

investigating

TestRail Cloud continues to experience issues with 502 or 504 gateway errors. We know this is frustrating and we’re working to resolve this as soon as possible. We have escalated this to our engineering team, and will provide an update as soon as more information becomes available.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue. We do not have an ETA for resolution at the present time, but expect to provide one shortly.

investigating

Some customers are experiencing gateway errors, e.g. '502 Bad Gateway' or '504 Gateway Time-out', when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Report: "Intermittent gateway issues / performance degradation"

Last update
resolved

This incident has been resolved.

monitoring

Earlier today we had some further reports of further 504 and 502 gateway errors affecting a small number of customers. We've identified the cause of the errors and implemented a fix. Service to TestRail Cloud has been fully restored and all instances are showing as online and operational. We will continue to monitor closely to ensure there are no other issues. If you are still receiving any 502 or 504 gateway errors, please email us at contact@gurock.com so that we can take a closer look at your specific instance.

monitoring

TestRail Cloud is fully operational, but we are continuing to monitor and investigate reports of intermittent 504 and 502 gateway errors, including for the TestRail API, as well as some other issues related to timeouts. We will post an update as soon as we have more information. If you or a teammate is experiencing any issues, please notify our Support team at contact@gurock.com so we can help triage.

Report: "TestRail Cloud instances in Maintenance mode"

Last update
resolved

This incident has been resolved.

monitoring

We have identified and resolved the issue. All instances are operational, however we are continuing to closely monitor the situation.

investigating

TestRail Cloud is currently experiencing issues. We know this is frustrating and we’re working to resolve this as soon as possible. We have escalated this to our engineering team, and will provide an update as soon as more information becomes available.

Report: "Gateway errors when accessing TestRail"

Last update
resolved

This incident has been resolved.

monitoring

Earlier today we had some further reports of further 504 and 502 gateway errors affecting a small number of customers. We've identified the cause of the errors and implemented a fix. Service to TestRail Cloud has been fully restored and all instances are showing as online and operational. We will continue to monitor closely to ensure there are no other issues. If you are still receiving any 502 or 504 gateway errors, please email us at contact@gurock.com so that we can take a closer look at your specific instance.

investigating

Some customers are experiencing gateway errors, e.g. '502 Bad Gateway' or '504 Gateway Time-out', when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Report: "Gateway errors when accessing TestRail"

Last update
resolved

This incident has been resolved.

investigating

Some customers are experiencing gateway errors, e.g. '502 Bad Gateway' or '504 Gateway Time-out', when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Report: "Undefined Column Name Bucket"

Last update
resolved

This incident has been resolved.

monitoring

We have deployed the fix to all customers and are monitoring for any further issues.

investigating

Our team has identified the issue and implemented a fix. All cloud customers will receive the fix shortly.

investigating

We are continuing to investigate this issue and will provide further updates shortly. In the meantime, users are able to workaround the problem by using the "Pass & Next" button to add a result instead of the "Add Result" button.

investigating

Some customers are experiencing an error when attempting to add test results. We are currently investigating the cause of this issue and will provide an update shortly.

Report: "Slow performance"

Last update
resolved

This incident has been resolved.

monitoring

We have identified the cause of the instability and implemented a fix which we are currently monitoring.

investigating

Some customers are experiencing slower than expected TestRail performance. We are currently investigating this issue.

Report: "Gateway and Access Denied errors when accessing TestRail"

Last update
resolved

This incident has been resolved.

monitoring

We have fixed the issue encountered by some customers earlier today in regards to Gateway Access issues. We are monitoring the situation.

identified

We have identified the issue in which some customers have experienced gateway errors (502 - Bad Gateway, 504 - Gateway Time-out, 403 - Access Denied) when attempting to access their TestRail Cloud instance. We have resolved the issue for several of our customer instances and expect to resolve the remaining instances shortly.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

Some customers are experiencing gateway errors, e.g. '502 Bad Gateway', '504 Gateway Time-out', or '403-Access Denied' when attempting to access their TestRail cloud instance. We are currently investigating this issue and will provide further updates shortly.

Report: "Some customers are experiencing 403 Access Denied messages"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified the cause of this issue and we are currently in the process of implementing a fix.

investigating

We are currently investigating this issue.

Report: "A small number of customers are experiencing slow response times"

Last update
resolved

The issue has been identified and a fix implemented. We will be continuing to monitor the situation.

investigating

We are currently investigating this issue.