Is Percy Down Right Now? Check if there is a current outage ongoing.

We are currently investigating elevated error rates affecting parts of our backend infrastructure. This may result in intermittent failures or delays in processing builds or visual comparisons. Our engineering team is actively working to identify and resolve the issue. We will provide an update as soon as more information is available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved

The issues has been identified

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

The issue has been resolved.

The fix has been rolled out and we are monitoring it.

We have identified the issue.

We faced a degradation of the service for a brief period for certain users, we have observed few builds failing intermittently during this brief period.

We are observing access denied snapshot on few Percy builds. We have identified the root cause, deployed a fix and monitoring it.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue, and have identified a potential fix.

We faced a degradation of the service for a brief period for some users.

This incident has been resolved. No jobs failed, the system was slower than usual and you might have seen longer build processing time.

A fix has been implemented and we are monitoring the results. We are waiting for backed up queues to get processed.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

A subset of users might have faced login failures while attempting to login to dashboard in this duration. Any already logged in sessions as well as APIs and builds were not affected. This incident is resolved.

This incident has been resolved.

We experienced heavy load in this duration - the queues are clearing up

We are currently investigating this issue.

This incident has been resolved.

We had jobs pending in queue causing delay's in build processing. We have fixed the scaling issue and queue is cleared.

We are currently investigating this issue.

This incident has been resolved. We experienced some unexpected technical issues that briefly impacted the availability of our website. Our team has identified the root cause and resolved it, and our services are back to normal. We apologize for any inconvenience this may have caused and appreciate your understanding. To make sure that these builds are not being used as baselines (base build) as a precaution we have marked these as "partial". If you encounter any further issues, please do not hesitate to contact our support team. Thank you for your patience and continued support.

We have fixed the issue and we are validating and monitoring the fix.

We are seeing incorrect images generated on Percy and we are investigating.

This is resolved.

This is resolved, we are currently monitoring queues and waiting for them clear backlog.

We are seeing some issues in percy causing a delay in sync of changes in plans and roles. The changes made in plans or roles would not reflect on percy immediately. This does not affect running percy builds or using percy review dashboard. We are working on investigating and fixing the issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We observed increased in build failure on projects containing Chrome and Firefox browsers after a recent infrastructure config change was deployed.

This incident has been resolved.

Due to a maintainence activity we are seeing some issues with Layout Comparisons(Alpha) feature, only on Safari, Edge and Mobile Browsers. We will update once this is stable, in the meanwhile Layout Comparison(Alpha) is working on Chrome and Firefox as intended.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Percy builds running on Safari on iPhone browser intermittently had prompts showing up causing unintended diffs. This issue first started occurring on 26th May and gradually became more frequent. The major impact on builds was observed between 30th May to 6th June. Although we started rolling out fixes for this issue by 30th May itself, due to the reactive nature of the fix some customers might have noticed these prompts till 6th June. We are sorry for any inconvenience caused due to this issue, if you still notice any such prompts, please reach out to support.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are noticing newer prompts in Safari on iPhone browser which are not covered by the fix we implemented in the last incident. We are currently investigating the issue.

Percy builds running on Safari on iPhone browser had popups showing up intermittently. We have rolled out a patch to address these popups and expect them to not show up in customer sessions anymore. We'll be actively monitoring the situation and taking appropriate steps to prevent such instances in the future.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

Percy API faced degraded performance from 17:17 PM to 19:47 PM PST on 26th March, 2023. API Latency increased and was caused due to increased backlog in our web server queue due to infrastructure scaling issues. This has been resolved now.

Percy API faced degraded performance from 16:26 PM to 19:08 PM PST on 6th March, 2023. API Latency increased and was caused due to increased backlog in our web server queue due to infrastructure scaling issues. This has been resolved now.

Percy builds running on Safari on iPhone showed “Connection is not private” and “Safari cannot open the page because the server cannot be found” pages instead of the actual website which was being tested. This issue was intermittent in nature and affected builds from 6:07 AM to 8:47 AM PST on 2nd March 23'. This happened because our infra was unable to process secure (HTTPs) requests due a configuration issue in our terminals. This issue has since been resolved and we have updated our monitoring to make sure we detect such issues early. We apologize for the inconvenience caused.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved on our cloud provider's end.

The issue has been identified on our cloud provider's end and we are monitoring the issue along with them.

We are currently investigating this issue.

We identified an issue in our rendering infrastructure which caused our render jobs to pile up and builds to slow down from 3:50 AM PST to 4:42 AM PST. No builds failed during this time and we have recovered the job backlog completely now. We will be reviewing our internal processes to make sure this doesn't happen again.

We are currently investigating this issue.

We identified certain organisations which faced intermittent build failures from 3:00 AM to 4:05 AM PT. This happened due to an internal infrastructure optimisation we did on our rendering infrastructure. We have rolled back since then and issue is resolved now. We apologise for the inconvenience.

This incident has been resolved. We apologize for the inconvenience.

A fix has been implemented and we are monitoring the results.

We have identified an issue related to a database change and are rolling out a fix.

This incident has been resolved.

We have identified an issue where our CDN provider is intermittently blocking some requests in Percy's rendering infrastructure, which is causing some renders to appear as a "Forbidden" page. We have made a configuration change and are monitoring the results.

This issue was caused by a bad database query that was shipped to our API. This part of our system was turned off to mitigate the issue. Many API requests would have timed out or 500 errored between the incident timeframe of: July 19 7:12 PM to 7:34 PM UTC. We're sorry for this issue.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue where our API is not responding.

Percy faced downtime due to outage at our CDN provider - Cloudflare's end from 11:34 PM to 12:44 AM PDT. During this time users might have faced intermittent issues while trying to access Percy dashboard and intermittent build failures. The issue is fixed now. We apologize for inconvenience.

We're investigating this issue.

Percy rendering infrastructure was down for Safari on iPhone beta platform which affected a subset of our customers enrolled in beta from 4.28 AM PDT to 7:10 AM PDT. This happened due to a deployment pipeline issue in our Safari on iPhone infra and has since been patched. We apologize for the inconvenience caused and will be doing an RCA on our end to avoid this in future.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We faced intermittent timeouts in ~30 Edge builds from 05:46 AM to 05:51 AM PDT (5 mins) due to a broken deploy. We apologize for the inconvenience caused and we are improving monitoring on our end to avoid these cases in the future.

We identified an issue introduced as part of a recent OS upgrade that was causing slowdowns in our rendering infrastructure. This has been resolved now, we will conduct internal root cause analysis to make sure this doesn't happen again.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Percy API faced degraded performance from 7:35 PM to 7.58 PM PT. 503s were caused due to increased backlog in our web server queue due to a set of bad requests. This has been resolved now.

This incident has been resolved.

We are investigating degraded performance on our API.

From 8:07 to 8:50 PT our API experienced an outage causing 503 and 504 responses to be served for most requests. The outage was caused by a misconfigured health check in our infrastructure due to which deploys in high traffic times led to our infrastructure getting incorrectly marked as unhealthy. We apologize for the inconvenience.

A fix has been implemented and we are monitoring the results.

We are currently investigating an API outage causing 503/504s to be served from percy API.

We faced elevated error rate in API for the following 2-3 minute intervals - 4:25 PM to 4:30 PM PT - 6:24 PM to 6:27 PM PT - 10:10 PM to 10:12 PM PT We have identified and fixed an issue in a previously applied patch.

This incident has been resolved. From 7:34 to 7:47 PT our API experienced an outage causing 503 and 504 responses to be served for most requests. The outage was caused by a memory leak in the API which caused processes to crash. We have put mitigations into place and will continue to monitor and improve our memory management systems to prevent this from reoccurring. We apologize for the inconvenience.

A fix has been implemented and we are monitoring the results.

We have identified the source of the issue and begun mitigation.

We are currently investigating an API outage causing 503/504s to be served from percy API.

Some customers received 503 responses when interacting with our API between these times: * Feb 7th 11:32pm UTC * Feb 8th 00:35am UTC We have since narrowed this issue down to resource contention on one of our Kubernetes clusters and the issue is now resolved.

We are investigating elevated 503 responses from our API.

We served intermittent 503s from Percy API from 9.20 PM to 9.21 PM PST.

This incident has been resolved.

Replicating a large change to our failover database is the cause of this latency. We are working to mitigate this now.

We are experiencing higher than usual database latency which has degraded our API performance.

Some builds that were being processed between 17:44 to 17:46 on Oct 7 UTC went onto fail. This issue is now resolved and future builds will succeed as usual.

This incident has been resolved as of last week. We have improved internal systems and monitoring for this issue, and will continue to watch it closely.

After root cause analysis and some infrastructure changes to address the issue, we have not seen any similar high latency issues in >12 hours. We will continue to monitor closely.

We have identified an issue with redis latency, which is causing bursts of 503/504s from Percy's API. We do not have a fix at this time but are investigating further and will post updates as we have them. We appreciate your patience and apologize for the inconvenience. This ops issue has been a tricky issue to resolve and we are coordinating with our redis provider on a resolution. We will leave this ticket open until we are sure the problem has been fully resolved.

We’ve identified a select group of projects that have intermittent or continuous build timeouts from September 28th to October 6th UTC. We’ve since rolled back a change on these projects which will resolve this issue. We’re sorry for the inconvenience this has caused and we’ll continue to closely monitor the impacted projects to ensure the issue is fully resolved going forward.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating errors occurring when builds are created.

This incident has been resolved. The backlog has been processed and we're back to normal capacity. Thanks for your patience.

We have scaled extra capacity to take care of the backlog faster than it would otherwise and it's working through the renders quickly. We will continue to monitor until resolved.

We have identified an issue with a backlog of rendering jobs. Builds may be processing more slowly than usually. We are working on a fix and will update soon.

This incident has been resolved.

We are monitoring the results of configuration changes which we believe have eliminated these intermittent issues. We will continue to monitor, and leave this issue open until fully resolved.

We are continuing to investigate this issue.

We are currently investigating an issue where intermittent 503/504s are being served by our API, which is impacting some customers.

This incident has been resolved.

The backlog of jobs has been completed and we are monitoring for any further issues.

Our Redis provider has mitigated the issue and we are now monitoring. We expect that builds will be slow to process or will timeout until the backlog has been worked through.

The issue has been identified and we are working on mitigating the issue.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

We identified an issue causing some builds to be marked as failed due to time out. A fix is being deployed and we are monitoring the situation.

We're investigating an issue that results in builds being marked as failed due to a time out. The error message seen on builds that fail this way is: "0 snapshots in this build took too long to render even after multiple retries."