Historical record of incidents for Codecov
Report: "Generalized latency issues throughout infrastructure"
Last updateWe have identified the problem as being with our cloud infrastructure provider and have contacted them for further diagnosis. Customer impact is delays in ingestion and intermittent failures for both US customers.
We are currently investigating latency problems across all of our infrastructure in US regions.
Report: "db maintenance"
Last updateWe will be undergoing scheduled maintenance during this time.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Extended Queue Processing Times"
Last updateWe are experiencing backups in our processing queues. This creates delays when processing coverage and test report upload files. This issue lasted from: 2025-05-14 8:00UTC to 2025-05-14 18:00UTC and has been fully resolved.
Report: "Extended Queue Processing Times"
Last updateWe are experiencing backups in our processing queues. This creates delays when processing coverage and test report upload files. This issue lasted from:2025-05-14 8:00UTC to 2025-05-14 18:00UTC and has been fully resolved.
Report: "Queue Slowdown"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Queue Slowdown"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "Queue Slowdown"
Last updateWe are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "DB Maintenance"
Last updateA fix has been implemented and we are monitoring the results.
Report: "Test Analytics Outage"
Last updateThis incident is resolved and Test Analytics is fully operational
We are currently experiencing downtime with Test Analytics. This issue began at 2025-05-01 22:03 UTC. We have identified the issue and are working toward resolution.
Report: "Test Analytics Outage"
Last updateWe are currently experiencing downtime with Test Analytics. This issue began at 2025-05-01 22:03 UTC. We have identified the issue and are working toward resolution.
Report: "System Degraded"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Slower processing"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Slack Notification issues"
Last updateThis incident has been resolved.
We are currently experiencing issues with sending notifications to Slack due to an ongoing incident: https://slack-status.com/2025-02/1b757d1d0f444c34
Report: "Degraded Performance"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Error processing uploads using the global upload token"
Last updateOn 28 January from 12:00 to 30 January 13:00 EST There was a disruption to global upload token validation that resulted in the Codecov system accepting uploads but internally denying the upload. This resulted in failed upload processing for repositories configured to use the global upload token. We now have increased monitoring for this scenario and have put measures in place to prevent it altogether. The underlying issue has been resolved and global upload token uploads should again be working as expected.
Report: "Queue backup resulting in delayed reporting"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Degraded UI"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Unable to login to Codecov"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
It looks like uploads are unaffected, user are not able to login to Codecov using any service at this time. We are investigating.
Report: "Delayed notifications"
Last updateThis incident has been resolved.
Queue levels are approaching normal. Notification times should be back to normal as well.
We are experiencing high volumes and are seeing some queue back ups resulting in longer than normal notification times.
Report: "Intermittent CLI Verification Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are having a problem with CLI distribution related to our caching layer. The cache is now bypassed and the issue is being investigated. This will have resulted in unexpected CLI verification failures when downloading/verifying the Codecov CLI.
Report: "Delays with Report Processing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Delays with Report Processing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Delays with Report Processing"
Last updateA large backlog of reports caused the system to lag behind processing. This has been remedied and the system is now fully operational.
The issue has been identified and a fix is being implemented.
Report processing is delayed. We are currently investigating the issue.
Report: "UI is not operational"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Reports failing"
Last updateFrom ~8:30 UTC - ~12:00 UTC Codecov failed to process reports. These reports will need to be uploaded again.
Report: "intermittent failures for uploads"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Uploads are failing"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Unable to login or very slow"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Degraded performance"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Intermittent 50X errors. We are looking into it.
Report: "Degraded Performance"
Last updateIntermittent 50X errors. We have mitigated the issue and are continuing to monitor.
Report: "Degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Issues with flags"
Last updateThis incident has been resolved.
There is an issue with our underlying flag reporting storage system. Fix is implemented and being monitored. This has resulted in some disruption to the UI.
Report: "Degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Issues with serving the UI"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Degraded performance with coverage uploads"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Intermittent failures"
Last updateA database query began unexpectedly taking large amounts of time to complete. Our SRE team is focused on improving this response time and working closely with our cloud provider to come to a resolution. We have implemented several fixes to mitigate the issues and are continuing to monitor and improve.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue with our database. High utilization is causing intermittent timeouts and 500s.
Report: "Issues viewing file contents for private repositories on app.codecov.io"
Last updateThis issue resulted from a change in our backend to better support GitHub App installations in our API. The PR tracking the revert and underlying code change can be found here: [https://github.com/codecov/codecov-api/pull/355](https://github.com/codecov/codecov-api/pull/355)
We have resolved this issue and will continue to monitor moving forward.
Currently we are experiencing an issue where project source code is not being fetched properly for private repos in the Codecov UI. This leads to an inability to view line coverage in the UI. We are investigating.
Report: "Continuing: Issue with Badges and Repo Dashboards"
Last updateWe have deployed fix for the underlying issue. Users should expect to see normal behavior for their badges and repo dashboards
A fix for incorrect data in coverage badges and repo overview pages was launched yesterday (January 22nd) and is continuing to be rolled out to all affected repositories through the course of today, January 23rd.
We have scheduled a maintenance for this Incident. Maintenance information can be found here: https://status.codecov.com/incidents/dcqgywtr2tpg
The underlying issue has been identified and we are implementing a fix that will be deployed through the course of today, January 22nd.
We are investigating downstream effects of a weekend incident affecting users' repo coverage badges and their repo overview pages. This incident appears as an incorrect head commit on a given branch.
Report: "Issue with Badges and Repo Dashboards"
Last updateWe have resolved this issue moving forward. For those repositories that were impacted by this issue, badges and repo dashboards should report correct data on the next upload to Codecov for the impacted repository.
There seems to be an issue with the Codecov badge not rendering when embedded. Additionally, there are difficulties in some instances viewing the dashboard for a repository. We are currently investigating this issue.
Report: "Emergency DB maintenance"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Issue Syncing Repositories and Orgs for New Accounts after GitHub App Installation"
Last updateThis incident has been resolved. The core issue was a recent was leading to an issue where a database connection could be closed before a new user's group and repo membership could be written to the database. This has been fixed and we've verified that new accounts should be syncing properly now after Github App Installation. For timeline purposes, this incident would have begun upon introduction of a recent change: Tue, 19 Dec 2023 15:53:34 GMT and was resolved on Thur 21 Dec 2023 21:00:00 GMT.
The issue has been identified and we are working on a fix.
We're currently investigating an issue where newly added Codecov accounts cannot properly sync their organizations or members after GitHub App Installation.
Report: "Degraded performance accessing the UI"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Errors on login"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Issues with notifications"
Last updateHigher than expected database load was causing some database transaction to time out. This was adversely impacting the notifications system and led to notification tasks timing out.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
Some notifications are experiencing issues. We are currently investigating.
Report: "Issues communicating with Github"
Last updateThis incident has been resolved.
GitHub is monitoring the issue. https://www.githubstatus.com/incidents/gqx5l06jjxhp
We are currently investigating this issue.
Report: "Issues logging in with Github"
Last updateThis incident has been resolved.
Workaround is in place and login is functional. We are continuing to monitor the system.
Problems appear to be related to Github's education api endpoint. A workaround fix is in progress.
We are currently investigating this issue.
Report: "Issues with the flags feature"
Last updateProblems with our timeseries database led to the flags tab responding slowly or timing out. This has been resolved.
Report: "Database issues"
Last updateThis incident has been resolved.
The database crashed and is in recovery mode. It should be back shortly.
Report: "Database Issues leading to Downtime or hung Uploads"
Last updateWe encountered an issue with our database that caused upload jobs to be delayed or fail. These jobs will need to be uploaded to Codecov again. We apologize for the inconvenience. This appeared to last approximately a minute.
Report: "Database Issues leading to Downtime or hung Uploads"
Last updateWe encountered an issue with our database that caused upload jobs to be delayed or fail. These jobs will need to be uploaded to Codecov again. We apologize for the inconvenience. The issue ran 2 mins from 6:30 pm UTC to 6:32 pm UTC
Report: "API unexpected errors"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Database Issues leading to Downtime"
Last updateWe have encountered an issue with our database that is causing a system wide outage. We have identified this issue and are working on a fix.
Report: "Unexpected timeouts"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Unexpected Queue Backup"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Issue with login"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.