CircleCI Incident History

The issue impacting customers who use GitHub as their VCS while GitHub was undergoing a service degradation (https://www.githubstatus.com/incidents/yjysg0xrl67m) has now been resolved. GitHub has resolved the underlying issue and the affected functionality has returned to normal. We thank you for your patience.

GitHub is currently experiencing a service degradation (https://www.githubstatus.com/incidents/yjysg0xrl67m). Customers who use GitHub as their VCS may experience impact to their CircleCI platform experience as a result, impacting checkout and workflows. We will provide another update as we have more information to share. Thank you for your patience.

## Summary On July 20, 2026 from 00:22 to 01:45 UTC, CircleCI customers using our GitHub integrations experienced failures running pipelines and experienced workflows getting stuck. During this incident, some pipelines triggered by users using GitHub failed to start or ran as error pipelines. Customers whose pipelines failed or ran as error pipelines during this window should re-run them. This was caused by an [upstream API degradation at GitHub](https://www.githubstatus.com/incidents/ph5nns5y4gxj), which impacted many GitHub APIs that CircleCI uses to trigger and run pipelines. By 01:45 UTC, GitHub APIs recovered, and customer pipelines ran normally. The original CircleCI status page can be found [here](https://status.circleci.com/incidents/9lvbbbs9l87b).  ## What Happened \(all times UTC\) Beginning 00:22 on July 20, 2026, GitHub began returning elevated errors for the following API requests: * `applications/*/token` * `repos/*/commits` * `repos/*/*/contents/*` * `repos/*/*/hooks` * `repos/*/*/hooks/*` * `repos/*/*/keys` * `repos/*/*/pulls` * `repos/*/*/statuses/*` CircleCI relies on these requests to correctly trigger and run pipelines. At 00:22, our internal monitoring alerted us to the problem. Our team began investigating, and found that a small number of pipelines belonging to GitHub projects either failed to start or ran as error pipelines. This included scheduled pipelines and workflows. In addition, a small number of customer workflows experienced stuck jobs and required a re-run of the workflow to fix. At 01:45, GitHub recovered and pipeline processing returned to normal operating levels. Only pipelines triggered during the incident window were affected, and customers should re-run those pipelines. ## Future Prevention and Process Improvement We are actively working to improve the resilience of pipeline processing during GitHub service disruptions to reduce the customer impact of similar incidents in the future. We are scoping features that will help customers recover with reduced manual steps after incidents resolve. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. Please reach out to our support team with any questions or concerns.

GitHub API error rates have recovered to normal levels. During the incident, some customers may have seen pipelines fail to start or not receive status updates. Some pipelines may be stuck in a running state. * You should re-push commits or use the UI to manually trigger pipelines that didn't start. * Pipelines in a stuck running state should be canceled and rerun. If you encounter any issues please contact CircleCI support.

We are seeing signs of recovery from the GitHub API. We will stay in a monitoring state to verify that the API error rate has recovered.

We are also seeing errors processing push event webhooks due to the upstream API outage. Users may need to retry pipelines for these push events.

We have identified an issue with GitHub API requests that may cause some customers to see workflows stuck in a running state or workflows fail to start. GitHub has reported incidents on the APIs https://www.githubstatus.com/incidents/ph5nns5y4gxj https://www.githubstatus.com/incidents/8vfyvq16hzh9 We are monitoring the stability of GitHub APIs and will update this page as more information is available.

Between 19:24 UTC on 22 July and 01:50 UTC on 23 July, customers using the gen3 preview machine (Linux VM) resource classes experienced delays and, in some cases, jobs failing to start. We have removed the gen3 resource classes while we address stability issues affecting them. Unfortunately, workflows that we were unable to provision compute capacity for during this period were cancelled. Affected customers should change their resource class from gen3 to gen2 and rerun those workflows. We thank you for your patience while our team worked to resolve this.

We have identified the issue and are working with our provider to resolve the availability of gen 3 resources. Gen 1 and Gen 2 are both fully operational.

We are continuing to investigate elevated wait times affecting customers using gen3 machine executors. Affected jobs may take longer than usual to start. We will provide another update as soon as we have more information to share.

We are continuing to investigate elevated wait times affecting customers using gen3 machine executors. Affected jobs may take longer than usual to start. We will provide another update as soon as we have more information to share.

We are investigating delays in starting jobs using the gen3 resource class.

Between 03:18 UTC and 06:04 UTC on July 22, 2026, a subset of customers were unable to access app.circleci.com. The issue has been resolved and access has returned to normal. We thank you for your patience while our team worked on implementing a fix.

A subset of customers may have experienced errors loading app.circleci.com starting at 03:18 UTC. A fix has been deployed and we are monitoring for full recovery. Thank you for your patience while our engineers confirm recovery. We will provide another update shortly.

An incident affecting GitHub's API caused CircleCI customers to experience errors loading the web app and signing in, along with pipelines and scheduled workflows failing to start or running late. GitHub has resolved the underlying incident (https://www.githubstatus.com/) and all affected functionality has returned to normal. Customers whose jobs or pipelines failed may rerun them. Scheduled pipelines and workflows that were missed during the incident will not run automatically and will need to be re-triggered. We thank you for your patience.

What's happening The upstream incident affecting GitHub has been mitigated and is now being monitored on GitHub's side. You can follow GitHub's status at https://www.githubstatus.com/. What can you expect Access to the CircleCI web app, sign-in, and pipeline and workflow processing are returning to normal. You may still see intermittent errors or delayed scheduled pipelines and workflows as systems stabilize. Thank you for your patience while we monitor for full recovery.

We are continuing to see high failure rates from GitHub APIs which are impacting login and workflow processing. We will continue to monitor the GitHub API for recovery.

We're investigating an issue where customers may see errors loading the CircleCI web app and signing in, along with pipelines and scheduled workflows failing to start or running late. This is related to an ongoing incident affecting GitHub. You can follow GitHub's status at https://www.githubstatus.com/incidents/gxycch3076xk. Our engineers are actively investigating. We will provide another update as soon as we have more information to share.

## Summary From 18:00 UTC on July 13, 2026 to 23:18 UTC on July 14, 2026, some customers running jobs on CircleCI's macOS fleet experienced jobs that hung or failed to progress during GitHub fetch steps. For these customers, GitHub fetch steps which would ordinarily take 1-2 minutes took 20-30 minutes, and timed out. The issue was caused by a capacity and routing problem on the network path between our Mac infrastructure provider and GitHub, and the associated intermittent networking issues slowed down fetches from GitHub. During the entire incident, our team worked directly with our Mac infrastructure provider to identify and reroute traffic away from the affected network paths. We resolved an initial occurrence at 19:40 UTC on July 13. The issue recurred at 21:52 UTC that same day, we reopened the incident, and it was fully resolved by 23:18 UTC on July 14. Our macOS fleet had a similar issue several weeks ago. From 20:18 UTC on June 24, 2026 to 03:09 UTC on June 25, 2026, customers experienced a [similar incident](https://status.circleci.com/incidents/gvysjmkf4ct9) affecting the ability of macOS jobs to reach GitHub. That incident was also caused by a network routing problem in our Mac infrastructure provider. We thank our customers for their patience while we worked through this incident. Please see below for specific actions which CircleCI and our Mac infrastructure provider will be taking. Please reach out to our support team with any questions or concerns. The status pages for this incident can be found [here](https://status.circleci.com/incidents/n1tc2lw9q7l0) and [here](https://status.circleci.com/incidents/7cnl777wp9qz). ## Background CircleCI's macOS jobs are hosted on a third-party Mac infrastructure provider. That provider connects to the broader internet, including services like GitHub, over multiple redundant upstream network paths. When one of those paths is experiencing reduced capacity or a routing problem, jobs that are fetching code or dependencies from a destination along that network path can slow down or hang intermittently, even when CircleCI's own platform, the third-party infrastructure provider, and the destination service are all operating normally. ## What Happened _\(All times UTC\)_ At 18:00 on July 13, some customers running macOS jobs began experiencing intermittent failures and delays fetching code and dependencies from GitHub. We opened an investigation and alerted our customers via our status page at 18:55. By 19:04, our Mac infrastructure provider identified increased latency on one of its network paths and rerouted traffic around it. Fetch times recovered, and we moved the incident to `Monitoring` at 19:29, and to `Resolved` at 19:40. At 20:44 UTC, customers reported that the issue had returned. We reopened the incident and updated our status page to `Investigating` at 21:52. Over the following two hours, we worked closely with our infrastructure provider to gather diagnostic data in an effort to isolate the affected network path. In the meantime, we published guidance recommending that customers increase their `no_output_timeout` setting to 15–20 minutes, to allow more time for GitHub fetches to complete during the intermittent slowdowns, and moved the status page to `Monitoring` at 23:35. After publishing the workaround guidance at 23:35 UTC on July 13, we expected the underlying network issue to improve overnight. It did not. At 13:59 UTC on July 14, we confirmed that customers were still experiencing intermittent failures and moved the status page back to `Investigating`. Our engineering team reproduced the failure directly using our own testing tools, which helped confirm this was a general `git-fetch` issue rather than something specific to any particular build tool, container image, or software update. At 15:35, we updated the status page to share that our infrastructure provider was testing changes to its network configuration to isolate the source of the issue. At 16:31, we updated the status page again to share that we'd identified the cause as reduced inbound bandwidth on a specific route, and that we were working with our provider to shift traffic to a different route. At 16:36, our infrastructure provider identified the affected network paths and shifted traffic away from them, fetch times recovered, and at 16:58, we updated the status page to `Monitoring`. By early afternoon, some customers were again seeing slow fetch times, and we moved the status page back to `Investigating` at 20:18. Our infrastructure provider identified two additional network paths with degraded performance and shifted traffic away from them by 22:07. Fetch times stabilized over the following hour, and we marked the incident `Resolved` at 23:18. In total, customers running Mac jobs may have experienced intermittent job slowdowns or failures for portions of the window between 18:00 UTC on July 13 and 23:18 UTC on July 14. Our infrastructure provider has since reported clean test results across its network and believes the underlying capacity issue originated further upstream, closer to GitHub, rather than within its own network. We are continuing to monitor this closely. ## Future Prevention and Process Improvement We are taking the following steps to prevent a recurrence and improve our response time: **We are building additional automated monitoring for macOS fleet GitHub-fetch performance.** Our extensive automated monitoring did not catch this incident. We are now adding end-to-end monitoring for GitHub fetches in particular, in order to detect similar issues proactively. **We are working directly with our Mac infrastructure provider on faster, more proactive detection.** We are asking our provider to build monitoring that can detect a degraded network path and reroute around it automatically, rather than relying on CircleCI to identify and request a reroute during an active incident. **We are turning the diagnostic tooling we built during this incident into a standing capability.** This will let us detect and reproduce this class of network failure on demand going forward, rather than assembling test infrastructure during an active incident. **We are reviewing our incident response process for a sufficient confirmation window before marking a network-related incident Resolved.** This incident briefly recurred after an early, premature resolution; we are formalizing a minimum period of confirmed-clean monitoring before closing incidents of this type. We will continue to provide regular updates along the way as investigation and remediation continues. **We are actively moving from a single provider for Mac infrastructure** **to multiple providers.** This will allow CircleCI to route incoming customer workloads to the most available and best-performing provider. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. Please reach out to our support team with any questions or concerns.

We continue to see significant VCS fetch time recovery over the last couple of hours. We're going to resolve this incident, and our engineering team will continue to monitor closely. We greatly appreciate your patience as we worked through this with our Mac infrastructure provider. Please reach out to our Support team if you have any issues.

Our Mac infrastructure provider has shifted more traffic to working network providers and we're starting to see significant recovery again in VCS fetch times. As always, our engineering team will continue to monitor this closely and we'll provide another update soon.

We're starting to see VCS fetch times rising again. Our engineering team is actively investigating and working with our Mac infrastructure provider to resolve this. Again, we appreciate your patience.

We're continuing to see slow, progressive recovery in VCS fetch times. We're going to continuing monitoring and update again soon. We appreciate your patience.

Our Mac infrastructure provider has shifted the inbound routes and we are seeing recovery in VCS fetch times. We will be continuing to monitor this issue over the next few hours. We'll update again soon.

We've identified the issue as some inbound routes from the affected VCS provider causing significantly lower available bandwidth which is negatively impacting repo fetch times. We're working with our Mac infrastructure provider to attempt to shift inbound traffic to a different route, however this process may take an extended period of time to complete. We will continue to provide updates as they become available. As we've shared in previous updates, increasing the no_output_timeout to 15-20 min continues to help: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit

Our Mac infrastructure provider is testing changes to their network configuration in an effort to isolate the source of the issue. We're actively validating whether this resolves the impact and will follow up with another update soon. In the meantime, increasing the no_output_timeout to 15-20 min continues to help: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit

Unfortunately there are still some customers that may be experiencing jobs that hang or fail to progress during Github fetch steps due to ongoing intermittent networking issues with our Mac infrastructure provider. Our engineers are actively working with them to resolve this. We appreciate your patience. In the meantime, we've seen success with increasing the no_output_timeout to 15 - 20 min to allow more time for Github fetches. This can be done by following this community guide: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit

We've identified the issue. Some customers may experience jobs that hang or fail to progress during Github fetch steps due to intermittent networking issues. Our engineers are actively monitoring. In the meantime, we've seen success with increasing the no_output_timeout to 15 - 20 min to allow more time for Github fetches. This can be done by following this community guide: https://support.circleci.com/hc/en-us/articles/360007188574-Build-has-Hit-Timeout-Limit

We've been able to reproduce the issue, but it still appears to be very much intermittent. We're continuing to work with our Mac infrastructure provider to help diagnose the issue. We'll update again soon.

We're still investigating these intermittent network issues. We'll update again shortly, we appreciate your patience.

We're received some reports of intermittent network issues between our mac infrastructure and a VCS provider again. We are investigating and will provide updates.

On July 13 between 15:30 UTC and 22:40 UTC, we experienced delays starting jobs on Docker (Gen 2) resource classes due to capacity constraints from our cloud provider. The same issue recurred on July 14 between 15:45 UTC and 22:50 UTC. Delays reached up to 25 minutes on July 13 and up to 24 minutes on July 14. The medium+ and 2 X-large+ resource classes saw the longest waits. Earlier updates in this incident reported figures based on average wait times, which didn't reflect the maximum impact some customers may have experienced. Wait times have now returned to normal. We are continuing to work with our cloud provider to add capacity ahead of the next peak period. Thank you for your patience.

On July 13 between 15:30 UTC and 22:40 UTC, we experienced delays starting jobs on Docker (Gen 2) resource classes due to capacity constraints from our cloud provider. Delays reached up to approximately 1 minute 48 seconds. We are continuing to experience these delays again since approximately 15:45 UTC. So far, delays have reached up to approximately 1 minute 19 seconds. We are working with our cloud provider to add capacity and will update this incident as the situation changes.

## Summary From 01:33 UTC to 05:18 UTC on July 14, 2026, customers with a Bitbucket-linked identity were unable to log in to CircleCI, and some customers experienced workflows that failed to start or update, due to a change in how Bitbucket's OAuth service reports account permissions. At 05:07 UTC, we deployed a fix that corrected how our systems read the updated permissions field. Some customers needed to log out and re-authenticate their Bitbucket integration after we deployed the fix. Our systems continued processing the backlog of affected jobs until 08:39 UTC. We thank our customers for their patience as we resolved this incident. Please reach out to our support team with any questions or concerns. The status page for this incident can be found [here](https://status.circleci.com/incidents/gsyjwybg477g). ## Background CircleCI supports logging in with a GitHub account, a Bitbucket account, or an email and password. When a customer logs in with a Bitbucket-linked identity, or when CircleCI needs to refresh the customer's access on their behalf, we exchange an authorization token with Bitbucket's OAuth service. This exchange includes a list of the permissions, or "scopes," the customer has granted us, which Bitbucket and CircleCI use to confirm what CircleCI is authorized to do on the customer’s behalf. ## What Happened \(All times UTC\) On April 8, 2026, [Bitbucket announced a change to its OAuth service](https://developer.atlassian.com/cloud/bitbucket/changelog/#CHANGE-3139): it would be renaming the field used to report a customer's granted permissions. Bitbucket ran a transition period during which both the old and new field names were available, then fully removed the old field name on May 4, 2026. Our systems had not been updated to recognize the new field name, so once Bitbucket fully phased out the old one, requests that depended on it began to fail. At 01:33 on July 14, 2026, our systems began failing to process permissions information returned for Bitbucket-linked accounts, because our systems still expected the old permissions structure. This caused login attempts for all Bitbucket-linked accounts to fail, including for customers who log in with GitHub but have a Bitbucket identity linked to their account. At 01:59, automated monitoring alerted our engineering team to a spike in errors on the affected systems. The team began investigating immediately, confirmed customer impact at 02:38, and alerted customers via our status page at 02:51. By 03:00, the team had isolated the failures to the renamed Bitbucket field. Beginning at 03:17, workflow status updates for Bitbucket pipelines belonging to customers with an expired access token began to be dropped. The Bitbucket permissions check is used broadly across our platform, and these permissions failures also affected some of our internal job-processing systems. This caused a subset of workflows to become stuck without a final status, and caused some pull requests to show missing or stuck status checks. At 04:21, the team deployed an initial fix that resolved the underlying issue, restoring the login flow and returning our internal permissions system to normal operation. Some affected customers needed to log out and log back in to pick up the fix. By 05:07, the team deployed two additional fixes so our systems would begin accepting Bitbucket's new permissions field format going forward. We resolved the incident at 05:18. Some customers did need to log out and re-authenticate their Bitbucket integration before their account fully recovered. Our systems continued processing a backlog of affected jobs, returning to normal levels by approximately 08:39. ## Future Prevention and Process Improvement We are taking the following steps to prevent a recurrence and improve our response time: **We are hardening our authorization code against upstream API changes.** This incident happened because our system did not gracefully handle a renamed field in a response from a third-party identity provider. We will be immediately updating our authorization code so that unexpected or missing fields from GitHub, Bitbucket, and GitLab are handled safely. **We are improving how we track upstream provider changes.** We currently already monitor changelogs from our identity providers for exactly this kind of breaking change, but we didn't add this particular provider change to our monitoring in time to catch it before it shipped. We are auditing and expanding this monitoring so provider announcements reach our team before they affect customers. **We are improving how we classify and communicate incidents in their earliest minutes.** This incident's initial classification did not immediately reflect its customer-facing severity. We are refining our incident tooling and guidance to help engineers identify and communicate customer impact more quickly. **We are improving the resilience of our workflow-processing pipeline.** A downstream failure in this incident caused some workflow status updates to be dropped rather than retried or clearly surfaced. We are reviewing this system's retry and error-handling behavior so similar downstream failures are more visible and easier to recover from. Customer experience is our top priority, and we commit to continually improving the reliability of our systems to match the trust that our customers place in us. Please reach out to our support team with any questions or concerns.

The issue where customers logging in with a Bitbucket-linked identity experienced login errors, and some customers experienced workflows that failed to start or did not update, has been resolved. What you may still experience and need to do - If you use Bitbucket as your login method, please log out and reauthenticate the CircleCI Bitbucket integration on your account. - If any of your workflows are stuck or showing missing status checks, please rerun them to pick up the fix. - If you continue to experience issues after taking these steps, please reach out to CircleCI Support. We thank you for your patience while our team worked on implementing a fix.

What's impacted Some customers logging in with a Bitbucket-linked identity were affected. This may also have included customers logging in with GitHub who have a linked Bitbucket identity on their account. Some customers also experienced workflows that were not starting or updating. What can you expect A fix has been deployed. We are monitoring the situation to confirm full recovery. Next update We will provide an update if anything changes

What's impacted Some customers logging in with a Bitbucket-linked identity are affected. This may also include customers logging in with GitHub but have a linked Bitbucket identity on their account. Additionally, some customers are experiencing workflows that are not starting or updating. What can you expect Affected customers may see login errors. Some customers may also notice workflows that fail to start or do not update as expected. Thank you for your patience while our engineers investigate. Next update We will provide another update as soon as we have more information to share

Our team continues to investigate login errors affecting some customers using Bitbucket as their identity provider.

We are currently investigating login errors affecting some customers using Bitbucket as their identity provider.

We've confirmed the fix with our infrastructure provider and successful tests on our end. If customers continue to see any failed runs, please run again for a successful connection and reach out to Support if any further issues.

Our Mac infrastructure provider has implemented a fix, we're starting to see recovery. We're monitoring and we'll update again shortly.

We're working with our Mac infrastructure provider to attempt a mitigation. Will update again shortly.

We've received reports of intermittent network issues between our mac infrastructure and a VCS provider. We are investigating and will provide updates.