Historical record of incidents for Travis CI
Report: "Scheduled Database Maintenance – April 26th Weekend"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
We will be performing essential database maintenance over the weekend of April 26th to upgrade our database systems.During this period, the Travis CI system may experience up to 16 hours of total downtime, and builds will not be executed. Please plan your workflows accordingly, as disruptions may occur within this timeframe.We understand that maintenance can be inconvenient, and we appreciate your patience as we work to enhance our services. Our team will strive to minimize the impact and restore full functionality as quickly as possible.For real-time updates during the maintenance period, please visit our status page.Thank you for your understanding.
Report: "Billing subsystem urgent maintenance"
Last updateWe will be performing a brief maintenance window to address an issue with our billing system related to user license consumption, as reported by some users. This maintenance will include cache optimization.We understand that downtime can be inconvenient, and we appreciate your patience as we work to enhance our services. Our team will strive to minimize any disruptions and restore full functionality as quickly as possible.Updates will be available on our status page throughout the maintenance period.Thank you for your understanding.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Trials not working"
Last updateThis incident has been resolved.
Trials are currently unavailable. The issue has been identified, and we are preparing a fix. In the meantime, please reach out to support@travis-ci.com if you have any questions.
Report: "DPLv2 dependency update causes DPLv2 deployments to fail"
Last updateBuild images updates and DPLv2 updates will drop continuously. DPLv2 will be promoted as default DPL gem version at Sep-30th.
Updates in DPLv2 and Travis CI build environments applied. We are monitoring. DPLv2 may be promoted to be default DPL function after Sep 14th.
TCI Enterprise server version 3.0.58 patch published and recommended to be used no later than by Sep-14th, 2024.
Updating build environment images still in progress - additional dependencies were identified and are worked on. Until build environment updates are finished, DPLv1 works normally, DPLv2 users are recommended to continue using uri gem update workaround. TCI Enterprise server version will receive patch 3.0.58 shortly. Patch will ensure stability of default DPL gem used while still allowing to select different DPL version - please watch enterprise changelog.
We have reverted DPLv2 gem from being latest stable in gem repository, 1.10.16 is now latest stable. This doesn't affect TCI Hosted users. This should prevent TCI Enterprise server environments not updated to TCIE v3.1 suffer from necessity to update individual .travis.yml files, via appointing specific dpl gem version. Image updates in progress and will be tested afterwards. Next update expected on September/2.
We have released DPLv2 as stable gem on Aug-28th into Travis gem collection. There's an issue for DPLv2 users and there's available workaround. The 'uri' gem updated 2 days ago caused DPLv2 2.0.5 not work correctly in Travis CI build environments due to dependency chain. This means some of your deployment recipes using 'deploy' and `dpl_version: 2.0.5` keyword can cease working. We recommend our DPLv2 users to add temporarily following snippet to your .travis.yml as a workaround: ``` before_deploy: - gem update uri ``` We are updating our build environments and will close this incident once that's done.
Report: "arm64 jobs not running"
Last updateThis incident has been resolved.
We are investigating an issue that is preventing arm64 jobs from running correctly.
Report: "s390x and ppc64le jobs not running"
Last updateThis incident has been resolved for Bionic and Focal builds. We're working on Jammy builds that should be fixed soon, updates will be posted to our changelog page.
Bionic and Focal builds should work now, fix deployed.
We have identified an issue that is preventing s390x and ppc64le jobs from running correctly. We’re actively working to address this problem.
Report: "Builds not being processed"
Last updateThis issue has been resolved and now builds are running as expected.
We are currently investigating an issue that is preventing some builds from working properly.
Report: "macOS builds failing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented
We have observed some issues with our Mac build process. Our team is investigating these issues and we will be providing further updates as more information becomes available
Report: "Build jobs from branches/tags on GitHub ran on main/master branch in Travis CI"
Last updateIncident resolved.
A fix for handling GitHub push events was implemented and we are monitoring the system now.
Issues with branch/tag based builds reported since Nov-9 have been confirmed to be on Travis CI side. We have identified the reason and confirmed it's due to GitHub push event handling on Travis CI side. Working on the fix. Build infrastructure is fully operational
Report: "GitHub sync blocked or lasting long - degraded performance"
Last updateData synchronization issues with GitHub have been fully resolved and all systems are operational
A fix is in place and we're monitoring.
We received information on issues with data synchronization with GitHub since Nov/1st. We have identified reason and are working on restoring system to normal operation.
Report: "Travis API errors."
Last updateOperation back to normal.
Issue found and fixed. Monitoring system.
Travis API errors discovered. Builds may be breaking.
Report: "Build job errors when acquiring repository from GitHub (GitHub SSH key update)"
Last updateThe issue seems resolved. If you encounter further roadbloacks in repositories using Custom SSH Key in Travis CI, specifically on git repository cloning stage, please validate if a matching key pair is still available at GitHub. If not, removing old Custom SSH Key in Travis CI repository settings and regenerating the Custom SSH Key in Travis CI may help unblock your build jobs.
We have deployed hotfix replacing main Github host RSA SSH keys and are monitoring the builds. Users with Custom SSH Key in Travis CI repository settings should be able to build correctly now.
Users utilizing Custom SSH Key in Travis CI private repository settings may experience build errors or warnings on git checkout. This is due main SSH keys being updated by GitHub (see https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ ). We are working on updating our build environments configuration to utilize updated GitHub SSH keys. Users not utilizing Custom SSH keys are not affected.
Report: "Synchronization with GitHub fails"
Last updateThis incident has been resolved. You may need to reinstall Travis CI GitHub App if you added a new repository or created a new account between 19 December 2022 12:06:34 and 20 December 2022 GMT 13:00:00.
User synchronization operational. The queue drain was finished at GMT 13:00. We will be monitoring the situation over next hours, looking for any remaining issues caused by the incident. You may need to reinstall Travis CI GitHub App if you added a new repository or created a new account between 19 December 2022 12:06:34 and 20 December 2022 GMT 13:00:00.
We're draining the pending messages to GitHub API queue. It may take some time before all affected accounts are enabled again.
The issue has been identified, fix is being implemented. Adding/activating new repos will not work for now.
We are currently investigating the issue. The symptoms are that either adding new repository or installing Travis CI GitHub App from scratch results in repositories not being activated and builds cannot be triggered.
Report: "Some build requests are not being triggered."
Last updateSome of the build requests they both triggered with API and Git push operations are not being processed. We are still investigating the root cause behind this issue.
Report: "Some builds are not being triggerred"
Last updateThis incident has been resolved.
We have implemented the fix and monitoring the issue.
We have identified the issue where some of the builds are not being triggered and got an undetailed error when they are triggered.
Report: "Some users are facing issues when triggering a build on Travis CI Platform."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Summary: Users on Old Travis CI pricing model: We have identified that the Users/Orgs on old Travis CI pricing model are unable to trigger builds. Description: Users are not able to trigger builds on Travis CI Platform with a message "We are unable to start your build at this time. You exceeded the number of users allowed for your plan. Please review your plan details and follow the steps to resolution." Our engineering team is continually working towards the resolution to this issue and we will provide further updates as soon as a fix is deployed.
The issue has been identified and a fix is being implemented.
Report: "travis-ci.com throws "504 Gateway Timeout" error"
Last updateThe main site https://travis-ci.com/ is back on field.
The main page at https://travis-ci.com is down. Your builds are still executed. We're reviewing the main site case, it may remain inactive for a full day or so. The API and https://app.travis-ci.com can be still accessed directly and you can run, view, and manage your builds as usual.
Report: "Builds are not triggering for app.travis-ci.com"
Last updateThis incident has been resolved. If you face any issues with builds on Travis Platform, you can reach our support staff at support@travis-ci.com
Our Infra team has implemented a fix that should resolve this issue. Thank you for your patience and cooperation!
Sorry, some users are unable to trigger builds on GitHub commits. We are working on it and hope to restore the functionality shortly. Sorry for the inconvenience!
Report: "Builds are not triggering for travis-ci.com"
Last updateThis incident has been resolved. If you face any issues with builds on Travis Platform, you can reach our support staff at support@travis-ci.com
Our Infra team has implemented a fix that should resolve this issue. Thank you for your patience and cooperation!
Sorry, some users are unable to trigger builds on GitHub commits. We are working on it and hope to restore the functionality shortly. Sorry for the inconvenience!
Report: "Some builds are being rejected"
Last updateThis incident has been resolved.
The fix has been implemented and build requests are being processed as usual. We are still monitoring the system
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Some build requests are being rejected and this causes builds to be triggered. We are currently investigating the issue.
Report: "Queued jobs are taking time to process"
Last updateThis incident has been resolved.
We are actively fixing this issue and will report back as soon this resolves. We are very sorry for the inconvenience in the meantime.
We are continuing to work on a fix for this issue.
We have identified the problem and a fix is in process.
Some builds may have trouble on the queued stage. We are currently investigating the issue.
Report: "Some builds are taking a long time to queue"
Last updateThis incident has been resolved.
We have identified the problem and queued jobs are already started processing.
Some builds may have trouble on the queued stage, we are currently investigating the issue.
Report: "PPC64 Jobs Are Failing"
Last updateThis incident has been resolved.
We observed some ppc64 jobs are taking longer to start. In addition, we've noticed an increased error rate. We are working to resolve these and will provide updates shortly.
Report: "Builds Queued for longer than normal"
Last updateThis incident has been resolved.
Our queue is currently draining fast and processing of builds should be back to normal
Builds are currently queued for longer than normal. We are investigating this issue and will provide an update shortly.
Report: "Some builds may be affected due to Google Incident"
Last updateThis incident has been resolved.
Due to the ongoing incident with Google's Compute Engine [1], some jobs may fail to run. We are monitoring the Google Status and will provide updates. Thank you [1] https://status.cloud.google.com/incident/cloud-networking/21006#21006002
Report: "Builds are not triggering for travis-ci.com"
Last updateWe are extremely sorry about the outage on this incident and we would like to share some action that has been taken; * Issue reported date: The issue has been reported on 24.09.2020 09:12 UTC. * Issue: **Unstable VCS service, high memory usage** * Reason: Issue in Bitbucket implementation, infinity loop when a user has been suspended or without permissions to the repository * Solution: Check suspended flag before API calls, infinite loop retries removed. * Fixed time: 24.09.2020 10:14 UTC * Action required from users: None * Issue status: Fixed Thank you for your understanding.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are investigating the root cause behind why builds are not triggered for travis-ci.com
Report: ".org Not processing builds"
Last updateBuilds are being processed normally.
The issue has been resolved and we are monitoring the system.
We are investigating reports of builds not being processed on .org.
Report: "Partial Outage on Notifications"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified issues with infrastructure handling notifications, which leads to these not been fired. A fix is being worked on.
Report: "Backlog of Jobs on .ORG"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating a large number of queued jobs on our .ORG infrastructure for MacOS builds
Report: "Queued Jobs on .COM"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an uptick in queued jobs on our .COM platform
Report: ".com API intermittent failure"
Last updateThis issue has been resolved.
A fix has been implemented and we are monitoring the results.
We have deployed the fixes and monitoring their effectiveness.
We have identified the cause and are working on a fix.
We are investigating reports of intermittent failures on API.
Report: "Degraded GitHub integration"
Last updateThe service are operating normally.
Our services are degraded due to GitHub's ongoing issues.
Report: "Intermittent build start or failure issue"
Last updateThe issue related to macOS builds has been resolved.
Investigation underway we are in close contact with the service provider.
You may experience issues with macOS builds due to an issue with our infrastructure service provider. We are closely looking into this issue and more to come!
Report: "Linux and Windows VMs not booting"
Last updateThe service provider issue has been resolved. The builds should be processed normally.
Our service provider is experiencing issues, and at the moment we are unable to boot VMs to run the builds on Linux and Windows.
Report: "API is down on .ORG"
Last updateThis incident has been resolved.
We are continuing to monitor our infrastructure and one of our upstream providers, Google [1]. We will provide additional updates once these systems are fully functional. Thank you [1] https://status.cloud.google.com/incident/cloud-networking/20004
We implemented a fix for the issue and are monitoring the results. Thank you
We are investigating an outage on the .ORG API. We will provide updates based on our findings. Thank you
Report: "GitHub status may not be posted on commits occasionally from builds using the legacy Services integration"
Last updateThe issue has been resolved. Please write to support@travis-ci.com if you encounter any similar problems. Thank you
We've deployed additional measures to mitigate this issue and are monitoring the effects. We will roll out additional fixes as required. Thanks for your patience
We have identified the source of these errors and are working on a fix.
We are investigating the reports of GitHub commit status not being posted. The issue only affects those repositories using the legacy Services integration (i.e., all .org repositories and some .com ones), and only under some circumstances. You can migrate your repositories to GitHub Apps on https://travis-ci.com/account/repositories.
Report: "Build Queues and Build Statuses"
Last updateThank you for your patience and understanding. This incident has been resolved by taking another look and tests about builds statuses and build trigger mechanism. The solution was deployed at 14:00 PM UTC.
The issue related to queued builds and build statuses is still partially down. We have investigated the issue and currently working on the components to up again.
Report: "Builds not starting/queuing or stucks on .COM"
Last updateThank you for your patience. The incident has been resolved and builds should behave as normal.
We have observed the root cause of the problem with the builds isn't starting due to misusage. Currently, we are recovering the infrastructure.
Report: "MacOS Builds Failing on .COM and .ORG"
Last updateWe observed an issue with a small portion of macOS builds, jobs routed to a particular infrastructure unit took longer than expected to start or failed. The issue was resolved at 18h31 UTC
Report: "MacOS Builds Failing on .COM and .ORG"
Last updatePlease see the MacStadium Post-Mortem \[1\] for details about this incident. Thank you! \[1\] [https://status.macstadium.com/incidents/l464rrkdg4yn](https://status.macstadium.com/incidents/l464rrkdg4yn)
Thank you for your patience. Our upstream provider has reported that a fix has been implemented [1]. We have also seen improvements on our end and so we are marking this issue resolved. Thank you [1] https://status.macstadium.com/incidents/l464rrkdg4yn
We have observed issues with Mac OSX builds that are failing on both .com and .org due to a major outage at our upstream provider. We are looking into this and will provide additional updates.
Report: "Internal Mac build errors"
Last updateThe Mac build capacity has been brought back to normal.
We have removed the affected hardware out of the executor pool. Requests should work as expected at this time.
We have identified an issue with a portion of Mac infrastructure which prevents builds from running correctly. We are working on mitigating the impact of this problem. Affected jobs would indicate "An error occurred while generating the build script." in the logs.
Report: "Linux builds not being processed for .com"
Last updateJobs have been processed. If you see problems, please contact us at support@travis-ci.com.
We are planning on restarting stuck jobs between 20:40 and 21:40 UTC and monitor the queue following the restart.
We are restarting the jobs stuck in "queued" state at this time. Affected users may restart them themselves. If you continue to see problems, please contact us at support@travis-ci.com.
We have confirmed that a portion of jobs requested today were not processed correctly. These jobs are stuck in the "queued" state. We confirmed that restarting affected jobs solves the issue.
Report: "Linux builds not being processed for .com"
Last updateJobs have been processed. If you see problems, please contact us at support@travis-ci.com.
We are restarting the jobs stuck in "queued" state at this time. Affected users may restart them themselves. If you continue to see problems, please contact us at support@travis-ci.com.
We have confirmed that a portion of jobs requested starting around 07:00 UTC today were not processed correctly. These jobs are stuck in the "queued" state. We confirmed that restarting affected jobs solves the issue.
We are investigating reports of Linux jobs not being processed for .com (both private and public).
Report: "Issue with IBM Z builds on Travis CI"
Last updateThis incident has been resolved.
We are seeing builds on IBM Z starting normally. A fix for both .com and .org has been implemented and we are monitoring the results.
We are investigating an issue with IBM Z builds.
Report: "Unreliable or intermittent network connectivity for Linux and Windows builds"
Last updateThis incident has been resolved.
The NAT gateway changes have been fully reverted. Network performance and reliability in your builds should be back to normal. We’ll be monitoring for a short while before closing this incident.
At this point, we've reverted half of the NAT gateway updates and a 50% of the Linux and Windows jobs currently processed have better network performance already. We'll provide another update as soon as this process is complete. Thanks for your patience.
We've identified an issue with the latest changes to update the NAT gateways. While this change was meant to improve maintainability, we are rolling it back due to the degraded network performance. We're expecting this provides network speed improvements within the next hour. We will provide additional updates at the top of the hour. Thank you
We are continuing to investigate this issue.
We are receiving reports of Linux and Windows builds where network connectivity is unreliable, intermittently working or slow. We are opening this incident to acknowledge the issue and also keep you posted on our work to solve it. Thank you for your patience.
Report: "Network Connectivity Errors"
Last updateThis incident has been resolved.
Builds are back to running normally. We will continue to monitor the situation for a short while before closing the incident.
Builds that were running on the affected pool have now finished and all builds are back to normal. We're monitoring closely and will post again once we're in the clear.
We have identified the production systems affected by this issue and have removed these from the pool. No new builds will be affected by this, existing builds in the pool will be allowed to finish but might have network connectivity issues. Please expect additional updates shortly. Thank you
We are continuing to investigate this issue.
We have received several reports of builds failing due to an unsuccessful TLS handshake to *whereami.travis-ci.com:443* on one of our production servers. We are opening this incident to acknowledge the issue and also keep you posted on our work to solve it. Thank you for your patience.
Report: "Slow Linux and Windows builds on travis-ci.org and travis-ci.com"
Last updateBacklogs have cleared on both sites hence we are resolving this incident.
We are seeing a backlog of Linux and Windows builds on both travis-ci.org and travis-ci.com. We are adding capacity as we speak and we’ll keep you posted on the results. Thank you for your patience.
Report: "Slow macOS builds on travis-ci.com and Linux & Windows builds on travis-ci.org"
Last updateAll backlogs have cleared now. Sorry again for the inconvenience and happy building!
macOS builds backlog on travis-ci.com has cleared. We are almost done processing the Linux and Windows backlog on travis-ci.org. Thank you for your patience!
We are seeing a backlog of - macOS builds on travis-ci.com - Linux and Windows builds on travis-ci.org We are looking into adding back capacity to process them.
Report: "Delayed posting of Github Status"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have cleared the back logs, and most status have now been posted. If your GitHub Apps is not posting statuses, please try disabling and re-enabling the repository, and restarting (parts of) the affected builds.
We have received reports of delayed posting of Github Statuses. We are working to clear the backlog of such that should have been sent out.
Report: "Slow booting Linux and Windows builds for both open source and private repositories"
Last updateBuilds are running smoothly and we have significant remaining headroom. Please send your commits our way! We are closing this incident for now and are thanking you again for you enduring patience.
We’ve added more capacity and have been able to process the accumulated backlogs on both .com and .org. We will monitor the situation for a short while to ensure everything is stable.
We are seeing backlogs of jobs growing on both travis-ci.org and travis-ci.com stemming from yesterday’s incident. We’ve been able to add more resources but not to the extent to be able to handle all build requests we receive during peak hours. Hence, your builds might be starting slower than usual. We are continuing to work with our infrastructure provider to improve the situation. We are sorry for the inconvenience this is causing you.
Report: "Slow booting Linux builds for both open source and private repositories"
Last updateBacklog on travis-ci.org has cleared has of 2019-07-03 2:14 UTC. Thanks again for your continued patience during this incident.
Backlog on travis-ci.com has cleared around 21:55 UTC. We are monitoring travis-ci.org's backlog closely.
Sorry for the lack of updates. It's been challenging to remain below the API rate-limit and the backlogs haven't been decreasing at a steady pace as previously mentioned. We now confirm that the backlog on .com is decreasing at a comfortable pace while the backlog on .org is over its peak. Thanks for hanging in there with us.
We’ve made progress in being able to stay under the API rate-limit and things are looking better on travis-ci.com. A higher backlog of jobs remains on travis-ci.org. We’ll keep you posted on the status of each backlog on a timely basis.
We are currently hitting the API rate-limit when using our Cloud Provider’s API that is preventing us from creating/deleting VMs. We are looking into getting the system back under the API rate-limit threshold. Thank you for your patience.
We are receiving reports of slow booting Linux builds. We are currently looking into it.
Report: "Maven connectivity issues"
Last updateWe have deployed the change to use Google's maven central mirror to improve the availability of Maven repositories.
The Sonatype team has safe listed all of our IP addresses and connections to Maven Central should now work. We will continue to monitor and please do not hesitate to contact us at support@travis-ci.com if you’re still observing Maven connectivity issues.
The Sonatype team has been investigating to safelist all of our IP addresses and confirm if this fixes the issues connecting to Maven Central: https://issues.sonatype.org/browse/MVNCENTRAL-4985. We'll continue to post updates as we have more information to share. For centralized reports of this situation please use the community forum at: https://travis-ci.community/t/continuous-maven-repo-403/3908/5
We're currently investigating an issue with Maven builds accessing Maven central, with error "403" and "Access Denied" We believe this could be related to the extending of IP ranges earlier this week and the need for them to be whitelisted by Maven. We've reached out to Sonatype, and we are working to resolve this issue as soon as possible.