Is CodeShip Down Right Now? Check if there is a current outage ongoing.

CodeShip builds are affected by an outage on one of the service providers.

This incident has been resolved.

We are continuing to monitor recovering builds. Please contact support if you continue to see issues.

Build triggering is stabilizing. You may need to stop and restart previous builds to run them again.

We are investigating intermittent build issues. During this time your build may not start or could be delayed before starting.

This incident has been resolved.

We are noting issues with Pro build image caching as part of this incident as well.

We continue to see issues with some Pro build machines not coming up and are continuing to monitor the open AWS incident. https://health.aws.amazon.com/health/status

We are seeing delays with AWS provisioning new build machines for Pro builds. Builds are still running, but in some cases your build may take longer to start.

This incident has been resolved.

Project build history is showing again and we continue to monitor.

We identified an issue blocking API requests and are making adjustments now.

Project build history is not showing and we are investigating. Using the build link listed in your CI provider might be a workaround to access build logs right now.

This incident has been resolved.

Instances are starting normally again and we are monitoring.

We are observing issues bringing up instances for CodeShip Pro builds.

This incident has been resolved.

We are monitoring intermittent availability issues caused by our infrastructure platform (https://status.heroku.com/). This issue was caused by their provider (AWS) having an issue with ELB APIs (which is now resolved).

We are continuing to monitor for any further issues.

We are monitoring intermittent availability issues caused by our infrastructure platform (https://status.heroku.com/).

This incident has been resolved.

Continuing to monitor https://status.aws.amazon.com and seeing improvement across our systems.

Seeing increased error rates from AWS and are monitoring https://status.aws.amazon.com.

This incident has been resolved.

AWS is reporting that systems are coming back up and we are seeing recovery across our systems. We will continue to monitor.

Continuing to monitor https://status.aws.amazon.com for outage updates. Build instability will continue while AWS works on recovery.

Seeing increased error rates from AWS and are monitoring https://status.aws.amazon.com.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Upstream providers indicated issues are resolved, but we are continuing to see some instability in our systems. We are working to further diagnose the problem.

Our service provider continues remediation efforts on the affected data services. (Heroku Incident - https://status.heroku.com/incidents/2361) (AWS is also flagging an issue with EBS volumes in US-EAST-1 - Heroku has not confirmed if that is the root cause as yet - although they have confirmed that it is an upstream provider issue for them too)

We are investigating an issue with the platform. Initial investigations indicate a problem with the data services offered by our platform provider. Heroku are aware of the incident - https://status.heroku.com/incidents/2361

This incident has been resolved.

A fix has been implemented and the CodeShip site is up again.

The CodeShip site is currently not reachable

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some builds are failing to clone the repository. The issue is currently being investigated

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Service should now be properly restored.

We are working to resolve issues preventing the CodeShip application from resolving. Service should be restored shortly.

This incident has been resolved.

A fix has been implemented and we are monitoring the system. If you noticed the build status page said "Trigger your first build" - then this was (incorrectly shown) because the build scheduling APIs were timing out.

We are investigating a build scheduling performance issue. There will be a short interruption to build process while we complete a system upgrade to alleviate stress on the system.

We have deployed a fix and confirmed it is working.

We have confirmed an issue with CodeShip Pro builds displaying the wrong UI and making logs inaccessible. We are in the process of restoring normal service for CodeShip Pro users.

We are hearing reports that some Pro builds are performing abnormally. Please contact us (support@codeship.com) with more information if you are seeing any odd behavior, and we will continue investigating.

This incident has been resolved.

Maintenance is complete and we are monitoring the system.

CodeShip Pro is under maintenance for about 60 minutes. During this time builds will be disrupted.

This incident has been resolved.

Builds are processing again and we continue to monitor.

We are currently investigating this issue.

This incident has been resolved.

We've put in place a workaround for the routing issue and are monitoring the behaviour of the system.

The issue has been isolated to a routing issue in our log system. Engineers are working on resolving the issue.

We are investigating an issue with CodeShip Basic builds not completing execution properly.

This incident has been resolved.

We are continuing to monitor build behaviour vs. AWS status updates. * the error rate on AWS backend services is continuing to drop, but has not reached normal levels (typically 0 in some parts of the system) * new builds are completing in a timely manner, although stalled builds may exist on your account * if your build has stalled, please cancel (and restart it if you need that build to complete)

AWS is now declaring their primary outage of Kinesis to be over - however they will take time to catch up. We use Kinesis for high availability (ordinarily!) of our build log systems. We are also dependent on CloudWatch to detect some service issues (and AWS has not declared that outage to be over yet). Builds are processing, but we are seeing long delays in pushing logs through the system - sometimes causing timeouts etc. We expect the behavior to improve over the next few hours and we will continue to monitor. Thank you for your patience.

There is an ongoing outage with multiple AWS services - please see https://status.aws.amazon.com. AWS is reporting that service restoration is underway and services are returning to normal operation (although throttled). This concurs with our internal service monitoring showing services returning to normal operation. You may experience some delay in build notifications and log access while services continue to be restored by AWS.

There is an ongoing outage with multiple AWS services - please see https://status.aws.amazon.com. AWS is currently reporting it may be several more hours until those services are fully restored. These system outages are directly impacting CodeShip customer builds (execution / monitoring / scaling). We are seeing incremental improvements in build processing as AWS restores services (even though they are not always updating their status) - we will update once we know more.

We are closely monitoring the various service outages reported by AWS: https://status.aws.amazon.com. The ECS and Kinesis outages are directly impacting CodeShip customer build logs and preventing builds from running. AWS is currently reporting it may be several more hours until those services are fully restored.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

We are performing upgrades to our application servers (component updates). We do not expect any disruption of service, but builds may be temporarily delayed as we update the platform.

This incident has been resolved.

We want to provide additional clarity on what’s been happening since the initial announcement, and what we have learned. First, we have engaged in a thorough review of the incident and of the CodeShip application security overall, and we have identified a series of improvements that we have started to implement. We have also completed rotating every key and secret across all of our systems, and we have implemented a new, aggressive, regular rotation as a short term measure, while we wait to complete the implementation of larger engineering efforts. On the investigation side, we have heard from customers that they identified unauthorized access by third parties using credentials connected to CodeShip. While this is obviously difficult to verify, especially for systems built on cloud services with very wide IP ranges, it underscores our request that all CodeShip customers need to immediately reset any key, source code credential, deployment credential, or authorization token that was ever potentially used on CodeShip prior to June 2020 - even if the CodeShip account is now inactive. If you are still in the process of resetting your credentials, we would like to emphasize that resetting the SSH key on your CodeShip projects is a critical step in order to resume normal functionality. To reset your SSH keys, you will need to go to `Project Settings` > `General` and click the red button that says `Reset SSH key`. Thank you for your prompt action, and we will be in further communication as the situation evolves.

Dear CodeShip users, We are reaching out to inform you of additional information we have uncovered as a result of our continuing investigation of the recent GitHub breach. To provide maximum transparency, we are reporting on the results of our investigation, the impact on users, actions you must take to protect yourself/your organization, and actions we will take to strengthen our security processes going forward. On Wednesday, September 16, 2020, CloudBees was notified by GitHub of suspicious activities targeting CodeShip business accounts connected to GitHub via the CodeShip GitHub app and now deprecated CodeShip OAuth tokens. CloudBees immediately initiated an investigation conducted by our security and engineering teams, and on September 27, we identified additional evidence of malicious activity against a failover CodeShip database. On September 29, we uncovered evidence to indicate that a malicious actor had access to this failover instance during the period of June 2019 to June 2020. At this time and to the best of our knowledge, we have no evidence of malicious activity or attempts within CodeShip systems since June 2020. *What type of data was affected?* The impacted accounts are those of CodeShip users. No other products or accounts were affected and CodeShip is in no way integrated with other CloudBees products or systems. *For all CodeShip users:* CodeShip users hashed account passwords, one-time password (OTP) recovery codes, and the OTP secret keys used to seed two-factor authentication all may have been exposed. Business contact information for invoicing purposes such as company contact name, company name, VAT number, postal address, phone number also may have been exposed. No payment information, such as bank account numbers or credit card numbers, was exposed. No other CloudBees product other than CodeShip was impacted. Also, the logging system was not accessed for any customers. *For CodeShip Basic users:* Any information contained in CodeShip users’ pipelines may have been exposed. This includes scripts, environment variables, access tokens and other similar data. *For CodeShip Pro users:* AES encryption keys may have been exposed. *Steps you should take* Although at this time we have no evidence that the data potentially exfiltrated has been used, all CodeShip users may have been affected (including free, Basic and Pro accounts) and should take the following steps: - Immediately rotate any keys or other secrets for cloud providers, third-party tools, or anything else that you used in your CodeShip pipelines. - If using CodeShip Pro, rotate your AES key and re-encrypt your secrets. - Immediately identify any other sensitive information that is stored in your pipelines and replace them within your pipelines and on any external systems. - Determine whether any of your systems accessible from CodeShip have experienced unauthorized access, by contacting your provider or carefully review your access records. - Verify that the source code held in repositories that are linked to your CodeShip account have retained their full integrity. - Reset your CodeShip 2FA: https://documentation.codeship.com/general/about/2fa/ At this time and to the best of our knowledge, we have no evidence of malicious activity or attempts within CodeShip systems since June 2020. We are continuing to monitor the situation. *Steps we are taking* As soon as we were notified by GitHub on September 16, we proceeded to rotate all our applications' internal secrets and rebuilt all our AWS AMIs. We are continuing to scrutinize our AWS security logs to monitor for suspicious activity, such as outbound connections to known malicious IPs. To date, we have not found any such activity. We want you to be assured that we are taking steps to increase the security strength of the CodeShip product, including but not limited to: - Validation that our product threat modeling and large-scope security reviews are systematically implemented. - Validation that the application of production security standards to all operational processes and artifacts is systematically implemented. - Enhancement of strict restrictions on access to production data and strict segregation of sensitive data. - Improvement of existing SIRT processes to ensure faster and better forensic investigation. *Who to contact * We will update here with any new developments. If you still have questions, please contact security@codeship.com. Last but not least, I’d like to apologize for the impact this is having on you. In the decade that CloudBees has been operating SaaS applications, we have always taken full responsibility for our products and we do so today. Please be assured that we will do everything we can to prevent this from happening again. - Sacha Labourey, CEO, CloudBees

On Wednesday, September 16, 2020, CloudBees was notified by GitHub of suspicious activities targeting certain CodeShip accounts connected to GitHub via the CodeShip GitHub app and now deprecated CodeShip OAuth tokens. If your GitHub credentials are impacted, you already received or will shortly receive a notification from GitHub informing you of this incident. The activities point to tokens being used to access the “/user/repos” GitHub API endpoint (https://docs.github.com/en/free-pro-team@latest/rest/reference/repos#list-repositories-for-the-authenticated-user), which is used to list users’ GitHub repositories, including private repositories. It is possible your repositories were cloned, so please contact GitHub support as soon as possible. Because the suspicious activities involve user tokens, as a first step in response we revoked all GitHub related tokens and SSH keys to keep all accounts protected. You need to reauthenticate CodeShip with GitHub immediately to avoid a service impact. *Action Required* - If you use GitHub to sign in to CodeShip, sign out of all CodeShip sessions and sign back in. - If you have GitHub projects setup on CodeShip, first remove and reinstall the CodeShip GitHub app: https://documentation.codeship.com/general/projects/builds-are-not-triggered/#github - Next, generate a new SSH key for each CodeShip project. This will enable CodeShip to clone the repository for builds again: https://documentation.codeship.com/general/projects/project-ssh-key/ We are continuing to investigate the underlying issue and will update our blog to provide more information as soon as we better understand any additional implications and potential root causes. Thank you. The CloudBees Team

Database was restored at 2020-08-29 08:25Z and all services were operational shortly thereafter.

Efforts continue to restore the database service.

There has been an underlying hardware failure for our main database. Our database service provider is failing over to new hardware.

The issue has been identified and a fix is being implemented.

We are currently investigating an outage on app.codeship.com

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

AWS is stabilizing and we are seeing Pro builds starting again. Continuing to monitor.

AWS is reporting EC2 issues which is causing Pro builds to be delayed.

This incident has been resolved.

Builds are stabilizing, but you may need to stop/restart any stuck builds. We are continuing to monitor.

Some builds are hanging on Basic. You may need to stop and restart builds that are not displaying logs. A fix is being implemented.

All services are operational and stable. The root cause has been identified as a CloudFlare service interruption - https://www.cloudflarestatus.com/incidents/b888fyhbygb8

https://codeship.com is back up and we are monitoring for further issues.

The https://codeship.com marketing site is down and we are investigating. https://app.codeship.com is separate and running normally.

This incident has been resolved.

GitHub webhooks are returning to normal levels and we are continuing to monitor.

We are seeing GitHub webhook delays which causes delays with builds starting. Monitoring https://www.githubstatus.com/incidents/phnch1rww464 for additional updates.

This incident has been resolved.

Our log processing is caught up and you can resume using the "new build details page" again.

We have identified the root cause, and our log processing queue is now catching up.

There is an issue with displaying build logs using the new build details page. As a temporary workaround, you can disable the new build details page from your settings - (https://app.codeship.com/user/edit) Scroll down to "New build details page" and click the "Turn off" button.

This incident has been resolved.

Basic builds should be returning to normal. If you have any further stuck builds please contact support@codeship.com and we will continue to monitor.

Some Codeship Basic builds are hanging due to the loss of some virtual machines. Stopping the build should allow other builds to continue.

https://codeship.com is restored.

https://codeship.com is now forwarding to a temporary location.

The https://codeship.com marketing site is down and we are investigating. https://app.codeship.com is separate and running normally.

This incident has been resolved.

Backlog of builds is being processed and we continue to monitor.

Builds are delayed and we are investigating.

This incident has been resolved.

Heroku is reporting that routing has stabilized. Continuing to monitor.

We are seeing another incident from Heroku that is impacting various CodeShip services. https://status.heroku.com/incidents/1974

This incident has been resolved.

CodeShip is back up and we are monitoring Heroku at https://status.heroku.com/incidents/1973

The CodeShip app is unreachable and we are investigating.

This incident has been resolved.

Customers should be able to log in again.

We are continuing to investigate this issue.

When attempting to log in to CodeShip customers are encountering an error page. We are investigating the issue and will provide an update when available.

This incident has been resolved.

A fix has been implemented and we are now catching up loading log messages into the storage system for the Beta UI.

We have identified the root cause and are implementing a fix for the errant behaviour.

If you're having issues displaying build logs, please revert to the traditional UI under your Personal Settings until this issue has been resolved.

Build execution has returned to normal

AWS has resolved the issue with spot instances and we are now monitoring the system as it returns to normal processing. Some builds may have been interrupted if they were impacted by the spot instance issues. We do not expect any further interruptions.

AWS is reporting issues with launching spot instances in EC2 that is impeding our ability to launch additional instances to run builds. AWS has identified the issue and is working on fixing. Builds may be delayed while AWS rectifies this issue.

This incident has been resolved, throughout our monitoring there have been no other failures due to this discovered.

Fix has been implemented, monitoring the results to ensure the incident is resolved.

CodeShip Basic builds are hanging on Preparing Bionic Build Container step, the issue has been identified and is currently being addressed.

This issue has been officially resolved.

Basic builds should be starting to run correctly again and we are monitoring.

Our team is still working to resolve this.

We are continuing to investigate this issue.

We are looking into this now.

This incident has been resolved.

Docker is reporting the registry issues are resolved and we are monitoring builds. https://status.docker.com/pages/533c6539221ae15e3f000031

Docker is reporting registry issues and this is causing Pro build failures. https://status.docker.com/pages/533c6539221ae15e3f000031

This incident has been resolved.

We're now monitoring the situation to ensure there are no further issues.

We are currently investigating this issue.

This incident has been resolved.

codeship.com is responding again.

The codeship.com marketing page is experiencing issues, but can be reached at http://cms.codeship.com.

This morning we saw performance problems with basic builds that slowed them down. We've resolved the issue now.

Outage occurred from ~ 10:10 AM UTC to ~ 10:30 AM UTC and was a result of 3rd party maintenance experiencing issues.

This incident has been resolved.

Our sites are back up.

Cloudflare is experiencing an outage and are working to resolve it: https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

On June 11th 2019, a responsible disclosure from an external security researcher made us aware of a vulnerability allowing access to an administrative interface showing internal queues where CodeShip Basic builds store and transmit messages between systems. Upon disclosure, we remedied the vulnerability within minutes to ensure that sensitive information waiting in those queues are not accessible without proper authentication and authorization. As a follow up, we did a thorough investigation on page access and were able to conclude that no unauthorized access to that page occurred outside of the security research. If you are a CodeShip Basic customer and you are concerned about this disclosure, please rotate your secrets, but we can confirm that no unauthorized access has occurred. (Rotating secrets regularly is good practice in general, and we advise our customers to do so regardless.) We're sorry for the issue - we take security very seriously and our team has been working as hard as possible to resolve the issue and put steps in place to prevent any possible recurrence in the future.

Bitbucket projects should be working again and we will prioritize additional updates to the integration.

New Bitbucket project creation is restored and we are continuing to work on remaining authentication issues today.

Working on updates to the Bitbucket integration.

The Bitbucket integration is not functioning and we are investigating.

This incident has been resolved.

Operational concerns with Bitbucket integration have been addressed and solutions are now live.

Bitbucket project updates are starting to run again for a limited number of projects.

The integration with Bitbucket is not functioning and we are working on a fix.

Queues have returned to normal.

We've addressed the problematic queues, and builds are now being processed normally.

We've identified the issue and are working to get customer queues flowing again. It only impacts a small number of customers.

We're currently investigating issues with stuck Basic builds that are blocking customer queues.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue. (app.codeship.com is fully functional)

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Service is now fully restored.

We are restoring stability and monitoring the problem.

We are seeing intermittent availability issues with the CodeShip application. We are investigating now and will update soon with more information.