Historical record of incidents for CloudBees
Report: "CloudBees Feature Management analytics"
Last updateWe're experiencing an elevated level of API errors related to the timescale database outage. https://status.timescale.com/
Report: "Platform services down"
Last updateIncident has been migrated to cloudbeesstatus.io status site - https://www.cloudbeesstatus.io/
We are currently investigating an issue in delivering services on the CloudBees platform
Report: "Issues with authentication"
Last updateWe have verified our fix is working.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We have identified the root cause of the issue and working towards a fix. Logins are currently working.
We are receiving errors affecting logins. We are currently investigating with our provider.
Report: "Can't log in to the CloudBees Feature Management web interface."
Last updateA configuration setting caused specific traffic to be rejected by our firewall, causing issues with authentication. The problem has been corrected and the incident is now resolved.
Logins to the web interface are working again. We are continuing to investigate.
We are aware that there are issues logging in to the CloudBees Feature Management web interface. We are investigating.
Report: "Authentication latency and timeouts"
Last updateThis incident has been resolved.
Our provider has implemented a fix and we are monitoring.
We are seeing extreme latency and timeouts on our authentication service that is causing login failures. We are currently investigating this with our provider.
Report: "Problem updating configurations"
Last updateEarlier today, for a few hours, we experienced an infrastructure problem which effected flag updates This problem is now fixed, and all flags should be updated with the latest configurations
Report: "Unable to login to CloudBees Feature Management"
Last updateThis incident has been resolved.
We have found the cause of the connectivity problems and applied a fix. The web interface and API are fully functional again and we will continue to monitor.
We have identified some connectivity (DNS) errors in our infrastructure and are working on a fix for this.
On further investigation, some API requests are failing. We are continuing to investigate the cause and identify a solution!
The CloudBees Feature Management API is operating normally. Whilst the web interface is unavailable, you can continue to make updates to Flags using the API: https://docs.cloudbees.com/docs/cloudbees-feature-management-rest-api/latest/introduction
We are aware of a problem logging into CloudBees Feature Management and are investigating the cause. Flag use by SDK and delivery of your targeting is unaffected.
Report: "CloudBees Feature Management impression recording suspended"
Last updateThis incident has been resolved.
We have identified the source of the problems that required taking analytics offline temporarily. Analytics functionality is now restored.
Following the earlier incident with the CloudBees Feature Management web interface, the impression analytics capability is temporarily offline whilst we investigate infrastructure problems.
Report: "Failures on User Login"
Last updateAll systems healthy again, and this incident is now resolved.
We have identified the source of the issue and resolved it. User login is back to stable behavior, and currently monitoring before closing the incident.
We are continuing to investigate this issue.
We have noticed sporadic failures in user login to our website app.cloudbees.com and Zendesk. Other apps such as app.rollout.io do not seem affected, but we're determining the extent of this issue.
Report: "CloudBees Feature Management"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
CloudBees Feature Management functionality is degraded due to an AWS outage. Affected are: - Delivery of flags configuration updates - GitHub Configuration-as-Code - Subscription management We, and Amazon are investigating.
Report: "Email-verification page having issues with re-send email button"
Last updateFix for the root cause has been rolled out to production. Email verification is working for all users again.
We've found out that re-sending the email via the button in email-verify page is continuously throwing an error, impacting new users signing up who click it. We've identified the root cause and rolling out a fix.
Report: "Cloudflare Service Issues"
Last updateThis incident has been resolved.
A fix has been implemented and CloudFlare is monitoring the results.
The issue has been identified and a fix is being implemented.
Cloudflare is investigating wide-spread issues with their services and/or network. Users may experience errors or timeouts reaching Cloudflare’s network or services.
Report: "CloudBees Feature Management Dashboard is not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Flag impression metrics/analytics may stop counting for a brief time period"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are in the process of running a short production infrastructure test that may result in flag impression metrics/analytics not updating for a short time period. We do not expect this to last for more than 15 to 30 minutes. During this time all other services will function as normal. We apologize in advance for any inconvenience caused.
Report: "Feature flag configurations not propagating to SDK clients"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
New feature flag configuration changes are not being propagated to SDK clients.
Report: "Advisor GCP Migration"
Last updateThis incident has been resolved.
Advisor is under maintenance.
Report: "DevOptics intermittent issues"
Last updateThis incident has been resolved.
We performed percussive maintenance on some of our infrastructure, and are now monitoring the system.
We are investigating intermittent issues with the web interface for the DevOptics platform.
Report: "Major provider outage"
Last updateThis incident has been resolved.
Now monitoring major providers status
Issue caused by major providers outage
We are continuing to investigate this issue.
There has been a provider outage and we're investigating right now. Currently our authentication service is down, affecting the web interface for most of our products.
Report: "Feature Management partially down"
Last updateThis incident has been resolved.
Some Feature Management services are beginning to return to normal. We are continuing to work on this.
Some Feature Management services are beginning to return to normal. We are continuing to work on this.
Some Feature Management services are beginning to return to normal. Working to restore the rest.
Report: "CloudBees CI Update Center timing out"
Last updateA fix for the underlying issue has been released and the system is now performing normally.
The root cause has been identified - a fix is being developed.
The CloudBees CI update center (jenkins-updates.cloudbees.com) is timing out due to increased load. Engineers are working to resolve the issue.
Report: "Auth0 outage"
Last updateThis incident has been resolved.
EU is recovering and error rates are less than 1%. US-1 is still experiencing partial outage. Our team is working on mitigation actions and estimated time for resolution. We apologize for the impact this is having on you and your users. Next update will be in 15 minutes
We are continuing to investigate this issue.
Investigating elevated login error rate
Report: "SS0 Provider outage"
Last updateOur SSO provider identified issues and took action to mitigate them.
Report: "www.cloudbees.com to GCP"
Last updateThis incident has been resolved.
www.cloudbees.com is under maintenance.
Report: "Feature flags are not rendered properly in the UI"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Feature Management web interface offline"
Last updateWe have resolved the underlying issue with Feature Management and restored all services.
Engineering has implemented a fix for the Feature Management control interface (app.rollout.io) Impression analytics is offline and we continue to troubleshoot this service.
Engineering has identified the root cause and is continuing to work on resolving the issue.
We have identified an issue with the Feature Management web interface and some associated services. Feature Flag delivery (to consumers) is fully operational. Engineers are working to correct the issue.
Report: "CloudBees Update Center and ancillary services offline"
Last updateAll services have been restored.
We are tracking an availability incident with our CloudBees Update Center and several ancillary services. Engineers are restoring service now.
Report: "Auth0 Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
You can now login to Cloudbees services including Cloudbees Feature management, Software Delivery Management, Devoptics and Support
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We are continuing to monitor the issue and will provides updates as soon as possible.
Please email support@cloudbees.com for any CloudBees-related support. Related Auth0 status: https://status.auth0.com/incidents/zvjzyc7912g5
CloudBees services currently impacted by Auth0 service outage include CloudBees Feature Management, CloudBees DevOptics, CloudBees Software Delivery Management, and the CloudBees Support Portal.
We are continuing to monitor for any further issues.
One of our third-party providers Auth0 for authentication is experiencing a major incident, we will monitor the outage and provide updates as services come back online
Report: "Investigating an issue with DevOptics entitlement synchronization"
Last updateAll entitlement / subscriptions are restored.
We are currently investigating this issue.
Report: "Authentication issue to https://app.codeship.com/"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "docs.cloudbees.com maintenance"
Last updateThe update is complete
We are performing system upgrades and maintenance on docs.cloudbees.com You may experience minor rendering issues while this upgrade is completed.
Report: "Feature Flags dashboard login not working as expected"
Last updateRoot cause identified - there was an issue of case sensitive emails, only one customer impacted and simple workaround was identified and verified to be working (use lower case letters in sign-in) . We are working to release a bug fix to solve the root issue.
some users reports issues with login in to the Feature Flags dashboard, we are investigating
Report: "Rollout - Degraded SAML service"
Last updateWe are closing this incident as the nature of the limitation has been identified. We will post an updated note when the system limitation is resolved (this is a high priority issue with multiple engineers working to resolve the issue). Until that time, please deactivate SAML restricted mode and create a new user on your account to maintain access. If you have any questions or concerns - please contact us via https://support.cloudbees.com
Existing SAML users can login, however re-linking your SAML user and new SAML user invites are currently not working.
Report: "Rollout incident"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
we’re currently investigating if any further features are effected
Report: "Maintenance occurring on GrandCentral / Login system DNS"
Last updateAll services have been restored.
Work is continuing - core login services have been restored - however entitlements are temporarily offline.
Hosting for GrandCentral is being migrated to a new provider. There will be a short interruption to service while DNS propagates and new certificates are issued (hence the "insecure site" warning).
Report: "Multiple systems not responding properly"
Last updateAll services are operational and stable. The root cause has been identified as a CloudFlare service interruption - https://www.cloudflarestatus.com/incidents/b888fyhbygb8 We have updated our status page note our support email address in the event our ZenDesk hosted site is not available.
Our monitoring systems show that all impacted services have recovered. We will continue to monitor this situation.
We are investigating an issue on several CloudBees systems that seems related to a ZenDesk / CloudFlare issue that is not being reported by them yet. This is a developing situation.
Report: "CloudBees Rollout service incident"
Last updateAll CloudBees Rollout (Feature Flags) services are now fully operational. We have not identified any data-loss or security impact from this outage. An outage post-mortem and corrective actions will be performed in due course. Thank you for your patience.
The Rollout core service (API/login/web) outage has been resolved and these services are now fully operational. However, Impression Analytics are not currently available, the engineering team are working to resolve this issue. We will continue to provide service updates on the status of Impression Analytics until the issue is resolved.
Apart from Impression analytics - which is currently not working - the service is back to operational. We're monitoring the situation.
Our Engineering Team has been successfully restored the database but still not 100% operational. More updates to follow.
IBM Compose update - "Virtual networking is up across all hosts in the cluster and the situation appears to be stable. We are slowly starting data/member capsules. Once those are up, we will start portals which will restore customer access" In parallel - CloudBees engineering teams are now working to restore the database service to our own infrastructure - with a view to failing over if Compose is not able to restore access in a timely manner.
We've recieved this message from our service provider: “At this point we are cautiously optimistic. Our engineers are close to having virtual networking up across all hosts in the cluster. So far so good. Once stable we will start bringing capsules back up.” More updates to follow.
Monitoring https://status.compose.com/ for further updates
Our service provider announced is going to take longer to recover the system. The current rough estimate for recovery is from 5 to 9 hours.
Our service provider has updated us about the situation and the rough estimate for recovery is 2 to 3 hours.
We have confirmed the database issue with our service provider and they are working to restore service. Service impact is that experiments can’t be updated, but existing flags are unaffected.
There is an outage with a 3rd party service. We're contacting them to see what's the situation.
Report: "SDM service degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "www.cloudbees.com - service disruption"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "jenkins-updates.cloudbees.com having issues"
Last updateThe issue has been identified and fixed. The service is now fully operational
Currently investigating https://jenkins-updates.cloudbees.com returning 401s when downloading plugins.
Report: "www.cloudbees.com is unavailable."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "docs.cloudbees.com is experiencing issues, will be back shortly"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Sendgrid email delivery having issues"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented
Report: "Auth0 Authentication Experiencing Degraded Performance"
Last updateThis incident has been resolved.
Auth0 has implemented some infrastructure changes on their side and performance seems to be improving.
We're suffering degraded Authentication service and are working with our provider to resolve.
Report: "ask.cloudbees.com scheduled maintenance"
Last updateMaintenance is now complete.
https://ask.cloudbees.com/ is currently undergoing scheduled maintenance upgrades We will update this status when maintenance is complete.
Report: "Rollout API"
Last updatePush configuration service has been restored and enabled for all customers
Engineering has implemented a fix and are monitoring while we progressively roll out push configurations to users.
Engineers are working on a solution but it will take some time to resolve and test. Push configuration will remain disabled for now. Please reach out to support with any concerns.
Engineers are investigating problems with pushing configurations to SDKs. We've disabled push configuration for now.
Report: "DevOptics UI not showing anything after user clicks on any gate in a ValueStream"
Last updateOn Tuesday 10/29/2019, users who are on Chrome version higher than 77.0.3865.120 and FireFox 69.0, 69.0.3, and 70.0 may have noticed that the ValueStream page turns blank when clicking on any gate. This issue prevents users from viewing the ticket and commit information at the gate level only. Display of the ValueStream graph and metrics was not impacted. We have resolved the issue and are currently monitoring.
Report: "Problem with Support Site Sign In - Workaround Available"
Last updateThe "Sign Up" button on support.cloudbees.com now works as expected.
If you are having trouble logging into the support site, please try going to support.cloudbees.com and clicking on 'My Activities'. This will get you logged in correctly. You can also email support@cloudbees.com or click on links from existing cases. We are working to fix the issues related to the 'Sign In' button
If you are having trouble logging into the support site, please try going to support.cloudbees.com and clicking on 'My Activities'. This will get you logged in correctly. You can also email support@cloudbees.com or click on links from existing cases. We are working to fix the issues related to the 'Sign In' button
Report: "Devoptics Scheduled Maintenance"
Last updateThis incident has been resolved.
DevOptics is undergoing planned maintenance and is unavailable at this time. Normal service will resume within a few minutes. Sorry for any inconvenience.
Report: "Slow processing times"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Heavy load of underlying systems is causing delays in rendering web UI portions of DevOptics data. Engineering is aware and actively looking into the issue.
Report: "Degraded Performance on DevOptics Run Insights"
Last updateThis incident has been resolved. There are cases of duplicated data ingested between 12:00 and 14:00 UTC 2019-05-21. All duplicate data has been removed.
We are currently investigating performance issues with DevOptics Run Insights.