Historical record of incidents for Red Hat
Report: "Errors when accessing console.redhat.com"
Last updateWhen accessing console.redhat.com, users may experience Sentry errors.
Report: "Spurious incident test messages"
Last updatePlease disregard the previous incident messages related to API OpenShift and OSIS/AIS. These messages were unintentionally sent during testing of new tooling. The issue has been resolved, and no further action is required.
Report: "API OpenShift Incident 06/11/2025"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Thank you Team Aurora for tackling this - only a test! :)
Report: "API OpenShift Incident 06/11/2025"
Last updateThank you Team Aurora for tackling this - only a test! :)
Report: "OSIS/AIS Incident 06/11/2025"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Thank you team Aurora for looking into this - only a test! :)
Report: "OSIS/AIS Incident 06/11/2025"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Team Aurora, thanks for looking into this! Only a test :)
Report: "OSIS/AIS Incident 06/11/2025"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Team Aurora, please look into this - and of course, this is only a test! Thank you!
Report: "DOWN - DRHC-001 - Homepage (Origin)"
Last updateWe are currently investigating this issue
Report: "console.redhat.com - Unable to login"
Last updateOn June 10th, between 10:06UTC and 10:16UTC, then 14:14UTC to 14:27UTC and 15:16UTC to 15:42UTC, users of console.redhat.com were unable to login. Incidents were related to an API rate limitation. Actions were taken to adjust the rate-limiting feature and no issue has been detected since then.
Report: "Hybrid Cloud Console chrome service database maintenance 2025-06-04"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The Red Hat Hybrid Cloud Console will undergo a DB upgrade forchrome service starting on 2025-06-04 at 5AM UTC (1 AM ET). The updates areexpected to last approximately 2 hours until 7AM UTC (3 AM ET).During this maintenance window, the console UI may briefly be unavailable.
Report: "Ansible Lightspeed Degraded Performance"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Ansible Lightspeed Degraded Performance"
Last updateWe are currently investigating this issue.
Report: "Hybrid Cloud Console quickstarts database maintenance 2025-06-02"
Last updateThe Red Hat Hybrid Cloud Console will undergo routine maintenance forthe quickstarts (learning-resources) database starting on 2025-06-02 at 10AM UTC (6 AM ET).The updates are expected to last approximately 3 hours until 1PM UTC (9 AM ET).During this maintenance window, the console should be fully operational and no disruptions should be expected.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Developers.redhat.com - Homepage Outage"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Developers.redhat.com - Homepage Outage"
Last updateWe are currently investigating this issue.
Report: "access.redhat.com - Support Cases and Ask Outage"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "access.redhat.com - Support Cases and Ask Outage"
Last updateWe are currently investigating this issue.
Report: "access.redhat.com - Support Cases Down"
Last updateWe are currently investigating this issue.
Report: "500 Internal Server Error When Reading Signatures"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "500 Internal Server Error When Reading Signatures"
Last updateWe are currently investigating this issue.
Report: "Partner Portal Outage"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
Report: "Partner Portal Outage"
Last updateWe are currently investigating this issue.
The issue has been identified and a fix is being implemented.
Report: "Quay.io Push Unavailable"
Last updateA fix has been deployed and Pushes are restored.
Quay.io has been moved to read-only mode while we work to implement a fix. During this time Pulls will continue to work, however, Pushes will be disabled until a fix is implemented. There is currently no timeline for Push restoration.
The issue has been identified. A primary key in the database has reached maximum value. We are currently working on validating a fix.
We are currently investigating this issue.
Report: "Quay.io Push Unavailable"
Last updateWe are currently investigating this issue.
Report: "Access.redhat.com & Catalog.redhat.com Partial outage"
Last updateWe identified and resolved a issue relating to a release of a new application. The change impacted some paths on access.redhat.com and the sigstore for container pulls on catalog.redhat.com. A fix was pushed at May 8, 2025, 05:05:05 AM UTC
Report: "Access.redhat.com & Catalog.redhat.com Partial outage"
Last updateWe identified and resolved a issue relating to a release of a new application. The change impacted some paths on access.redhat.com and the sigstore for container pulls on catalog.redhat.com. A fix was pushed at May 8, 2025, 05:05:05 AM UTC
Report: "connect.redhat.com container factory component outage"
Last updateThis incident has been resolved.
connect.redhat.com container factory component outage
Report: "connect.redhat.com container factory component outage"
Last updateconnect.redhat.com container factory component outage
Report: "Quay.io Intermittent 502s"
Last updateQuay.io saw a brief disruption in Pushes and Pulls stemming from an increase in traffic. We have taken action and error rates have decreased.
Report: "access.redhat.com outage"
Last updateaccess.redhat.com was intermittently unavailable
Report: "access.redhat.com outage"
Last updateaccess.redhat.com was intermittently unavailable
Report: "Connect.redhat.com Outage"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
An outage has been detected on connect.redhat.com. It impacts account management, the container factory, and partner subscriptions on connect.redhat.com. We are currently working to restore the functionality.
Report: "Connect.redhat.com Outage"
Last updateAn outage has been detected on connect.redhat.com. It impacts account management, the container factory, and partner subscriptions on connect.redhat.com. We are currently working to restore the functionality.
Report: "Quay.io Intermittent 502s"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Quay.io Intermittent 502s"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "MWNX-001 - repository.jboss.org"
Last updateRoot certificate was missing, the service is up with old certificate now.
The root certificate was missing
Team is currently investigating this issue.
Report: "MWNX-001 - repository.jboss.org"
Last updateTeam is currently investigating this issue.
Report: "Hybrid Cloud Console chrome service database maintenance 2025-03-31"
Last updateThe Red Hat Hybrid Cloud Console will undergo routine maintenance forchrome service starting on 2025-03-31 at 7AM UTC (3 AM ET). The updates areexpected to last approximately 6 hours until 1PM UTC (9 AM ET).During this maintenance window, the console should be fully operational and no disruptions should be expected.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Quay.io Intermittent 502s"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified an issue that is causing an elevated rate of 502s on pushes and pulls. We are currently working on a fix.
Report: "Quay.io Intermittent 502s"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified an issue that is causing an elevated rate of 502s on pushes and pulls. We are currently working on a fix.
Report: "Registry: Unrecognized signature format"
Last updateThis incident has been resolved.
We have identified an issue preventing some content from being pulled due to bad signatures. The issue was caused by a recent change, which we have now rolled back. However, some incorrect signatures remain and will require cleanup. Our team is working on resolving this, and we will provide updates as we make progress. Error message you might experience: unrecognized signature format, starting with binary 0x2d
Report: "Registry: Unrecognized signature format"
Last updateThis incident has been resolved.
We have identified an issue preventing some content from being pulled due to bad signatures. The issue was caused by a recent change, which we have now rolled back. However, some incorrect signatures remain and will require cleanup.Our team is working on resolving this, and we will provide updates as we make progress.Error message you might experience: unrecognized signature format, starting with binary 0x2d
Report: "Quay.io Intermittent 502s on Pulls"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Quay.io Intermittent 502s on Pulls"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Hybrid Cloud Console User Access (RBAC) database maintenance 2025-03-25"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The Red Hat Hybrid Cloud Console will undergo routine maintenance for User Access (RBAC) starting on March 25th, 2025 at 9 a.m. UTC (5 a.m. ET). The updates are expected to last approximately 5 hours until 2 p.m. UTC (10 a.m. ET).During this time, write actions to RBAC (creating or editing user groups, or custom roles) will be unavailable. We will implement Blue/Green Deployments to ensure near zero-downtime for read operations. All other services that depend on RBAC, as well as any services and tenants using the RBAC API or User Access in HCC, should not be impacted.
Report: "Hybrid Cloud Console Playbook-Dispatcher database maintenance 2025-03-24"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The Red Hat Hybrid Cloud Console will undergo routine maintenance forPlaybook-Dispatcher starting on 2025-03-24 at 9PM UTC (4 PM ET). The updates areexpected to last approximately 3 hours until 12PM UTC (7 PM ET).During this time, users will not be able to create or update playbook runs throughthe Playbook-Dispatcher service.
Report: "Customer Portal Downloads Outage"
Last updateThis incident has been resolved.
A dependent service is undergoing a major outage and the Customer Portal Downloads page is offline.
Report: "Customer Portal Downloads Outage"
Last updateThis incident has been resolved.
A dependent service is undergoing a major outage and the Customer Portal Downloads page is offline.
Report: "ROSA HCP clusters cannot be provisioned."
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "catalog.redhat.com - page content failing to load"
Last updateThis incident has been resolved.
We are currently investigating an issue where page content fails to load on catalog.redhat.com.
Report: "ROSA HCP clusters cannot be provisioned in the us-east-2 region"
Last updateProvisioning delays for HCP clusters in us-east-2 have been resolved.
Update to follow within 30 minutes.
Report: "Catalog.redhat.com - Container catalog"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "access.redhat.com - Support Cases Outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Monitor Portal Search"
Last updateWorking fine.
Nodes are restarted, up and running. Will monitor for some time.
There was an issue with solr GC(garbage collector).
Search is down, we are currently investigating issue.
Report: "Unclassified SBR alert"
Last updateThis incident has been resolved.
A restart has occurred, issue is believed to be fixed. Awaiting and monitoring to make sure everything is clear.
We have received SBRC-001 - Search - unclassified-sbr-alert, the concerned team is working on it. We have identified the RCA and have been monitoring it.
Report: "Intermittent Push & Pull Failures"
Last updateThis incident has been resolved.
We have implemented a change related to reducing load on our database and have seen a dramatic decrease in errors related to pushing and pulling. We will continue to monitor performance today.
We are currently investigating an issue of elevated 502s (Bad Gateway) on pushes & pulls
Report: "registry.connect.redhat.com and registry.access.redhat.com 503 Certificate Error"
Last updatePulls for registry.connect.redhat.com and registry.access.redhat.com generated 503s due to expired certificate. The certificates have been rotated and pulls have been restored.
Report: "Red Hat Phone Services - Outage"
Last updateThis incident has been resolved.
Red Hat is currently experiencing issues with phone service. There is a total outage of phone services in the United States and there are intermittent outages in APAC regions, This has affected all teams, including Customer Service and Support. Please use email or the Customer Portal Case Management system for communication. We are actively working to resolve this issue.
Red Hat is currently experiencing issues with phone service in the United States. This has affected all teams, including Customer Service and Support. Please use email or the Customer Portal Case Management system for communication. We are actively working to resolve this issue.
Report: "Intermittent Push & Pull Failures"
Last updateThis incident has been resolved.
We have reduced the amount of 502s overall and pulls and pushes should now be succeeding at a higher rate, however, 502 rates are still elevated. We will continue to work to reduce the number of 502 responses and monitor.
We have identified an issue with a node in the cluster. We have removed that node and are seeing a decrease in 502s and are transitioning to monitoring.
We have received reports of increased 502s when pulling and pushing images on quay.io. We are currently working on implementing a fix.
Report: "Intermittent Push & Pull Failures"
Last updateThis incident has been resolved.
We believe this issue is related to resource contention in the cluster. We've scaled up the amount of resources and are currently monitoring usage.
The issue has been identified and a fix is being implemented.
We have received reports of increased 502s when pulling images on quay.io. We are currently investigating.
Report: "catalog.redhat.com - Catalog App outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Customer Portal commenting Platform is down"
Last updateThe incident has been resolved.
The issue has been identified and the concerned team is working on it. https://access.redhat.com/api/v1/comments/trending-content
Report: "Developers.redhat.com - Downloads outage"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Labs facing subscription issue"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Pull Failures on registry.redhat.io"
Last updateOn 2024-12-13 an issue was identified that impacted OpenShift clusters configured with proxy settings. During this time there were failures in pulling container images due to a "Forbidden" error. As a part of a backend transition, the image signature verification process was misconfigured to redirect to catalog.redhat.com. This domain is not currently documented as a requirement in the OpenShift firewall configuration guide and as a result proxy configurations that do not explicitly allow access to catalog.redhat.com blocked the request causing image pulls to fail. This configuration has been corrected and pulls to registry.redhat.io have been restored.
Report: "Quay.io Builds Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue where builds are stuck waiting for available build worker.
Report: "DOWN - CGW-001 - Download Manager Ping and DOWN - CGW-001 - Download Manager Pageload"
Last updateTHis incident has been resolved and was caused due to https://redhat.service-now.com/surl.do?n=INC3272853
We are continuing to investigate this issue.
Downloads are not working on Developers, so it's fairly major outage.
Report: "DOWN - CPLA-001 - Labs app - satelliteupgradehelper"
Last updateThis has been resolved now and the probable reason for it might be due to https://redhat.service-now.com/surl.do?n=INC3272853. Marking this incident as resolved
The subscription API is down, we are working on a patch for temporary solution
Report: "subscription.rhsm.redhat.com elevated error rate"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "access.redhat.com - Support Cases Outage"
Last updateThis incident has been resolved. Salesforce has confirmed that our instance has been updated with the fix and is fully operational. Most of the functionalities appears to be functioning smoothly.
A fix has been implemented and we are monitoring the results.
The team has rolled back all production instances and is conducting final checks.
The rollback is confirmed to successfully resolve the issue and the team will continue to implement the rollback across the remaining instances. The team has developed a script to perform this rollback on the impacted instances in groups to speed up the resolution.
Salesforce is seeing positive results following the completion of the rollback on a number of instances. The rollback is being done in a staggered manner due to the scale of the issue. As rollbacks complete, affected customers should begin to see improved performance. The rollback efforts are expected to take several hours. To create support cases during the outage, please use techsupport@redhat.com
The outage was caused by a Salesforce issue, where some changes led to experiencing high levels of Network traffic, which affected database stability. Now, the Salesforce team successfully rolled back the changes and the outage is resolved now and it is operational.
We are currently investigating this issue.
Report: "Partial Outage for Openshift Cluster Manager"
Last updateThis incident has been resolved.
We have verified long-term fixes: OCPBUGS-43821 (4.14), OCPBUGS-43856 (4.13), and OCPBUGS-43872 (4.12). Until those are available for updates, SRE recommends customers enable the Backup and DR API (backupdr.googleapis.com) in your Google Cloud Platform account, if not already enabled, as a preliminary mitigation. Please see https://access.redhat.com/solutions/7092988 for details. We will update this incident once cluster updates are available for OpenShift Dedicated.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We've identified an error impacting cluster provisioning of OpenShift Dedicated clusters on GCP for versions 4.14 and below. While the issue is investigated, SRE recommends customers enable the Backup and DR API (backupdr.googleapis.com) in your Google Cloud Platform account, if not already enabled, as a preliminary mitigation. Please see https://access.redhat.com/solutions/7092988 for details.
Report: "access.redhat.com - Homepage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Outage for subscription.rhsm.redhat.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "catalog.redhat.com - Catalog App Outage"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Quay.io Builds Intermittent Failures"
Last updateThis incident has been resolved.
A fix has been implemented and we no longer observing increased failures.
We are continuing to work on a fix for this issue. Builds that have failed should be run again since the failures are intermittent.
The issue has been identified and a fix is being implemented.
We continue to investigate the root cause, but builds we can confirm builds are intermittently working.
We are currently investigating an issue where builds are showing an error related to "Preparing build node."
Report: "OpenShift Update Service (OSUS) Outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have implemented a fix and are seeing error rates decrease. We are continuing to monitor the situation.
We are continuing to investigate this issue.
We are currently investigating an issue with the OpenShift Update Service.
Report: "access.redhat.com - Support Cases Outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "access.redhat.com - Support Cases Outage"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Quay.io Builds Unavailable"
Last updateThe build system has been restored. We will be working to add additional alerting and monitoring in the coming days.
We are currently investigating an issue where builds are showing an error related to "Preparing build node."
Report: "Unclassified SBR alert"
Last updateThis incident has been resolved.
We have received SBRC-001 - Search - unclassified-sbr-alert, the concerned team is working on it. We have identified the RCA and have been monitoring it.