Historical record of incidents for Platform9
Report: "Management Plane UI is unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Management Plane is unreachable, our team is actively working on identifying and fixing the issue.
Report: "Intermittent Management Plane Instability"
Last updateThis incident has been resolved.
A fix has been implemented and we are continuing to monitor the environments
We're currently investigating an issue related to intermittent management plane instability which may manifest as being unable to login via the UI, API failures, and/or nodes being disconnected.
Report: "Upstream Issue: Red Hat Quay Image Registry Down"
Last updateRed Hat has marked the incident as resolved.
Container image pulls may fail for images hosted by Red Hat Quay image registry due to an ongoing incident. For more information, see: https://status.redhat.com/incidents/qh68rjfg6xs6 Nodes where the images are already cached should continue to function for newly-created containers.
Report: "[Intermittent] Degraded Access to Management Planes"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've identified a related bottleneck where we're observing network congestion on the backend and we've formulated a plan to attempt to mitigate it. The management plane may still be unstable at this time until we've taken these full mitigation steps.
We are continuing to investigate this issue.
We are currently investigating an issue impacting multiple customers' ability to access their management plane instances. Attempts will sometimes result in 502 errors. We will continue to update this post as more details become available.
Report: "DNS Resolution Issues"
Last updateThe incident may be considered resolved. No further interruptions have been observed nor reported.
Our Operations team has taken steps to resolve the issue. No further interruptions are anticipated at this time. Our team will continue to monitor the affected environments.
We've identified the issue with the automated DNS component which was causing issues properly synchronizing CNAME records for our Platform9 management plane instances. Our Operations team is taking steps to restore.
We're currently investigating reports of issues related to DNS resolution, especially with regard to the Platform9 management plane endpoint. Hosts may intermittently or temporarily show as disconnected. Cluster workloads should remain unaffected.
Report: "Majority of Free Tier users are unable to access their system"
Last updateThis incident has been resolved.
A fix has been implemented and environments should start recovering soon
We are currently investigating the issue
Report: "Some management planes are intermittently unreachable"
Last updateThis incident has been resolved.
We are investigating issues with our infrastructure provider.
Report: "(Retroactive) Connectivity issue reaching management planes"
Last updateThere were intermittent issues reaching management planes between 2021-12-15 15:22 - 2021-12-15 16:02 UTC. This resulted from an Internet connectivity outage at our hosting provider (Amazon Web Services). Connectivity has now been restored.
Report: "Platform9 management planes are unreachable"
Last updateThe network issue has been resolved.
The issue has been identifed to be a network issue with our Data Center service provider. They are working to resolve it.
Certain customers cannot reach their management plane deployments. We are currently investigating the issue.
Report: "Platform9 management planes are unreachable"
Last updateNo issues reported after monitoring the fix for 5 hours. With customers able to access their management plane, the issue stands resolved.
The situation is still being monitored. So far, no issues have been reported.
Our vendor has identified and implemented a fix. Affected customers are now able to access their management plane.
Certain customers cannot reach their management plane deployments. It has been identifed to be a network issue with our Data Center service provider. They are working to resolve it
Report: "Platform9 management planes are unreachable"
Last updateAll Customer Management Planes are Available.
A fix has been implemented and we are monitoring the results.
Connectivity has been restored
We have detected that the majority of our customers cannot reach their management plane deployments. We are investigating the issue.
Report: "Management plane is unreachable for some customers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
Management plane is unreachable for some customers. We have identified the issue and are implementing fix.
Report: "Elevated error rates on some customer Management Planes"
Last updateThis incident has been resolved.
Customer Management Planes should be available again. We will continue to monitor the situation
We are currently investigating the issue
Report: "Elevated error rates on some customer management planes"
Last updateThis incident has been resolved.
A fiber optic issue was identified at our upstream datacenter provider and has been fixed. We are continuing to monitor the situation.
An issue was resolved by network provider. We are monitoring for issues
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are seeing increased error rates for certain customers and are currently investigating to identify the root cause.
Report: "A small number of customers are experiencing connectivity issues to the management plane"
Last updateConnectivity to management planes has been restored
We have identified a hardware failure that is impacting a small number of customers management planes. We are replacing the hardware and restoring access shortly.
Report: "Management plane is unreachable for some customers"
Last updateThis incident has been resolved.
The issue should be resolved and we are monitoring the health of the environment
The storage outage is resolved and customer management planes will be recovering shortly
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Elevated error rates on some customer management planes"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have identified the root cause and addressed the issue. We are monitoring the situation. We apologize for the inconvenience
We are currently investigating this issue.
Report: "Multiple customer regions inaccessible"
Last updateThe incident is resolved. If you are still experiencing any problems, please submit a ticket at pf9.io/support
We are continuing to monitor for any further issues.
We have issued a fix to restore connectivity to customer regions. We are monitoring the systems now.
We are currently investigating the issue and will share further updates as time progresses.
Report: "Multiple Customer Regions Inacessible"
Last updateThis incident has been resolved.
All systems should be restored and we are now monitoring for any lingering issues.
We have identified the storage issue that caused this outage and are now recovering systems.
We have identified a fix and are working on recovering customer control planes.
We are currently investigating network connectivity loss to some customer management planes.
Report: "An infrastructure related issue occurred that affected some customers environments."
Last updateAll issues have been resolved and an root cause has been found.
An infrastructure related issue occurred that affected some customers environments. All systems should be up and running now. We are monitoring the situation and working on an RCA.
Report: "Platform9 Software Upgrade"
Last updateThis incident has been resolved.
The previous notification regarding the software upgrade is a false alarm. Apologies for the false alarm. The cloud is alive and healthy. If you have any questions, please feel free to submit a ticket at support.pf9.io or contact support@platform9.net
Report: "Some customers are experiencing connectivity loss to the control plane"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A resolution is in place.
We have identified the underlying issue. A hardware failure caused the initial outage and the automated failover system was not able to restore network connectivity. We are working on restoring connectivity as soon as possible.
We are investigating the cause and will update this page shortly
Report: "Connectivity issues reaching the Platform9 control plane"
Last updateAll systems are back online. We will provide a root cause as soon as we have it.
We are continuing to monitor for any further issues.
We are still seeing some systems being impacted by the outage and have not recovered. We are working on recovering these environments.
We are continuing to monitor for any further issues.
Most customer environments should be back online now. We apologize again for the inconvenience
We have replaced a network device and are working to recover customer environments.
We are continuing to investigate the network outage. We appreciate your patience and apologize for the inconvenience.
We are investigating network issues impacting access to the control plane for several customers.
Report: "Some customers may be experiencing issues connecting to their control plane"
Last updateThis incident has been resolved.
We are investigating the problem and will provide an update shortly.
We have identified the problematic service and have restored the affected environments
Report: "Intermittent connectivity issues to the management plane"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating intermittent issues with API calls to the Platform9 management plane
Report: "Intermittent connectivity issues"
Last updateThis incident has been resolved.
We have identified and resolved a networking issue.
Some customers are experiencing intermittent connectivity issues. We are investigating the issue.
Report: "Issues loading data in the control plane"
Last updateConnectivity was restored at 14:40 PST. Systems continue to be operational.
The outage resulted from a brief loss of upstream connectivity to our Database systems. The connectivity has been restored
We're investigating issues where customers are not able to see data in the Platform9 UI
Report: "Intermittent connectivity issues to the Platform9 management plane"
Last updateThis incident has been resolved.
We have replaced the affected device and are monitoring the situation. The service should be operational now.
Some customers may be experiencing intermittent issues reaching the management plane. We are investigating the cause.
Report: "Some customers are experiencing issues connecting to their management plane"
Last updateThis incident has been resolved.
We have fixed the network connectivity issues to the backend storage
The outage is related to storage connectivity for some customer environments. We are continuing to investigate. We apologize for the inconvenience.
We are currently investigating network connectivity loss to some customer management planes.
Report: "Some customers are experiencing issues connecting to their management plane"
Last updateThis incident has been resolved.
The issues have been resolved, we are monitoring for stability.
The issue has been identified and we are working on a fix.
We are continuing to investigate the issue regarding network connectivity to customer management planes.
We are currently investigating network connectivity loss to some customer management planes.
Report: "Workloads inaccessible"
Last updateThis issue has been resolved. If you are still experiencing connectivity access to your Platform9 control plane, contact Platform9 support at support.pf9.io
This issue has been resolved. If you are still experiencing connectivity access to your Platform9 control plane, contact Platform9 support at support.pf9.io
We have issued a fix for the issue at hand and are monitoring all workloads impacted. If you are still experiencing errors accessing your workloads, contact Platform9 Support at support.pf9.io
We have identified the issue preventing workloads from being inacessible and are working on a fix.
Report: "Systems degraded"
Last updateAll impacted systems are reporting as healthy. If you are experiencing any issues with your Platform9 Control plane, please contact support at support.pf9.io
The underlying issue has resolved, and we are currently monitoring all impacted control planes.
We are currently investigating an issue in the Platform9 Cloud that is impacting customer control planes.
Report: "Management planes not reachable"
Last updateThis incident has been resolved.
We have restored connectivity. Please let us know if you continue to experience issues.
We have identified and resolved an issue with our storage system. We will begin restoring systems as soon as possible.
We are investigating a loss of connectivity to Platform9 management planes
Report: "Management plane API/UI slow or unresponsive"
Last updateThis incident has been resolved.
There was a temporary connectivity issue with our datacenter provider.
We are investigating an issue with network connectivity to our databases.