Historical record of incidents for Catalyst Cloud
Report: "CCKS Kubernetes Cluster: creation and upgrades"
Last updateWe have identified an issue with initial creation of Kubernetes clusters where these are not progressing to completion. This issue appears to relate to an external provider and we expect clusters to progress once the access is resolved. Upgrade actions of existing clusters will also stall, but running clusters are otherwise not affected and can scale and run as normal.
Report: "Network degregation in HLZ"
Last updateFull performance was restored at approx 0410 NZST.
A small number of instances have been affected by a network performance reduction in HLZ beginning from approx 0340 NZST.
Report: "Experiencing issue with new instances"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Experiencing issue with new instances"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Issue affecting some customer servers in nz-hlz-1"
Last updateThis incident has been resolved.
The virtual machines have all been migrated and are now be running. We are keeping an eye on things but expect no further issues.
A compute hypervisor has crashed, some customer virtual machines are impacted. We are currently migrating customer workloads.
The issue has been identified and a fix is being implemented.
Report: "Issue affecting some customer servers in nz-hlz-1"
Last updateThis incident has been resolved.
The virtual machines have all been migrated and are now be running. We are keeping an eye on things but expect no further issues.
A compute hypervisor has crashed, some customer virtual machines are impacted. We are currently migrating customer workloads.
The issue has been identified and a fix is being implemented.
Report: "Maintenance of support ticketing system"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be carrying out maintenance on our support ticketing system during this window. The support panel in our dashboard may not be usable at times during this window.If you need to raise an urgent support ticket, and the panel isn't available, and have premium support, please use that process to raise a ticket.
Report: "Emergency Network Service Maintenance"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be migrating routers from one of our servers. The affected routers will experience one network interruption of up to 60 seconds.
Report: "Block storage service degraded in nz-hlz-1"
Last updateBetween 05:43 and 05:57 a disk failure impacted the NVMe block storage service in the Hamilton region causing delayed or failed read and write requests. The impacted disk was removed from service by automated systems and latency recovered at 05:57.
Report: "Network service has degraded performance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We have detected an issue with the creation of network resources in the nz-por-1 region. The issue may impact the creation of compute servers.
Report: "NVME"
Last updateNVME performance issue resolved at approx. 12:15 NZDT.
NVME block storage experienced brief degradation beginning from approx 12:13 NZDT.
Report: "Degraded performance for block storage in nz-por-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Service API partial outage in the nz-por-1 region"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The following API services are affected: * Compute * Networking * Block Storage * Load Balancers * Cloud Orchestration * Kubernetes cluster creations/updates/deletions (existing clusters are operating normally) Workloads are continuing to operate normally; however you may be unable to make any changes to workloads at this time. The following API services are operating normally: * Identity & Access Management * Object Storage
We are currently investigating this issue.
Report: "Intermittent issues with block storage in nz-hlz-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Network hardware issue porirua"
Last updateWe have replaced the failed hardware. Full service has returned.
We are still working on the replacement hardware.
We are about to replace the broken hardware. In the meantime a mitigation is in place that has greatly reduced the risk of outages.
We are working on replacing the network hardware but continue to experience issues with hdd block storage in porirua
The issue has been identified and a fix is being implemented.
We are experiencing a hardware issue with our networking fabric in the nz-por-1 region. Some services might experience degraded performance.
Report: "Reduced b1.standard block storage performance in nz-por-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Networking connectivity of GPU compute servers"
Last updateThis issue has been resolved - GPU instances are now functioning normally.
Following an upgrade of the compute service, there is an issue that effects the network connectivity of GPU compute servers. All GPU servers will be prevented from having network access. Please note that other compute flavors are not affected.
Report: "Billing services unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "NVME Block Storage Performance Degradation Porirua"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Block storage degraded performance in Hamilton"
Last updateOur fix was succesfull and volumes have been behaving normally since 14:30. We are monitoring for any further issues.
We have had an incident with block storage in Hamilton. We have implemented a fix and are monitoring the result. The symptoms would have been very long access times on nvme volumes.
Report: "Block Storage Maintenance in Porirua"
Last updateBlock storage maintenance is now complete.
We are performing block storage maintenance in Porirua.
Report: "Block storage maintenance in Porirua"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some block storage volumes using the b1.standard storage tier may be experiencing slower read/write times than normal. We are currently investigating the issue.
We are performing maintenance activities on block storage in Porirua. There is no expected impact on workloads.
Report: "Billing dashboard and user management pages partially unaccessible"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Unable to create new load balancers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Intermittent networking issues"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Block Storage maintenance in Hamilton"
Last updateBlock storage maintenance is now complete
We are upgrading some block storage components in Hamilton this week. There is no expected user impact.
Report: "Network Transit issues"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Our upstream ingress protection (cloudflare) is currently experiencing issues in New Zealand. Follow https://www.cloudflarestatus.com/incidents/7znjbdb5bdcd for more frequent updates.
Report: "The APIs in nz-por-1 are experiencing a degraded performance and are slow to serve requests"
Last updateThis incident has been resolved.
Our fix was successful. Please re-try any actions that failed during this incident.
A fix has been implemented. Requests are likely to be slow for a while longer, we will continue monitoring and update status page once this period is over.
We have identified the issue and are working on a solution.
Customers may experience issues in launching, resizing or unshelving instances.
We are currently investigating this issue.
Report: "Intermittent dashboard connectivity issues"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Compute service API issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Failing to create new instances
Report: "Some volumes were inaccessible in the Porirua region"
Last updateA bug in a profiling tool resulted in long access times for some volumes in the Porirua. Some volumes experienced timeouts for a short period until this tool was disabled.
Report: "Issues with compute API in nz-hlz-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Experiencing issues with compute API in nz-hlz-1, we are investigating
Report: "Servers are unable to be created in the nz-hlz-1"
Last updateThis incident has been resolved.
The fix has been applied and compute has been restored. We are monitoring the situation.
We have identified the cause and are in the process of fixing the issue
We are currently investigating this issue.
Report: "Networking interruptions in Porirua Region"
Last updateThe issue with DNS servers has been resolved.
A fix has been implemented and we are monitoring the results.
We believe we have identified the issue with DNS and we are implementing a fix. We are continuing to investigate the interruptions.
We are seeing interruptions in outgoing networking and DNS for the Porirua region. We are investigating and will provide an update soon.
Report: "Dashboard performance degradation"
Last updateThis incident has been resolved.
We are currently experience intermittent degraded performance in our dashboard.
Report: "Networking issues in Porirua Region"
Last updateThis incident has been resolved.
Full network service has resumed and we are monitoring.
We are currently investigating the issue
Report: "Instance DELETE and MIGRATE issues in Porirua"
Last updateThis issue has been resolved.
The cause of the issue has been identified and an emergency change is being applied. A small number of customer instances in this region will be restarted during this process.
Some of our server are experiencing an issue which prevents VMs on these server from being DELETED or MIGRATED.
Report: "Transient network issues"
Last updateThis incident has been resolved.
A fix has been implemented upstream, and we are monitoring the results.
The issue has been narrowed down to a PoP in Auckland. We are continuing to work with the affected upstream to resolve this.
We are continuing to work with the affected upstream on this issue.
Some customers may be experiencing transient packet loss, due to an issue with an upstream provider. Further updates will be provided soon.
Report: "Octavia errors in nz-hlz-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Octavia in nz-hlz-1 is experiencing some issues, we are investigating.
Report: "Unexpected hypervisor restart"
Last updateThe hypervisor is fully operation and all of the instances that were running on it have been restarted.
A fix has been implemented and we are monitoring the results.
All of the instances that were running on the affected hypervisor have been restarted.
During compute maintenance one hypervisor rebooted unexpectedly. The instances running on the hypervisor suffered an unplanned shut down.
Report: "DNS issues affecting some customers"
Last updateThe problem have been resolved.
We are in the process of reverting this change of behaviour and expect the old behaviour to be restored everywhere within the hour.
We have identified a DNS issue affecting some customers in the Hamilton region. Engineers are investigating a change in name resolution behaviour that was introduced yesterday, which may affect customers who filter DNS queries on remote servers using ACLs.
Report: "DNS resolution failures"
Last updateWe have flushed the cache in our DNS servers across all regions in order to ensure that the incorrect data published by InternetNZ is no longer available. For details about the .NZ DNS issues, see: https://status.internetnz.nz/incidents/gq1c6slz3198
Report: "API issues in nz_wlg_2"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Block Storage API issues in Porirua"
Last updateAll queued updates have been applied. Please contact support if you are still experiencing any issues.
Engineers are still monitoring the recovery of some queued network updates that were delayed.
Engineers have detected issues that are impacting many API operations across multiple API services. We are restoring services currently.
We are investigating issues with the Block Storage API in the Porirua region. Some Volume Create/Update/Delete operations are failing. Engineers are investigating.
Report: "Degraded IO performance in NZ-WLG-2"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified, and we expect IO performance to be resolved. This may decrease briefly over the next 15-30 min., but will return to normal following this. Further updates will be provided.
This incident is ongoing, further updates will be provided soon.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Object storage issue"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Compute and Networking in Porirua region"
Last updateThis incident has been resolved.
Engineers have restored the affected compute workloads. If you continue to have problems with your workloads, please contact support.
Engineers are continuing to restore workloads.
We understand the cause of the compute incident in Porirua and engineers are working to restore workloads.
We have identified compute failures in the Porirua region. Engineers are working to restore workloads.
We are investigating issues with Compute and Networking services in the Porirua region.
Report: "DNS issue in nz-hlz-1"
Last updateThis incident has been resolved.
We are currently experiencing intermittent DNS issues in the nz-hlz-1 region. Some services might experience momentary disruption.
Report: "API and Dashboard issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "API Issues in nz-por-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "API Issues in nz-por-1"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring
We're aware of issues with the API in the nz-por-1 region and are investigating.
Report: "Network outage in NZ_WLG_2 region"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Block storage API issue"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We have implemented a fix and are monitoring to confirm resolution.
We have identified the issue and are working towards resolution.
We are investigaing
Report: "Network transit issue in Porirua"
Last updateThis incident has been resolved.
A issue has been identified and resolved. We are continuing to monitor the situation.
We have identified an issue with our network transit in our nz-por-1 region and are working on resolving this.
Report: "APIs degraded"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Object Storage Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We have