Instaclustr

Is Instaclustr Down Right Now? Check if there is a current outage ongoing.

Instaclustr is currently Operational

Last checked from Instaclustr's official status page

Historical record of incidents for Instaclustr

Report: "Provisioning Failures for AWS Clusters provisioned in customer managed AWS accounts"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented

investigating

We are currently experiencing issues with provisioning clusters on the AWS Provider on customer managed AWS accounts through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Provisioning Failures for AWS Clusters provisioned in customer managed AWS accounts"

Last update
Resolved

This incident has been resolved.

Monitoring

A fix has been implemented and we are monitoring the results.

Identified

The issue has been identified and a fix is being implemented

Investigating

We are currently experiencing issues with provisioning clusters on the AWS Provider on customer managed AWS accounts through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Increased Failure Rate of Apache Cassandra Backups"

Last update
resolved

This incident has been resolved.

monitoring

We have seen a consistent reduction in errors rates from Apache Cassandra backup events over the past 3 days, however, we will continue to closely monitor. We expect to provide another update in the next 24 hours.

monitoring

We are continuing to observe a reduction in errors rates from Apache Cassandra backup events, however, we will continue to closely monitor. We expect to provide another update in the next 24 hours.

monitoring

We are continuing to observe a reduction in errors rates from Apache Cassandra backup events, however, we will continue to closely monitor. We expect to provide another update in the next 24 hours.

monitoring

A fix has been deployed to all Apache Cassandra nodes. Initial results indicate the problem has been resolved, however, we will continue to closely monitor. We expect to provide another update in the next 24 hours.

monitoring

We have started rolling out a fix, this is expected to take a few hours to be applied to all Apache Cassandra nodes, we will be monitoring the progress and effectiveness of the rollout closely.

identified

The issue has been identified and we are preparing to rollout a fix.

investigating

We are currently seeing an elevated rate of Backup Failures for AWS nodes for our Apache Cassandra offering. Currently we are expecting that these backups will continuously retry and eventually succeed, however this will be visible in the Instaclustr console and APIs as failed backup events. We are actively monitoring and working on a solution to this, and will provide more updates as investigation continues. If you have any questions or concerns please reach out via support@instaclustr.com

Report: "Instaclustr Terraform Provider Issue"

Last update
resolved

This incident has been resolved.

monitoring

We have applied the fix and are monitoring the result.

identified

We have identified and are applying the fix.

investigating

We are currently experiencing a validation issue with Instaclustr Terraform Provider. You may see the error message "Internal validation of the provider failed" due to this issue and may not be able to manage your clusters using Instaclustr Terraform Provider. Our team is working on resolving this problem and we apologise for any inconvenience caused. Cluster Management API and Management Console are still operational and can be used to manage your clusters. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "New Clusters may get Stuck during the Provisioning Stage and the Monitoring API is experiencing elevated latencies and error rates"

Last update
resolved

This incident has been resolved.

monitoring

We are actively monitoring the situation concerning Monitoring API latencies and are pleased to announce that the latency levels are currently aligning with our expectations. We will maintain our oversight and provide updates as new information becomes available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

monitoring

We have successfully resolved the issue that was causing provisioning clusters to get stuck during the provisioned stage. Currently, we are continuing to monitor the situation regarding Monitoring API latencies, and we are pleased to report that latency levels are returning to normal. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

monitoring

We are monitoring the results of the fix and we are seeing reduced latencies in our Monitoring API. We anticipate that latency levels will return to normal in the next few hours and are continuing to monitor the outcome of the fix. Existing customer clusters are unaffected. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

investigating

We have implemented a fix that has resulted in improved latencies for the Monitoring API. Our team is continuously investigating a permanent fix to resolve this issue. We are also still seeing issues with clusters stuck during the Provisioning stage, and investigating a fix for this as-well. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

investigating

We are still seeing elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused.

identified

The issue has been identified and a fix is being implemented.

investigating

We are also investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

investigating

We are currently experiencing issues with provisioning clusters on through the Cluster Management API and Management Console. New Clusters created via the Instaclustr Console and Cluster Management API may get stuck during the Provisioning stage. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster Provisioning Failure for Azure Provider"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently experiencing issues with provisioning clusters on Azure provider through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Issues with setting up ClickHouse S3 integrations"

Last update
resolved

This incident has been resolved.

identified

We are currently experiencing issues with enabling ClickHouse S3 integrations through the Cluster Management API and Management Console. Our team has identified a fix and is actively working on resolving this problem. We apologise for any inconvenience caused. If you need an S3 integration set up before the fix is deployed, please contact Instaclustr support. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Transient Elasticsearch Cluster Provisioning Failures for Google Cloud Platform"

Last update
resolved

A fix has been deployed and the issue has been resolved.

identified

The issue has been identified, and a fix is being worked on

investigating

We are currently experiencing transient issues with provisioning new ElasticSearch clusters on Google Cloud Platform through the Cluster Management API and Management Console. No other applications are affected by this (including OpenSearch). Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "[Completed] Monitoring and Prometheus APIs Maintenance"

Last update
resolved

Maintenance is now completed, and the Monitoring and Prometheus APIs are operating normally.

investigating

Maintenance is still in-progress, but is taking a bit longer than expected. We expect it to be complete by 03:00 UTC time.

investigating

The maintenance is now under way, and should be completed within 2 hours.

investigating

We will be conducting maintenance on our Monitoring and Prometheus APIs for a duration of roughly **2 hours**. The maintenance is scheduled to take start on **2024-11-28** at **23:00** and finish on **2024-11-29** at **01:00** (UTC time). During this period, there may be brief disruptions to our Monitoring and Prometheus APIs, which could include failed requests that need to be retried, or latency spikes. During this maintenance, existing customer clusters will continue to operate normally, and support will be accessible through the support portal (support.instaclustr.com) and email (support@instaclustr.com). Our internal monitoring will remain active throughout the maintenance, and our support team will address any issues as per normal operations. If you have any concerns or questions regarding the impact of this maintenance, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "ClickHouse provisioning temporarily unavailable"

Last update
resolved

This incident has been resolved.

identified

We have identified the root cause and the team is currently working on a fix. Further updates will be provided as they become available.

investigating

Provisioning of ClickHouse clusters is currently unavailable. This affects provisioning through the Console and through the Instaclustr Provisioning API. Provisioning of all other offerings is unaffected and no impact on existing running clusters including ClickHouse.

Report: "Monitoring API Latency and Error Issues"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Customers unable to create new clusters via console in certain configurations"

Last update
resolved

This incident has been resolved.

identified

We have identified further configurations that cannot be provisioned via the managed console, these are: - Provisioning Kafka with Tiered Storage is not possible (all cloud providers) - Provisioning OpenSearch with Searchable Snapshots on Google Cloud Platform is not possible - Provisioning OpenSearch with Cross-cluster Replication is not possible (all cloud providers) We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (https://support.instaclustr.com) or email (support@instaclustr.com).

identified

Users of the Google Cloud Platform (GCP) Marketplace are currently unable to create new clusters via the management console, existing clusters and other cloud services (Amazon Web Services, Azure and Google Cloud) are unaffected. Users of GCP Marketplace can still use our Cluster Management API or Terraform provider to provision new clusters. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (https://support.instaclustr.com) or email (support@instaclustr.com).

Report: "Monitoring API Latency and Error Issues"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Malformed Email Notifications"

Last update
resolved

This incident has been resolved.

identified

We have identified the root cause and the team is currently working on a fix. Further updates will be provided as they become available.

identified

We have identified an issue where some email notifications are displaying malformed messages. As a result, customers may receive email notifications that are not properly formatted, potentially affecting readability and the ability to understand the content. New and existing customer clusters are unaffected. The Management Console, Cluster Management API, Monitoring API, Prometheus API and Support Portal are still operational. Additional details will be provided as they become available. If you are affected by this issue, please contact us via the support portal (support.instaclustr.com) or email (support@instaclustr.com) for assistance and we will provide a workaround.

Report: "Delays in Sending Customer Notification Emails"

Last update
resolved

This incident has been resolved.

identified

We are currently investigating issues with the outbound email function in our Management Console. As a result, our system is experiencing delays in sending customer notifications via email. This includes account verification and password reset emails. Our team is actively investigating the issue and we apologise for any inconvenience caused. New and existing customer clusters are unaffected. The Management Console, Cluster Management API, Monitoring API, Prometheus API and Support Portal are still operational. Additional details will be provided as they become available. If you are affected by this issue, please contact us via the support portal (support.instaclustr.com) or email (support@instaclustr.com) for assistance and we will provide a workaround.

Report: "Service Disruptions for Management Console and APIs"

Last update
resolved

This incident is now resolved.

investigating

We are currently investigating elevated error rates affecting the email delivery system. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Customers requests submitted from the console or API are affected. For ex: add nodes, add new DataCentre etc. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com).

Report: "Private network Redis cluster provisioning issues"

Last update
resolved

We are pleased to inform you that the issue affecting the provisioning of Redis clusters with the Enterprise feature "Private Network Cluster" has been resolved. The fix has been successfully deployed and tested. We appreciate your patience and understanding as we worked to resolve this matter. If you experience any further issues or have any questions, please do not hesitate to reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

identified

We have identified the root cause of the issue and are currently testing a fix. We anticipate releasing the fix within the next few hours. Thank you for your patience and understanding as we work to resolve this matter. We will provide another update as soon as the fix is deployed. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

investigating

Creating a Redis cluster with the Enterprise feature Private Network Cluster, will fail to provision the cluster. We are currently investigating the issue. There is no impact to existing Redis clusters. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Service Disruptions for Deprecated Provisioning API v1"

Last update
resolved

This incident has been resolved. Rate limits for the provisioning v1 API have been restored to the original limit of 750 requests/minute. All Instaclustr APIs are operating as expected with standard API rate limits.

investigating

We have increased our rate limits for our provisioning v1 API to 350 requests/minute and are currently working towards adjusting it further with the goal of restoring to original rate limits. Rate limits for Terraform Provider v1 usage have been completely restored to the original limits of 750 requests/minute. Existing customer clusters and the Management Console continue to be unaffected. We will provide further updates as we continue to restore our rate limits. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com).

investigating

We are currently facing ongoing degradation of service on our deprecated provisioning v1 API, which is affecting both direct API calls and interactions with Terraform v1. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. The Management Console and Cluster Management API are still operational. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com).

Report: "Support Email Service Degradation"

Last update
resolved

This incident has been resolved.

identified

We are aware that some users may be experiencing issues sending emails to our support email (support@instaclustr.com) due to a problem with an upstream provider. Our team is currently investigating the problem and we apologise for any inconvenience caused. New and existing customer clusters are unaffected. Additional details will be provided as they become available. If you have an urgent matter, please create a ticket through support.instaclustr.com.

Report: "Support Email Unavailable"

Last update
resolved

The fix from the upstream provider has been rolled out, and the issue has been resolved.

identified

The issue has been identified and we're waiting on a fix being rolled out by our upstream provider.

investigating

We are aware that some users may be experiencing issues sending emails to our support email (support@instaclustr.com) due to a problem with an upstream provider. Our team is currently investigating the problem and we apologise for any inconvenience caused. New and existing customer clusters are unaffected. Additional details will be provided as they become available. If you have an urgent matter, please create a ticket through support.instaclustr.com.

Report: "Service Disruptions for Management Console and Cluster Management API"

Last update
resolved

A fix has been applied, and latencies on both the Management Console and Cluster Management API have returned to baseline levels.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating elevated latencies affecting our Management Console and Cluster Management API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. Our Monitoring API and Prometheus API is unaffected. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Azure Central US Region Operational Issues"

Last update
resolved

This incident has been resolved as indicated by updates from Azure.

monitoring

We are continuing to see recovery in our Provisioning systems as indicated by updates from Azure. We are actively monitoring the situation and will provide additional details as they become available. Azure status is tracked at https://azure.status.microsoft/en-us/status/. Please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com) if you have any further questions.

identified

We are currently experiencing service degradation and potential disruptions in the Azure Central US region. This incident is a result of an ongoing issue with the cloud provider's infrastructure. You will be unable to provision new clusters or nodes in Azure Central US region. All nodes in the Azure Central US region are unavailable. Node resize and replace, Monitoring API and Prometheus API for nodes in Azure Central US region are also affected. Individual impacted customers are being contacted via support tickets. Clusters in other Azure regions and cloud providers are unaffected. We are actively monitoring the situation and will provide additional details as they become available. Azure status is tracked at https://azure.status.microsoft/en-us/status/. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Service Disruptions for Management Console and Cluster Management API"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating elevated latencies affecting our Management Console and Cluster Management API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. Existing customer clusters are unaffected. Our Monitoring API and Prometheus API is unaffected. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Failure when adding a data centre to an existing Redis cluster in GCP or Azure"

Last update
resolved

This incident has been resolved.

identified

We are currently experiencing issues with adding a data centre through the Cluster Management API and Management Console for Redis clusters that are hosted with GCP or Azure. Add data centre requests for Redis clusters will be accepted but the nodes for the new data centre will not reach running. We have identified a manual workaround for the issue, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com) to action this. Our team is actively working on resolving the underlying problem and we apologise for any inconvenience caused. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Private Network Cluster Provisioning Failure for AWS"

Last update
resolved

Provisioning of new AWS Private Network clusters is working again now.

investigating

We are currently experiencing issues with provisioning Private Network clusters on AWS through the Cluster Management API and Management Console. Provisioning requests will be accepted but the cluster nodes will not reach running. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. At this stage, existing customer clusters appear to be unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "GCP Operational Issues"

Last update
resolved

This incident has been resolved.

monitoring

Google is reporting a fix and we're seeing services returning to normal.

identified

We are currently experiencing service degradation and potential disruptions in the GCP cloud provider across all regions. This incident is a result of an ongoing issue with the cloud provider's infrastructure. You will be unable to provision new clusters or nodes, node replacement operation and resize operations are also unavailable. You are still able to access your clusters if you are running a standard configuration. Individual impacted customers are being contacted via support tickets. We are actively monitoring the situation and will provide additional details as they become available. GCP status on this issue can be tracked at https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#guoaCiLrsny92g8eoeyR If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster restores are failing on Azure AZ provider"

Last update
resolved

The fix identified has been implemented, and is working as expected. The issue has been resolved.

identified

We are noticing some failures in restoring to New Clusters in the Azure AZ provider. We have identified the problem and are working on an urgent fix. We apologise for any inconvenience caused. If you have any concerns or questions regarding how this may affect you, or in case you needed any assistance, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cadence Node Deletion Failure"

Last update
resolved

We have rolled out a fix which has resolved this issue

identified

We are currently experiencing failure with deleting Cadence nodes via the cluster resize feature through the Management Console. We have identified the problem and are working on an urgent fix. We apologise for any inconvenience caused. Creating new Cadence clusters is unaffected. Additional details will be provided as they become available. If you have an urgent matter or would like to downsize your Cadence clusters, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster Deletion Failure for clusters on Azure Provider"

Last update
resolved

The issue preventing the deletion of clusters on Azure via the Cluster Management API and the Management Console has been successfully resolved following the deployment of the fix. If you have any concerns or if you encounter any issues, please don't hesitate to reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently experiencing issues with deleting clusters on Azure through the Cluster Management API and Management Console. The fix has been identified and our team is actively working on resolving this problem and we apologise for any inconvenience caused. No customer clusters are known to be impacted. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Metrics unavailable"

Last update
resolved

We have resolved an issue may have caused the unavailability of metrics from our Monitoring API and Prometheus API. As a result of this issue there may be missing metrics during the time window of 15:07 to 16:27 AEST. If you have any concerns or if you encounter any issues, please don't hesitate to reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster provisioning delayed"

Last update
resolved

Following the implementation of the fix, cluster provisioning performance has returned back to normal. If you have any concerns or if you encounter any issues, please don't hesitate to reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

identified

The issue has been identified and we are currently working on applying a fix. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

investigating

We are currently experiencing issues with provisioning clusters. Provisioning clusters is taking an extended amount of time. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Elevated latencies and error rates with our Cluster Management API and Monitoring API"

Last update
resolved

After rolling out the fix, the elevated latencies and error rates affecting our Monitoring API and Cluster Management API have been successfully resolved and we have seen a return to normal performance levels. If you have any concerns or if you encounter any issues, please don't hesitate to reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

identified

We have identified the issue causing elevated latencies and error rates on the Monitoring API and Cluster Manager API, and are currently rolling out a fix.

investigating

We've identified that Cluster Management API is experiencing elevated latencies and error rates as well. We are actively investigating the issue and we apologise for any inconvenience caused.

investigating

We are currently investigating elevated latencies and error rates with our Monitoring API. Our team is actively working to resolve the issue and we apologise for any inconvenience caused. We will provide updates as soon as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Provisioning on AWS is unavailable"

Last update
resolved

After AWS has fixed their IAM Operation issue, we have tested the new node provisioning, replace and resize features using AWS platform is working fine.

identified

Currently, AWS is focused on mitigating the issue resulting in propagation delays to reduce the API error rate first, after that they will shift toward full resolution, and then towards understanding root cause and preventing recurrence. During this issue, customers may also be unable to load portions of the IAM Management Console, or may receive a message when attempting to navigate to the IAM Management Console homepage that says “IAM service page is currently unavailable”. We will continue to provide additional updates as we have them, or within the next 60 minutes.

identified

According to AWS following are the affected AWS services: Informational (16 services) AWS Amplify, AWS App Runner, AWS Client VPN, AWS CloudFormation, AWS IAM Identity Center, AWS IoT Core, AWS Organizations, AWS Service Catalog, AWS Systems Manager, AWS Verified Access, Amazon API Gateway, Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, Amazon Elastic Load Balancing, Amazon VPC IP Address Manager, Reachability Analyzer

identified

AWS is reporting increased error rates and latencies for IAM APIs. Due to these errors, Instaclustr users could see AWS provisioning issues. New nodes provisioning, replace with new instances and resize functionality will be effected. The issue is occurring due to AWS IAM service increased error rates. AWS is investigating increased error rates for AWS Identity and Access Management (IAM). Authentication and authorization of existing users, credentials, roles, policies are not impacted.

Report: "Private Network Cluster Provisioning Failure for AWS in EU_WEST_3"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently experiencing issues with provisioning clusters on AWS through the Cluster Management API and Management Console. It appears only EU_WEST_3 is affected at this stage. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster and instance Provisioning Failure for AWS"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are currently experiencing issues with provisioning clusters on AWS through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Issue with Terraform Provider v2.0.96"

Last update
resolved

This incident has been resolved.

identified

We are currently experiencing errors when using the Instaclustr Terraform Provider v2.0.96 Our team is actively working on resolving this problem and we apologize for any inconvenience caused. While we actively work on a resolution, we recommend using version 2.0.95. This temporary measure will allow you to continue your operations without interruption. We will provide additional details as they become available. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Azure All Regions Operational Issues"

Last update
resolved

This incident has been resolved and provisioning services are fully functional again.

identified

We are currently experiencing service degradation and disruptions in all the Azure AZ regions This incident is a result of an ongoing issue with the cloud provider's infrastructure. You will be unable to provision, delete or resize clusters or nodes. Existing, running clusters and nodes are unaffected. We are actively monitoring the situation and will provide additional details as they become available. Azure AZ status is tracked at https://azure.status.microsoft/en-us/status/. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Issue with Terraform Provider v2.0.94"

Last update
resolved

This incident has been resolved.

identified

We are currently experiencing errors when using the Instaclustr Terraform Provider v2.0.94. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. While we actively work on a resolution, we recommend using any version of our Terraform provider prior to 2.0.94. This temporary measure will allow you to continue your operations without interruption. We will provide additional details as they become available. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster restores are failing for Postgres"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are currently monitoring the results.

identified

We have identified a solution and are currently working on a fix

investigating

We are currently experiencing errors when submitting a restore request for Postgres clusters through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster Provisioning Failure for AWS"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

We are currently experiencing issues with provisioning all clusters on AWS through the Cluster Management API and Management Console. Our team is actively monitoring the situation and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Cluster Resize Failure"

Last update
resolved

This incident has been resolved.

identified

We are currently experiencing issues with resizing clusters through the Cluster Management API and Management Console. We have identified the fix and are actively working on resolving this problem. We apologise for any inconvenience caused. The Monitoring API and Prometheus API are operational. If you need to resize your clusters, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com). We can manually resize your clusters for you.

Report: "Cluster Provisioning Failure for Cadence"

Last update
resolved

The issue has been resolved. We have identified a reasonable workaround to resume Cadence cluster and node provisioning on the affected regions. Please reach out to Instaclustr Support if you still see any issues.

monitoring

We have identified a reasonable workaround to resume Cadence cluster and node provisioning on the affected regions. We will still continue to monitor the status updates from the third party provider.

identified

Provisioning of new Cadence clusters and nodes is currently experiencing issues in some regions. We have identified this to an upstream incident with a third party provider. We will monitor this issue as it progresses, existing clusters and running nodes are unaffected.

investigating

We are currently experiencing issues with provisioning Cadence clusters through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "Issues provisioning new clusters and nodes"

Last update
resolved

This incident has been resolved.

monitoring

The upstream incident with a third-party provider is resolved. We are monitoring the results.

identified

Provisioning of new clusters and nodes is currently elevated error levels, we have identified this to an upstream incident with a third party provider. We will monitor this issue as it progresses, existing clusters and running nodes are unaffected.

Report: "Issues provisioning new clusters"

Last update
resolved

This incident has been resolved

investigating

Provisioning of new clusters and new nodes is currently taking longer than usual and in some cases failing due to acknowledge issues with one of our service providers (quay.io). We are monitoring the issue and will advise when resolved. See http://status.quay.io/ for quay.io status.

Report: "Provisioning Failures for AWS private network clusters"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified the root cause and are currently testing a fix for this issue.

identified

We are currently experiencing issues with provisioning clusters on AWS through the Cluster Management API and Management Console. Our team is actively working on resolving this problem and we apologise for any inconvenience caused. Existing customer clusters are unaffected, and the Monitoring API and Prometheus API are operational. Additional details will be provided as they become available. If you have an urgent matter, please reach out to us via the support portal (support.instaclustr.com) or email (support@instaclustr.com).

Report: "OpenSearch cluster creation fails through the Console for Run-In-Your-Own-Account clusters"

Last update
resolved

This incident has been resolved.

identified

We are continuing to test the fix for this issue.

identified

OpenSearch cluster creation is currently failing at the submission step for Run-In-Your-Own-Account clusters when the provisioning is attempted through the Instaclustr Console. The root cause has been identified and the fix is being tested. Provisioning through the Instaclustr API and Terraform Provider remain unaffected. There is no impact to existing clusters.

Report: "Backups failing for restored GCP clusters"

Last update
resolved

Investigations and issue resolution on affected GCP clusters have been finalised.

monitoring

A fix has been rolled out to resolve backup issues on newly restored GCP clusters and we are investigating impact and resolution for existing affected clusters.

investigating

GCP clusters restored since September 18 2023 are seeing backup failures. We are currently investigating the issue.

Report: "AWS US_EAST_1 Operational Issue"

Last update
resolved

This incident has been resolved.

identified

We have observed service degradation in the AWS region US_EAST_1. This is due to an ongoing AWS issue, we are monitoring the situation as it unfolds - See https://health.aws.amazon.com/health/status for further details.

Report: "instaclustr.com website is unavailable"

Last update
resolved

This incident has been resolved.

identified

The Instaclustr.com website is currently unavailable, please note that the console is still available and can be accessed via the following link https://console2.instaclustr.com/ All platform features and functionality are currently operational and are available, clusters (both new and existing) are unaffected by this outage, supporting systems such as monitoring and our APIs remain available and continue to operate as expected. Support remains available and can be contacted via the support portal at https://support.instaclustr.com/

Report: "Azure outage - Australia East region"

Last update
resolved

All impacted Azure clusters were recovered.

monitoring

Azure has recovered most of its impacted services. We will be aiming to ensure the impacted clusters are back to normal. --- Current Status: With 99% of storage services and 99% of impacted Virtual Machines back online and healthy, we are actively investigating remaining issues with individual downstream services to confirm their recovery status. Our Storage team are making progress on one specific storage scale unit that is still experiencing isolated issues. Our SQL team are investigating a potential issue with an underlying Service Fabric dependency. Our Cosmos DB team are investigating why some services have not fully recovered. Despite these remaining investigations, the majority of customers and services should already be recovered. Further updates will be provided in 60 minutes, or as events warrant.

identified

Azure is investigating this issue actively - https://azure.status.microsoft/en-us/status/ --- Impact Statement: Starting at 08:30 UTC on 30 August 2023, a subset of customers with workloads hosted in the Australia East region may be experiencing difficulties accessing or managing some resources deployed in this region. Current Status: We are experiencing impact related to a cooling issue for a sub-section of a single data centre in the Australia East region. This is resulting in connectivity and availability issues for some Storage and Compute resources in this region. Additional Azure services with dependencies on these resources may also experience impact related to this. We are actively working onsite to mitigate the cooling issue, and updates will be provided in an hour or as events warrant.

investigating

We have observed an impact to clusters on Azure in Australia East region due to an ongoing Azure outage - https://azure.status.microsoft/en-us/status/. No Azure cluster in other regions are impacted for now.

Report: "Degraded platform performance"

Last update
resolved

We are unable to recover cluster metrics for some clusters between 3:19AM- 3:25AM UTC. We will be performing an investigation into this issue, and looking to remediate it going forward. Availability for other metrics remains as usual. Remaining services have been remediated.

monitoring

We are investigating a related issue where some cluster metrics between 3:19AM UTC - 3:25AM UTC are not available via our monitoring API or console metrics view.

monitoring

We've identified the issue, and are remediating remaining services that are still affected. The management console, monitoring API and cluster management API appear to be working as expected.

investigating

We are investigating some issues in our platform that are causing availability issues on our services. Operations may be affected across multiple services. No impact to customer clusters is expected.