Historical record of incidents for Elastic Cloud
Report: "Kibana access/logout issue"
Last updateWe have identified an issue where customers may be seeing more frequent logouts or messages about invalid access tokens in Kibana. This is impacting deployments that have multiple Kibanas only. We are working on the remediation and will update again in an hour. If you run into this issue, reloading the page should resolve the issue.
Report: "Serverless project creation in GCP"
Last updateAt this tine we have identified an issue which is preventing Serverless projects from being created in GCP. We are currently investigating the situation
Report: "Customers Unable to Create BYOK Deployments in Azure"
Last updateThe Elastic Cloud Hosted service is experiencing issues with the creation of new BYOK (bring-your-own-key) deployments in Azure regions. This only affects users attempting to create a new BYOK deployment in Azure regions. Existing Azure BYOK and non-BYOK deployments are not impacted. The team is working on identifying the root case.
Report: "New serverless project creations delayed in aws-eu-west-1"
Last updateWe are aware of delays in serverless project creations in AWS eu-west-1 region. The root cause for the issue has been identified and the team is working on a remediation. We will post an update within the next hour.
Report: "Delay in projects creation in Azure East US (Virginia)"
Last updateConfirming delays in projects creation in Azure East US has been resolved.
Services have stabilized, and performance has returned to normal. We’re conducting final checks to ensure everything is fully resolved.
We are currently investigating an issue causing delays in project creation for customers in the Azure East US (Virginia) region. Our team is actively working to identify the root cause and implement a solution. We will provide updates here as soon as more information becomes available.
Report: "Delay in projects creation in Azure East US (Virginia)"
Last updateWe are currently investigating an issue causing delays in project creation for customers in the Azure East US (Virginia) region. Our team is actively working to identify the root cause and implement a solution. We will provide updates here as soon as more information becomes available.
Report: "Loading Issues for Deployments"
Last updateIssue has been resolved, systems are operational.
The fix has been deployed, we keep monitoring the situation.
Some customers deployments edit page on the Elastic Cloud Console are currently stuck in a loading state. The root cause has been identified, and the team is working on a solution
Report: "Loading Issues for Deployments"
Last updateSome customers deployments edit page on the Elastic Cloud Console are currently stuck in a loading state. The root cause has been identified, and the team is working on a solution
Report: "Elastic Cloud: Loading Issues for Deployments and Projects"
Last updateThis incident has been resolved.
A fix has been deployed, and some customers' deployments and projects are loading as expected again.
A fix has been deployed, and some customers' deployments and projects are loading as expected again.
Deployments and projects on the Elastic Cloud homepage are currently stuck in a loading state. The root cause has been identified, and the team is working on a solution
Report: "Elastic Cloud: Loading Issues for Deployments and Projects"
Last updateDeployments and projects on the Elastic Cloud homepage are currently stuck in a loading state. The root cause has been identified, and the team is working on a solution
Report: "Issue with capture and display of deployment metrics in AWS eu-west-1"
Last updateWe have completed our work to restore deployment metrics in AWS eu-west-1 and the service is back to normal.
We have applied a fix to restore the display of deployment metrics in AWS eu-west-1 and the service is recovering. We will continue monitoring for further impact and provide an update in one hour, or sooner if there is a change in status.
We are investigating a potential service impact with the capture and display of deployment metrics. Our team is actively investigating the cause so that they can mitigate and restore the service. We believe customers in AWS eu-west-1 may be affected. We will post an update in one hour, or sooner if there is a change in status.
Report: "Issue with capture and display of deployment metrics in AWS eu-west-1"
Last updateWe are investigating a potential service impact with the capture and display of deployment metrics. Our team is actively investigating the cause so that they can mitigate and restore the service. We believe customers in AWS eu-west-1 may be affected. We will post an update in one hour, or sooner if there is a change in status.
Report: "Deployments in azure-southafricanorth are not able to be modified or created."
Last updateThis incident has been resolved.
The network connectivity has been restored and we see that the service availability has been fully restored. We continue to monitor the situation and await Azures final confirmation of network connectivity restoration. We will post an update in the next two hours or sooner if we have additional details to share.
Our investigations with Azure engineers have concluded that a peering issue between Orange S.A. (AS5511) and Azure is causing this issue. Azure is working with Orange S.A. to resolve this. Any connection towards this region going through the Orange S.A. may experience connectivity issues. This may also include access to elasticsearch, kibana, APM endpoints. Azure has not yet publicly acknowledged this network issue on their status page. We will post an update in the next two hours or sooner if we have additional details to share.
We're still investigating the connectivity issue in the azure-southafricanorth region together with engineers from Azure. This connectivity issue is impacting customers' ability to start, stop, or modify deployments. We will post an update in the next two hours or sooner if we have additional details to share.
We're still investigating a connectivity issue in our azure-southafricanorth region that is impacting customers' ability to start, stop, or modify deployments. We will post an update in an hour or sooner if we have additional details to share.
We're still investigating a connectivity issue in our azure-southafricanorth region that is impacting customers' ability to start, stop, or modify deployments. We will post an update in an hour or sooner if we have additional details to share.
We're currently investigating a possible connectivity issue in our azure-southafricanorth region that is impacting customers' ability to start, stop, or modify deployments. We will post an update in an hour or sooner if we have additional details to share.
Report: "Deployments in azure-southafricanorth are not able to be modified or created."
Last updateWe're currently investigating a possible connectivity issue in our azure-southafricanorth region that is impacting customers' ability to start, stop, or modify deployments. We will post an update in an hour or sooner if we have additional details to share.
Report: "Past Due Invoice Emails Incorrectly Sent"
Last updateWe are aware of an issue that caused some credit card customers to receive a "Past Due Invoice" email for their April 1 invoice, despite a successful payment. This email was sent in error and does not indicate that successfully paid invoices will be charged again. We have identified the root cause and are working with a vendor to understand options to prevent its recurrence. We apologize for the inconvenience.
We are aware of an issue that caused some credit card customers to receive a "Past Due Invoice" email for their May 1 invoice, despite a successful payment. This email was sent in error and does not indicate that successfully paid invoices will be charged again. We have identified the root cause and are working with a vendor to understand options to prevent its recurrence. We apologize for the inconvenience.
Report: "Past Due Invoice Emails Incorrectly Sent"
Last updateWe are aware of an issue that caused some credit card customers to receive a "Past Due Invoice" email for their April 1 invoice, despite a successful payment. This email was sent in error and does not indicate that successfully paid invoices will be charged again. We have identified the root cause and are working with a vendor to understand options to prevent its recurrence. We apologize for the inconvenience.
Report: "Delays in the Managed Intake Service on Azure Serverless"
Last updateThis incident has been resolved.
We have mitigated the problem and we are seeing signs of recovery. We will be monitoring the situation for the next our before resolving the incident
We are aware of delays processing events in the Managed Intake Service on Elastic Cloud Serverless on Azure and we are investigating. We will provide an update within 1 hour or sooner if applicable
Report: "Delays in the Managed Intake Service on Azure Serverless"
Last updateThis incident has been resolved.
We have mitigated the problem and we are seeing signs of recovery. We will be monitoring the situation for the next our before resolving the incident
We are aware of delays processing events in the Managed Intake Service on Elastic Cloud Serverless on Azure and we are investigating. We will provide an update within 1 hour or sooner if applicable
Report: "slow loading times on the elastic.co webpage"
Last updateThis incident has been resolved.
We have implemented a fix to mitigate the problems with website latency on www.elastic.co Metrics indicate that the performance has improved, and returned to expected levels. Monitoring will continue.
We are continuing to investigate some alerts and reports of latency on our www.elastic.co website. Some users may experience a delay with the page loading. We will provide an update within 1 hour
We are aware that some users may experience slow loading times on the elastic.co webpage. Our engineers are currently investigating, and will provide an update within 1 hour
Report: "slow loading times on the elastic.co webpage"
Last updateThis incident has been resolved.
We have implemented a fix to mitigate the problems with website latency on www.elastic.coMetrics indicate that the performance has improved, and returned to expected levels.Monitoring will continue.
We are continuing to investigate some alerts and reports of latency on our www.elastic.co website.Some users may experience a delay with the page loading.We will provide an update within 1 hour
We are aware that some users may experience slow loading times on the elastic.co webpage. Our engineers are currently investigating, and will provide an update within 1 hour
Report: "Cloud MFA emails may be delayed"
Last updateOur service has fully recovered from the upstream SendGrid outage.
We've observed that our platform's MFA email reliability has improved. We are continuing to monitor the status of our MFA provider and SendGrid for updates. https://status.sendgrid.com/incidents/6fj45kvpw8n3
Elastic is currently aware of delivery delays for MFA emails while logging into https://cloud.elastic.co. We have identified the issue as an upstream outage of Sendgrid, which is used by our MFA provider to send emails - https://status.sendgrid.com/ We are currently monitoring the situation and will provide an update within 1 hour.
Report: "Cloud MFA emails may be delayed"
Last updateOur service has fully recovered from the upstream SendGrid outage.
We've observed that our platform's MFA email reliability has improved. We are continuing to monitor the status of our MFA provider and SendGrid for updates. https://status.sendgrid.com/incidents/6fj45kvpw8n3
Elastic is currently aware of delivery delays for MFA emails while logging into https://cloud.elastic.co.We have identified the issue as an upstream outage of Sendgrid, which is used by our MFA provider to send emails - https://status.sendgrid.com/We are currently monitoring the situation and will provide an update within 1 hour.
Report: "Auto-Ops returning 404 in some deployments"
Last updateThe Auto-Ops links are working correctly at this time.
We have identified the issue and are working on rolling out a fix.
We are aware that some users may be receiving a 404 error when trying to open AutoOps deployment status pages. If you experience this error, you can work around it by opening the page in a new tab.
Report: "Auto-Ops returning 404 in some deployments"
Last updateThe Auto-Ops links are working correctly at this time.
We have identified the issue and are working on rolling out a fix.
We are aware that some users may be receiving a 404 error when trying to open AutoOps deployment status pages. If you experience this error, you can work around it by opening the page in a new tab.
Report: "Disruption to instance memory and disk usage on cloud.elastic.co for 7.x customers"
Last updateThis incident has been resolved.
We have deployed a fix for this issue and are monitoring to verify 7.x Elasticsearch deployments are properly reporting memory and disk usage both on cloud.elastic.co and via the deployments API.
We've identified the issue and have begun deploying the fix into several regions. We anticipate to resolve the issue in the remaining regions within the next hour.
We've determined that some customers running 7.x Elasticsearch deployments may have issues seeing Instance memory and disk usage on cloud.elastic.co and through the deployments API. We're continuing to investigate the issue.
Report: "Disruption to instance memory and disk usage on cloud.elastic.co for 7.x customers"
Last updateThis incident has been resolved.
We have deployed a fix for this issue and are monitoring to verify 7.x Elasticsearch deployments are properly reporting memory and disk usage both on cloud.elastic.co and via the deployments API.
We've identified the issue and have begun deploying the fix into several regions. We anticipate to resolve the issue in the remaining regions within the next hour.
We've determined that some customers running 7.x Elasticsearch deployments may have issues seeing Instance memory and disk usage on cloud.elastic.co and through the deployments API. We're continuing to investigate the issue.
Report: "Elastic Cloud Serverless - Anomaly detection job impact"
Last updateThis incident has been resolved.
The root cause has been identified and addressed. We are currently monitoring the system to ensure stability and performance.
We have determined that some customers on Elastic Cloud Serverless may see incorrect anomaly counts on anomaly detection jobs and some detection rules using filters may not be working correctly. We have also determined that time shifts in lens may be incorrect. Additionally, if you are using Elasticsearch directly and are using a filters aggregation, have a top level query, but have not turned on the "other_bucket", your results may be incorrect. We have identified a fix for the issue and are validating the change before deploying to Elastic Cloud Serverless.
We have determined that some customers on Elastic Cloud Serverless may see incorrect anomaly counts on anomaly detection jobs and some detection rules using filters may not be working correctly. We are continuing to investigate the issue.
Report: "Elastic Cloud Serverless - Anomaly detection job impact"
Last updateThis incident has been resolved.
The root cause has been identified and addressed. We are currently monitoring the system to ensure stability and performance.
We have determined that some customers on Elastic Cloud Serverless may see incorrect anomaly counts on anomaly detection jobs and some detection rules using filters may not be working correctly. We have also determined that time shifts in lens may be incorrect.Additionally, if you are using Elasticsearch directly and are using a filters aggregation, have a top level query, but have not turned on the "other_bucket", your results may be incorrect.We have identified a fix for the issue and are validating the change before deploying to Elastic Cloud Serverless.
We have determined that some customers on Elastic Cloud Serverless may see incorrect anomaly counts on anomaly detection jobs and some detection rules using filters may not be working correctly. We are continuing to investigate the issue.
Report: "Signup issues via GCP Marketplace"
Last updateGCP has mitigated the Procurement API outage. We are working to reconcile any state updates that were attempted during the outage window, including new customer signups. If any further outage of the API occurs, we will create a new incident.
We're continuing to work with GCP towards resolution. Further updates will be posted as information becomes available.
We're working with GCP on mitigating the incident by rolling back access controls that were applied to their Procurement API. We will provide an update in 1 hour or upon mitigation.
We are currently investigating issues with our access to GCP's Procurement API, which is blocking new customer signups via GCP Marketplace. We will provide an update in 3 hours, or sooner if appropriate.
Report: "Signup issues via GCP Marketplace"
Last updateGCP has mitigated the Procurement API outage. We are working to reconcile any state updates that were attempted during the outage window, including new customer signups. If any further outage of the API occurs, we will create a new incident.
We're continuing to work with GCP towards resolution. Further updates will be posted as information becomes available.
We're working with GCP on mitigating the incident by rolling back access controls that were applied to their Procurement API. We will provide an update in 1 hour or upon mitigation.
We are currently investigating issues with our access to GCP's Procurement API, which is blocking new customer signups via GCP Marketplace. We will provide an update in 3 hours, or sooner if appropriate.
Report: "Credit card customers receiving incorrect past due invoice emails"
Last updateWe are aware of an issue that caused some credit card customers to receive a "Past Due Invoice" email for their April 1 invoice, despite a successful payment. This email was sent in error and does not indicate that successfully paid invoices will be charged again. We have identified the root cause and are working with a vendor to understand options to prevent its recurrence. We apologize for the inconvenience.
We are currently investigating reports that some credit card customers incorrectly received "Past Due Invoice" emails . We are working to identify the root cause and will provide updates as soon as more information is available.
Report: "Availability issues with cloud.elastic.co"
Last updateWe received alerts regarding difficulty accessing the Elastic Cloud Console, cloud.elastic.co, from specific locations due to a CDN caching issue starting at 19:29 UTC. This issue has been remediated as of 19:54 UTC.
Report: "Deployment Monitoring Impacted"
Last updateThe missing data has been successfully back-filled. All our systems are fully operational and we consider this incident resolved.
We have identified the issue of the ingestion pipeline and successfully mitigated it. Deployment observability data is currently being back-filled and will be fully available again soon. We continue to monitor the situation and will confirm full service restoration once the data is successfully back-filled.
We are seeing delayed ingestion of platform metrics in our azure-australiaeast region. This may impact the observability data of our customers elasticsearch deployments. This may impact deployment scale ups and scale downs, but does not impact data ingestion or any functionality of a running elasticsearch cluster. We are investigating the root cause and will inform again in the next hour.
Report: "elastic.co website unavailable"
Last updateThe elastic.co site is now operating as expected. We have monitored it's health for the past hour and are now setting this incident to resolved.
We've deployed a fix to the elastic.co website and are now seeing optimal performance. We will continue to monitor the health and performance of the site for the next hour.
Our engineers have identified an issue impacting parts of our elastic.co webpage. We are testing and validating a fix for this issue. Intermittent impact to the elastic.co webpage and subpages may still be present while we work towards resolution. Our next update will be in 1 hour, or sooner if appropriate.
Our engineers continue to drive a resolution for the elastic.co webpage. Intermittent impact may be observed on the elastic.co webpage and subpages, where they may fail to load. Our next update will be in 1 hour, or sooner if appropriate.
Our engineers continue to drive a resolution for the elastic.co webpage. Intermittent impact may be observed on the elastic.co webpage and subpages, where they may fail to load. Our next update will be in 1 hour, or sooner if appropriate.
Our engineers continue to drive a resolution for the elastic.co webpage. Intermittent impact may be observed on the elastic.co webpage and subpages, where they may fail to load. Our next update will be in 1 hour, or sooner if appropriate.
Our engineers are working towards a resolution for the elastic.co webpage. During this time you may be unable to load the elastic.co webpage and its subpages. We will provide an update within 1 hour, or sooner if appropriate.
Our engineers are continuing to investigate the issue with elastic.co. During this time you may be unable to load the elastic.co webpage. We will provide an update within 1 hour, or sooner if appropriate.
We are currently aware of issues loading the elastic.co webpage and are investigating. We will provide an update within 1 hour or sooner if applicable.
Report: "Azure East US Project Creation"
Last updateThis incident has been resolved.
Operability has largely recovered, however we expect that there may be some intermittent issues with the ability to create new projects until the issue with the upstream cloud provider has been resolved. We will continue to monitor the issue and will issue an update in a few hours.
We have identified a root cause to the issue and are continuing to investigate. Project availability appears to be returning to normal, however an issue with the upstream cloud service provider is ongoing and may continue to affect the ability for customers to create new projects in the interim. We will continue to monitor the problem and provide and update once the issue has been mitigated.
We are currently investigating project availability in Azure East US. Customers may be experiencing difficulty creating projects. We will report back as soon as we have more information.
Report: "Ingest Scaling Issue"
Last updateThe incident has been resolved. Customer traffic should be back to normal.
We are experiencing an issue that may cause some portion of our customer traffic to be rejected. We have identified the issue and are working toward a resolution.
Report: "Trial users unable to leave organization to join existing organization"
Last updateA fix for the issue has been implemented. Users should be able to leave a trial and join another organization without issue.
We are still working on rolling out the fix. Next update will be in four hours, or sooner if new information is available.
We are still working on rolling out the fix. Next update will be in four hours, or sooner if new information is available.
We have identified the cause of the issue and we are working on a fix. Next update will be in four hours, or sooner if new information is available.
We're investigating an issue facing new users when they sign up for a trial, but are unable to leave that organization without creating a deployment or project. This leads to billable usage, which blocks the user from being able to leave that trial to join another organization.
Report: "Issue with AWS Marketplace Sign Up Process"
Last updateThe issue with Elastic AWS Marketplace integration has cleared.
AWS Marketplace Sign Ups are now unblocked. We're continuing to monitor the AWS Marketplace service health.
We are currently investigating an issue on our AWS Marketplace product page which is blocking new customers sign ups.
Report: "Connectivity instability affecting customers traversing over internet"
Last updateThe traffic issue has been mitigated, and this incident is now resolved.
We are investigating and responding to our monitoring of connectivity instabilities with Azure West-Europe. Only customers who traverse over the internet to reach Elastic Cloud in Azure Westeurope may be experiencing connectivity instability. We will continue to monitor and investigate the cause along with Azure. We will provide updates once we have new information about the incident.
We are investigating and responding to our monitoring of connectivity instabilities with Azure West-Europe. Only customers who traverse over the internet to reach Elastic Cloud in Azure Westeurope may be experiencing connectivity instability.
We are investigating and responding to our monitoring of connectivity instabilities with Azure West-Europe. Only customers who traverse over the internet to reach Elastic Cloud in Azure Westeurope may be experiencing connectivity instability. We will continue to monitor, and investigate the cause along with Azure.
Report: "AWS Marketplace Signups Unavailable"
Last updateThis issue has been resolved.
We have identified the cause of the issue and are engaged with AWS support to resolve it. We confirmed that there is no impact to accounts with existing subscriptions. We will post an update in four hours, or sooner if new information is available.
We have identified an issue where our AWS Marketplace listing has been removed. We are investigating and engaging with AWS Support. We believe there is no impact to subscribed accounts, but new customers are not able to sign up through the marketplace. We will post an update in an hour, or sooner if new information is available.
Report: "Stale Performance Metrics Data"
Last updateThis issue is resolved. All data has been ingested and we will continue to actively monitor for normal Operation over the next few hours.
We've addressed the issue. Ingestion has resumed and we have already processed the backlog of logs. We will continue to monitor ingestion and post an update in approximately 2 hours.
We are aware of an issue that may cause performance data in the deployments page of the cloud UI to be stale. We are working to address the problem and will post an update in approximately 2 hours.
Report: "System Emails Not Sending"
Last updateThis incident has been resolved.
We've identified the issue and have started processing the backlog of email. We will continue to monitor the situation until the backlog is cleared.
We are aware of an issue where system emails are not being delivered. We are investigating and will provide an update in two hours on our investigation.
Report: "Issues with creating new projects and updating existing projects"
Last updateThe services have recovered and the incident is resolved.
We are aware of issues with creating new projects and updating existing projects. We are investigating this and working towards mitigation actions.
Report: "Disruptions to usage data in us-east-1"
Last updateNo new issues observed, resolving.
All utilization data has recovered. We will continue to monitor and resolve the incident if no further issues.
We continue to investigate the disruptions. An update will be provided in 1h, or sooner.
We are investigating potential disruptions with utilization data in us-east-1. This could delay disk usage, JVM memory pressure, and billing usage information. We will post an update in 1h, or sooner.
Report: "Disruptions in ap-southeast-2 (Asia Pacific, Sydney)"
Last updateThere have been no new issues during monitoring window. Resolving.
We have mitigated the issue on our side. We have no longer seen any storage issue in the last 8 hours and are continuing to monitor the situation until we get final confirmation from AWS that their issue has been resolved.
We continue to work on the mitigation provided by AWS with well over half of the impacted deployments completed. We will provide another update in 1h, or sooner.
We continue to work on the mitigation provided by AWS with over half of the impacted deployments completed. We will provide another update in 1h, or sooner.
AWS has confirmed a degraded disk/storage issue affecting i3 instances in the ap-southeast-2a zone. We are working through a mitigation they provided. We will provide another update in 1h, or sooner.
We're working closely with AWS and have identified failures with i3 instances in ap-southeast-2a. We will provide another update in 1h, or sooner.
We are still investigating disruptions in ap-southeast-2a. Potential impact include: Search results being incomplete. Ingesting, updating, or deleting data may fail. An update will be provided in 1h, or sooner.
We are investigating disruptions in ap-southeast-2, Asia Pacific (Sydney). We will post a new update in 1h, or sooner.
Report: "Availability issues with aws-eu-north-1c (AWS Stockholm)"
Last updateAWS has confirmed that they have entirely resolved the outage.
AWS has mitigated the incident on their end. However, we will leave the incident open in Monitoring status until AWS confirms they have completely resolved the outage. AWS is posting regular updates on its public status page here: https://health.aws.amazon.com/health/status. We will post the next update when new information is available.
AWS is experiencing a network-related outage in AZ c of the AWS Stockholm (eu-north-1) region. As a result, non-HA Elastic Cloud deployments running in that zone may be unavailable. Deployments in the AWS Stockholm that are configured for HA remain available at this time. AWS is working on mitigating the incident. We will continue to monitor and provide updates as we have them.
Report: "Kibana Security Solutions Page is degraded"
Last updateThis incident has been resolved.
A fix has been deployed and we are monitoring to ensure no further disruptions.
The team has identified the root cause of the issue affecting the operation of Kibana Security Solution pages and is actively working on a fix. We expect the fix to be deployed later today. We will provide an update once the fix has been successfully deployed and verified.
The team has identified that the issue causing degraded Kibana operation when users visit security solutions UI pages is only affecting users with Kibana instances configured with less than 2GB of memory. The team is working on a fix for the issue. The next update will happen once we have more information about the fix.
We are aware of an issue that is causing degraded Kibana operation when users visit security solutions UI pages. Our engineers are investigating and working on a fix. Next Update will be within the next hour.
We are aware of an issue that is causing degraded Kibana operation when users visit security solutions UI pages. Our engineers are investigating and working on a fix
Report: "elastic.co Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and the elastic.co website is available again
We are continuing to investigate this issue. We will provide an update in 1 hour, or sooner if appropriate.
We are aware of an issue resulting in elastic.co being unavailable. We are currently working on a resolution.
Report: "Ubuntu/Debian and Fedora package repositories disruption"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results. Next update will be in 1 hour.
We have identified the root cause for the metadata change on the Debian/Ubuntu and Fedora repositories and we are in the process of reverting this change. Next update will be in 1 hour
We are investigating a problem with the metadata of our Ubuntu/Debian and Fedora repositories which may affect hosts performing unattended updates. Next update will be in one hour.
Report: "Potential delays in project creation"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the cause of this problem, and mitigating action has been taken. The situation is being monitored, and further details will be posted within 30 minutes.
We are investigating an issue where customers trying to create new Serverless projects may experience slowness and delays. These may result in a failed operation to create the project. Further information will be posted with the next 30mins.
Report: "Trial customers are unable to spin up deployments directly from the onboarding page"
Last updateAll issues related to this incident have been resolved and trial account deployment generation should now be working as expected.
We have deployed a fix for this issue and are monitoring to ensure that trial deployments created from the onboarding page are now spinning up properly.
We are aware of an issue where new trial customers attempting to create their first deployment directly from the onboarding page are unable to do so. We have identified the cause and are working to remediate this issue as quickly as possible. We will post an updated status in approximately 2 hours.
Report: "Customers Unable to Ingest Date Using AWS Firehose"
Last updateDeployment is done for AWS and still in progress for GCP and Azure, and we are not seeing any traffic from older version in these regions. Issue is fixed for new version, marking the incident as Resolved.
The fix has been deployed and tested in Staging. We are now rolling the deployment to Production, all regions, all CSPs. Next update in 2 hours.
We have identified the cause of the issue and are still working to deploy a fix. We will give another status update in 2 hours.
We have identified the cause of the issue and are still working to deploy a fix. We will give another status update in 2 hours.
We have identified the cause of the issue and are working to deploy a fix. We will give another status update in 2 hours.
We are aware of an issue where deployments may be unable to ingest data using AWS Firehose. We are actively investigating the cause of this and will post back in 2 hours or sooner if we have an update.
Report: "User management console Performance page shows incorrect values for the CPU usage"
Last updateWe have resolved this issue. Customers should no longer be seeing incorrect values for CPU usage.
In our investigations, we have identified this issue as a cosmetic error that should not impact the performance or operation of deployments. Our teams are continuing to investigate the issue. We will provide further updates once they are available.
We are continuing to investigate this issue. We are going to post an update in 2 hours.
We are continuing to investigate this issue. We are going to post an update in an hour.
The CPU Usage data reported on the Monitoring Performance section of the User management console could be incorrect, for some of the deployments. This is just a reporting error and doesn't reflect the actual CPU usage. We are investigating the issue. Meanwhile, please use Logs and metrics feature on the User management console to enable stack monitoring, if you need more accurate CPU usage data.
Report: "Azure network outage in South Central US"
Last updateAzure confirmed that they have entirely resolved the outage. We have resolved all the infrastructure issues caused by the upstream Azure outage.
We have resolved all of the infrastructure issues caused by the upstream Azure outage. However, we will leave the incident open in Monitoring status until Azure confirms they have completely resolved the outage. Azure is posting regular updates on its public status page here: https://azure.status.microsoft/en-us/status. We will post the next update when new information is available.
We continue to see periodic infrastructure issues as a result of the ongoing Azure incident. Azure is posting regular updates on its public status page as it works to resolve the problem: https://azure.status.microsoft/en-us/status. We have mitigated the significant majority of the impact for our customers but will keep our incident open until the upstream issue is fully resolved. Elastic Cloud deployments configured for HA continue to be available. Non-HA deployments may be impacted. Due to our inability to reliably provision new infrastructure in this zone, it is possible that attempts to scale up or create new deployments will not succeed. We will continue to monitor and mitigate the impact on our customers as best we can while we wait for Azure to address the root cause. We will post an update in the next four hours or sooner if new information is available.
We continue to see periodic infrastructure issues as a result of the ongoing Azure incident. We have mitigated the significant majority of the impact for our customers. Elastic Cloud deployments configured for HA continue to be available. Non-HA deployments may be impacted until Azure resolves the upstream issue. Azure is indicating that it may be several hours before the incident is fully resolved. We will continue to monitor and mitigate the impact to our customers as best we can while we wait for Azure to address the root cause. We will post an update in the next four hours, or sooner if new information is available.
We are continuing to work to remediate the infrastructure impacted by the Azure outage. We will provide an update in one hour, or sooner if we have additional information to share.
Azure has mitigated the incident on their end. We are continuing to work to restore the health of a small number of impacted deployments in the region. We will provide an update in one hour, or sooner if we have additional information to share.
Azure has mitigated the incident on their end. We are now working to restore the health of any impacted deployments in the region. We will provide an update in one hour, or sooner if we have additional information to share.
Azure is working to mitigate the outage impacting zone 2 of the South Central US region. Elastic Cloud deployments configured for HA continue to be available. Non-HA deployments may be impacted until Azure resolves the upstream issue. We will provide updates as soon as we have them.
Azure is experiencing a network-related outage in zone 2 of the South Central US region. As a result, non-HA Elastic Cloud deployments running in that zone may be unavailable. Deployments in the South Central US that are configured for HA remain available at this time. Azure is working on mitigating the incident. We will continue to monitor and provide updates as we have them.
Report: "Traffic filter application delay in AWS Frankfurt (aws-eu-central-1)"
Last updateWe investigated reports of traffic filter applications being delayed in aws-eu-central1. We identified a misbehaving service and reverted it to a known good state. We saw the backlog of traffic filter applications is processed, which resolves the incident.
Report: "Deployment creation problems in AWS region"
Last updateThis incident has been resolved.
We have identified, and fixed, a misconfiguration which caused these templates to be temporarily unavailable. Deployment creation in our AWS regions, using any template, is now possible. We are monitoring the situation.
We are currently investigating an issue with the creation of deployments in AWS using any of the following templates : - storage optimized - storage optimized dense - cpu optimized - vector search optimized (arm) Existing deployments are unaffected.
Report: "Some usage statement PDFs may be unavailable"
Last updateAll backlog usage reports have been completed, and all November usage reports have been delivered.
We're still processing the backlog of reports, but they should finish within the next 8 hours. We'll post an update when all of the reports are completed.
We're still processing the backlog of reports but they should finish sometime in the next 8 hours. We'll post an update when all of the reports are completed.
We have identified the cause of the issue and implemented a solution. Usage pdfs for the month of November are being generated. We will post another status update in 2 hours.
We've identified an issue with usage statements not properly showing in the User Console for some customers. Another update will be provided within 1h.
Report: "Ubuntu/Debian and Fedora package repositories are temporarily unavailable"
Last updateThis incident has now been resolved.
We're continuing to monitor the package repositories for 8.x.
A fix has been implemented, all package repositories should be working as expected. We'll continue to monitor the situation
The problem has been identified and we're trying to mitigate it. We expect a recovery within the next 3 hours.
We've identified that pulling 8.X packages are impacted while 7.X packages should continue to work expected. We're trying to fix the situation as soon as possible
We are aware that some of our public repositories (Ubuntu/Debian and Fedora) are temporarily unavailable and are working towards resolving this.
Report: "Synthetics monitor issues in GCP us-east4"
Last updateThis incident is resolved now.
We have identified and fixed the underlying issue. We can confirm that synthetic monitors are running fine again in the us-east4 region. We are continuing to monitor the situation.
We can confirm that synthetics monitors from the us-east4 regions are not running successfully since 6:30 UTC today. We are still investigating the underlying issue and will provide the next update by 12:30 UTC at latest.
Some customers may experience issues executing synthetics monitors in GCP us-east4 region. We're currently investigating this issue and will provide a further update by 10:30 UTC.
Report: "Issues with creating or scaling up High Storage deployments on us-west-1 AWS region"
Last updateThis incident has been resolved.
A fix has been implemented by AWS and we're monitoring the situation to confirm that scaling High Storage deployments is working as expected.
We are investigating an issue with creating or scaling up High Storage deployments across us-west-1 AWS region. We're following up with AWS on the details. Next update will be in one hour, or sooner if appropriate.
Report: "Issues with creating or scaling up High Storage deployments on multiple AWS regions"
Last updateAll related capacity issues were resolved earlier today and monitoring found no new issues.
We are still working with AWS to resolve capacity constraints in a handful of regions. Next update in 4 hours, or sooner if appropriate
We are continuing to work with AWS to mitigate the provisioning failures across all impacted regions. Next update will be in one hour, or sooner if appropriate
We are continuing to work with AWS to mitigate the provisioning failures. Next update will be in one hour, or sooner if appropriate
We have identified the root cause for the failed scale up and creation deployment operations that we were seeing and we are working with AWS to mitigate this. Next update will be in one hour, or sooner if appropriate
We are still investigating issues with creating or scaling up High Storage deployments or those with Warm or Cold data tier across multiple AWS regions. We're following up with AWS on the details. Next update will be in one hour, or sooner if appropriate
We are investigating an issue with creating or scaling up High Storage deployments across multiple AWS regions. We're following up with AWS on the details. Next update will be in one hour, or sooner if appropriate
Report: "Errors pulling 8.15.4 and 8.16.0 packages"
Last updateThis incident has been resolved.
The 8.16.0 packages are now available in https://artifacts.elastic.co/packages/8.x
8.16.0 is still unavailable in https://artifacts.elastic.co/packages/8.x. As a workaround, it can be downloaded from https://www.elastic.co/downloads/. We will provide an update once it is available on https://artifacts.elastic.co/packages/8.x
We've fixed a corrupted manifest. Elastic Cloud is no longer impacted and fully functional. Adding https://artifacts.elastic.co/packages/8.x and pulling 8.15.4 packages is also working. Pulling 8.16.0 packages are still impacted and being worked on. An update will be provided in 1 hour, or less if the situation changes.
We've identified that 8.15.4 packages are also impacted. If you have already added https://artifacts.elastic.co/packages/8.x/apt you can still pull packages other than 8.16.0 or 8.15.4, however attempts to add this as a new repository will fail. We're continuing to work on a resolution. An update will be provided in 1 hour, or less if the situation changes.
We are investigating errors when pulling 8.16.0 apt packages from artifacts.elastic.co. This also impacts Elastic Cloud customers.
Report: "Known issue with APM Data Ingestion v8.15.0"
Last updateWe have implemented restrictions in Elastic Cloud blocking the problematic upgrade paths. The small number of impacted customers have been notified via an email containing information on the mitigation steps to follow. These steps are also documented in: https://www.elastic.co/guide/en/observability/8.15/apm-known-issues.html#_upgrading_to_v8_15_x_may_cause_ingestion_to_fail https://www.elastic.co/guide/en/observability/8.15/apm-known-issues.html#_upgrading_to_v8_15_0_may_cause_apm_indices_to_lose_their_lifecycle_policy
There is a known issue where upgrading to v8.15.0 may cause APM indices to lose their lifecycle policy. It is suggested to upgrade to at least v8.15.1. Please see this article for more detail and support on how to navigate your upgrades: https://www.elastic.co/guide/en/observability/8.15/apm-known-issues.html#_upgrading_to_v8_15_0_may_cause_apm_indices_to_lose_their_lifecycle_policy There is a known issue where upgrading from v8.12.2 or below directly to any 8.15 version causes APM data ingestion to be rejected. It is suggested to first upgrade to v8.14.3 and from there upgrade to version 8.15.1 or higher. Please see this article for more detail and support on how to navigate your upgrades: https://www.elastic.co/guide/en/observability/8.15/apm-known-issues.html#_upgrading_to_v8_15_x_may_cause_ingestion_to_fail
There is a known issue where upgrading to v8.15.0 may cause APM indices to lose their lifecycle policy. It is suggested to upgrade to at least v8.15.1. Please see this article for more detail and support on how to navigate your upgrades: https://www.elastic.co/guide/en/observability/8.15/apm-known-issues.html#_upgrading_to_v8_15_0_may_cause_apm_indices_to_lose_their_lifecycle_policy There is a known issue where upgrading from v8.12.2 or below directly to any 8.15 version causes APM data ingestion to be rejected. It is suggested to first upgrade to v8.14.3 and from there upgrade to version 8.15.1 or higher.
Report: "Some public repositories are temporarily unavailable"
Last updateThis incident has been resolved. All impacted repositories have been made public, and their forks have been restored.
As previously communicated, all repositories except for Kibana have been changed back to public, and their forks have been restored. We are awaiting further input from GitHub Support for the Kibana repository but expect it may take several hours (or more) to be changed back to public. We will post updates here as we have them.
All repositories except for Kibana have been changed back to public and their forks have been restored. We are awaiting further input from GitHub Support for the Kibana repository.
All repositories except for Kibana have been changed back to public. GitHub is in the process of restoring forks for those repositories. We are awaiting further input from GitHub Support for the Kibana repository.
Our teams are executing on the restoration path for returning our impacted git repositories to a public state and are moving closer to remediation. This was caused as part of an internal change task where some of our public git repositories hosted under Elastic's GitHub organisation were marked as private. As a result, when attempting to access these repositories an error may be seen.
As part of an internal change task, some of our public git repositories hosted under Elastic's GitHub organisation were marked as private. As a result, when attempting to access these repositories an error may be seen. Our teams are working on the restoration path for returning our impacted git repositories to a public state.
As part of an internal change task, some of our public git repositories hosted under Elastic's GitHub organisation were marked as private. As a result, when attempting to access these repositories an error may be seen. Our teams are working to restore these with urgency.
We are aware that some of our public repositories are temporarily unavailable and are working towards resolving this.
Report: "Connectivity issues between gcp-me-west1 to various Azure regions"
Last updateThe connection issues we were observing have subsided and connectivity between gcp-me-west1 and Azure is back to normal. We will continue to investigate the cause of this disruption from the involved CSPs.
We have observed a disruption in connectivity between gcp-me-west1 and a number of Azure regions. We are still monitoring the situation and awaiting more information on the cause of this disruption from the involved CSPs. We will post back in 2 hours or sooner if we have more details. There are no issues between other GCP regions or AWS regions and we have not observed any impact to customer deployments.
We have observed a disruption in connectivity between gcp-me-west1 and a number of Azure regions. We're investigation the cause of this disruption and are engaging the involved CSPs. We will post back in 2 hours or sooner if we have more details. There is no issue between other GCP regions or AWS regions and we have not observed any impact to customer deployments.
We have observed a disruption in connectivity between gcp-me-west1 and a number of Azure regions. We're investigation the cause of this disruption and are engaging the involved CSPs. We will post back in 2 hours or sooner if we have more details.
Report: "Customers Unable To Process Training Payments"
Last updateA fix has been applied and the incident has been resolved
We are still working to test and deploy a fix for this issue. We will have another update in approximately 2 hours.
We are still working to test and deploy a fix for this issue. We will have another update in approximately 2 hours.
We are aware of an issue that is inhibiting customers' ability to pay for trainings from our portal. We have identified the issue and are actively working on a fix.
Report: "Microsoft Login to Elastic Cloud Not Working for All Users"
Last updateThis incident has been resolved.
We have successfully identified and mitigated the issue causing failed logins using the Microsoft Provider. Logins with Microsoft to Elastic Cloud should work fine again. If you are still experience issues, please make sure to clear your session and try again. In case you are still not able to login using your Microsoft Credentials, please reach out to Elastic Support. We are monitoring the situation and will check potentially failed login attempts.
We got reports from some users that the Microsoft login functionality for Elastic Cloud is not working. We are actively looking into it and will post an update in an hour.