Historical record of incidents for Splunk Observability Cloud EU0
Report: "Degraded performance accessing the Splunk APM Interface"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating a degradation in performance while using the Splunk APM Interface. Parts of the Splunk APM Troubleshooting experience like the service map, map breakdowns, Tag Spotlight and charts within Splunk APM may be impacted. Trace data ingest is not impacted.
Report: "Intermittent login failures for customers using Unified identity"
Last updateThis incident has been resolved.
Customers using unified identity may experience intermittent failures while logging into Splunk Observability cloud web interface. Datapoint Ingest is not affected. We are investigating and will provide an update shortly.
Report: "Intermittent login failures for customers using Unified identity"
Last updateCustomers using unified identity may experience intermittent failures while logging into Splunk Observability cloud web interface. Datapoint Ingest is not affected. We are investigating and will provide an update shortly.
Report: "Degraded performance accessing the Splunk APM Interface"
Last updateWe are investigating a degradation in performance while using the Splunk APM Interface. Parts of the Splunk APM Troubleshooting experience like the service map, map breakdowns, Tag Spotlight and charts within Splunk APM may be impacted. Trace data ingest is not impacted.
Report: "Unable to ingest approximately 1% of metric datapoints between 3:44PT and 4:50am PT"
Last updateFrom 3:44am to 4:50am PT an issue prevented approximately 1% of metric datapoints from being ingested into Splunk Observability Cloud, this has since been resolved.
Report: "Unable to ingest approximately 1% of metric datapoints between 3:44PT and 4:50am PT"
Last updateFrom 3:44am to 4:50am PT an issue prevented approximately 1% of metric datapoints from being ingested into Splunk Observability Cloud, this has since been resolved.
Report: "Splunk APM Monitoring, Troubleshooting MetricSets and Trace Data are Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A degradation in the performance of a key backend component of Splunk APM is causing both Troubleshooting, Monitoring MetricSets and Trace Data to be delayed by more than five minutes. No data is being dropped at this time but data for the APM Troubleshooting page, Tag Spotlight experience, Trace Analyzer and other APM pages, as well as metrics created from traces and APM detectors are all delayed.
Report: "Splunk APM Monitoring, Troubleshooting MetricSets and Trace Data are Delayed"
Last updateA degradation in the performance of a key backend component of Splunk APM is causing both Troubleshooting, Monitoring MetricSets and Trace Data to be delayed by more than five minutes. No data is being dropped at this time but data for the APM Troubleshooting page, Tag Spotlight experience, Trace Analyzer and other APM pages, as well as metrics created from traces and APM detectors are all delayed.
Report: "Splunk Observability Cloud Web Interface Intermittently Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Resolving team are actively working on fix implementation
Resolving team are actively working on fix implementation
A fix is still being continuously implemented.
A fix is still being continuously implemented.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are investigating intermittent unavailability of the Splunk Observability Cloud web application. Data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Splunk Observability Cloud Web Interface Intermittently Unavailable"
Last updateWe are investigating intermittent unavailability of the Splunk Observability Cloud web application. Data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Splunk APM Trace Data Being Dropped"
Last updateA degradation of the trace data ingest path for Splunk APM is causing some trace spans to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
Report: "Splunk APM Trace Data Being Dropped"
Last updateA degradation of the trace data ingest path for Splunk APM is causing some trace spans to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
Report: "Splunk Synthetics Google Chrome Upgrade"
Last updateSplunk Synthetic Monitoring will update Google Chrome and Chromium to version 135.0.7049.84-1 for Browser tests on 4/22 at 8 am PST. We periodically auto-update to newer versions of Google Chrome/Chromium when available. Due to differences between browser versions, Synthetics test behavior or timings can sometimes change and may require updates to your configured steps.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Report: "Log Observer Connect Search Failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We're experiencing failures for searches related to Log observer. We will provide an update as soon as possible.
Report: "Log Observer Connect Search Failures"
Last updateWe're experiencing failures for searches related to Log observer. We will provide an update as soon as possible.
Report: "Unable to fetch Splunk RUM sessions"
Last updateThe issue has been resolved.
We have identified the root cause of this issue and are working on fixing it. We will provide an update once we have applied the fix.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Unable to fetch Splunk RUM sessions"
Last updateWe are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Charts not Loading for Some Customers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Charts for a subset of customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
Report: "Splunk Synthetics runners are unavailable in the Stockholm region"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are seeing signs of recovery and are continuing to monitor.
Splunk Synthetics runners are unavailable in the Stockholm region due to an AWS EC2 instance outage in the eu-north-1 region.
Report: "Lag in APM metrics availability"
Last updateThe incident is resolved and we are fully operational.
We are still actively investigating the issue.
We are continuing to investigate this issue.
We are investigating a lag in the creation and availability of APM metricsets and tag spotlights.
Report: "Lag in APM metrics availability"
Last updateWe are currently investigating this issue.
Report: "Charts and Detectors Delayed"
Last updateThis incident has been resolved.
The system has now recovered, and we are continuing to monitor.
The recovery has been progressing as expected and will continue over the next few hours. We will continue to provide updates
The recovery is ongoing at a steady pace. We will continue to monitor and provide updates.
The fixes continue to take effect and we expect the system to recover at a steady pace over the next few hours. We will continue to monitor and provide updates
Additional fixes are now deployed and starting to take effect. We will continue to monitor and provide updates
We are continuing to deploy the additional fixes and will provide further updates as they start taking effect
Additional fixes are now implemented and are in the process of being deployed. We will continue to provide updates.
We are in the process of implementing additional fixes and will continue to provide updates
The fix has been deployed. We are monitoring it and will continue providing updates.
We are continuing to deploy the fix and will provide further updates as it starts taking effect.
The fix has been implemented and is in the process of being deployed.
We are continuing to make progress on a fix for this issue.
Customers may be experiencing delays in charts and detectors that rely on property and tag updates on metric time series. Datapoint ingest is not affected. We have identified the issue and are actively working on implementing the fix.
Report: "Issue ingesting metrics"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Elevated Error Rate from the Splunk APM and RUM API"
Last updateWe are investigating an elevated rate of errors occurring while interacting with the Splunk APM API. Trace data ingest is not impacted.
Report: "Splunk Synthetic Monitoring updated Google Chrome to version 125"
Last updateSplunk Synthetic Monitoring updated Google Chrome to version 125 for Browser tests on July 18 at 12:30 PM EDT. We periodically auto-update to newer versions of Google Chrome when available. Due to differences between browser versions, test behavior or timings can sometimes change and may require updates to your test steps.
Report: "Splunk Log Observer Interface Unavailable"
Last updateThis incident has been resolved.
Splunk Log Observer Interface is operational.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate the major outage of Splunk Log Observer web application.
We are continuing to investigate the major outage of Splunk Log Observer web application.
We are continuing to investigate the major outage of Splunk Log Observer web application. Correction from previous message: Log data ingest is not impacted at this time.
We are continuing to investigate the major outage of log data ingest.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are investigating a major outage of the Splunk Log Observer web application. Log data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Delayed metrics from GCP"
Last updateThe issue with Cloud Monitoring metrics has been resolved for all affected users as of Monday, 2024-07-15 10:52 US/Pacific.
We are experiencing an issue syncing cloud metrics from GCP. Some metrics from GCP could be delayed or dropped. This is due to a GCP issue. See status page here: https://status.cloud.google.com/incidents/ERzzrJqeGR2GCW51XKFv
Experiencing an issue syncing cloud metrics from GCP. Some metrics from GCP could be delayed or dropped.
Experiencing an issue syncing cloud metrics from GCP. Some metrics from GCP could be delayed.
Report: "Datapoints Being Dropped"
Last updateThis incident has been resolved.
Datapoint ingest was affected and we are dropped a minor number of datapoints from 2:55PM GMT until 3:09PM GMT. We have rectified the issue and are monitoring the situation.
Report: "Splunk RUM Interface Unavailable"
Last updateSplunk RUM web application was unavailable between 12:37am - 12:57am UTC. Session data ingest was not impacted.
Report: "Splunk RUM Interface Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are investigating a major outage of the Splunk RUM web application. Session data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Splunk RUM Trace data Being Dropped"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
A degradation of the ingest path for Splunk RUM is causing Session data to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
Report: "Splunk Observability Cloud Web Interface Unavailable"
Last updateWe ran into issues with availability of Splunk Observability Cloud UI from 3:36am - 4:40 am PDT. Ingest was not impacted and this incident is completely resolved.
Report: "Splunk Observability Cloud Web Interface Unavailable"
Last updateWe ran into issues with availability of Splunk Observability Cloud UI from 11:01pm - 11:33 pm PDT. Ingest was not impacted and this incident is completely resolved.
Report: "MetricTimeSeries Creation Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Customers may experience delays in seeing newly created metric time series. Datapoint ingest is not affected for existing time series. We are investigating and will provide an update shortly.
Report: "Splunk Observability Synthetic Monitoring Results Processing delayed"
Last updateThis incident has been resolved.
A fix has ben implemented and we are monitoring the results.
We are investigating delays in processing Splunk Observability Synthetic Monitoring test results. Tests are still running, but metrics may be delayed until the issue is resolved.
Report: "Charts Loading Slowly"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Charts for a subset of customers may be loading slowly. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
Report: "Splunk Log Observer Interface Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating a major outage of the Splunk Log Observer web application. Log data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Splunk APM Exemplar Search errors out"
Last updateWe are experiencing a degradation in the availability of trace exemplars. When clicking on charts to search for trace exemplars, results may not be available. It may not be possible to view individual traces.
Report: "Splunk Observability Cloud Web Interface Unavailable"
Last updateThe Splunk Observability Cloud web application was unavailable between 21:17 UTC and 21:37 UTC. Data ingest and alerting was not impacted.
Report: "Notifications are delayed"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Alerts and notifications are delayed
Report: "Configuration analysis of Splunk APM MetricSets was failing"
Last updateA bug was preventing the creation of new MetricSets configurations or disabling/modifying existing MetricSets configurations. The computation of existing MetricSets was not affected.
Report: "Alerts being sent for some detector rules previously disabled via API"
Last updateThis incident has been resolved.
We have identified a bug that is causing alerts to be sent for detector rules that have been previously disabled via the API. This issue does not impact detectors that have rules disabled via the UI. Starting January 18, some detectors that had rules disabled via the API (such as through Terraform) continued to send alerts. We are currently rolling out a fix to production systems and expect to resolve the issue within the next 3 hours.
Report: "Alert Modal not displaying plot lines"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We’re investigating an issue with the Alert Modal chart not displaying plots. Only charts within Alert Modals are impacted. Alerting, notification, and general charting are not impacted.
Report: "Synthetics - Degraded processing rate with run count."
Last updateCertain scheduled synthetics runs may not have been processed during this time. This has now been resolved and the system should be back to normal.
We are currently experiencing reduced run counts and are working to remediate the issue.
Report: "Splunk RUM Sessions Are Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
A degradation in the performance of the Splunk RUM trace data ingestion pipeline is causing the processing and storage of sessions to be delayed by more than five minutes starting approximately at 14:22 UTC. No data is being lost at this time, but the most recent data may not be available.
Report: "Splunk APM Trace Data Being Dropped"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
A degradation of the trace data ingest path for Splunk APM is causing trace spans to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
Report: "Charts not Loading for Some Customers"
Last updateCharts for a subset of customers may not have been loading between 7:14AM and 8:10AM UTC / 12:14AM and 1:10AM PDT. Datapoint ingest was not affected. The issue has been resolved and systems are now operational.
Report: "Splunk Enterprise Cloud customers not able to sign up for Splunk Observability Cloud"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Customers using Splunk Enterprise Cloud SSO are currently not able to sign up for Splunk Observability Cloud. Splunk Cloud SSO customers who are already signed up, are facing intermittent issues accessing Log Observer Connect. The problem has been identified and a fix is in progress
Report: "Datapoints Being Dropped"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Datapoint ingest is affected and we are dropping datapoints. We are investigating and will provide an update every 15 mins.
Report: "Datapoints Being Dropped"
Last updateFully resolved as of 1:53pm PT.
Datapoint ingest is affected and we are returning high error rates which can lead to dropped datapoints. We are actively remediating the issue and will provide an update every 15 mins.
Report: "Splunk Log Observer: Logs Are Delayed"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
A degradation in the performance of the Splunk Log Observer data ingestion pipeline is causing the processing and storage of logs to be delayed by more than five minutes. No data is being lost at this time, but the most recent data may not be available. Search is unavailable at this time.
Report: "Datapoints Being Dropped"
Last updateThe incident was fully resolved as of 10:45am PT.
Datapoint ingest is affected and we are returning high error rates which can lead to dropped datapoints. We are actively remediating the issue and will provide an update every 15 mins.
Report: "Splunk Log Observer Interface Unavailable"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are investigating a major outage of the Splunk Log Observer web application. Log data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Splunk Observability Cloud Login Not Working Intermittently"
Last updateThe fix is stable and the outage is fully resolved.
A fix has been implemented for this issue and we are monitoring the application. The ability to login to the Splunk Web Application as well as searching for business objects should be working again.
The issue has been identified and a fix is being implemented.
We are continuing to investigate the outage in the Splunk Web Application. All O11y Products are affected with intermittent issues.
We are continuing to investigate the issue related to logging into the Splunk O11y Cloud web application. Some customer may also experience intermittent issues searching for business objects like detectors, alerts, users, events, and more.
We are investigating a major outage of the Splunk Observability Cloud web application. The ability to login to the application is intermittently failing. We will provide an update as soon as possible.
Report: "Datapoints Were Dropped"
Last updateDatapoint ingest was dropping up to 5% of data from 2:45 to 2:55 pm PT. This issue is now resolved.
Report: "Charts not Loading"
Last updateThe incident has been resolved.
a fix has been implemented and we are monitoring the results.
We continue to work on remediating the issue.
We have identified a root cause
We have identified the issue.
Charts for all customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
Report: "Splunk Log Observer Charts Interface Unavailable"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented
We are continuing to investigate this issue.
We are investigating a partial outage of the Splunk Log Observer charts. Log data ingest is not impacted at this time. We will provide an update as soon as possible.
Report: "Splunk Synthetics runs missing/delayed in the Lagos region"
Last updateThis incident has been resolved.
Splunk Synthetics runs are now functioning normally in the Lagos region.
The issue has been identified and we are beginning to see signs of recovery. We will continue monitoring progress.
We are currently investigating an issue wherein Splunk Synthetics runs are missing or delayed in the AWS Lagos region. Other regions are functioning normally.
Report: "Unable to create/manage Splunk LO Connect connections"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are investigating an issue wherein users are unable to create and manage Splunk LO Connect connections. There is no impact on functioning of existing LO Connect connections.