Is Splunk Observability Cloud US1 Down Right Now? Check if there is a current outage ongoing.

We are continuing to investigate this issue.

A degradation in the performance of the Splunk RUM metrics processing pipeline is causing Monitoring MetricSets, Session Data, and Trace data to be delayed by more than five minutes. Charts and detectors built from Monitoring MetricSets are impacted

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Customers may experience intermittent timeout issues logging into app.us1.signalfx.com. We are investigating and will provide an update shortly. A small number of datapoints are not accepted at ingest, which may result in delays in some detectors.

Datapoint ingest is affected and we are dropping datapoints. We are investigating and will provide an update every 15 mins.

A fix has been implemented and we are monitoring the results.

Customers may experience intermittent timeout issues logging into app.us1.signalfx.com. We are investigating and will provide an update shortly. A small number of datapoints are not accepted at ingest, which may result in delays in some detectors.

Datapoint ingest is affected and we are dropping datapoints. We are investigating and will provide an update every 15 mins.

Customers may experience intermittent timeout issues logging into app.us1.signalfx.com. We are investigating and will provide an update shortly.

Datapoint ingest is affected and we are dropping datapoints. We are investigating and will provide an update every 15 mins.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating a degradation in performance while using the Splunk APM Interface. Parts of the Splunk APM Troubleshooting experience like the service map, map breakdowns, Tag Spotlight and charts within Splunk APM may be impacted. Trace data ingest is not impacted.

This incident has been resolved.

Customers using unified identity may experience intermittent failures while logging into Splunk Observability cloud web interface. Datapoint Ingest is not affected. We are investigating and will provide an update shortly.

Customers using unified identity may experience intermittent failures while logging into Splunk Observability cloud web interface. Datapoint Ingest is not affected. We are investigating and will provide an update shortly.

We are investigating a degradation in performance while using the Splunk APM Interface. Parts of the Splunk APM Troubleshooting experience like the service map, map breakdowns, Tag Spotlight and charts within Splunk APM may be impacted. Trace data ingest is not impacted.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

A degradation of the ingest path for Organization metrics data points is causing data points to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.

Cloud metrics may be delayed up up 30 minutes in time window: 2:10 - 2:37 AM PST. There is no data loss, but data might be delayed.

The issue has been identified and a fix is being implemented.

A degradation of the ingest path for Organization metrics data points is causing data points to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.

Cloud metrics may be delayed up up 30 minutes in time window: 2:10 - 2:37 AM PST. There is no data loss, but data might be delayed.

This incident has been resolved.

The system has recovered and we are continuing to monitor

We are actively implementing backend improvements to address delays in event availability. This multi-step process is expected to take several hours. Engineering teams across regions are coordinating to monitor progress and ensure full functionality is restored. Our next update will be provided by 10:00 AM UTC on May 9, 2025

We are continuing to work on the issue. Alert notifications are being sent out on time. A small percentage of the events behind those notifications are delayed being created and are not available to see in the user interface.

Efforts to resolve the issue are ongoing, and we’ll continue to share updates along the way

We're making steady progress on the fix and will keep you informed as more details emerge

We are continuing to work on a fix and will provide updates as more information becomes available

We are continuing to work on a fix for this issue

Starting at 10:30a PT some small percentage of alerts may have been delayed by up to 2-3 hours. The root cause is known and a fix is being worked on.

Starting at 10:30a PT some small percentage of alerts may have been delayed by up to 2-3 hours. The root cause is known and a fix is being worked on.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A degradation in the performance of a key backend component of Splunk APM is causing both Troubleshooting, Monitoring MetricSets and Trace Data to be delayed by more than five minutes. No data is being dropped at this time but data for the APM Troubleshooting page, Tag Spotlight experience, Trace Analyzer and other APM pages, as well as metrics created from traces and APM detectors are all delayed.

A degradation in the performance of a key backend component of Splunk APM is causing both Troubleshooting, Monitoring MetricSets and Trace Data to be delayed by more than five minutes. No data is being dropped at this time but data for the APM Troubleshooting page, Tag Spotlight experience, Trace Analyzer and other APM pages, as well as metrics created from traces and APM detectors are all delayed.

Splunk Synthetic Monitoring will update Google Chrome and Chromium to version 135.0.7049.84-1 for Browser tests on 4/22 at 8 am PST. We periodically auto-update to newer versions of Google Chrome/Chromium when available. Due to differences between browser versions, Synthetics test behavior or timings can sometimes change and may require updates to your configured steps.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're experiencing failures for searches related to Log observer. We will provide an update as soon as possible.

We're experiencing failures for searches related to Log observer. We will provide an update as soon as possible.

From 10:35 to 10:43 PT charts may have been slow or not loaded. Ingestion and processing of data was not impacted. The issue is resolved at this time.

From 10:35 to 10:43 PT charts may have been slow or not loaded. Ingestion and processing of data was not impacted. The issue is resolved at this time.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

A fix has been applied and systems are now functioning as expected.

We have identified the root cause of this issue and are working on fixing it. We will provide an update once we have applied the fix.

We are currently investigating this issue.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Notifications for Alerts by HTTP and email are being delayed. We are actively investigating the issue.

Notifications for Alerts by HTTP and email are being delayed. We are actively investigating the issue.

This incident has been resolved.

Event creation may be delayed by 2 hours for some customers, alerting notification is not impacted but events for alerts may not be available in the UI until this incident is resolved.

Event creation may be delayed by 2 hours for some customers, alerting notification is not impacted but events for alerts may not be available in the UI until this incident is resolved.

This incident has been resolved.

We have identified the issue and are starting to see services return to operational.

We are currently experiencing issues with Charts and APM and are investigating the issue.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

This incident has been resolved.

We are continuing to monitor for any further issues.

We are seeing signs of recovery and are continuing to monitor.

Splunk Synthetics runners are unavailable in the Stockholm region due to an AWS EC2 instance outage in the eu-north-1 region.

Between 11:30 AM and 3:00 PM (PST) on 2/10, customers experienced slow chart loading times and API responses. Some API requests returned errors. Data ingestion remained unaffected. We have identified and resolved the root cause.

Some customers were experiencing slow chart loading times, and some charts may not load. There may be delays when creating new detectors as well. Data ingestion and the performance of opened charts and existing detectors remained unaffected. We have identified the root cause and implemented fixes. Our team is working on a long-term solution to prevent these performance issues. This issue has occurred multiple times over the past three days: once on 2/4 for 3 minutes, three times on 2/5 totaling 14 minutes, and four times on 2/6 totaling 56 minutes. The last occurrence was 3:53-4:11 PM (PST) on 2/6.

Charts for a subset of customers may be loading slowly between 04:05 PM to 04:15 PM Pacific Time. Portion of catrs may have failed to load between same time window. Datapoint ingest was not affected. Issue has been fully resolved since.

Starting at 3:37PM PT until 5:40PM PT our systems experienced an issue in which UI objects such as charts, detectors, and dashboards may not have been indexed when created or updated in the index when updated, resulting in potentially stale results when searching UI objects. The issue has since been fully resolved, no data was lost, all indices are now up-to-date. Alerting and telemetry data were not impacted.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A degradation of the ingest path for Splunk RUM is causing session data to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.

This incident has been resolved.

We are continuing to investigate this issue.

We are experiencing delays in creation and updates to dimensions. We are actively investigating the root cause.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are experiencing delays in creation and updates to dimensions. We are actively investigating the root cause.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating an elevated rate of errors occurring while interacting with the Splunk APM API. Trace data ingest is not impacted.

This incident has been resolved.

Customers may be experiencing delays in some charts and detectors for new MTSes. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

A degradation of the ingest path for Splunk RUM is causing < 1% of session data to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.

We are currently investigating this issue.

Log Observer web application was unavailable for some customers between 3:29am PDT and 3:51am PDT. Log data ingest was not impacted. The system has since recovered and is operating as expected.

This incident has been resolved.

A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be delayed by more than five minutes. Service, endpoint, and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted. A degradation in the performance of the Splunk APM data ingestion pipeline is causing the processing and storage of raw trace data to be delayed by more than five minutes. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

Less than 1% of alerts may have experienced latency from 7:27 am PST – 8:35 am PST.

Some users may have experienced latency in chart loading starting at 8:12 am PST and ending at 8:39 am PST

We were able to resolve this issue. Ingestion was delayed only from AWS tooling due to upstream issues with the cloud provider - https://health.aws.amazon.com/health/status

We have identified the cause and are currently resolving the issue.

We are currently experiencing lag when ingesting data and are investigating the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

AWS is currently experiencing an outage. We will continue to monitor until the AWS issue is resolved. The latest information can be found here: https://health.aws.amazon.com/health/status

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A degradation in the performance of the Splunk APM trace processing pipeline is causing 5% - 10% of Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

We had degraded performance of the Splunk APM web application and UI was failing to load intermittently, between 2024-08-28 9:30AM Pacific and 2024-08-28 9:45AM. We have since recovered and APM UI is now fully available.

This incident has been resolved.

The system is recovering and we are continuing to monitor.

We are investigating a degradation of the Splunk Observability Cloud application. Data ingest is not impacted at this time. We will provide an update as soon as possible.

Splunk Synthetic Monitoring updated Google Chrome to version 125 for Browser tests on July 18 at 12:30 PM EDT. We periodically auto-update to newer versions of Google Chrome when available. Due to differences between browser versions, test behavior or timings can sometimes change and may require updates to your test steps.

The issue with Cloud Monitoring metrics has been resolved for all affected users as of Monday, 2024-07-15 10:52 US/Pacific.

We are experiencing an issue syncing cloud metrics from GCP. Some metrics from GCP could be delayed or dropped. This is due to a GCP issue. See status page here: https://status.cloud.google.com/incidents/ERzzrJqeGR2GCW51XKFv

Experiencing an issue syncing cloud metrics from GCP. Some metrics from GCP could be delayed or dropped.

Experiencing an issue syncing cloud metrics from GCP. Some metrics from GCP could be delayed.

A fix has been implemented and full functionality has been restored. Past missing events have been filled in and will be visible as expected

A fix has been implemented and we are monitoring the results. During the incident, utilizing our APIs or UIs for visualizing alerts, muting rules or session logging events may be missing some information that will be later filled in. However, all notifications should still have been generated and sent externally as expected and on-time. This includes Slack, ServiceNow and JIRA integrations as well as emails. Expected resolution by 21:30 PT

A fix has been implemented and we are monitoring the results. During the incident, utilizing our APIs or UIs for visualizing alerts, muting rules or session logging events may be missing some information that will be later filled in. However, all notifications should still have been generated and sent externally as expected and on-time. This includes Slack, ServiceNow and JIRA integrations as well as emails. Expected resolution by 21:00 PT.

A fix has been implemented and we are monitoring the results. During the incident, utilizing our APIs or UIs for visualizing alerts, muting rules or session logging events may be missing some information that will be later filled in. However, all notifications should still have been generated and sent externally as expected and on-time. This includes Slack, ServiceNow and JIRA integrations as well as emails. Expected resolution by 21:30 PT.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results.

Event alerts are delayed by 2 hours with some small number of events lost. UI and ingest are unaffected. The problem has been identified and we are working on a resolution. Updates will be provided every 15 minutes.

This incident has been resolved.

We are continuing to monitor the environment.

We are continuing to monitor the environment. Some of the write operations may be degraded while systems continue to recover.

A fix has been pushed. The interface and log-in latency has improved. We are monitoring the results.

A fix has been pushed. The interface and log-in latency has improved. We are monitoring the results.

We are continuing to develop a fix. We will issue another update shortly.

This issue has been identified and a fix is being implemented. We will issue another update shortly.

We are continuing to investigate this issue. We will issue another update shortly.

We are aware that the interface and log-ins may be slow or unresponsive. We are actively investigating the issue. We will issue another update shortly.

This incident has been resolved.

Datapoint ingest is affected and we are experiencing some delay in ingest. We have identified the issue and are working to resolve the problem. Updates will be provided every fifteen minutes.

This incident has been resolved.

We are continuing to investigate this issue.

Charts on APM Profiling for a subset of customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Charts in APM Profiling for a subset of customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

This incident has been resolved as of 15:13 PDT. Charts and alerts are now operational.

After further review, in addition to charts not loading for a subset of customers, some alerts are delayed, some alerts may be sending false positives, and some alerts may not be sent for some customers. The fix that is being rolled out will fix the alerts as well as charts not loading. We are continuing to monitor the roll out of the fix.

The issue has been identified and a fix is being implemented.

Starting at 1:50p PT, charts for a subset of customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

Following the monitoring period, no further issues have been observed and the incident is now considered as Resolved.

A fix has been implemented at 04:14 PT, the services are now recovered and we are monitoring the results.

There are also issues since 3:21a PT in the Monitoring MetricSets. A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be corrupted and represent only a portion of the traffic and activity they are meant to measure on our customer’s services, endpoints, traces and workflows.

As of 3:18a PT this morning, charts for a subset of customers may not be loading. Some detectors may not be able to start. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

Charts for all customers may not be loading from 10:44 am - 10:55 am. Alert notifications are delayed. Datapoint ingest is not affected. This issue is completely resolved now.

We ran into issues with availability of Splunk APM UI from 2:06am - 2:14 am PDT. Trace data ingest was not impacted and this incident is completely resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are investigating an intermittent issue with Splunk Observability Cloud web application. Data ingest is not impacted at this time. We will provide an update as soon as possible.

We ran into issues with availability of Splunk APM UI from 10:54am - 11:04 am PDT. Trace data ingest was not impacted and this incident is completely resolved.