Is Splunk On-Call Down Right Now? Check if there is a current outage ongoing.

This incident is now resolved. The Splunk On-Call web client and mobile applications are fully operational. We apologize for the inconvenience. Please reach out to the Splunk On-Call support team with any questions. https://help.splunk.com/en/splunk-observability-cloud/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

We continue to monitor the situation. We are still experiencing high alert lag. The Web Client and Mobile Apps are now accessible. Thanks for your patience while we continue to work this issue to resolution.

Update: A fix has been implemented and we are monitoring the issue. The On-Call timeline is still experiencing loading issues. We are seeing high alert-lag across the platform.

We've identified this issue related to a deploy inadvertently impacting our databases and are working to stabilize. The Web Client is coming back on line now. Thank you for your patience while we coming back on line.

The Splunk On-Call portal is currently unavailable via web and mobile. We are actively working to resolve this issue as quickly as possible. If you have any immediate questions or concerns, our support team is standing by to respond: 1 (855) 775-8657 https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

This incident is now resolved. The Splunk On-Call web client and mobile applications are fully operational. We apologize for the inconvenience. Please reach out to the Splunk On-Call support team with any questions. https://help.splunk.com/en/splunk-observability-cloud/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

We continue to monitor the situation. We are still experiencing high alert lag. The Web Client and Mobile Apps are now accessible.Thanks for your patience while we continue to work this issue to resolution.

Update: A fix has been implemented and we are monitoring the issue. The On-Call timeline is still experiencing loading issues. We are seeing high alert-lag across the platform.

We've identified this issue related to a deploy inadvertently impacting our databases and are working to stabilize. The Web Client is coming back on line now. Thank you for your patience while we coming back on line.

The Splunk On-Call portal is currently unavailable via web and mobile. We are actively working to resolve this issue as quickly as possible.If you have any immediate questions or concerns, our support team is standing by to respond: 1 (855) 775-8657https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

This incident is now resolved. Alert Ingestion - Inbound email is now fully operational. We apologize for this inconvenience. If you continue to experience issues with your inbound email alerts, please reach out to On-Call Support as soon as possible! https://docs.splunk.com/observability/en/sp-oncall/admin/get-started/spoc-support.html

A fix has been implemented and we are monitoring the results. Stand by for confirmation

We have identified the source of this issue and are actively working to deploy a fix.

Emails sent to the Generic Email Endpoint Integration are not being ingested by On-Call. These emails are not creating incidents or other alerts as expected. All other Integrations and On-Call incident operations are fully operational at this time. We are currently investigating this issue. Please monitor this Statuspage for the most up to date information, we will post another update shortly

Emails sent to the Generic Email Endpoint Integration are not being ingested by On-Call. These emails are not creating incidents or other alerts as expected. All other Integrations and On-Call incident operations are fully operational at this time.We are currently investigating this issue. Please monitor this Statuspage for the most up to date information, we will post another update shortly

This incident has been resolved. Single Sign On (SSO) is fully operational. We apologize for this inconvenience. If you continue to experience issues with your SSO connections, please reach out to On-Call Support as soon as possible! https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

A fix has been implemented and we are monitoring the solution. SSO is back to an operational state.

The issue has been identified and a fix is being implemented.

Please bear with us while we investigate this issue—impacted components Single Sign On (SSO) Updates to follow. We apologize for this inconvenience. Please reach out to On-Call Support with any questions. https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

The issue has been identified and resolved; apologies for any inconvenience! Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We are investigating reports of issues with our MS Teams integration, specifically posts from On-Call to MS Teams. Please bear with us as we continue to look into this issue.

This incident has been resolved. Page loading in the Web client has returned to normal functionality. All systems are operational. Thank you for your continued patience as we worked this issue to resolution. Please reach out to On-Call Support with any questions. https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Our recent fix has resolved much of the issue, but we will remain in a monitoring state for more testing. If you are experiencing loading issues when navigating to the Teams tab, refreshing the page from your browser should alleviate the issue. Again, we apologize for this inconvenience. If you continue to experience issues with page loading in the web client, don't hesitate to get in touch with On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

A fix has been implemented and we are monitoring the results. We apologize for this inconvenience. If you continue to experience issues with page loading in the web client, don't hesitate to get in touch with On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/ Updates to follow.

Please bear with us while we investigate this issue—impacted pages: Users, Settings, Teams. Updates to follow. We apologize for this inconvenience. Please reach out to On-Call Support with any questions. https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

This incident has been resolved. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

The issue has been identified as upstream from the On-Call platform but impacting our CPU utilization. Mitigation is underway and we are now monitoring the results. Our engineering teams continue to work on this issue with urgency. We will provide updates as soon as we learn more information. Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Stay tuned for further updates as we work towards a complete resolution. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Impacted Functionality: -Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality. -UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API -UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics. Please be advised that our engineering teams are working urgently on this issue. We will provide updates as soon as we learn more information. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We continue to investigate this issue. Please note that only a subset of instances are affected by this issue and no disruption to incident creation and notifications out to on-call end-users have been detected. Apologies for any inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We are continuing to investigate this issue.

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Impacted Functionality: -Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality. -UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API -UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics. Please be advised that our engineering teams are working on this issue with urgency. We will provide updates as soon as we learn more information. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

This incident has been resolved. Timeline events are no longer missing, all systems fully operational.

A fix has been implemented and we are monitoring the results

Splunk On-Call is currently investigating an issue resulting in missing timeline events for approximately 8% of our customer base. These events display ingested alerts and incident actions such as paging and acknowledgments. The actual paging functionality is unaffected. As a workaround, please leverage the Splunk On-Call Response Metrics, On-Call Review, and Incident Frequency reports to obtain your alert event information. We apologize for this inconvenience. Please reach out to On-Call Support with any questions. https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Our provider is no longer experiencing SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. This incident has been resolved. All systems are operational. If you're still experiencing issues with SMS please reach out to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/ We apologize for this inconvenience.

The Splunk On-Call notification provider continues to experience SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. Our provider is working with the carrier partner to resolve the issue. Splunk On-Call will continue to monitor and provide updates as soon as more information becomes available. As mentioned, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid single points of notification failure. Documentation is found here: Paging Policies: https://help.victorops.com/knowledge-base/paging-policy-setup/ Notification Types: https://help.victorops.com/knowledge-base/notification-types/ We apologize for this inconvenience. Please reach out to On-Call Support with any questions! https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Update: This issue affects shortcode SMS to the domestic US (AMER). Our provider continues to work with carrier partners to resolve this issue. You may track the progress toward the resolution of this issue here: https://status.twilio.com/incidents/p6jvzrw06911

The issue has been identified. Our provider is working with carrier partners to resolve the issue. In the meantime, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid a single point of failure. Documentation is found here: Paging Policies: https://help.victorops.com/knowledge-base/paging-policy-setup/ Notification Types: https://help.victorops.com/knowledge-base/notification-types/ We apologize for this inconvenience. Please reach out to On-Call Support with any questions! https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

The issue has been identified and resolved! Apologies for any inconvenience! Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We are continuing to investigate this issue.

We are currently investigating a problem regarding the performance of our MS Team integration. Please bear with us while we troubleshoot this issue — updates to follow.

This incident has been resolved. Reporting is fully operational. We apologize again for any inconvenience. Please direct any questions regarding this incident to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

UPDATE: Reporting back to Operational status. We continue to monitor our Reporting data and will be keeping this incident open for the time being. Thanks again for your patience while we work to completely resolve this incident.

UPDATE: Remediation complete; watching the latency decrease. Thanks again for your patience while we work this issue to resolution.

Apologies for the Reporting inconvenience! Current Status: - Data is up to date as of 6:00 AM (UTC) - No reporting data lost - Data replication is in progress and will be 100% available once complete Please direct any questions to Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We have identified the issue and are working towards a resolution.

We are currently experiencing lag in our report tooling. New Incidents will not show-up in any reports until the incident is resolved.

This incident has been resolved. All systems are operational. If you're still experiencing an issue with loading large lists of on-call users in the Incident tab under "Team: All Teams" please refresh the page and give it a few seconds. If the problem continues, please reach out to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

The issue has been identified and a fix is being implemented. Please stay tuned for further updates.

We are continuing to investigate this issue.

Splunk On-Call is currently experiencing an isolated disruption to the Incident Page for large instances. We are actively working to resolve this issue as quickly as possible. Please use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, you may contact our support team as outlined here: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/ Thank you for your patience while we work this issue to ground.

This incident has been resolved. All On-Call systems are operational. Thank you for your patience while we completed our testing. For questions regarding this incident, please contact Splunk On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We are still monitoring the status of this incident. Testing continues. All systems are operational.

We continue to monitor the issue but have moved our iOS SSO Mobile login back to operational status. Thanks again for your patience while we continue to work the root cause of this issue to ground.

We are continuing to monitor for any further issues.

We've identified the issue and have implemented a fix. Moving this incident to a monitoring status. iOS SSO mobile login should now be available. Updates to follow. Apologies again for any inconvenience.

We are currently investigating an issue with iOS mobile SSO log-in. For immediate access to Splunk On-Call please use the Web Client until we can identify the root cause of this issue. Again, this is only affecting iOS mobile. Thanks your your patience while we troubleshoot this issue.

Host Update - "The original issue has been successfully resolved." Accordingly, we will be resolving this incident. Please reach out Splunk On-Call Support with any questions or concerns regarding this incident! https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Host Update - "Site availability is continuing to improve. We are actively monitoring and continue to see a full recovery in progress. Sep 14, 2023 - 12:12 PDT" ... Splunk On-Call will resolve this Statuspage incident when our host gives the all clear. For all intents and purposes, our Knowledge Base is back to fully operational.

We are continuing to monitor for any further issues.

A fix has been implemented by our host and the Knowledge Base (https://help.victorops.com/) is once again available. If you are still experiencing loading issues please clear your cache and refresh the page. Thank you!

The issue has been identified by our KB host and they are working as quickly as possible towards resolution. Thanks for your patience while we/they restore functionality.

We are currently investigating an issue with our Knowledge Base hosting service. The ability to contact Support is unaffected and fully operational. We will provide updates as soon as they're available.

This incident has been resolved. Apologies for the inconvenience -- Team Incidents Dashboard is back to operational.

Apologies for the inconvenience. We are investigating a loading issue on the Incidents Tab affecting the Team Incidents Dashboard. Please refer to the People Pane in the Timeline tab for this information. Updates to follow.

Testing has been completed and this incident has been resolved. Duration of Impact: (109 min) 7:49 AM to 9:38 AM (MST) - This event was a visual impact on the Timeline in the UI. - No Alerts were dropped during this time - Detailed Alert information in the affected time range may be found via the Incident Frequency Report in our in-product reporting features. Again, apologies for this disruption and thank you for your patience while we worked the issue to the ground. Please reach out to Splunk On-Call Support with any questions regarding this event. https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Current Status: We remain in a "monitoring" state for this issue. - Impacted customer Timelines have returned to fully operational - System testing continues We'll provide a more detailed update here as soon as the issue is fully resolved. Apologies again for this inconvenience and thank you for your patience while we work this issue towards resolution.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results. Timelines are back to operational. Thank you for your patience while we continue to work on this issue. Please stay tuned for further updates.

We are continuing to investigate this issue.

We are currently investigating Timeline and Mobile App loading issues. Please bear with us as we investigate this issue. Please stay tuned for updates.

This incident has been resolved. All systems are operational. Apologies again for this disruption.

A fix has been implemented and we are monitoring the results.

We are working with our provider to regain access. Please bear with us while we work this issue to resolution. Our apologies for this inconvenience.

Currently experiencing issues with SSO login for all providers. Please bear with us while we work toward a resolution.

This incident has been resolved.

MS Teams Add-On Maintenance finished. Thanks for your patience while we completed this work

Update: Related to the Timeline issues experienced over the weekend, the MS Teams integration is in temporary maintenance. Please bear with us while we work towards a permanent fix.

The MS Teams integration is back to operational status. We continue to monitor Timeline performance and are working towards a resolution. Please reach out to Support via in-product Chat, the Support Portal, and our hotline (1-855) SPLUNK-S or (1-855) 775-8657 if you are experiencing issues with On-Call Timeline performance. Updates to follow.

We are continuing to work on a fix for this issue.

This issue has been identified. MS Teams is impacted. Please bear with us while we implement a fix.

We are currently investigating disconnects and loading issues to the On-Call timeline that may affect the ability to ACK and RESOLVE active incidents. If you are experiencing any issues with incident actions in the Timeline, please reach out to On-Call Support via https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/ Our Support hotline may be reached here: (1-855) SPLUNK-S or (1-855) 775-8657 Thank you for your patience while we work this issue to ground. Updates to follow.

Thanks for your patience while we worked this issue to the ground. Chat has returned to fully operational status.

Apologies for this inconvenience! Currently, our in-product Live Chat feature is experiencing issues. We are investigating. For immediate Support, please use the Splunk Support Portal (https://login.splunk.com/) or the Call Center: https://www.splunk.com/en_us/about-splunk/contact-us.html#customer-support

This incident has been resolved and the API service is operating normally. For further information on this incident please visit: For more information please visit: https://health.aws.amazon.com/health/status Apologies for this inconvenience.

Please be aware: we elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. For more information please visit: https://health.aws.amazon.com/health/status Updates to follow.

This incident has been resolved. Apologies for the inconvenience.

We are actively researching a geographically isolated issue related to Phone Notification delivery challenges to the +86 (China) country code. We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve this situation. Relevant updates will be provided to this Status Page as they arrive. As we continue to research this issue, we recommend that Splunk On-Call end-users located in China diversify their Splunk On-Call Personal Paging Policy to include Email, SMS, and Push notification delivery as alternatives to Phone. Our apologies for any inconvenience this situation may be causing you and your end-users. Updates to follow.

RESOLVED: Successful phone call routing to carriers in China (country code +86) has resumed. The changes Splunk On-Call has implemented to avoid zero-second duration calls are outlined below and in our updated Knowledge Base article, found here: https://help.victorops.com/knowledge-base/mobile-app-getting-started/#phone-notifications-to-china-country-code-86 IMPORTANT NOTE: In response to this situation, On-Call took technical measures to modify call handling for end-users within the +86/China Country Code. These measures were implemented in an effort to prevent/avoid telecommunications restrictions and zero-second duration calls associated with the +86/China Country Code: **After three attempts to notify via PHONE CALL to +86, we will cease any further attempts to contact the end user for a period of two hours. After two hours, we will make another three attempts, beginning the cycle again. . . . We sincerely appreciate your continued patience as we worked to resolve this issue.

Although we have not received formal notification back from our telecommunications provider on this front, we are currently reviewing our logs and seeing successful call delivery into the +86/China Country Code. IMPORTANT NOTE: In response to this situation, On-Call took technical measures to modify call handling for end-users within the +86/China Country Code. These measures were implemented in an effort to prevent/avoid telecommunications restrictions and zero-second duration calls associated with the +86/China Country Code: **After three attempts to notify via PHONE CALL to +86, we will cease any further attempts to contact the end user for a period of two hours. After two hours, we will make another three attempts, beginning the cycle again. . . . We sincerely appreciate your continued patience as we (continue) to work this issue to ground.

Working in collaboration with our current telecommunications provider for Phone notification service into +86/China, we're taking steps this week to significantly reduce (or) eliminate "zero-duration" calls from our inbound +86/China call volume. In doing so, we believe we can improve the quality of our inbound call volume into +86/China in specific response to their stringent requirements around inbound Phone calls. NOTE: The above effort will require code development, testing, and production deployment coordination and is not a guarantee that the current Phone notification issues to end-users in the +86/China Country Code will be resolved. Nonetheless, we're working with urgency to have these updates deployed into production later this week. In the interim, we strongly encourage all Splunk On-Call end-users to diversify their Personal Paging policies to include PUSH, EMAIL, and SMS notification methods. We will continue to update our Status Page as relevant information arrives. We sincerely apologize for the ongoing challenges.

Thank you for your patience while we continue to work with our telecommunication provider to investigate phone notification service disruption to the (+86) China region. As mentioned, this disruption is limited to Phone notifications to the (+86) region. We strongly encourage all On-Call end users in this region to diversify their personal paging policies to include Email, SMS and Push notification delivery as alternatives to Phone. Our apologies for any inconvenience this situation may be causing you.

As we continue to investigate the +86 (China) Phone notification issue with our telecommunications providers, we strongly encourage Splunk On-Call end-users to diversify their Personal Paging Policies to include Push, Email, and SMS as notification methods.

We are actively researching a geographically isolated issue related to Phone Notification delivery challenges into the +86 (China) country code. We have escalated this issue to our telecommunications providers and will continue to stay actively engaged in all efforts to resolve this situation. Relevant updates will be provided to this Status Page as they arrive. As we continue to research this issue, we recommend that Splunk On-Call end-users located in China diversify their Splunk On-Call Personal Paging Policy to include Email, SMS, and Push notification delivery as alternatives to Phone. Our apologies for any inconvenience this situation may be causing you and your end-users. Updates to follow.

This incident has been resolved.

AWS has posted that they have resolved the issue affecting Internet connectivity to the US-WEST-1 Region. Connectivity within the region was not affected by this event. The issue has been resolved and the service is operating normally.

AWS has updated their StatusPage to state that they have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. They have seen some improvement to Internet connectivity in the last few minutes but will continue to work towards full recovery.

The AWS US-WEST-2 Region is currently experiencing Internet connectivity issues and are investigating at the moment. These connectivity issues are impacting our services but we will continue to monitor their status and update ours accordingly.

We are moving our Splunk On-Call status page to a Resolved state, in relation to this issue. We'll continue to monitor this situation. If you have any immediate questions in reference to your Splunk On-Call instance, please don't hesitate to reach out to our Support team at spoc-support@splunk.com. We're here to help. Thanks for your patience in relation to this issue and apologies for any inconvenience.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring results.

The issue has been identified and a fix is in progress

We’re currently investigating an issue around email ingestion. We will be providing additional updates on our status page. Apologies for this inconvenience! If you have any immediate questions, please reach out to the Splunk On-Call Support team: spoc-support@splunk.com

This incident has been resolved. Reporting UI and Reporting API are operational. Thanks for your patience while we worked this issue to resolution.

A fix has been implemented and we are now monitoring.

The issue has been identified and a fix is being implemented. Apologies for the inconvenience and thank you for your patience.

Alert ingestion, incident creation, and notifications are unaffected and operating normally. The availability of newly occurring data in our reporting user interfaces as well as our reporting API is currently delayed beyond the expected amount. Recent events may not be available for viewing in reports or accessible via API. More precise details will be shared in subsequent updates. We are actively working to resolve this issue as quickly as possible. Use the "Subscribe to Updates" option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by to respond via email victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

We’re currently monitoring an issue that has affected less than 1% of outbound notifications (all message delivery types: Phone, SMS, Push, and Email) during an isolated period of time. A fix is in progress and should be resolved soon. We will be providing additional updates on our status page upon the completion of our internal review. Apologies for this inconvenience! If you have any immediate questions, please reach out to the Splunk On-Call Support team: victorops-support@splunk.com

**Basic Timeline & Incident Overview**: Starting at approximately 6:18 AM \(Mountain\) on the morning of 02.26.21, the _Splunk On-Call_ \(SpOC\) platform began experiencing delays in the outbound delivery of notifications. These delays were directly related to a critical service disruption associated with one of our primary telecommunications providers. Although the service disruption was primarily affecting outbound Phone and SMS notifications, the situation also \(temporarily\) delayed our delivery of Push notifications, as well. At approximately 7:40 AM \(Mountain\), actions taken by the SpOC Engineering team alleviated the Push notification delays and returned Push notification delivery to standard operational efficiency. The proper delivery of SMS notifications resumed at approximately 8:30 AM \(Mountain\). Phone notifications were queued and being delivered at a roughly 40-minute delay rate until approximately 9:30 AM \(Mountain\). The overall incident timeline was from approximately 6:18 AM \(Mountain\) - 9:30 AM \(Mountain\) on the morning of 02.26.21. After internal testing and direct communication with the telecommunications provider in-question, we deemed the incident resolved at 9:30 AM \(Mountain\) on the morning of 02.26.21. . . . As with any and all such incidents, the appropriate _Splunk On-Call_ \(SpOC\) will conduct a collaborative and intensive Post Incident Review \(PIR\) process aimed at both preventative measures and improved responsiveness toward addressing any future issues. If you have any immediate questions or concerns, please contact the _Splunk On-Call_ \(SpOC\) Support Team at: **victorops-support@splunk.com** Once again, we sincerely apologize for any unintended inconvenience this incident may have caused.

The issue has been resolved. All message delivery types (Phone, SMS, Push, and Email) and associated notifications are fully functional. We will be providing additional updates on our status page upon the completion of our internal review. If you have any immediate questions, please reach out to the Splunk On-Call Support team: victorops-support@splunk.com We sincerely apologize for any unintended inconvenience this issue may have caused.

We are continuing to investigate this issue.

SMS message delivery is improving significantly (to) fully functional. Phone messaging is still delayed. We are continuing to investigate the issue. Updates to follow.

We are continuing to investigate this issue.

As we continue to investigate this issue, we recommend leveraging PUSH Notification in your Splunk On-Call personal paging policy settings. For guidance: https://help.victorops.com/knowledge-base/paging-policy-setup/ The service disruption we are continuing to troubleshoot is primarily affecting Phone and SMS messaging. Updates to follow.

We are continuing to investigate this issue.

Please be advised that message delivery and alerting associated with the Splunk On-Call platform are significantly delayed as a result of this issue. As we continue to investigate and work this issue to ground, we recommend that you directly evaluate any monitoring solutions you may have integrated with Splunk On-Call to better determine the health and status of your associated systems, applications, and/or platform. Updates to follow.

Update: The service affecting issues that we are actively investigating appear to be associated with a service disruption being experienced by one of our communications partners, Twilio: https://status.twilio.com/ We are continuing to pursue the investigation and resolution of this issue with the highest levels of urgency. Updates to follow.

We are continuing to investigate this issue.

We are actively investigating a service affecting issue with the message delivery and alerting associated to our platform. We are approaching this issue with the highest level of urgency and the appropriate parties are actively engaged in troubleshooting the issue. Updates to follow as soon as they are available. f you have any immediate questions, please contact the Splunk On-Call Support team: victorops-support@splunk.com

Thanks for your patience! A fix has been implemented for Slack. If you continue to experience any problems with your integration please do not hesitate to reach out us: victorops-support@splunk.com.

We continue to work on this identified issue and hope to have a fix implemented soon. Please reach out to victorops-support@splunk.com with any questions regarding your Slack integration. Thanks again for your patience while we work towards a resolution.

We are continuing to investigate this issue.

This Slack re-authentication issue has been identified. Please continue to NOT REAUTHENTICATE on the integrations page. If you are experiencing an issue with your Slack integration, please reach out to victorops-support@splunk.com for immediate help. Thanks for your patience while we continue to work on this issue to resolution!

Currently experiencing an issue with Slack integration re-authorization on the integrations page. Please do not attempt to reauthorize at this moment. We are investigating this issue and will provide updates as soon as possible.

This incident has been resolved.

We are continuing to monitor for any further issues.

We are monitoring slowdowns on the Splunk On-Call billing page. The page might be slow to load for some customers. Please stay tuned for updates.

We are moving our VictorOps status page to a Resolved status in relation to this issue. We'll continue to monitor this situation via the following status page updates: https://www.cloudflarestatus.com/ If you have any immediate questions in reference to your VictorOps instance, please don't hesitate to reach out to our Support team at victorops-support@splunk.com We're here to help. Thanks for your patience in relation to this issue and sorry for any inconvenience.

We are monitoring a performance issue associated with a third-party partner that may be creating intermittent performance challenges on the VictorOps platform. For more information, please reference the following status page: https://www.cloudflarestatus.com/ If you have any immediate questions, please feel free to contact the VictorOps Support Team at victorops-support@splunk.com

This incident has been resolved. For more detailed information on this incident, please reference the July 17th incident details listed on the following third-party status page: https://www.cloudflarestatus.com/ If you have any immediate questions in relation to your VictorOps instance and this incident, please don't hesitate to reach out to the VictorOps Support team at: victorops-support@splunk.com Thank you.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are experiencing upstream internet connectivity challenges impacting customers’ ability to log into VictorOps. Please stay tuned for further information on our status.victorops.com and Twitter.

After monitoring our remediation efforts surrounding this incident, we are moving this incident to a resolved status. Brief Incident Overview: From approximately 4:30 PM (Mountain) to 5:35 PM (Mountain) on Sunday (04.19.20), we experienced service disruption challenges that may have adversely affected the overall VictorOps end-user experience. We sincerely apologize for any unintended inconvenience this issue may have caused. If you have any immediate questions surrounding this incident, please contact the VictorOps Support Team at: victorops-support@splunk.com

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are actively investigating a service disruption and will be posting updates as efficiently as possible. If you have any immediate questions, please contact the VictorOps Support Team: victorops-support@splunk.com

We have updated our Slack integration and confirmed the resolution of this issue. Our apologies for any unintended inconvenience this issue may have caused. If you have any questions or need any assistance, please don't hesitate to reach out to the VictorOps + Splunk Support Team: victorops-support@splunk.com

We are currently investigating an issue with our Slack integration. New customers attempting to configure their Slack integration may be encountering an error message during the initial set-up process. In addition, existing customers may encounter an error message if they revoke and re-enable their Slack integration. We are working to resolve the above issues as efficiently as possible. Updates to follow. If you have any immediate questions, please contact the VictorOps + Splunk Support Team: victorops-support@splunk.com

We received notification that a planned maintenance event surrounding two third-party entities (CloudFlare and AWS) on Thursday, March 12th created some unforeseen, intermittent, network-level performance challenges that adversely affected VictorOps and other affiliated providers leveraging their services. We have received confirmation that those issues have been resolved and we're subsequently moving our Status Page Performance Monitoring incident to a resolved status. We apologize for any unintended inconvenience this situation may have cause for you and your colleagues. If you have any immediate questions, please don't hesitate to contact the VictorOps Support team at support-victorops@splunk.com

We have noted intermittent Internet performance issues this afternoon and evening along the eastern seaboard of the United States. These sporadic and isolated issues have temporarily affected the ingestion and delivery performance of email and phone alerting and notifications. We are continuing to actively monitor this situation and will provide relevant updates to our Status Page if and when applicable. If you have any immediate questions, please contact our support team at victorops-support@splunk.com. We are here to help.

This incident has been resolved.

We are currently experiencing an isolated service disruption. The effected areas are currently being investigated by our Engineering Team in order to identify the issue and implement a solution. We are actively working to resolve this issue as quickly as possible and we will send an additional update within the next 20 minutes. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by via email at victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

Apologies for any inconvenience. This incident has been resolved.

A fix is being implemented and we will continue to update you as we monitor the results. To clarify, both Test and On and Off-call notifications are affected.

We are continuing to work on a fix for this issue.

We've identified the issue and are working towards a resolution. We can confirm that off-call push notifications and test phone call notifications are degraded. Thanks for your patience while we sort this out and please stay tuned for further updates.

We are currently experiencing an isolated service disruption. The effected areas are currently being investigated by our Engineering Team in order to identify the issue and implement a solution. We are actively working to resolve this issue as quickly as possible and we will send an additional update within the next 20 minutes. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by via email at victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing an isolated service disruption. The effected areas are currently being investigated by our Engineering Team in order to identify the issue and implement a solution. We are actively working to resolve this issue as quickly as possible and we will send an additional update within the next 20 minutes. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by via email at victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

VictorOps is currently experiencing an isolated disruption and we are actively working to resolve this issue as quickly as possible. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by via mail at victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

Due to modifications Atlassian made to comply with GDPR, changes were needed to be implemented to accpet these modifications. We've been actively monitoring the implemented fix and the overall issue. Based on positive technical observations, we're moving this incident from Monitoring to Resolved. Thank you for your patience during this time. Please contact the VictorOps + Splunk Support Team if you have any immediate questions: victorops-support@splunk.com

A fix has been implemented and we are monitoring the results.

We have identified the issue (JIRA configuration change) and are actively working on a resolution. Please contact the VictorOps + Splunk Support Team if you have any immediate questions: victorops-support@splunk.com Additional updates to follow.

We are actively investigating challenges with our JIRA Webhook Integration. We will continue to update our status page as additional information arrives. Apologies for any unintended inconvenience. Please contact the VictorOps + Splunk Support Team if you have any immediate questions: victorops-support@splunk.com Updates to follow.

The SMS delivery problem has been resolved by our SMS provider.

Our SMS provider has been working to resolve the delivery problem, and SMS messages are now restored. We will continue to monitor the situation.

We are continuing to work on a fix for this issue.

We are working with our SMS service provider. They are currently investigating SMS delivery failures at their end.

We are currently investigating delivery failures of SMS notifications.

We detected the issue due to the Cloudflare outage at 7:49am MDT \(UTC -6\). The VictorOps Technical Support Team was notified at 7:53am MDT. After verifying the issue and acknowledging Cloudflare’s status we created our StatusPage incident at 8am MDT. Cloudflare offered resolution at 8:11am MDT and VictorOps became fully operational. We remained in a monitoring state until 9:12am MDT, upon that time we resolved the incident. Please see the following link to the Cloudflare StatusPage for the 7/2/19 incident: [https://www.cloudflarestatus.com/incidents/9wjyx63y2xsy](https://www.cloudflarestatus.com/incidents/9wjyx63y2xsy) And please find the most recent explanation from Cloudflare here: [[https://blog.cloudflare.com/cloudflare-outage/](https://blog.cloudflare.com/cloudflare-outage/)](https://blog.cloudflare.com/cloudflare-outage) As always, we welcome and encourage you to reach out to the VictorOps Support team with any questions: [victorops-support@splunk.com](mailto:victorops-support@splunk.com)

We've been actively monitoring the implemented fix and the overall situation at the moment. Based on positive technical observations, we're moving this incident from Monitoring to Resolved. Thank you for your patience during this time. Again, we sincerely apologize for any inconvenience this issue may have caused. As a follow-up to this issue, our teams will be conducting a Post Incident Review and will follow-up with our findings on this StatusPage. If you have any immediate questions in reference to this incident, please don't hesitate to contact our Support Team with an email to victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

Portal access has been restored.

We are continuing to work on a fix for this issue.

The VictorOps portal is currently unavailable via web and mobile. We are ingesting alerts, processing incidents, and delivering notifications to users as normal. You can ack or resolve incidents by responding to phone call and SMS notifications. Cloudflare is facing network issues (which is in turn having downstream effects on those who utilize its services, including VictorOps) and is working towards a resolution. Please see https://www.cloudflarestatus.com/ We are actively working to resolve this issue as quickly as possible. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by to respond via email victorops-support@splunk.com or submit the form on our Contact Support page (https://victorops.com/contact-support/).

On Monday, from 6:01am to 6:30am MDT \(UTC -6\) VictorOps was affected by the upstream connectivity challenges that impacted large swaths of the internet. For more details on the incident please visit the [Cloudflare Status page](https://www.cloudflarestatus.com/), and [Cloudflare's explanation of what happened](https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/). Please know that at VictorOps we take any service disruption very seriously. We have conducted our own internal post-incident review about this event and have taken measures to refine and improve to our response time to incidents like the one we experienced on Monday. As always, we welcome and encourage you to reach out to the VictorOps Support team with any questions: [victorops-support@splunk.com](mailto:victorops-support@splunk.com)

This incident has been resolved. VictorOps was affected by an upstream route leak impacting CloudFlare IPs. Please refer to the CloudFlare StatusPage for further detail: https://www.cloudflarestatus.com/. Apologies for any challenges and inconveniences. VictorOps has returned to full functionality.

We are continuing to monitor for any further issues.

We have identified upstream issues that impacted the VictorOps domain. We are continuing to monitor any issues.

We are currently investigating accessibility issues for some users.

We discovered discrepancies in our Reporting database that we are continuing to take actions toward remediating. Please be advised that end-users may still encounter discrepancies in their VictorOps Reporting as we continue to work toward resolving this issue. Please look toward our Reporting component on this Status Page for updates in relation to this incident. If you have any immediate questions, please contact the VictorOps Support team at: victorops-support@splunk.com

This incident has been resolved! Thanks for your patience.

We are investigating the issue. Please stand by for updates

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The main VictorOps timeline and incident views: We are investigating intermittent performance challenges with our web-client.

This incident has been resolved.

UPDATE: T-Mobile customers must call 611 and request to remove VictorOps numbers from Scam ID and Scam Block for whitelisting. The list of numbers VictorOps will use for notification may be found in the following link: https://help.victorops.com/knowledge-base/victorops-phone-numbers/

A fix has been implemented and we are monitoring the results.

If you are a T-Mobile end-user experiencing Phone Call challenges with your VictorOps notifications, we recommend that you please reference T-Mobile's Scam ID & Scam Block instructions: https://support.t-mobile.com/docs/DOC-38784 For whitelisting purposes, the list of numbers VictorOps will use to contact you may be found in the following link: https://help.victorops.com/knowledge-base/victorops-phone-numbers/ Please know that the VictorOps Support team is also working with T-Mobile in an effort to more gracefully address this situation. Please stay tuned to our Statuspage for further updates. We apologize for any unintended inconvenience you may be experiencing in relation to this issue. If you have any immediate questions - please don't hesitate to contact the VictorOps Support Team at: victorops-support@splunk.com

The incident is now resolved. For reference, the service disruption we experienced occurred from 10:20AM - 10:39AM (Mountain) this morning. End-users may have experienced performance challenges with our Web and Mobile clients during this time period as well as alerting. We apologize for any unintended inconvenience. If you have any immediate questions in relation to this incident, please contact our Support team at: victorops-support@splunk.com Again, our apologies for any unintended inconveniences this issue may have caused.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Please be advised that we are investigating a service disruption associated to our Web and Mobile clients. We are actively investigating the issue. If you have any immediate questions, please contact our Support team at victorops-support@splunk.com We will be providing Status Page updates as they arrive.

All components have returned to operational, though we are still monitoring activity closely.

We identified a disruption in the service related to a deployment at approximately 10:22 am MT. We are now closely monitoring the issue. Please contact victorops-support@splunk.com for any questions or concerns.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

Isolated organizations are experiencing service disruption in the main portal.

Production monitoring of the new short code number (see below) has proven successful. Accordingly, we are moving this incident to resolved. Please see below for additional details: . . . In response to the isolated SMS notification challenges we were encountering on the AT&T Network, we have provisioned a new short code number. This new short code number should allow for SMS notifications to reach those AT&T end-users that were previously encountering notification challenges. For reference, the new short code number is: 842867 If you are continuing to encounter SMS notification challenges, please reach out to VictorOps Support team at: victorops-support@splunk.com Once again, we apologize for any inconvenience this issue may have caused.

In response to the isolated SMS notification challenges we were encountering on the AT&T Network, we have provisioned a new short code number. This new short code number should allow for SMS notifications to reach those AT&T end-users that were previously encountering notification challenges. For reference, the new short code number is: 842867 If you are continuing to encounter SMS notification challenges, please reach out to VictorOps Support team at: victorops-support@splunk.com We apologize for any inconvenience this issue may have caused.

We are currently investigating isolated reports of end-users on the AT&T network that are failing to receive SMS notifications via the VictorOps platform. Please know that we are actively working on resolving this issue. Please contact the VictorOps Support team with any immediate questions: victorops-support@splunk.com We apologize for any unintended inconvenience. Updates to follow.

The incident is now resolved. For reference, the service disruption we experienced occurred from 5:00 PM - 5:16 PM (Mountain) this evening. End-users may have experienced performance challenges with our Web and Mobile clients during this time period. We apologize for any unintended inconvenience. If you have any immediate questions in relation to this incident, please contact our Support team at: victorops-support@splunk.com Again, our apologies for any unintended inconveniences this issue may have caused.

We have identified the issue and performance has been restored to both Web and Mobile clients. Additional updates to follow.

Please be advised that we are investigating a service disruption associated to our Web and Mobile clients. We are actively investigating the issue. If you have any immediate questions, please contact our Support team at victorops-support@splunk.com We will be providing Status Page updates as they arrive.

This incident has been resolved.

We are continuing to monitor for any further issues.

Impact: No alerts / escalations / paging dropped. Mobile was not impacted. Approximate down time: 11:55am to 12:06pm

5 min 20 sec downtime. Webc

We are currently investigating the issue.