Is Totango Down Right Now? Check if there is a current outage ongoing.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating an issue with Customer Data Hub Processing Delays

This incident has been resolved.

We are currently experiencing slower-than-usual processing of updates. Our team is actively investigating the issue and working on a resolution. We appreciate your patience and will provide further updates as soon as possible.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing issues with creating collections when using list attributes. Our team is actively investigating the cause. Thank you for your patience as we work to resolve this

This incident has been resolved.

HTML characters getting appended in campaign email content. This issue is currently being investigated on high priority.

This incident has been resolved.

A fix for the issues reported with bulk updating/ clearing attribute values have been implemented and currently operational. Issue with Populating campaign sender signature and URL is being actively investigated

1. Issues populating Campaign sender signature and URL 2. Issues updating/ clearing attribute values via bulk action

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

The issue has been resolved.

We are currently investigating an issue within Outlook's Touchpoint Connect. Email-To-Touchpoint without the Outlook connect should be working as expected.

This incident has been resolved.

An issue has been identified and fix has been implemented that has improved most of the performance issues. But we are monitoring the results and working towards a permanent solution.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue

This incident has been resolved. Thank you for your patience.

We are currently experiencing an issue with email sending from Totango. This issue is affecting all campaigns and other emails, and no emails are being sent at this time. Our team is actively investigating the issue, and we will provide updates as soon as we have more information.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are continuing to investigate this issue. The issue with Tasks and SuccessPlays has been resolved.

We are continuing to investigate this issue. The issue with Tasks and SuccessPlays has been resolved.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved, however some emails may not have been received by Totango today. Thank you for your patience and flexibility.

A fix has been implemented and we are monitoring the results.

The Totango staff is experiencing issues with our email inboxes. Please know that we are working to resolve the issue, however we are not receiving any emailed communication. We apologize for the inconvenience.

This incident has been resolved.

We are currently investigating an issue in the Email-to-Touchpoint feature.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

A fix was identified and the team is monitoring results.

We have identified issues related to Account assignment misalignment and are actively investigating.

On March 30, the US Cyber Defense Agency echoed a new critical backdoor vulnerability in a linux common package (XZ-utils library). CVE-2024-3094. What could be the affect? Under certain conditions, this backdoor could allow a malicious actor to break authentication, allowing the attacker to gain access to the affected system What did Totango do since the announcement? Totango scanned all our Linux instances, in order to check whether we are vulnerable and take the recommended actions to minimize the risk Current status After completing the scanning and take the relevant actions - all Totango systems are secured and not vulnerable.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to monitor for any delays in the user integration jobs which are known to be impacting some customers.

Issues with Account Integrations are being monitored and a fix is in place. We are continuing to monitor for any delays in the user updates which are known to be impacting some customers.

The issue with the account integration job has been resolved. A fix has been implemented for user integration jobs we are monitoring the results.

The issue with the account integration job has been resolved. We are currently investigating an issue with user integration jobs. We will provide additional information as soon as possible.

A fix has been implemented and we are monitoring the results.

We are currently investigating the issue. The issue only occurs with the account integration job. We will provide additional information as soon as possible.

We are investigating an issue in the data pipelines, integration jobs appear to be stuck. The issue only occurs with the account integration job. We will provide additional information as soon as possible.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

**Event Description:** On November 20, 2023, a critical issue was reported concerning the Single Sign-On \(SSO\) mechanism using Microsoft services. Customers began experiencing difficulties in logging in, with multiple complaints received by the support team at approximately 13:00 UTC. **Timeline:** * **13:00, Nov 20, 2023**: Notification received from support about multiple customer complaints regarding SSO login issues. * **13:40, Nov 20, 2023**: The root cause of the issue was identified. * **13:55, Nov 20, 2023**: Issue was resolved. **Root Cause:** The issue was traced back to the expiration of the secret keys for our SSO application in Azure. This expiration, which likely occurred around November 18th, went unnoticed as there were no alerts or notifications generated for this event. **Steps to Resolution:** * A new secret key was generated and updated in the parameter store. **Lesson Learned and Preventive Actions:** 1. **Engagement with Azure Support**: We will be reaching out to Azure support to investigate why no alerts were triggered for the key expiration. 2. **Implementation of New Alert System**: An alert has been added in New Relic \(NR\) to specifically monitor for key expiration errors. This action has already been completed.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating reports of failed login attempts.

This incident has been resolved.

Our team has identified an issue with ongoing SuccessPlays which will cause a delay in tasks creation. We are working to fix the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We have currently identified an issue with Successplays and Tasks which may be causing SP announcements to not be triggered.

This incident has been resolved. Thank you for your patience during this time.

Campaigns and SuccessPlays have been re-enabled and we are monitoring the results. Thank you for your patience.

Campaigns and SuccessPlays have been re-enabled and we are monitoring the results. Thank you for your patience.

Campaigns and SuccessPlays have been re-enabled and we are monitoring the results. Thank you for your patience.

As part of the monitoring, we are deploying another fix which demands pausing the processing of Campaigns and SuccessPlays. Thank you for your patience while we are looking into it.

We have currently identified issues with processing Campaigns and SuccessPlays. Thank you for your patience while we are looking into it.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We have currently identified issues with processing Campaigns and SuccessPlays. Thank you for your patience while we are looking into it.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue with application performance. You may experience slower than usual load times while we work to resolve this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

The issue has been identified, the issue occurred while trying to activate campaigns and a fix is being prepared.

We are currently investigating an issue with campaigns, should happen only within teams that contain a filter.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results.

We are currently investigating the degraded performance issue that is impacted EU services.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our team has identified an issue with ongoing SuccessPlays which will cause a delay in tasks creation. We are working on a fix which will be deployed shortly.

# **Event Description**  Success Plays unexpectedly fired in the US region on April 19, 14:15 UTC, due to the use of an old Redis cluster. ‌ # **Findings & Timeline** ‌ _**Reported: April 19, 14:15 UTC The Totango system monitoring detected the issue.**_  _**Identified: April 19, 15:20 UTC The root cause was identified as a DNS change from the new Redis cluster to the old one.**_ _The application team immediately stopped all SuccessPlay triggering. The SuccessPlay engine at that time already triggered a few thousands tasks and events \(as part of the task creation emails which were sent to users\)._ ‌ _The application team then created a cleanup plan, which included deleting duplicated tasks and events and reverting attribute changes._ _**Resolved:** Apr 21, 19:58 UTC_ _Duplicated task deletion is completed_ _**April 21, 20:51 UTC**_  _**Cleanup plan execution was completed in full, including events deletion and reverting attribute changes.**_ ‌ ‌ **Root Cause:** _An old Redis cluster was being used actively instead of the new one due to a DNS change, causing SuccessPlays to fire unexpectedly._ ‌ _Corrective Action_ * _Deprecate and isolate old components when a replacement is created - Completed_ * _Implement a monitoring system to alert on any manual changes performed - In Progress \(ETA: June 30th\)_ * _Enforce a single config file for all application DNSs - Planned \(ETA: August 2023\)_ * _Create a dedicated runbook to tackle similar cases in the future - In Progress \(ETA: May 5th\)_ * _Implement anomaly detection flows to identify changes in automation triggering behavior, pause it, and alert about it - In Progress \(ETA: June 1st\)_

All tasks and events have been deleted and all attributes have been reverted to their previous state unless otherwise requested.

Task deletions have completed successfully and any attributes should be reverted to their previous state unless otherwise requested.

We are continuing our work to remove tasks and have started work on reverting any attribute changes that were made.

We are working to delete any duplicated tasks created as a result of this issue.

SuccessPlays are now active and should begin to process again. We are working on making sure systems return to normal operations.

We are planning to re-enable SuccessPlays in the next 15-30 minutes. Another update will be posted following this action. As always we appreciate your patience.

We have identified the cause for SuccessPlays firing unexpectedly. We have currently stopped SuccessPlay processing on all services while we work to remediate the issues for those services which have been impacted. We will provide further updates as our fixes are implemented. Thank you for your patience.

We are currently investigating an issue regarding Plays.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented

We are currently investigating an issue related to Customer Data hub, we expect loading issues and upload returning an error.

A fix was implemented and SuccessPlays are now processing. No data loss was experienced. Thank you for your patience.

We are currently working on a fix. Thank you for your patience.

Our team is currently investigating some delays with ongoing SuccessPlay processing.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

**Event Description:** A delay in campaign delivery was noted for our US services. **Root Cause:** Our third-party email delivery service experienced a temporary outage starting at 7am GMT which caused time-based campaign deliveries to be delayed. After working with the provider, we saw a restoration of service around 5pm GMT which allowed our systems to return normal operations. **Preventive actions:** * We will be working with our provider to better understand the cause for the outage and work with them on ensuring it does not impact our clients moving forward.

The issue with the campaigns in the US has been resolved.

Our third party provider has resolved the issues with mail delivery and we are currently monitoring as the systems begin to catch up. We apologize for the inconvenience this may have caused.

We are continuing to work on a fix for a delay in the following components: 1. Sending Emails from campaigns. 2. Inviting users via Touchpoints. 3. Replying to Touchpoints via email. Please note that our third party is receiving these messages, we do not expect any data loss. Once the issue is resolved, we expect all messages to be delivered

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

Due to third party issues we are not able to send campaigns in our US site. We are working with their support to resolve this issue as soon as possible.

This incident has been resolved.

We need to perform unplanned maintenance on our tasks database which will temporarily prevent the creation of tasks as well as viewing them. We expect this should not take more than 20 minutes. Updates to follow.

**Event Description:** An outage occurred after Sunday’s deployment affecting accessibility on Totango’s Web application such as failure to view account profiles as well as integration connection errors with code 9004. The issue occurred between 08:12 and 08:47 AM UTC and was resolved at 08:55 AM UTC. **Root Cause:** Code defect with Totango’s new API throttling caused it to enter a deadlock and failed the user to access the UI API request or connection. Issue resolved by an immediate Hotfix. **Preventive actions:** * Will be adding new alerts on the test environment to identify these incidents before reaching production and prevent them from happening. * Created an internal new tool that should help capture new API errors on the test environment before deployment.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.