Historical record of incidents for Segment
Report: "Batch computations impacted"
Last updateWe've identified an incident affecting batch computations, broadcasts, backfills, CSV downloads, audience previews, and prediction jobs. Our engineering teams are actively investigating and working on a resolution.
Report: "Destination delivery delay"
Last updateWe are investigating an issue with destination delivery delay. This appears due to downstream outages in various cloud providers.
Report: "Engage event processing delay"
Last updateWe are experiencing increased event data processing delay in Engage and are actively investigating.
Report: "Engage functionality is experiencing degradation and not usable at the moment"
Last updateThis incident has been resolved.
Our engineering teams are actively investigating the root cause and working to restore full functionality. We will provide updates as more information becomes available.
We are currently investigating this issue.
Report: "Engage functionality is experiencing degradation and not usable at the moment"
Last updateOur engineering teams are actively investigating the root cause and working to restore full functionality. We will provide updates as more information becomes available.
We are currently investigating this issue.
Report: "Requesting Security Codes and Some Email/Slack notifications may intermittently be delayed"
Last updateThis incident has been resolved.
There may be periods of delay when requesting a security code (for email login or generating a token) or other alerting notifications. Notifications are eventually delivered (spikes of 15 minute delay). There is no impact on event delivery.
Report: "Requesting Security Codes and Some Email/Slack notifications may intermittently be delayed"
Last updateThere may be periods of delay when requesting a security code (for email login or generating a token) or other alerting notifications. Notifications are eventually delivered (spikes of 15 minute delay). There is no impact on event delivery.
Report: "Customer web interface issue"
Last updateThis incident has been resolved
We are fully back live.
We are currently investigating an issue with the Segment web application and public API.
Report: "Customer web interface issue"
Last updateWe are currently investigating an issue with the Segment web application and public API.
Report: "Audience UX issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating an issue with the Audience UX not loading for realtime audiences. There is no impact to data delivery.
Report: "Audience UX issue"
Last updateWe are investigating an issue with the Audience UX not loading for realtime audiences. There is no impact to data delivery.
Report: "Accuracy of scores for certain predictive traits"
Last updateAll new model scores have been successfully generated. The Predictions Overview page will automatically update as additional data becomes available for the new predictions. We appreciate your patience as the updated results are reflected.
We have deployed a fix, and the correct model scores are now being generated. Please note that the conversion history on the Predictions Overview page may still display incorrect results; we are actively working to resolve this issue.
The issue has been identified and a fix is being implemented.
We've identified a bug that is affecting the accuracy of scores for certain predictive traits. This issue may temporarily reduce the performance of those scores. Our engineering team is actively working on a fix and we will provide further updates as progress is made.
Report: "Delay in profile data processing"
Last updateThis incident has been resolved.
Processing delay is recovering and we are continuing to monitor closely.
We detected an issue that affected identity resolution and profile data update may be delayed. We have deployed a change to address the issue and are closely monitoring.
Report: "Delay in profile data processing"
Last updateWe detected an issue that affected identity resolution and profile data update may be delayed. We have deployed a change to address the issue and are closely monitoring.
Report: "Accuracy of scores for certain predictive traits"
Last updateThe issue has been identified and a fix is being implemented.
We've identified a bug that is affecting the accuracy of scores for certain predictive traits. This issue may temporarily reduce the performance of those scores. Our engineering team is actively working on a fix and we will provide further updates as progress is made.
Report: "Snowflake incident impacting Segment Warehouse integrations"
Last updateThis incident has been resolved.
We are currently monitoring potential impact due to the current Snowflake incident. https://status.snowflake.com/incidents/9mwvw50z4wnm For customers using Snowflake in the impacted regions, they may see failed warehouse syncs in Reverse ETL, Linked Audiences, Linked Events, Profile Sync, and Snowflake Destination. These will be retried in the next scheduled sync. More details available in Snowflake incident. For any questions, please reach out to the Segment support team.
Report: "Processing delays for real-time Audiences, Computed Traits, and Journeys"
Last updateBetween 6:18am ET and 8:28am ET we experienced delays of up to 1h 42m processing messages for some Engage Audiences, Computed Traits, and Journeys. No data has been lost, and the system has fully recovered. Some audience and journey conditions using time windows of 1 hour or less may not have output values during this incident.
Report: "Small portion of EU connections traffic is lagging behind."
Last updateLag has fully recovered.
We've mitigated the issue and we're monitoring the recovery.
We've identified an issue causing a small portion of EU connections traffic to lag behind up to an hour.
Report: "Delivery Failure Samples are Unavailable"
Last updateThis incident has been resolved.
As we investigate a technical issue, Delivery Samples in Event Delivery, Delivery Overview and in Failed RETL syncs are temporarily unavailable.
Report: "Error Samples in Event Delivery and Delivery Overview are temporarily unavailable"
Last updateThis incident has been resolved.
As we investigate an issue, Delivery Samples in Event Delivery and Delivery Overview are unavailable.
Report: "Warehouse Destination Delay"
Last updateThis incident has been resolved.
We have identified an issue with Warehouse Destinations that affects US workspace customers. Data delivery is delayed. A fix has been implemented and we are monitoring the result.
Report: "Delay in Profiles Identity Resolution"
Last updateThe delay in the identity resolution pipeline has been resolved.
Fix is deployed. We are seeing improvements and will continue to monitor the recovery.
We have identified the case for the issue and are working on a fix.
We have detected in an increase in delay in identity resolution and are actively investigating.
Report: "Processing delays for real-time Audiences, Computed Traits, and Journeys"
Last updateThis incident is now resolved, and Audiences, Journeys and Computed Traits are processing data as expected.
We are continuing to monitor the recovery process and estimate that we will be fully back to real-time within the next 30 minutes.
We are continuing to monitor the recovery process and estimate that we will be fully back to real-time within 4 hours.
We're currently experiencing delays of up to 7 hours processing messages for some Engage Audiences, Computed Traits, and Journeys. The issue has been identified and the recovery process has started. No data has been lost. Some audience and journey conditions using time windows of 7 hours or less may not have output values during this incident.
Report: "Generative AI features unavailable"
Last updateThis incident has been resolved.
Generative Audiences, Suggested Mappings, and Functions Writer and Analyzer are currently unavailable due to an AI vendor outage.
Report: "Engage batch computations, Predictions and Recommendations may produce incomplete results."
Last updateThe missing links have been fully recovered. Batch computations and Predictions will automatically include the updates in the subsequent runs. There has been no impact to the Profile API and real-time computations. We apologize for the inconvenience.
We identified and resolved an issue where a small percentage of profiles lost links post-merge, affecting event attribution to canonical profiles. There is no data loss. We are working to recover the missing links for accurate profile and computations updates.
The issue has been identified.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Some Unify and Engage customers are experiencing inability to create new Audiences and Traits"
Last updateThis incident has been resolved
We are currently investigating this issue.
Report: "Identifier Traits not populating as Identifiers for Custom ID Rules"
Last updateOn Nov 11, from 6:38AM UTC to Nov 13 12:45PM UTC, customers may have experienced an issue where their users were unable to get custom identifiers added to their profiles. Our engineers have identified and deployed a fix. We apologize for any inconvenience this may have caused.
Report: "Sync Failures of AWS destinations"
Last updateThe issue has now been resolved and Syncs to amazon-kinesis-firehose, amazon-personalize, amazon-lambda and amazon-eventbridge are now flowing.
We have also now recovered AWS Lambda. We will continue working through and monitoring the issue.
amazon-kinesis-firehose and amazon-eventbridge have recovered - we will continue monitoring while working to recover amazon-lambda
We have identified an issue with Syncs to amazon-kinesis-firehose, amazon-personalize, amazon-lambda and amazon-eventbridge and working to resolve.
Report: "Processing delays for real-time Audiences, Computed Traits, and Journeys"
Last updateWe have successfully mitigated the issue and are fully back to real-time processing.
We are continuing to monitor the recovery process and estimate that we will be fully back to real-time within 1 hour.
We're currently experiencing delays of up to 4 hours processing messages for 53.67% of Engage Audiences, Computed Traits, and Journeys. The issue has been identified and the recovery process has started. We estimate 38% of the affected computations will be back to real-time in the next 30 minutes, and the remainder will be back to real-time within 3 hours. No data has been lost.
Report: "Scheduled maintenance on Nov 2nd from 10 AM PST - 1 PM PST"
Last updateMaintenance complete.
We will be conducting scheduled maintenance on our core configuration management database. During this period, the App and Public API will be read-only and login functionality will not be working.
Report: "EU - Event Delivery Delayed"
Last updateWe identified and resolved an isolated event delivery delay in our EU environment that caused delays for 10% of connections traffic up to 40 minutes.
Report: "Syncs to Data Lakes are taking longer than expected."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the result.
We have identified an issue with Data Lakes that affects some customers. Syncs to Data Lakes are taking longer than expected. No data loss has occurred.
Report: "Engage Batch Computations High Error Rate"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing a high error rate with Engage batch computations.
Report: "Increased Engage List destinations error rate"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Delay in sending events to third party destinations"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating the issue.
Report: "Increased error rates of Profiles API in EU"
Last updateThis incident has been resolved.
We have identified an issue and deployed a fix. Profiles API in EU is recovering.
We detected increased error rate in Profile API for our EU region service and are actively investigating.
Report: "Data processing delay in Twilio Engage"
Last updateThis incident has been resolved.
We detected a an issue that's causing a delay in data processing in Engage. A fix has been deployed and we are monitoring closely.
Report: "Amazon Kinesis Stream delays"
Last updateDelivery delays to Amazon Kinesis of over an hour due to AWS outage.
Report: "Databricks Reverse ETL and Destination Sync delays"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating an issue with Databricks ReverseETL and Destination that is affecting all customers.
Report: "Incident impacting EU"
Last updateAll systems have recovered. We have resolved this incident
We identified the root cause and fixed the issue. Our systems are recovering. We continue to monitor the recovery
We have an incident impacting many products in the EU. We are investigating this. Multiple products such as Engage, rETL, Profile Sync are impacted in the EU. At this time the scope of impact is a delay in the impacted services.
Report: "Heap.io Partner Outage"
Last updateHeap.io confirmed that the AWS outage affecting data delivery operation is now resolved.
Our partner Heap.io is having an outage. We are monitoring the situation. For heap specific information visit https://www.heap.io/status
Report: "Engage Premier Email Template Creation and Editing Issue"
Last updateA fix has been implemented, and we are continuing to monitor the result.
The Segment engineering team has acknowledged an incident affecting customers with Engage Premier. These customers are currently unable to create new email templates or edit existing ones. When attempting to do so, an error message stating "Something went wrong" appears. We are actively working on resolving this issue and will provide updates as soon as possible. Thank you for your patience.
Report: "Segment has identified an issue where recently created Engage audiences are not correctly syncing to downstream list destinations"
Last updateThis incident has been resolved.
We have identified a root cause and are working to mitigate the issue.
Report: "Intercom Source fails to fetch "Conversations" collection"
Last updateThe connectivity to Intercom fetching "Conversations" collection is restored.
Segment is seeing timeout errors when calling Intercom "Conversations" collection API endpoint. We started the investigation by reaching out to Intercom for assistance.
Report: "Clearbit Outage"
Last updateThe connectivity to Clearbit destination is now fully recovered.
We are currently investigating the issue. The API endpoint at Clearbit is returning unexpected errors.
Report: "Destination Delivery Delays"
Last updateRecovery from delivery latency is now fully resolved.
Event delivery for streaming connections is delayed, roughly 50% of connections are currently experiencing up to 30 minutes of delivery delay.
Report: "A small Percent of destinations are experiencing lag delivering events to their integrations"
Last updateThis incident has been resolved.
We have routed new traffic away from the degraded infrastructure, but events received in the past hour may see elevated latency.
Report: "Unify and Engage pipeline delay"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing delay in processing data onto Unify profiles and Twilio Engage (US). Engineering team has identified and mitigating an underlying issue.
Report: "Realtime audiences: delays syncing to list destinations, incorrect counts"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for the delayed syncs and incorrect counts. No data has been lost.
We are continuing to work on a fix for this issue.
We have identified the issue causing the delay and currently working on fixing it.
Report: "EU traffic delayed for 2 hours"
Last updateAs of 12:00 PM UTC on 11.03.2024, our Segment Data Collection Activation -Streaming Sources service was impacted for 2 hours. Support team has received reports that EU customers were not receiving events in their sources since roughly 12:00pm UTC. A traffic surge caused this incident. Platform has finished processing through the surge. There is no data loss. The impact was a 2 hour delay for EU traffic. EU Systems are back to normal. The team continues to monitor it.
Report: "Processing delays for real-time Audiences, Computed Traits, and Journeys"
Last updateThis incident has been resolved.
This incident has been resolved.
We have identified the issue and implemented a fix. Processing has resumed and we are monitoring recovery.
We're currently experiencing delays processing Engage Audiences, Computed Traits, and Journeys created between Aug 8th 2022 and Aug 19th 2022, as well as those created between Aug 22 2022 and March 14th 2023. Our engineers are working on identifying a solution. No data has been lost.
Report: "Error accessing customer workspaces - Workspace not found"
Last updateThis Workspace access issue is now resolved, and there was no data loss as a result of this incident. Thank you again for your patience while we worked through this.
A fix has been deployed, and workspace access has now resumed. We are monitoring the situation for stability before resolving. Thank you for your patience.
We are continuing to make progress on the root cause of the errors. Workspace access is improving, with occasional errors experienced. We will continue to provide updates as more information becomes available. Thank you for your patience.
We are making progress on restoring access to workspaces and resolving the API errors. We will continue to provide updates as more information becomes available. Thank you for your patience.
We are actively working to restore access to workspaces and resolve the API errors. Please note, there is no impact on data ingestion or delivery to destinations.
Customers attempting to access their workspaces within the web app on US and EU are met with an error message. We're investigating the issue and will post updates as soon as we have further information.
Report: "Syncs delayed for Snowflake warehouse destination"
Last updateWe experienced an intermittent issue while ingesting data into Snowflake destination which caused sync delays for a small subset of customers. The issue is now resolved with no data loss reported.
Report: "Source API Keys page not loading"
Last updateThis incident has been resolved
A fix has been implemented and we are monitoring the results
The issue has been identified and a fix is being implemented
We are currently investigating an issue affecting Source API Keys page, which is not loading due to an error
Report: "Processing delays for real-time Audiences, Computed Traits, and Journeys"
Last updateThis issue is resolved. Audiences, Computed Traits, and Journeys are now all processing in real-time.
We have identified the issue and implemented a fix. Processing has resumed and we are monitoring recovery.
We're currently experiencing delays processing Engage Audiences, Computed Traits, and Journeys created before February 2021, between May 19 2022 and July 18th 2022, and between July 13 2023 and July 19 2023. Our engineers are working on identifying a solution. No data has been lost.
Report: "Segment Web Interface Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating an issue with the Segment web interface and public api that is affecting all customers. Data ingestion is not affected.
Report: "Unify and Engage pipeline delay"
Last updateWe experienced an delay processing data onto Unify profiles and into Twilio Engage (US). Engineering team has identified and mitigated an underlying issue. Data processing delay recovered after 4.5 hours.
Report: "Segment's Catalog Search is not working"
Last updateThis incident has been resolved
A fix has been implemented and we are monitoring the results
The team has identified the issue and a fix is being implemented. We will provide an update you soon.
We are currently investigating an issue affect the Segment Search functionality. Our team is working on identifying a fix and we will update this status as soon as we have more information.
Report: "Intermittence in the loading of actions mapping UI"
Last updateIssue has been identified and resolved.
The actions mapping UI is experiencing issues. We are currently investigating them.