Historical record of incidents for Blueshift
Report: "Errors in campaigns graphs in the dashboard (US datacenter only)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are seeing errors with displaying campaigns graphs in various dashboard pages.
Report: "Errors in campaigns graphs in the dashboard (US datacenter only)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are seeing errors with displaying campaigns graphs in various dashboard pages.
Report: "Latencies in One-time and Recurring Campaigns (US datacenter only)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Latencies in One-time and Recurring Campaigns (US datacenter only)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Observing elevated event latencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Observing backlogs in segmentation data"
Last updateThis incident has been resolved.
The system is stable now but the backlogs will take some time to get processed. Keeping this incident open until backlogs are done.
The issue has been identified and a fix is being implemented.
Observing backlogs in segmentation data processing. This will manifest as delayed data visibility during segmentation.
Report: "Bee Editor Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating the issue. This editor is a third-party component and we are tracking the outage on their status page: https://beefree.statuspage.io/
Report: "Elevated API Latencies on the US datacenter (api.getblueshift.com)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Dashboard (app.getblueshift.com) not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Database Cluster Latency"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Blueshift API Interruption and Latency"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Seeing latencies and queries failing for dashboard metrics"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing delayed campaigns"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Elevated API Latencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing elevated API latencies"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Delays in segment triggered & one-time/recurring campaigns"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing increased API Latencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing login issues on our EU datacenter (app.eu.getblueshift.com)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are creating a hotfix for this issue.
We are currently seeing few failed logins on our EU datacenter.
Report: "Campaigns delays due to orchestration service outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Campaigns interrupted due to orchestration service outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing database errors in our USA datacenter (app.getblueshift.com)"
Last updateThis incident has been resolved.
We have tweaked some settings on the database that were causing the errors. We are monitoring now.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "(EU only) Emergency database reboot for app.eu.getblueshift.com"
Last updateThis incident has been resolved.
Reboot is complete. Dashboard is functioning normally now.
Reboot started
We need to perform an emergency reboot of our master database. This will take about 10 minutes. Errors will be seen on the dashboard during this time. This procedure is required because we need to enable some settings on the database that are a prerequisite to our longer scheduled database migration maintenance procedure that is scheduled on Friday, 5th January 2024.
Report: "Customers API Outage for api.eu.getblueshift.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing errors in dashboard"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Issue is related to reporting & graphs. Other dashboard functionality is working normally.
We are currently investigating this issue.
Report: "Delayed campaigns in our EU datacenter"
Last updateCampaigns that use Static/Dayparting/Send time optimization are delayed. This is specific to the EU datacenter. The USA datacenter is operating normally.
Report: "Few one time/recurring campaigns and segment triggered campaigns were delayed"
Last updateA small number of one time/recurring campaigns and segment triggered campaigns were delayed. Approximate duration for the delay was 1 hour. We have fixed the issue and services are operating normally now.
Report: "Observing delayed campaigns"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Intermittent api errors on the EU endpoint api.eu.getblueshift.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Ongoing Latencies for Blueshift API (related to AWS Outage)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We continue to see AWS related network latencies. We are monitoring the state. Quoting from the AWS status text for more clarity. https://health.aws.amazon.com/health/status Network Connectivity Issues Sep 18 8:01 PM PDT We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We continue to see improvement to network mapping latencies but they are not at normal levels yet. Other AWS services are also starting to see recovery as network mapping latencies improve. We will continue to keep you updated until network mapping latencies have returned to normal levels.
We are currently investigating this issue. Associated AWS outage link: https://health.aws.amazon.com/health/status
Report: "Seeing intermittent errors on event api"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Blueshift dashboard not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing increased latencies for internal stats service"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Internal stats services are experiencing failure"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We have identified the cause of the outage. We are working on recovering the service
Investigating an issue with the stats service. We are working to make it functional again.
Report: "Seeing increased latencies for Blueshift API api.getblueshift.com"
Last updateThis incident has been resolved.
The associated services have been scaled appropriately and API latencies seem to be recovering.
Report: "Event ingestion pipeline is experiencing transient processing capacity issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Delays in segment triggered campaigns"
Last updateWe saw some delays in segment triggered campaigns. This was identified to a product bug that was resolved by rolling out a hotfix today.
Report: "Delays in event triggered campaigns"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Issue with campaigns errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are investigating increased errors for segment triggered campaigns.
Report: "Issue with Event Triggered Campaigns"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Issue with dashboard not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Delays and errors for customer apis"
Last updateWe experienced delays and errors to customer apis for about an hour from 9:45 UTC - to - 10:45 UTC today.
Report: "Delays and errors for users with One-time/Recurring campaigns"
Last updateA critical security update was applied to our caching instances. Campaigns were paused during the actual process of the update. These were unpaused after the update was complete, however, there were some intermittent failovers of nodes in one of the larger cache clusters causing intermittent connection losses from the campaign services depending on it. As a result, few campaigns were delayed and some portion of users errored out.
Report: "Delayed Processing Campaign Stats"
Last updateDelayed stats have been processed and this issue is now resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "High Load on Database Cluster"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
Campaigns are enabled for all accounts now.
We are continuing to work on a fix for this issue.
All campaigns had been paused to enable us to investigate the issue. We are now unpausing campaigns for certain accounts in a phased manner.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Currently seeing high load on our database systems. We are currently investigating the overall cause of the high load but there is no external impact.
Report: "Issue with dashboard not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Dashboard is loading now.
We are currently investigating this issue.
Report: "Delayed Campaigns"
Last updateThis incident has been resolved.
All event backlogs are cleared now.
Events are unpaused for all accounts now. There is some backlog and it should be drained in about an hour.
We have unpaused more events. The backlogs should clear in about 2 hours.
All campaigns have been unpaused now. However events have been unpaused only partially - we are working on unpausing events completely.
This is taking more time.
Started unpausing. Will take 15 mins to get back to normal operations.
Will start unpausing in another 15 mins. This is taking a bit longer.
Campaigns & events will remain paused for another 30 mins. i.e. until 8 am UTC
The issue has been identified and a fix is being implemented.
We need to perform unscheduled & urgent maintenance on one of our database clusters. For this we need to pause campaigns & events.
Report: "Messaging delays for campaigns"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are in recovery mode now, we should have everything unpaused in another 30 mins.
We are pausing all campaigns to give the issue time to resolve. ETA is about 2 hrs from now.
We are currently investigating this issue.
Report: "Event processing delayed"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "event processing delayed"
Last updateThis incident has been resolved.
Backlogs are continuing to recover and we are monitoring the situation as it is improving.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Event processing delayed"
Last updateThe system has stabilized, and the issue has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Seeing increased latencies in event triggered campaigns"
Last updateThe latencies have been stabilized, event-processing, and campaigns are functional again. We are looking at additional long-term strategies to mitigate the risk of related latencies.
We have slowly ramped traffic back up and are monitoring the fix, latencies have reduced and stabilized.
We have identified the latency issue and are working to recover our systems.
We have paused event-triggered and segment-triggered campaigns
Report: "Messaging delays for campaigns"
Last updateDelays have been resolved, working on a long-term mitigation strategy.
We are monitoring the short term solution while working on our long term mitigation strategy.
We have identified the root cause and have developed a short term and long term solution plan.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We have noticed some delays for campaign messaging, and are investigating the root cause.