Blueshift

Is Blueshift Down Right Now? Check if there is a current outage ongoing.

Blueshift is currently Operational

Last checked from Blueshift's official status page

Historical record of incidents for Blueshift

Report: "Errors in campaigns graphs in the dashboard (US datacenter only)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are seeing errors with displaying campaigns graphs in various dashboard pages.

Report: "Errors in campaigns graphs in the dashboard (US datacenter only)"

Last update
Resolved

This incident has been resolved.

Monitoring

A fix has been implemented and we are monitoring the results.

Identified

The issue has been identified and a fix is being implemented.

Investigating

We are seeing errors with displaying campaigns graphs in various dashboard pages.

Report: "Latencies in One-time and Recurring Campaigns (US datacenter only)"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Latencies in One-time and Recurring Campaigns (US datacenter only)"

Last update
Resolved

This incident has been resolved.

Update

We are continuing to monitor for any further issues.

Monitoring

A fix has been implemented and we are monitoring the results.

Identified

The issue has been identified and a fix is being implemented.

Investigating

We are currently investigating this issue.

Report: "Observing elevated event latencies"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Observing backlogs in segmentation data"

Last update
resolved

This incident has been resolved.

monitoring

The system is stable now but the backlogs will take some time to get processed. Keeping this incident open until backlogs are done.

identified

The issue has been identified and a fix is being implemented.

investigating

Observing backlogs in segmentation data processing. This will manifest as delayed data visibility during segmentation.

Report: "Bee Editor Outage"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating the issue. This editor is a third-party component and we are tracking the outage on their status page: https://beefree.statuspage.io/

Report: "Elevated API Latencies on the US datacenter (api.getblueshift.com)"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Dashboard (app.getblueshift.com) not loading"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Database Cluster Latency"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Blueshift API Interruption and Latency"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

Report: "Seeing latencies and queries failing for dashboard metrics"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing delayed campaigns"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Elevated API Latencies"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing elevated API latencies"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Delays in segment triggered & one-time/recurring campaigns"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing increased API Latencies"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing login issues on our EU datacenter (app.eu.getblueshift.com)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are creating a hotfix for this issue.

investigating

We are currently seeing few failed logins on our EU datacenter.

Report: "Campaigns delays due to orchestration service outage"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Campaigns interrupted due to orchestration service outage"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing database errors in our USA datacenter (app.getblueshift.com)"

Last update
resolved

This incident has been resolved.

monitoring

We have tweaked some settings on the database that were causing the errors. We are monitoring now.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "(EU only) Emergency database reboot for app.eu.getblueshift.com"

Last update
resolved

This incident has been resolved.

monitoring

Reboot is complete. Dashboard is functioning normally now.

identified

Reboot started

investigating

We need to perform an emergency reboot of our master database. This will take about 10 minutes. Errors will be seen on the dashboard during this time. This procedure is required because we need to enable some settings on the database that are a prerequisite to our longer scheduled database migration maintenance procedure that is scheduled on Friday, 5th January 2024.

Report: "Customers API Outage for api.eu.getblueshift.com"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing errors in dashboard"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

Issue is related to reporting & graphs. Other dashboard functionality is working normally.

investigating

We are currently investigating this issue.

Report: "Delayed campaigns in our EU datacenter"

Last update
resolved

Campaigns that use Static/Dayparting/Send time optimization are delayed. This is specific to the EU datacenter. The USA datacenter is operating normally.

Report: "Few one time/recurring campaigns and segment triggered campaigns were delayed"

Last update
resolved

A small number of one time/recurring campaigns and segment triggered campaigns were delayed. Approximate duration for the delay was 1 hour. We have fixed the issue and services are operating normally now.

Report: "Observing delayed campaigns"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Intermittent api errors on the EU endpoint api.eu.getblueshift.com"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Ongoing Latencies for Blueshift API (related to AWS Outage)"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We continue to see AWS related network latencies. We are monitoring the state. Quoting from the AWS status text for more clarity. https://health.aws.amazon.com/health/status Network Connectivity Issues Sep 18 8:01 PM PDT We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We continue to see improvement to network mapping latencies but they are not at normal levels yet. Other AWS services are also starting to see recovery as network mapping latencies improve. We will continue to keep you updated until network mapping latencies have returned to normal levels.

investigating

We are currently investigating this issue. Associated AWS outage link: https://health.aws.amazon.com/health/status

Report: "Seeing intermittent errors on event api"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Blueshift dashboard not loading"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing increased latencies for internal stats service"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Internal stats services are experiencing failure"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

We have identified the cause of the outage. We are working on recovering the service

investigating

Investigating an issue with the stats service. We are working to make it functional again.

Report: "Seeing increased latencies for Blueshift API api.getblueshift.com"

Last update
resolved

This incident has been resolved.

monitoring

The associated services have been scaled appropriately and API latencies seem to be recovering.

Report: "Event ingestion pipeline is experiencing transient processing capacity issues"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating this issue.

Report: "Delays in segment triggered campaigns"

Last update
resolved

We saw some delays in segment triggered campaigns. This was identified to a product bug that was resolved by rolling out a hotfix today.

Report: "Delays in event triggered campaigns"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Issue with campaigns errors"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are investigating increased errors for segment triggered campaigns.

Report: "Issue with Event Triggered Campaigns"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Issue with dashboard not loading"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Delays and errors for customer apis"

Last update
resolved

We experienced delays and errors to customer apis for about an hour from 9:45 UTC - to - 10:45 UTC today.

Report: "Delays and errors for users with One-time/Recurring campaigns"

Last update
resolved

A critical security update was applied to our caching instances. Campaigns were paused during the actual process of the update. These were unpaused after the update was complete, however, there were some intermittent failovers of nodes in one of the larger cache clusters causing intermittent connection losses from the campaign services depending on it. As a result, few campaigns were delayed and some portion of users errored out.

Report: "Delayed Processing Campaign Stats"

Last update
resolved

Delayed stats have been processed and this issue is now resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "High Load on Database Cluster"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

Campaigns are enabled for all accounts now.

identified

We are continuing to work on a fix for this issue.

identified

All campaigns had been paused to enable us to investigate the issue. We are now unpausing campaigns for certain accounts in a phased manner.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

Currently seeing high load on our database systems. We are currently investigating the overall cause of the high load but there is no external impact.

Report: "Issue with dashboard not loading"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

Dashboard is loading now.

investigating

We are currently investigating this issue.

Report: "Delayed Campaigns"

Last update
resolved

This incident has been resolved.

monitoring

All event backlogs are cleared now.

monitoring

Events are unpaused for all accounts now. There is some backlog and it should be drained in about an hour.

monitoring

We have unpaused more events. The backlogs should clear in about 2 hours.

monitoring

All campaigns have been unpaused now. However events have been unpaused only partially - we are working on unpausing events completely.

monitoring

This is taking more time.

monitoring

Started unpausing. Will take 15 mins to get back to normal operations.

identified

Will start unpausing in another 15 mins. This is taking a bit longer.

identified

Campaigns & events will remain paused for another 30 mins. i.e. until 8 am UTC

identified

The issue has been identified and a fix is being implemented.

investigating

We need to perform unscheduled & urgent maintenance on one of our database clusters. For this we need to pause campaigns & events.

Report: "Messaging delays for campaigns"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are in recovery mode now, we should have everything unpaused in another 30 mins.

identified

We are pausing all campaigns to give the issue time to resolve. ETA is about 2 hrs from now.

investigating

We are currently investigating this issue.

Report: "Event processing delayed"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "event processing delayed"

Last update
resolved

This incident has been resolved.

monitoring

Backlogs are continuing to recover and we are monitoring the situation as it is improving.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Event processing delayed"

Last update
resolved

The system has stabilized, and the issue has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Seeing increased latencies in event triggered campaigns"

Last update
resolved

The latencies have been stabilized, event-processing, and campaigns are functional again. We are looking at additional long-term strategies to mitigate the risk of related latencies.

monitoring

We have slowly ramped traffic back up and are monitoring the fix, latencies have reduced and stabilized.

identified

We have identified the latency issue and are working to recover our systems.

investigating

We have paused event-triggered and segment-triggered campaigns

Report: "Messaging delays for campaigns"

Last update
resolved

Delays have been resolved, working on a long-term mitigation strategy.

monitoring

We are monitoring the short term solution while working on our long term mitigation strategy.

identified

We have identified the root cause and have developed a short term and long term solution plan.

investigating

We are continuing to investigate this issue.

investigating

We are continuing to investigate this issue.

investigating

We have noticed some delays for campaign messaging, and are investigating the root cause.