Historical record of incidents for LoyaltyLion
Report: "Partial API outage and delays in data processing"
Last updateThis incident has been resolved. All our systems are operational and the data backlog has been cleared.
A fix has been implemented in our caching layer. We continue to monitor the results.
Our systems remain stable. We are still working through the data processing backlog whilst continuing to monitor.
An issue in our caching layer led to elevated error rates and response times. We have identified the issue and stabilised our systems. We're now working through the data processing backlog while monitoring.
Report: "Delays in data processing"
Last updateThis incident has been resolved.
We are currently investigating an issue with delays in data processing, this could cause delays to tier evaluations.
Report: "Job queue partial outage"
Last updateWe observed a performance degradation in one of our job queueing subsystems. As part of recovery to normal operations, we - paused processing on customer tier evaluation between 2024-03-02T13:30 UTC and 2024-03-02T19:00 UTC - paused processing on outbound webhook delivery between 2024-03-02T14:30 UTC and 2024-03-03T14:30 UTC - implemented controls on unusual traffic from Shopify Tier assignment and webhook delivery are now functioning as normal. We plan to gradually deliver the webhooks from the paused period over the next few days.
Report: "SDK outage"
Last updateOur SDK components went offline due to an issue with our CDN provider. We migrated to our backup CDN and full functionality was restored. This only impacted shopper-facing functionality (i.e. claiming rewards on-site), no API or backend requests were impacted
Report: "Increased API and platform latency"
Last updateThis incident has been resolved.
The situation has been resolved, we're continuing to monitor
We've taken steps to triage and performance is stabilising.
We're investigating increased API and platform latency. The merchant account is unaffected.
Report: "Data Processing Delays"
Last updateWe've caught up with the backlog of jobs and our platform is back to normal.
Our data processing infrastructure is running behind which is causing delays data synchronisation and events such as email sending, customer search and integrations. We've identified the issue and we're working on it.
Report: "Data Processing Delays"
Last updateThis incident has been resolved.
Our background job processing infrastructure is running behind which is causing delays in events such us email sending, customer search and integrations. We've identified the issue and we're working on it.
Report: "Data Processing Delays"
Last updateOur background job processing infrastructure is running behind which is causing delays in events such us email sending and integrations. We've identified and addressed the problem, and we're working through the backlog of jobs.
Report: "Partial API and platform outage"
Last updateThis incident has been resolved.
We're further increasing capacity to account for additional traffic over the BFCM period
We are currently investigating this issue.
Report: "Elevated platform and API response time"
Last updateA misconfigured cache caused elevated response times for our platform and API routes.
Report: "Elevated platform response time"
Last updateThis incident has been resolved.
We've identified the issue and applied a fix. We're currently monitoring to make sure our systems are fully recovered.
We are currently investigating elevated response time in our API and platform.
Report: "Elevated platform response time"
Last updateAll systems are back to normal. We'll continue to monitor and perform a root cause analysis.
Most functionality has been restored. We're still working on a few background services.
Shopper-facing functionality has been restored. We're continuing our investigation in order to restore all services.
We are currently investigating elevated response time in our API and platform.
Report: "Partial API outage"
Last updateThe situation has now been resolved
We are continuing to investigate this issue.
We're currently investigating elevated error responses from the LoyaltyLion API
Report: "API and platform outage"
Last updateWe saw an outage beginning at 23:58 BST on May 1st. Service was partially restored at 00:34 BST May 2nd and normal service levels resumed at 01:38 BST. We're still investigating the root cause of the outage and will update this incident as we learn more.
Report: "Partial API and platform outage"
Last updateBetween 17:10-17:20 API and Platform endpoints experienced a partial outage. Some API requests may have failed during this period. By 17:20 normal service was restored.
Report: "Merchant account sign in issues"
Last updateShopify merchants unable to sign in for ~90 minutes; no shopper-facing functionality was impacted
Report: "Partial API and platform outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Partial API and platform outage"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Partial API and platform outage"
Last updateWe are currently investigating this issue.
Report: "Degraded merchant account & API performance"
Last updateThis incident has been resolved.
We've rolled out our fix and are monitoring the results
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
We've identified the root cause of the issue and are working on a fix.
We're currently experiencing degraded performance on the merchant account; load times for both customer facing functionality and aggregate information in the merchant account are elevated.
Report: "Job processing delays"
Last updateWe've caught up with our backlog and all delayed jobs have now been processed.
We've just rolled out a fix and have resumed background job processing; we'll give another update when we've cleared the backlog.
We've identified the root cause of the issue; we've triaged the problem and are working on a permanent fix.
We're currently investigating an issue relating to our background job processing. You may see delays in the following areas: * Customer email delivery * Launching loyalty tiers * Imports and exports Core customer-facing functionality including earning points, claiming rewards, and customer UI is unaffected.
Report: "Partial API and platform outage"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Elevated response times for API and platform services"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Our API and SDK platform is currently experiencing degraded performance
Report: "Elevated response times for API and platform services"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "SDK functionality impaired"
Last updateThis incident has been resolved.
SDK components should now be back online. We're monitoring the situation and will deploy additional resources as needed.
We are continuing to investigate this issue.
LoyaltyLion's on-page SDK components currently have impaired functionality. They may not render, or will render as though a customer is not logged in. We're investigating the issue.
Report: "Partial API & Platform outage"
Last updateSome API and Platform requests will have failed between 19:07 - 21:03 UTC.
Report: "Partial API and platform outage"
Last updateFor a period of about 10 minutes our API and platform experienced degraded functionality and may have returned error responses for API calls and authenticated platform requests.
Report: "Outage"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Our merchant and shopper facing platform and API are currently experiencing an outage. We've identified the problem and are working to restore service shortly
Report: "Merchant account outage"
Last updateWe've resolved the issue and all merchant-facing aspects of LoyaltyLion are now operational.
Merchant-facing aspects of LoyaltyLion are currently experiencing issues. All shopper-facing aspects are unaffected. Engineers are investigating.
Report: "Shopify embedded POS app outage"
Last updateThe Shopify apps team have fixed the issue and LoyaltyLion and all other POS apps should be working again. You don't need to update the Shopify POS app.
Shopify are currently experiencing an issue with embedded POS apps. The LoyaltyLion embedded app and any other POS apps you have installed may fail to open and will time out with an "Application failed to launch" message. We have raised this issue with Shopify and it has been acknowledged; a fix should be incoming.
Report: "Merchant account outage"
Last updateWe had a temporary outage in the merchant account which affected the rules area. Access was restored promptly and no data was lost.