Historical record of incidents for Chord Commerce
Report: "Chord Documentation Outage"
Last updateOur documentation site (https://docs.chord.co/) is experiencing an outage. We have identified the source of the outage and will notify customers once the documentation site is back up.
Report: "Chord Activations are currently unavailable on the platform."
Last updateThis incident has been resolved. Chord Activations functionality has been restored on our platform.
We are currently investigating this issue and will provide an update once the issue is identified and fixed.
Report: "Systemwide Outage"
Last updateThe incident has been resolved. The Chord data platform and legacy services are now fully operational.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Chord is currently experiencing an unexpected service outage. Our technical teams are actively investigating the issue and working to restore full functionality as quickly as possible. Users may experience login failures in the Chord data platform and legacy services.
Report: "Chord API Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Gitlab outage delaying some data jobs"
Last updateThis incident has been resolved.
Gitlab services have been restored and all data jobs will resume normal schedules.
While Gitlab is down some data jobs will be delayed. Those jobs will resume as soon as Gitlab service is restored, and no data will be lost.
Report: "Chord.co marketing site outage"
Last updateAWS has instituted fixes, and Netlify and other integration partners are reporting improvements/fixes on their end as well. Chord.co is up and running again, and we will continue to monitor our products.
So far, API and store behavior appear to be unaffected. We will update if this changes.
The main chord.co marketing site is currently not responding. Two major upstream providers (AWS and Netlify) are reporting outages. We will keep an eye on the situation, as well as assess if this affects our main product.
Report: "Chord Hub Timeouts"
Last updateThis incident has been resolved.
Our engineers are currently investigating root cause, and requests to a specific Redis cache are timing out, causing a cascade of timeouts downstream. We are currently working on a fix for this.
Some analytics requests from the Chord Hub are currently timing out for some customers.
Report: "Data Pipeline Delays for Some Customers"
Last updateWe're considering this incident fully resolved.
The upstream issue at the vendor appears to have been resolved. We'll leave this incident in Monitoring for the next hour or so to ensure we don't see any rebound.
The vendor is making progress on a fix and reporting a decrease in issues they're seeing, but are not declaring an all-clear yet.
Due to issues with a third party vendor, some data pipelines are being delayed from their normal schedules. We're tracking the incident at the vendor and will update here when we believe we're all clear.
Report: "docs.chord.co is unavailable"
Last updateThis incident has been resolved.
The outage appears to be resolved. We will continue to monitor.
Chord's documentation, docs.chord.co, is currently unavailable due to an outage of our website provider.
Chord's documentation, docs.chord.co, is currently unavailable due to an outage of our website provider.
Report: "Intermittent Issues in Chord OMS"
Last updateHeroku is back, and scheduled jobs are operating again.
Heroku is currently experiencing downtime to their API and dashboard. This is impacting scheduled batch jobs, so some scheduled processes will experience delays until they get back online. Our OMS and APIs are still online and operational. We will provide updates if anything changes.
We've received alerts of a partial outage throughout Chord OMS systems. So far this has affected some background tasks and may intermittently affect the availability of stores. We're working to get confirmation of suspected issues with a vendor upstream.
Report: "Intermittent connectivity issues in the Chord Staging OMS admin interface"
Last updateThis incident has been resolved.
The issue is rooted in the importer worker, so it has been quarantined. Aside from that functionality, Staging should be functioning.
The staging OMS administrative interface is experiencing intermittent downtime. We are currently investigating this issue.
Report: "Service Update"
Last updateAll affected jobs are now recovered.
Service is being restored and scheduled jobs have recovered. We are continuing to monitor to ensure these jobs complete successfully.
We are still monitoring the outage and awaiting updates from our service provider. Our applications remain operational, except that scheduled background jobs are currently affected and will resume when service is restored.
We are currently monitoring an outage at one of our service providers, which is not affecting our applications at this time. We will continue to monitor the situation closely and will update this page if our services are impacted.
Report: "Intermittent connectivity issues in the Chord Staging OMS admin interface"
Last updateThe issue has been resolved. All OMS staging services should be operating normally.
We are still investigating the issue, but can confirm the Staging OMS API should still be operating as expected.
The staging OMS administrative interface is experiencing intermittent downtime. We are currently investigating this issue.
Report: "CircleCI Security Breach"
Last updateCircleCI disclosed their service was subject to a significant security incident. They are advising all users to rotate their secrets as soon as possible. Chord is resolving any risks to our platform and urgently advises its users to change all of their impacted keys. https://circleci.com/blog/january-4-2023-security-alert/
Report: "OMS Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We believe we have a fix. We are seeing less errors and will continue to monitor.
We are continuing to investigate this issue.
We have increased the database performance, but you may still see intermittent errors.
We are experiencing an outage in our OMS
Report: "Elevated API Errors"
Last updateThere was an issue with elevated level of API errors at 11am ET which resulted in some customers seeing 500 errors. This issue was resolved a few minutes after the hour.