Historical record of incidents for Transcend
Report: "Degraded performance of our Backend API [EU Region]"
Last updateThe DSR backlog is now down to normal levels. Incoming DSRs should process as per usual.
We have successfully scaled our infrastructure up to accommodate the increased API demand in the EU region. Backend performance is now healthy. However, the secondary effect of the recent backend performance degradation is that a backlog of DSRs have accrued. The backlog is actively being processed now. We will keep this incident open until that DSR processing pipeline returns to normal levels.
We have identified a fix and are scaling our infrastructure to accommodate extreme API demand. The first-order impact to load times in the Admin Dashboard were fixed overnight and we are monitoring results this morning (Pacific Time). The second-order impact on DSR processing times is our focus this morning, and infrastructure scaling is underway.
We are currently investigating performance issues with our backend API.
Report: "Consent Deployment is failing for Certain Regions"
Last updateOur CDN provider has identified and fixed a cache purging issue. Updates to Consent bundles should now deploy as expected. For more information, see https://www.cloudflarestatus.com/incidents/vft3jf8tvnv2
We're continuing to investigate with our provider. Note that this issue only impacts publishing updates to airgap.js bundles - existing consent deployments continue to function as expected.
We've determined this is an issue with our CDN Provider, and are currently helping their escalation team investigate further.
We are currently investigating an issue with our CDN cache invalidation, which impacts Consent deployments.
Report: "Increased Latency in EU-Hosted Region"
Last updateThis issue has been resolved, with latency returning to normal levels since 1:28 PM PDT.
A fix has been implemented and we are monitoring the results.
Our systems in the EU are under increased load and we are scaling up to meet demand.
We are currently investigating this issue.
Report: "Email Sending is not working"
Last updateThis incident has been resolved.
Email sending has resumed.
We are continuing to investigate this issue.
Email sending is occasionally failing. We are investigating the issue.
Report: "Email Sending is not working"
Last updateEmail sending was temporarily broken, causing DSR submission to fail.
Report: "Email Sending is not working"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
Email sending to data subjects is currently not working. This prevents new DSRs from being submitted from the Privacy Center or from the Admin Dashboard for select customers who are not self-hosting Sombra.
Report: "Backend downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating some partial downtime on our backend API in the EU
Report: "Privacy Center Email Login Issues"
Last updateDuring a code release, users could not log in to the Privacy Center via email.
Report: "Database performance degradation"
Last updateThis incident has been resolved.
We have applied a fix that we believe will restore functionality fully, and are continuing to monitor errors.
Our previous fix did not resolve the issue. Admin Dashboard and Backend API continues to return 5xx errors, and we have escalated the incident to "Partial Outage".
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "Multi tenant Sombra is unresponsive and returning 5xx status"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Report: "Some views under the Privacy Requests tab are failing to load for customers in the EU region."
Last updateThis incident has been resolved.
A few views under the Privacy Requests tab are failing to load. We have determined the root cause of the issue and are working on deploying the fix.
Report: "Backend downtime"
Last updateThis incident appears resolved with the upstream AWS CloudFront service. This appears to have affected users primarily in us-east-1 accessing AWS CloudFront content: https://news.ycombinator.com/item?id=36775816
Our backend has appeared healthy consistently. Our frontend is experiencing unrelated issues caused by upstream downtime in certain edge locations with AWS CloudFront. This appears consistent with what we are seeing on the AWS page of https://downdetector.com/status/aws-amazon-web-services/
We are continuing to monitor for any further issues.
The backend is continuing to look healthy in recent tests
We have confirmed that the API is responding again for users. We are continuing to monitor the application for overall health
We have noticed an issue with our backend API where requests are failing to complete. We have identified the issue and are rolling out a fix.
Report: "Admin Dashboard login issues for specific user accounts"
Last updateOur metrics show that the issue has been resolved. Reloading the page will ensure you are on the latest version of our application.
A fix has been implemented and we are monitoring the results.
We believe we have identified a cause for the username/password login issues, and are working on a patch
Certain accounts on the admin dashboard may have issues logging into the Transcend Admin Dashboard. These users are often those that login with then username/password authentication mechanism in the EU region of Transcend's backend. The steps that have resolved the issues for users have been: - Try logging out and back in if they are already logged in - Try clearing their browser cookies and localStorage if they are able - Try logging in in an Incognito or private browser window
Report: "Degraded Performance on the Admin Dashboard"
Last updateThis incident appears to be resolved, and all affected users should be able to hard reload their browsers to use the updated version
The deploy was successful, and we are now monitoring metrics. Users are recommended to hard reload their pages to ensure they are on the latest version.
We are aware of performance degradation on the admin dashboard, and are attempting to deploy a fix. In the meantime, users may attempt to reload the page and then logout and back in if they see errors relating to their sessions expiring. We will provide updates on this incident as we progress on improving the performance
Report: "Consent Manager Pages are failing to load for some users"
Last updateThis incident has been resolved.
We have identified the root cause and are implementing a fix
Report: "Partial Outage on Backend API"
Last updateThis incident has been verified to be resolved
A fix has been deployed, and we initial results show all requests succeeding. We will continue to monitor systems over the coming minutes
We have identified an issue with our Backend API that is causing some requests to timeout.
Report: "Degraded performance and occasional backend failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.