Degraded performance of TMS Project Management Component (EU and US DC) between August 31 2:26 PM CEST and August 31 5:57 PM CEST
Last updatepostmortemSep 4 · 08:45 UTC
## Introduction We would like to share details about an incident that affected Phrase TMS on August 31, 2026. Between 2:26 and 5:57 PM CEST, some users of the legacy Project page were unable to use Tools-menu actions such as creating or editing jobs and running analyses, and clicking to open a job did not launch the CAT web editor as expected, instead returning users to the project page. This post-mortem explains what happened, how it was resolved, and what we are doing to prevent it from happening again. ## Timeline * **August 31, 2026 at 2:26 PM CEST** – A code change reached production containing a defect that broke script execution on the Project page for any user who loaded it. * **August 31, 2026 at 2:49 PM CEST** – The first customer reports came in describing Tools-menu buttons as disabled and jobs failing to open in the editor. * **August 31, 2026 at approximately 3:10 PM CEST** – Our team identified the root cause of the issue. * **August 31, 2026 at 3:48 PM CEST** – A fix for the underlying defect was completed. * **August 31, 2026 at 4:04 PM CEST** – The fix was verified in a pre-production environment. * **August 31, 2026 at 5:18 PM CEST** – Deployment of the fix to production began. * **August 31, 2026 at 5:57 PM CEST** – The fix was fully live in production and normal functionality was restored for all affected customers. ## Root Cause The incident was caused by a code change intended to fix an unrelated, minor display issue on shared project pages. That change altered how a value was inserted into a script embedded directly in the page. The system that renders the page automatically encodes values for safety, but that encoding does not distinguish between a value being placed in regular page content versus inside a script. As a result, the embedded script's syntax was silently broken once the change reached production. Because browsers stop executing any further code on a page once they encounter invalid script syntax, every script placed after that point on the page stopped running — not just the part related to the original change. This is why customers experienced what looked like two separate problems \(disabled menu buttons and jobs failing to open in the editor\) that were, in fact, downstream effects of the same single defect. The issue was not caught before release because the verification performed at the time confirmed that the underlying data being inserted was correct, but did not load the actual page in a browser to confirm it rendered and executed correctly end-to-end. ## Actions to Prevent Recurrence 1. **Fix deployed** – The underlying defect was corrected and deployed to production the same day it was identified. 2. **Engineering guidance updated** – We have updated our internal engineering documentation to clearly describe this specific failure pattern and the correct, safe way to handle it, so this category of mistake is caught during code review going forward. 3. **Automated detection improvements underway** – We are working on adding automated monitoring for this class of front-end failure, so similar issues can be detected and addressed before customers are affected, rather than relying on customer reports.
Reported by Phrase on their status page.