Summary:
On October 6th, 2026, at 1:00 PM PT, our monitoring alerted us to a high number of connection
errors in the production environments shortly after a hotfix was deployed. Preliminary engineering
investigations coupled with internal and external reports of platform loading errors confirmed
platform reachability issues. A platform incident was declared at 1:11 PM PT, and the hotfix was
rolled back, along with a targeted code fix deployment to address the underlying code error. All
services were fully restored by 2:09 PM PT the same day.
Severity:
Sev-1
Scope:
All Firstup customers were affected, with customers in US2 and EU platforms experiencing a
more prolonged service disruption due to circular restart dependencies that had to be addressed
separately from the hot fix rollback. Customers in the US1 platform experienced a brief disruption
of service primarily accompanied by degraded application responsiveness.
Impact:
- US1: From 1:00 PM PT to 1:19 PM PT, service was degraded and the platform processed up to 70% less work than normal.
- US2: From 1:00 PM PT to 2:09 PM PT (69 minutes), users couldn't log in to Studio, Employee Experience or the support portal, and extended message delivery times.
- EU: From 1:00 PM PT to 1:26 PM PT (26 minutes), users had the same problems as in US2.
Root Cause:
The root cause for this incident was identified to be the deployment of a hotfix that was built on an
updated shared base software image. That image included newer networking and encryption
libraries that don't work with the web server setup of one of our core services, so the service
crashed. While this problem had already been identified and protected against in our regular
release processes, it had not been protected against in the hotfix deployment process branch.
Our systems and services health checks didn't catch the failure until the healthy copies of the
impacted service had already been replaced. Additionally, startup dependencies between two
backend services increased the time to recovery and was more pronounced in the US2 platform.
Mitigation:
In US1 and EU, the deployed hotfix was rolled back and services were restored by 1:19 PM PT
and 1:26 PM PT, respectively. In US2, and in addition to the hotfix rollback, we deployed a code
fix that:
- removed the startup dependency between the two backend services
- locked the base software image to the last working version
This fully restored service by 2:09 PM PT. The same fix then went to all regions in a scheduled
release at 6:50 PM PT on October 6th, 2026, with no customer impact.
Recurrence Prevention:
To prevent this incident from happening again due to the same root cause, Firstup has done (or
committed to doing) the following:
- Lock and version shared base software images in every build process, including hotfixes, so that what is test is exactly what is released.
- Roll out changes in stages, starting with one region or part of capacity, so problems are caught before they reach every region.
- Strengthen health checks so they confirm the service is actually working before it takes traffic.
- Removed the identified startup dependencies between the two core services in the US1 and EU platforms by deploying the code fix during our maintenance window on October 6th, 2026.
- Conduct a review of any remaining dependencies for similar risks.
- Tighten the hotfix process. Hotfixes that aren't deployed promptly will need re-approval, and related changes across services will be deployed together.
- Update monitoring and recovery runbooks in every production region.
