[Piano Analytics] Delay in generating yesterday's data
Last updatepostmortemOct 30 · 11:12 UTC
Incident explanation
- Due to inconsistencies in our delivery process, two services with incompatible versions were deployed to production during a full redelivery of all services, aimed at quickly resolving a security vulnerability
- These services are part of our data ingestion pipeline, which became blocked. As all consolidation and aggregation jobs depend on this step, they could not be computed, delaying data availability.
Action performed
- Both services were rolled back to their last stable versions, which unblocked the process.
Actions to come
- Strengthen validation of this part of the pipeline during deliveries.
Set up an on-call rotation for the D-1 data feed (details to be defined).
Reported by Piano Software on their status page.
