Schematic Incident History

On July 8 between 3:44–4:18pm ET, a subset of customers experienced elevated latency on flag checks, event processing, and API requests. Some requests continued to see intermittent, milder latency elevation into the evening until our permanent fix was in place. We want to share what we found and what we've changed. The root cause was one node in our caching tier reaching its network bandwidth limit. Traffic through this node had been growing gradually for a few weeks, and this particular cloud service limit was not in the set of underlying cache metrics we monitor most closely \(CPU, memory, connection counts, etc\) which all were healthy throughout this incident. A large data operation that afternoon pushed the node over the threshold, at which point network throttling caused the elevated latency customers experienced. In response, we have upgraded the caching cluster with no downtime; it is now comfortably overprovisioned. Latency returned to normal immediately and has been stable since. We also added new monitoring and alerting \(for this metric and several others\), and have already shipped several efficiency improvements that reduce cache traffic outright, with more in progress. We're sorry for the disruption. Reliability of flag checks is something we know you build on, and we've treated this accordingly.

From the time of 3:44-4:18pm EST a subset of customers experienced elevated latency of flag checks, event processing, and the API. The root cause was identified, the issue has been resolved. At around 4:30 PM we received a large number of events that overwhelmed our caching system. This is a hot path in our system that increased API latency substantially for a subset of customers. We're putting some additional mitigations in place now to further prevent similar issues.

From the time of 3:44-4:18pm EST a subset of customers experienced elevated latency of flag checks, event processing, and the API. The root cause was identified, the issue has been resolved. At around 4:30 PM we received a large number of events that overwhelmed our caching system. This is a hot path in our system that increased API latency substantially for a subset of customers. We're putting some additional mitigations in place now to further prevent similar issues.