Potential for some missed messages for subscribers in US East PoP
Last updatepostmortemAug 28 · 17:46 UTC
Problem Description, Impact, and Resolution
On Tuesday, August 25th, 2026, at 12:10 UTC, our internal monitoring alerted us to an issue where a very small subset of messages within one availability zone of one region (our US East point of presence) may not have been immediately delivered to subscribers in that same region. All messages were persisted normally within PubNub Persistence service, thus no message data was lost. Traffic to or from any other region or AZ was not affected, and no other PubNub services were affected. No customers reported impact.
Root Cause
During planned internal load testing, a group of internal routing servers was removed from service. Due to a configuration flaw, their network addresses were not fully retired and remained cached by our publishing layer.
This presented no problem for the most part. However, one address was subsequently reassigned to an unrelated internal component that responded normally on the same protocol and port. Because that response appeared successful, the publishing layer received no error, and messages sent to that address were routed incorrectly.
A rolling restart of the affected servers cleared the outdated addresses at 13:25 UTC.
Mitigation Steps and Recommended Future Preventative Measures
- Once the affected servers were identified, a rolling restart was performed to clear the outdated routing addresses.
- Following an investigation that confirmed the root cause, a fix ensuring outdated internal routing addresses are correctly retired was deployed to all publishing servers worldwide on August 25th, 2026 at 22:00 UTC.
Reported by PubNub on their status page.