Multiple products in us-central1-b are experiencing network service degradation.
Last updateresolvedSep 29 · 16:09 UTC
Addendum to Incident Report
This addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected:
- Compute Engine VMs: VMs running in the affected data center lost network connectivity to resources located outside of their datacenter. From a Compute Engine perspective, this incident only affected a subset of VMs in us-central1-b and us-central1-f.
- Zonal and Regional Products Based on Compute Engine Products with dependencies on VMs in the affected datacenter experienced symptoms like connection timeouts or service unavailability because the underlying VMs lost network connectivity. Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker.
- Google Kubernetes Engine (GKE): GKE clusters whose control plane had dependencies on Compute Engine VMs in the affected clusters experienced errors and timeouts when administrators or nodes attempted to connect to the Kubernetes API server. Consequently, deployments might have been delayed, Kubernetes API operations might have failed, and new nodes might not have registered with the control plane (causing node pool operational delays).
- Cloud Run and App Engine: Workloads in the us-central1 region experienced elevated serving latency, 5xx aborted request errors, and outbound connectivity degradation. To prevent severe request failures, traffic was diverted to other clusters. Diverting requests to other clusters temporarily caused elevated latency or errors in those clusters.Once the impacted cluster was restored, workload demand for these new backends created a 'thundering herd' of concurrent instance initializations. This sudden surge bypassed warm caches and overwhelmed internal file-serving tiers, extending symptoms until emergency capacity was provisioned.
- Other Google APIs and Services, in us-central1 or a multi-region that includes us-central1, were affected if they had dependencies on software tasks in the affected clusters and weren't able to use software tasks in an unaffected cluster. Symptoms included connection timeouts or service unavailability. Examples of other Google APIs and services are: Google Cloud Bigtable, BigQuery, Google SecOps SOAR, Cloud Spanner, and Cloud Firestore. For some services, symptoms were limited to temporary latency spikes and service unavailability until workloads shifted to unaffected clusters.
- Private Service Connect: Clients (in any region) experienced connectivity issues involving PSC endpoints with dependencies in us-central1, in specific circumstances: - PSC endpoints for Google APIs were affected in the same way as noted in the Google APIs and services section. - Producer/consumer PSC endpoints were affected if the producer had load balancers or backend VMs in the affected clusters of us-central1.
- Hybrid Connectivity: Some Cloud Router, Cloud VPN, and Cloud Interconnect VLAN attachments in us-central1 were affected in specific circumstances: - Cloud Router BGP software tasks for some Cloud Routers in us-central1 experienced a Cloud Router maintenance event if their BGP software tasks were running in the affected clusters. - Cloud VPN tunnel software tasks running in the affected clusters lost network connectivity. This caused some Cloud VPN tunnels in us-central1 to go down until a replacement tunnel task in a different cluster was assigned. - Resources (in any region) sending packets through Cloud Interconnect VLAN attachments in us-central1 might have experienced intermittent connectivity issues in certain situations where network flows weren’t already programmed.
- Envoy-based load balancers and other products: Load balancers and other products that depended on managed Envoy tasks of a proxy-only subnet in us-central1 experienced increased latency or connectivity issues if enough Envoy tasks were in the affected clusters. The following products were affected: - Regional external application load balancers in us-central1 - Regional internal application load balancers in us-central1 - Cross-region internal application load balancers with proxies in us-central1 - Regional external proxy network load balancers in us-central1 - Regional internal proxy network load balancers in us-central1 - Cross-region internal proxy network load balancers with proxies in us-central1 - Secure Web Proxy in us-central1
Reported by Google Cloud Platform (Americas) on their status page. View vendor report ↗