Harness Incident History

1. postmortemSep 10 · 07:21 UTC

   # Summary On September 3, 2026, between approximately 2:54 AM and 3:44 AM PDT, customers running pipelines on Prod1 experienced delays in pipeline execution graph rendering and execution status updates. Pipeline execution itself was not affected — pipelines continued to run and complete — but the visual graph and status information in the Harness UI lagged behind actual execution progress. Harness engineering identified the cause, added processing capacity, and restored normal graph and status updates. Extended monitoring confirmed full recovery the following day. # Root Cause A planned database failover activity in the Prod1 region temporarily increased network latency between the pipeline execution service and its database while traffic was briefly served cross-region. This reduced the rate at which the service could process pipeline orchestration events, and a processing backlog began to build. Pipeline execution graphs and status indicators in the UI depend on these orchestration events being processed in near real time. As the backlog grew, graph rendering and status updates fell increasingly behind actual execution progress, and some graphs had to be rebuilt rather than served from cache. Pipeline execution itself does not depend on this same processing path, so pipelines continued to run and complete throughout the incident. # Impact * Affected users: Customers with pipeline executions running on Prod1 during the incident window. * Symptom: Delayed pipeline execution graph rendering and delayed execution status updates in the Harness UI. * Not reported as impacted: Pipeline execution itself — pipelines continued to run and complete. * Data: No data loss, corruption, or exposure was identified. * Suggested workaround: None required from customers. The issue was resolved entirely by Harness engineering. * Duration of impact: Approximately 45–50 minutes of degraded graph and status visibility, from 2:54 AM to 3:44 AM PDT on September 3, 2026. Harness continued monitoring after the backlog cleared and confirmed full recovery. # Remediation ## Immediate Harness engineering added processing capacity to the affected service and restarted it, which cleared the event-processing backlog and restored normal pipeline execution graph and status updates. # Action Items | **Action Item** | **Objective** | | --- | --- | | Queue based alarms | We are raising priority for queue based alarms so we can take faster remediation actions \(increase capacity\) |

Reported by Harness on their status page.

1. postmortemSep 9 · 23:23 UTC

   ### Summary On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.  ### Customer Impact * Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out. * No data loss or corruption occurred. ### Root Cause The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures. ### Mitigation * Disabled the capability across all environments * Temporarily increased platform capacity to restore stability ### Next Steps To prevent recurrence, Harness will: 1. **Permanently fix the capability** by profiling and eliminating the sub-optimal code path and query 2. **Improve detection** by enhancing alerting for resource-intensive queries on high-frequency platform paths

Reported by Harness on their status page.

1. postmortemSep 9 · 18:22 UTC

   ## **Summary** Between 12:34am PST and 12:38am PST on 5th September, the Delegate service manager experienced some elevated exceptions when attempting to write to the database. Consequently, delegate connections were dropped, causing them to disconnect. Delegate automatically re-attempts registration back to the `delegate service manager`  and majority of the delegates got connected back after the incident. For Docker and ECS delegates the automatic restart is not enabled unless these delegates have health monitoring enabled. For these delegates a manual restart is needed and was recommended.  Post restart the delegate would re-connect and the issue was resolved. ## **Root cause** On Prod2 cluster we identified a performance bottleneck in the delegate service that, under certain conditions, can increase database write latency and delay heartbeat processing which leads to delegates being disconnected. ## **Impact** All K8s delegates and \`Docker/ECS\` delegates got connected back immediately within 4 mins and started to function normally. The impact can be scoped to those specific types of delegates that didn’t have health monitoring enabled.  ## **Remediation** * Immediate: We have added additional monitoring and increased resources for handling the influx of traffic.  * Permanent: We have identified a hotspot in the code that can cause high latency when writing to a database which we are actively working on resolving.  ## **Action Items** To prevent such issues from happening again, Harness will work on the following: 1. Increased targeted monitoring and alerting to initiate timely mitigation and prevent this from happening again. 2. Fix the identified delegate service managers database client reconnect failures 3. Fix the hotpots that can cause query latency.

Reported by Harness on their status page.

1. resolvedAug 26 · 09:10 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. postmortemSep 2 · 22:15 UTC

   ## Summary Between August 27 and August 28, 2026, customers experienced an issue where some pipelines, deployments, and related resources appeared as not found in the Harness UI and API, even though the underlying data remained intact. ‌ The issue occurred during a planned internal infrastructure update that affected communication between internal platform services. As a result, requests that depended on account, organization and project scope resolution were unable to complete successfully, which led to incorrect not found responses being returned to customers for existing entities. ‌ Engineering identified the issue, rolled back the change, and restored normal service. No customer data was lost or deleted during the incident. ‌ ## Root Cause The issue was caused by a configuration error introduced during a planned internal service routing update in Production. An internal platform service responsible for resolving account, organization, and project context was unable to validate requests from other Harness services after the change was applied. Because that validation step is required before many entity reads and pipeline-related actions can proceed, the failed requests surfaced to customers as not found errors for resources that continued to exist normally. The issue was limited to the affected production environment and was resolved by reverting the change and restoring the previous service communication path. ‌ ## Impact * Some customers saw existing pipelines, deployments, and related entities appear as not found in the UI and API. * Some pipeline-related operations, including execution progression, webhook-triggered starts, scheduled trigger evaluation, and entity listing, were temporarily disrupted. * The issue affected availability and visibility of existing entities, but it did not remove data or change customer configurations. * No unauthorized access occurred, and no customer data loss was observed. ## Remediation * **Immediate:** Reverted the infrastructure configuration update and restored the previously working service communication path. * **Recovery validation:** Verified that affected entity lookups, pipeline operations, and dependent APIs were functioning normally after rollback. * **Permanent:** Corrected the configuration handling associated with the update so similar issues do not interfere with service-to-service authentication in future rollouts. ## Action Items To prevent such issues from happening again, Harness will  1. Improve configuration validation by enhancing the pre-deployment tests to verify internal service communication before shifting production traffic. 2. Enhance monitoring and alerting for internal authentication failures so issues can be detected earlier. 3. Improve error handling so dependency failures are less likely to appear to customers as resource not found errors.

Reported by Harness on their status page.

1. postmortemAug 27 · 23:36 UTC

   ## Summary * Starting at **23:42 UTC** on August 23, 2026, several FME customers reported failures loading the FME UI. * FME UI Artifacts served from the CDN expired due to a retention policy, causing FME UI to fail to load. * Any flag request changes through the API, change delivery, and the data pipeline continued to work with no interruption. ## Root Cause * The FME UI is served from a CDN. The UI artifacts got evicted due to a retention policy, causing the UI to fail to load for all users. ## Impact * The FME UI was unable to load for all users across all production environments. ### What was not impacted? * SDK functionality and runtime flag evaluation * Admin API calls * Customer flag configuration data * No data loss occurred ## Remediation * FME UI got restored in the CDN through a deployment * Recovery confirmed across all production environments before closing the incident. ## Action Items * Improve the asset retention policy so that the currently active version is never subject to eviction.

Reported by Harness on their status page.

1. postmortemAug 26 · 17:20 UTC

   **Summary** On 19 August 2026 between 12:35 and 17:29 UTC, the Harness Application Security service experienced a significant disruption affecting both the customer-facing console and the data ingestion pipeline in the SaaS Production and US1 regions. ‌ **Root Cause** The internal configuration service that supplies runtime settings to nearly every other component became overloaded and entered a repeated restart cycle. Because so many services depend on it, the effects were broad: console pages such as protection policies, posture views, activity logs, API inventory, and custom policy failed to load or timed out, and downstream processing stalled while waiting for configuration it could not obtain. # **Customer impact** | **Dimension** | **Detail** | | --- | --- | | Console \(UI\) impact | Multiple pages failed to load or timed out, including protection policies, posture event pages and posture views inside dashboards and insight pages, activity log queries, API inventory screens, custom policy, and sensitive-data views and widgets. | | Ingestion impact | Security telemetry processing degraded severely and, in some paths, stopped entirely. Consumer lag grew across normalisation, grouping, anomaly detection, generation, and related processing stages. | | Data loss | A subset of telemetry ingested during the disruption was permanently dropped. | ‌ **Mitigation** Several intermediate mitigations additional CPU and memory, relaxed health-check thresholds, a database restart, and a larger connection pool ameliorated the issue. Disabling the new feature in both affected regions restored throughput sharply and durably. The incident was resolved at 17:29 UTC. ‌ # **Preventive actions** The following actions are committed and tracked internally to completion. The feature that triggered this incident remains disabled and will not be re-enabled until the work below is complete and validated. | **Action** | | --- | | | | OPtimize the code by tuning parameters such as cache eviction and retention , evaluate cursor-based pagination for bulk rule retrieval as rule counts grow | | Add a purpose-built database index for the service-scoping access pattern | | Remediate pipeline recovery semantics so consumers replay safely after position-marker loss instead of skipping backlog | | Mandate staged rollout for configuration overrides that alter downstream request patterns: low-volume cluster, then mid-volume, then high-volume | | Add backpressure and concurrency protection to the configuration service: circuit breaking, bounded queues, and timeout isolation | | Enhance observability by Instrumenting more detailed metrics |

Reported by Harness on their status page.

1. postmortemAug 26 · 04:57 UTC

   # Summary On 20 August 2026, beginning at approximately 15:00 UTC, the Harness platform experienced widespread performance degradation across all production environments. Pipeline executions that normally complete in around two minutes took seven to ten minutes. Continuous Delivery, Continuous Integration, pipeline orchestration, and Feature Management & Experimentation were all affected. Google Cloud Platform experienced a multi-product incident in the us-west1 region affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk I/O. Harness production infrastructure runs on persistent disks in that region. The degradation raised database operation latency from approximately 2 ms to over 10 ms at the 95th percentile, which in turn caused message-queue processing lag and propagated to every service that depends on timely database access. ‌ # Impact This was a degradation, not an outage. Pipelines continued to execute and complete successfully throughout; they were slow rather than failing. No data was lost, and no customer work was dropped as a result of this incident. # **Root cause** Harness production infrastructure in the affected environments runs on Google Cloud Platform persistent disks in the us-west1 region. When that storage layer degraded, the effect propagated through the platform in a predictable chain: **Persistent-disk I/O degradation in us-west1.** Google Cloud Platform experienced a multi-product incident affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk performance. This was an infrastructure failure in the provider’s environment, outside Harness’s control. # **Preventive actions** Although Harness cannot prevent a cloud provider infrastructure failure. The actions below are aimed at detecting one faster and being better positioned to act on it. | **Action** | | --- | | Continue routine pre-testing of targeted cross-region database failovers, as performed during this incident, to keep failover readiness verified rather than assumed | | Assess full-stack multi-region failover readiness for future scenarios in which cross-region latency would be unacceptable |

Reported by Harness on their status page.

1. postmortemAug 23 · 21:45 UTC

   ### Summary On August 20, 2026, between 10:24 and 14:55 UTC, a subset of FME writes failed. Writes made from the FME UI and writes made with Harness access tokens \(PATs and SATs\) were not affected. Runtime flag evaluation continued to work normally. The issue was mitigated by reverting a recent authentication change in a shared governance service, and affected writes returned to normal by 14:55 UTC. Status: [https://status.harness.io/incidents/rhthgm7d5dkz](https://status.harness.io/incidents/rhthgm7d5dkz) ### Root Cause A change in how a shared governance service authenticated inbound calls resulted in some FME writes being rejected. Those writes used service-to-service credentials that the governance service could no longer verify after the change. FME surfaces a governance failure to the client as HTTP 499, the same status used when a governance policy intentionally denies a change. Because 499 is a valid, expected response in that deny path, the failures did not look like an outage on our alerts, and the incident was identified from customer reports rather than internal detection. ### Impact * A subset of FME writes failed during the window, primarily those made using legacy Split API keys or change request scheduling. * Writes made from the FME UI were not impacted. * Writes using Harness access tokens \(PATs and SATs\) were not impacted. * Runtime flag evaluation continued normally. * No data loss occurred. Failed writes did not apply. ‌ ### Remediation Reverted the governance-service authentication change. Affected writes returned to normal immediately. ### Action Items To prevent such issues from happening again, * Harness will return a distinct error \(not 499\) when a write fails because governance could not be evaluated, so it is not confused with an intentional policy denial. * Add alerting on the governance evaluation call itself, rather than relying on the client-facing status code. * Expand authentication support for policy evaluations. * Expand automated coverage for additional write scenarios.

Reported by Harness on their status page.

1. postmortemAug 21 · 04:19 UTC

   # **Summary** Between 25 July and 4 August 2026, pipeline execution dashboards and overview pages in the Harness Prod 2 and Prod 3 clusters displayed data that was between behind real time. Pipelines themselves continued to build, deploy, and execute normally throughout; the issue was confined to how quickly execution records were copied into the database that serves reporting and dashboard views. ‌ **No customer data was lost.** Every affected record remained durably stored and was replayed into the analytics datastore once the underlying limitation was removed. Harness migrated the affected clusters to a horizontally scalable, queue-backed version of the replication component on 1 August 2026 and completed targeted data backfills for all affected accounts. # **Root cause** Harness maintains a change-data-capture component that continuously replicates pipeline execution records from the primary operational datastore into a separate time-series datastore optimised for dashboards and reporting queries. Dashboards read exclusively from the analytics datastore. When replication falls behind, dashboards render an accurate but older view of the world, while execution itself is unaffected. This was caused by sharp, sustained increase in database write volume from another Harness platform module sharing the same replication path exceeded the throughput ceiling of the older, single-instance version of that component still running in Prod 2 and Prod 3. A backlog formed and grew. ‌ ‌ # **Preventive actions** Harness has completed or committed to the following actions to prevent such issues. | **Action** | | --- | | Fine tune the replication lag alerting so that any delay beyond a defined threshold is notified | | Add a replication lag panel to the standard platform monitoring board so pipeline health is visible to on-call by default | | Reduce write amplification from co-tenant modules through per-module rate limiting or entity filtering on the replication stream |

Reported by Harness on their status page.

1. postmortemAug 17 · 18:40 UTC

   ## **Summary** On August 6, 2026 \(morning PDT\), some customers running pipelines in the Prod2 production environment observed pipeline executions that stopped making progress — stages that did not advance and produced no further output or status updates. The issue was reported by affected customers. Harness engineers identified the cause, mitigated the impact, and pipeline executions returned to normal operation. The issue was caused by a self-referential pipeline expression. A Git webhook triggered a pipeline that referenced the contents of the webhook payload, and the payload itself contained further copies of that same expression. Each round of expression resolution therefore produced more expressions to resolve, doubling the amount of work each time. This exhausted the resources of the service instance processing that execution, and other executions assigned to the same instance were unable to progress while it was in that state. ## **Impact** During the incident window \(approximately 6:11 AM to 11:23 AM PDT on August 6, 2026\): * Some customers' pipeline executions on Prod2 stalled mid-execution and made no further progress. * Affected executions produced no new step output or status updates, and had to be aborted and re-run after mitigation. * Behavior was limited to executions being processed by the affected service instance — pipelines handled by other instances continued to execute normally. There was **no data loss**. Pipeline definitions, execution history, and stored state were unaffected. The majority of pipelines on Prod2 continued to execute successfully throughout the incident; the primary impact was that some in-flight executions could not complete and needed to be re-run once the issue was mitigated. ## **Root Cause** Harness pipelines support expressions that are resolved at runtime — for example, an expression that inserts the contents of the Git webhook payload that triggered the pipeline. In this case, a Git commit message contained the literal text of the payload expression itself, twice, and the pipeline referenced that same payload expression. Because the commit message is part of the webhook payload, resolving the expression inserted the entire payload — including the two literal copies of the expression carried in the commit message. Those newly inserted copies were then treated as expressions to be resolved, and each pass inserted two more full copies of the payload. The size of the value being processed, and the work required to process it, therefore doubled on every pass and grew exponentially rather than converging. Harness has a safeguard intended to stop exactly this: expression resolution is bounded by a maximum nesting depth, beyond which resolution halts and the pipeline fails with an explicit error. A defect in that safeguard meant the limit was not applied in this specific self-referential case, so resolution continued unchecked. Expression resolution runs inline on the threads that start pipeline steps. As each pass consumed progressively more memory and CPU without ever completing, the service instance performing that work stopped making progress, and every execution assigned to that instance stalled — which is what customers reported. ## **Mitigation** Harness completed the following immediate mitigation steps: * Identified the pipeline and the expression pattern responsible for the runaway resolution. * Stopped the affected service instance so that it would take on no further work. The remaining healthy instances picked up and processed queued executions normally. * Confirmed that pipeline executions returned to normal and closed the incident. These actions restored normal pipeline execution behavior and resolved the customer-facing impact. ## **Action Items** To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented: * Fix the defect in the expression depth and loop-detection safeguard so that self-referential expressions are caught and fail fast with a clear error instead of consuming resources without bound. * Prevent payload expressions from being resolved out of trigger payload content, removing the self-referential path entirely. * Tighten the maximum expression nesting depth and evaluate explicit loop detection in addition to the existing depth limit. * Enhance automated tests in pre-production environments that reproduce self-referential expression patterns and verify that the safeguard detects and stops them. * Add monitoring for this pattern in pipeline executions so that it is detected proactively.

Reported by Harness on their status page.

1. postmortemAug 10 · 18:24 UTC

   ## Summary Customers on Prod1, Prod2, and Prod3 \(US\) clusters experienced failures when loading SEI 2.0 dashboards on August 6, 2026, from 7:22 AM PDT to 9:03 AM PDT. Customers calling the SEI 2.0 API also experienced similar failures. No customer data was lost, and ingestion of all integration data continued to work uninterrupted. SEI customers using 1.0 were not impacted. ## Root Cause The incident was caused by resource exhaustion on the nodes serving queries. This resource degradation developed in a pattern that did not cross our existing alerting thresholds early enough to provide sufficient warning or allow mitigation before customer impact occurred. ## Impact Customers on Prod1, Prod2, and Prod3 \(US\) clusters were unable to load SEI 2.0 dashboards during the incident window. **Duration:** August 6, 2026, 7:22 AM PDT – 9:03 AM PDT \(~1 hour 41 minutes\) ### What was not impacted? * Data ingestion and processing * SEI 1.0 customers * Integrations and metadata flows No customer data was lost. ## Remediation Upon identifying the root cause, our team took immediate corrective action by adding capacity to restore the affected systems. Services were fully recovered, and all dashboards resumed normal operation at 9:03 AM PDT. ## Action Items To prevent from such issues happening again, Harness is/has Proactively added capacity updates have been applied to prevent this issue from recurring #### Enhanced Monitoring and Alerting Additional monitoring and alerting have been put in place to detect anomalies early, focused on a leading indicator, which in this case was thread pool exhaustion, before they can impact dashboard availability and data rendering. #### System Patch in Progress We are working with our vendor to apply a patch to remediate this and similar issues completely.

Reported by Harness on their status page.

1. postmortemAug 13 · 00:50 UTC

   # **Summary** On July 31, 2026, artifact uploads performed through pipeline in the EU1 cluster began failing with an authentication error. Uploads initiated manually \(outside of a pipeline\) were not affected, and the ability to retrieve existing artifacts \(downloads\) was also unaffected — this was isolated to the specific pipeline upload path in one cluster. # **Impact** * Artifact uploads performed through pipeline in the EU1 cluster failed with an authentication error for approximately 4 hours and 34 minutes. * Retrieving existing artifacts \(downloads\) was not affected. * Manually uploading artifacts outside of a pipeline was not affected. * Other clusters/regions were not affected by this issue. # **Root Cause** The component responsible for handling pipeline-based artifact uploads is distributed as a container image. In the EU1 cluster, this image is retrieved from an internal registry that mirrors a public image source; in other clusters, the same image is retrieved directly from the public source. A publishing error in our release process caused a new build of this component to be published using a version label that was already in use, rather than being assigned a new, unique version. As a result, two different images ended up associated with the same version label in the public source. Our internal registry mirrors images from the public source via an automated replication process. Because of how that replication was triggered, it copied the original \(earlier\) image associated with that version label rather than the corrected one. This meant the EU1 cluster — which pulls from the internal mirror — ended up running a different, defective image than other clusters, which pull directly from the public source and therefore received the corrected image. The defective image contained an authentication issue that caused pipeline uploads to fail. # **Mitigation** * Reverted the affected account to the last known-good version of the upload component, immediately restoring pipeline uploads. * Published a corrected, permanent version of the component to resolve the issue across all clusters. # **Next steps** ‌ * Fix the upload step to remove the underlying container-related defect that made this failure mode possible. * Update our release pipeline for this component so that publishing an image can never overwrite an existing version — every publish must create a new, distinct version going forward.

Reported by Harness on their status page.