Harness Incident History

1. resolvedOct 28 · 17:10 UTC

   Starting 10/28/2026 10:10 AM CI jobs on internal VM runners intermittently hung or expired without logs, affecting some customers in prod1/2. The resolution is to upgrade the new runner version https://github.com/drone-runners/drone-runner-aws/releases/tag/v1.0.0-rc.318 (Docker image: drone/drone-runner-aws:1.0.0-rc.318)

Reported by Harness on their status page.

1. resolvedOct 27 · 22:24 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. postmortemOct 23 · 18:21 UTC

   Summary

   On October 19, 2026, between approximately 12:10 AM and 12:55 AM PDT, customers on Prod4 experienced errors and intermittent unavailability of Harness Pipelines services. Heavy pipeline executions drove a sharp rise in CPU and memory use on the pipeline service, and for a short period all service instances were unavailable. Harness engineering scaled the service both vertically and horizontally, which restored service. A fix for the underlying behavior is being rolled out.

   Root Cause

   On October 19th, high number of concurrent executions with very large execution graphs drove a sharp increase in cluster resources. The service stores and retrieves this full execution graph for visualization purposes and further progressing the pipelines to next states. Our auto-scalaers on Prod4 took longer than usual for scaling resources due to a sharp rise in CPU and memory. 

   As a part of the fix, the execution graph update will now process graph changes incrementally instead of the full graph. It is being rolled out in a phased manner and will be completely rolled out at the earliest.

   Impact

   • Affected users: Customers on Prod4.
   • Symptom: unavailability of pipeline functionality in the Harness platform.
   • Data: No data loss.
   • Suggested workaround: None required from customers. Harness engineering resolved the issue.
   • Duration of impact: Approximately 20 minutes of full unavailability and degraded performance for 25 minutes. Harness continued monitoring after recovery and confirmed platform stability with customers.

   Remediation

   Immediate

   Harness engineering scaled the pipeline service both vertically (more resources per instance) and horizontally (more instances), and removed instances in an unhealthy state. The instance count stabilized and normal service was restored. The system will stay scaled up till the fix is completely rolled out. 

   Permanent fix and Action Items

   To prevent such issues from happening again, Harness is

   1. Tuning the auto heal/scaling process so that impact is mimimal to none.
   2. Optimizing the execution graph flow so that large graphs can be process faster.

Reported by Harness on their status page.

1. ResolvedOct 19 · 08:21 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. resolvedOct 15 · 10:27 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. ResolvedOct 15 · 10:27 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. resolvedOct 15 · 08:15 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. postmortemOct 23 · 03:16 UTC

   . .

Reported by Harness on their status page.

1. ResolvedOct 13 · 04:29 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. postmortemOct 23 · 03:06 UTC

   tldr: \(There was a brief rate limiting by maven central\) More details to follow

Reported by Harness on their status page.

1. ResolvedOct 12 · 16:31 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. resolvedOct 8 · 10:53 UTC

   This incident has been resolved.

Reported by Harness on their status page.

1. postmortemOct 23 · 02:12 UTC

   1. Summary

   On October 6, 2026, pipelines containing an IaCM stage were unable to start on Prod1 for approximately two hours. Affected users saw an error when the pipeline attempted to create its execution plan. Pipelines that did not use an IaCM stage continued to run normally throughout.

   The underlying cause was an internal logging defect in the service that handles IaCM stage execution: under a specific, rare timing condition, that service's own logging system could end up waiting on itself indefinitely, which in turn stalled the broader process responsible for preparing pipeline execution plans. The issue was triggered by an unusually large burst of log activity from one customer's pipeline. We resolved the immediate impact by rolling the service back to its previous stable version, and a permanent code fix is currently in testing.

   2. Impact

   • All pipelines using an IaCM stage on Prod1 were unable to start for approximately two hours.
   • Multiple customers on the Prod1 cluster were affected.
   • Pipelines that did not use an IaCM stage were not affected and continued running normally.
   • No data was lost as a result of this incident.
   • As a side effect of the rollback used to resolve this incident, an unrelated fix for one customer (extending how long a certain authentication token remains valid during long-running pipelines) was temporarily reverted. That fix was reapplied the same evening.

   3. Root Cause

   The service responsible for IaCM pipeline stages logs its activity through an internal queue: log messages are placed on the queue, and a single background worker continuously reads from it and writes the messages out. Separately, that service has a filter that can rewrite the severity of certain log messages back onto that same queue.

   Under a specific, narrow timing condition, this created a deadlock, if the queue happened to be completely full at the exact moment a rewritten message was being submitted back onto it, the one worker responsible for freeing up space on the queue would itself get stuck waiting for space to free up, a condition it could never resolve, since it was the only thing that could have freed that space. Once that happened, every other part of the service that needed to log anything also became stuck waiting, which cascaded into the pipeline-plan-creation process failing for any pipeline using an IaCM stage.

   4. Resolution

   • Rolled the service back to its last stable version, which resolved the immediate incident.

   5. Preventive Actions

   1. Optimize the underlying defect: Optimize and tune the logging filter so that rewriting a message's severity happens on the thread that generated the log line, rather than being resubmitted through the same background worker's queue, eliminating the deadlock condition entirely.
   2. Enhance monitoring on logging-queue health: add metrics and alerting on this internal queue so that this class of saturation pages our on-call team automatically rather than relying on customer reports.
   3. Evaluate limits and perform high stress tests with very large pipelines:

Reported by Harness on their status page.

1. resolvedOct 6 · 20:00 UTC

   On October 7, 2026, from 08:00 UTC to 11:00 UTC (1:30 PM to 4:30 PM IST), some pipeline executions on the Prod 7 cluster saw intermittent failures in Policy (OPA) step evaluations.

Reported by Harness on their status page.

1. postmortemOct 10 · 01:57 UTC

   Summary

   Customers on Prod 2 experienced intermittent failures of a few Verify steps at startup. Affected steps failed shortly after launch with a message indicating Continuous Verification(CV) could not reach an internal Harness service. The issue stemmed from a recent service update that introduced connection pooling to improve performance and reliability with a keep-alive setting incompatible with the receiving service's timeout.

   No customer data was lost or corrupted.

   Customer Impact

   Customers using CV Verify steps on Prod-2 saw intermittent step failures during the incident window. The failure rate was low — the majority of executions completed successfully throughout the window. Pipelines that did not use CV Verify steps were unaffected.

   Root Cause

   A recent CV service update introduced connection pooling to an internal platform component, with a keep-alive duration longer than that component's idle timeout. Connections idle long enough were silently closed by the server; the CV service did not detect this and had no retry logic for such failures, causing them to surface directly as Verify step failures. The issue was intermittent because it only affected connections that happened to be idle long enough to expire.

   Mitigation

   The connection keep-alive duration was reduced to fall within the receiving service's idle timeout. Failures dropped to near zero immediately, and no further step failures of this type have been observed on any production environment.

   Next Steps

   To prevent recurrence, Harness will:

   • Fix connection handling: Tuned the default keep-alive and enabled automatic retry on connection failures so stale-connection errors are handled transparently.
   • Add resilience at step startup: Introduce retry with backoff for the internal service call at the Verify step start, so isolated transient failures do not surface as immediate step failures.
   • Improve alerting: Enhanced monitoring for internal connection error rates.

Reported by Harness on their status page.

1. postmortemOct 9 · 18:48 UTC

   Summary

   On Tuesday, 6 October 2026, IaCM pipelines that use the OpenTofu plugin in Prod1 failed intermittently for a production accountAffected runs injected secrets successfully, then later steps timed out or failed authentication. The same pipeline often succeeded on retry, with no configuration change. Typical failing executions ran for about 90 minutes.

   Other Harness modules were not broadly impacted. We have no report of customer data loss. Pipelines that finished inside the token lifetime were unaffected by this defect.

   Root Cause

   The IaCM stages minted authentication tokens that expired after one hour. Policy checks later in a long run then rejected the token, so the step timed out or failed authentication.

   That one-hour lifetime was fixed. Runs of about 90 minutes crossed the one-hour mark, which is why the failures showed up on longer pipelines and often cleared on a shorter retry.

   A separate set of plan errors in the same account came from credential setup in the customer workspace and is outside this incident.

   Impact

   • Product area: IaCM pipelines using the IaCM plugins talking to OPA sdk.
   • Customer experience: intermittent timeouts and authentication failures after secret injection, on runs longer than about one hour.
   • Environment: Prod1, one production account
   • Data: No data loss reported.

   ‌

   Remediation

   • Immediate: Opened a status page for the affected Prod1 account, and shipped an interim IaCM manager change for one class of the observed failures while the plugin fix was built.
   • Permanent: Shipped an OpenTofu plugin hotfix so token expiry is configurable. In production the lifetime was raised to about 6 hours, so a long pipeline does not outlive the token. The hotfix was validated in QA and deployed to Prod1.

   We apologise for the disruption caused. Please reach out to your Harness support contact if you have any further questions.

Reported by Harness on their status page.

1. postmortemOct 9 · 16:17 UTC

   Okta Application SP-Initiated SSO Login Failures

   ‌

   Summary

   ‌

   Between October 5, 2026, 18:18 UTC and October 6, 2026, 00:26 UTC, some Harness customers were unable to sign in to the Harness Hosted MCP server via SAML Single Sign-On (SSO). The main Harness Portal user authentication remained fully operational. Affected users saw login errors or were redirected to an invalid SSO page instead of completing sign-in.

   ‌

   The issue was caused by breaking changes in a version upgrade of the Harness identity service HarnessID. We fixed it with a series of hotfixes, and SSO login was fully restored for all affected customers.

   ‌

   Impact

   ‌

   Users signing in to Harness MCP via SAML SSO were unable to complete authentication. Existing MCP sessions that relied on SSO could not be refreshed, so some users had to sign in again and could not. Non-SSO logins to Hosted MCP worked during most of the incident. No customer data was lost or exposed.

   Customers using their own generic SAML 2.0 application configuration in Okta were not affected.

   Root Cause

   On September 29, 2026, an updated version of the Harness application was published to the Okta Integration Network. The update added support for registering multiple reply (ACS) URLs, so that customers running Harness across several regions or environments could configure more than one callback URL.

   ‌

   Harness includes an AssertionConsumerServiceURL value in the SAML authentication request it sends to Okta. Before this update, Okta did not validate that value for application instances without additional ACS URLs configured and instead returned the SAML assertion to the application's default ACS URL. After the update, this validation became active across all existing application instances, including those where the customer had never configured an additional ACS URL. Because the requested value was not present in the instance's registered list, Okta returned an HTTP 400 error.

   ‌

   The updated application went through Okta's standard OIN publishing review, which includes backward-compatibility verification against existing installations of the published integration. This change in validation behavior for instances with no additional ACS URLs configured was not surfaced during that review, so it reached customers at the time of release.

   Remediation

   Immediate Response

   ‌

   Two workarounds were published on the status page: sign in from the Harness tile in Okta (IdP-initiated login), which was unaffected, or register the application's Okta SSO URL in the Additional ACS URLs field of the Harness Okta application. Harness engaged each reported affected customer directly and validated their sign-in.

   ‌

   Resolution

   ‌

   Rolling the application back to the previous version was deliberately avoided, as that would only defer the same disruption to the next release. Every affected customer was instead unblocked through the ACS URL registration, and we are working with Okta to make the ACS URL validation conditional again, so that instances with no additional ACS URLs configured behave exactly as they did before, and no customer action is required.

   Action Items

   To prevent such issues from happening again, Harness will take the following action items

   ‌

   1. Working with Okta to restore conditional ACS URL validation, so instances with no additional ACS URLs configured behave exactly as before and no customer action is required.
   2. Publish a dedicated configuration guide for the Harness OIN application in Harness documentation, including reply (ACS) URL requirements.
   3. Add an independent pre-release verification step against existing customer configurations for any future update to the published Okta application, alongside Okta's own review.
   4. Migrate user authentication to Harness ID, which removes the dependency on this validation behavior entirely.

Reported by Harness on their status page.

1. postmortemOct 7 · 18:30 UTC

   1. Summary

   Starting September 29, 2026, customers who sign in to Harness through the published Harness application in the Okta Integration Network (OIN) were unable to complete SP-initiated SSO login. Affected users received an HTTP 400 error from Okta instead of being signed in.

   The issue was introduced by a change in how Okta validated the requested reply (ACS) URL for existing installations of the published Harness application. Workarounds were published within an hour of the first report, and every affected customer was unblocked.

   2. Impact

   ‌

   • Impact was limited to customers using the published Harness application from the Okta Integration Network for SP-initiated login (starting sign-in from Harness). Those attempts returned an HTTP 400 error from Okta.
   • IdP-initiated login, started from the Harness tile in Okta, continued to work throughout.
   • Customers using their own generic SAML 2.0 application configuration in Okta were not affected.
   • No customer data was lost or exposed.

   3. Root Cause

   On September 29, 2026, an updated version of the Harness application was published to the Okta Integration Network. The update added support for registering multiple reply (ACS) URLs, so that customers running Harness across several regions or environments could configure more than one callback URL.

   Harness includes an AssertionConsumerServiceURL value in the SAML authentication request it sends to Okta. Before this update, Okta did not validate that value for application instances that had no additional ACS URLs configured, and simply returned the SAML assertion to the application's default ACS URL. After the update, this validation became active on all existing instances of the application, including those where the customer had never configured an additional ACS URL. Because the requested value was not present in the instance's registered list, Okta rejected the authentication request with an HTTP 400 error.

   The updated application went through Okta's standard OIN publishing review, which includes backward compatibility verification against existing installations of a published integration. This change in validation behavior for instances with no additional ACS URLs configured was not surfaced during that review, so it reached customers at release.

   4. Remediation

   Immediate Response

   Two workarounds were published on the status page: sign in from the Harness tile in Okta (IdP-initiated login), which was unaffected; or register the application's Okta SSO URL in the Additional ACS URLs field of the Harness Okta application. Support engaged each affected customer directly and validated sign-in with them.

   ‌

   Resolution

   Rolling the application back to the previous version was deliberately avoided, as that would only defer the same disruption to the next release. Every affected customer was instead unblocked through the ACS URL registration, and we are working with Okta to make the ACS URL validation conditional again, so that instances with no additional ACS URLs configured behave exactly as they did before and no customer action is required.

   5. Action Items

   To prevent such issues from happening again, we are working on

   Action Item
   Working with Okta to restore conditional ACS URL validation, so instances with no additional ACS URLs configured behave exactly as before and no customer action is required.
   Publish a dedicated configuration guide for the Harness OIN application in Harness documentation, including reply (ACS) URL requirements.
   Add an independent pre-release verification step against existing customer configurations for any future update to the published Okta application, alongside Okta's own review.
   Migrate user authentication to Harness ID, which removes the dependency on this validation behavior entirely.

Reported by Harness on their status page.

1. postmortemOct 2 · 22:40 UTC

   Summary

   Between approximately 11:37 AM and 3:01 PM PDT (18:37–22:01 UTC) on September 29, 2026, customers using Harness CI self-hosted Linux cloud runners in prod1/prod2 (us-west1) experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code.

   Root Cause

   This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there.

   Impact

   • Customers running self-hosted Linux CI builds in us-west1 saw intermittent build/pipeline failures, not a full outage.
   • Observed customer-facing errors included Git fetch timeouts, e.g. Failed to connect to github.com port 443: Connection timed out

   Remediation

   • Immediate: Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 (partially limited by central1 capacity); restarted the affected Cloud NAT gateways, which stabilized pipeline networking.

   Action Items

   1. Secure additional NAT gateway/IP capacity (In progress)
   2. Improve NAT saturation detection and alerting ahead of customer impact.
   3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.

Reported by Harness on their status page.

1. postmortemOct 2 · 05:37 UTC

   Summary

   Customers on Prod-2 experienced elevated failure rates in Continuous Verification pipelines. Verification jobs failed to complete or reported errors because CV data-collection workers could not communicate with the platform service that coordinates their work. This was caused by an unexpected traffic spike to one of our backend systems. 

   Service was restored by scaling out the affected platform component. No data loss or corruption occurred.

   ‌

   Customer Impact

   Customers using Continuous Verification on Prod-2 saw verification pipeline jobs fail or time out during the incident window. A significant fraction of verification job instances that ran during this period ended in an operational failure. Pipelines not using CV continued to operate normally.

   ‌

   Root Cause

   A large automated provisioning run created a significant number of persistent CV monitoring workers in a short window, which overwhelmed a shared platform component.

   The sudden increase in concurrent outbound calls exhausted the available network connections on each service instance.  

   ‌

   Mitigation

   The maximum scaling limit for the CV service was raised, allowing it to expand capacity to meet demand. Failures stopped shortly after the scale-out was completed.

   ‌

   Next Steps

   To prevent recurrence, Harness will:

   • Increase capacity headroom and add provisioning guardrails: Raise the CV service scaling limit to maintain headroom under burst loads, and introduce rate limiting for bulk provisioning of monitored services.
   • Improve Resiliency: Add exponential backoff and jitter for worker retries, and use a reusable connection pool to reduce overhead and prevent transient failures from compounding under high concurrency.
   • Improve monitoring and alerting: Enhance alerting for abnormal per-account worker growth rates, so similar conditions are detected and acted on earlier.

Reported by Harness on their status page.

1. postmortemOct 13 · 04:50 UTC

   .

Reported by Harness on their status page.

1. postmortemOct 2 · 22:05 UTC

   Summary

   On September 22, 2026, three of our outbound network gateways in one region became unhealthy and began rejecting a small percentage of new outbound connection attempts. The issue was resolved the same day problem completely with no further recurrence or follow-on impact.The underlying cause was a saturated internal traffic-monitoring buffer on those gateways, which caused their health checks to respond too slowly and get marked unhealthy. 

   Impact

   • Only three gateway instances in one region were affected; other regions and other gateways in the same region were not impacted.
   • There was no cascading failure , the issue was fully resolved with a single round of gateway restarts, and no follow-on issues occurred afterward.

   Root Cause 

   Our outbound network gateways run an internal component that monitors network traffic for operational visibility. Under sustained high traffic, this component's internal buffer became saturated, which created backpressure severe enough to slow down the gateway's own health-check responses beyond the timeout our load balancer allows. Once that timeout was exceeded, the load balancer marked the affected gateways unhealthy and began rejecting a portion of new connections destined for them.

   Actual packet forwarding continued to work normally throughout the incident. This was specifically a health-check and monitoring-buffer issue, not a failure of the gateways' core forwarding capability.

   Resolution

   The three affected gateway instances were manually restarted, which cleared the saturated monitoring buffer and immediately restored normal health-check behavior. Connection rejections stopped as soon as the restart completed, and no further intervention was needed.

   Preventive Actions

   Action
   Replace the current traffic-monitoring approach on all gateways with a more efficient, lower-overhead implementation that cannot saturate in this way, and that alerts well before any customer-visible impact.
   Implement automatic, periodic replacement of long-running gateway instances on a fixed schedule, so no instance can accumulate the kind of age-related degradation seen in this incident.
   Migrate gateway instances to network-optimized infrastructure with substantially more dedicated network capacity, better suited to sustained high-throughput traffic.
   Enhance existing monitoring buffer's utilization while the longer-term replacement is rolled out.
   Enhance alerting specifically on gateway health-check response latency, to catch early signs of degradation before it becomes customer-visible.

Reported by Harness on their status page.

1. postmortemOct 2 · 22:10 UTC

   Same as the previous root cause.

Reported by Harness on their status page.

1. postmortemOct 2 · 22:08 UTC

   This was a customer specific issue.

Reported by Harness on their status page.

1. postmortemAug 7 · 19:32 UTC

   Summary

   CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways.

   Impact

   • CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services (e.g., GitHub, Bitbucket) over our outbound network gateways.
   • The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume.
   • No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue.
   • us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue.

   Root Cause

   ‌

   Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details (source/destination address and port). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways

   ‌

   Action Items

   To prevent such issues from happening again Harness will,

   Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..

Reported by Harness on their status page.