Harness Incident History

This incident has been resolved.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

## Summary On July 14, 2026, certain Harness Feature Flag \(FF\) Classic customers on the Prod1 and Prod2 production environments were unable to view Feature Flags in the Harness platform. Affected requests returned an authorization error \(HTTP 403\), so Feature Flags were not visible for those users until access was restored. The behaviour was caused by a planned security update that began requiring an additional Feature Flags permission for related read operations. Users whose roles did not yet include that permission were correctly denied access, which appeared as a product failure. Harness temporarily rolled back the Feature Flags service change to restore access, updated the required permissions for affected users and roles, and confirmed that Feature Flag visibility returned to normal. The stronger permission checks remain in place going forward. ## Impact During the customer-reported incident window on July 14, 2026 \(status page approximately 11:25 UTC to 11:57 UTC\): * Certain Feature Flag Classic customers on **Prod1** and **Prod2** were impacted. * Affected users could not view Feature Flags in the Harness UI / API and received **403 Forbidden** responses. * Impact was limited to users and roles that did not yet have the updated Feature Flags permission required by the security change. * A public status page update was posted for Prod1 and Prod2 Feature Flags. There was **no data loss**, no change to stored feature flag configurations, and no impact to Feature Flag evaluation for applications whose SDK credentials and permissions were unaffected. Customers and users with the required permission continued to operate normally. ## Root Cause Harness deployed a planned Feature Flags authorization update that enforces the `ff_targetgroup_view` permission on target-related read paths used when viewing Feature Flags. This enforcement is intentional and remains the expected behaviour. Users and roles that had not yet been granted `ff_targetgroup_view` received 403 responses and could not see Feature Flags. From the customer’s perspective this looked like an outage; it was an authorization denial due to missing required permissions after the security update. ## Mitigation Harness completed the following mitigation steps: * Temporarily rolled back the Feature Flags service in Prod2 and Prod1 \(and aligned Prod0\) to restore access quickly while permissions were corrected. * Updated / granted the required `ff_targetgroup_view` permission for affected users and roles. * Verified that Feature Flag visibility returned to normal and closed the status page incident. These actions restored customer access. The permission enforcement introduced by the security update **remains in effect** going forward; the lasting fix is correct permission assignment, not removal of the check. ## Preventive Actions To avoid a recurrence of this issue, Harness is taking the following actions: * Keeping the stronger Feature Flags permission checks in place as the permanent security posture. * Ensuring role and permission updates for `ff_targetgroup_view` are applied with \(or before\) similar authorization changes in production.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Certain users are unable to see feature flags in prod1 and prod2

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are investigating an issue impacting AIDI dashboards. Users may experience increased load times or intermittent failures when accessing dashboards. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available.

# Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

# Summary Between June 19 and July 17, 2026, the built-in Git Clone step — and any pipeline step using the drone-git clone plugin — failed on ARM64 Kubernetes build infrastructure with the error exec /usr/local/bin/clone: exec format error. AMD64 \(Intel/AMD\) builds, Windows builds, and the VM containerless binary path were not affected. The root cause was a defect in our internal image publishing process that caused ARM64-tagged drone-git images to actually contain AMD64 binaries. We identified and mitigated the issue the same day it was reported by reverting the drone-git image to the last known-good version. No customer action or configuration change was required. # Root Cause On June 19, 2026, a security remediation restructured how the drone-git image is built. AMD64 builds were updated correctly, but the ARM64 build pipeline didn't build ARM64 files directly — it adapted the AMD64 build file via text substitution and compiled it on ARM64 infrastructure. The June 19 change altered the AMD64 file so that substitution silently no-op'd instead of failing, so the pipeline published an image tagged ARM64 whose Git Clone and Git LFS binaries were still compiled for AMD64. # Impact * Affected: The built-in Git Clone step, and any pipeline step using the drone-git clone plugin, running on ARM64 Kubernetes build infrastructure, across all accounts, between July 6 and July 17, 2026. * Symptom: Builds failed at the Git Clone step with exec /usr/local/bin/clone: exec format error. * Not affected: AMD64 \(Intel/AMD\) Kubernetes and VM builds, Windows builds, the VM containerless execution path, and our hardened image variant. # Mitigation We reverted the drone-git image version used across all affected services to the last known-good release. This fully resolved the ARM64 execution failures; no customer configuration changes were required. # Next Steps To prevent such issues from happening again. * Rebuild the ARM64 image publishing pipeline to build our dedicated ARM64 build files directly, rather than adapting the AMD64 build files. * Enhance automated post-publish validation to every image release: verify binary architecture matches the image tag, and run a functional smoke test before an image is considered releasable. * Expand automated test coverage to include ARM64 Kubernetes build scenarios. * Remove the affected intermediate image versions from circulation once the corrected release is validated.

This incident has been resolved.

We are currently investigating this issue.

# Summary On July 9, 2026, some customers in the Harness Prod3 cluster experienced pipeline failures over a ~2.5-hour window, despite making no changes in their pipelines. Shell Script steps failed when fetching scripts from the Harness File Store, and Kubernetes deployment steps failed when retrieving secret encryption details. # Impact 1. **Affected users**: Some customers using the Harness Prod3 cluster reported pipeline execution failures. 2. **User impact:** Customers experienced failures in Shell Script and Kubernetes deployment steps. Impacted pipelines required manual retries after the incident was resolved. 3. **Scope:** This was not a platform-wide outage. The issue was isolated to specific APIs in our internal service within the Prod3 cluster. # Root Cause The incident was caused by degraded performance in a downstream service. This led to a storm of timeout errors in our internal service, which caused these failures. # Mitigation Engineering identified the change that caused this issue in the downstream dependency service and deployed a patch to restore it. Once deployed, pipeline executions returned to normal success rates. # Preventive Measures & Next Steps 1. Added additional monitoring and alerting on the downstream service to detect performance degradation earlier. 2. Reviewing timeout and retry logic in the internal service to improve resilience against downstream latency spikes. 3. Evaluating circuit breaker patterns to prevent cascading failures from downstream dependencies.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

## Summary Between July 7 and July 8, 2026, customer accounts hosted in our Prod 3 environment experienced intermittent slowness and failures when loading pipelines and resolving templates. The underlying cause was a sharp, sustained increase in internal feature-flag lookup traffic from our template-processing service to our core platform backend service. The issue presented across two separate days. On Day 1 \(July 7\), our team restored service through infrastructure level mitigations by restarting affected services and adding capacity. Because that underlying cause was still present, the same failure mode recurred on Day 2 \(July 8\). Deeper investigation on Day 2 identified the specific feature flag and code path responsible, and engineering shipped a fix so that flag check is now served from local cache. This fix was deployed across all production clusters and fully resolved the issue; it has not recurred since We are auditing all other feature flags on the platform for the same caching gap, and are implementing process and monitoring improvements described in the Preventive Measures section below to catch this class of issue earlier and reduce the risk of recurrence. ## Impact **Symptoms:** Intermittent slowness or failure loading pipelines; slow or failed template validation and resolution; intermittent connectivity/timeout errors between platform services. **Functional impact:** Pipeline executions and pipeline-studio operations that depend on template resolution were delayed or failed during the active windows described in the timeline above. Some executions required manual retry ## Root cause Our core backend service could not keep up with the request arrival rate because of a sustained, uncached feature-flag lookup driving a large and growing volume of internal traffic following a recent release. ‌ ## Mitigation * Immediate mitigation \(Day 1 and Day 2\): removed the saturated backend instance\(s\) from service and added capacity to relieve the acute bottleneck. * Permanent fix \(Day 2\): updated the template-processing service so the responsible feature-flag check reads from local cache instead of calling the backend service on every lookup, eliminating the excess internal traffic at its source. * The fix was deployed to all production clusters and verified through a return to normal internal traffic volumes and response times. ## Preventive Measures & Next Steps * Feature-flag caching audit: We are reviewing all feature flags used in high-frequency code paths across the platform to identify and close any similar caching gaps before they can cause a repeat of this issue. * Release validation: We are strengthening pre-release load testing for new feature-flag-gated code paths so that call-volume regressions of this kind are caught before reaching production. * Capacity alerting: We are adding proactive alerting on backend request-queue saturation so that this class of bottleneck is caught and mitigated automatically, before it affects customer-facing response times.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue in Prod3 cluster where Harness platform components are loading slower than expected.

## Summary Between July 7 and July 8, 2026, customer accounts hosted in our Prod 3 environment experienced intermittent slowness and failures when loading pipelines and resolving templates. The underlying cause was a sharp, sustained increase in internal feature-flag lookup traffic from our template-processing service to our core platform backend service. The issue presented across two separate days. On Day 1 \(July 7\), our team restored service through infrastructure level mitigations by restarting affected services and adding capacity. Because that underlying cause was still present, the same failure mode recurred on Day 2 \(July 8\). Deeper investigation on Day 2 identified the specific feature flag and code path responsible, and engineering shipped a fix so that flag check is now served from local cache. This fix was deployed across all production clusters and fully resolved the issue; it has not recurred since We are auditing all other feature flags on the platform for the same caching gap, and are implementing process and monitoring improvements described in the Preventive Measures section below to catch this class of issue earlier and reduce the risk of recurrence. ## Impact **Symptoms:** Intermittent slowness or failure loading pipelines; slow or failed template validation and resolution; intermittent connectivity/timeout errors between platform services. **Functional impact:** Pipeline executions and pipeline-studio operations that depend on template resolution were delayed or failed during the active windows described in the timeline above. Some executions required manual retry ## Root cause Our core backend service could not keep up with the request arrival rate because of a sustained, uncached feature-flag lookup driving a large and growing volume of internal traffic following a recent release. ‌ ## Mitigation * Immediate mitigation \(Day 1 and Day 2\): removed the saturated backend instance\(s\) from service and added capacity to relieve the acute bottleneck. * Permanent fix \(Day 2\): updated the template-processing service so the responsible feature-flag check reads from local cache instead of calling the backend service on every lookup, eliminating the excess internal traffic at its source. * The fix was deployed to all production clusters and verified through a return to normal internal traffic volumes and response times. ## Preventive Measures & Next Steps * Feature-flag caching audit: We are reviewing all feature flags used in high-frequency code paths across the platform to identify and close any similar caching gaps before they can cause a repeat of this issue. * Release validation: We are strengthening pre-release load testing for new feature-flag-gated code paths so that call-volume regressions of this kind are caught before reaching production. * Capacity alerting: We are adding proactive alerting on backend request-queue saturation so that this class of bottleneck is caught and mitigated automatically, before it affects customer-facing response times.

This incident has been resolved.

A fix has been implemented in one of our microservice responsible for dealing with pipeline templates which has resolved the issue. We are continuing to monitor the issue.

We are currently investigating an issue in Prod3 cluster where pipeline components are loading slower than expected.