Buildkite logo and current status indicator

Buildkite Incident History

Operational

buildkite.com ↗12 top-level components

checked Oct 27, 2026 10:52 AM UTC · Buildkite's official status page

Buildkite is up and running.

Buildkite is currently operational with all systems functioning normally.

All components operational

Be the first to know

Statusfield watches Buildkite alongside the rest of your stack and tells you when its status changes. Email, Slack, Microsoft Teams, PagerDuty, and webhooks.

30-day free trial — no card needed. First-time customers; continued monitoring requires a paid subscription.

Buildkite incident history

15 reports published by Buildkite in the last 30 days

When Buildkite breaks, it is typically resolved in 1h 4m — median across 4 resolved incidents over 30 days.

October 26, 2026

Elevated API latency and error rates

Last update
  1. resolvedOct 26 · 17:07 UTC

    Between 17:07-17:19 UTC, our API endpoints experienced elevated latencies and errors due to issues with a shared service. Our monitoring systems detected these issues, which triggered automated system remediations and resulted in the service self-healing. Although our APIs are once again stable, we are continuing to investigate the cause of this very brief service degradation.

Reported by Buildkite on their status page.

October 24, 2026

Elevated error rates

Last update
  1. resolvedOct 24 · 23:40 UTC

    Between 23:16 and 23:27 we saw elevated error rates across our platform due to a scheduled database upgrade. That upgrade was not intended to cause any impact, but due to an unforeseen dependency on this database customers experienced error rates averages 2.5% during this period.

Reported by Buildkite on their status page.

October 22, 2026

Delays with Hosted Agents jobs

Last update
  1. resolvedOct 22 · 17:37 UTC

    Processing of Hosted Agents jobs are no longer delayed.

Reported by Buildkite on their status page.

October 20, 2026

Buildkite response to the eMudhra certificate compromise

Last update
  1. resolvedOct 21 · 22:38 UTC

    All services that power Buildkite have been updated to distrust the affected certificates, and updated releases are available for the components we distribute that run in customer infrastructure: Buildkite agent: v4.7.0 and v3.138.2 Buildkite CLI: v3.64.2 bktec: v3.3.1 elastic-ci-stack-for-aws: v7.8.0 agent-stack-k8s: v0.52.0 We recommend updating to these versions to prevent exposure to the affected certificate providers.

Reported by Buildkite on their status page.

Delayed notifications

Last update
  1. resolvedOct 20 · 21:05 UTC

    Adding additional capacity has resolved the notifications latency.

Reported by Buildkite on their status page.

October 19, 2026

Delayed Hosted Agents instance creation

Last update
  1. resolvedOct 19 · 15:45 UTC

    We received reports from a few customers that Hosted Agent jobs were delayed in starting. We investigated and found this related to a GitHub incident with webhooks that caused throttling in Hosted Agent job creation once webhooks were once again being ingested. Any affected Hosted Agents customers would have seen delays in instance creation between 15:45-17:12UTC.

Reported by Buildkite on their status page.

October 6, 2026

Degraded checkout performance with Hosted Agents

Last update
  1. resolvedOct 6 · 23:59 UTC

    The issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents.

Reported by Buildkite on their status page.

October 1, 2026

Small number of incoming GitHub Webhook failing

Last update
  1. resolvedOct 1 · 06:57 UTC

    The error rate has returned to normal levels.

Reported by Buildkite on their status page.

Small number of incoming GitHub Webhook failing

Last update
  1. ResolvedOct 1 · 06:57 UTC

    The error rate has returned to normal levels.

Reported by Buildkite on their status page.

September 30, 2026

Delayed notifications

Last update
  1. resolvedSep 30 · 02:32 UTC

    Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved.

Reported by Buildkite on their status page.

Delayed notifications

Last update
  1. ResolvedSep 30 · 02:32 UTC

    Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved.

Reported by Buildkite on their status page.

September 28, 2026

macOS hosted agent jobs not starting

Last update
  1. resolvedSep 28 · 01:30 UTC

    Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.

Reported by Buildkite on their status page.

macOS hosted agent jobs not starting

Last update
  1. ResolvedSep 28 · 01:30 UTC

    Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.

Reported by Buildkite on their status page.

September 22, 2026

Issues with webhook ingestion, GraphQL queries

Last update
  1. postmortemSep 28 · 04:12 UTC

    Service Impact

    On September 22, 2026, between 19:09 and 19:32 UTC, customers experienced failures ingesting webhooks and executing GraphQL queries. Some web application requests also failed or timed out.

    During this period, 2.0% of requests to our webhook ingestion endpoints and 20.3% of requests to our external GraphQL endpoint returned HTTP 500 errors.

    Webhook ingestion errors ceased by 19:30 UTC, and external GraphQL errors returned to background levels by 19:32 UTC.

    Incident Summary

    While investigating a separate API performance issue, we made a manual configuration change intended to rollback a routing change for one database proxy. The command instead changed the routing configuration for another database proxy, disrupting access to a database shard used internally by Buildkite.

    Some customer-facing operations search across all database shards to locate their shard, where routing information isn't available in the request. Those requests round robin through available shards, and could encounter the unavailable shard and wait for a database connection to time out. This extended the impact beyond our internal workloads.

    We reverted the incorrect configuration and verified that the database proxy had healthy endpoints and load-balancer targets.

    Two contributing factors allowed the change to cause an outage: the command was not independently checked before execution, and our infrastructure accepted a Service configuration that selected the wrong database proxy workload. The cross-shard lookup behaviour then allowed the disruption to affect requests for other customers.

    This configuration-change outage was separate from the API performance issue we were originally investigating.

    Changes we're making

    • Safer operational procedures: We have updated our runbook and introduced an explicit second person review requirement for manual production commands.
    • Automated safeguards: We’ve added validation to prevent database proxy routing changes from selecting the wrong workload.
    • Improved failure isolation: We’ve been working on an ongoing initiative to make database shard discovery more resilient, eliminating the remaining cases where an unavailable shard may impact requests for other customers.

Reported by Buildkite on their status page.

Issues with webhook ingestion, GraphQL queries

Last update
  1. ResolvedSep 22 · 20:50 UTC

    Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC.

Reported by Buildkite on their status page.