Alchemy Incident History

This incident has been resolved.

We are continuing to investigate this issue.

A subset of requests on Hyperliquid Mainnet routed through our Asia Pacific region are seeing increased delays and elevated error rates.

This incident has been resolved.

A subset of eth_getLogs requests on Monad Mainnet routed through our EU Central region are seeing increased delays.

This incident has been resolved.

We are continuing to investigate this issue.

We are investigating degraded performance affecting requests on Sui Mainnet.

Between 22:35 and 22:55 UTC on July 30, a subset of requests on Robinhood Mainnet experienced elevated latency. This incident has been resolved.

This incident has been resolved.

A subset of eth_sendRawTransaction requests on Hyperliquid Mainnet are seeing increased delays for requests routed through our EU Central region.

Between 13:30 UTC on July 28 and 08:00 UTC on July 29, HYPERLIQUID\_MAINNET requests served from us-east experienced elevated latency, and `eth_sendRawTransaction` returned a high rate of errors. Read methods remained available throughout, with elevated response times. **Impact** Transaction submission was the primary failure mode. During the peak period, the large majority of `eth_sendRawTransaction` requests either timed out or returned an error, and clients retrying submissions saw sustained failures. Read traffic on the network was slower than normal but continued to be served successfully. Latency was elevated for all HYPERLIQUID\_MAINNET request types for the duration of the event. **Root cause** A majority of our Hyperliquid full nodes in us-east lost peering with the network. The nodes were unable to complete state synchronization handshakes with upstream peers -- connection attempts were rate limited, timed out, or rejected because peer connection slots were full. Affected nodes fell behind and were removed from the healthy serving pool. Recovery required the upstream peer pool to free capacity; feeding new peers into the configuration and pruning node state did not restore peering. Two factors extended the impact: * With that capacity gone, all us-east traffic concentrated on the remaining archive node pool, which pushed latency up further and made `eth_sendRawTransaction` the first method to fail. * On Hyperliquid, transactions are not propagated peer-to-peer. They are relayed to a network-level submission endpoint that can rate limit, which means the transaction submission path is sensitive to network conditions independent of which infrastructure serves the request. **Resolution** We began routing `eth_sendRawTransaction` to secondary infra as soon as the issue was identified, but this did not restore transaction submission. Submissions relayed through secondary infra continued to fail at a comparable rate, since that path depends on the same network-level submission endpoint. A subset of nodes began to recover from around 01:30 UTC on July 29, when the first `hlnode` in us-east returned to a healthy state as upstream peer capacity freed up. Server-side errors from our serving layer dropped off around the same time, though submissions continued to fail through a mix of remaining unhealthy nodes and rejections at the network's submission endpoint. Peering re-established across the rest of the pool over the following hours. Latency returned to normal levels by approximately 08:00 UTC, request volumes and error rates returned to baseline, and all us-east nodes were confirmed healthy shortly afterward. **Follow-up** We are exploring running our own validator and sentry nodes for Hyperliquid so that peering no longer depends on availability in the shared public peer pool.

This incident has been resolved.

Some requests to Hyperliquid Mainnet routed through our US East region may be experiencing elevated latency. We are investigating.

This is resolved. Base Mainnet's Safe head has caught up and data batching to Ethereum L1 is back to normal. Block production was unaffected throughout. See the Base status page for their full update: https://status.base.org/incidents/j6y12tshwkcg

This is a network-wide issue on Base Mainnet, not specific to Alchemy. The Safe head is delayed because data batching to Ethereum L1 is lagging. New block production on Base Mainnet is progressing normally, and requests against the latest block are unaffected. Only queries and workflows that depend on the safe or finalized block tag will see staleness. Refer to the Base status page for the latest updates: https://status.base.org/incidents/j6y12tshwkcg

We are investigating a network-wide stall affecting safe and finalized blocks on Base Mainnet.

Nodes have recovered and services are healthy again.

Nodes were unable to progress, team carried out a restart of the nodes and are seeing recovery and improved performance on requests. Actively monitoring.

A subset of requests on Stable Mainnet are experiencing degraded

Nodes have been rescyned with stable binary and services are back to healthy.

A part of our Cronos nodes were down due to a binary issue.

We are continuing to investigate this issue.

A subset of requests on Cronos Mainnet are experiencing availability issues.

This incident has been resolved.

Full nodes (at head) are seeing good traffic and stable performance! Archive nodes are still being worked on - ETA 1 hour if things go well. We'll continue to provide updates as we make progress!

The team is aware Stable Mainnet nodes are down, actively investigating and recovering the nodes via snapshots.

A subset of requests on Stable Mainnet are experiencing degraded performance.

This incident has been resolved.

A subset of requests on Robinhood Mainnet routed through our Asia Pacific region may be experiencing increased latency.

Traffic has been fully restored to the US West region. This incident is now resolved!

We identified the root cause and the affected service has been failed over. We are monitoring for full recovery!

We're investigating an issue impacting WebSocket connectivity in US West region.

Resolved: All impacted networks, including Mode Sepolia (testnet), have fully recovered and caught up. This incident is now resolved.

Update: All impacted networks have recovered except Mode Sepolia (testnet), which is still degraded. We are continuing to monitor Mode Sepolia.

Update: Zora has recovered. Mode mainnet has recovered; Mode Sepolia (testnet) is still degraded. Ronin, BOB, Codex, and Superseed remain impacted. We are continuing to monitor.

We are investigating a network-wide stall affecting Zora, Ronin, BOB, Codex, Mode, and Superseed networks.

We are investigating a network-wide stall affecting Zora, Ronin, BOB, Katana, Mode, Metal, and Superseed networks.

We have confirmed full recovery across our BNB Smart Chain Mainnet infrastructure. All nodes are healthy and serving traffic normally, with response times and error rates back to baseline. This incident is now resolved. Thank you for your patience!

We experienced node-level failure on BNB Mainnet which caused a subset of our nodes to become unavailable. As a result, increased load was placed on our remaining infrastructure, further contributing to degraded response times. A node binary fix was deployed and we are seeing recovery.

We are investigating elevated error rates and increased latency on BNB Smart Chain Mainnet. Some requests may fail or respond slower than usual. Our team is actively working on a fix and we will post updates here as we learn more.

This incident has been resolved.

We are currently investigating an issue impacting transaction propagation on Soneium Mainnet.

Between approximately 23:20 UTC on July 17 and 03:30 UTC on July 18, some Polygon Mainnet requests experienced slower than normal response times, mainly for log queries. The issue was limited to the us-east-1 region. The cause was an unusually heavy request load that put pressure on part of our infrastructure in the region. We added extra capacity and rebalanced traffic, after which performance returned to normal. The issue is fully resolved. We apologize for the disruption.