Sequencing Delay on GIWA Sepolia Testnet
Last updatepostmortemSep 3 · 04:54 UTC
## Overview On 26 August 2026 at 04:44 UTC, the leader sequencer on GIWA Sepolia testnet began responding slowly to block construction requests. Requests repeatedly exceeded the time allotted for building a block, block intervals widened, and blocks produced in this state did not include user transactions. As block intervals widened, chain time fell behind wall-clock time. Once that gap moved outside the conditions for peer-to-peer block propagation, blocks produced by the leader were no longer propagated to the network. Nodes other than the sequencer stopped tracking the chain head, and RPC responses and the block explorer remained fixed at a stale state. **Block production itself did not stop.** Block numbers were continuous throughout, and **there was no reorg, no safe-head or finality violation, and no funds were at risk.** ## Impact - **Transaction processing** — Most blocks produced during this window did not include user transactions. Submitted transactions were accepted but were delayed before being included in a block. - **Public RPC endpoints** — Reads continued to be served, though against a stale state. - **Block explorer** — Indexing fell behind, so recent blocks and transactions were not displayed. - **L1 batch submission** — Unaffected. Blocks that were produced were posted to L1. ## Timeline - **04:44:17** — The last block under normal conditions was committed. Block intervals were normal up to this point. - **04:44:18** — The leader sequencer's response to a block construction request exceeded its time limit. That block production attempt was voided. - **04:44:19–25** — In line with the high-availability configuration, leadership transferred to another sequencer and then returned to the original one. The same symptom was observed on the sequencer it transferred to. - **04:45 onward** — Widened block intervals persisted, and blocks continued to be produced without user transactions. - **04:46:02** — The gap between chain time and wall-clock time moved outside the conditions for peer-to-peer propagation. From this point, blocks produced by the leader were not propagated to the network, and other nodes stopped tracking the chain head. - **04:57** — We became aware of the incident and began investigating. - **05:00–06:10** — We narrowed the affected area while applying mitigations. During this period block production recovered temporarily and then degraded again, repeatedly. - **06:13–06:18** — Mitigations took effect, the backlog of blocks began propagating, and nodes caught up to the chain head. - **06:24** — Per-block transaction throughput returned to normal. ## Root Cause ### 1. Trigger — Slow response to block construction requests (root cause not isolated) A GIWA sequencer must return a block within a fixed time after receiving a block construction request. That time is set in relation to the block time, and if it is exceeded the block production attempt is voided. From 04:44:18 UTC, these responses began to slow. Response time degraded to roughly ten times its normal value, and about a quarter of requests exceeded the limit. Responses were within normal range up until the immediately preceding block, with no sign of gradual deterioration. **We were not able to isolate the root cause of the slowdown.** Our investigation ruled out the following against measured data. - **Resource exhaustion** — Compute, memory, and disk utilisation were all within normal range at the onset of the incident. - **Increased load** — Per-block transaction counts and transaction inflow were unchanged immediately before and after onset. - **Block size** — Blocks of the same size were processed within normal time on many occasions under normal conditions. - **Hardware fault on a single node** — The same symptom appeared after leadership transferred to a different sequencer. Records that would have shown which internal step was delayed were not being captured. We have expanded instrumentation so that this can be determined if the condition recurs. ### 2. Voided block production attempts accumulated delay When a block construction response exceeds its time limit, that block production attempt is voided and the sequencer retries the same block. Through this process, actual block intervals widened beyond the block time. Meanwhile, the time recorded in a block is a fixed value that advances from the previous block by the block time. Consequently, when actual block intervals exceed the block time, **chain time falls behind wall-clock time by that difference.** In this incident, that difference accumulated continuously. Blocks produced in this state also did not include user transactions. When the chain has fallen behind, there is no time available to fill a block, so production follows a path that yields a minimally populated block. This persisted until the condition was resolved. ### 3. Node sync stalled once propagation conditions were no longer met Peer-to-peer block propagation applies a condition intended to prevent re-propagation of stale blocks. If the time recorded in a block is more than a certain margin behind the current time, that block is excluded from propagation. At 04:46:02 UTC, the accumulated difference exceeded that margin. From that point, **blocks produced by the leader were not propagated to the network.** Other nodes received no new blocks and stopped tracking the chain head, which is the direct reason RPC and the block explorer remained fixed at a stale state. No self-recovery path exists in this state. The time recorded in a block is inherited from its predecessor, so restarting components or replacing the leader does not narrow a gap that has already opened. In practice there were periods after 05:00 when block production speed recovered temporarily, but because propagation was blocked, this was not observable externally. **This is also why the high-availability configuration did not help during this incident.** Chain progression is reflected in whether a sequencer is judged healthy, so when the chain has fallen behind, any sequencer elected as leader is judged the same way. Because this was a condition shared across all sequencers rather than a fault on one node, replacing the leader did not resolve it. ## Remediation **Recovery action (around 06:10 UTC)** — We adjusted block propagation conditions so the backlog of blocks could propagate, and increased resource allocation for the sequencer execution layer. The backlog then propagated to nodes, chain head tracking resumed, and transaction throughput returned to normal at 06:24 UTC. The following work has been applied. - **Guaranteed transaction processing under delayed conditions** — Adjusted the block production path so that blocks continue to include transactions even when the chain has fallen behind. - **Recalculated block propagation conditions** — Applied uniformly across all nodes with a value that covers the maximum delay observed in this incident. Block propagation is maintained when delay occurs, so nodes can continue tracking. - **Expanded instrumentation for the block construction path** — Enabled detailed measurement so the delayed step left undetermined in section 1 can be identified if the condition recurs. - **Reviewed the high-availability configuration** — Adjusted health criteria so that standby sequencers remain in a normal state while leadership transfer is in progress. The purpose of this work is not to eliminate the delay itself but to ensure that when it occurs, **transaction processing and block propagation are maintained and nodes can catch up automatically.** We intend to determine the root cause if the condition recurs, using the expanded instrumentation. ## If You Operate a Node If you run a GIWA node yourself, note that during this incident blocks produced by the sequencer were not propagated over peer-to-peer, so **chain head tracking fell behind regardless of node-side configuration.** Once propagation resumed, nodes caught up automatically without any action required. For normal operation, we recommend maintaining stable execution layer peer-to-peer connectivity. Obtaining blocks depends on the execution layer peer-to-peer network, so recovery may be slower under delayed conditions if your connection targets are restricted. ## Notes - GIWA Sepolia is a test network intended for testing and validation. Its characteristics and terms of use are defined in the [Testnet Terms of Use](https://docs.giwa.io/terms-and-policies/testnet-terms-of-use). - Network status and incident history are available at [status.giwa.io](https://status.giwa.io), where you can also subscribe to updates. - The impact window above reflects the full incident duration and is not specific to any individual user or service. Actual impact varied depending on the endpoint used and how it was queried.
Reported by GIWA on their status page.