Grandine cannot cross the Gloas fork: a zero-count envelope livelock and an unrelated OOM

All ten Grandine nodes on glamsterdam-devnet-9 are stuck at head slot 7199 — the last pre-Gloas slot — and have never imported a single post-fork block. Two independent defects are in play: a sync livelock in which Grandine issues an ExecutionPayloadEnvelopesByRange request with count: 0 for the empty fork-boundary slot 7200 and retries it forever, and a hard OOM in which resident memory jumps from a flat ~12 GB plateau to 25–30 GB inside a single five-second interval, killing the process roughly every two minutes. The evidence argues against the two being causally linked. A third, contributing problem: every Grandine host is running the :glamsterdam-devnet-8 image, not the :develop trunk the devnet-9 spec sheet calls for — and that mutable tag silently moved twice during the run.

What we observe

Grandine is not partitioned and not misconfigured at the network layer. grandine-reth-1 holds 193 peers and advertises the correct fork digest 0x6ebb965e — the same digest its healthy peers advertise. It simply never advances:

WARN validator::validator:739: too many empty slots after head: 7199 + 32 < 12299

The beacon API confirms nothing past the fork boundary was ever stored. Slot 7199 is present; the first Gloas slot and the first post-fork block are both absent:

GET /eth/v1/beacon/headers/7199  ->  200
GET /eth/v1/beacon/headers/7200  ->  404
GET /eth/v1/beacon/headers/7203  ->  404   {"code":404,"message":"block not found"}

GET /eth/v1/node/syncing
{"data":{"head_slot":"7199","sync_distance":"5090","is_syncing":true,"is_optimistic":false,"el_offline":false}}

Slot 7200 is the first Gloas slot (fork epoch 225 × 32). Slots 7200, 7201 and 7202 were all missed network-wide; the first post-fork block is at slot 7203.

Root cause analysis

Bug 1 — zero-count ExecutionPayloadEnvelopesByRange livelock at the fork boundary

Grandine constructs a sync batch of length zero for the empty fork-boundary slot and then waits for a response that can never arrive:

DEBUG p2p::sync_manager: retrying batch SyncBatch {
    target: ExecutionPayloadEnvelope, direction: Forward,
    peer_id: PeerId("16Uiu2HAmHZD15GtPknoBUeauCezzT2dFLQbSDS2ZJYh8wUxHw2Kc"),
    start_slot: 7200, count: 0, retry_count: 161, response_received: false,
    data_columns: None, is_delayed: false }

DEBUG p2p::network: sending ExecutionPayloadEnvelopesByRange request
    (app_request_id: Application(2244), peer_id: 16Uiu2HAm2w9NKS2ccezMt8ANj1ZY7guSJme77uaEBBSctCkx2LTU,
     request: ExecutionPayloadEnvelopesByRangeRequest { start_slot: 7200, count: 0 })

DEBUG p2p::network: reporting peer: 16Uiu2HAmHZD15GtPknoBUeauCezzT2dFLQbSDS2ZJYh8wUxHw2Kc
    Mid Tolerance Error SyncService ExpiredSyncBatch

The loop is: build a zero-length batch → send it → the peer returns nothing → the batch expires (ExpiredSyncBatch, response_received: false) → the peer is downscored → retry against a different peer. Forever.

The defect is specific to the envelope pipeline. Aggregating every SyncBatch Grandine logged across all ten hosts over six hours, count: 0 appears only for ExecutionPayloadEnvelope — never for Block, never for DataColumnSidecar:

And the zero-count batches are overwhelmingly anchored at the fork boundary. Of every zero-count batch in a six-hour window, 19,157 are at start_slot: 7200 and span all ten hosts; the remainder are single-host stragglers at pre-fork slots in the low hundreds.

Root cause (hypothesis, high confidence). Under EIP-7732 an execution payload envelope exists only for a slot that contains a block with a payload. Slots 7200–7202 are empty, so the envelope count for a range anchored at 7200 evaluates to zero. Grandine appears to derive the batch length from the number of envelopes it expects rather than from the slot span, and does not treat a zero-length result as "nothing to fetch, advance the cursor" — it materialises a batch and enters the request/expire/retry cycle. What would confirm it: the batch-construction path in p2p::sync_manager that computes count for an ExecutionPayloadEnvelope target; a unit test with a fork boundary followed by ≥1 empty slot should reproduce it without a network.

No backoff cap and no give-up condition. The retry counter increments monotonically within a process lifetime and never resets except on restart. On grandine-reth-1, a process 56 minutes old had reached retry 259 — roughly one retry every 13 seconds, each one downscoring a peer. Hourly maxima on the same host reached 3,878 on the longer-lived pre-swap binary:

hour (UTC)   max retry counter
2026-09-02 20:00        1285
2026-09-02 22:00        1816
2026-09-03 00:00        2336
2026-09-03 02:00        2854
2026-09-03 04:00        3401
2026-09-03 06:00        3878
2026-09-03 07:00           4   <- counter resets: new process after the 06:52 image swap
2026-09-03 08:00         241

The reset visible in the 07:00 bucket independently corroborates the 06:52 UTC image swap described below.

Bug 2 — OOM: a discrete 13–17 GB allocation at process age ~110–120 s

$ sudo dmesg -T | grep 'Out of memory'
[Thu Sep  3 08:45:53 2026] Out of memory: Killed process 273622 (grandine)
    total-vm:3474850168kB, anon-rss:31194000kB, file-rss:128kB, shmem-rss:0kB,
    UID:1006 pgtables:64924kB oom_score_adj:0

Sampling /proc/<pid>/status every five seconds across full process lifetimes shows this is not a gradual leak and not "the 4M-validator registry simply does not fit". Resident memory sits on a flat plateau at ~12 GB and then makes a single enormous jump inside one sampling interval:

grandine-geth-1 — one complete life (5 s sampling)
  age=  5s   1.17 GB
  age= 15s   6.18 GB
  age= 40s   6.42 GB     <- plateau
  age= 50s  11.57 GB     <- step up after anchor-state load
  age=105s  12.15 GB     <- flat for ~55 s
  age=110s  15.71 GB     <- spike begins
  age=115s  29.57 GB     <- +13.9 GB in 5 s -> SIGKILL
  (new pid, age=1s, 0.24 GB)

grandine-geth-1 — previous life        grandine-besu-1
  age=106s  12.17 GB                    age=112s  12.17 GB
  age=111s  12.17 GB                    age=117s  20.68 GB
  age=116s  25.58 GB  -> dead              age=122s  dead

For contrast, the one host that survives is perfectly flat — 13.10 GB ±7 MB across a 170-second sample taken at process age 3092–3259 s:

grandine-reth-1 (uptime 3384 s)
  age=3092s  13,097,284 kB
  age=3147s  13,104,820 kB
  age=3202s  13,102,772 kB
  age=3259s  13,101,628 kB

So Grandine's steady state on this network is ~12–13 GB, comfortable on a 32 GB host. The kill comes from a discrete bulk allocation of 13–17 GB triggered at process age ~110–120 s, about 60 s after the anchor state at slot 7136 is loaded. There is no panic, no graceful shutdown and no error line — the process is simply gone between one log line and the next, consistent with SIGKILL.

Root cause: unresolved. We can characterise the event precisely but not yet attribute it to a code path. What would confirm it: a heap profile or allocator statistics over the first three minutes of startup, or running the node under a container memory limit below the host's RAM so the allocation fails and produces a backtrace instead of a kernel kill. The reproduction window is narrow and reliable — every ~2 minutes on any of the nine crash-looping hosts.

Are the two bugs causally linked? The evidence says no.

The obvious hypothesis is that the livelock drives the memory growth — an unbounded retry queue, or gossip objects for 5,100 unprocessable slots piling up in a delayed-object buffer (Grandine does log fork_choice_control::mutator: retrying objects delayed until slot 12299). We tested it against the one node that survives, and it does not hold:

The node that livelocks hardest is the node that does not OOM. grandine-reth-1 issues 16× more zero-count envelope requests and carries a retry counter two orders of magnitude higher than the crash-looping host, yet its memory is flat to within 7 MB over three minutes. Both hosts log delayed-object retries at an identical rate (~5/min), so that buffer is not the discriminator either. If the livelock drove the memory growth, grandine-reth-1 would be the worst-affected node; it is the least affected.

We therefore treat these as two independent defects. Fixing the livelock will not stop the OOM, and fixing the OOM will leave the client still unable to cross the fork. What distinguishes grandine-reth-1 — why it cleared the ~115 s allocation once and now never re-triggers it — is unresolved and is the single most useful thing to investigate next.

Contributing factor — wrong image, on a tag that moved twice mid-run

All ten Grandine hosts run registry-1.docker.io/ethpandaops/grandine:glamsterdam-devnet-8. The devnet-9 spec sheet lists Grandine as :develop trunk. These nodes are running a devnet-8 build on devnet-9.

Worse, that tag is an ordinary mutable Docker tag, and it demonstrably moved twice during this run under Watchtower:

Two consequences for anyone reading restart counts on these hosts. First, Watchtower creates a new container rather than restarting the existing one, so RestartCount resets to zero on every image swap. The values we observed (47–63) are restarts since 06:52 UTC, not since genesis, and understate the true crash rate. Second, "Grandine 2.0.5" on this devnet refers to two different binaries; any bug report must name the digest.

The zero-count livelock is present in both images — the request pattern is continuous in the logs from at least 2026-09-02 20:57 UTC (the edge of our query window; it predates that) through 2026-09-03 08:57 UTC, straddling the 06:52 swap. The OOM crash-loop is specific to 4a9f26f09401.

The binary also does not recognise nine of devnet-9's configuration variables, and computes a null next-fork digest:

WARN runtime::grandine_args:1247: unknown configuration variables:
  ["ATTESTATION_PROPAGATION_SLOT_RANGE, ATTESTATION_SUBNET_COUNT, GAS_LIMIT_SCHEDULE,
    HEZE_FORK_EPOCH, HEZE_FORK_VERSION, INCLUSION_LIST_DUE_BPS, MAX_REQUEST_INCLUSION_LIST,
    MAX_TRANSACTIONS_BYTES_PER_INCLUSION_LIST, MIN_SLOTS_FOR_INCLUSION_LISTS_REQUESTS"]

INFO eth2_libp2p::discovery:565: Updating the ENR next fork digest next_fork_digest=0x00000000

This does not by itself explain either bug — the fork schedule is supplied by --configuration-directory and Grandine reports configuration matches the one in configuration file — but it means no result here should be read as a verdict on Grandine trunk. The livelock needs re-testing against a :develop build before it is filed against current code.

Impact

• Grandine has zero coverage on devnet-9. No Grandine node has validated a single Gloas block. The client is untested against every EIP in the fork.
• 10,000 validators offline (10 hosts × 1,000), roughly 1.0% of the ~1,001,087 active set. Small next to the 307,000 lost to the Prysm split, but it compounds the same participation deficit that is holding the network below the 2/3 finality threshold.
• Peer scoring pollution. Every retry downscores a peer with Mid Tolerance Error SyncService ExpiredSyncBatch for correctly declining to answer a malformed zero-count request. At ~1 retry/13 s/node across 10 supernodes, that is a steady stream of unearned penalties against healthy nodes.
• Downstream tooling breakage. Assertoor logs error processing execution block: could not add block to cache against every grandine-* client.

How to verify / reproduce

Bug 1 — livelock. Should reproduce without a network: run a Gloas fork transition in which the first fork slot (and ideally the next two) contain no block, then drive a Grandine node to sync across that boundary. Expect a SyncBatch { target: ExecutionPayloadEnvelope, start_slot: <fork_slot>, count: 0 } that never completes. On the live network:

# zero-count batches, by target — count:0 appears only for ExecutionPayloadEnvelope
SELECT extract(Body,'target: ([A-Za-z]+)') AS target,
       extract(Body,'count: ([0-9]+)')     AS cnt,
       count() AS c, uniq(ResourceAttributes['host.name']) AS hosts
FROM external.otel_logs
WHERE Timestamp >= now() - INTERVAL 6 HOUR
  AND ResourceAttributes['network']  = 'glamsterdam-devnet-9'
  AND ResourceAttributes['ethereum_cl'] = 'grandine'
  AND match(Body, 'SyncBatch \{')
GROUP BY target, cnt ORDER BY c DESC

Bug 2 — OOM. Reproduces every ~2 minutes on any of the nine crash-looping hosts. Sample RSS at 5 s resolution to catch it; anything coarser than ~10 s will miss the spike entirely and make it look like a clean exit:

PID=$(pgrep -x grandine); awk '/VmRSS/{print $2}' /proc/$PID/status

# and always cross-check the kill against the kernel, not docker:
sudo dmesg -T | grep 'Out of memory: Killed process'

Recommended action

1. Grandine team — livelock. Treat a zero-length envelope range as "nothing to fetch, advance the cursor", not as a batch to dispatch. Guard batch construction so count == 0 is never sent, add a retry ceiling with a give-up path, and stop attributing ExpiredSyncBatch to the peer when the request we sent was degenerate.
2. Grandine team — OOM. Investigate the allocation at process age ~110–120 s that takes RSS from ~12 GB to 25–30 GB in under five seconds on a 4M-entry validator registry. Start from what grandine-reth-1 does differently, since it clears the same point once and then stays flat at 13.1 GB indefinitely.
3. ethpandaops — image. Move the Grandine hosts to :develop as the spec sheet specifies, and re-test. Until then, no finding here should be filed against Grandine trunk.
4. ethpandaops — pinning. :glamsterdam-devnet-N tags are being treated as pinned but are ordinary mutable tags; this one moved twice mid-run, once ~1 hour before the fork. Pin by digest, and disable Watchtower for the duration of a fork test.
5. Diagnostics. Anything that reads container exit status on this fleet needs to consult dmesg; docker inspect reports OOMKilled:false ExitCode:0 for kernel OOM kills and will silently mislead.

Evidence appendix

Livelock, verbatim

[2026-09-03T08:43:07.173Z] DEBUG p2p::sync_manager:1249: [Sync Peers: 188/189] add execution payload
  envelope request by range (app_request_id: Application(2230),
  peer_id: 16Uiu2HAmHZD15GtPknoBUeauCezzT2dFLQbSDS2ZJYh8wUxHw2Kc, range: 7200..7200, retries: 161)

[2026-09-03T08:43:22.620Z] DEBUG p2p::sync_manager:1249: [Sync Peers: 191/192] retrying batch
  SyncBatch { target: ExecutionPayloadEnvelope, direction: Forward,
  peer_id: PeerId("16Uiu2HAmHZD15GtPknoBUeauCezzT2dFLQbSDS2ZJYh8wUxHw2Kc"),
  start_slot: 7200, count: 0, retry_count: 161, response_received: false,
  data_columns: None, is_delayed: false },
  new peer: 16Uiu2HAm2w9NKS2ccezMt8ANj1ZY7guSJme77uaEBBSctCkx2LTU

[2026-09-03T08:43:22.620Z] DEBUG p2p::network:2996: reporting peer:
  16Uiu2HAmHZD15GtPknoBUeauCezzT2dFLQbSDS2ZJYh8wUxHw2Kc Mid Tolerance Error SyncService ExpiredSyncBatch

[2026-09-03T08:43:23.752Z] DEBUG p2p::sync_manager:1249: ... retries: 163

Correct fork digest, correct peers, no progress

[2026-09-03T08:43:18.547Z] DEBUG p2p::network:1490: sending Status response ... local: V2(StatusMessageV2 {
  fork_digest: 0x6ebb965e, finalized_epoch: 223,
  head_root: 0x3d90ebc2db79c0c39f2457a5554c1c14b0ee18a744bc603d94c11463c2843365,
  head_slot: 7199, earliest_available_slot: 6816 }))
    ... remote peer at head_slot: 12214, same fork_digest 0x6ebb965e

The final seconds before an OOM kill — no error, no panic

08:59:57.001  WARN validator::validator:739: too many empty slots after head: 7199 + 32 < 12299
08:59:57.167 DEBUG p2p::sync_manager: [Sync Peers: 10/10] remove peer (16Uiu2HAm25Kp7bJy...)
08:59:57.587 DEBUG p2p::sync_manager: [Sync Peers: 12/12] remove peer (16Uiu2HAmN8fPimuJ...)
08:59:58.003  INFO http_api_utils::logging:75: produced response (HTTP/1.1 200 OK) to (GET /metrics ...)
--- process killed; RSS 29.57 GB ---
09:00:00.568  WARN runtime::grandine_args:1216: both --configuration-directory and --boot-nodes specified
09:00:00.585  INFO runtime::runtime:1269: starting beacon node

Startup sequence preceding the spike

08:41:57.8  process start                                          (age    0s)
08:41:58.5  INFO runtime::runtime:1493: started loading validator keys
08:42:41.0  INFO runtime::runtime:180: loaded 1000 validator key(s)  (age   43s)
08:42:42.8  INFO fork_choice_control::storage:1115: loading validators from disk
08:42:47.8  INFO pubkey_cache:187: decompressing new validator keys for state at slot: 7136
08:42:50.5  INFO fork_choice_control::storage:258: loaded state at slot 7136   (age   53s)
08:42:51.4 DEBUG fork_choice_control::storage:96: verified 5058 pending deposit signatures
            ... ~60 s of flat ~12 GB RSS ...
            (age ~110-120s) RSS 12 GB -> 25-30 GB -> SIGKILL

Node configuration

Grandine/2.0.5-1a910389/x86_64-linux
image: registry-1.docker.io/ethpandaops/grandine:glamsterdam-devnet-8
custody mode: Super          --subscribe-all-data-column-subnets
custody_group_count: 128     archival interval: 32 epochs
back-sync enabled: false     slasher enabled: false
hosts: 32 GB RAM, CL + EL co-resident (EL using 0.4-2.8 GB)