At the Gloas fork boundary (epoch 225 = slot 7200 = 2026-09-02 16:00:00 UTC), Prysm computed a different post-fork beacon state than every other consensus client and rejected the first post-fork block. 307 of 310 Prysm validator nodes — plus all four bootnodes, which also run Prysm — are frozen at head slot 7199, the last pre-Gloas slot. Their execution clients are frozen with them at block 6581/6582. This removes ~31% of the active stake from the network and is, on its own, sufficient to make finality impossible. We have ruled out configuration drift: the stuck nodes' runtime spec, fork schedule and ENR eth2 field are byte-identical to healthy nodes. This is a state-transition implementation bug.
Participation was 95% at epoch 224 and collapsed to 56% at epoch 225 — the fork epoch — and has been flat at 55–63% for 155 epochs since. The chain last justified epoch 224 and last finalized epoch 223, both immediately pre-fork.
| Epoch | Participation | Blocks | Note |
|---|---|---|---|
| 224 | 95% | 30/32 | last pre-fork epoch |
| 225 | 56% | 15/32 | Gloas fork |
| 226–379 | 55–63% | 13–24/32 | flat, no recovery |
Slots 7200, 7201 and 7202 were all missed. The first block on the Gloas chain is slot 7203, block root 0x3cd7ca12…d20f15, proposed by validator 2865618 (a Lighthouse node). Every live Prysm node rejects it, in a loop, forever:
WARN initial-sync: Block processing failure
error=validator execution and consensus: failed to validate consensus state transition function:
could not execute state transition: could not validate state root,
wanted: 0x11e6bf3205722cc299100ac2863958a5c16611815101dc8d810b33136374efc5
received: 0x2458f99eebeb5cfab87fed85230e5b1ef32041c1539c59c4f1489c10c799a6e0
firstSlot=7203 lastSlot=7203
root=0x3cd7ca120a6cc81537b24f8e62656ac2ac892a47884607d41a73087701d20f15In Prysm's validateStateRoot, wanted is the locally computed post-state root and received is the root declared in the block. We fetched the canonical header for slot 7203 from a healthy Lighthouse node: its state_root is 0x2458f99e… — the value Prysm rejects. Prysm is the outlier, not the network.
| Cohort | Hosts | Head slot | Status |
|---|---|---|---|
| Prysm validator nodes | 307 | 7199 | frozen, looping |
| Prysm validator nodes | 3 | — | beacon down / unreachable |
| Bootnodes (also Prysm) | 4 | 7199 | frozen, looping |
| Lighthouse / Nimbus / Lodestar / Teku | 680 | ~12 216 | crossed the fork cleanly |
100% of live Prysm nodes are stuck. The three that are not at slot 7199 are down for unrelated reasons, not ahead of it.
This was the first thing we tested, because it is the cheapest explanation and it would have meant no client bug at all. It is ruled out. We pulled /eth/v1/config/spec from a stuck Prysm node and from a healthy Lighthouse node and diffed all shared keys:
keys only in PRYSM: 21 (constants Prysm exposes and Lighthouse does not)
keys only in LIGHTHOUSE: 9 (and vice versa)
shared keys with DIFFERENT values: 1
DEPOSIT_CONTRACT_ADDRESS prysm='0x00000000219ab540356cBB839Cbe05303d7705Fa'
lighthouse='0x00000000219ab540356cbb839cbe05303d7705fa'The only shared-key difference is EIP-55 checksum casing of the deposit contract address, which is cosmetic. Every Gloas-relevant constant is identical:
GLOAS_FORK_EPOCH 225 == 225
GLOAS_FORK_VERSION 0x80124218 == 0x80124218
PTC_SIZE 512 == 512
MAX_PAYLOAD_ATTESTATIONS 4 == 4
PAYLOAD_ATTESTATION_DUE_BPS 7500 == 7500
ATTESTATION_DUE_BPS_GLOAS 2500 == 2500
MAX_SIGNED_EXECUTION_PAYLOAD_BID_SIZE 196932 == 196932
DOMAIN_PTC_ATTESTER 0x0c000000 == 0x0c000000
DOMAIN_BEACON_BUILDER 0x0b000000 == 0x0b000000
MIN_BUILDER_WITHDRAWABILITY_DELAY 64 == 64
CHURN_LIMIT_QUOTIENT_GLOAS 32768 == 32768The fork schedules are identical too — both report the Gloas transition 0x70124218 → 0x80124218 at epoch 225. Both nodes load the same --chain-config-file=/network-config/config.yaml.
The 311 stuck Prysm nodes are spread across all seven execution clients:
Geth 129 Nethermind 92 reth 45 besu 25 erigon 12 ethrex 6 nimbus-el 3Those same seven EL clients run without incident under Lighthouse, Nimbus, Lodestar and Teku on the Gloas chain, and a separate sweep found zero invalid-block, state-root-mismatch or block-access-list errors across the EL tier in a 3-hour window. The fault is isolated to the consensus client. The EL freeze is a consequence: a stalled CL stops driving forkchoiceUpdated.
A fleet-wide image-churn problem exists on this devnet (watchtower replaced containers on 763 hosts in the 75 minutes before the fork — geth 412, nethermind 300, erigon 40, lodestar 30, grandine 10). Prysm was not among them. That matters here, because it closes off the alternative explanation that the fleet forked on a different binary than the one that was tested.
bootnode-1 beacon created 2026-09-02T05:25:16Z started 05:25:16Z restarts=0
prysm-nethermind-2 beacon created 2026-09-02T05:28:09Z started 16:01:28Z restarts=1
both: image sha256:274dc1e88061f1933369c608809aa04d9ae6ff82050b471eaeb41b25a2ab2bfb
image created 2026-09-02T05:10:50ZOne digest, deployed ~05:25–05:28 UTC on 2026-09-02 from an image built at 05:10 UTC the same day — roughly 10.5 hours before the fork. The cleanest evidence is bootnode-1: it ran the entire fork transition with restarts=0 on digest 274dc1e8… and is stuck at slot 7199. The failure is therefore attributable to a single, identifiable Prysm build, and the repro below can name one digest with confidence.
prysm-nethermind-2 — the host used as the primary evidence node for the state-root divergence below — did restart once, at 16:01:28 UTC, 88 seconds after the fork boundary. It came back on the same image digest and still stuck at slot 7199, so the restart is a symptom of the fork failure rather than a binary swap, and it does not weaken any measurement taken from that host.
As a loose observation, not an established pattern: lodestar-geth-2's beacon restarted at 16:00:58 UTC — 58 seconds after the boundary — closely mirroring the Prysm host, on a different client and a different image. We have not checked whether this generalises across the fleet and are not claiming that it does.
Prysm's pre-fork state is correct. We queried the state root at slot 7199 on both a stuck Prysm node and a healthy Lighthouse node:
prysm-nethermind-2 (stuck) lighthouse-geth-1 (healthy)
slot 7199 0xe4af38e3...c80e6daa 0xe4af38e3...c80e6daa IDENTICAL (measured)
slot 7200 404 "no block exists at slot 7200" 0x48b23aa6481e1800658b4744299bbc845566ee1ac65161328accc9190480a88a
slot 7201 404 0x20cef3c0c341ad70ca53f78cc6826d89b4a2160bc2670b1df5994175046dbf4b
slot 7202 404 0xc5f9e3328e6abd3a200ddbed70f030c993f09cff31687fbacf0077926f6fd345
slot 7203 404 0x2458f99eebeb5cfab87fed85230e5b1ef32041c1539c59c4f1489c10c799a6e0The slot-7199 identity is a direct measurement on the stuck node, not an inference: Prysm holds byte-for-byte the same pre-fork state as the healthy chain.
Beyond 7199 the API vector closes. Prysm returns 404 "State not found: no block exists at slot 7200" for 7200, 7201 and 7203 — it will not advance a state through empty slots to serve this endpoint, whereas Lighthouse does. The comparison that would localise the bug therefore cannot be made over the beacon API and has to be run offline (see below).
The first state-transition failure anywhere in the Prysm fleet is at 2026-09-02 16:00:39 — 39 seconds after the fork boundary, ~3 seconds into slot 7203, on the gossip path (ERROR sync: Could not handle p2p pubsub). We swept all Prysm beacon logs from 15:58 to 16:40 UTC: there is no earlier error at slots 7200–7202.
Prysm's silence at slots 7200–7202 does not exonerate the state upgrade. Those slots are empty, and the empty-slot advance performs no state-root comparison — there is no block header to check against. The first externally observable checkpoint after the fork is block 7203. The divergence is therefore somewhere inside the bundle process_slots(7199 → 7203) + process_block(7203), which contains three slot advances, the epoch 224→225 transition, the upgrade_to_gloas state upgrade, and the first Gloas block body. We cannot narrow it further from outside the client.
Hypothesis: the divergence is in upgrade_to_gloas, specifically in the initialisation of new Gloas BeaconState fields, rather than in Gloas block processing.
Why we favour it. The observed signature — exact agreement at 7199, then a wrong root at the very first post-fork state-root check — is what an incorrectly initialised state upgrade produces: the hash-tree-root of the post-upgrade state is wrong immediately, and every subsequent root inherits the error, regardless of block contents. A bug confined to Gloas block processing would more typically be sensitive to what is in the block, and we would expect at least some post-fork blocks to be accepted. None are.
There is a matching open upstream PR: prysm#17380, "Explicitly set bid fields when upgrading to Gloas". That is, by title, a fix for exactly this class of defect. The devnet-9 spec sheet also tracked Prysm as running consensus-spec v1.7.0-alpha.12 rather than the devnet's target v1.7.0-alpha.14, with prysm#17383 (bump to alpha.14) still open.
We have not verified this. We can see that Prysm's post-fork state root is wrong; we cannot see which field of the state is wrong from outside the process. Treat the PR references as candidates, not as a diagnosis.
The beacon API cannot do this (Prysm 404s on empty slots), so run it in a harness. Take the network's slot-7199 state, advance it through the fork boundary with Prysm's own code, and compare the root:
# 1. Pull the pre-fork BeaconState SSZ from a NAMED healthy node.
# Note: checkpointz on this devnet is DOWN (pinned to a static list of now-dead
# nodes, "no nodes found"), so checkpoint sync is NOT a usable source here.
curl -H 'Accept: application/octet-stream' \
http://<lighthouse-geth-1>:5052/eth/v2/debug/beacon/states/7199 -o state_7199.ssz
# 2. In a Prysm harness, load it and advance to the first Gloas slot:
# state = BeaconState.unmarshal(state_7199.ssz) # pre-Gloas, epoch 224
# state = process_slots(state, 7200) # epoch transition + upgrade_to_gloas
# root = hash_tree_root(state)
# 3. Compare against the network:
# 0x48b23aa6481e1800658b4744299bbc845566ee1ac65161328accc9190480a88aOnce wedged, each Prysm node falls into a self-sustaining failure mode:
| Cohort | Validators | Share of active stake |
|---|---|---|
| Prysm stuck at slot 7199 (307 nodes × 1000) | 307 000 | 30.7% |
| Prysm down / unreachable (3 nodes) | 3 000 | 0.3% |
| Grandine (separate defect, reported separately) | 10 000 | 1.0% |
| Teku beacons down (13 nodes) | 13 000 | 1.3% |
| Lighthouse (7) + Lodestar (3) down | 10 000 | 1.0% |
| Total offline | 343 000 | 34.3% |
Against ~1 001 087 active validators, the participation ceiling is 65.7%. The finality threshold is 66.7%. Observed participation is 55–63%, with the residual gap attributable to Teku's sick hosts. Even if every other issue on the network were fixed, the chain still could not finalize while Prysm is down. Prysm alone is 30.7 points of the 34.3-point deficit.
All four bootnodes run Prysm/v7.1.8-c1d69e8 and are stuck at slot 7199. We decoded the live ENR eth2 field from a stuck bootnode and from a healthy Lighthouse node:
bootnode-1 (stuck @ 7199) fork_digest=0x6ebb965e next_fork_version=0x80124218 next_fork_epoch=FAR_FUTURE
lighthouse-geth-1 (healthy, tip) fork_digest=0x6ebb965e next_fork_version=0x80124218 next_fork_epoch=FAR_FUTUREThey are identical — and this is expected behaviour, not a misconfiguration. compute_fork_digest derives the digest from the fork version for the current wall-clock epoch, not from the node's head, so a node that failed the transition still advertises the correct post-Gloas digest. (The value checks out cryptographically: the Gloas base digest 0x20523e64 XOR the EIP-7892 blob-parameters digest 0x4ee9a83a gives 0x6ebb965e, the exact on-wire value.) The consequence worth recording is architectural: fork-digest isolation structurally cannot protect a network against a failed fork transition — it separates nodes on different configured forks, not nodes that disagree about the resulting state. Three consequences follow:
We tested the intuitive worry — that 312 nodes advertising the live fork digest would clog peer tables and starve gossip — and it does not hold. Resolving every connected peer's IP back to the inventory, against a no-pruning baseline of 30.9% (312 of 1008 nodes stuck):
| Host | Connected | Stuck@7199 | Share | Publish failures / h |
|---|---|---|---|---|
| teku-geth-3 | 93 | 37 | 40% | 0 |
| teku-besu-11 | 94 | 2 | 2% | 77 |
| nimbus-geth-1 | 153 | 4 | 3% | 0 |
| lighthouse-geth-1 | 201 | 1 | 0.5% | 0 |
| no-pruning baseline | — | — | 30.9% | — |
Peer scoring works, on every client: by-client stuck shares are lighthouse 0.6%, nimbus 3.4%, lodestar 8.7%, teku 9.2% — all far below the 30.9% a non-pruning node would show. Lighthouse carries 201 connected against 498 disconnected.
More decisively, the correlation runs the wrong way for the poisoning theory. The host with the most stuck peers logs the fewest publish failures. teku-geth-3 sits at 40% stuck and logs zero Failed to publish attestation in an hour, while teku-besu-11 at 2% stuck logs 77. Across a wider sweep the dedicated gossip analysis measured r(stuck-peer share, publish failures) = −0.271 against r(CPU%, publish failures) = +0.931. The attestation-publish failures are driven by CPU exhaustion on the affected hosts, not by peer-table poisoning; see the dedicated gossip report.
An earlier revision of this report suggested that stuck nodes crowd out useful peers on hosts that fail to prune, and that stopping them would relieve the healthy network. Both are wrong and are retracted. Peer scoring already isolates the stuck nodes, and stopping them would not improve attestation throughput. This does not soften the conclusion — it sharpens it: fixing the Prysm state transition is the only lever on finality.
All commands are read-only.
# 1. Confirm the freeze and that it is exactly the last pre-Gloas slot
curl -s localhost:5052/eth/v1/node/syncing
# {"data":{"head_slot":"7199","sync_distance":"5017","is_syncing":true,...}}
# 2. Confirm the pre-fork state matches the network (it does)
curl -s localhost:5052/eth/v1/beacon/states/7199/root
# 0xe4af38e333ab280c8d77eb20ab5d49e36fffc84a8601c61bc0bcf113c80e6daa on BOTH prysm and lighthouse
# 3. Rule out config drift
diff <(curl -s PRYSM:5052/eth/v1/config/spec) <(curl -s LIGHTHOUSE:5052/eth/v1/config/spec)
curl -s localhost:5052/eth/v1/config/fork_schedule # identical on both
# 4. The rejected block, from a healthy node
curl -s LIGHTHOUSE:5052/eth/v1/beacon/headers/7203
# state_root = 0x2458f99eebeb5cfab87fed85230e5b1ef32041c1539c59c4f1489c10c799a6e0
# = the value prysm reports as "received" and refuses
# 5. The failure loop
docker logs --tail 60 beacon 2>&1 | grep 'could not validate state root'Fleet-wide, over the shipped OTel logs:
SELECT min(Timestamp), uniq(ResourceAttributes['host.name']), count()
FROM external.otel_logs
WHERE Timestamp BETWEEN toDateTime('2026-09-02 15:58:00') AND toDateTime('2026-09-02 16:40:00')
AND ResourceAttributes['network'] = 'glamsterdam-devnet-9'
AND ResourceAttributes['ethereum_cl'] = 'prysm'
AND match(Body, 'could not validate state root')
-- first occurrence: 2026-09-02 16:00:39, i.e. 39s after the fork, 3s into slot 7203For the Prysm team, the highest-value repro does not need the devnet running — only the slot-7199 state SSZ, which is large but static. See the offline procedure in the Root cause section: process_slots(state_7199, 7200), compare against 0x48b23aa6…. That isolates the state upgrade from block processing in one step and keeps working after this devnet is torn down. Pull the state from a named healthy node — checkpointz is down on this network, so checkpoint sync is not available as a source.
network glamsterdam-devnet-9 (chain 7013099983)
genesis 2026-09-01 16:00:00 UTC (MIN_GENESIS_TIME 1788274800 + 3600s GENESIS_DELAY)
Gloas fork epoch 225 = slot 7200 = 2026-09-02 16:00:00 UTC
CL image registry-1.docker.io/ethpandaops/prysm-beacon-chain:develop
CL image digest sha256:274dc1e88061f1933369c608809aa04d9ae6ff82050b471eaeb41b25a2ab2bfb
CL image created 2026-09-02T05:10:50Z
CL deployed 2026-09-02 05:25-05:28 UTC (~10.5h before the fork), restarts=0 through the boundary
CL version Prysm/v7.1.8-c1d69e8 (linux amd64)
observed 2026-09-03 08:50 UTC, epoch ~380, slot ~12230slot 7199 Canonical proposer 513048 graffiti "prysm-reth-37" <- last pre-Gloas block
slot 7200 Missing proposer 662856 <- fork boundary, empty
slot 7201 Missing proposer 1314750 <- empty
slot 7202 Missing proposer 3703783 <- empty
slot 7203 Canonical proposer 2865618 graffiti "Lighthouse/v8.2.2-f4fa28b" <- first Gloas block
slot 7204 Orphaned proposer 893732 graffiti "teku-nethermind-18"All four proposers listed above were checked against the live registry and are active_ongoing with 32 ETH effective balance — the high indices come from the 3M additional-mnemonic validators and are legitimately active, not exited. Proposer selection is not implicated.
healthy (lighthouse-geth-1) prysm
slot 7199 0xe4af38e333ab280c8d77eb20ab5d49e36fffc84a8601c61bc0bcf113c80e6daa IDENTICAL (measured)
slot 7200 0x48b23aa6481e1800658b4744299bbc845566ee1ac65161328accc9190480a88a 404 (no block at slot)
slot 7201 0x20cef3c0c341ad70ca53f78cc6826d89b4a2160bc2670b1df5994175046dbf4b 404 (no block at slot)
slot 7202 0xc5f9e3328e6abd3a200ddbed70f030c993f09cff31687fbacf0077926f6fd345 404 (no block at slot)
slot 7203 0x2458f99eebeb5cfab87fed85230e5b1ef32041c1539c59c4f1489c10c799a6e0 404 (no block at slot)
block 7203 header (canonical, from healthy node):
root 0x3cd7ca120a6cc81537b24f8e62656ac2ac892a47884607d41a73087701d20f15
parent_root 0x3d90ebc2db79c0c39f2457a5554c1c14b0ee18a744bc603d94c11463c2843365
state_root 0x2458f99eebeb5cfab87fed85230e5b1ef32041c1539c59c4f1489c10c799a6e0
proposer 2865618
prysm computes for the same block:
state_root 0x11e6bf3205722cc299100ac2863958a5c16611815101dc8d810b33136374efc5 <- WRONG2026-09-02 16:00:39.461 ERROR sync: Could not handle p2p pubsub
error=validator execution and consensus: failed to validate consensus state transition function:
could not execute state transition: could not validate state root ...
fork boundary = 2026-09-02 16:00:00 UTC (slot 7200)
slot 7203 = 2026-09-02 16:00:36 UTC
first error = 2026-09-02 16:00:39 UTC -> 3s into slot 7203, on arrival of the block via gossip
no state-transition error logged at slots 7200, 7201 or 7202 (all empty; no root check performed)312 nodes frozen at head slot 7199:
by CL: prysm 307 | bootnode 4 (also prysm) | grandine 1 (separate defect)
CL versions: Prysm/v7.1.8-c1d69e8 x311
Grandine/2.0.5-1a910389 x1
by EL: Geth 129 | Nethermind 92 | reth 45 | besu 25 | erigon 12 | ethrex 6 | nimbus-el 3
EL block heights: 6581 (x156), 6582 (x156)
All seven EL clients are represented -> the fault is CL-side.
The same seven ELs operate normally under Lighthouse, Nimbus, Lodestar and Teku.bootnode-1 head=7199 Prysm/v7.1.8-c1d69e8 EL block 6581
bootnode-2 head=7199 Prysm/v7.1.8-c1d69e8 EL block 6581
bootnode-3 head=7199 Prysm/v7.1.8-c1d69e8 EL block 6581
bootnode-4 head=7199 Prysm/v7.1.8-c1d69e8 EL block 6581
container timing (bootnode-1): beacon created 05:25:16Z, restarts=0 -> ran the whole fork untouched
execution created 14:54:27Z (66 min pre-fork, watchtower); beacon not touched
live ENR eth2 field, decoded:
bootnode-1 fork_digest=0x6ebb965e next_fork_version=0x80124218 next_fork_epoch=FAR_FUTURE
lighthouse-geth-1 fork_digest=0x6ebb965e next_fork_version=0x80124218 next_fork_epoch=FAR_FUTURE
-> indistinguishable at the discovery layer- 1008-host ansible sweep (read-only) of /eth/v1/node/{syncing,version,peer_count},
/eth/v1/beacon/headers/head, /eth/v1/beacon/states/head/finality_checkpoints,
eth_blockNumber / eth_syncing / web3_clientVersion / net_peerCount, and docker state
- second fleet sweep for kernel OOM state (dmesg), because docker inspect reports
OOMKilled:false ExitCode:0 for kernel OOM kills and cannot be trusted for this
- log aggregation over external.otel_logs via panda clickhouse (42.6M rows / 2h for this network)
- targeted per-host verification against named hosts for every claim above