We tested the hypothesis that the 312 nodes stuck on the pre-Gloas chain are poisoning
healthy nodes' peer tables and causing the fleet's attestation-publish failures. It is false.
Peer scoring is working across every client: against a 30.9% fleet baseline of stuck nodes, healthy
nodes hold 0.6–9.2% stuck peers. The host with the worst peer table in our sample
(teku-geth-3, 39.8% stuck) has zero attestation failures, while the host with the most failures
(teku-geth-1, 780) has a clean 4.6%. What actually predicts attestation-publish failure is
CPU saturation (r = +0.93). Separately, we resolved the fork-digest question: the stuck nodes
advertise the correct Gloas digest because compute_fork_digest is derived from the
wall clock, not from the node's chain head — so no fork-digest configuration could ever have
partitioned them.
Every connected peer's IP resolved back to the devnet inventory via
/eth/v1/node/peers?state=connected. A node performing no pruning would sit near
the fleet baseline of 30.9% (312 stuck of 1009 hosts).
| Consensus client | Hosts | Peer slots | Held by stuck nodes | Share | Median connected |
|---|---|---|---|---|---|
| teku | 6 | 521 | 48 | 9.2% | 90 |
| lodestar | 4 | 795 | 69 | 8.7% | 198 |
| nimbus | 4 | 594 | 20 | 3.4% | 155 |
| lighthouse | 4 | 812 | 5 | 0.6% | 203 |
| fleet baseline (no pruning) | — | — | — | 30.9% | — |
Every client is far below the no-pruning baseline. Peer scoring is successfully purging nodes
that never relay. Lighthouse is the most aggressive — lighthouse-besu-1 reports 201
connected against 498 disconnected, i.e. active rejection.
| Host | CL | Connected | Stuck | Share | Publish failures (2 h) |
|---|---|---|---|---|---|
| teku-geth-3 | teku | 93 | 37 | 39.8% | 0 |
| teku-reth-2 | teku | 99 | 5 | 5.1% | 86 |
| teku-geth-1 | teku | 65 | 3 | 4.6% | 780 |
| teku-besu-11 | teku | 100 | 2 | 2.0% | 281 |
| teku-nethermind-1 | teku | 76 | 1 | 1.3% | 0 |
| teku-geth-12 | teku | 88 | 0 | 0.0% | 81 |
| lodestar-geth-2 | lodestar | 198 | 23 | 11.6% | 0 |
| lodestar-geth-1 | lodestar | 199 | 20 | 10.1% | 0 |
| lodestar-reth-1 | lodestar | 196 | 15 | 7.7% | 0 |
| lodestar-besu-1 | lodestar | 202 | 11 | 5.4% | 0 |
| nimbus-geth-5 | nimbus | 154 | 8 | 5.2% | 6 |
| nimbus-besu-1 | nimbus | 157 | 6 | 3.8% | 28 |
| nimbus-geth-1 | nimbus | 156 | 4 | 2.6% | 299 |
| nimbus-reth-1 | nimbus | 127 | 2 | 1.6% | 271 |
| lighthouse-besu-1 | lighthouse | 205 | 3 | 1.5% | 0 |
| lighthouse-nethermind-1 | lighthouse | 204 | 2 | 1.0% | 0 |
| lighthouse-geth-1 | lighthouse | 202 | 0 | 0.0% | 0 |
| lighthouse-reth-1 | lighthouse | 201 | 0 | 0.0% | 0 |
| Client | Failed | Succeeded | Failure rate | Hosts affected |
|---|---|---|---|---|
| teku | 7,685 | 36,091 | 17.56% | 37 |
| nimbus | 25,982 | 1,271,041 | 2.00% | 71 |
| lighthouse | 0 | 516,705 | 0.00% | 0 |
| lodestar | 0 | 14,658 | 0.00% | 0 |
Both stuck and healthy nodes advertise fork digest 0x6ebb965e. We derived what that
digest actually is. The devnet's fork versions are correctly distinct
(FULU_FORK_VERSION: 0x70124218, GLOAS_FORK_VERSION: 0x80124218), and with
genesis_validators_root = 0x064e977b… the base digests are:
FULU 0x70124218 -> base fork_digest 0xae929e67
GLOAS 0x80124218 -> base fork_digest 0x20523e64
EIP-7892 blob-parameter digest, BLOB_SCHEDULE {EPOCH: 0, MAX_BLOBS_PER_BLOCK: 21}:
sha256(serialize(BlobParameters(0, 21)))[0:4] = 0x4ee9a83a
0x20523e64 XOR 0x4ee9a83a = 0x6ebb965e <-- exactly the digest seen on the wire
So 0x6ebb965e is the Gloas digest. The stuck nodes — whose chain head is
slot 7199, three slots before the fork — are correctly advertising the post-fork digest.
That is not a bug and not a devnet misconfiguration: per the p2p specification,
compute_fork_digest takes the fork version for the current wall-clock epoch from the
fork schedule, not the fork version of the node's own head. A node whose clock reads epoch 380 computes
the Gloas digest whether or not it managed to process the fork.
Fork-digest isolation partitions nodes running a different fork schedule — a stale binary or a wrong config. It cannot partition nodes that share the schedule and fail to execute the transition. No configuration change to this devnet would have automatically fenced off the stuck nodes. Application-layer peer scoring is the only mechanism that can, and — per the table above — it is doing its job.
If stuck peers occupying gossip slots caused publish failures, stuck-peer share should predict failures. It does not. Across six Teku hosts:
r(stuck-peer share, publish failures) = -0.271 (Spearman +0.143) r(CPU%, publish failures) = +0.931 (Spearman +0.943) r(CPU%, stuck-peer share) = -0.486 (Spearman -0.086)
The single cleanest refutation: teku-geth-3 holds 39.8% stuck peers — by far
the worst peer table we measured — and logs zero attestation-publish failures.
teku-geth-1 holds a healthy 4.6% and logs 780. All four Lodestar hosts carry
5–12% stuck peers and log zero failures.
We also tested the coordinating hypothesis that overload prevents peer scoring from running,
making a poisoned table a symptom rather than a cause. That is not supported either: the worst-pruning
host (teku-geth-3, 39.8%) has the lowest CPU of the six Teku hosts at 84%, and
r(CPU, stuck-peer share) is negative. teku-geth-3's peer table is anomalous, and on this
evidence it is also harmless. We could not explain it and we are not going to guess.
| Host | Beacon CPU | Stuck-peer share | Publish failures |
|---|---|---|---|
| teku-geth-1 | 761% | 4.6% | 780 |
| teku-besu-11 | 584% | 2.0% | 281 |
| teku-reth-2 | 281% | 5.1% | 86 |
| teku-geth-12 | 263% | 0.0% | 81 |
| teku-nethermind-1 | 209% | 1.3% | 0 |
| teku-geth-3 | 84% | 39.8% | 0 |
The ordering is monotonic in CPU and unrelated to peer composition. This is consistent with the sibling Teku report's finding of heap pressure and a wedged tick scheduler: a beacon node that cannot keep up with the slot clock misses the publish deadline. The gossip mesh is a victim here, not the cause. Peer count is a plausible secondary factor — every host with ≥196 connected peers logs zero failures, and Teku's 65–100 peers must cover 64 attestation subnets — but our sample cannot separate low peer count from high CPU, since both track the same client.
Nimbus's failures are real but were mis-scaled in the original framing. The exact string count over
2 h is 25,982 Attestation not sent across 71 of 80 hosts — but against
1,271,041 successful Attestation sent, a 2.00% failure rate. The error is
named explicitly and is not the same shape as Teku's:
NTC 2026-09-03 07:22:16.601+00:00 Attestation not sent topics="message_router" attestation="(committee_index: 4, attester_index: 2863700, data: (slot: 11811, index: 1, beacon_block_root: 35ca5a1f, source: 224:a8ce3d31, target: 369:35ca5a1f), signature: b0d8e79c)" error="No peers on libp2p topic"
Nimbus peer tables are clean (1.6–5.2% stuck) and Nimbus CPU is low (3.6–12%), so neither of the two mechanisms above explains it. Nimbus runs a median of 155 connected peers against Lighthouse's 203; with 64 attestation subnets, a thinner peer set raises the probability that some subnet momentarily has no mesh peer. We flag this as the leading hypothesis but did not confirm it — confirming it needs per-subnet mesh-size sampling over time, which we did not run. The 2.00% rate makes this a low-priority item regardless.
The incident report stated: 343,000 validators offline = 34.3%, a 65.7% ceiling against observed 55–63%, with "the residual gap is Teku's sick hosts". The first half stands; the attribution of the residual was wrong and is only partly resolved.
Quantifying the gossip contribution from the measured rates:
| Source | Validators affected | Share of active set |
|---|---|---|
| Verified offline (stuck / down / OOM-looping) | 343,000 | 34.3% |
| Teku publish failures (17.56% of ~126,000 live Teku validators) | ~22,000 | ~2.2% |
| Nimbus publish failures (2.00% of 80,000) | ~1,600 | ~0.16% |
That revises the ceiling from 65.7% to roughly 63.3%. Observed participation over 16 sampled epochs averages 56.4% (range 41–63%), so the top of the observed range now matches the model but the mean does not. Testing the remaining gap against on-chain inclusion capacity — only 17.5 of 32 slots produce a block on average, and participation is measured from attestations included in blocks — gives:
Pearson r(blocks proposed, participation%) = +0.546 over n=16 epochs blocks proposed: mean 17.5/32, range 11-23 participation: mean 56.4%, range 41-63%
A moderate positive correlation, explaining roughly 30% of the variance. So reduced block production is a real contributor but not a complete explanation. The residual is partially unresolved and we say so rather than assigning it. Attestation loss from the missing 40% of slots, inclusion-distance effects during the leak, and per-epoch committee-shuffling variance are all plausible and untested.
The remediation implication changes too. Because peer scoring is already isolating the stuck nodes effectively, stopping them would not measurably improve the health of the 370 nodes that crossed the fork. The original argument for stopping them early does not survive this measurement. Fixing Prysm remains the only action that restores finality.
# 1. Peer-table composition on any node: resolve peer IPs against the inventory
curl -s localhost:5052/eth/v1/node/peers?state=connected \
| jq -r '.data[].last_seen_p2p_address' \
| grep -oE '/ip4/[0-9.]+|/ip6/[0-9a-f:]+'
# cross-reference against ansible/inventories/devnet-9/inventory.ini
# a node with no pruning sits near 30.9%; all measured clients sit at 0.6-9.2%
# 2. Confirm the digest is Gloas, not a stale pre-fork digest
python3 -c "
import hashlib,struct
g=bytes.fromhex('064e977bb09f5fe8ef114bdca09d8ccec07c412c4d41b6879c4c7207d9780073')
base=lambda v: hashlib.sha256(bytes.fromhex(v)+b'\x00'*28+g).digest()[:4]
blob=hashlib.sha256(struct.pack(' 6ebb965e
# 3. Failure rate per client (not raw counts - the raw counts mislead)
# external.otel_logs, position(Body,'Attestation not sent')>0
# vs position(Body,'Attestation sent')>0
# 4. The falsifying pair
# teku-geth-3: 39.8% stuck peers, 0 failures
# teku-geth-1: 4.6% stuck peers, 780 failures
Methodological warning. A substring search for not sent or a regex alternation
containing subnet matches Nimbus's successful Attestation sent lines,
because the full log line contains the word subnet further along. That inflates the
apparent failure count by roughly 50×. Match the literal string and always compute a rate against
the success count.
No peers on libp2p topic separately and at low
priority, with per-subnet mesh sampling.312 nodes at head slot 7199 (last pre-Gloas slot; fork epoch 225 = slot 7200) prysm 307 bootnode 4 (all four bootnodes run Prysm) grandine 1 fleet: 1009 hosts -> 30.9% baseline
sending Status ... fork_digest: 0x6ebb965e, head_slot: 7199, earliest_available_slot: 7136
received Status ... fork_digest: 0x6ebb965e, head_slot: 12209, earliest_available_slot: 0
identical digest; the handshake cannot distinguish them
teku 290,630 lines 41 hosts grandine 39,922 lines 10 hosts <- stuck pre-Gloas prysm 1,574 lines 22 hosts <- stuck pre-Gloas
FULU_FORK_VERSION: 0x70124218 FULU_FORK_EPOCH: 0
GLOAS_FORK_VERSION: 0x80124218 GLOAS_FORK_EPOCH: 225
BLOB_SCHEDULE:
- EPOCH: 0
MAX_BLOBS_PER_BLOCK: 21