glamsterdam-devnet-7 — Payload-Side Orphaning + EL Differential Testing

Date: 2026-08-12 · Network: glamsterdam-devnet-7 (GLOAS/ePBS, mainnet preset, 12 s slots)
Tooling: panda CLI — xatu clickhouse-raw, external.otel_logs, ethnode direct RPC, panda execute sandbox
Scope: (1) root-cause of payload-side block orphaning, (2) 6-client EL differential testing against the Glamsterdam EIPs

Part 1 — Payload-side orphaning: a single withholding builder

Summary

Up to 50% of slots were losing their execution payload. Cause: builder_index=34 wins the bid
auction with a flat, inflated bid and then reveals the payload far past the reveal deadline, so the PTC
unanimously votes the payload absent and the slot resolves Empty.

Beacon-block orphaning was never the problem (~1%). The network kept finalizing at 2 epochs behind
throughout — this is throughput loss, not a liveness failure.

Mechanism

Under EIP-7732 the slot splits into two phases (~/devnets/glamsterdam-devnet-7/CHAOS_TESTING.md):

t=0.0s  proposer broadcasts BeaconBlock (bid only, no payload)
t=6.0s  PAYLOAD_DUE_BPS=5000            -> builder MUST reveal the envelope
t=9.0s  PAYLOAD_ATTESTATION_DUE_BPS=7500-> PTC votes {payload_present, blob_data_available}
t=12s   next slot

Builder 34 bids a flat 1,000,000,000 against honest builder bids of ~3.0–3.4e8, wins, then reveals late.
Caught directly in the bidpool log on nimbus-reth-1:

Updated highest bid for slot and parent  topics="bidpool"
previous_value=199463976 previous_builder=31 bid_slot=209688 builder_index=34 bid_value=1000000000

Evidence — reveal timing per builder (6 h window)

PTC verdict: builder 34 → 0 present / ~102 absent. Every other builder → ~105 present / 0 absent.

Timeline — a stepped parameter sweep

The precision is the tell: across 140 slots the reveal spread is min 11555 / p50 11557 / p90 11560 — a
5 ms band. That is a programmed offset, not network jitter. Combined with the unchanging flat bid and
clean on/off phasing, this is a deliberate parameter sweep (matching experiment E1 in the chaos plan),
not a broken builder.

Attribution is open. Builder 34 is not one of the four buildoor-* hosts — those collectively own
indices 2, 3, 24–32 (confirmed via component=bid-creator logs) — and index 34 emits no bid-creator
log line anywhere in external.otel_logs. It is outside our otel coverage.

Spec-conformance result: the 6.0 s deadline is enforced exactly

Mapping PTC verdict against reveal time (median across 44 sentries, 10 h window):

No payload revealed after t=6.0 s ever achieved a Full verdict. The cliff is sharp and unanimous across
every CL client on the network. Split votes sit in a narrow 5.5–9 s band — exactly the propagation jitter
expected either side of a hard cutoff. This confirms EIP-7732 payload-timeliness enforcement.

Two observability gaps found

• beacon_synthetic_payload_status_resolved is stale — last write 2026-07-31. The Full/Empty/Skipped
  classifier with PTC tallies was blind through the entire incident.
• Dora undercounts. It reported missedPayloads of 0–1 per epoch while a third to a half of slots were
  losing payloads; it appears to count only never-revealed payloads, not revealed-too-late ones.

Part 2 — EL differential testing (6 clients)

Clients: geth, besu, nethermind, reth, erigon, ethrex (via teku-<el>-1). nimbus-el excluded — see D1.

Headline: the EVM consensus surface is clean; every divergence found is in the RPC/debug layer

EIP-8024 detail

All 6 clients agree and match the spec, including every invalid-immediate boundary
(90 < x < 128 for DUPN/SWAPN, 81 < x < 128 for EXCHANGE), the 0x5b/JUMPDEST case, and stack underflow.

EIP-2780 intrinsic gas — execution is exact everywhere

Spec decomposition: TX_BASE_COST 12 000, COLD_ACCOUNT_ACCESS 3 000,
TX_VALUE_COST 4 244 + TRANSFER_LOG_COST 1 756 = 6 000.

The CREATE/CREATE2 per-step gasCost spread (geth/besu/erigon/ethrex 11 000 · nethermind 194 600 ·
reth 831 613) is attribution only — the total gas consumed by the call is byte-identical across all six.
Similarly, besu reporting gasCost=0 for DUP11–16/SWAP10–16 was a stack-underflow reporting artifact; with
adequate stack depth all six report 3.

Divergences found (all RPC-layer)

D1 — nimbus-el rejects params: null (JSON-RPC strictness)

$ panda ethnode exec-rpc glamsterdam-devnet-7 teku-nimbusel-1 web3_clientVersion
Error: JSON-RPC error -32700: RequestParam must be either array or object, got=Null

All 6 other clients accept a null/omitted params. Reproduced on all 4 nimbus-el instances
(teku-, lighthouse-, nimbus-, prysm-nimbusel-1). Breaks generic tooling — evm.call silently
returns 0x. nimbus-el could not be included in any differential below.

D2 — nethermind debug_traceCall fails without an explicit gasPrice

$ panda ethnode exec-rpc ... teku-nethermind-1 debug_traceCall '[{"data":"0x60015460015400"},"latest",{}]'
{"error":"tracing failed: max fee per gas less than block base fee: maxFeePerGas: 0, baseFee: 13", ...}

geth/besu/reth/erigon/ethrex all tolerate maxFeePerGas: 0 for a trace. With an explicit gasPrice
nethermind's output matches geth exactly (same gas, same structLogs).

D3 — ethrex debug_traceCall defaults to callTracer and rejects the standard tracer name

Default (no tracer specified) returns a call frame, not structLogs:

{"from":"0x00..00","gas":"0x11e1a300","gasUsed":"0x33390","input":"0x6001...","to":"0xbd77...","type":"CREATE"}

And the de-facto standard name is rejected:

Invalid params: unknown variant `structLogger`, expected one of `callTracer`, `prestateTracer`, `opcodeTracer`

ethrex names the opcode logger opcodeTracer. Every other client defaults to the struct logger.

D4 — nethermind eth_call reports success on an EIP-2780 runtime-gas OOG

Fresh-account value transfer (intrinsic 21 000, +183 600 runtime state gas = 204 600):

Nethermind returns empty success across the whole 21 000–204 599 range. EIP-2780 does say a runtime OOG
leaves the transaction valid (included, reverted) — so nethermind's model is defensible for a
transaction — but for eth_call a caller cannot distinguish "succeeded" from "reverted out of gas".
This masks failures that all five other clients surface, and would cause tools to under-fund transactions.

D5 — eth_estimateGas disagrees with the EIP-2780 decomposition

Execution is exact on all six (above); the estimator is not:

• geth, reth keep a legacy 21 000 floor — over-estimating a self-send by +9 000 (+75%) and a
  zero-value send by +6 000 (+40%). EIP-2780's whole point is that these paths are now cheaper.
• besu, ethrex add a constant +156…+165 margin whenever COLD_ACCOUNT_ACCESS is involved.
• erigon is the only client returning the exact executable minimum on the fresh-account path
  (204 600 — which also matches a real observed transfer of 204 600 gas); the other five over-estimate
  by +2 791.

D6 — eth_call gas caps differ

Deploying 49 152 bytes (75.2 M gas at CPSB 1530) with gas=300M: geth/reth/erigon return OOG
(~50 M RPC cap), besu/nethermind/ethrex proceed. The EIP-7954 64 KiB limit itself is confirmed on
geth/besu/erigon (MAXCODE at 65 537, OK at 65 536); the others are masked by cap/OOG behaviour.

Cross-check against the known-bug register

Checked ~/devnets/glamsterdam-devnet-7/BUGS.md (862 lines) and EDGECASES.md (272 lines).
D1–D6 are all previously unrecorded. Nearest neighbours:

• BUG-019 — geth debug_traceBlockByHash default-callTracer panic. Same subsystem as D3 but a
  different method and failure mode.
• BUG-013 / BUG-061 — nethermind intrinsic-gas defects (stale memo overcharging self-sends;
  EIP-2780 stateRoot mismatch). This run found no nethermind intrinsic-gas divergence in execution —
  boundaries are exact — suggesting those are fixed in the deployed image, though D4 is adjacent.

Method note

Each differential ran the identical request against all six clients concurrently and compared results;
anything flagged was then re-tested against the spec-derived expectation before being called a divergence.
Two intermediate results were discarded as instrumentation errors: a per-opcode gas sweep that read the
wrong trace index (returning empty data, not agreement), and a code-size test under-provisioned on gas.

Reproduce the headline builder finding:

panda clickhouse query-raw clickhouse-raw "
SELECT builder_index, uniqExact(slot) AS slots, round(quantile(0.5)(pms)) AS p50_ms,
       uniqExactIf(slot, pms > 12000) AS slots_late
FROM (SELECT slot, builder_index, min(propagation_slot_start_diff) AS pms
      FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v1_events_execution_payload
      WHERE meta_network_name='glamsterdam-devnet-7'
        AND slot_start_date_time >= now() - INTERVAL 6 HOUR
      GROUP BY slot, builder_index)
GROUP BY builder_index ORDER BY slots DESC"

Iteration 2 (2026-08-12, later) — precompiles, SSTORE state gas, 63/64 rule

Clean surfaces (no divergence)

Sanity-checked: identity precompile 0x04 echoed the 32-byte payload correctly on all six.

D7 — SSTORE per-step gasCost splits 3-3 on the EIP-8037 state-gas charge

Cold slot, 0→1:

The delta is exactly 97 920 = 64 bytes × CPSB 1530 — the EIP-8037 state-gas charge for creating a new
storage slot. nethermind/reth/ethrex fold it into the SSTORE step; geth/besu/erigon attribute it elsewhere.

Not a consensus divergence. Verified two independent ways:

total gas consumed   : 317 612 on all 6 clients (identical)
min executable gas   : 317 612 on all 6 clients (independent bisect via eth_call)

This is the same class as the CREATE/CREATE2 attribution spread, but ~9× larger (97 920 vs 2 791 gas)
and it hits the single most-traced opcode in gas-analysis tooling. Any differential harness comparing
per-opcode gasCost across clients will report false positives on every SSTORE to a fresh slot.

Running tally

• Consensus surface: still clean. Every divergence found so far — CREATE/CREATE2, SSTORE,
  DUP/SWAP-underflow, INVALID — resolves to debug-tracer attribution once totals are compared.
• RPC/debug layer: 7 real divergences (D1–D7), none previously recorded in BUGS.md / EDGECASES.md.

Iteration 3 — SLOTNUM, SELFDESTRUCT, access lists, error shapes

Clean surfaces

SLOTNUM was verified against the header at blocks head−0/1/5/50 — exact match every time, so the opcode
and the header field are consistent and the EIP-7843 implementation agrees across all six clients.

D8 — eth_createAccessList: besu and ethrex omit account-only accesses

Storage accesses are handled correctly everywhere; account accesses are dropped by two clients.

besu/ethrex also report a correspondingly lower gasUsed (0x33429 vs 0x3398d for the BALANCE probes).

Impact. A transaction built from besu's or ethrex's access list will not pre-warm those accounts, so at
execution it pays COLD_ACCOUNT_ACCESS (3 000 under EIP-8038) instead of WARM_ACCESS (100) — the access
list silently fails to do its job, and the returned gasUsed understates the real cost. This is the
consensus-adjacent RPC users call specifically to price EIP-8038 warm/cold access.

D9 — memory-expansion OOG returns three different JSON-RPC error codes

MSTORE at offset 2³²:

Code 3 is the standard execution reverted code — ethrex reports a gas/memory halt as a revert, so tooling
that branches on the error code will misclassify it as a revert rather than an out-of-gas condition.

Minor note

eth_call against a block ~500 deep failed on geth and ethrex while besu/nethermind/reth/erigon served it —
a historical-state availability difference (node config), not a client defect.

Running tally after 3 iterations

• Consensus surface: clean. No divergence has survived verification — CREATE, SSTORE, DUP/SWAP and
  INVALID all reduce to debug-tracer attribution once totals are compared.
• RPC/debug layer: 9 divergences (D1–D9), none recorded in BUGS.md / EDGECASES.md.
• EIPs verified conformant across all 6 clients: 8024, 2780, 7843, 8246, plus precompiles 0x01–0x12,
  the 63/64 rule, and the full 256-opcode accept/reject and gas-cost surface.

Iteration 4 — EIP-7708 transfer logs, eth_simulateV1, getProof, transient storage

Clean surfaces

EIP-7708 is live and correctly encoded: emitter 0xfffffffffffffffffffffffffffffffffffffffe, topics[0] =
keccak256('Transfer(address,address,uint256)'), from/to zero-padded in topics[1..2]. All six clients
return an identical log set over the sampled range.

D10 — eth_simulateV1 + traceTransfers double-counts EIP-7708 transfer logs, four different ways

Simulating one plain value transfer:

traceTransfers predates EIP-7708 and synthesises its own ETH-transfer logs under the magic address
0xeeee…eeee. Now that the protocol itself emits a real transfer log, the two overlap — and no two client
families resolve the overlap the same way: three double-count, one substitutes, one errors, one doesn't
implement the method.

Impact. Any indexer or simulator using eth_simulateV1 with traceTransfers will double-count every ETH
transfer on geth/reth/erigon, silently lose the protocol log on besu, and hard-fail on nethermind. With
traceTransfers=false all five agree — so the bug is confined to, and caused by, the flag.

This looks like a genuine spec-interaction gap rather than a single client defect: EIP-7708 and the
eth_simulateV1 traceTransfers feature need an explicit rule about which log wins.

D11 — erigon eth_getProof rejects non-padded storage keys

$ ... eth_getProof '["0x2F76…F97F",["0x0"],"latest"]'
erigon: -32602 invalid argument 1: json: cannot unmarshal hex string of odd length into hexutil.Bytes

geth, besu, nethermind, reth and ethrex all accept the short form 0x0. With a full 32-byte padded key
erigon works and returns the same proof as the others — confirming it is purely input-encoding strictness.

Notes / non-findings

• reth eth_getProof is fine. An earlier failure at block head−2 was historical-state availability,
  not a missing method — it serves latest correctly. Not a divergence.
• ethrex does not implement eth_simulateV1 at all (-32601).
• Historical-state depth varies by node (geth/ethrex could not serve eth_call ~500 blocks back while
  besu/nethermind/reth/erigon could) — node configuration, not a client defect.

Running tally after 4 iterations

• Consensus surface: still clean. Nothing has survived verification across four rounds.
• RPC/debug layer: 11 divergences (D1–D11), none recorded in BUGS.md / EDGECASES.md.
• EIPs verified conformant across all 6: 8024, 2780, 7843, 7708, 8246, plus precompiles 0x01–0x12,
  transient storage, the 63/64 rule, and the full 256-opcode accept/reject and gas-cost surface.

Iteration 5 — EIP-7928 BAL header, EIP-8282 predeploys, receipts, feeHistory

Clean surfaces

EIP-8282 addresses confirmed live on-chain. Both v7.0.0 predeploys carry identical bytecode on every
client at 0x0000BFF46984E3725691FA540A8C7589300D8282 (deposit) and
0x000064D678505AD48F8CCB093BC65613800E8282 (exit) — corroborating that the HackMD page, which still shows
the old deposit and the devnet-6 exit address, is the stale source.

D12 — ethrex omits blockTimestamp from log objects in eth_getBlockReceipts

Same receipt count, same receipt-level keys, but 156 differing fields across 226 receipts — all from one
missing key inside each log:

receipt[8].logs[0]  blockTimestamp:  geth "0x6a7cf6d0"   ethrex <absent>

geth, besu, nethermind, reth and erigon all populate blockTimestamp on every log object (confirmed present
in eth_getLogs output on all five: address, blockHash, blockNumber, blockTimestamp, data, logIndex, removed, topics, transactionHash, transactionIndex). Indexers that read timestamps straight off receipt logs
get null on ethrex and must fall back to a separate block lookup per receipt.

D13 (minor) — trace depth numbering differs for empty sub-calls

A CALL to an account with no code: besu and ethrex emit a trace step at depth=2, while geth, nethermind,
reth and erigon report only depth=1. Total gas is 206 922 on all six, so this is a tracer convention
difference, not a consensus one — but it breaks depth-based call-tree reconstruction across clients.

Non-finding (recorded so it isn't re-chased)

eth_feeHistory is NOT divergent. A digest comparison split the clients 3-3
(geth/nethermind/erigon vs besu/reth/ethrex) on a pinned block, but field-by-field the only difference is
JSON number rendering in blobGasUsedRatio:

geth/nethermind/erigon : [0,   0,   0,   0.7142857142857143]
besu/reth/ethrex       : [0.0, 0.0, 0.0, 0.7142857142857143]

Same numeric values — integer vs float serialization of zero. baseFeePerGas, baseFeePerBlobGas,
gasUsedRatio, oldestBlock and reward all agree exactly.

An earlier feeHistory comparison in this iteration also raced on latest (each client resolving it to a
different head), which briefly looked like agreement. Both runs were redone pinned to a fixed block number.

Running tally after 5 iterations

• Consensus surface: still clean. Five rounds, nothing has survived verification.
• RPC/debug layer: 13 divergences (D1–D13), none recorded in BUGS.md / EDGECASES.md.
• EIPs verified conformant across all 6: 8024, 2780, 7843, 7708, 7928 (header), 8246, 8282, plus
  precompiles 0x01–0x12, transient storage, the 63/64 rule, and the full 256-opcode surface.

Iteration 6 — state overrides, EIP-7702 delegation, tx validation, bad blocks

Clean surfaces

D14 — ethrex does not implement eth_call state overrides

$ ... eth_call '[{...},"latest",{"0x…AA":{"code":"0x602a60005260206000f3"}}]'
ethrex: -32000 Invalid params: Expected one or two params and 3 were provided

ethrex rejects the third parameter outright rather than ignoring it. The same call without overrides
succeeds, so this is a missing feature, not a malformed request. State overrides underpin simulation
tooling (wallet previews, transaction simulation, MEV research) — none of it works against ethrex.
This also blocked ethrex from the EIP-7702 delegation tests above.

D15 — ethrex does not implement txpool_status

-32601 Method not found. geth/besu/nethermind/reth/erigon all serve it. (Their pending/queued counts
differ widely — 0x42/0x401 on geth vs 0x9/0x4e on reth — but txpool contents are node-local, not
consensus state, so that is expected, not a divergence.)

Interop observation — eth_sendRawTransaction rejection codes are inconsistent

Same malformed input, six different answers:

The JSON-RPC spec does not mandate exact codes here, so this is not a defect — but code-branching clients
cannot treat -32000 vs -32602 as meaningful, and besu collapses every malformed input to a bare
"Invalid params" with no diagnostic detail.

Observation — debug_getBadBlocks is non-empty on three clients, all historical

All entries are historical — the highest is block 0xf112 (61 714) against a current head of 188 633,
i.e. ~127 k blocks back. Nothing live. geth and ethrex share bad-block hash 0x472ac1409983e8afcd6950a1…,
which points at a genuinely invalid block rather than a client-specific rejection.

Given the age, these are consistent with the already-documented invalid-block era in BUGS.md
(BUG-013 nethermind intrinsic-gas invalid blocks, BUG-023 geth builder sealing payloads its own validator
rejects). Not reported as new. Worth noting that the error field is null on every entry across all
three clients, so the buffer preserves the block and RLP but not the rejection reason.

Discarded test

One probe in this iteration (eth_call with an override but no to and empty data) was malformed and
returned uninformative 0x — discarded rather than interpreted. The corrected form is test B above.

Running tally after 6 iterations

• Consensus surface: still clean. Six rounds, nothing has survived verification.
• RPC/debug layer: 15 divergences (D1–D15), none recorded in BUGS.md / EDGECASES.md.
• ethrex is the clear outlier: 6 of the 15 findings are ethrex-only gaps (D3 tracer naming, D10
  simulateV1 absent, D12 blockTimestamp, D14 state overrides, D15 txpool_status, plus the D9 error code).
• EIPs verified conformant across all 6: 8024, 2780, 7702 (delegation indicator), 7843, 7708, 7928
  (header), 8246, 8282, plus precompiles, transient storage, the 63/64 rule and the 256-opcode surface.

Iteration 7 — EIP-8037 code deposit, CREATE2, receipt shapes

EIP-8037 code-deposit repricing at CPSB = 1530 — confirmed exactly, all 6 clients

Deploying N bytes via init code, measuring total gas (not per-step attribution):

The marginal cost is exactly CPSB = 1530 gas per code byte, and every client agrees at every size. (The
1 551 first-byte figure and the 1 530.3 at 100 bytes include one-off memory-expansion cost for the larger
RETURN, not a pricing difference.) This is the strongest positive consensus result so far — the headline
EIP-8037 repricing is implemented identically across all six clients.

Other clean surfaces

D12 (extended) — ethrex's missing blockTimestamp also affects eth_getTransactionReceipt

Originally found in eth_getBlockReceipts; it is the same omission in the single-receipt method:

Receipt-level keys are identical (14) on all six — the gap is confined to log objects, and it affects both
receipt RPCs. Any indexer reading log timestamps needs a per-block fallback on ethrex.

Interop note — eth_protocolVersion is not comparable across clients

A deprecated legacy method with no agreed semantics — besu/nethermind/erigon report an eth wire version
(68/71/72) while reth reports 5. Recorded as an observation, not a defect.

Discarded tests

• A first RETURNDATASIZE probe pushed CALL arguments in the wrong order; all six failed identically, so
  it showed no divergence but proved nothing. Re-run correctly above (result: 32 everywhere).
• A receipt probe landed on a block with no transactions and returned nothing; re-run against a block
  containing logs.

Running tally after 7 iterations

• Consensus surface: clean. Seven rounds, no divergence has survived verification.
• RPC/debug layer: 15 divergences (D1–D15), none recorded in BUGS.md / EDGECASES.md.
• EIPs verified conformant across all 6: 2780, 7702 (delegation indicator), 7708, 7843, 7928 (header),
  8037 (CPSB 1530 code deposit), 8024, 8246, 8282, plus precompiles, transient storage, CREATE2
  derivation, return-data semantics, the 63/64 rule and the full 256-opcode surface.

Iteration 8 — log-bloom verification, gas oracles, debug range methods

Log bloom independently verified (not just cross-client agreement)

Rather than comparing clients to each other, the block's logsBloom was recomputed from scratch —
keccak256 of each log address and topic, low 11 bits of the first three byte-pairs, set into a 2048-bit
filter — and compared against each client's header:

block 188912, 314 logs (including EIP-7708 SYSTEM_ADDRESS transfer logs)
computed bloom: 0x322d88a6e8ba9522a60d961de012dd3a…
geth ✓  besu ✓  nethermind ✓  reth ✓  erigon ✓  ethrex ✓   (header == computed on all 6)

This confirms EIP-7708 transfer logs are correctly folded into the block bloom filter on every client — a
result cross-client comparison alone could not establish, since all six could have been consistently wrong.

Other clean surfaces

D16 — debug_storageRangeAt: two implementations, two different response shapes

The two clients that implement it disagree on the pagination field — nextKey (a cursor) vs complete
(a boolean). Tooling written against one cannot page through the other.

D17 — debug_accountRange: three shapes including a bare null

reth returning null rather than an error or a result object is the notable one — a caller sees a
successful response carrying nothing.

Observation — eth_gasPrice: besu suggests 10× the others

eth_gasPrice is an implementation-defined fee-oracle suggestion, not consensus state, so this is not a
defect — but a wallet defaulting to besu's suggestion on this network would bid 10× the going rate.

Builder 34 — stopped again

Two consecutive clean windows — the withholding paused around 00:00 UTC, matching the on/off cycling seen
all evening rather than a permanent stop.

Running tally after 8 iterations

• Consensus surface: clean. Eight rounds, no divergence has survived verification.
• RPC/debug layer: 17 divergences (D1–D17), none recorded in BUGS.md / EDGECASES.md.
• EIPs verified conformant across all 6: 2780, 7702, 7708 (incl. bloom inclusion), 7843, 7928, 8024,
  8037 (CPSB 1530), 8246, 8282, plus precompiles, transient storage, CREATE2, KECCAK256 and memory-expansion
  pricing, MCOPY, return-data semantics, the 63/64 rule and the full 256-opcode surface.

Iteration 9 — a real client panic, and a bug that has gone away

🔴 NEW BUG — erigon panics on eth_feeHistory when newestBlock is "safe" or "finalized"

The first genuine client crash found in this campaign — not an RPC shape or completeness gap.

$ panda ethnode exec-rpc glamsterdam-devnet-7 teku-erigon-1 eth_feeHistory '["0x4","safe",[25]]'
Error: JSON-RPC error -32000: method handler crashed

Underlying panic (from external.otel_logs):

[EROR] RPC method eth_feeHistory crashed: runtime error: makeslice: len out of range

This is happening organically on the live network — 14 occurrences in the last 24 h across three
erigon nodes, without anyone probing for it:

Only latest/earliest and numeric blocks work; the two checkpoint tags crash the handler. The RPC layer
recovers (no node death), but any tool asking erigon for fee history at the safe or finalized head gets a
hard failure. Not present in BUGS.md or EDGECASES.md.

Reproduce:

for el in geth besu nethermind reth erigon ethrex; do
  echo -n "$el: "; panda ethnode exec-rpc glamsterdam-devnet-7 teku-$el-1 eth_feeHistory '["0x4","safe",[25]]' 2>&1 | head -c 60; echo
done

✅ BUG-019 (geth callTracer panic) — no longer reproducing

BUGS.md records debug_traceBlockByHash with the callTracer panicking
index out of range [0] with length 0, at ~57–74 panics/day on each of 7 geth nodes through 2026-07-27.

Current status:

• Zero index out of range panics network-wide in the last 24 h (ClickHouse external.otel_logs).
• debug_traceBlockByNumber with {"tracer":"callTracer"} succeeded on 13/13 consecutive recent blocks
  on teku-geth-1.

Caveat worth stating: the bug needs a block containing a top-frame-charge OOG tx, so clean block sampling
alone would not prove a fix — but the absence of the panic signature in 24 h of logs, against a prior
baseline of ~60/day/node, is strong evidence it has been fixed upstream. Recommend re-verifying and moving
BUG-019 to resolved.

D18 — besu accepts invalid eth_feeHistory reward percentiles

The spec requires reward percentiles in [0, 100]. besu returns a result instead of an error.

D19 (minor) — erigon returns a degenerate response for blockCount ≥ 2^63

eth_feeHistory with blockCount = 0x8000000000000000 (or 2^64-1) returns
{"oldestBlock":"0x0","gasUsedRatio":null} on erigon rather than an error. geth/nethermind/reth clamp and
return valid data; besu and ethrex reject with -32602/-32000.

Instrumentation note

An attempt to trace full blocks (227 txs) across all six clients in one sandbox process was killed
(exit 137, OOM). Block-level traces must be checked one client at a time with status-only inspection —
switched to that approach.

Builder 34 — resumed

Back after the ~25-minute pause, same 11 556 ms reveal. Cycling continues.

Running tally after 9 iterations

• Consensus surface: clean. Nine rounds, no consensus divergence has survived verification.
• 1 real client bug found (erigon eth_feeHistory panic) + 19 RPC/debug divergences (D1–D19),
  none previously recorded.
• 1 documented bug appears fixed (BUG-019).

Iteration 10 — network-wide malformed eth2 RPC requests hitting every teku node

🟠 NEW — teku rejects ~13,000 incoming eth2 RPC requests per day as malformed

A log-signature sweep surfaced two exceptions firing constantly on all 7 teku hosts:

Per-host over 24 h: buildoor-teku-nethermind-1 1 746 · teku-erigon-1 1 664 · teku-ethrex-1 1 616 ·
teku-nimbusel-1 1 260 · teku-besu-1 1 228 · teku-reth-1 932 · teku-geth-1 672. Continuous across the whole
window (first 01:33, last 01:32 next day) — steady, not a burst.

Stack — these are incoming requests being decoded, not responses:

RpcException$ExtraDataAppendedException: [Code 1] Extra data appended to end of message
  at LengthPrefixedPayloadDecoder.complete(LengthPrefixedPayloadDecoder.java:124)
  at RpcRequestDecoder.complete(RpcRequestDecoder.java:68)
  at Eth2IncomingRequestHandler.readComplete(Eth2IncomingRequestHandler.java:89)
  at RpcHandler$Controller.onRemoteWriteClosed(RpcHandler.java:344)

The appended bytes decode to a Snappy stream:

extra data: 0x 0008 ff060000734e61507059 010c00000175
              │    │                     └─ chunk type 0x01 (uncompressed), len 0x00000c
              │    └─ Snappy stream-identifier frame: 0xff, len 6, "sNaPpY"
              └─ varint length prefix (8)

That is a complete, well-formed second length-prefixed Snappy message trailing the first — consistent with a
peer either pipelining a second request onto a stream (the eth2 RPC spec is one-request-per-stream) or
re-emitting the stream identifier per message. Teku correctly rejects it; the sending client is at fault.

Teku does not log the offending peer ID on these lines, so attribution to a specific client is not
established — that is the obvious next step (packet capture, or a teku build that logs the peer on decode
failure). Recorded here as a network-level observation with the byte-level evidence.

Not present in BUGS.md or EDGECASES.md.

Correction — some erigon panics in the log were self-inflicted

The 4 makeslice panics on teku-erigon-1 timestamped 00:59:58 were caused by my own probes in
iteration 9, not organic traffic. The iteration-9 finding still stands on the organic occurrences that
predate any testing:

Two signatures investigated and dismissed

• unwrap (656–1 618 per host) — a false positive of my own regex: it matches unwrapped: inside
  Netty's ByteBuf.toString(). No relation to Rust unwrap() panics. The underlying lines are the teku
  RPC exceptions above.
• panic: runtime error: invalid memory address or nil pointer dereference on teku-reth-1
  (2026-08-12 05:00:36, single occurrence) — this is a Go panic, but reth is Rust and every surrounding
  log line is teku (Java). It therefore comes from a Go sidecar on that host, not from reth. Left
  unattributed rather than mis-assigned.

Builder 34 — still cycling

Running tally after 10 iterations

• Consensus surface: clean. Ten rounds, no consensus divergence has survived verification.
• 1 client bug (erigon eth_feeHistory panic), 1 network-level protocol issue (malformed eth2 RPC
  requests to teku), 19 RPC/debug divergences, 1 documented bug apparently fixed (BUG-019).
• The log-sweep technique has now produced 2 of the 3 most significant findings; probe-based differentials
  have produced the RPC-surface catalogue.

Iteration 11 — attribution of the teku RPC framing issue (and a downgrade)

⬇️ CORRECTION — the teku eth2-RPC exceptions are foreign-peer teardown noise, not a devnet client bug

Iteration 10 flagged ~13 000 ExtraDataAppendedException + MessageTruncatedException per day on teku
hosts as a possible client protocol defect. Attribution now shows a much more mundane cause. The
exceptions sit inside a tight, repeating log sequence on every occurrence sampled:

02:05:00.757  Connected to peer 16Uiu2HAmAWey…Vine via /ip4/49.12.123.25/tcp/48934
02:05:00.757  onConnectedPeer() 16Uiu2HAmAWey…Vine
02:05:00.758  Rejecting inbound connection from 16Uiu2HAmAWey…Vine because IRRELEVANT_NETWORK
02:05:00.945  Failed to disconnect from 16Uiu2HAmAWey…Vine cleanly.
02:05:00.945  CompletionException: RpcException$ExtraDataAppendedException …
02:05:00.945  Disconnected from peer 16Uiu2HAmAWey…Vine

Peers from other networks find these nodes over discv5, open a stream, and get rejected with
IRRELEVANT_NETWORK; the decode exception is thrown while tearing that connection down. The "extra data"
is the tail of a message from a foreign network's stream, not a malformed request from a devnet client.

Quantitative support (24 h, per teku host). Each exception emits two log lines (the
CompletionException and its Caused by:), so the exception count is half the line count:

Near 1:1 on geth and erigon, but 2–3× on besu and ethrex — so foreign-network peers explain much but
not demonstrably all of the volume. Stated as-is rather than overclaimed.

Severity: downgraded to benign operational noise. It is DEBUG-level logging of connections the node is
correctly refusing. No action beyond awareness; it is not evidence of a devnet client emitting malformed
frames.

The teku-nethermind-1 zero was a logging artifact

That host reported 0 exceptions and 0 IRRELEVANT_NETWORK — not different behaviour, just a different
log level:

teku-nethermind-1 runs at INFO; both signatures are DEBUG-level, so they are simply not collected there.
Consequence for the iteration-10 table: those per-host counts partly measured log verbosity, not peer
behaviour. Any per-host comparison in external.otel_logs must first check DEBUG-line volume.

Builder 34 — paused again

Running tally after 11 iterations

• Consensus surface: clean. Eleven rounds, no consensus divergence has survived verification.
• 1 client bug (erigon eth_feeHistory panic on safe/finalized) — still the only genuine defect.
• 19 RPC/debug divergences (D1–D19), none previously recorded.
• 1 documented bug apparently fixed (BUG-019).
• 2 candidate findings withdrawn after deeper analysis (eth_feeHistory 0-vs-0.0 rendering; this
  RPC-framing issue). Roughly half of all digest-level candidates have dissolved under drill-down — the
  method's dominant failure mode is over-reporting, and the verification step is doing the real work.

Iteration 12 — block-context opcodes

🔴 D20 (real bug) — ethrex's GASLIMIT opcode returns i64::MAX in eth_call

Verified stable and non-incidental:

• Same value at every block tested (189 452, 189 478, 189 482) — not a one-off.
• Independent of the call's own gas parameter (0x200000 and 0x11E1A300 both yield 2⁶³−1), so it is
  not echoing the request; ethrex is populating the block environment's gas limit with i64::MAX.
• The other five all match eth_getBlockByNumber(...).gasLimit exactly.

Scope. This is the eth_call / simulation environment. Block execution is almost certainly correct —
ethrex nodes agree on state roots and the network finalizes, which it could not do if GASLIMIT were wrong
during real execution. But any contract reading GASLIMIT (gas-budgeting logic, batch sizing, DoS guards)
gets a nonsense value when simulated against ethrex, and any tool comparing simulated to on-chain behaviour
will diverge. Not in BUGS.md or EDGECASES.md (the register's GasLimit hits are CL targetGasLimit
issues, unrelated).

Reproduce:

for el in geth besu nethermind reth erigon ethrex; do
  echo -n "$el: "
  panda ethnode exec-rpc glamsterdam-devnet-7 teku-$el-1 eth_call \
    '[{"data":"0x4560005260206000f3","gas":"0x200000","gasPrice":"0x3b9aca00"},"latest"]'
done

Clean surfaces — every other block-context opcode agrees

GASLIMIT is the only divergent member of the block-context family — which strengthens the case that it
is a specific ethrex defect rather than a general environment-construction difference.

EXTCODEHASH / EXTCODESIZE edges — all unanimous

Builder 34 — longest pause yet

Four consecutive clean windows (01:45, 02:00, 02:15, 02:30 UTC), all 0% past-due, after 15.2% at 01:30.
Previous pauses ran ~25–30 min; this one is ≥60 min.

Running tally after 12 iterations

• Consensus surface: clean. Twelve rounds, no consensus divergence has survived verification.
• 2 client bugs: erigon eth_feeHistory panic on safe/finalized; ethrex GASLIMIT = i64::MAX.
• 20 RPC/debug divergences (D1–D20), none previously recorded.
• 1 documented bug apparently fixed (BUG-019); 2 candidates withdrawn after drill-down.

Iteration 13 — D20 scope confirmed, call-context semantics clean

D20 confirmed deeper — the ethrex GASLIMIT bug is in block-environment construction, not eth_call

Repeating the probe through debug_traceCall and reading the value actually pushed onto the stack:

The same wrong value appears in the tracer's stack view, not just the eth_call return. That rules out a
return-value serialization quirk and places the defect in the block environment ethrex builds for simulated
execution — shared by eth_call and debug_traceCall alike.

Clean surfaces

The EXTCODECOPY past-end case is a classic divergence spot (zero-fill vs revert vs truncated copy) and all
six handle it correctly.

Log sweep — nothing new

A 6-hour sweep for makeslice, index out of range, nil pointer, FATAL, stack backtrace and Rust
thread panics returned exactly one host with 4 hits — and those are the eth_feeHistory panics at
00:59:58 that my own iteration-9 probes caused. No new organic panics anywhere on the network in 6 h.

Builder 34 — apparently stopped

Seven consecutive clean windows, ~1 h 45 m of silence. Every previous pause ran 25–30 minutes, so this is a
qualitative change — the withholding experiment looks concluded rather than merely between cycles. Total
observed run: ~2026-08-12 14:10 → 08-13 01:30 (~11 h), across two delay settings (36.06 s, then 11.557 s).

Running tally after 13 iterations

• Consensus surface: clean. Thirteen rounds, no consensus divergence has survived verification.
• 2 client bugs: erigon eth_feeHistory panic (safe/finalized); ethrex GASLIMIT = i64::MAX
  in both eth_call and debug_traceCall.
• 20 RPC/debug divergences (D1–D20), none previously recorded.
• 1 documented bug apparently fixed (BUG-019); 2 candidates withdrawn after drill-down.

Iteration 14 — independent invariant verification (EIP-7708 negative cases, EIP-8037 gas dimensions)

Instead of cross-client comparison, this round checked internal invariants each client must satisfy on its
own. Two "failures" appeared identically on all six — and both turned out to be correct EIP behaviour that
a naive invariant check would have mis-reported as a bug.

✅ EIP-7708 negative cases verified — zero violations

The spec requires a transfer log for a nonzero-value transfer to a different account, and no log for
reverted transactions. Classifying every value-transferring tx in a block by (receipt status, log present):

104 value-transferring transactions classified per client, zero violations on all six. The 40 logless
transfers are all status = 0x0 contract-creation transactions that reverted — exactly the spec's
"no log emitted on revert" case. This tests EIP-7708's negative branch, which the earlier bloom and
eth_getLogs checks did not reach.

✅ EIP-8037 / 7976 two-dimensional gas accounting confirmed

Identical on all six clients. The header's regular-gas figure is not the sum of receipt gas — receipts
report total gas charged (regular + state), while the header carries only the regular dimension per the
EIP-8037/7976 rework. Σ receipt.gasUsed == cumulativeGasUsed[-1] exactly, so receipts are internally
consistent; the gap is purely the header's excluded state gas.

Methodological note: header.gasUsed == Σ receipt.gasUsed is a valid invariant on mainnet and a
false one on devnet-7. A differential harness carrying mainnet assumptions would flag this on every block,
on every client. Worth recording for anyone building devnet-7 tooling.

Other invariants — all hold on all six

Log sweep and builder 34

A 6-hour sweep over makeslice / index out of range / nil pointer / FATAL / stack backtrace
returned only the 4 self-inflicted eth_feeHistory panics from iteration 9. Nothing new.

Builder 34: 0 past-due slots in every 30-minute window since 01:30 UTC (~2 h). Confirmed stopped.

Running tally after 14 iterations

• Consensus surface: clean. Fourteen rounds, no consensus divergence has survived verification.
• 2 client bugs: erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX.
• 20 RPC/debug divergences (D1–D20).
• EIPs verified conformant across all 6: 2780, 7702, 7708 (positive, bloom and negative cases),
  7843, 7928, 7976/8037 (CPSB 1530 and header gas dimensions), 8024, 8246, 8282.

Iteration 15 — cryptographic root verification, and a blob-announcement size mismatch

✅ withdrawalsRoot independently recomputed — correct on all 6 clients

Implemented a minimal Merkle-Patricia trie (rlp + eth_hash; no trie lib in the sandbox) and rebuilt the
withdrawals root from the withdrawal list, rather than comparing clients to each other.

MPT self-test first: empty-trie root reproduces the canonical
0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 ✓ — so the implementation is sound
before trusting its output.

The 16-withdrawal blocks exercise branch nodes (keys 0x80, 0x01…0x0f, 0x10), not just a single
leaf — so this tests the real trie structure, not a degenerate case. Header withdrawalsRoot is
cryptographically confirmed on every client.

🟡 NEW — network-wide blob-transaction announcement size mismatch (all 8 geth nodes)

Every geth node logs WARN Announced transaction size mismatch continuously:

The dominant size pair is unambiguous:

WARN Announced transaction size mismatch  peer=7f7021da…  tx=3d8ed0..8b909f  size=137,567  ann=6487
137,567 − 6,487 = 131,080 ≈ 131,072 = 128 KiB = exactly one blob

So a peer announces the blob-excluded transaction size while geth computes the blob-included size
(or vice versa — the inverse pairing size=6487 ann=137,567 also appears, 361×). This is the eth/68
announcement-size convention for type-3 blob transactions being applied inconsistently across clients.

Impact is low — geth warns and continues — but it is constant network-wide noise and indicates a genuine
spec-interpretation split on how size is computed for blob transactions. Only geth logs the warning, so
the sending client cannot be identified without a peer-ID→client mapping; that is the follow-up.
Not in BUGS.md or EDGECASES.md.

Two signatures checked and dismissed

• state root (3 500–17 800 per host) — nimbus-el's routine DBG … Validating block … blk="(parentHash…
  lines, which merely contain a stateRoot field. Normal per-block logging, not errors. The 5× spread
  between hosts is log verbosity again, not behaviour.
• INVALID / bad block / Failed to import — no host exceeded the noise threshold.

Builder 34

Zero past-due slots in every 30-minute window since 01:30 UTC (~3 h). Remains concluded.

Running tally after 15 iterations

• Consensus surface: clean. Fifteen rounds, nothing has survived verification.
• 2 client bugs (erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX),
  1 network-wide interop split (blob announcement size), 20 RPC/debug divergences.
• Independently verified (not just cross-client agreement): log bloom recomputation, withdrawals-root
  MPT recomputation, EIP-7708 positive + negative cases, EIP-8037 header gas dimensions, receipt/log
  ordering invariants.

Iteration 16 — attributing the blob-announcement mismatch (and refining iteration 15's conclusion)

⬇️ REFINEMENT — the announcement-size mismatch is not a cross-client convention split

Iteration 15 read the mismatch as "a peer announces the blob-excluded size while geth computes the
blob-included size", implying one or more clients use the wrong eth/68 convention. Attribution shows
otherwise.

Built a peer-ID → client-name map from admin_peers across four geth nodes (48 unique peers), then joined
it against the peer IDs in the Announced transaction size mismatch warnings over 6 h:

Geth warns about announcements from other geth nodes more than about anyone else. A convention split
between clients cannot explain geth↔geth mismatches. The most consistent reading is that geth compares its
blob-inclusive local size against the blob-exclusive announced size that type-3 propagation
legitimately uses (blobs travel separately from the transaction), so the warning fires structurally rather
than because a peer misbehaved.

Revised severity: cosmetic/logging, not an interop defect. The 137,567 − 6,487 = 131,080 ≈ 128 KiB
arithmetic from iteration 15 still stands and is what makes the blob explanation compelling — but the
attribution flips the conclusion from "some client is wrong" to "geth's warning is comparing two different
quantities". Worth reporting upstream as noisy/misleading logging rather than as a client bug.

(Caveat: 43% of warnings map to peer IDs not present in any sampled admin_peers snapshot — peers that
had disconnected by sampling time. The attributed share is a sample, not the full population.)

Builder 34

Zero past-due slots in every 30-minute window since 01:30 UTC (~3.5 h). Remains concluded.

Running tally after 16 iterations

• Consensus surface: clean. Sixteen rounds, nothing has survived verification.
• 2 client bugs: erigon eth_feeHistory panic on safe/finalized; ethrex GASLIMIT = i64::MAX
  in eth_call and debug_traceCall.
• 20 RPC/debug divergences (D1–D20).
• 3 candidate findings withdrawn or downgraded after deeper analysis: eth_feeHistory 0-vs-0.0
  rendering; teku RPC framing (foreign-peer teardown noise); blob announcement size (geth-side accounting).
• Independently verified against spec/crypto rather than cross-client agreement: log bloom
  recomputation, withdrawals-root MPT recomputation, EIP-7708 positive + negative branches, EIP-8037 header
  gas dimensions, receipt/log ordering invariants.

Standing observation on method: three of the six most promising leads dissolved or downgraded on
follow-up. Every one was caught by asking "what else could produce this?" before publishing — attribution
(who actually sent it), timestamps (was it my own probe), log level (is the host even emitting this), and
spec re-reading (is my invariant mainnet-only). The sweep generates candidates cheaply; the disconfirmation
step is where the accuracy comes from.

Iteration 17 — transactionsRoot cryptographically verified; raw-transaction RPCs missing on two clients

✅ transactionsRoot independently recomputed — correct on all 6

Rebuilt the transaction trie from raw transaction bytes (fetched via
eth_getRawTransactionByBlockNumberAndIndex, so no re-implementation of typed-transaction RLP and therefore
no encoding risk of my own), keyed by rlp(index):

block 190 274, 56 transactions
computed transactionsRoot = 0x8e76d8a3a76e71abcaae381ebc7001b8dce731db465ddc0c1ae6b8b83be7457a
geth ✓  besu ✓  nethermind ✓  reth ✓  erigon ✓  ethrex ✓

MPT self-tested against the canonical empty root first. 56 transactions produce a trie with real branch and
extension structure, and the sampled transactions include type-0x03 (blob) transactions — so typed
transactions are covered. Every client's header transactionsRoot is cryptographically confirmed.

That makes three header commitments now verified by recomputation rather than by cross-client agreement:
logsBloom, withdrawalsRoot, transactionsRoot.

D21 — besu and ethrex do not implement the raw-transaction RPCs

The four that implement them return byte-identical output. These are the standard way to obtain canonical
transaction bytes for re-broadcast, offline signature verification, trie reconstruction and archival — none
of which can be done against besu or ethrex without re-encoding the transaction by hand from
eth_getTransactionByHash fields (error-prone for typed transactions, and exactly the risk this method
exists to remove). Not in BUGS.md or EDGECASES.md.

Builder 34

Zero past-due slots in every 30-minute window since 01:30 UTC (~4 h). Remains concluded.

Running tally after 17 iterations

• Consensus surface: clean. Seventeen rounds, nothing has survived verification.
• 2 client bugs: erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX.
• 21 RPC/debug divergences (D1–D21).
• 3 candidates withdrawn/downgraded after deeper analysis.
• Independently verified (recomputation, not agreement): logsBloom, withdrawalsRoot,
  transactionsRoot, EIP-7708 positive + negative branches, EIP-8037 header gas dimensions,
  receipt/log ordering invariants.

ethrex remains the outlier: 8 of the 21 divergences are ethrex-only gaps (D3 tracer naming, D10 no
eth_simulateV1, D12 missing blockTimestamp, D14 no state overrides, D15 no txpool_status, D20
GASLIMIT, D21 raw-tx RPCs, plus the D9 error code). Its consensus behaviour has been correct in every
single test — all three trie roots, every opcode, every gas figure. The gaps are confined to RPC surface
completeness.

Iteration 18 — receiptsRoot verified; builder 34 RESUMED (correcting "concluded")

🔴 CORRECTION — builder 34 was not concluded; it resumed at 05:47:48 UTC on the original delay

Iterations 13–17 reported the withholding experiment as concluded after ~4 h of clean windows. That was
premature. It restarted:

Last slot before the gap: 01:30:12. First slot after: 05:47:48. The delay reverted from 11.557 s back to
36.058 s — so this is a third distinct phase, not a continuation of phase 2.

What I got wrong: I treated "longest pause yet" as evidence of conclusion. A 4-hour gap was simply
longer than the 25–90 minute gaps seen earlier; the operator's cadence had no upper bound I could infer from
five prior samples. The correct statement throughout would have been "paused, duration unknown" — an absence
of activity is not evidence of termination, and I should not have upgraded it to "concluded" merely because
the gap was the longest observed so far.

✅ receiptsRoot independently recomputed — correct on all 6

Fourth header commitment verified by recomputation. Encoding
typeByte ‖ rlp([status, cumulativeGasUsed, logsBloom, logs]), key rlp(index):

block 190 434, 23 receipts
computed receiptsRoot = 0x9619c3601b30a1aa4d13fb1b745010f18a9e882e58b8d189a0f4ac75c1172021
geth ✓  besu ✓  nethermind ✓  reth ✓  erigon ✓  ethrex ✓

Matched first attempt with the standard four-field receipt body — a useful negative result in itself:
despite EIP-7928/8037 adding block access lists and a state-gas dimension, devnet-7 receipts are not
extended in the receipt trie. The extra data (blobGasPrice, blobGasUsed, effectiveGasPrice, …)
appears in the JSON-RPC view only, not in the consensus-committed encoding.

Four header commitments now cryptographically verified

Running tally after 18 iterations

• Consensus surface: clean. Eighteen rounds; all four header commitments verified cryptographically.
• 2 client bugs: erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX.
• 21 RPC/debug divergences (D1–D21).
• 3 candidates withdrawn/downgraded, 1 status call corrected (builder 34 "concluded").

Iteration 19 — state-root proof verification, and a third client bug

✅ Account Merkle proof verified against the header stateRoot

The strongest independent check yet: rather than trusting eth_getProof, the returned proof was walked
and verified — every node re-hashed and checked against the expected reference, the path checked against
keccak(address) nibble by nibble, and the terminal leaf compared to
rlp([nonce, balance, storageHash, codeHash]).

block 190 589, account 0x2F7626bB…F97F, proof depth 7
geth ✓  besu ✓  nethermind ✓  ethrex ✓    (verified=True, leaf value matches)

This validates the state commitment itself, not just block-level data — the account balance and nonce
each client reports are cryptographically bound to the stateRoot in its header.

That makes five independent verifications: logsBloom, withdrawalsRoot, transactionsRoot,
receiptsRoot, and now an account proof against stateRoot.

🔴 D22 (real bug) — erigon eth_getProof returns an internal error for any non-head block

Only the head block works. Every other depth fails — and the message is not "state unavailable", it is
erigon computing a proof and then detecting its own root mismatch. It is building the proof against the
wrong state.

Contrast with the other five:

Impact. eth_getProof is the primitive for light clients, bridges and any state verification at a
finalized block — which is by definition not the head. On erigon that is unusable. The failure is at least
safe (it errors rather than serving an invalid proof), but the distinction between reth's explicit
window limit and erigon's self-detected inconsistency is the difference between a documented constraint and
a defect. Not in BUGS.md or EDGECASES.md.

Reproduce:

hd=$(panda ethnode exec-rpc glamsterdam-devnet-7 teku-geth-1 eth_blockNumber | tr -d '"')
for off in 0 1 2; do b=$(printf '0x%x' $((hd-off)))
  echo -n "head-$off: "; panda ethnode exec-rpc glamsterdam-devnet-7 teku-erigon-1 \
    eth_getProof "[\"0x2F7626bBDb8c0f9071bC98046Ef6fDed2167F97F\",[],\"$b\"]" 2>&1 | head -c 80; echo
done

Builder 34 — phase 3 ran ~23 minutes, now paused (duration unknown)

Phase 3: 05:47:48 → ~06:10, at ~20–22% of slots — a markedly lower rate than phases 1–2 (which peaked near
50%), at the original 36.06 s delay. Now paused again; duration unknown (not calling it concluded).

Running tally after 19 iterations

• Consensus surface: clean. Nineteen rounds; four header commitments plus a state proof verified
  cryptographically.
• 3 client bugs: erigon eth_feeHistory panic on safe/finalized; ethrex GASLIMIT = i64::MAX;
  erigon eth_getProof broken for non-head blocks.
• 22 RPC/debug divergences (D1–D22).
• 3 candidates withdrawn/downgraded, 1 status call corrected.

Iteration 20 — complete state-verification chain, all 6 clients

✅ Storage proofs and code hash verified — the chain now runs header → storage slot

Extending the proof-walking to the two remaining links:

block header
  └─ stateRoot
       └─ account proof (walked, every node re-hashed)          ✓ verified iteration 19
            └─ account leaf = rlp([nonce, balance, storageHash, codeHash])
                 ├─ storageHash → storage proof, path = keccak(slot)   ✓ THIS ITERATION
                 │                  leaf = rlp(value)
                 └─ codeHash    → keccak(eth_getCode(addr))            ✓ THIS ITERATION

Against the EIP-8282 deposit predeploy 0x0000BFF4…8282, slots 0 and 1:

Every client's reported storage values and contract bytecode are cryptographically bound, through two
levels of Merkle proof, to the stateRoot in its own block header. All six pass.

(erigon succeeds here because the query targets latest; per D22 its eth_getProof fails at any earlier
block — consistent with, not contradicting, that finding.)

Seven independent verifications now complete

Not one of these relies on clients agreeing with each other — each is checked against the cryptography.

Instrumentation note

First run crashed: some clients return storageProof[].key in short hex (0x0) rather than the padded
32-byte form, so bytes.fromhex failed. Padding to 64 hex chars fixed it. Minor, but a real cross-client
formatting inconsistency in eth_getProof output that any proof-verifying tool must handle.

Builder 34 — phase 3 continues at a much reduced rate

Intermittent 5–7% bursts rather than the sustained 20–50% of earlier phases. Still active; duration
unknown.

Running tally after 20 iterations

• Consensus surface: clean. Twenty rounds. Seven cryptographic verifications, all passing on all six
  clients — the strongest possible statement that these clients agree on the correct answer, not merely
  with each other.
• 3 client bugs: erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX; erigon eth_getProof
  broken for non-head blocks.
• 22 RPC/debug divergences (D1–D22).
• 3 candidates withdrawn/downgraded, 1 status call corrected.

Iteration 21 — filter lifecycle and trace_* ; a fourth client bug

🔴 D23 (real bug) — erigon eth_getFilterLogs returns 0 while its own eth_getLogs returns 902

Same node, same block range, same filter object — two methods, two different answers:

This is self-inconsistency within one client, which is what makes it unambiguous — no spec
interpretation is required to see that a node returning 902 logs for a range should not return 0 for the
same range through a filter handle. eth_newFilter succeeds and eth_uninstallFilter reports true, so
the filter exists; it simply yields nothing. Any tool using the filter API for log retrieval silently gets
empty results on erigon. Not in BUGS.md or EDGECASES.md.

D24 — ethrex requires the optional topics field in filter objects

eth_getLogs   {"fromBlock":…,"toBlock":…}   → ethrex: -32000 Expected parameter: topics is missing
eth_newFilter {"fromBlock":…,"toBlock":…}   → ethrex: -32000 Expected parameter: topics is missing

topics is optional in the JSON-RPC spec, and all five other clients accept its absence. This
retroactively explains an unresolved error from iteration 5, where ethrex alone failed an unfiltered
eth_getLogs — at the time I recorded it as "ethrex errored" without a cause. The cause is this.

D25 — ethrex does not implement eth_newBlockFilter, trace_block or trace_call

All -32601 Method not found.

D26 — eth_getFilterChanges semantics diverge on first poll

For a filter over a closed historical range, the first getFilterChanges call returns:

reth returns the full matching set; the other four return nothing. The spec ("changes since last poll") is
genuinely ambiguous for a range that is already entirely in the past, so I am not calling either side
wrong — but a consumer polling getFilterChanges gets completely different behaviour depending on client.

trace_* (parity-style) support matrix

* nethermind requires an explicit gasPrice, exactly as with debug_traceCall (D2) — the same root cause
across both tracing families, not a per-method quirk. geth has never implemented parity traces; that is
expected, not a finding.

Clean

eth_getBlockByNumber for a non-existent block (head + 1 000 000) returns null on all six — correct
and unanimous. Filter IDs are opaque and differ in form (geth/besu/reth 128-bit random, nethermind a counter
0x6, erigon structured) — not a defect.

Builder 34

7.5% at 06:45, then 0% across 07:00, 07:15, 07:30. Paused again; duration unknown.

Running tally after 21 iterations

• Consensus surface: clean (7 cryptographic verifications, all six clients).
• 4 client bugs: erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX; erigon eth_getProof
  non-head; erigon eth_getFilterLogs returns nothing.
• 26 RPC/debug divergences (D1–D26).
• 3 candidates withdrawn/downgraded, 1 status call corrected, 1 earlier unexplained error now
  explained (iteration 5 ethrex eth_getLogs).

Iteration 22 — transaction/block field shapes; ethrex blockTimestamp omission is systematic

D12 (broadened again) — ethrex omits blockTimestamp from transaction objects too

First seen in eth_getBlockReceipts (iteration 5), then eth_getTransactionReceipt (iteration 7). It is
not receipt-specific — it is every object that should carry it:

Key-set diffs are otherwise exactly identical across all six (21 keys on the transaction object,
extra=[] missing=[] for the other four) — blockTimestamp is the single systematic omission. Consumers
reading timestamps from transaction or log objects need a per-block fallback on ethrex everywhere.

Clean — type-3 blob transaction shape

blobVersionedHashes and maxFeePerBlobGas are present and the full key set matches across all six
(modulo blockTimestamp). eth_getTransactionCount at a historical block: 0x28f on all six.
eth_getUncleCountByBlockNumber: 0x0 on five.

D27 — ethrex does not implement eth_getUncleCountByBlockNumber (-32601)

D28 (note) — eth_coinbase / eth_accounts behave differently on all six

Five different responses to eth_coinbase across six clients. These are deprecated, node-local methods with
no consensus meaning, so this is an interop note rather than a defect.

One thing worth flagging: nethermind is the only client exposing an account through eth_accounts on a
publicly-reachable RPC endpoint. I checked it before drawing any conclusion — the account holds 0 balance
and no code, so it is almost certainly nethermind's auto-generated node/miner key rather than a funded
key. Not a security finding on this devnet, but the behavioural difference is worth knowing if the same
configuration is carried to a network where that key matters.

Builder 34

0% past-due across 07:15, 07:30, 07:45, 08:00 — paused ~1 h 15 m. Duration unknown.

Running tally after 22 iterations

• Consensus surface: clean — 7 cryptographic verifications, all six clients.
• 4 client bugs: erigon eth_feeHistory panic; ethrex GASLIMIT = i64::MAX; erigon eth_getProof
  non-head; erigon eth_getFilterLogs returns nothing.
• 28 RPC/debug divergences (D1–D28).
• 3 candidates withdrawn/downgraded, 1 status call corrected, 1 earlier unexplained error explained.