Four client bugs, taken apart

1 · The discovery pipeline

The fleet emitted 2.70 billion log lines over the window. You cannot read that, and you cannot grep it usefully either — 83 % of it is one client on a debug build, and 49 % of all execution-layer lines are two clients logging user RPC errors at WARN. The signal came from a narrower path:

Three traps that would have produced wrong answers

Worth stating plainly, because each one silently returns a plausible result:

The general shape: a check that cannot fail is not a passing check. Before trusting a zero, confirm the field can ever be non-zero.

2 · geth — a panic that costs 11,000 blocks

core/filtermaps · process-fatal

How it was found

A crash census across the fleet turned up 13 process-fatal aborts, and 9 of them were the same panic on one node out of fifteen — the one running a block builder under sustained transaction load. A bug that hits 1 of 15 identical nodes is almost always about timing, not configuration.

The lead-in was the giveaway. Every abort was preceded within tens of milliseconds by head churn, and RawReceipts was then called with exactly head + 1 — the same off-by-one relationship all four times:

10:38:09.102  Shorten chain  del=2 number=97,824
10:38:09.121  Extend chain   add=2 number=97,826
10:38:09.130  Shorten chain  del=1 number=97,825   <-- head lands on 97,825
10:38:09.146  panic: invalid block number          <-- +16 ms, asked for 97,826

What is actually wrong

The intuitive guess — “the chain view went stale” — is wrong, and worth discarding explicitly: ChainView is immutable in its head number. The stale thing is a flag.

Why it is worth fixing

A panic here is not a dropped request — the process dies with a dirty trie. On restart geth logs Head state missing, repairing and rewinds to the last persisted state. Measured cost across the four aborts analysed:

if l.delimiter && l.blockNumber >= cv.HeadNumber() {
    return false
}

3 · nimbus-eth2 — one bad answer, poisoned forever

fork choice · still unfixed upstream

How it was found

A node reported up = 1 on both scrape targets and had produced zero blocks in 20 hours. That contradiction is the entire discovery. Everything else followed from asking why two sources disagreed.

The health endpoint did know — /eth/v1/node/health was returning 206, not 200. Nobody was watching it.

What is actually wrong

An INVALID response to one engine_forkchoiceUpdatedV4 — for a payload that engine_newPayloadV5 had returned VALID for seconds earlier — put the node into a state it could never leave. Four defects compound:

The transition to the permanent state is exact, not approximate. A viability check falls back to justified.epoch + 2 >= currentSlot.epoch; 3840 + 2 = 3842 holds through epoch 3842 and fails the instant epoch 3843 begins — slot 122,976 = 13:55:12 UTC, matching the logged current_slot to the second.

The operational lesson, learned the hard way

The first diagnosis said this node needed a resync — reasoning that it wedged immediately on reloading its persisted state, so a restart would reload the same state. That was wrong, and reading the source is what caught it: fork choice is rebuilt in memory from the finalized head at startup, and a fresh node only inherits invalid from a parent. So a plain restart starts clean.

It was then tested on the live node. A single docker restart beacon:

Full recovery in 25 minutes, versus hours for a resync. The cheaper action was also the correct one, and only reading the code revealed that.

4 · nethermind & besu — announce what you will serve

devp2p transaction announcements · two clients, one invariant

How they were found

geth logs a warning and disconnects a peer whose announced transaction size disagrees with the transaction it then delivers. Those warnings were running at 1,392 per geth node per day — about ten times the rate at which two already-known versions of this bug had been filed. So the question was not “is there a bug” but “who is the new offender”.

The log line names a peer by node ID, not by client. Attribution was rebuilt from first principles — take each node's self-announced enode public key, hash it, join on the ID — and cross-checked against admin_peers. That produced two different offenders with two different signatures, which is what revealed these were separate bugs rather than one.

The invariant both violate

The size you announce must be the size of the bytes you will serve.

Receivers validate the delivery against the announcement and use it to budget response sizes. Both clients announced one encoding and served another.

Same symptom, completely different causes

And these two compound with a third, already-known bug: besu never dials out. Every force-drop is a connection besu cannot re-establish — it can only wait to be re-dialled. Neither bug alone would be crippling; together they turn a recoverable disconnect into sustained isolation. Three of the eth/72-capable pairings ended up effectively partitioned:

An honest complication

The devp2p spec still says the announced size is the “consensus encoding” — pre-blob wording, in a section the spec itself marks as unfinished. Read literally, nethermind's code comment is quoting the spec correctly. So the filing does not argue “the spec says elided”. It argues that the announced size must match the delivered bytes, because that is what every receiver validates against — and that nethermind already demands exactly that of its own peers on eth/68. A filing that overstates its authority gets dismissed; one that names the ambiguity and argues around it does not.

5 · Ground truth: what the spec should say

Patching two clients treats the symptom. The actual defect is upstream of all of them — the specification does not unambiguously define what the announced size is, and where it does define it, it contradicts what every implementation does.

What the four documents actually say

The rule that resolves it

The announced size must be the byte length of the transaction exactly as that peer will return it in PooledTransactions, on the negotiated protocol version.

This is the only definition that satisfies what the field is for. EIP-5793 introduced it so nodes could “load balance or throttle peers” and avoid blindly broadcasting large transactions — which requires the number to be the bandwidth the fetch will actually cost. A consensus-encoding size is off by three orders of magnitude for a blob transaction and is useless for that job.

Proposed replacement text for caps/eth.md

An earlier draft of this section proposed wording that an adversarial review then took apart. Five defects are worth recording, because each is a trap anyone writing this text would fall into:

The revised text states the rule once, generically, and puts the per-type detail where a future fork can extend rather than rewrite it:

#### Pooled Encoding

Certain transaction types carry auxiliary data which is required to validate the
transaction in the pool, but which is not part of the transaction as it appears in a
block. Transactions of such a type therefore have a second, 'pooled' encoding, used by
[PooledTransactions]. ... we refer to transactions in this encoding using `pooled-tx_n`.

For a transaction whose `tx-type` defines no wrapped form, the pooled encoding is
identical to `tx`. Types which do define one - currently only type `0x03`, introduced by
[EIP-4844] - are encoded as:

    wrapped-tx = tx-type || rlp([
        tx-payload-body, wrapper-version: P,
        blobs: [blob_1: B_131072, ...],
        commitments: [commitment_1: B_48, ...],
        proofs: [proof_1: B_48, ...],
    ])

... On chains where [EIP-7594] is not active, the version-0 wrapper defined by [EIP-4844]
is used instead: it omits `wrapper-version` and carries one blob proof per blob. The two
forms are distinguished by the number of elements in the RLP list.

--- and, under NewPooledTransactionHashes ---

`txsize_n` is the byte length of the announced transaction in the [pooled encoding], on
the protocol version negotiated for this connection. It is the length of:

- the RLP encoding of `legacy-tx`, for legacy transactions;
- `tx-type || tx-data`, for typed transactions whose type defines no wrapped form;
- the whole `wrapped-tx`, for types which do.

The RLP string header which frames a typed transaction as an element of the enclosing
[PooledTransactions] list is not counted.

A peer must announce the size of the transaction it holds. A receiver may recompute
`txsize_n` from a transaction it has been served - evaluating it at the protocol version
on which the announcement was made, which need not be the version it was delivered on -
and may disconnect a peer whose announcement disagrees.

The structural test is not SSZ but EIP-8094, which proposes stripping sidecars from PooledTransactions entirely. Under it txsize collapses back to the plain consensus size — and the invariant sentence above survives with zero edits while the old per-type prose would need a rewrite. Three live drafts (8094, 8141, SSZ), one structure that absorbs all three.

Does this survive the SSZ migration?

This was the main worry, and the answer is cleaner than expected. No SSZ EIP conflicts. EIP-6404/6465/6466/7495/7688 contain zero references to NewPooledTransactionHashes, PooledTransactions, or any announcement size; EIP-6404 explicitly defers the channel question (“RLP and SSZ transactions may clash when encoded… use only a single format within one channel”).

The hazard one would brace for — a container whose serialized length is not unique — is also gone: EIP-7495 was retitled ProgressiveContainer, serialization is now identical to Container, and active_fields is a static type property that is never serialized. One value, one length. Under SSZ the invariant actually becomes more valuable, because EIP-6404's Transaction has no sidecar concept at all — “the byte length of what this peer would serve” is the only definition that still means something when the type system has no answer.

Why this shape, and not the alternatives

What this means for the two filings

They should not be filed the same way, which is the practical payoff of doing this analysis before pressing send:

• besu — file as a straightforward bug. It announces one encoding and serves another of the same kind. Internally inconsistent under every reading; no spec question arises.
• nethermind — file as alignment with de-facto network behaviour, explicitly conceding that the literal devp2p text supports their implementation, and paired with a devp2p issue to fix the text. A filing that opens “you violate the spec” invites the accurate reply “no, we implement it — geth doesn't”, and gets closed.
• devp2p — file the spec clarification. This is the fix that prevents the next client from doing the same thing.

6 · Why each fix is safe

A fix that trades a crash for silent corruption is not a fix. Each patch was checked against the same three questions: does it change behaviour on the paths that were already working, does it use an existing error route rather than inventing one, and does the test actually pin the bug?

The strongest test technique here

Two of the three patches were validated by putting the bug back and confirming the new test fails. A test that passes both with and without your fix is testing nothing, and you cannot tell the difference without trying it. It costs one minute.

7 · What generalises

1. up = 1 does not mean “working”. A node reported healthy through 20 hours of producing nothing. Alert on function — “has this validator proposed or attested recently” — not on liveness. The signal that would have caught it already existed and was unwatched.
2. A check that cannot fail is not a passing check. A NULL column made “no missed payloads” the answer to a question that was never actually asked. Before trusting a zero, confirm the field can ever be non-zero.
3. Rebuild attribution from first principles when a conclusion rests on it. The peer logs name a node ID, not a client. Deriving the mapping from public keys — rather than reusing a prior mapping — is what separated two distinct bugs that shared one symptom.
4. Read the source at the deployed commit. Mutable image tags meant the fleet moved 42 times in four days. Reading HEAD would have explained the wrong binary.
5. Be willing to be wrong out loud. Six intermediate conclusions in this investigation were wrong and were caught by verification: an assumed root cause, a miscounted crash total, a wrong protocol version, a “one-line fix” that was six files, a misread ramp that was a step — and a resync recommendation that would have cost hours of unnecessary work. Every one was caught by checking rather than by intuition.

Teardown of four bugs found on glamsterdam-devnet-8, 2026-08-28 → 08-31. Companion to the weekend incident report. Fix branches are pushed with issue and PR text drafted; filing upstream is the maintainers' and operators' call. Line numbers and evidence are in the filing documents.