Under ePBS a proposer can ship the builder's payload or one its own execution client built. Three days of glamsterdam-devnet-7 says the local path is holding up well: it fills blocks as fully as the builders do, reveals the payload 13× earlier, and never once lost a payload to the timeliness committee. Along the way the data ranks all six execution clients on how much they pack and how fast they validate — and surfaces one client with a transaction-pool gap worth closing.
glamsterdam-devnet-7 runs Gloas/ePBS on 12-second slots. Each slot the proposer publishes a beacon
block containing a bid; the winning builder then reveals the execution payload envelope, and a
~100-member payload-timeliness committee (PTC) votes at t+9 s on whether that payload showed up. The
proposer picks between builder bids and a payload its own EL built — self-building, recorded
on-chain with builder_index = 2^64-1.
The devnet carries four buildoor builder instances (two ethrex-backed, one geth-backed,
one nethermind-backed) plus 35 validating nodes spanning 7 execution clients and 6 consensus clients.
Over the window, 4,812 payloads were self-built and 15,743 came from a builder, so both
paths have enough samples to compare.
Transaction demand was not constant, which turns out to be useful. Three regimes are visible in the data: a baseline period at ~88 txs/block, a spam wave from 2026-07-20 14:00 to 2026-07-21 14:00 UTC peaking above 700 txs/block, and a post-spam period at ~104 txs/block. Every claim below is checked against all three.
Comparing self-built payloads across clients has an obvious trap: a proposer normally self-builds only when its local block beats the builder's bid, so the self-built sample is pre-filtered to each client's good blocks. That bias is not uniform — it depends entirely on the consensus client's bid-selection policy.
And the policies are wildly different. Teku self-builds 83.5% of the time; every
other consensus client self-builds 6–10%. Teku is effectively always taking its local payload, which
makes the six teku-<el>-1 nodes an unbiased, randomized control group: same
consensus client, same slot schedule, same network — the only variable is the execution client.
Everything in the leaderboard below is measured on that cohort.
Six execution clients, 3,384 self-built payloads from teku nodes. Five of them cluster tightly between 69 and 135 transactions — a good result, and a reminder that local building is a real alternative to the builder path on this devnet. reth sits apart at 6, consistently enough that it reads as one systematic cause rather than variance: its 75th percentile (10 txs) is below every other client's 25th.
Gas tells the same story with a different unit: a reth self-built payload burns a median of 0.52 M gas against 7.9–14.6 M for the others, on a 300 M gas limit. Gas per transaction is normal (88 K vs 101–130 K), so reth is not picking unusually cheap transactions — it simply has far fewer of them to pick from.
The blob column is the most useful diagnostic. Every other client's self-built payloads carry a median of 5 to 10 blobs; reth's carry zero. A clean zero rather than a low number is a helpful clue — it suggests one ingress or validation path that blob transactions never clear, not a tuning problem in block packing.
The obvious alternative explanation is that reth is simply being conservative and more demand would close the gap. The spam wave lets us test that, and it doesn't: the control cohort tracked demand upward — besu to 802 txs, geth to 750, ethrex to 652, nimbus-el to 464, nethermind to 336 — while reth moved from 5 to 33 and back to 5. The ratio is comparable to its peers'; the absolute level isn't. That is useful, because it narrows the search: whatever limits reth is upstream of demand rather than a packing heuristic.
The hourly view shows the shape over the full window. The grey trace is the network's median payload; the orange trace is reth's self-built payloads alone. The y-axis is logarithmic, so the two traces are roughly a decade apart throughout.
ethrex has its own dip, but a time-bounded one: its self-built payloads sat at ~13 txs through the baseline window and recovered to ~108 from 2026-07-20 15:00 UTC onward, matching its peers ever since. Unlike reth's, this looks like a condition that resolved — worth confirming what changed on those nodes rather than treating it as a live defect.
Block building is downstream of the transaction pool, so we checked the pool directly.
txpool_status across every node that exposes it, sampled during the post-spam period, when
the network was steadily producing ~104-transaction blocks:
geth, nethermind and besu nodes each hold 85–194 pending transactions and healthy queued sets; reth nodes hold 1 to 8. That reframes the finding in reth's favour: its payload builder is doing its job correctly and packing everything available to it. The block is an accurate reflection of an almost-empty pool, so the question is why transactions aren't reaching or surviving that pool — a much narrower problem than "block building is broken".
Worth noting what is not affected: reth's liveness on this devnet is fine. It missed 3 of 3,488 assigned slots (0.1%, the second-best rate measured), had 3 orphaned blocks, and its newPayload times sit with everyone else's at ~40 ms. This is a narrow gap in a client that is otherwise keeping up with an unreleased fork.
The strongest hypothesis, and the next thing to test, is devnet-7's intrinsic-gas rework
(EIP-2780 /
EIP-8037), which moved state-dependent costs
out of intrinsic gas and into runtime charges at the top frame. A pool-side validity check still using
pre-fork intrinsic-gas rules would reject most of this network's traffic while the block-import path,
which uses the consensus rules, happily accepts the same transactions inside other clients' blocks —
which is exactly the asymmetry observed. Nothing surfaces at info level; confirming it
needs RUST_LOG=reth_transaction_pool=debug on one node.
Filling a block is only half of what an execution client does in a slot; the other half is
validating everybody else's. Dora records a per-client newPayload time for every slot,
and because every client validates the same payload each slot, this is a genuinely
like-for-like comparison — 21,134 slots and roughly 760,000 individual observations.
The ordering is close to the inverse of the fill leaderboard. reth is the fastest validator on the network at a 14 ms median, with nethermind a hair behind at 15 ms; nimbus-el is the slowest at 119 ms. That matters for how the earlier finding should be read: reth's execution engine is not slow, and its payload builder is not slow. The gap is narrowly located in what reaches the pool.
The gap widens with block size, which is where a devnet comparison earns its keep — mainnet blocks rarely reach the sizes the spam wave produced here.
At the largest blocks the spread is 8×: reth validates a >40 M gas payload in a median of 105 ms
and nethermind in 132 ms, while geth takes 312 ms, besu 393 ms and nimbus-el 839 ms. None of these are
close to dangerous on a 12-second slot, and all of them are pre-release numbers on an unreleased fork —
but the scaling curves are the useful artefact here, and they are worth re-measuring as gas limits
climb. erigon does not report newPayload timings to Dora, so it has no row.
engine_getPayload — where the tail actually livesValidation is the well-watched half. The other half is engine_getPayload: the call a
proposer makes when it asks its own execution client to hand over a block. Under ePBS that call sits on
the critical path twice over — once for every proposer deciding whether to self-build, and once for
every builder assembling a bid — and unlike newPayload it is not retried or
pipelined. If it is slow, the slot is simply late.
Lighthouse exports a per-method engine-API histogram, so its seven nodes give the same controlled comparison the teku cohort gave for fill: one consensus client, one call, every execution client. Roughly 660–720 calls per client over the window — one per proposal.
Medians are unremarkable — 6 to 92 ms, all fine. The p99 is where the clients separate, and by 60×. nethermind (182 ms) and geth (195 ms) hold their tails tight. besu reaches 974 ms, nimbus-el 1.12 s, and ethrex 1.99 s. A two-second worst case to produce a block is the kind of number that is invisible in a median and decisive in a slot.
One honest caveat before reading too much into the ranking: getPayload cost scales with
what goes into the payload, and reth's payloads are small, for the reason §05 lays out. Its 34 ms
p99 is partly a consequence of that, not independent evidence of speed. The clients at the other end
of the table are the meaningful signal here.
Splitting the same measurement by demand regime shows these are not fixed costs. Under the spam wave every client's p99 rose; two of them rose a long way.
ethrex's getPayload p99 went from 197 ms at baseline to 3.99 s under load,
a 20× degradation, with its median moving from 87 ms to 593 ms. besu's p99 went 195 ms → 991 ms and its
median 78 ms → 555 ms. Both recovered when the spam stopped.
Hold that number for the next section. The builder that collapsed to 1–3 transaction blocks
through the spam wave was buildoor index 0 — an ethrex-backed builder — while the
geth-backed builder kept packing 600–800. A builder has a hard deadline to assemble
a bid; if getPayload starts taking multiple seconds, the builder ships whatever it has.
Two independent measurements, one explanation — and a much better lead than "the buildoor integration
is flaky". After the pool gap, this is the most actionable thing in the report.
Setting reth aside, how do the two build paths compare head to head? Under normal load, closely: both settle at ~103 transactions per payload in the post-spam period and ~88 in the baseline period. Under load they diverge sharply, and not in the builder's favour.
Through the spam wave, builder payloads fell to a median of 2–34 transactions while
proposers' own execution clients were packing 450–700. That is not the whole builder fleet failing —
it is one instance. Builder index 1 (geth-backed) packed 600–800 throughout; builder index 0
(ethrex-backed) produced 1–3 transaction blocks for the wave's entire duration and only recovered when
the spam stopped. Since standalone ethrex nodes self-built 652-transaction payloads over the same
hours, the block builder itself was clearly capable. §06 offers a likelier mechanism: ethrex's
engine_getPayload p99 reached 3.99 s during exactly these hours, and a
builder working to a bid deadline will ship whatever it has when the call runs long.
On timeliness, the metric ePBS actually scores, self-building wins outright. A self-built payload envelope hits the network at a median of 341 ms into the slot; a buildoor payload lands at 4,359 ms. Neither is close to the 9-second PTC deadline in normal operation, but the margin is 26× larger for local building.
That margin shows up in outcomes. Of 4,812 self-built payloads, zero were voted
absent by the PTC. Of 15,743 builder-built payloads, 138 (0.88%) were — every one of
them from a builder deliberately configured to withhold. From
buildoor-lodestar-ethrex-1's own log:
Published execution payload envelope slot=56520 ... builderIndex=2, isSelfBuild=false, dataColumns=128, delaySec=16.027, sentPeers=8
At 16 s the envelope arrives ~7 s past the PTC deadline, so the committee votes payload-absent unanimously and the payload is orphaned while the beacon block stays canonical. That is the experiment working as designed. It is worth stating plainly that the proposers it hit were spread across every execution client in proportion to node count — reth 7, nimbus-el 7, besu 6, geth 4, ethrex 3, nethermind 3, erigon 1 over the 36 affected slots we sampled. Nothing about the withholding outcome is client-specific.
Nothing here is a finished diagnosis — it localises the problem to reth's pool and leaves the mechanism open. Four cheap steps would settle it, and we're happy to run any of them and share the output:
RUST_LOG=reth_transaction_pool=debug on one reth node and capture the
rejection reason for a transaction that other clients accept. That single log line decides whether
this is an intrinsic-gas validation issue or a gossip/ingress one.eth_sendRawTransaction and see whether it is accepted into the pool — separates ingress
from validation without waiting for gossip.engine_getPayload under sustained load to find what takes it from
197 ms to 3.99 s at p99 — and check whether buildoor's bid deadline is what truncates its blocks.
besu's 5× tail growth over the same window is worth the same look.Every figure here comes from three sources joined on slot number: Dora's slot API for payload
contents and proposer identity, Xatu for builder index, envelope-gossip timing and PTC votes, and
direct node RPC for the mempool snapshot. Self-built is defined as
builder_index == 18446744073709551615 and cross-checked against the payload's
extraData — the two agree on 20,555 of 20,555 canonical slots. Engine-API latencies come
from Prometheus: newPayload medians and gas-bucket scaling from Dora's per-client slot
timings, and the getPayload and newPayload quantiles from lighthouse's
execution_layer_request_times histogram, which labels by method and execution client — so
that comparison is one consensus client measuring all of them. The deathstar node is
excluded from all per-client statistics; it is an adversarial-testing node, not a client sample.
# slot contents, proposer identity, extraData (paged by max_slot) curl "https://dora.glamsterdam-devnet-7.ethpandaops.io/api/v1/slots?limit=1000&max_slot=<N>" # builder index, envelope reveal timing, PTC votes — 3 days panda clickhouse query-raw clickhouse-raw " SELECT slot, any(builder_index) FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v2_beacon_block WHERE meta_network_name='glamsterdam-devnet-7' AND slot_start_date_time > now() - INTERVAL 3 DAY GROUP BY slot" # envelope reveal time, p10 across all 47 xatu sentries panda clickhouse query-raw clickhouse-raw " SELECT slot, quantile(0.1)(propagation_slot_start_diff) FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v1_events_execution_payload_gossip WHERE meta_network_name='glamsterdam-devnet-7' AND slot_start_date_time > now() - INTERVAL 3 DAY GROUP BY slot" # PTC verdict per slot panda clickhouse query-raw clickhouse-raw " SELECT slot, uniqExactIf(validator_index, payload_present=1) yes, uniqExactIf(validator_index, payload_present=0) no FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v1_events_payload_attestation WHERE meta_network_name='glamsterdam-devnet-7' AND slot_start_date_time > now() - INTERVAL 3 DAY GROUP BY slot" # mempool snapshot panda execute <<'PY' from ethpandaops import ethnode print(ethnode.execution_rpc("glamsterdam-devnet-7","teku-reth-1","txpool_status",[])) PY # engine_getPayload p99 per execution client, 3 days (lighthouse cohort) panda prometheus query devnets " histogram_quantile(0.99, sum by (execution_client, le) ( increase(execution_layer_request_times_bucket{ network='glamsterdam-devnet-7', method='get_payload'}[3d])))" # the withholding builder's own admission panda clickhouse query-raw clickhouse-raw " SELECT Timestamp, Body FROM external.otel_logs WHERE ResourceAttributes['network']='glamsterdam-devnet-7' AND ResourceAttributes['host.name']='buildoor-lodestar-ethrex-1' AND Body LIKE '%Published execution payload envelope%' AND Timestamp > now() - INTERVAL 1 HOUR"
Caveats. Median tx count is a demand-dependent measure; that is why the control cohort and the three-regime split exist, and why the leaderboard is reported per regime as well as overall. Non-teku consensus clients self-build so rarely that their self-built samples are selection-biased toward good blocks — they are shown in the appendix table for completeness but are not the basis of any claim. erigon is excluded from every ranking: its validators were activated ahead of the nodes finishing sync, so its missed-slot and proposal counts measure the test setup rather than the client. The mempool figures are a single snapshot, not a time series. And this is a devnet running an unreleased fork against specs that were still moving a fortnight ago — every client here is chasing a target that changed under it. A gap on devnet-7 says nothing about a client's released builds, and finding one at this stage is the entire point of running the devnet.