Who fills blocks? Round two.

Eight days after the first look at self-built blocks, we ran a second, heavier spam wave — 23 hours above 850 transactions per block, up from round one's ~700 peak. The builder collapse that defined round one did not repeat: every fix the first report asked for shows up in this data. But the load found two new places to break. Proposers started skipping slots — 1,187 of them, in a pattern that points at specific CL pairings — and the builder path turns out to quietly drop 1 in 47 of the payloads it wins, while the self-build path has still never lost one.

01The setup

Same experiment, harder push, changed cast

The measurement is the same as round one: glamsterdam-devnet-7 runs Gloas/ePBS on 12-second slots, proposers choose each slot between a buildoor builder bid and a payload their own execution client built, and a ~100-member payload-timeliness committee (PTC) votes at t+9 s on whether the winning payload actually showed up. Self-built payloads are identified from the bid's builder index and cross-checked against the payload's extraData — the two agree on 29,672 of 29,672 revealed payloads. (One mechanical change since round one: the old 264−1 self-build sentinel is gone; a self-built bid now simply carries no builder index. Scripts that still filter on the sentinel will find three slots and conclude self-building died. It didn't.)

Four things about the network are different, and they matter for reading the charts:

• The spam wave was heavier and longer: 2026-07-27 09:00 → 07-28 08:00 UTC, median 850–1,020 txs/block sustained for 23 hours, self-built payloads peaking at an hourly median of 1,614. Round one peaked around 700.
• The builder fleet changed: three active builders — b0 and b2 ethrex-backed, b1 geth-backed. The nethermind-backed instance is deployed but won essentially no bids in the window. The delaySec=16 withholding experiment from round one is retired.
• erigon is now a real sample. Its round-one numbers were a test-setup artefact (validators activated before sync finished). This window it proposed 1,589 times, missed 0.7% of its duties, and joins every ranking.
• An adversarial node called deathstar now proposes — and gets ~74% of its blocks orphaned, on purpose, ~200 a day through 07-28. It is excluded from every per-client statistic, same as round one.

The window is 2026-07-26 00:00 → 07-30 10:23 UTC, split into baseline (33 h), spam (23 h) and post-spam (50 h). Post-spam demand settled at ~120–190 txs/block — noticeably above the ~103 it held before the wave, so "post" is not "back to baseline" this time.

02The second wave

Both build paths held. That's the headline round one couldn't write.

Round one's most dramatic chart was the builder path collapsing to 2–34 transaction blocks for a full day while self-builders packed 450–700. That did not happen again. Under a heavier wave, all three builders kept packing — hourly medians of 772–984 transactions through the spam window — and the two paths tracked each other the whole way. The fix implied by round one's §06 (ethrex's engine_getPayload collapsing under load, starving the builder against its bid deadline) appears to be in and working: ethrex's spam-regime getPayload p99 went from 3.99 s in round one to 0.95 s now, against a bigger wave.

The chart also shows the week's two other incidents, for orientation: the 07-23 afternoon dip to ~8 txs/block is the geth payload-divergence event (a known, separately-tracked bug — geth-built payloads were being rejected by nethermind and nimbus-el validators; prysm nodes orphaned 159 blocks that day), and the small notch on 07-24 is a one-day teku-nimbusel incident (144 orphans). Both predate this report's analysis window, which starts 07-26.

03The control group, again

Teku still anchors the comparison — but lodestar changed its mind

As in round one, consensus-client bid policy decides who self-builds, and teku still effectively always prefers its local payload: 81.3% self-build rate (83.5% in round one). Its seven teku-<el>-1 nodes remain the unbiased control cohort — same consensus client, same slots, only the execution client varies — and now include teku-erigon.

The interesting mover is lodestar: 6.0% → 21.3%, and it is not a policy change. The jump ramps in over 07-25 morning (9% → 30% in ten hours), identically on all six lodestar nodes, with no deploy and no restart — every node ran the same build (v1.44.0/94954a2) untouched from 07-22 to 07-28. Nor is it a value preference: when lodestar takes a bid its log shows the bid worth ~2× its local block, and on self-built slots there is no "Selected builder bid block" line at all. Lodestar self-builds when no eligible bid is on hand at its decision point — a bid must match its view of the parent block — so what actually grew on 07-25 is the fraction of slots where the builders' pre-slot bids don't line up with lodestar's parent view in time. Bid values, bid-gossip timing (~380 ms before slot start) and base fee (pinned at 7 wei) were all flat across the change, and every other CL's no-bid fallback rate stayed put at 1.7–4.6% — which makes this a lodestar-specific bid-eligibility or timing behaviour worth a look in its own right. Either way, self-build spec coverage now rests on teku plus an involuntary lodestar contribution.

04The leaderboard

Median transactions in a self-built payload — teku cohort

Six of seven clients now cluster between 83 and 192 transactions at a median — everyone drifted up with the heavier ambient load, and erigon slots into the middle of the pack on its first clean measurement. reth improved from 6 to 15 — and is still an order of magnitude below its peers, with a 75th percentile (48) beneath everyone else's 25th.

Two genuinely encouraging details in reth's row. Its blob median moved off the clean zero of round one to 1 — blob transactions have started clearing whatever was blocking them, which weakens the "one hard ingress wall" reading. And its gas-per-transaction is unremarkable (90 K), so what it packs is normal traffic; there is simply ten times too little of it. The diagnosis from round one — the payload builder is fine, the transaction pool is starved — still stands, and §05 shows the pool is still where the gap lives.

The gap still holds at every load level

Under the wave, the control cohort scaled to 822–1,102 median transactions; reth went 3 → 48 → 24. The ratio to its peers is roughly constant across a 10× demand change, which keeps the conclusion from round one intact: the limit is upstream of demand and upstream of the block builder.

05The mempool, revisited

Still near-empty on reth — with one curious exception

txpool_status across every node that exposes it, sampled 2026-07-30 ~10:30 UTC, with the network producing ~120–145 transaction blocks. erigon, newly measurable, holds a healthy pool. Five of six reth nodes hold 1–11 pending transactions where their neighbours hold 92–340.

The exception is grandine-reth-1 at 128 pending — the first reth pool observed holding a normal transaction count. One node, one snapshot, but it is a lead: if one reth instance can accumulate a pool, the difference between it and the other five (peering, configuration, restart timing, version) is a much smaller search space than "all of reth's ingress". The round-one ask — RUST_LOG=reth_transaction_pool=debug on one node to capture a rejection reason — is still the fastest way to close this, now with a natural A/B pair to compare.

06The builder path drops payloads

504 canonical slots whose payload never arrived — zero of them self-built

This is the finding that replaces round one's withholding experiment, and it is organic. A canonical beacon block whose execution payload is never revealed leaves an empty slot: the chain keeps the block, the PTC votes the payload absent, and every transaction in it evaporates. In this window that happened on 533 canonical slots (1.8%). For the 504 of them where the winning bid is recorded, the split is absolute: every single one was builder-won. No self-built payload has gone absent in either measurement window — 0 for 11,294 across both reports.

The failure is not uniform across the fleet. Builder 0 fails to reveal 3.9% of the bids it wins — 4× the rate of its two peers — and its own log says why:

level=warning msg="Reveal vote gate never opened, withholding payload"
  component=reveal-service slot=114688 threshold=60   ← buildoor-prysm-ethrex-1

This is the honest-builder safety rule working as designed: the builder reveals only once it has seen enough attestation weight on the proposer's beacon block, and if that view never crosses the threshold before the deadline, it keeps the payload. So the question isn't "why does buildoor drop payloads" but "why does b0's view of attestations fail to cross 60% in time, 4× more often than b1's or b2's?" The suggestive detail: b0 is the builder whose consensus-side view comes from prysm — the CL with a known envelope-sync wedge on this devnet — while b1 rides lighthouse and b2 lodestar. A per-builder comparison of attestation-arrival timing would settle it, and unlike round one's diffuse "buildoor is flaky", this is one node, one threshold, one log line.

Two scale notes. The absolute numbers rose post-spam (71/day → ~170/day) mostly because b0 also started winning far more bids post-spam, not because its failure rate exploded. And at network level this is the difference between the two build paths' reliability: a builder-won slot delivered its payload 97.9% of the time this window; a self-built slot, 100.0%.

07The new failure mode: missed slots

The wave stopped breaking payloads and started breaking proposals

Round one's wave produced almost no missed slots. This one produced 1,187 — 3.7% of the window — and they are not evenly spread. They concentrate hard in the spam window (07-27 06:00 → 07-28 12:00), on specific consensus–execution pairings:

Read the besu column of that chart as a controlled experiment: the same execution client, on the same network, missed 282 slots behind nimbus, 217 behind lodestar, 57 behind lighthouse, 18 behind teku and 14 behind prysm. That 20× gradient across consensus clients cannot be a pure EL problem. Meanwhile nethermind's misses were almost entirely the nimbus pairing (276 of 280) and geth's mostly lodestar (184 of 232) — while reth, ethrex, erigon and nimbus-el nodes sailed through the same wave missing almost nothing (9–44 each).

The shape that fits: under load, block production on the slower-getPayload execution clients (besu, and to a lesser degree geth and nethermind under spam — §08) runs long, and nimbus and lodestar's proposal paths give up where teku and prysm's wait. Whether that is a timeout constant, a missing async path, or resource contention on those specific VMs is exactly the kind of thing worth one focused look while the next wave runs — it is the largest single source of lost slots on the devnet and it did not exist as a failure mode in round one. Worth noting besu also missed round one entirely with zero misses, so this is new behaviour under the new, heavier wave, not a chronic besu trait.

08Engine API performance

Validation: the ordering shuffled since round one

Dora records per-client newPayload timings for every slot — like-for-like, since every client validates the same payload. Round one's ranking (reth fastest, nethermind close, geth/besu mid-pack) has reshuffled, partly because blocks are bigger now:

nethermind now leads at 19 ms with reth at 22 ms. The mover is geth: from mid-pack to the second-worst tail — its p90 (343 ms) and lighthouse-measured p99 (1.52 s) are now besu-territory and worse, and the gas-bucket view below shows why: above 80 M gas its median validation hits 279 ms, 4–5× nethermind and reth. nimbus-el remains last at every size, breaking half a second at the top bucket. erigon, measurable for the first time, lands mid-pack.

getPayload: round one's two fires are out; the tails that remain

Same lighthouse-cohort measurement as round one: one consensus client, every execution client, ~900 proposals each. Both of round one's headline getPayload findings are resolved in this data — ethrex's under-load collapse (p99 3.99 s → 0.95 s) and besu's failure to recover after the wave (post-spam p90 727 ms → 179 ms). What remains is a familiar shape: medians all comfortable at 6–89 ms, tails separating by 8×.

Under spam, nimbus-el (1.37 s) and besu (0.97 s) still brush the region where a builder or proposer working to a deadline starts making bad choices — §07 suggests some CLs already are. And one oddity for completeness: reth's post-spam p99 jumped to 541 ms from a near-zero base — tiny payloads make its fast numbers easy, which makes a half-second worst case stand out. Worth a glance if it persists once its pool fills.

09Reveal timing

The builder path got 2.3× faster — and one builder reveals five seconds in

Self-built payload envelopes reach the network at a median of 392 ms into the slot; builder payloads at 1,895 ms — down from 4,359 ms in round one, a 2.3× improvement in builder reveal latency. The margin over the 9-second PTC deadline is comfortable on both paths, and only a single revealed payload in the whole window arrived past 9 s (round one: 126, all from the retired withholding experiment).

The builder histogram is bimodal, and the two humps are two builders: b0 and b2 (ethrex-backed) reveal around 1.2–2.5 s, while b1 (geth-backed) consistently reveals at ~5.1–5.3 s, baseline or spam, busy or idle. That is still 4 s clear of the deadline, but it is a fixed behaviour worth understanding before anyone raises the stakes — a builder that idles at t+5 s has less headroom for a bad day than one that idles at t+1.5 s.

10Other things the data coughed up

• Block-level orphans stayed negligible once deathstar is excluded: 12 orphaned blocks in 31,917 slots — the identical count to round one. Fork choice remains boring, which is what you want it to be.
• deathstar's campaign didn't destabilise anyone. ~1,350 deliberately orphaned proposals over the week, ~200/day, wound down on 07-29; the network absorbed all of it without a finality wobble.
• The Xatu sentry fleet had an outage 07-24 → 07-27 (median sentries per slot: 15 on 07-26, 42+ from 07-27). That is why this report's window starts 07-26 and why round-one-style gossip-timing analysis of the 07-23/24 incidents isn't in it.
• The lodestar-ethrex pairing is gone from the validator inventory. The node that spent late July with ~99% of its proposals orphaned no longer exists in the current mapping, and because Dora resolves proposer names retroactively, its historical duties now display under other lodestar node names — that is what lodestar-geth-1 "missing" 142–214 slots a day on 07-22 → 24 and then proposing normally from 07-25 actually is. Close the incident as "reassigned", not "fixed": the fork-choice question it raised was never answered.

11Scorecard — round one → round two

12What would close this out

• reth pool: same ask as round one — RUST_TX_POOL debug logging on one starved node — plus the new A/B: diff grandine-reth-1 (128 pending) against any starved sibling. Config, peer set, uptime, version.
• b0's reveal gate: compare attestation-arrival timing on buildoor-prysm-ethrex-1 vs the lighthouse- and lodestar-fed builders during a busy hour; if prysm's attestation view is late, this is a one-line threshold or a prysm fix, and 366 empty slots a week come back.
• Missed-slot pairings: instrument one nimbus-besu and one lodestar-geth proposal end-to-end under the next wave (getPayload issue → block publish or give-up), and pull the CL-side proposal timeout constants for nimbus and lodestar vs teku.
• geth validation tail: profile newPayload on >80 M gas blocks — 279 ms median at the top bucket is new since round one and the trend line matters more than the number.
• b1's fixed 5-second reveal: find out whether it's configuration or code; it is the least margin any honest party keeps on this network.
• lodestar's eligible-bid shortfall: debug-log the bid rejection reason at decision time on one lodestar node (parent mismatch vs arrival timing vs cutoff) and correlate with whatever changed network-side over 07-25 morning — the ramp was gradual, so the cause is a drifting condition, not a switch.

13Method & reproduction

Same joins as round one: Dora's slot API for payload contents, proposer identity and per-client execution timings; Xatu ClickHouse for the winning bid's builder index, envelope-gossip timing and payload-status resolution; lighthouse's per-method engine-API histograms in Prometheus for getPayload/newPayload quantiles; direct node RPC for the mempool snapshot. Self-built = no builder index on the canonical bid (the round-one sentinel is retired), cross-checked against extraData: 29,672 of 29,672 agree. Payload-absent = canonical beacon block with no execution payload, cross-checked against Xatu's beacon_synthetic_payload_status_resolved (status EMPTY) where its coverage overlaps. deathstar is excluded from all per-client statistics.

# slot contents, proposer identity, extraData, per-client exec times (paged)
curl "https://dora.glamsterdam-devnet-7.ethpandaops.io/api/v1/slots?limit=1000&max_slot=<N>"

# winning bid per slot — NULL builder_index = self-built (sentinel is retired)
panda clickhouse query-raw clickhouse-raw "
  SELECT slot, any(builder_index) FROM \`glamsterdam-devnet-7\`.canonical_beacon_block_execution_payload_bid
  WHERE meta_network_name='glamsterdam-devnet-7' AND slot BETWEEN 82800 AND 114716 GROUP BY slot"

# payload-absent slots, authoritative (coverage starts 2026-07-28)
panda clickhouse query-raw clickhouse-raw "
  SELECT slot FROM \`glamsterdam-devnet-7\`.beacon_synthetic_payload_status_resolved
  WHERE meta_network_name='glamsterdam-devnet-7' AND status='EMPTY' GROUP BY slot"

# envelope reveal time, p10 across sentries
panda clickhouse query-raw clickhouse-raw "
  SELECT slot, quantile(0.1)(propagation_slot_start_diff)
  FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v1_events_execution_payload_gossip
  WHERE meta_network_name='glamsterdam-devnet-7' AND slot BETWEEN 82800 AND 114716 GROUP BY slot"

# the reveal gate, in the builder's own words
panda clickhouse query-raw clickhouse-raw "
  SELECT Timestamp, Body FROM external.otel_logs
  WHERE ResourceAttributes['network']='glamsterdam-devnet-7'
    AND ResourceAttributes['host.name']='buildoor-prysm-ethrex-1'
    AND Body LIKE '%withholding payload%' AND Timestamp > now() - INTERVAL 6 HOUR"

# engine_getPayload p99 per execution client (lighthouse cohort), by regime
panda prometheus query devnets "
  histogram_quantile(0.99, sum by (execution_client, le) (
    increase(execution_layer_request_times_bucket{
      network='glamsterdam-devnet-7', method='get_payload'}[23h])))"   # @ regime end

# mempool snapshot
panda execute <<'PY'
from ethpandaops import ethnode
print(ethnode.execution_rpc("glamsterdam-devnet-7","grandine-reth-1","txpool_status",[]))
PY

Caveats. Median transaction count remains demand-dependent; the control cohort and regime split exist for exactly that reason, and "post-spam" demand (~120–190 txs) is higher than round one's, which inflates every client's round-two medians a little — compare ratios, not raw counts, across reports. The Xatu sentry outage means 07-26 reveal timings ride on ~15 sentries instead of ~45; per-slot p10s that day are noisier. 29 payload-absent slots have no bid row in the canonical table and are attributed to neither path. The mempool figures are a single snapshot. And the standing caveat carries over verbatim: this is a devnet running an unreleased fork — a gap here says nothing about released builds, and surfacing gaps at this stage is the entire point.