The bid that arrived on time and validated late

The round-two block-building report left one question open: why did lodestar's self-build rate jump from 6% to 21% on 07-25, with no deploy, no restart, and builder bids worth twice its local blocks? This report answers it, from source code and per-node evidence. The bids were on the wire 388 ms before the slot, correctly keyed, everywhere. What changed is inside the node: lodestar's own gossip validation of the bid — the only path into the pool its proposal flow reads, exactly once, at slot start — developed a tail that crosses the deadline. When the proposer looked, the bid was still in the pipeline on one slot in five.

01What the code says

There is no value comparison. There is one map lookup.

lodestar's Gloas proposal flow (produceBlockV4, packages/beacon-node/src/api/impl/validator/index.ts) decides the build path in two steps, both synchronous:

Three properties of this code decide everything downstream. First, the value comparison everyone assumes exists is not implemented yet — the code says so itself:

// TODO GLOAS: respect builderSelection (MaxProfit, BuilderAlways, ExecutionAlways, etc.) to let
// the user control bid source preferences and value comparison. Also add external builder api
// support when it is implemented.

An eligible bid is taken unconditionally; self-build is purely the no-bid fallback. Second, the pool lookup demands an exact triple — slot, parent beacon-block root, and parent execution-block hash (which encodes the proposer's FULL/EMPTY view of the parent payload):

getBestBid(slot, parentBlockHash, parentBlockRoot) {
  // exact (slot → parentRoot → parentHash) map walk; no fuzzy matching
  return this.bidByParentHashByParentRootBySlot.get(slot)?.get(parentBlockRoot)?.get(parentBlockHash) ?? null;
}

And third — the load-bearing one — the read happens once, at slot start, with no wait, no re-check, and no event-driven wake-up. A bid that enters the pool one millisecond after this line runs might as well not exist.

The only path into that pool is gossip validation (chain/validation/executionPayloadBid.ts), a 13-check gauntlet that includes two expensive steps: a state regen on the bid's parent branch (chain.regen.getBlockSlotState(..., {dontTransferCache: true})) and a BLS signature verification. The bid's clock therefore runs: publish → gossip hops → validation queue → regen → BLS → pool. The proposal flow's clock runs: slot start → one lookup. Whoever finishes first wins, and nothing retries.

02The bid's timeline

buildoor fires at −400 ms. The margin was never large.

The builder side is equally deterministic. buildoor's scheduler (pkg/p2p_bidder/scheduler.go, defaults in pkg/config/default.go) publishes its first bid at −400 ms before slot start and its last at −100 ms, with a 500 ms re-bid interval — meaning in practice each builder gets one shot per slot. Bidding is additionally gated on the builder's payload being built, the proposer's gossiped preferences being cached, and stops the moment a block for the slot is seen.

Measured against this design, the network did its part with room to spare: bids were first seen by Xatu sentries at a median of −388 ms, every day, with no drift — and the distribution is identical for slots where lodestar took the bid and slots where it didn't. The race is not on the wire. It is inside the node, in the ~390 ms between arrival and the proposal flow's single read.

03Eliminating everything else

Before pointing at validation latency, every other explanation was tested and killed:

The funnel below is the same elimination expressed on the self-built slots themselves. Nearly every self-built slot had a bid that existed, was early, was keyed to the exact parent root and parent hash the proposer used, and — where the proposing node's own event stream lets us check — was validated by that very node, just too late.

Two real but minor contributors are worth naming, because they explain the days the main line doesn't. The builder circuit breaker — which post-Gloas counts payload non-reveals, not missed blocks — activated 5–9 times on every lodestar node on 07-23 (the geth payload-divergence day) and briefly on 07-30, and on minimal preset its window is just 8–15 slots with 2–3 allowed faults, so a short burst of withheld payloads flips it. That is what pushed 07-23 to 32% self-builds while validation-lateness alone predicts ~15%. And ~4% of self-built slots genuinely had no bid for their parent branch — the builders' occasional miss, not lodestar's.

04The proof

The proposing node's own clock, slot by slot

Xatu's execution_payload_bid events come from each node's own beacon-API event stream — a lodestar node emits the event only after its gossip validation admits the bid to the pool. That timestamp, joined per slot to the node that proposed, is the decision's actual input, and it splits the two outcomes with almost no overlap:

The separation is not marginal. When the bid was taken, the node had it validated at a median of −365 ms — about 25 ms after wire arrival, in line with every other client. When the node self-built, its validation of the same-keyed bid completed at a median of +510 ms, p90 +1.03 s — half a second to a full second after the only moment it would ever be consulted. The lateness is bimodal, not a smear: when validation loses, it loses by hundreds of milliseconds, which points at discrete stalls (a queue behind an expensive item, an event-loop pause, a regen cache miss) rather than a uniformly slower pipeline.

05The drift

The tail is the self-build rate

Plotting the daily fraction of bids a lodestar node validated late against lodestar's daily self-build rate turns the round-two mystery into a single curve. Every other consensus client sits at 0.4–0.9% late throughout — flat, for the same bids, on the same wire.

What the curve does not explain is its own cause. The tail fattened over 07-25 morning (2.4% late on 07-22 → 24% by 07-26) with no code change — and, notably, the 07-28 restart onto a new commit did not reset it, which rules out a slow leak tied to process uptime. Whatever drifted is workload-shaped: candidate suspects are the per-bid state regen (the one expensive, chain-state-dependent step in the gauntlet), gossip-queue contention from the growing message volume, or BLS batch pressure from the 512-member PTC — all of which grew as the devnet's traffic did. Separating them needs lodestar's own metrics (gossip_validation_queue_*, regen timings), which is the concrete next step; the mechanism itself is no longer in question.

06What this means

• Nothing is losing blocks. The fallback works: every one of those self-built payloads landed, revealed at ~390 ms, and none was ever voted absent. On this devnet the failure is invisible without exactly this analysis.
• On mainnet economics this is forfeited revenue. The bids left on the table were worth ~2× the local blocks. A validator running this code against real builders would silently under-earn on every slot the race is lost — ~20% of them at current tail size.
• The window is structurally ~0. buildoor's last bid (−100 ms) and lodestar's read (t≈0) leave less than 100 ms of designed margin, all of which is consumed by validation. Either side can fix it: lodestar by re-checking the pool (or waiting briefly / waking on pool insert) before falling back, and by getting the regen out of the per-bid hot path; buildoor by bidding earlier and actually re-bidding. Any one of these erases the effect.
• The devnet accidentally built the right experiment. A CL that always takes bids would never reveal this; a CL that always self-builds wouldn't either. lodestar's no-margin fallback made its own validation latency observable on-chain — worth keeping as a test pattern, deliberately next time.

07Method & reproduction

Sources read at the exact running versions: lodestar 94954a2 (produceBlockV4, executionPayloadBidPool.ts, validation/executionPayloadBid.ts, builderCircuitBreaker.ts) and buildoor c5339a5 (p2p_bidder/scheduler.go, config/default.go). Chain data: Dora slots API for proposer/status, canonical bid table for the winning bid and each slot's payload hash, beacon_api_eth_v1_events_execution_payload_bid for per-node bid sightings (the event fires post-validation, which is what makes the per-node join meaningful), and external.otel_logs for lodestar's decision and circuit-breaker log lines. Parent-hash matching uses each parent slot's canonical payload hash (FULL view). Slots on 07-24→26 ride the reduced Xatu sentry fleet; per-day values those days are thinner but directionally consistent.

# the decision, in lodestar's own words (bid taken vs self-build has no 'Selected' line)
panda clickhouse query-raw clickhouse-raw "SELECT Timestamp, Body FROM external.otel_logs
  WHERE ResourceAttributes['network']='glamsterdam-devnet-7'
    AND ResourceAttributes['host.name']='lodestar-besu-1'
    AND Body LIKE '%Selected builder bid block%' AND Timestamp > now() - INTERVAL 1 HOUR"

# per-implementation bid validation lateness (the smoking gun)
panda clickhouse query-raw clickhouse-raw "
  SELECT meta_consensus_implementation, round(median(propagation_slot_start_diff)) p50,
         round(quantile(0.9)(propagation_slot_start_diff)) p90,
         round(100*countIf(propagation_slot_start_diff >= -20)/count(),1) late_pct
  FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v1_events_execution_payload_bid
  WHERE meta_network_name='glamsterdam-devnet-7'
    AND slot_start_date_time > now() - INTERVAL 1 DAY
  GROUP BY meta_consensus_implementation"

# circuit breaker activations
panda clickhouse query-raw clickhouse-raw "SELECT toStartOfDay(Timestamp) d, count() FROM external.otel_logs
  WHERE ResourceAttributes['network']='glamsterdam-devnet-7'
    AND ResourceAttributes['host.name'] LIKE 'lodestar-%'
    AND Body LIKE '%circuit breaker activated%'
    AND Timestamp > now() - INTERVAL 7 DAY GROUP BY d"

# buildoor's bid window defaults
git -C ~/repos/buildoor grep -n "BidStartTime\|BidEndTime" pkg/config/default.go