How long does bid validation take?

Companion charts to the bid-race investigation. Per bid message, latency is each node's own beacon-event emission (which fires after gossip validation admits the bid to the pool) minus the earliest emission anywhere on the network — so it bundles one gossip hop with queueing and validation. The same bids traverse the same mesh for every client, which makes the cross-client comparison a controlled one: 2.58 M node-level observations, 2026-07-22 → 07-31, buildoor's own sentry excluded.

01The percentile curves

02The two modes

03The drift, at 6-hour resolution

Reading the three together: validation itself is cheap — every client, lodestar included, clears half of all bids in well under 25 ms. What lodestar has that nobody else has is a second mode: ~16% of bids take 400–1,500 ms, a hump, not a tail. The proposal flow reads the bid pool ~390 ms after the bid hits the wire, so everything in that hump arrives too late and becomes a self-built block. The hump first appears late on 07-22, hardens through 07-25, and survives the 07-28 restart — whatever feeds it is workload-shaped, not accumulated process state. The two named suspects remain the per-bid state regen and gossip-queue contention; deciding between them needs lodestar's internal queue metrics rather than external observation.

04Reproduction

# per-implementation bid pipeline latency (node emission − first network sighting)
panda clickhouse query-raw clickhouse-raw "
WITH per_node AS (
  SELECT slot, builder_index, parent_block_hash, meta_client_name AS node,
         any(meta_consensus_implementation) AS impl, min(propagation_slot_start_diff) AS t
  FROM \`glamsterdam-devnet-7\`.beacon_api_eth_v1_events_execution_payload_bid
  WHERE meta_network_name='glamsterdam-devnet-7'
    AND slot_start_date_time > now() - INTERVAL 1 DAY AND meta_client_name NOT LIKE '%buildoor%'
  GROUP BY slot, builder_index, parent_block_hash, meta_client_name),
firsts AS (SELECT slot, builder_index, parent_block_hash, min(t) AS t0 FROM per_node GROUP BY slot, builder_index, parent_block_hash)
SELECT impl, round(quantile(0.5)(p.t - f.t0)) p50, round(quantile(0.9)(p.t - f.t0)) p90,
       round(quantile(0.99)(p.t - f.t0)) p99
FROM per_node p INNER JOIN firsts f USING (slot, builder_index, parent_block_hash)
GROUP BY impl ORDER BY p50"