Fast Confirmation Rule on Gloas — safe-block behaviour under payload withholding

Network: glamsterdam-devnet-7 (production) · Date: 2026-07-22 · Author: ethPandaOps

Follow-up to FCR Breakout #11: testing FCR implementations on a Glamsterdam devnet, specifically the
invariant "the safe block on EL is at least one slot behind a confirmed beacon block."

Summary

1. Four CL clients implement FCR, not three. Lighthouse has a complete implementation
   (--enable-fast-confirmation) that had not come up in the breakout discussion. Teku and Prysm have none.
2. All three available implementations — nimbus, lighthouse, lodestar — are now enabled on this devnet and
   are confirming at 1–2 slots behind head.
3. The invariant holds in practice. Under a builder withholding ~25% of payloads, no client ever handed
   the EL a hash for an unrevealed payload, safe never moved backwards, and there were no engine-API
   errors.
4. Lighthouse's FCR code resolves the safe hash from the confirmed block's own payload, with the parent
   only as a fallback — the inverse of what specs/gloas/fast-confirmation.md requires, and the inverse
   of what its own code comment says.
5. The precondition for that to matter exists on this network: payloads revealed late and then voted
   absent by the PTC survive on geth and besu as non-canonical side blocks, and are missing entirely on reth.
6. consensus-specs #5449 adds a
   safe_execution_block_hash property to the FCR test vectors. That is the instrument that settles
   own-vs-parent per client deterministically — devnet observation cannot.

1. The spec

specs/gloas/fast-confirmation.md is identical at v1.7.0-alpha.12 (the tag this devnet pins) and at
consensus-specs master (e310d1c0d, 2026-07-21). It was last modified 2026-05-27. No open PR changes the
rule.

### Modified `get_safe_execution_block_hash`

# Note: In Gloas, only the parent payload of a confirmed beacon block is safe.

def get_safe_execution_block_hash(fcr_store: FastConfirmationStore) -> Hash32:
    safe_block = fcr_store.store.blocks[fcr_store.confirmed_root]
    # [Modified in Gloas:EIP7732]
    return safe_block.body.signed_execution_payload_bid.message.parent_block_hash

The rule is unconditional — always the parent, whether or not the confirmed block's own payload was
revealed. The companion get_node_for_root modification returns payload_status=PAYLOAD_STATUS_PENDING in
Gloas, i.e. FCR treats a confirmed block's payload as pending by construction.

Lineage: #5197 introduced the Gloas rule;
#5278 moved the definitions out of
fork_choice/safe-block.md into the per-fork fast-confirmation.md files. fork_choice/safe-block.md no
longer exists — client code linking it is citing a deleted file.

2. Network under test

A builder withholding payloads throughout is the ideal condition for this test — the Gloas parent-hash rule
exists specifically to handle unrevealed payloads.

3. Implementation inventory

Checked against each client's glamsterdam-devnet-7 branch and the running image:

Lighthouse ships a dedicated consensus/fast_confirmation crate, an SSE fast_confirmation event
(beacon-APIs#616), EF fork-choice test wiring and a beacon_fcr_* metrics family. Nimbus has been running
FCR in production on this devnet by default, feeding a near-head safe block to all seven EL clients.

4. Behaviour under withholding

All four CLs sampled simultaneously against the same EL (besu), each safe hash resolved back to the
beacon slot whose bid produced it — 64 samples, slots 56945–56964, 4 withheld slots in window:

Across the full data set — 303 samples, ~60 slots, 15 withheld:

• 0 samples on any client where safe equalled an unrevealed payload hash
• 0 instances of safe moving backwards
• 0 ExecutionForkChoiceUpdateFailed / Failed to update execution head outside the rollout restart
  windows

The three FCR implementations are behaviourally indistinguishable on this network. Non-FCR clients sit
35–44 slots back, as expected from the justified checkpoint.

The justified / finalized path is correct on every client

Both checkpoints tested were verified FULL — the bid's block_hash was canonical on the EL — so this is
not an artefact of empty checkpoint blocks:

Verified on lighthouse, lodestar, teku, prysm and nimbus.

5. Lighthouse resolves the safe hash from the confirmed block's own payload

beacon_node/beacon_chain/src/canonical_head.rs:

// Resolve the confirmed block's execution payload hash for the EL `safe_block_hash`.
// This MUST be the parent block hash for Gloas, per the spec.
let confirmed_block_hash = confirmed_node
    .execution_status
    .block_hash()                                     // confirmed block's OWN payload hash
    .or(confirmed_node.execution_payload_parent_hash) // parent — only as a fallback
    .ok_or(FastConfirmationError::NodeHasNoBlockHash(fcr.confirmed_root))?;

The comment states the requirement; the expression inverts it. Two hash-level samples, taking lighthouse's
own beacon_fcr_confirmed_root_slot as the confirmed slot and reading safe live:

confirmed_slot=56657  safe=#51227 0xe982d6f04420
    bid.block_hash        = 0xe982d6f04420   <- safe matches this
    bid.parent_block_hash = 0x7363e4f3717e

confirmed_slot=56755  safe=#51286 0x812b01e782   -> own block_hash again

Because the fallback branch is taken whenever the payload was never revealed, this converges on the spec's
answer in the fully-withheld case — which is why §4 shows no divergence. The gap is the window in which a
payload has been revealed but fork choice has not yet marked the block empty.

For contrast, nimbus is explicit and unconditional —
beacon_chain/consensus_object_pools/attestation_pool.nim:

# In Gloas a bid's `block_hash` is only on the EL's canonical chain when fork
# choice marked the slot's block full; the bid's `parent_block` must be on it.
safeBlockRoot = pool.forkChoice.retrieve_fast_confirmed_root()
...
elif safeBlock.get.slot.epoch >= pool.dag.cfg.GLOAS_FORK_EPOCH:
  pool.dag.loadExecutionAndParentBlockHash(safeBlock.get).parentHash.get(...)

Lodestar's getSafeExecutionBlockHash (packages/fork-choice/src/forkChoice/safeBlocks.ts) has no
explicit Gloas branch — it returns confirmedBlock.executionPayloadBlockHash and falls back to
ZERO_HASH, relying on the implicit parent semantics of that ProtoBlock field, and its doc comment links the
deleted fork_choice/safe-block.md. Making the Gloas case explicit would remove the dependence on an
implicit invariant.

6. Why this matters here: revealed-then-excluded payloads

Withheld ≠ never revealed. Slot 56678:

• 08:55:52 — the payload envelope for slot 56678 is published (0x518ed79e88…, builderIndex=2).
• The PTC still votes payload_present = false — the reveal was past the attestation deadline.
• Fork choice skips it; the next bid's parent_block_hash is 0xc528099af4…, slot 56677's payload.

What the ELs did with that payload:

The excluded payload survives as a non-canonical side block on geth and besu, and does not exist on reth.
Naming such a payload as safeBlockHash would point the EL's safe tag at a non-canonical block, or at a
hash the EL has never seen. This is exactly what the parent rule prevents, and this network produces the
precondition at a ~25% rate.

7. Operational findings

Two lodestar nodes stalled after the rollout restart. lodestar-geth-1 and lodestar-ethrex-1 were at
head (~56795) at 09:19; after restarting at 09:29 both fell back to exactly slot 56351 and stopped:

Syncing - ? left - 0.00 slots/s - slot: 56941 - head: (slot -590) 0xefab…e12a
  - exec-block: syncing(51026 0x2f70…) - finalized: 0x139f…7937:1759 - peers: 2

Both sit in a CL-behind / EL-syncing deadlock with 2–6 peers and 0 slots/min. The other six lodestar nodes
restarted in the same window and recovered normally. They need a resync.

lighthouse-nimbusel-1 never confirms. beacon_fcr_confirmation_delay_slots = 87, versus 1 on every
other lighthouse node, and its EL safe is pinned to finalized. Not a sync problem — the beacon is at
head. It also has the highest revert count in the fleet.

Revert churn. beacon_fcr_revert_to_finalized_total climbs ~117 per lighthouse node over ~54 minutes —
roughly one revert every 4–5 slots. safe recovers immediately each time, so it is not user-visible, but the
rate is worth understanding.

Metric caveat for cross-client work. Nimbus's beacon_safe_slot comes from retrieve_fast_confirmed_bid
and reports the slot of the block whose payload is published as safe — not the confirmed beacon slot. It
is not comparable to lighthouse's beacon_fcr_confirmed_root_slot, which is the confirmed root's slot.

Flag scope. The buildoor inventory groups are children of the CL groups, so the FCR flags also land on
buildoor-lighthouse-geth-1 and buildoor-lodestar-ethrex-1.

8. Recommendations

1. Land consensus-specs #5449. It checks safe_execution_block_hash directly in the FCR vectors and is
   the only way to settle own-vs-parent per client deterministically.
2. Lighthouse: reconcile the code with its own comment — the Gloas path should return
   execution_payload_parent_hash unconditionally.
3. Lodestar: add an explicit Gloas branch to getSafeExecutionBlockHash instead of relying on implicit
   ProtoBlock field semantics, and fix the dead spec link.
4. Grandine: rebase grandinetech/grandine#656 onto glamsterdam-devnet-7 and publish an image so a third
   independent implementation can be exercised.
5. Test cross-client confirmation agreement. Nobody has yet compared which block nimbus, lighthouse and
   lodestar confirm at the same slot. That is where a genuine FCR disagreement would surface, and it is a
   more valuable next test than further safe-hash sampling.
6. Resync lodestar-geth-1 and lodestar-ethrex-1; investigate why FCR cannot confirm on the nimbus-eth1
   pairing.

Appendix — reproducing the evidence

# Spec provenance
cd consensus-specs && git fetch upstream --tags
git diff v1.7.0-alpha.12 upstream/master -- specs/gloas/fast-confirmation.md   # empty
git log --format="%h %ad %s" --date=short upstream/master -- specs/gloas/fast-confirmation.md

# Network health
panda dora overview glamsterdam-devnet-7

# FCR state
panda prometheus query devnets 'beacon_fcr_confirmation_delay_slots{network="glamsterdam-devnet-7"}'
panda prometheus query devnets 'beacon_fcr_confirmed_root_slot{network="glamsterdam-devnet-7"}'
panda prometheus query devnets 'lodestar_fast_confirmation_confirmed_epoch{network="glamsterdam-devnet-7"}'
panda prometheus query devnets 'beacon_fcr_revert_to_finalized_total{network="glamsterdam-devnet-7"}'

# Withheld slots: PTC votes live in the NEXT block's payload_attestations
panda ethnode beacon-get glamsterdam-devnet-7 lighthouse-besu-1 /eth/v2/beacon/blocks/<slot>
#   -> .data.message.body.payload_attestations[].data.payload_present
#   -> .data.message.body.signed_execution_payload_bid.message.{block_hash,parent_block_hash,builder_index}

# EL safe / finalized
panda ethnode exec-rpc glamsterdam-devnet-7 lighthouse-besu-1 eth_getBlockByNumber '["safe", false]'

# The excluded payload, retained as a non-canonical side block
panda ethnode exec-rpc glamsterdam-devnet-7 lighthouse-geth-1 eth_getBlockByHash \
  '["0x518ed79e887fb16c4c8d3e143be57cd3268ad37b58e12dbeb1f0a1f195ca8d3a", false]'
panda ethnode exec-rpc glamsterdam-devnet-7 lighthouse-geth-1 eth_getBlockByNumber '["0xc824", false]'

# Stalled lodestar nodes
panda prometheus query devnets \
  'rate(beacon_head_slot{network="glamsterdam-devnet-7",consensus_client="lodestar",job="consensus_node"}[15m])*60'

Spec: specs/gloas/fast-confirmation.md ·
#5197 ·
#5278 ·
#5449

Client source (all @ glamsterdam-devnet-7):
lighthouse beacon_node/beacon_chain/src/canonical_head.rs ·
lodestar packages/fork-choice/src/forkChoice/safeBlocks.ts ·
nimbus beacon_chain/consensus_object_pools/attestation_pool.nim