All four builder nodes are dead: the ePBS builder path has never run

All four buildoor-* hosts on glamsterdam-devnet-9 are the only 16 GB machines in a fleet of 1,002 × 32 GB machines. They cannot hold the network's ~4M-entry validator registry, so their beacon nodes OOM continuously and the builder service has never been able to reach one. Zero bids have been submitted at any point in the devnet's life, and zero blocks on this chain have been builder-built. devnet-9's headline feature is EIP-7732 enshrined PBS; the external builder path it exists to exercise has produced nothing. This is an infrastructure sizing defect, not a client bug.

What we observe

Every buildoor host is 8 vCPU / 15,998 MB / 315 GB. Every other node in the fleet is 8 vCPU / 32,102 MB / 640 GB.

Resident set size at the moment of the kill is 12.8–14.6 GB on a 15,998 MB box — 80–91% of all RAM on the machine, before the OS, the EL and the sidecars get any. Sibling 32 GB hosts run the identical images stably at 12–14 GB resident.

Why the restart counters understate this

Two independent measures, taken over different windows, corroborate each other closely:

Extrapolating the observed rate over the 41 hours since genesis gives roughly 1,100 / 690 / 290 beacon-node deaths on the three crash-looping hosts respectively — order-of-magnitude figures, since the rate need not have been constant. The one directly-measured near-lifetime number is buildoor-teku-nethermind-1: its container has survived every watchtower wave since 2026-09-01 20:56 and has accumulated 563 restarts in 36 hours.

The dmesg retention windows are themselves evidence of the churn. The three crash-looping hosts have flooded their ring buffers down to the last 18–74 minutes, while buildoor-lodestar-ethrex-1 — the one that is not crash-looping — still holds its complete history back to boot on 2026-08-31 21:58:

host                          uptime since        oldest dmesg line       retained
buildoor-lighthouse-geth-1    2026-08-31 21:56    2026-09-03 08:44:08     18 min of 59 h
buildoor-teku-nethermind-1    2026-09-01 00:12    2026-09-03 08:33:57     28 min of 56 h
buildoor-prysm-ethrex-1       2026-08-31 22:16    2026-09-03 07:48:14     74 min of 58 h
buildoor-lodestar-ethrex-1    2026-08-31 21:58    2026-08-31 21:58:02     complete

buildoor-lodestar-ethrex-1 is a distinct failure mode rather than a healthy host: it OOM-killed twice at 2026-09-01 16:30:57 and 16:43:35 — within 45 minutes of genesis — and has not run far enough since to allocate a large state. Its beacon head is stuck at slot 31.

The builder service itself is healthy but blind. Its log is dominated by failure to reach its own local beacon node:

level=warning msg="Event stream connection error, reconnecting..." component=cl-client
  error="failed to connect to event stream: Get \"http://beacon:5052/eth/v1/events?topics=payload_attributes\":
  dial tcp [2a03:b0c0:1:e0:0:1:b298:e00b]:5052: connect: connection refused" topic=payload_attributes

Message shapes for the buildoor container since genesis, across all four hosts:

Root cause analysis

1. The sizing defect — a missing entry in a two-word allow-list

Droplet size is chosen from a single boolean in terraform/devnet-9/digitalocean.tf:

variable "digitalocean_supernode_size" { default = "s-8vcpu-32gb-640gb-intel" }
variable "digitalocean_fullnode_size"  { default = "s-8vcpu-16gb" }

size = vm.supernode ? var.digitalocean_supernode_size : var.digitalocean_fullnode_size

And supernode is resolved by this three-tier heuristic (digitalocean.tf:218–223):

# Supernode: explicit > bootnode/mev > validator_count >= 128
supernode = (
  node.supernode != null ? node.supernode :
  can(regex("(bootnode|mev)", node.name)) ? true :
  (node.count > 0 ? (node.validator_end - node.validator_start) / node.count >= 128 : false)
)

Now trace a buildoor entry from terraform/devnet-9/nodes.tf:

{ name = "buildoor-prysm-ethrex",    count = 1, cloud = "digitalocean", builder_start = 0 },
{ name = "buildoor-lighthouse-geth", count = 1, cloud = "digitalocean", builder_start = 1 },
{ name = "buildoor-lodestar-ethrex", count = 1, cloud = "digitalocean", builder_start = 2 },
{ name = "buildoor-teku-nethermind", count = 1, cloud = "digitalocean", builder_start = 3 },

1. node.supernode is not set → null → fall through.
2. regex("(bootnode|mev)", "buildoor-prysm-ethrex") → no match → fall through.
3. validator_end - validator_start = 0, so 0 / 1 = 0 >= 128 → false.

→ supernode = false → s-8vcpu-16gb. Every validator-bearing entry in nodes.tf carries an explicit supernode = true, so all 1,000 of them get the 32 GB box. The bootnodes carry no supernode key either, but the name matches the (bootnode|mev) allow-list and they get 32 GB.

The defect is that buildoor was never added to that allow-list. It is a newer node class introduced for ePBS; bootnode and mev are the two older zero-validator classes that were.

2. Why the heuristic is wrong in principle, not just incomplete

The fallback tier keys off validators per node as a proxy for "does this box need to be big". That proxy was defensible on an 84,000-validator devnet. It is not defensible here.

A buildoor signs for zero validators, but it still runs a full CL and a full EL, and it must hold the same ~4M-entry validator registry as every other node on the network — devnet-9 ships 1,000,000 active validators plus 3,000,000 additional exited validators, a ~560 MB genesis.ssz. The heuristic conflates how many keys a node signs with and how large a state it must hold. On a 4M-validator network those two quantities are unrelated, and the first is a useless predictor of the second.

This is why the failure is total rather than marginal: sibling 32 GB hosts running the identical client images are stable at 12–14 GB resident, which is comfortably over the 16 GB box's usable ceiling once the OS and the EL are accounted for.

3. The consequence chain

The failure is strictly downstream of the sizing, and the timeline shows it:

Before genesis there was no large state to hold and the hosts were fine. connection refused appears at 271 in the window before genesis, then jumps 18× to 25,709 in the first window after genesis — the moment the ~4M-validator registry had to be resident. It has climbed steadily since and has never recovered.

The chain of causation is therefore:

16 GB droplet
  -> beacon node cannot hold the ~4M-validator registry
     -> kernel OOM-kills the beacon (up to 27 times per hour, ~1,100 deaths since genesis)
        -> buildoor's SSE event stream to http://beacon:5052 is refused (228,263 times)
           -> no payload_attributes, no head, no proposer_preferences
              -> buildoor synthesizes attributes (61,105 times) instead of receiving them
                 -> the p2p bidder service cannot run
                    -> zero bids, ever

The bidder service's own lifecycle messages corroborate this. Initializing p2p bidder service… / p2p bidder service started appear 6 times each across the four hosts, and the last occurrence is 2026-09-01 10:39 UTC — five hours before genesis and twenty-nine hours before the Gloas fork. The last thing it said was No builders in beacon state (pre-Gloas), waiting for registration. It has not successfully initialised since.

Impact

Confirmed from the chain side. Sampling 20 slots spread across the whole post-fork range (7203 → 12299) on a healthy Lighthouse node:

sampled=20  self_built=13  builder_built=0  missing=7

Every present block carries the "no builder" sentinel in its Gloas bid:

slot 12292: builder_index=18446744073709551615  value=0
slot 12295: builder_index=18446744073709551615  value=0
slot 12297: builder_index=18446744073709551615  value=0
slot 12299: builder_index=18446744073709551615  value=0

18446744073709551615 is 2^64 - 1 (UINT64_MAX), the sentinel for a self-built payload, and value = 0 confirms no bid was paid. The 7 missing slots are consistent with the network-wide ~40% miss rate caused by the separate Prysm and Grandine fork-transition failures.

Concretely, this means the following went untested on the largest ePBS devnet run to date:

• The builder bid gossip and selection path (signed_execution_payload_bid with a real builder_index and non-zero value).
• Builder payment and the PTC (payload-timeliness committee) attestation flow against a genuinely external builder.
• Builder registration, builder_deposits and builder_exits — the buildoors never registered, and the corresponding Dora contract indexers have no data to show.
• EIP-8282 builder execution requests, which have no builder to originate them.
• Cross-client builder/proposer interop — the four buildoors deliberately pair four different CL/EL combinations (prysm-ethrex, lighthouse-geth, lodestar-ethrex, teku-nethermind) precisely to cover this, and none of them ran.

Client teams who assumed their builder path was being exercised on devnet-9 should assume it was not.

Open question: the buildoor slot counter is 4,800 slots ahead

Reported separately because it is not explained by the sizing defect and may be an independent bug.

All four buildoors report a proposal_slot roughly 4,800 slots higher than the real chain slot. At wall clock 1788426019 the true chain slot is 12301; all four buildoors report 17099–17102. The offset is 17100 - 12301 = 4799 slots = 16 hours.

The counter is not drifting. Over 13 consecutive hours all four advance at exactly 300 slots/hour — one slot per 12 s, the correct chain rate — and stay bit-identical to each other:

hour (UTC)   buildoor-lighthouse-geth-1   buildoor-teku-nethermind-1   delta/h
2026-09-02 21:00        13800                      13800                 300
2026-09-02 22:00        14100                      14100                 300
...
2026-09-03 07:00        16800                      16797                 300
2026-09-03 08:00        17098                      17099                 298

Four independent hosts advancing in lockstep at the exact chain rate indicates a clock-driven counter with a wrong origin, not a free-running synthesis loop that drifted — a loop would desynchronise. The arithmetic points at the origin:

real genesis      1788278400 = 2026-09-01 16:00:00 UTC
implied origin    1788278400 - (4800 x 12) = 1788220800 = 2026-09-01 00:00:00 UTC

Sixteen hours before genesis is exactly midnight of the genesis day, which is what you would get from a genesis value truncated from a datetime to a date. Consistent with this, buildoor is passed no genesis argument at all — it must read genesis from the CL, and the CL has been unreachable since genesis:

run --state-db=/data/buildoor.sqlite --builder-key-index=1
    --cl-client=http://beacon:5052 --el-engine-api=http://execution:8551
    --el-rpc=http://execution:8545 --api-port=8085 ...

The synthesized timestamp field differs between hosts for the same slot number — buildoor-teku-nethermind-1 emits wall-clock-now while the other three emit correct_genesis + slot × 12 (hence 16 h in the future) — so the two values are not derived identically across hosts.

How to verify / reproduce

Host sizing, from the repo — no cluster access needed:

grep -n "digitalocean_supernode_size\|digitalocean_fullnode_size" terraform/devnet-9/digitalocean.tf
sed -n '218,223p' terraform/devnet-9/digitalocean.tf     # the supernode heuristic
grep -n "buildoor" terraform/devnet-9/nodes.tf           # no supernode key, no size override

Confirm on the hosts (read-only):

ansible buildoor -m shell -a 'nproc; free -m; df -BG /'
# -> 8 vCPU / 15998 MB / 315 GB on all four

ansible buildoor -m shell -a "docker inspect beacon --format '{{.RestartCount}}'"
sudo dmesg -T | grep 'Out of memory: Killed process'

docker inspect beacon --format '{{.Created}} {{.RestartCount}}'
sudo dmesg -T | grep 'Out of memory: Killed process'   # intervals give the true rate

Confirm zero bids over the devnet's whole life:

SELECT toStartOfInterval(Timestamp, INTERVAL 6 HOUR) AS t, count() AS total,
       countIf(match(Body,'(?i)(bid submitted|submitted bid|bid accepted|bid published)')) AS bids,
       countIf(match(Body,'(?i)connection refused')) AS conn_refused
FROM external.otel_logs
WHERE Timestamp >= toDateTime('2026-08-30 00:00:00')
  AND ResourceAttributes['network'] = 'glamsterdam-devnet-9'
  AND ResourceAttributes['container.name'] = 'buildoor'
GROUP BY t ORDER BY t

Confirm zero builder-built blocks:

curl -s http://localhost:5052/eth/v2/beacon/blocks/<slot> \
  | jq '.data.message.body.signed_execution_payload_bid.message
        | {builder_index, value}'
# -> {"builder_index":"18446744073709551615","value":"0"}  for every post-fork slot

Recommended action

1. Resize the four buildoor hosts to the supernode size. The one-line fix is to add supernode = true to the four buildoor-* entries in terraform/devnet-9/nodes.tf; that is explicit and survives any later change to the heuristic.
2. Fix the heuristic for future devnets. Add buildoor to the (bootnode|mev) allow-list in digitalocean.tf and in the matching block in hetzner.tf. Better still, stop using validator count as the proxy — on a network with a multi-million-entry registry, every node that runs a CL needs the large box regardless of how many keys it signs with.
3. Add a launch-time assertion. A check that every host running a beacon container has at least the supernode memory size would have caught this before genesis. Related: the disk is also halved (315 GB vs 640 GB), which has not bitten yet but will on a longer-running devnet.
4. Re-run the ePBS builder scenarios once resized. Nothing about EIP-7732's external builder path has been exercised, so devnet-9's primary objective is currently unmet independently of the Prysm and Grandine fork failures.
5. Refer the 16-hour slot offset to the buildoor authors as an open question, to be reproduced against a healthy beacon node.

Evidence appendix

Host specification, all four buildoors

buildoor-lighthouse-geth-1 | cpus=8 mem=15998MB disk=315G
buildoor-teku-nethermind-1 | cpus=8 mem=15998MB disk=315G
buildoor-prysm-ethrex-1    | cpus=8 mem=15998MB disk=315G
buildoor-lodestar-ethrex-1 | cpus=8 mem=15998MB disk=315G

fleet RAM distribution (1007 hosts probed): {32 GB: 1002, 31 GB: 1, 16 GB: 4}
                                             the four 16 GB hosts are exactly the buildoors

Kernel OOM, buildoor-lighthouse-geth-1 — every kill retained in dmesg

[Thu Sep  3 08:44:08 2026] Out of memory: Killed process 663293 (lighthouse) rss=14569044kB
[Thu Sep  3 08:46:24 2026] Out of memory: Killed process 663845 (lighthouse) rss=14566156kB
[Thu Sep  3 08:48:33 2026] Out of memory: Killed process 664494 (lighthouse) rss=14555076kB
[Thu Sep  3 08:50:56 2026] Out of memory: Killed process 665212 (lighthouse) rss=14564456kB
[Thu Sep  3 08:53:00 2026] Out of memory: Killed process 665833 (lighthouse) rss=14559872kB
[Thu Sep  3 08:55:04 2026] Out of memory: Killed process 666398 (lighthouse) rss=14531944kB
[Thu Sep  3 08:57:23 2026] Out of memory: Killed process 666983 (lighthouse) rss=14546176kB
[Thu Sep  3 08:59:31 2026] Out of memory: Killed process 667551 (lighthouse) rss=14540720kB
[Thu Sep  3 09:01:55 2026] Out of memory: Killed process 668207 (lighthouse) rss=14509260kB

nine kills in 17m47s -- and that is the entire dmesg buffer; older records already wrapped
docker inspect beacon -> restarts=56  exit=0  oom=false     # docker's view is wrong twice over:
                                                            # OOMKilled is false, and the count
                                                            # reset at the 07:00 watchtower swap

Builder service lifecycle — last successful start was before genesis

count  hosts  first seen           last seen            message
    6      4  2026-08-31 14:43     2026-09-01 10:39     Initializing p2p bidder service...
    6      4  2026-08-31 14:43     2026-09-01 10:39     p2p bidder service started
    6      4  2026-08-31 14:43     2026-09-01 10:39     No builders in beacon state (pre-Gloas), waiting for registration
    7      4  2026-08-31 14:43     2026-09-01 10:39     Timing defaults applied

genesis was 2026-09-01 16:00:00 UTC — every one of these predates it

Builder service, current state (buildoor-teku-nethermind-1)

level=warning msg="Event stream connection error, reconnecting..." component=cl-client
  error="... dial tcp [2a03:b0c0:2:f0:0:1:ea8f:a00b]:5052: connect: connection refused" topic=proposer_preferences
level=warning msg="Cannot resolve proposer for synthesized attributes, keeping the source slot's"
  component=builder-service slot=17031
level=info msg="No payload attributes received for slot (block missing?), re-using the last available attributes"
  attrs_from=17030 component=builder-service missing_slots=1 parent_hash=edcacc26da776f03 slot=17031

Scope note

This report covers the buildoor hosts and the ePBS builder path only. The network-wide loss of finality on devnet-9 is caused by separate, unrelated failures in the Prysm and Grandine Gloas fork transitions, reported elsewhere. The buildoor failure is independent of both: it began at genesis, twenty-four hours before the fork.