glamsterdam-devnet-9 · incident report

Fleet-wide memory exhaustion at a 4M-entry validator registry

P1 — Capacity

Component fleet capacity / all consensus clients Observed 2026-09-03 08:50–11:10 UTC Network glamsterdam-devnet-9 (chain 7013099983) Fork Gloas @ epoch 225

glamsterdam-devnet-9 is the first Glamsterdam devnet at 1,000,000 active validators and a ~4,000,000-entry registry. 275 of 1,007 hosts have kernel OOM kills. The dominant victim is Prysm (255/308 hosts, 320 events, 28–31 GB RSS on 32 GB hosts). Prometheus time series resolve the mechanism precisely: Prysm's resident set steps up +11.2 GiB in the single hour of the Gloas fork transition (14.20 → 25.36 GiB median) and then sits flat at 23–27 GiB for the next 17 hours, leaving ~5 GiB of headroom on a 32 GB host. Lighthouse, on identical hosts, stepped only +2.5 GiB. The memory is not consumed by the stuck-sync retry loop and it is not a progressive leak — Prysm nodes restarted after the fork are equally stuck yet sit flat at 13–15 GiB. A rolling restart reclaims ~10 GiB per host and would stop the OOMs today, independently of the state-root fix. The other two OOM-prone clients fail for different reasons and need different remedies: Grandine dies to a discrete five-second allocation from a 12 GB steady state, and the buildoors are simply on 16 GB hosts.

What we observe

275 of 1,007 hosts have at least one kernel OOM kill recorded in dmesg. The distribution is not uniform — it is bimodal by client:

Consensus client	Hosts OOM-killed	Rate	Total OOM events
prysm	255 / 308	83%	320
grandine	10 / 10	100%	89
buildoor (16 GB hosts)	4 / 4	100%	29
bootnode (runs Prysm)	1 / 4	25%	1
lighthouse	3 / 430	0.7%	6
nimbus	1 / 80	1.3%	1
teku	1 / 140	0.7%	1
lodestar	0 / 30	0%	0

Kernel-reported victims across the fleet: beacon-chain (Prysm) 257, grandine 10, lighthouse 4, java (Teku) 2, nimbus_beacon_n 1, MainThread (Lodestar) 1.

Resident set by client at 1M active / 4M registry

Beacon-container memory across the fleet sweep at 10:48 UTC (docker stats), split by whether the node is at the chain tip:

Client	State	n	median GiB	p90 GiB	max GiB
lighthouse	at tip	423	15.96	17.19	19.98
teku (-Xmx12g)	at tip	106	14.11	15.95	18.07
lodestar (devnet-8 tag †)	at tip	26	11.83	12.34	12.81
nimbus	at tip	80	8.19	9.16	16.74
prysm	stuck @ 7199	306	16.04	—	28.34
bootnode (prysm, no validators)	stuck @ 7199	4	21.23	—	23.02

† Lodestar's hosts run ethpandaops/lodestar:glamsterdam-devnet-8 rather than the :unstable trunk the devnet-9 spec requires, so its figure is a devnet-8-era build, not a trunk memory profile.

Read naively, Prysm's median (16.04 GiB) looks unremarkable next to Lighthouse's (15.96 GiB). That median is misleading: it mixes two populations with very different histories, which the next section separates.

Root cause analysis

Prysm: a one-time +11 GiB step at the fork transition, never released

Prometheus process_resident_memory_bytes for the consensus_node job, 30-hour window, median over 6 Prysm hosts (prysm-geth-100…105) and 6 Lighthouse hosts (lighthouse-geth-100…105), all on identical s-8vcpu-32gb-640gb-intel droplets. All 6 Prysm hosts ran continuously through the fork (RestartCount 0, uptime 27.6 h):

                        prysm (GiB)      lighthouse (GiB)
  09-02 13:00              13.90              12.50
  09-02 14:00              14.72              12.47
  09-02 15:00              14.48              12.46
  09-02 16:00              14.20              12.65     <-- GLOAS FORK (slot 7200)
  09-02 17:00           25.36              15.19
  09-02 18:00              24.10              15.89
  09-02 20:00              24.89              16.24
  09-03 00:00              25.18              16.95
  09-03 04:00              25.32              18.05
  09-03 08:00              23.39              14.98
  09-03 09:00              24.41              15.73

Prysm steps +11.16 GiB in one hour across the fork boundary, then plateaus. It does not keep climbing: over the following 17 hours the median oscillates between 23.4 and 25.7 GiB with no trend. Lighthouse, on the same hosts and the same fork, steps +2.54 GiB. (The Lighthouse drop at 09-03 07:00 is its container being replaced by watchtower at 06:52 UTC, which also explains why Lighthouse's uptimes are ~2 h while Prysm's are 17–28 h: Prysm's image digest was stable for the whole run, so watchtower had nothing to swap.)

Ruling out the stuck-sync retry loop

Prysm nodes are wedged in an initial-sync loop, re-attempting the same failing block at slot 7203 indefinitely (see the companion state-root divergence report). The obvious hypothesis is that this retry loop accumulates states. It does not. The same Prometheus query over 13 Prysm nodes that were restarted after the fork — equally stuck, equally looping, same binary:

  09-02 16:00              13.02     <-- GLOAS FORK
  09-02 17:00              15.20
  09-02 20:00              12.29
  09-03 00:00              14.57
  09-03 04:00              14.22
  09-03 09:00              13.55

Flat at 12–15 GiB for 17 hours. These nodes are in exactly the same failing retry loop and never approach the OOM threshold. The retry loop is therefore not the memory consumer. The +11 GiB is acquired by executing the fork transition itself and is then retained for the lifetime of the process.

The most likely retained object is the pre-fork/post-fork BeaconState pair plus the epoch-transition working set: at ~4M registry entries a single Gloas BeaconState is on the order of several GiB, and holding both sides of the upgrade plus intermediate copies accounts for the observed step. This specific attribution is a hypothesis — confirming it needs a Go heap profile (/debug/pprof/heap) from a node that lived through the fork, which we did not take because the profiling endpoint is not exposed on these nodes.

Why the OOM counts sit on the restarted population

Cross-referencing current resident set against lifetime dmesg OOM counts closes the loop:

Prysm/bootnode cohort	hosts	median OOMs	total OOMs
Beacon ≥22 GiB now (ran through the fork, never restarted)	61	0	22
Beacon <18 GiB now (restarted since)	223	1	278

The low-memory hosts carry the OOM history precisely because they were OOM-killed and restarted, which reset them to the 13–15 GiB baseline. The 61 hosts still sitting at ≥22 GiB have mostly not died yet — they are the next cohort to go. With ~5 GiB of headroom on a 32 GB host, any transient (a peer burst, the EL's page cache, a GC cycle) tips them over.

Note also that OOM kills predate the fork on some hosts, during the initial load of the ~560 MB genesis.ssz: prysm-geth-100 was killed at 30.7 GB on Sep 1 19:06 UTC, three hours after genesis. So Prysm has two distinct high-water regimes at this registry size — genesis load and fork transition — and the 32 GB host clears neither with comfortable margin.

The pattern across clients — three different failure modes, not one

Two groups separate cleanly on OOM rate. Lighthouse (3/430), Nimbus (1/80), Teku (1/140) and Lodestar (0/30) essentially never die: their steady state at 1M active validators is 8–18 GiB and their fork-transition step is small (Lighthouse +2.5 GiB). Prysm (255/308), Grandine (10/10) and the buildoors (4/4) always do. But the three failing cases are mechanically different, and conflating them would produce the wrong remediation:

Client	Mode	Signature	Fixed by more RAM?
prysm	High-water ceiling	+11.2 GiB step at the fork, then a flat 23–27 GiB plateau; ~5 GiB headroom on 32 GB	Yes — 64 GB clears it with margin
grandine	Allocation bug	Flat ~12 GB for ~55 s, then 12 → 25–30 GB inside a single 5-second interval at process age 110–120 s; steady state only 12–13 GB	No — masks it at best
buildoor	Undersized host	The only 16 GB hosts in a fleet of 32 GB hosts; a terraform fall-through	Yes — it is a one-line config fix
teku	Heap-capped	Bounded by -Xmx12g, so it degrades into GC thrash and a wedged event loop instead of being killed	Partly — needs a higher -Xmx, not a bigger host

The Grandine distinction is the important one for planning. Its steady state is 12–13 GB — among the leanest in the fleet — and it is killed by a discrete bulk allocation that roughly doubles its resident set in under five seconds, about 60 s after the slot-7136 anchor-state load, with no panic or error line emitted. That is not "the 4M registry does not fit". Moving Grandine to a 64 GB host would stop the kills without fixing anything: the allocation itself is the defect. See the Grandine report for the full lifetime traces.

Host sizing

Observed: 1,002 hosts at 32 GB, 4 at 16 GB. From terraform/devnet-9/digitalocean.tf:

variable "digitalocean_supernode_size" { default = "s-8vcpu-32gb-640gb-intel" }
variable "digitalocean_fullnode_size"  { default = "s-8vcpu-16gb" }
...
size = vm.supernode ? var.digitalocean_supernode_size : var.digitalocean_fullnode_size

# supernode resolution (digitalocean.tf:218)
supernode = (
  node.supernode != null ? node.supernode :
  can(regex("(bootnode|mev)", node.name)) ? true :
  ... validator_count >= 128 ...
)

Every validator group in nodes.tf carries an explicit supernode = true, and bootnode matches the regex("(bootnode|mev)") fallback, so both land on 32 GB. The four buildoor-* entries match none of the three branches and fall through to the 16 GB fullnode default.

Impact

• Prysm's 307 nodes cannot be kept alive at this registry size on 32 GB hosts without periodic restarts. 320 OOM events and counting; 61 hosts are currently sitting within ~5 GiB of the ceiling.
• Each OOM restart costs a full re-load of the 4M-entry registry, so a node spends minutes unavailable per event — compounding the participation deficit on top of the fork failure itself.
• Because docker inspect reports these as clean exits, the fleet's own health signals under-report the problem. Any dashboard or alert keyed on State.OOMKilled shows zero.
• For future devnets: 32 GB is the wrong sizing for a 4M-entry registry if Prysm or Grandine is in the mix. It is adequate for Lighthouse, Nimbus, Lodestar and (heap-capped) Teku.

How to verify / reproduce

# The kills — note docker will NOT show these
sudo dmesg -T | grep 'Out of memory: Killed process'
docker inspect beacon --format '{{.State.OOMKilled}} {{.State.ExitCode}}'   # prints "false 0"

# The fork step, per client
panda prometheus query-range devnets \
  'process_resident_memory_bytes{job="consensus_node",network="glamsterdam-devnet-9",
    instance=~".*(prysm|lighthouse)-geth-10[0-5]"}' \
  --start now-30h --end now --step 30m

# Live resident set on one host
PID=$(docker inspect beacon --format '{{.State.Pid}}'); grep VmRSS /proc/$PID/status

# Fleet-wide OOM sweep (read-only)
ansible all -m shell -a "sudo dmesg -T | grep -c 'Out of memory: Killed process'"

To confirm or kill the retained-BeaconState hypothesis, take curl localhost:8080/debug/pprof/heap from a Prysm node with ≥22 GiB RSS that ran through the fork, and compare against one restarted afterwards. That single measurement would turn the attribution from hypothesis into fact.

Recommended action

1. Immediate, zero-risk: rolling-restart the 61 Prysm hosts currently above 22 GiB. This reclaims ~10 GiB each and stops the next wave of OOMs. It does not fix the fork divergence — those nodes stay stuck — but it stops the churn and the repeated 4M-registry reloads.
2. Prysm team: the +11.16 GiB step across the fork transition is worth a heap profile regardless of the state-root bug. At mainnet registry scale this is the difference between fitting and not fitting on a standard node.
3. Separate "needs more RAM" from "has an allocation bug" before sizing anything. Only Prysm and the buildoors are capacity problems that hardware fixes. Grandine is not: its steady state is 12–13 GB and it dies to a discrete sub-five-second doubling, so a bigger host would hide the defect rather than resolve it. Teku needs a higher -Xmx, not a bigger host.
4. Sizing for the next devnet at this registry size: 64 GB for the validator fleet, or 32 GB only for clients with a demonstrated <20 GiB fork-transition high-water mark. Measure the fork step, not just the steady state — Lighthouse's and Prysm's steady states were within 0.1 GiB of each other the day before the fork, and diverged by 10 GiB an hour after it.
5. Give the buildoors supernode = true in nodes.tf, or add buildoor to the regex fallback in digitalocean.tf:218.
6. Alerting: key node-death alerts on dmesg / node_vmstat_oom_kill rather than docker inspect's OOMKilled, which is false for every kill in this incident.

Evidence appendix

Verbatim kernel OOM records:

prysm-nethermind-22:
[Wed Sep  2 00:50:03 2026] Out of memory: Killed process 77935 (beacon-chain)
  total-vm:33842460kB, anon-rss:28020500kB, file-rss:568kB, shmem-rss:0kB,
  UID:1006 pgtables:56192kB oom_score_adj:0
[Wed Sep  2 15:16:22 2026] Out of memory: Killed process 106198 (beacon-chain)
  total-vm:42520744kB, anon-rss:30464308kB, file-rss:360kB, shmem-rss:0kB,
  UID:1006 pgtables:62372kB oom_score_adj:0

prysm-geth-100:
[Tue Sep  1 19:06:19 2026] Out of memory: Killed process 34218 (beacon-chain)
  total-vm:37655344kB, anon-rss:30737916kB, file-rss:0kB, shmem-rss:0kB,
  UID:1006 pgtables:66888kB oom_score_adj:0

grandine-geth-1:
[Thu Sep  3 08:45:53 2026] Out of memory: Killed process 273622 (grandine)
  total-vm:3474850168kB, anon-rss:31194000kB, file-rss:128kB, shmem-rss:0kB,
  UID:1006 pgtables:64924kB oom_score_adj:0

Live VmRSS from /proc/<pid>/status, 09-03 11:04 UTC, by uptime cohort (all Prysm, all 32 GB hosts, single stable image digest):

  ~17.0 h uptime (restarted after the fork)   n=17  median 13.96 GiB  range  8.29 - 16.42
  ~27.6 h uptime (ran through the fork)       n= 8  median 24.04 GiB  range 15.58 - 27.60

  All 5 hosts at >=24 GiB are in the 27.6 h cohort.

A measurement that did not work, recorded so it is not repeated: sampling VmRSS twice over a 180-second window to estimate a growth rate produced values from −34 to +116 GiB/h across hosts, including +20 to +29 GiB/h for healthy Lighthouse nodes that are demonstrably stable. A three-minute window captures allocator and GC jitter, not trend, and at these magnitudes the noise swamps the signal entirely. The uptime-cohort comparison above, and then the Prometheus range query, are what actually resolved the question — and the Prometheus series overturned the cohort reading, which on its own looked like a steady ~0.95 GiB/h leak (Pearson r = 0.79 against uptime) rather than the step function it really is.

Sampling rate matters in the other direction too. Prysm's failure is a one-hour step, so a 30-minute Prometheus step resolves it fine. Grandine's is a bulk allocation that completes inside five seconds, and any sampling coarser than ~10 s misses it entirely — the node simply appears to vanish, which combined with docker inspect's false OOMKilled: false reads as a clean exit. Match the sampling interval to the failure you are hunting: 30 min for plateaus and steps, ≤5 s for spikes.