devnet-9's stated purpose is to run client trunk branches through a Gloas fork. For two consensus clients that did not happen. Grandine was silently left behind by the image bump that moved every other client to trunk, and all 31 Lodestar hosts were deliberately rolled back to a devnet-8 tag and redeployed in one coordinated action finishing 27 minutes before the fork — a change pushed to the fleet half an hour before it was committed to git, with a three-word commit message and no stated reason. Both clients' devnet-9 results are therefore not results about trunk. Two further deviations, Reth and Nimbus-EL, are documented and intentional and get a clean bill of health.
Audited from external.otel_logs
(ResourceAttributes['container.image.name'], 90-minute window, grouped by
container.name), then confirmed host-side with docker inspect.
| Client | Hosts | Image actually running | Spec sheet claims | Verdict |
|---|---|---|---|---|
| lighthouse | 429 | lighthouse:unstable | ✅ :unstable | match |
| nimbus | 80 | nimbus-eth2:unstable | ✅ :unstable | match |
| prysm | 313 | prysm-beacon-chain:develop | ❓ :develop | match |
| teku | 46 | teku:master | ✅ :master | match |
| lodestar | 31 | lodestar:glamsterdam-devnet-8 | ✅ :unstable | silent deviation |
| grandine | 10 | grandine:glamsterdam-devnet-8 | ❓ :develop | silent deviation |
| Client | Hosts | Image actually running | Spec sheet claims | Verdict |
|---|---|---|---|---|
| geth | 413 | geth:master | ✅ :master | match |
| nethermind | 300 | nethermind:master | ✅ :master | match |
| besu | 79 | besu:main | ✅ :main | match |
| erigon | 40 | erigon:main | ✅ :main | match |
| ethrex | 22 | ethrex:main | 🔨 :main | match |
| reth | 140 | reth:glamsterdam-devnet-8 | ❌ pinned | documented |
| nimbusel | 10 | nimbus-eth1:glamsterdam-devnet-8 | ❌ pinned | documented |
The policy statement is unambiguous (md/glamsterdam-devnet-9.md:8):
🚨 Client teams - please get your trunk branches ready. devnet-9 runs trunk images (`:master` / `:main` / `:unstable` / `:develop`)
Reth and Nimbus-EL are recorded as ❌ pinned in the EL tracker
(md/glamsterdam-devnet-9.md:71), so their devnet-8 tags are expected and correctly
disclosed. No action needed for those two. Lodestar and Grandine are not: the CL tracker
(line 81) claims ✅ :unstable and :develop respectively.
Every client image on this devnet resolves through a single file,
ansible/inventories/devnet-9/group_vars/all/images.yaml, via
default_ethereum_client_images.<client>. Its history explains both deviations,
and they have different causes.
devnet-9's inventory was created wholesale from devnet-8. The creating commit says so in its own message:
commit 5644dce 2026-08-19 13:34:59 +0000 Barnabas Busa
devnet-9: 1000-node mainnet-distribution stress test
...
Ansible: inventory copied from devnet-8 (devnet-8 client images kept), 1M active
+ 3M exited-spray validators shuffled into a 4M-entry genesis registry, ...
At that commit every client except Nimbus and Teku was on :glamsterdam-devnet-8.
That is a legitimate, disclosed starting state — the defect is in what happened next.
Twelve days later a single commit moved the fleet to trunk:
commit 1324aa2 2026-08-31 10:05:43 +0000 parithosh
bump images
- lighthouse: .../lighthouse:glamsterdam-devnet-8 + lighthouse: .../lighthouse:unstable
- lodestar: .../lodestar:glamsterdam-devnet-8 + lodestar: .../lodestar:unstable
- prysm: .../prysm-beacon-chain:glamsterdam-... + prysm: .../prysm-beacon-chain:develop
- prysm_validator: .../prysm-validator:glamsterdam-... + prysm_validator: .../prysm-validator:develop
teku: .../teku:master (unchanged, already trunk)
grandine: .../grandine:glamsterdam-devnet-8 (UNCHANGED — left behind)
- besu: .../besu:glamsterdam-devnet-8 + besu: .../besu:main
- geth: .../geth:glamsterdam-devnet-8 + geth: .../geth:master
- erigon: .../erigon:glamsterdam-devnet-8 + erigon: .../erigon:main
- nethermind: .../nethermind:glamsterdam-devnet-8 + nethermind: .../nethermind:master
reth: .../reth:glamsterdam-devnet-8 (unchanged — intentionally pinned)
nimbusel: .../nimbus-eth1:glamsterdam-devnet-8 (unchanged — intentionally pinned)
- ethrex: .../ethrex:glamsterdam-devnet-8 + ethrex: .../ethrex:main
Nine entries were bumped; three were left. Two of the three (Reth, Nimbus-EL) are the documented
pins. The third, Grandine, is the last line of the consensus-layer block and is the only CL not
touched. Nothing in the commit, the file, or the spec sheet records a decision to hold Grandine
back — and the spec sheet's own pre-launch checklist still carries the item unticked
(md/glamsterdam-devnet-9.md:157):
## Open items before launch - [ ] prysm + grandine trunk confirmation.
So Grandine's trunk status was explicitly flagged as unconfirmed before launch and was never resolved. The devnet launched, and forked, with that question still open. This is an omission that the process caught and then dropped, not a deliberate pin.
The most recent commit on the repository reverts Lodestar to the devnet-8 tag:
commit 0e07117 2026-09-02 15:56:20 +0000 parithosh
swap lodestar image
- lodestar: registry-1.docker.io/ethpandaops/lodestar:unstable
+ lodestar: registry-1.docker.io/ethpandaops/lodestar:glamsterdam-devnet-8
The Gloas fork boundary is slot 7200 = 2026-09-02 16:00:00 UTC. The commit lands
3 minutes 40 seconds before it. The commit message is three words; there is no body, no
linked issue, and no PR — git log --all --grep=lodestar returns no other
devnet-9 commit that would explain it.
The deployment is earlier still, and it was the entire cohort. A
docker inspect sweep of all 31 Lodestar hosts shows every one recreated inside a single
10 minute 21 second window, all on the devnet-8 image:
beacon container Created, all 31 lodestar hosts, sorted:
15:22:19 lodestar-nethermind-7 (1 host)
15:25:49 – 15:26:24 lodestar-{geth-1,3,4,6..12}, {nethermind-1,2,3,5,6,9},
{reth-1..4}, besu-1, ethrex-1,
buildoor-lodestar-ethrex-1 (24 hosts)
15:30:31 – 15:30:38 lodestar-{geth-2,5,13}, besu-2 (4 hosts)
15:32:40 lodestar-nethermind-4, lodestar-nethermind-8 (2 hosts)
window 2026-09-02 15:22:19 → 15:32:40 UTC — 31/31 hosts, 0 exceptions
image on every host: ethpandaops/lodestar:glamsterdam-devnet-8
The wave-shaped distribution is the signature of an ansible run with parallel forks,
not of independent restarts. The whole Lodestar cohort was swapped and redeployed in one
coordinated action, finishing 27 minutes 20 seconds before the fork boundary, and starting
34 minutes before the change was committed to git. The running state led the repository, so
for that half hour images.yaml did not describe the fleet: anyone reading git during
the fork window would have concluded Lodestar was on :unstable.
This redeploy is invisible in watchtower's logs — a sibling
investigation confirmed watchtower logged no event for it, because an ansible redeploy recreates
containers directly. Reconstructing "what was deployed when" on this fleet therefore requires a
docker inspect sweep of container Created timestamps joined against
watchtower logs; watchtower logs alone show only the subset of changes watchtower itself made.
Two hosts restarted close to the boundary — lodestar-geth-2 at
16:00:58, 58 seconds after the fork, and lodestar-geth-9 at 16:09:07. The other
29 either never restarted or restarted many hours later, so this was not a cohort-wide crash at the
fork; Lodestar crossed on the freshly deployed image.
:glamsterdam-devnet-8 tags are not pins — they are moving targetsThe name suggests a frozen devnet-8 artifact. It is not. Both mismatched images were rebuilt during this devnet's run:
lodestar:glamsterdam-devnet-8 digest sha256:8bf3694f59b41b91f769e1fd8ee8a3cc1c45e3f70690458c2a49689fdc322035 built 2026-09-02 14:49:38 UTC — 71 minutes before the fork grandine:glamsterdam-devnet-8 digest sha256:4a9f26f09401a19c6de2156599b547169753ec305bac221471d8fa55860a9e14 built 2026-09-03 06:36:33 UTC — ~14.5 hours AFTER the fork
A sibling investigation independently found that watchtower swapped all ten Grandine hosts twice — at T−65 and T−55 minutes before the fork, and again at 09-03 06:52 — because watchtower re-pulls mutable tags on a 15-minute poll. So these clients ran neither trunk nor a stable pin: they ran whatever was last pushed to a devnet-8-named tag, and that content changed underneath them mid-run. The Grandine binary that failed the fork and the one crash-looping now are different builds.
A sibling agent noted Grandine reporting unknown configuration variables. The full list confirms the binary predates the config it was given:
WARN runtime::grandine_args:1247: unknown configuration variables:
["ATTESTATION_PROPAGATION_SLOT_RANGE, ATTESTATION_SUBNET_COUNT, GAS_LIMIT_SCHEDULE,
HEZE_FORK_EPOCH, HEZE_FORK_VERSION, INCLUSION_LIST_DUE_BPS,
MAX_REQUEST_INCLUSION_LIST, MAX_TRANSACTIONS_BYTES_PER_INCLUSION_LIST,
MIN_SLOTS_FOR_INCLUSION_LISTS_REQUESTS"]
Five of the nine are post-Gloas keys — the Heze fork and EIP-7805 inclusion lists.
A build that does not recognise HEZE_FORK_EPOCH is older than the network config it is
being handed. This ties the image finding to the client finding: it is direct evidence that the
Grandine binary on this fleet is behind the devnet-9 configuration, independent of what the tag is
called.
Stated precisely, and no further:
The two defects the Grandine report documents — the zero-count
ExecutionPayloadEnvelopesByRange livelock at the fork boundary, and the discrete
12 GB → 30 GB bulk allocation that OOM-kills every host — were observed on
grandine:glamsterdam-devnet-8, not on :develop. Neither should be filed
against Grandine trunk without a re-test. They may well reproduce there; that is unknown. What
is known is that devnet-9 has produced no evidence about Grandine trunk crossing Gloas,
which was the pre-launch open item and remains open.
Lodestar crossed the fork successfully: 27 of 31 nodes at the chain tip, 3 down, 0 stuck at slot
7199, and 0 OOM kills across all 30 hosts — the best OOM record of any client on the fleet.
That is a real result for lodestar:glamsterdam-devnet-8 at commit
34b3f47a0189739630d95fb883c1c26239359527 (v1.46.0). It is
not evidence that Lodestar :unstable crosses Gloas cleanly. The rollback happened
before the fork, so :unstable was never exercised across the boundary at all.
The reverse inference also deserves care: because the rollback is undocumented, we cannot tell
whether lodestar:unstable was rolled back because it was misbehaving pre-fork.
If it was, that is a Lodestar trunk defect that went unrecorded and is now invisible. This is the
single most important thing for the team to answer, and only the person who made the change
can.
Both are documented as ❌ pinned in the spec sheet's EL tracker. Their devnet-8
tags are expected. The only caveat that applies is the general one above: a
:glamsterdam-devnet-8 tag is mutable, so "pinned" overstates the guarantee even where
the pin is intentional. Worth switching to digests, but there is no discrepancy between intent and
reality here.
Lighthouse, Nimbus, Prysm, Teku, Geth, Nethermind, Besu, Erigon and Ethrex all run the tags the
spec sheet claims. In particular, the Prysm state-transition divergence at slot 7203 is a genuine
trunk finding — prysm-beacon-chain:develop is what the sheet specifies, and a
sibling investigation confirmed the digest did not change across the fork. That report stands
unqualified.
Fleet-wide audit of what is actually running, from the log pipeline:
panda clickhouse query-raw clickhouse-raw "
SELECT ResourceAttributes['ethereum_cl'] AS cl,
ResourceAttributes['container.image.name'] AS img,
uniq(ResourceAttributes['host.name']) AS hosts
FROM external.otel_logs
WHERE Timestamp >= now() - INTERVAL 90 MINUTE
AND ResourceAttributes['network']='glamsterdam-devnet-9'
AND ResourceAttributes['container.name']='beacon'
GROUP BY cl,img ORDER BY cl,hosts DESC"
Host-side confirmation, including the build date that exposes a mutable tag:
ansible -i inventories/devnet-9/inventory.ini lodestar-besu-1 -m shell -a \
"docker inspect beacon --format '{{.Config.Image}} {{.Created}}';
docker image inspect \$(docker inspect beacon --format '{{.Config.Image}}') \
--format '{{index .RepoDigests 0}} {{.Created}}'"
Cohort-wide deployment window — the measurement that distinguishes a one-host redeploy from a fleet-wide one:
ansible -i inventories/devnet-9/inventory.ini lodestar -f 40 -m shell -a \
"docker inspect beacon --format '{{.Created}} {{.State.StartedAt}} {{.Config.Image}}'"
Config-drift history:
git log --follow -p -- ansible/inventories/devnet-9/group_vars/all/images.yaml
lodestar:unstable swapped out at
15:26 UTC on 2026-09-02? If it was failing, that is an unrecorded trunk defect and the most valuable
unlogged fact in this incident. If it was precautionary, say so in the sheet.:develop before filing either fork-transition bug. The
livelock and the OOM must be reproduced on trunk, or the reports must be explicitly scoped to the
devnet-8 build and its digest sha256:4a9f26f0….:glamsterdam-devnet-N tags are mutable and
demonstrably moved twice mid-run. Anywhere the intent is "pinned" — Reth, Nimbus-EL, and the
already-digest-conscious tracoor entry, which carries the comment
# pinned: floating tag + IfNotPresent won't re-pull — use
image@sha256:…. That comment shows the failure mode was already understood in one place
and not generalised.:unstable and Grandine :develop while the fleet ran
neither. A five-line query against images.yaml, or against the live log pipeline as
above, would have caught both before genesis.Created timestamps.| When | What |
|---|---|
| 08-19 13:34:59 | 5644dce devnet-9 inventory created from devnet-8; all client images kept at :glamsterdam-devnet-8 (disclosed in commit message) |
| 08-31 10:05:43 | 1324aa2 "bump images" — 9 entries moved to trunk; grandine, reth, nimbusel left |
| 09-01 16:00:00 | Genesis |
| 09-02 14:49:38 | lodestar:glamsterdam-devnet-8 rebuilt (digest 8bf3694f…) |
| 09-02 14:55–15:05 | watchtower swaps all 10 Grandine hosts (sibling finding, T−65/T−55 min) |
| 09-02 15:22:19 → 15:32:40 | All 31 Lodestar hosts recreated on the devnet-8 image — whole cohort switched, ansible redeploy, invisible to watchtower |
| 09-02 15:56:20 | 0e07117 "swap lodestar image" committed — 34 min after the redeploy began, 24 min after it finished |
| 09-02 16:00:00 | Gloas fork, slot 7200 |
| 09-03 06:36:33 | grandine:glamsterdam-devnet-8 rebuilt again (digest 4a9f26f0…) |
| 09-03 06:52 | watchtower rolls the new Grandine image onto all 10 hosts |
lodestar-besu-1
Sep-02 15:26:21.860 info: Lodestar network=testnet, version=v1.46.0/34b3f47,
commit=34b3f47a0189739630d95fb883c1c26239359527
grandine-reth-1
client version: Grandine/2.0.5-1a910389/x86_64-linux
INFO runtime::grandine_config:125: network: mainnet
ansible/inventories/devnet-9/group_vars/lodestar.yaml:28
lodestar_container_image: "{{ default_ethereum_client_images.lodestar }}"
ansible/inventories/devnet-9/group_vars/grandine.yaml:34
grandine_container_image: "{{ default_ethereum_client_images.grandine }}"
ansible/inventories/devnet-9/group_vars/all/images.yaml:5,10 (current HEAD)
lodestar: registry-1.docker.io/ethpandaops/lodestar:glamsterdam-devnet-8
grandine: registry-1.docker.io/ethpandaops/grandine:glamsterdam-devnet-8
This report covers image-vs-spec agreement only. The Grandine, Prysm, Teku, buildoor, watchtower, memory, gossip and service-tier findings are covered in their own reports; the only claim made here about them is the scoping consequence stated under What this invalidates.