glamsterdam-devnet-9 · incident report

Watchtower replaced client binaries on 76% of the fleet inside the hour before the Gloas fork

P1 — Operational / measurement integrity

Component fleet operations · ethereum-node-docker-watchtower Observed 2026-09-03 08:50 UTC · rev 2 Network glamsterdam-devnet-9 (chain 7013099983) Fork Gloas @ epoch 225

Every glamsterdam-devnet-9 host runs ethereum-node-docker-watchtower with --interval=900, polling Docker Hub every 15 minutes and replacing the execution, beacon, validator and xatu-sentry containers whenever a floating tag moves. Over the ~34 h of the run it replaced client containers 4,694 times across 972 of 1,008 hosts. 763 hosts — 76 % of the fleet — had a client binary swapped in the 75 minutes immediately before the Gloas fork. Prysm's binary was not among them: watchtower performed zero Prysm replacements after 09-02 01:00 UTC, and the binary that met the fork was in place 10.5 h beforehand, so the Prysm state-root divergence is not a watchtower artifact. Grandine's was: all 10 Grandine hosts were rebuilt at T−65 and T−55 min, and again 15 h later, meaning the binary that failed the fork and the binary now crash-looping are two different images.

Correction — this revision supersedes the first publication

The first version of this report stated that the Prysm image was "deployed 09-02 00:42–00:57 UTC, 15 hours before the fork". That was wrong in what it measured. 00:42 was the first Found new image detection event inside a 34-hour query window, for digest f7324c1cf6c8 — not the digest that ran the fork, and not a container creation time. Direct docker inspect on the hosts shows the Prysm containers that met the fork were created 05:25–05:28 UTC (T−10.5 h) from image sha256:274dc1e8…, built 05:10:50 UTC, and were deployed by an ansible playbook run, not watchtower.

The substantive conclusion is unchanged and is now on firmer evidence: no Prysm binary swap across the fork. Corrected below, along with a new finding this reconciliation exposed — watchtower is not the only thing replacing containers on this fleet (§3).

What we observe

Watchtower emits Found new image whenever a watched tag resolves to a new digest, then stops the container and creates a replacement. Aggregating those events for ResourceAttributes['network']='glamsterdam-devnet-9' over the run:

Image (all `ethpandaops/`)	Distinct digests	Replacement events	Watchtower detection window (UTC)
erigon:main	6	240	09-02 10:08 → 09-03 06:44
nethermind:master	5	1,473	09-02 00:37 → 09-02 21:37
besu:main	3	226	09-02 00:38 → 09-03 06:49
lighthouse:unstable	2	1,647	09-02 00:49 → 09-03 07:01
grandine:glamsterdam-devnet-8	2	20	09-02 14:55 → 09-03 06:55
geth:master	1	413	09-02 14:49 → 09-02 15:00
prysm-beacon-chain:develop ^†	1	268	09-02 00:42 → 09-02 00:57
prysm-validator:develop ^†	1	266	09-02 00:43 → 09-02 00:58
nimbus-eth2:unstable	1	80	09-03 06:50 → 09-03 06:59
lodestar:unstable	1	61	09-02 14:49 → 09-02 15:02

^† Read this column as "when watchtower detected a tag move", not "when the binary was deployed". Two caveats apply specifically to the Prysm rows. First, the window is truncated: the query looked back 34 h, and earlier Prysm waves exist outside it (e.g. prysm-validator:develop → 52e7b99abb13 at 09-01 22:11), so "distinct digests" is a floor, not a count for the whole devnet. Second and more important, the digest watchtower recorded here (f7324c1cf6c8) is not the digest that ran the fork — see §3 and the Prysm card below.

The 75 minutes before the fork

Gloas activates at epoch 225 = slot 7200 = 2026-09-02 16:00:00 UTC (genesis 2026-09-01 16:00 UTC + 24 h). The replacement waves land as follows:

T−70 min  14:49:24   lodestar:unstable      e8fce775bfb9    30 hosts  (beacon + validator)
T−70 min  14:49:54   geth:master            87871b65d5ca   412 hosts
T−70 min  14:50:00   erigon:main            bd5ac27ad631    40 hosts
T−65 min  14:55:13   grandine:gd-8          ad678fd104dd     7 hosts
T−55 min  15:05:00   grandine:gd-8          ad678fd104dd     3 hosts   (all 10 now swapped)
T−55 min  15:05:00   nethermind:master      3c0c7d227a3b    13 hosts
T−50 min  15:10:00   nethermind:master      3c0c7d227a3b   282 hosts
T−45 min  15:15:00   nethermind:master      3c0c7d227a3b     5 hosts
========  16:00:00   GLOAS FORK — epoch 225, slot 7200  ========
T+2h05m   18:05:00   nethermind:master      e0d1e1f0911d     2 hosts
T+2h10m   18:10:00   nethermind:master      e0d1e1f0911d   281 hosts
T+2h10m   18:10:00   erigon:main            16725d9962c9    36 hosts
T+5h37m   21:37:00   nethermind:master      b9f6bf4c6461   300 hosts
T+14h52m  09-03 06:52 grandine:gd-8         4a9f26f09401    10 hosts
T+14h55m  09-03 06:55 lighthouse:unstable   32b007fcc377   393 hosts  (beacon + validator)
T+14h50m  09-03 06:50 nimbus-eth2:unstable  58b22bdac7bd    80 hosts

Fleet churn, quantified

763 of 1,008 hosts (76 %) had a client container replaced between T−75 min and the fork.
340 hosts were replaced again in the first 6 h after the fork, while the network was already below the finality threshold.
972 of 1,008 hosts (96 %) were touched at least once; 4,694 replacement events total.
These are lower bounds. They count watchtower replacements only, and watchtower is not the only mechanism replacing containers on this fleet (§3).

Root cause analysis

1. The configuration is doing exactly what it was told to

From ansible/inventories/devnet-9/group_vars/all/all.yaml:

ethereum_node_docker_watchtower_enabled: true
ethereum_node_docker_watchtower_containers_list:
  - execution
  - beacon
  - validator
  - xatu-sentry
docker_watchtower_container_additional_args:
  - --interval=900
  - --stop-timeout=300s
  - --include-restarting

ansible/inventories/devnet-9/group_vars/buildoor.yaml adds buildoor to that list on the four builder hosts. all.yaml also sets ethereum_node_images_always_pull: true.

Three properties of this config matter:

--interval=900 — every host independently polls the registry every 15 minutes. There is no maintenance window, no coordination, and no awareness of the fork schedule.
--include-restarting — containers that are already crash-looping are eligible for replacement too, so a client stuck in a restart loop gets its binary swapped underneath it.
The watched list is precisely the consensus-critical set: execution, beacon, validator.

2. Every client tag on this devnet is mutable — including the ones treated as pinned

From ansible/inventories/devnet-9/group_vars/all/images.yaml, not one client image is pinned to a digest or an immutable version:

lighthouse: ethpandaops/lighthouse:unstable
lodestar:   ethpandaops/lodestar:glamsterdam-devnet-8
nimbus:     ethpandaops/nimbus-eth2:unstable
prysm:      ethpandaops/prysm-beacon-chain:develop
teku:       ethpandaops/teku:master
grandine:   ethpandaops/grandine:glamsterdam-devnet-8
besu:       ethpandaops/besu:main
geth:       ethpandaops/geth:master
erigon:     ethpandaops/erigon:main
nethermind: ethpandaops/nethermind:master
reth:       ethpandaops/reth:glamsterdam-devnet-8
nimbusel:   ethpandaops/nimbus-eth1:glamsterdam-devnet-8
ethrex:     ethpandaops/ethrex:main

The trunk tags (:master, :main, :unstable, :develop) moving is expected and intended — devnet-9 is explicitly a trunk devnet. The sharper problem is the :glamsterdam-devnet-N tags. The devnet spec sheet records reth and nimbus-EL as "❌ pinned", and Grandine, reth, nimbus-EL and (from commit 0e07117) lodestar all point at glamsterdam-devnet-8. These read as version pins, but they are ordinary mutable Docker tags:

grandine:glamsterdam-devnet-8 resolved to two different digests during this run — ad678fd104dd at 09-02 14:55, then 4a9f26f09401 at 09-03 06:52. A tag the team is treating as frozen was rebuilt and redeployed twice mid-devnet, once inside the hour before the fork.

The repository already contains the counter-example. images.yaml pins tracoor: …:0.0.2-glamsterdam-devnet-8 with the comment # pinned: floating tag + IfNotPresent won't re-pull. The floating-tag hazard is understood; the reasoning was applied to one tooling image and not to the thirteen client images.

3. Watchtower is not the only source of binary churn — ansible redeploys are invisible in these logs

Reconciling the Prysm timestamp against the hosts exposed a mechanism this report initially missed. docker inspect on three Prysm hosts:

bootnode-1          sha=sha256:274dc1e8…  created=2026-09-02T05:25:16Z  started=05:25:16Z        restarts=0
prysm-geth-22       sha=sha256:274dc1e8…  created=2026-09-02T05:26:16Z  started=05:26:16Z        restarts=0
prysm-nethermind-2  sha=sha256:274dc1e8…  created=2026-09-02T05:28:09Z  started=16:01:28Z        restarts=1
image sha256:274dc1e88061f1933369c608809aa04d9ae6ff82050b471eaeb41b25a2ab2bfb  built 2026-09-02T05:10:50Z

Watchtower logged zero events mentioning any Prysm image after 09-02 01:00 UTC (count=0). So the 05:25–05:28 replacement was not watchtower. On prysm-geth-22, beacon and validator were created 0.4 s apart (05:26:16.416 and 05:26:16.773) while every other container on the host kept its earlier creation time — the signature of a role-scoped ansible playbook run, deploying an image built 15 minutes earlier at 05:10:50.

This was not a fleet-wide run: teku-geth-3's beacon dates from 09-01 20:49 and nimbus-geth-26's from the 09-03 06:56 watchtower wave. But it was not the only such run either — lodestar-geth-2 had both its beacon and execution containers recreated at 09-02 15:30, i.e. 30 minutes before the fork, which likewise appears nowhere in the watchtower logs.

Consequence for this incident

A fleet manifest reconstructed from watchtower logs alone is incomplete. It misses every ansible-driven container replacement — which on this run includes the entire Prysm fleet at T−10.5 h and at least some Lodestar hosts at T−30 min. Any "what was running when?" question must join watchtower logs (what the registry moved) with per-host docker inspect (what was actually deployed and when).

4. Watchtower replaces containers, which corrupts the restart-count signal

Watchtower does not docker restart. It emits Stopping container … signal=SIGTERM followed by Started new container — a brand-new container from the new image. Two consequences for anyone debugging this incident:

RestartCount resets to zero on every watchtower update. The restart counts observed in the fleet sweep (Grandine 47–61, buildoor-teku-nethermind-1 558) are since the last watchtower swap, not since the node was deployed. Crash rates are systematically understated.
Container age is watchtower's clock, not the client's. The fleet-wide docker ps reading of "beacon Up 2 hours" at 08:45 UTC is the 06:52–06:55 lighthouse/grandine/nimbus wave, not a client event. Confirmed directly: lighthouse-geth-1 runs sha256:32b007fcc377… created 2026-09-03T06:55:11Z, and grandine-geth-1 runs sha256:4a9f26f09401… created 2026-09-03T06:52:24Z.

Impact

Measurement integrity — the primary harm

devnet-9 has two stated goals: a clean Gloas fork transition from a large pre-fork state, and a deliberate non-finality / inactivity-leak test. Both are measurements. A measurement requires that the system under test hold still.

Instead, 76 % of the fleet had a consensus-critical binary replaced in the 75 minutes before the fork, and 340 more hosts were swapped in the 6 h after it while the network was already in the leak. The concrete costs:

The fork transition is not reproducible. There is no single fleet manifest to point at. Answering "what was running when the chain failed to finalise?" requires reconstructing per-host digests from watchtower logs, which is what this report had to do.
Restart and uptime telemetry is unreliable for every other report in this incident, for the reasons above.
A leak-recovery measurement is not currently possible. Recovery from an inactivity leak is a multi-hour observation; with a 15-minute poll interval and trunk tags that moved 5–6 times a day, the binary set will change several times inside any recovery window.
Client teams receive ambiguous bug reports. A report saying "Grandine crash-loops on devnet-9" is not actionable without the digest, because two Grandine digests ran during the window.

Did watchtower cause the fork failures? Two different answers

Prysm — negative result

No. Watchtower performed zero Prysm image replacements after 09-02 01:00 UTC. The binary that met the fork was deployed by ansible, not watchtower, and was in place well before it:

Quantity	Measured value	Relative to fork
Image digest	sha256:274dc1e88061f1933369c608809aa04d9ae6ff82050b471eaeb41b25a2ab2bfb	—
Image built	2026-09-02 05:10:50 UTC	T−10 h 49 m
Containers created	2026-09-02 05:25:16 – 05:28:09 UTC	T−10 h 32 m
Last watchtower Prysm event	2026-09-02 00:58 UTC (digest `f7324c1cf6c8`)	T−15 h
Watchtower Prysm events after 01:00	none (count = 0)	—

Behaviour across the boundary, measured directly: bootnode-1 and prysm-geth-22 both show restarts=0 with StartedAt equal to their 05:25/05:26 creation time — they ran straight through the fork on the same process. prysm-nethermind-2 shows restarts=1 with StartedAt=2026-09-02T16:01:28Z, 88 s after the boundary, on the same digest — a symptom of the fork failure, not a binary swap.

The Prysm state-root divergence at slot 7203 therefore happened on a binary that had been running unchanged for 10.5 hours across roughly 3,100 slots of pre-fork operation. It is a genuine client bug, not a mid-run binary swap. This strengthens the Prysm report rather than complicating it.

Grandine — genuine correlation, causation not established

Correlated, and it materially complicates the analysis. All 10 Grandine hosts were replaced with ad678fd104dd at 14:55 (7 hosts) and 15:05 (3 hosts) — 65 and 55 minutes before the fork. Grandine then failed to cross the fork.

This is not evidence that watchtower caused the failure. The ExecutionPayloadEnvelopesByRange { start_slot: 7200, count: 0 } livelock is a deterministic logic error at the fork boundary and would be expected to reproduce on any build with that code path. What the swap does establish is that nobody validated the exact binary that met the fork: the image was pushed and deployed inside the final hour, with no soak time.

The consequence that matters for the Grandine report: the binary that failed the fork (ad678fd104dd) and the binary now OOM-crash-looping (4a9f26f09401, deployed 09-03 06:52) are different images. The 30–31 GB RSS behaviour observed today was measured on a build that was not present at the fork. Both digests must be named in any reproduction request.

One further piece of config churn in the same window, noted for completeness and outside the scope of this report: commit 0e07117 "swap lodestar image" (lodestar:unstable → lodestar:glamsterdam-devnet-8) was authored 2026-09-02 17:56 +0200 = 15:56 UTC, four minutes before the fork. lodestar:glamsterdam-devnet-8 never appears in the watchtower logs, so the running Lodestar containers were not moved to it by watchtower; whether a playbook run deployed it is not determined here.

How to verify / reproduce

Reconstruct the per-image digest history for any window (read-only):

panda clickhouse query-raw clickhouse-raw "
SELECT toStartOfInterval(Timestamp, INTERVAL 5 MINUTE) AS t,
       extract(Body, 'image=([^ ]+)')      AS image,
       extract(Body, 'container=([^ ]+)')  AS ctr,
       extract(Body, 'new_id=([0-9a-f]+)') AS new_id,
       uniq(ResourceAttributes['host.name']) AS hosts
FROM external.otel_logs
WHERE Timestamp >= toDateTime('2026-09-02 13:00:00')
  AND Timestamp <= toDateTime('2026-09-02 19:00:00')
  AND ResourceAttributes['network'] = 'glamsterdam-devnet-9'
  AND ResourceAttributes['container.name'] = 'ethereum-node-docker-watchtower'
  AND match(Body, 'Found new image')
GROUP BY t, image, ctr, new_id ORDER BY t"

Confirm what a given host is actually running right now:

ansible grandine-geth-1 -m shell -a \
  "docker inspect beacon --format 'img={{.Config.Image}} sha={{.Image}} created={{.Created}}'"

# grandine-geth-1 → img=…/grandine:glamsterdam-devnet-8
#                   sha=sha256:4a9f26f09401a19c6de2156599b547169753ec305bac221471d8fa55860a9e14
#                   created=2026-09-03T06:52:24Z

Note that Update session completed … updated=N (103,743 rows over the run) is the per-poll heartbeat and is emitted whether or not anything changed. Filter on Found new image for actual replacements, and on Started new container to count container recreations.

Do not use watchtower logs alone

Watchtower logs answer "when did the registry tag move, and did watchtower act on it". They do not capture ansible-driven replacements, which on this run included the whole Prysm fleet. A container's true provenance needs both halves — creation time from the container, build time from the image:

ansible <host> -m shell -a \
  "docker inspect beacon --format 'sha={{.Image}} created={{.Created}} started={{.State.StartedAt}} restarts={{.RestartCount}}';
   docker image inspect \$(docker inspect beacon --format '{{.Image}}') --format 'built={{.Created}}'"

A container whose created timestamp has no corresponding Found new image line within a few seconds was replaced by something other than watchtower. Beacon and validator created within a second of each other, with the rest of the host untouched, indicates a role-scoped playbook run.

Recommended action

Disable watchtower for the remainder of devnet-9. The opt-out already exists and is a one-line change:
```
# ansible/inventories/devnet-9/group_vars/all/all.yaml
ethereum_node_docker_watchtower_enabled: false
```
This is the highest-value action: the network is currently below the finality threshold, and every further swap wave adds churn and makes the leak measurement less interpretable.
Pin client images by digest for fork-transition and leak tests, and capture the manifest from the hosts. Resolve each tag to a @sha256:… at launch and record it in images.yaml; then snapshot the actual per-host digests with a docker inspect sweep at T−1 h, since the config file records intent and only the hosts record fact. This keeps the trunk-devnet intent (the digests are still trunk builds) while making the run reproducible and giving client teams an unambiguous artifact to reference.
Stop treating :glamsterdam-devnet-N tags as pins. They are mutable and grandine:glamsterdam-devnet-8 demonstrably moved twice mid-run. Either pin them by digest or rename them so they are not mistaken for frozen builds. The existing tracoor comment in images.yaml is the right instinct applied to the wrong subset of images.
If watchtower stays enabled between devnets, add a fork freeze window. No image replacement from T−24 h to T+24 h around a scheduled fork epoch, and none during a declared leak test.
Reconsider --include-restarting. Swapping the binary under a crash-looping container destroys the evidence needed to diagnose why it was crash-looping, and resets RestartCount, hiding the severity of the loop.
When filing client bugs from this incident, quote the digest, not the tag. Specifically: Prysm prysm-beacon-chain:develop @ sha256:274dc1e88061f1933369c608809aa04d9ae6ff82050b471eaeb41b25a2ab2bfb (built 09-02 05:10:50 UTC, deployed 05:25–05:28 UTC, unchanged across the fork); Grandine fork failure on ad678fd104dd, current OOM loop on 4a9f26f09401. Take the digest from docker inspect on a host, not from the watchtower logs.

Evidence appendix

Raw watchtower log lines

time=2026-09-02T14:55:13Z level=info msg="Found new image" container=beacon \
  image=registry-1.docker.io/ethpandaops/grandine:glamsterdam-devnet-8 new_id=ad678fd104dd…

time=2026-09-02T15:10:00Z level=info msg="Found new image" container=execution \
  image=registry-1.docker.io/ethpandaops/nethermind:master new_id=3c0c7d227a3b…

time=…Z level=info msg="Stopping container" container=execution id=… signal=SIGTERM timeout=…
time=…Z level=info msg="Started new container" container=execution \
  image=registry-1.docker.io/ethpandaops/nethermind:master new_id=…

time=…Z level=info msg="Watchtower 1.x.x using Docker API v1.x"
time=…Z level=info msg="Next scheduled run: … in 14 minutes 5x seconds"
time=…Z level=info msg="Update session completed" failed=0 notify=no scanned=N skipped=0 updated=N

Event volumes over the run (≈34 h)

Signal	Count
`Update session completed` (15-min heartbeat, all hosts)	103,743
`Found new image` (actual replacements)	4,694
Distinct hosts with ≥1 replacement	972 / 1,008
Hosts replaced in T−75 min → fork	763
Hosts replaced in fork → T+6 h	340

Scope note

This report covers fleet image churn only. The Prysm state-root divergence, the Grandine envelope livelock and OOM, the Teku heap and tick-scheduler stalls, the buildoor host sizing, and the service-tier failures are covered in their own reports. The cross-cutting findings that belong to those reports are: (a) Prysm's binary — sha256:274dc1e8…, built 09-02 05:10:50 UTC, deployed 05:25–05:28 UTC — was stable across the fork, with prysm-nethermind-2's single restart at 16:01:28 UTC landing 88 s after the boundary on the same digest; and (b) Grandine's fork failure (ad678fd104dd) and its current OOM loop (4a9f26f09401) occurred on two different digests.

Revision history

Rev 2 (2026-09-03) — corrects the Prysm deployment timestamp from "00:42 UTC, 15 h before the fork" (a watchtower detection event for a digest that never ran the fork) to the measured container creation time of 05:25–05:28 UTC, T−10.5 h, on sha256:274dc1e8…; adds §3 on ansible-driven replacements that watchtower logs do not capture. Conclusion unchanged: no Prysm binary swap across the fork. Rev 1 is superseded.

Compiled from a 1008-host ansible sweep of every node's beacon and engine API, a second sweep for kernel OOM state, and aggregation over external.otel_logs via panda clickhouse. Genesis 2026-09-01 16:00 UTC (MIN_GENESIS_TIME 1788274800 + 3600 s GENESIS_DELAY); Gloas fork epoch 225 = slot 7200 = 2026-09-02 16:00 UTC.