The devnet-9 service tier is loud but mostly healthy: of ten degraded workloads, six are
faithfully reporting the consensus-layer damage described in the sibling reports and will recover on their own
once Prysm and Grandine cross the fork. Four will not. The independent failures are a
missing-TLS-certificate fault on a subset of per-node rpc-* endpoints (which
breaks spamoor, erpc and the faucet's health checks against healthy nodes),
checkpointz serving 404 because it is pinned to a static upstream list that is now mostly dead nodes,
forky being unable to parse Lighthouse's fork-choice dump at all, and
forkmon returning 503. Nothing here is blocking consensus; the cost is that the tooling a client
dev would reach for to debug the fork failure is itself partly broken.
All counts below are scoped to k8s.namespace.name = 'glamsterdam-devnet-9' over the two hours
ending 2026-09-03 08:58 UTC. This scoping matters: workload names such as tracoor,
erpc, forky and checkpointz are reused across namespaces
(glamsterdam-devnet-8, bal-devnet-2) on the same cluster. Filtering on
k8s.workload.name alone mixes networks — an unscoped query attributes roughly 20,000 bal-devnet
error lines to tracoor that have nothing to do with devnet-9.
| Workload | Lines | Error-matching | level=ERROR | level=warn | Verdict |
|---|---|---|---|---|---|
| slashoor | 85,680 | 80,954 | 0 | 80,958 | consequence |
| tracoor | 325,168 | 65,369 | 316 | 559 | consequence |
| …-xatu-cbt | 1,895,003 | 21,061 | 1 | 169,474 | consequence |
| dora | 17,361 | 14,100 | 65 | 15,457 | consequence |
| forky | 27,645 | 4,783 | 4,783 | 0 | independent |
| erpc | 4,777 | 4,768 | 0 | 0 | independent |
| checkpointz | 1,759 | 1,750 | 1,750 | 0 | independent |
| assertoor | 2,801 | 1,693 | 0 | 1,693 | consequence |
| powfaucet-agents | 4,108 | 1,380 | 0 | 0 | independent |
| dugtrio | 6,388 | 1,268 | 0 | 1,268 | consequence |
| spamoor | 1,134 | 720 | 0 | 729 | independent |
The two loudest workloads are the two least broken. slashoor emits zero
level=ERROR lines — all 80,954 are level=warning SSE reconnects. tracoor
emits 316 true errors out of 325,168 lines; the rest are level=debug. Meanwhile
checkpointz (1,750 lines) and forky (4,783 lines) are ~100% level=ERROR
and are genuinely down or blind. A regex for (error|fail) over Body ranks these
exactly backwards.
| Endpoint | HTTP | Latency | State |
|---|---|---|---|
| faucet.… | 200 | 0.86 s | working |
| assertoor.… | 200 | 0.72 s | working |
| dora.… | 200 | 6.9 s | up, badly degraded latency |
| dora.…/api/v1/epoch/latest | 200 | 8.5 s | up, badly degraded latency |
| forkmon.… | 503 | 0.21 s | down |
| checkpointz.… | 404 | 0.21 s | down — no checkpoint sync available |
rpc-* endpoints — independent bugThis is the most consequential independent finding, because it breaks tooling against nodes that are perfectly healthy. A direct handshake against two per-node RPC endpoints fails before HTTP:
$ curl -sv https://rpc-lighthouse-reth-21.srv.glamsterdam-devnet-9.ethpandaops.io/
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, unrecognized name (624):
* OpenSSL/3.0.13: error:0A000458:SSL routines::tlsv1 unrecognized namerpc-lighthouse-reth-21 http=000 tls=1 <- handshake refused
rpc-teku-geth-19 http=000 tls=1 <- handshake refused
rpc-lighthouse-geth-1 http=401 tls=0 <- normal (401 = auth required)unrecognized name is the server rejecting the SNI: nginx has no certificate for that hostname.
The affected hosts are not the stuck ones — lighthouse-reth-21 and teku-geth-19 are on
the post-fork chain. This single fault explains three downstream workloads:
spamoor: client check failed: Post "https://eth:***@rpc-teku-geth-N…": remote error: tls: unrecognized name (112 per endpoint)erpc: {"code":"ErrUpstreamRequest", … "cause":{"code":"ErrEndpointTransportFailure","message":"failure when sending request to …"}} against rpc-lighthouse-reth-21, rpc-lighthouse-reth-50, rpc-teku-geth-19powfaucet: RPC endpoint lighthouse-geth-N health check failed (164)Most likely the per-host nginx-proxy-cert-linker / cert-loader chain failed to issue or
link a certificate on those nodes. It will not self-heal when the fork split is fixed.
The public endpoint returns HTTP 404: the network has no working checkpoint-sync URL, so there is no supported way to rebuild a node that has fallen past weak subjectivity — precisely the operation the devnet needs most right now. The logs name its configured upstreams:
level=error msg="failed to bootstrap node.. will retry in 5m0s" error="client is not active" upstream=prysm-geth-52
level=error msg="failed to bootstrap node.. will retry in 5m0s" error="client is not active" upstream=teku-nethermind-19
level=error msg="failed to bootstrap node.. will retry in 5m0s" error="client is not active" upstream=teku-geth-11
level=error msg="Failed to fetch finality after epoch transition" error="client is not active" node=teku-besu-1
level=error msg="Failed to fetch finality after epoch transition" error="client is not active" node=grandine-erigon-1
level=error msg="Failed to fetch finality after epoch transition" error="client is not active" node=grandine-geth-2
level=error msg="Failed to fetch finality when polling" error="client is not active" module=beacon/defaultEvery one of prysm-geth-52, teku-nethermind-19, teku-nethermind-23,
teku-besu-1, grandine-erigon-1 and grandine-geth-2 is a node whose beacon API
is confirmed down in the fleet sweep. The discriminator asked for: checkpointz is not failing because the
network has no healthy node — 370 nodes are at the tip. It is failing because it is pinned to a small static list
that happens to be full of dead ones, with no failover to the healthy majority. That design gap is
independent of the fork and should be fixed regardless.
4,783 lines, 100% level=ERROR. The dominant failure is not a dead node — it is a parse failure
against healthy ones:
level=error msg="Failed to fetch frame" component=source/beacon_node
error="failed to get fork choice dump: failed to parse fork choice
invalid JSON: invalid value for validity: : unrecognised fork choice validity: "
source_name=lighthouse-besu-1Lighthouse's /eth/v1/debug/fork_choice is returning a fork-choice node whose
validity field is the empty string, where forky expects one of the known enum values.
It reproduces on all seven Lighthouse sources forky polls, one per EL pairing:
lighthouse-erigon-1 160 lighthouse-geth-1 156
lighthouse-besu-1 158 lighthouse-nethermind-1 156
lighthouse-nimbusel-1 158 lighthouse-reth-1 154
lighthouse-ethrex-1 156The uniformity across every EL pairing rules out an EL-side cause and points at Lighthouse itself. Given that
Gloas/ePBS introduces new payload-availability states into fork choice, an unmapped enum member serialising as
"" is the natural explanation. This is worth filing against Lighthouse — and it means
fork-choice visualisation is unavailable for the entire Lighthouse cohort, which is 430 of 1,008 nodes and the
largest healthy cohort on the network. A secondary, smaller forky failure (not ready to fetch frames
source_name=teku-besu-N, 193) is a consequence of the wedged Teku nodes.
Returns HTTP 503 Service Temporarily Unavailable at the ingress with only 214 error lines logged,
i.e. the pod is not serving rather than erroring in a loop. Needs a direct look at the deployment; nothing in the
log stream explains it.
80,954 lines, all level=warning, all the same shape — SSE stream reconnects against beacon nodes
that are down:
level=warning msg="SSE connection error, reconnecting" component=stream
endpoint="https://bn-grandine-nethermind-1.srv.glamsterdam-devnet-9.ethpandaops.io"
error="unexpected status code: 502" package=beacon
level=warning msg="SSE connection error, reconnecting" component=stream
endpoint="https://bn-prysm-geth-50.srv.glamsterdam-devnet-9.ethpandaops.io"
error="failed to connect to SSE: Get …"Ranked by volume, the failing endpoints are exactly the known-broken set — all ten Grandine nodes (OOM restart loop), plus the Teku and Prysm nodes whose beacon API is down:
grandine-nethermind-2 4351 grandine-geth-1 3964 lighthouse-geth-41 2467
grandine-erigon-1 4311 grandine-reth-1 2705 grandine-geth-4 2421
grandine-geth-2 4285 teku-geth-39 2617 lighthouse-geth-147 2409
grandine-besu-1 4197 teku-reth-9 2562 nimbus-geth-21 2367
grandine-nethermind-1 4006 teku-geth-21 2523 prysm-geth-50 2350slashoor is working correctly. It is an accurate alarm, not a fault.
Only 316 level=ERROR lines in 325,168. The error-matching bulk is level=debug
Failed to fetch sync status … 502 Bad Gateway (3,194) and 504 Gateway Time-out (892)
against the same dead nodes. But the dominant log volume is a sustained fork-choice re-index storm:
time (10-min bucket) block_reindex state_reindex
2026-09-03 07:00 4,519 4,519
2026-09-03 07:20 9,271 9,271
2026-09-03 07:40 4,750 4,750
2026-09-03 08:00 4,840 4,840
2026-09-03 08:20 4,929 4,929
2026-09-03 08:40 10,076 10,076
2026-09-03 08:50 5,108 5,108msg="Queueing up a fresh execution block trace index after a beacon chain reorg" target_slot=N
msg="Queueing up a fresh beacon state index from reorg event" event_depth=N event_old_head_block=H …That is a steady ~8/s of reorg-triggered re-index work with two spikes to ~17/s, which is the fork-choice churn
of a non-finalising network reflected in the forensics tier. tracoor is not broken; it is being asked to re-index
continuously and is keeping up. Also present: Dropping queue item … error="item not available: execution
block H has not been revealed" (330 for lodestar-reth-N, 194 for lighthouse-besu-N)
— an ePBS-specific case where the payload envelope for a block was never revealed.
level=warning msg="error fetching contract logs for fork N (N-N): requested block range is in the future"
contract-indexer=withdrawals
… contract-indexer=builder_exits / consolidations / builder_depositsDora's EL contract indexers request a block range derived from a head they learned from one node, then poll an endpoint that resolves to a different node. With 312 nodes' ELs frozen at block ~6,581 and the rest at ~12,200, a range valid for one is 5,600 blocks in the future for the other. Resolves itself when the split does. The 6.9 s frontend and 8.5 s API latency are a separate, milder concern worth its own look.
21,061 error-matching lines against 1,895,003 total is a 1.1% rate, and exactly one line is
level=ERROR. The recurring shape is a retry, not a failure:
level=warning msg="Failed to calculate initial position, will retry"
error="transformation dependency has not been initialized: glamsterdam-devnet-9.…"CBT models are waiting on upstream transformations that cannot advance while the chain does not finalise. Answering the question directly: this is normal background retry behaviour amplified by non-finality, not a real problem. It needs no action.
assertoor: error processing execution block: could not add block to cache against
bootnode-N, grandine-geth-N, prysm-reth-N — the stuck cohort exactly.
dugtrio: beacon block stream error: EOF from teku-geth-N (266) and
teku-nethermind-N (167), plus stream error: … INTERNAL_ERROR; received from peer from
nimbus-geth-N (130) — the wedged Teku nodes dropping long-lived streams.
checkpointz returns 404, so there is no supported way to
rebuild a node past weak subjectivity — the exact recovery operation this devnet needs.# Namespace-scoped service error census (workload names are NOT unique across networks)
panda clickhouse query-raw clickhouse-raw "
SELECT k8s.workload.name w, count() c,
countIf(match(Body,'(?i)(error|fatal|panic|fail)')) errs,
countIf(SeverityText IN ('ERROR','FATAL','CRIT')) lvl_err
FROM internal.otel_logs
WHERE Timestamp >= now() - INTERVAL 2 HOUR
AND k8s.namespace.name='glamsterdam-devnet-9'
GROUP BY w ORDER BY errs DESC"
# Reproduce the TLS fault (fails before HTTP; compare against a working host)
curl -sv https://rpc-lighthouse-reth-21.srv.glamsterdam-devnet-9.ethpandaops.io/ # unrecognized name
curl -sv https://rpc-lighthouse-geth-1.srv.glamsterdam-devnet-9.ethpandaops.io/ # 401, TLS fine
# Reproduce the forky/Lighthouse parse failure directly
curl -s https://bn-lighthouse-geth-1.srv.glamsterdam-devnet-9.ethpandaops.io/eth/v1/debug/fork_choice \
| jq '.fork_choice_nodes[] | select(.validity == "")' | head
# User-visible endpoint state
curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://checkpointz.glamsterdam-devnet-9.ethpandaops.io/
curl -o /dev/null -w '%{http_code} %{time_total}s\n' https://forkmon.glamsterdam-devnet-9.ethpandaops.io/rpc-* and
bn-* hostname for an unrecognized name handshake alert and repair the
nginx-proxy-cert-linker / cert-loader chain on the affected hosts. Highest value of the
four: it is silently removing healthy nodes from spamoor, erpc and the faucet.validity serialising as ""
from /eth/v1/debug/fork_choice. Confirm whether it is a new Gloas fork-choice state that has no
string mapping.slashoor, tracoor, assertoor, dugtrio, the dora contract indexers and xatu-cbt are all downstream of the Prysm and Grandine fork failures and the resulting dead nodes. They need no work of their own. Consider temporarily raising the log level on slashoor and tracoor so their reconnect churn stops masking new signal while the split persists.
One cross-cutting note for the devops team: because workload names are shared across namespaces on this
cluster, any service-tier query that filters on k8s.workload.name without also pinning
k8s.namespace.name will silently blend devnet-8, devnet-9 and bal-devnet-2 traffic.
All figures from internal.otel_logs, namespace glamsterdam-devnet-9, two-hour window
ending 2026-09-03 08:58 UTC. Endpoint probes taken directly at 08:57–08:59 UTC. Node health states
(which nodes are down or stuck) are carried over from the 1,008-host ansible fleet sweep
described in the companion reports; this report independently confirms them from the service side, where the
failing endpoint names match the stuck cohort exactly.
Scope note: this report covers the k8s service tier only. The Prysm state-transition divergence, the Grandine sync livelock and OOM loop, the Teku heap exhaustion, the builder-node failures and the watchtower churn are covered in the sibling reports for this incident.