Teku: canonical blocks marked invalid, starved attestation subnets, and a 12 GB heap ceiling

Teku is running three independent failures on glamsterdam-devnet-9, and the most serious one was not the one we set out to investigate. Alongside a real heap-exhaustion problem (-Xmx12g against a ~4M-entry validator registry) and a peer-table starvation problem that is not caused by it, we found that Teku nodes transiently mark canonical blocks as having "failed validation" and then reject every attestation voting for them — 75,380 rejected gossip attestations across 12 hosts in three hours. We verified seven distinct rejected block roots against a healthy Lighthouse node: all seven are canonical: true. That is a consensus-level divergence, not a resource problem.

What we observe

Across 140 Teku nodes in the one hour to 2026-09-03 08:50 UTC:

* measured over a 3-hour window

host                     noPeers   heap   ticks  taskQ  pubFail
teku-besu-3                  534      0     158      0      529
teku-geth-1                  351      6     523    238      379
teku-besu-6                  127     58     215     53      223
teku-ethrex-1                 73      0      46      0       73
teku-reth-15                  68      0     179      0       68
teku-reth-6                   41   4436     185      0       44
...
teku-besu-4                    0      0    7173      0        0   <- wedged
teku-nethermind-19             0      0    7170      0        0   <- wedged
teku-besu-8                    0      0    7186      0        0   <- wedged
teku-nethermind-36             0      0    7184      0        0   <- wedged
teku-geth-44                   0  12891       0      0        0   <- pure heap

Root cause analysis

A. Teku transiently marks canonical blocks INVALID most serious

Twelve Teku hosts entered a state where they consider a recent canonical block to have "failed validation", and then reject every gossip attestation and aggregate that votes for it:

WARN - Rejecting gossip message on topic /eth2/6ebb965e/beacon_attestation_61/ssz_snappy,
  reason: Attestation votes for a block that failed validation:
  0xe981e6888897963968d5f91f69ddd4a681468f10d54129dbd512f4b6be2790e3

ERROR - Validator *** Produced invalid attestation for slot 11979. Invalid reason:
  Attestation votes for a block that failed validation:
  0xb43addd02eea934abdd00ebca65373ce48b571d935598040af8e20bb3ab470d6

We resolved every one of those roots against a healthy Lighthouse node (lighthouse-geth-1, at the tip, execution_optimistic: false). All of them are canonical:

Note that slots 12286, 12287 and 12288 are consecutive. teku-besu-6 was rejecting the execution payload envelopes of three canonical blocks in a row on the ePBS /eth2/6ebb965e/execution_payload/ssz_snappy topic:

WARN - Rejecting gossip message on topic /eth2/6ebb965e/execution_payload/ssz_snappy,
  reason: Execution payload envelope block with root
  0x8b661d9528286c11f8a417408f8dce550d1802c23a6ff28f625884cc51c2a0b8 is invalid

What we could establish about the mechanism. The condition is episodic and self-resolving — each host shows a burst lasting 2–70 minutes and then recovers:

teku-besu-3     36,062 events   07:07:10 -> 08:16:51
teku-erigon-2   11,636 events   07:14:45 -> 07:16:48   (2 min)
teku-besu-6     11,618 events   08:50:14 -> 08:58:18
teku-geth-51     8,861 events   07:11:36 -> 07:26:01
teku-geth-34     8,469 events   07:16:29 -> 08:16:47

Immediately before teku-besu-6's burst, its log shows fork choice missing its deadline and blocks importing after the attestation cutoff — but importing successfully:

08:44:24.654 WARN - Late Block Import *** Block: 5b5aff82…(12214) Proposer: 2952226 Result: success
08:44:56.295 WARN - Timeout waiting for fork choice to complete for slot 12224 (limit: 1500 ms).
                    Continuing without waiting.
08:45:17.828 ERROR - Failed to process pending attestation dependent on 0x5b5aff82…

We could not confirm an EL-side cause. Every engine-API payload status Teku actually logged in this window was VALID ([NewPayload] got quick payload status payloadStatus="&{Status:VALID …}", [ForkChoiceUpdated] … status=VALID), and the sibling investigation found zero invalid-block, state-root-mismatch or BAL errors across all seven execution clients in the same period. So this is not the EL returning INVALID. The available evidence points at Teku's own block/envelope validation path reaching a terminal "failed validation" verdict under fork-choice timeout pressure, where the optimistic-sync rules would require it to stay optimistic rather than latch INVALID — but we did not obtain a log line naming the specific validation predicate that failed, so the precise trigger is unresolved and needs Teku-side diagnosis.

B. Attestation subnet starvation — independent of the heap problem

This was the hypothesis we most expected to fold into the heap story. It does not.

70% of the affected hosts have no heap pressure whatsoever, and the four most heap-starved nodes show zero subnet errors — they are too wedged to even attempt a publish. The two phenomena are not causally linked.

It is also not one bad subnet. All 64 attestation subnets are affected (6,483 events over 3 h across 41 hosts); subnet 33 is merely the modal value at 276 events:

subnet:events(hosts)
 0:97(15h)   1:174(18h)   2:132(17h)   ...  33:276(30h)  34:169(29h)  ...
54:260(29h)  58:129(26h)  60:197(26h)  ...  63:104(19h)
=> 65 distinct topic values, i.e. the full 0..63 subnet space

Root cause: peer-table poisoning by the stuck cohort. We sampled the peer table of a healthy Teku node and correlated every connected peer's IP against the inventory addresses of the 312 nodes stuck at slot 7199 (all Prysm nodes, all four bootnodes, Grandine):

teku-geth-3   connected peers: 80
              of which stuck-at-7199 (prysm / bootnode / grandine): 36
              => 45.0% of the peer table is dead weight

Those nodes advertise the same fork digest 0x6ebb965e, so they are accepted as valid peers, occupy slots in a bounded peer table, and never relay post-fork attestations. With 45% of a peer table inert, individual gossipsub subnet meshes routinely end up with no usable peer and the publish fails outright.

C. Heap exhaustion → GC thrash → wedged event loop

Confirmed and real, but narrower than the other two. The setting is ours, not a Teku default:

ansible/inventories/devnet-9/group_vars/teku.yaml:32:  JAVA_OPTS: "-Xmx12g"

Against a registry of roughly 4M entries (1,000,000 active + 3,000,000 exited) on hosts with 32 GB of RAM, a 12 GB heap ceiling is too low. Two distinct end states:

• Wedged — teku-besu-8, teku-besu-4, teku-nethermind-19, teku-nethermind-36: JVM resident at 13.8 GB against the 12 GB ceiling, CPU ~0.5%, dropping ~7,180 ticks/hour (every single 500 ms tick), beacon API unresponsive. The event loop is blocked, not busy.
• Thrashing — teku-erigon-3 at 2,455 Java heap space/h and 97% CPU; teku-geth-44 at 12,891/h. Alive, burning the CPU in GC.

Host memory is available: free -g on these nodes shows 31 GB total, ~16 GB used, ~14 GB available. The ceiling is the binding constraint, not the hardware.

Impact

• 14 of 140 Teku beacon APIs are unresponsive; their validators (~13,000 keys) contribute nothing.
• Failure mode A is the one with network-wide reach: a node that rejects attestations for canonical blocks both withholds its own votes and refuses to relay other validators' votes. At 75,380 rejections in 3 hours it is actively suppressing attestation propagation, which depresses measured participation beyond the offline-node count.
• Failure mode B costs attestations on otherwise perfectly healthy nodes, and will persist for as long as a third of the network sits on the pre-fork chain.
• Teku is not alone in losing attestations: Nimbus logs 26,119 Attestation not sent events across 71 of its 80 hosts in the same window. The publish-side problem is fleet-wide; Teku is simply the client that names the cause in its logs.

How to verify / reproduce

A — canonical blocks marked invalid. Take any root from a rejection line and resolve it on a node that crossed the fork:

panda clickhouse query-raw clickhouse-raw "
  SELECT ResourceAttributes['host.name'], Timestamp, substring(Body,1,320)
  FROM external.otel_logs
  WHERE Timestamp >= now() - INTERVAL 3 HOUR
    AND ResourceAttributes['network']='glamsterdam-devnet-9'
    AND match(Body,'failed validation') ORDER BY Timestamp DESC LIMIT 10"

curl -s http://localhost:5052/eth/v1/beacon/headers/<root> | jq '.data.canonical, .data.header.message.slot'
# expected on a healthy node: true

B — peer-table poisoning. Correlate a Teku node's peer table against the stuck cohort's inventory addresses:

curl -s http://localhost:5052/eth/v1/node/peers \
  | jq -r '.data[] | select(.state=="connected") | .last_seen_p2p_address' \
  | grep -oP '/ip[46]/\K[^/]+' | sort > /tmp/peers.txt
# intersect with ansible_host / ipv6 of the prysm_*, bootnode and grandine_* inventory groups

C — heap. docker stats --no-stream beacon against JAVA_OPTS; a node pinned at the ceiling with near-zero CPU is wedged, one at ~100% CPU is thrashing.

Recommended action

1. Teku team — highest priority. Failure mode A is a consensus divergence, not a resource issue. A Teku node latching "failed validation" on a block the rest of the network holds canonical, and then censoring attestations for it, needs a root-cause diagnosis on the Teku side. The trigger correlates with Timeout waiting for fork choice to complete … Continuing without waiting and late block imports; the engine API reported VALID throughout. Please confirm whether the block/envelope validation path can reach a terminal INVALID verdict on timeout or on a missing ePBS envelope, where optimistic-sync rules require staying optimistic.
2. devops — raise -Xmx. ansible/inventories/devnet-9/group_vars/teku.yaml:32 is -Xmx12g on 32 GB hosts with ~14 GB free. -Xmx20g or higher is warranted for a 4M-entry registry. This fixes mode C only, and will not touch A or B.
3. devops — mode B is a symptom of the fork split. No Teku-side change will fix a peer table that is 45% inert. It resolves when the 312 stuck nodes are recovered or removed from the network. Until then, treat Teku's subnet-publish failures as expected collateral.
4. Nimbus team — cross-check. 26,119 Attestation not sent across 71 of 80 hosts suggests the same peer-table starvation is costing Nimbus attestations silently.
5. Restart the 4 wedged nodes (teku-besu-8, teku-besu-4, teku-nethermind-19, teku-nethermind-36) after the heap change — they will not recover on their own.

Evidence appendix

Wedged hosts — every tick dropped

teku-besu-8          7,186 dropped ticks/h   beacon mem 13.82 GiB / 31.35 GiB   CPU 0.52%
teku-besu-4          7,173 dropped ticks/h
teku-nethermind-19   7,170 dropped ticks/h
teku-nethermind-36   7,184 dropped ticks/h
JAVA_OPTS=-Xmx12g ; docker inspect: status=running restarts=3 oom=false memlimit=0

Heap thrash

teku-geth-44    12,891 Java heap space/h  (0 dropped ticks, 0 subnet failures)
teku-reth-6      4,436 /h
teku-geth-29     3,077 /h
teku-erigon-3    2,455 /h   beacon mem 14.03 GiB / 31.35 GiB   CPU 97.40%

Other Teku error signatures in the window

ERROR - Unhandled exception in timer tick handler                       5,921 (1 host)
ERROR - Unexpected rejected execution due to full task queue
        in p2p-async-scheduler-N                                          877 (6 hosts)
ERROR - Execution Client request failed. Make sure the Execution
        Client is online and can respond to requests.                     928 (5 hosts)
ERROR - Failed to update fork choice.                                     461 (4 hosts)
FATAL - Exiting due to fatal error in prometheus-http-N-N                   8 (4 hosts)
ERROR - Java heap space                                                     3 (3 hosts)

Scope note

Read-only investigation throughout: ad-hoc ansible shell/script modules, panda clickhouse aggregation over external.otel_logs, and direct beacon API reads. No playbook was run, no node was restarted, no configuration was changed.