Relay CL sessions (Titan/Ultrasound Lighthouse) drop after days and do not re-dial

[swarna1101]

Summary

Relay gateways (titan-prod-ethereum-gateway-*, ultrasound-prod-ethereum-gateway-*) sit in GCP with no local CL. We give Titan/Ultrasound our libp2p peer IDs; their Lighthouse beacon nodes dial us. After a few days (or after a gateway restart) mump2p_gateway_cl_peers goes to 0 until they restart their BNs. We then ping them again. This blocks any long-term relay latency analysis.

This is not a mump2p mesh issue. Mesh peers stay up. Only the CL libp2p session dies, and neither side redials.

Current status (2026-08-20 ~09:15 UTC)

All six mainnet relays are healthy right now (v1.1.1, paired_with=relay):

Host	Public IP (CL listen 33212)	CL peers	mump2p peers	libp2p blocks / 15m
titan-…-us	136.115.160.220	2	36	~73
titan-…-eu	34.107.100.106	2	42	~73
titan-…-asia	34.143.245.193	2	54	~73
ultrasound-…-us	35.194.60.207	3	42	~72
ultrasound-…-eu	34.141.56.233	3	56	~73
ultrasound-…-asia	35.247.172.22	2	43	~74

Gateway processes last started 2026-08-11 ~15:50 UTC (~8.7d ago). CL sessions on Titan have only been back since 2026-08-18 ~22:00 UTC.

14-day CL peer timeline (mump2p_gateway_cl_peers)

Titan (all 3 regions, same shape):

• until 2026-08-07 13:00 UTC: 9 CL peers
• 2026-08-07 14:00 → 2026-08-11 11:00: 0
• 2026-08-11 11:00–16:00: brief 2 (matches our gateway restart)
• then 0 again until 2026-08-18 22:00
• 2026-08-18 22:00 → now: 2 (stable)

Ultrasound:

• mostly 0 / 1 until 2026-08-12 14:00, then 2–3 and stable since

How they are configured

Infra: ansible,cluster_ethereum_prod, **gateway_only: true** (skips eth-node/eth-cl-proxy`).

# host_vars/titan-prod-ethereum-gateway-us-node-1.yml
gateway_id: titan-prod-ethereum-gateway-us-1
gateway_cluster_id: optimum_ethereum_mainnet_v0_1
gateway_only: true

direct_cl_peers is intentionally unset on relay/hub hosts. From roles/optimum-gateway/templates/app_conf.yml.j2:

Set only on hermes-paired hosts… This also activates a connect-time allowlist that closes every peer not listed, so it must stay unset on hub and relay gateways.

So relays accept inbound CL connections only. We never persist Titan/Ultrasound multiaddrs, so we never outbound-dial them.

Why it stays down until they restart

Documented Lighthouse/Nimbus behaviour: they do not re-dial the gateway after the TCP session dies or after a gateway restart. Docs: Direct CL Peers, Troubleshooting.

On hermes we work around that with direct_cl_peers (gateway keeps retrying). On relays we cannot use that field as-is because it is also an allowlist (onPeerConnected closes anyone not listed — setup_libp2p_host.go).

v1.0.2 advertised custody group count 8 so PeerDAS-aware Lighthouse is less likely to prune us as a useless peer. That does not restore a dead session. After days (idle timeout, BN rotation, LH peer manager, our restart) the link is gone until they restart / re-add --boot-nodes + --trusted-peers.

What we need

A reconnect story for inbound-only relay CLs that does not require pinging Titan/Ultrasound every time cl_peers hits 0.

Options to evaluate (not decided):

1. Split config: outbound keep-alive / re-dial list that is not an allowlist, so relays can pin known CL multiaddrs without dropping other inbound CLs.
2. Partner runbook: require Lighthouse --trusted-peers and --boot-nodes (not a one-shot connect), --semi-supernode, and tell us when they rotate BNs. Still brittle if they forget.
3. Alert + automation: cl_peers == 0 for >N minutes on paired_with=relay pages us; optional gateway-side redial if we store last-seen CL addrs from inbound connects (may go stale on BN IP change).

How to watch

• Mimir: mump2p_gateway_cl_peers{gateway_label=~"titan-.*|ultrasound-.*"}
• Also mump2p_gateway_cl_peer_disconnected_total, block_arrival_libp2p_ms_count

[swarna1101]

Status

No gateway code change. Relays never set direct_cl_peers (it is also an allowlist, and was hermes-only in ansible), so we never re-dial after a CL drop or a gateway restart. Lighthouse does not re-dial us unless they keep persistent --boot-nodes + --trusted-peers.

Done

• Infra: getoptimum/infrastructure#519 - ansible can render direct_cl_peers from relay host_vars; Lists are still empty until we have their BN listen multiaddrs (inbound source ports are not dialable).
• Gitops: getoptimum/gitops#567 - alert when titan/ultrasound cl_peers < 2 for 30m (== 0 missed a multi-day partial drop). Existing zero-peer alert still covers a full loss.

Still needed (Titan + Ultrasound)

1. Confirm BN flags are persistent (--boot-nodes + --trusted-peers + --semi-supernode), not a one-shot connect.
2. Send each BN’s listen multiaddr so we can fill host_vars and roll ansible. Do not ship a partial list - allowlist would drop any BN not listed.

Close when: lists are deployed, one titan gateway restart recovers cl_peers without pinging them, and the same for ultrasound in at least one region.