Departed peer can remain listed as state: serving after repeated node-identity churn

[ndizazzo]

Summary

After a node identity leaves a mesh, the creator can end up listing it again as state: serving with
a populated model list and no observed RTT, long after the peer is gone.

Observed on the creator while exactly one peer process existed on that host:

id: f90d9d1d10 | mesh1   | state: serving | rtt_ms: 1
id: 9dd38366b6 | carrack | state: serving | rtt_ms: 1      <- live process, self id 9dd38366b6
id: 4cc4d7bbbf | carrack | state: serving | rtt_ms: None   <- no corresponding process anywhere

Corroborated at the same moment:

• pgrep -af "mesh-bundle/mesh-llm serve" on that host returned exactly one process, whose own
  /api/status reported node_id: 9dd38366b6.
• That live node's own peer view was correct: 2 peers, no duplicate.
• So the phantom existed only in the creator's view.

Signals on the phantom: rtt_ms: None and a stale first_joined_mesh_ts (1789065385138 vs
1789068069115 for the live node). It had been stopped roughly 45 minutes earlier.

Root-cause hypothesis (grounded in the code, not yet confirmed)

DEAD_PEER_TTL is 300 seconds (mesh/peer_state.rs), and its own doc comment states the
assumption that appears to have failed:

After this period the entry expires silently and the peer can be re-discovered through normal
gossip propagation. If the peer is genuinely gone, no bridge peer will mention it and it stays
forgotten.

The phantom outlived the tombstone by ~40 minutes, so the likely sequence is:

1. Peer dies. Heartbeat evicts it and records a tombstone in MeshState::dead_peers.
2. After 300s the tombstone expires.
3. A third party's gossip still carries a stale announcement for that EndpointId.
4. The peer is re-learned transitively. last_mentioned is refreshed and state is taken from the
   stale announcement, which said serving.
5. No connection is ever established, so rtt_ms stays None, and nothing demotes the entry.

That also explains why it needed churn and a third node: with three nodes, one of them can keep
rebroadcasting a stale announcement for a departed identity past the tombstone window. Note that
each restart in the observed session wiped ~/.mesh-llm/key, so every restart was a genuinely new
EndpointId and multiple ids per hostname is legitimate; the bug is the resurrection, not the
duplication.

This is a hypothesis. It has not been confirmed, and no PR is attributed. #1673 touched adjacent
code (transitive_peer_update_*), which makes it a reasonable place to look first, but that change
concerns the memory field rather than liveness, so it is a starting point and not a finding.

Impact

No routing impact was observed: with the phantom present, 15 consecutive inference requests across
three models all returned HTTP 200 from the correct live nodes, and /v1/models was correct on all
three nodes.

The impact is operator-facing. A peer list that reports a nonexistent node as actively serving
overstates mesh capacity and would mislead capacity or availability decisions, and it is exactly the
kind of thing that wastes time during an incident.

What demonstrably already works (must not regress)

A single clean stop was handled correctly in the same session:

warn  💛 Heartbeat: 78ab1fe122 unreachable (1/2), will retry
warn  💔 Heartbeat: 78ab1fe122 unreachable (2 failures), removing + broadcasting death
warn  ⚠️ Peer 78ab1fe122 died — removing and broadcasting

And a legitimate rejoin with the same invite restored state: serving, the model union, and
cross-node routing. Any fix must keep both of these intact, which is why they appear as explicit
guard tests below.

---

Fix plan (TDD)

The mesh module already has a substantial test harness under
crates/mesh-llm-host-runtime/src/mesh/tests/ (including gossip/, announcement_unique.rs,
direct_path.rs, connections.rs), so tests go there rather than needing new scaffolding.

One structural note: dead_peers: HashMap<EndpointId, Instant> and last_mentioned: Instant use
wall-clock Instant directly, so the resurrection decision is not currently unit-testable in a
time-dependent way. Phase 1 therefore extracts a pure decision function before asserting behavior on
it, which is the smallest change that makes the bug expressible as a test.

Phase 0 — Confirm the mechanism and settle whether it is a regression

Two things to establish before writing a fix, because they change its shape:

1. Confirm the resurrection path. Reproduce with three nodes (below) and capture, on the creator,
   the gossip that re-adds the departed id: enable --swarm-capture on all three and look for a
   peer_direct_add/peer_direct_update for the dead EndpointId after its
   removing + broadcasting death. Record which node mentioned it.
2. Establish regression status. Run the same churn sequence against a v0.76.0-only mesh and a
   current-main mesh and compare. Regression status is currently UNVERIFIED; do not attribute a
   PR until this says so.

Reproduction sequence used when this was found:

1. Create a private mesh on node A. Join nodes B and C.
2. Stop node B and rejoin it with a fresh identity each time (wipe ~/.mesh-llm/key), several
   times, including at least one join from a different binary version.
3. Wait past DEAD_PEER_TTL (>5 minutes) after the final stop.
4. Compare node A's /api/status peer list against processes actually running on each host.

Phase 1 — RED: make the decision testable, then assert it

Extract the resurrection decision into a pure function so time and gossip provenance are inputs:

// mesh/peer_state.rs
pub(super) enum RelearnDecision { Accept, RejectStaleMention, RejectTombstoned }

pub(super) fn classify_transitive_relearn(
    tombstoned_for: Option<Duration>,   // None if no tombstone
    announcement_age: Duration,         // age_ms_at_received
    has_connection: bool,
) -> RelearnDecision

Then write these failing tests in mesh/tests/peer_state.rs:

1. expired_tombstone_does_not_resurrect_peer_from_a_stale_mention
   Tombstone older than DEAD_PEER_TTL, announcement older than the recorded death, no connection.
   Expect RejectStaleMention.
   Fails today: the peer is accepted once the tombstone lapses.

2. relearn_requires_an_announcement_newer_than_the_recorded_death
   Same but with an announcement newer than the death. Expect Accept, so genuine recovery on
   another path still works and the fix is not a blanket block.

3. live_tombstone_still_rejects_regardless_of_announcement_age
   Guards the existing 300s behavior.

4. transitively_relearned_peer_is_not_reported_as_serving_without_a_connection
   Asserts the operator-visible symptom directly: a peer re-learned only by gossip, with no
   connection and no observed RTT, must not surface as state: serving. Whether it is represented as
   a distinct "known, unverified" state or simply not listed is the design decision to make here.
   Fails today.

5. peer_with_no_connection_and_no_observed_rtt_is_not_routing_eligible
   The safety property. Routing was unaffected in the observed incident, and this pins that down so
   the phantom can never become consequential.

Phase 2 — RED: guard tests for the paths that already work

Write these before touching the implementation, and require them green throughout:

6. direct_rejoin_clears_the_tombstone_and_restores_serving
   The verified-working rejoin path: same identity reconnecting directly must clear the tombstone and
   return to serving.

7. clean_stop_still_demotes_then_evicts
   The verified-working eviction path: standby with empty serving_models, then eviction after the
   heartbeat failure threshold.

8. distinct_identities_from_one_hostname_remain_separate_peers
   Multiple EndpointIds from the same hostname are legitimate and must not be collapsed, so a fix
   does not "solve" this by deduplicating on hostname.

Phase 3 — GREEN: minimum change

1. Route transitive re-learn through classify_transitive_relearn() and reject mentions that are not
   newer than the recorded death.
2. Retain the death timestamp beyond the reconnect-suppression window, so "do not reconnect for 300s"
   and "do not believe stale mentions of this id" become separate lifetimes. A bounded LRU of recent
   deaths is enough; it does not need to be permanent.
3. Do not report serving for a peer with no connection and no observed RTT. Derive that from
   connection state rather than from the last announcement's contents.
4. Make sure nothing in routing selects a peer in that state.

Phase 4 — Real multi-host validation

Unit tests cannot prove this one either, because it needs real gossip across three nodes and real
timing past a 5-minute window. Run on three real hosts with packaged binaries.

Scenario	Expected
Steady 3-node mesh	each node lists exactly the other two, all with an observed RTT
Stop one node, wait 2 min (< TTL)	entry demotes to standby / serving_models: [], then evicts
Stop one node, wait 10 min (> TTL)	stays evicted; no resurrection, no serving entry with rtt_ms: None
Churn: 4 stop/rejoin cycles with a fresh identity each, then wait 10 min	peer count on every node equals live nodes; zero entries with rtt_ms: None
Legitimate rejoin with same identity	returns to serving, union restored, cross-node inference works
Mixed-version node in the churn	same as above (the observed incident included a v0.76.0 node)

For each row, capture /api/status from every node (the incident was only visible on the
creator), plus pgrep output per host as ground truth, plus the --swarm-capture JSONL.

Phase 5 — Regression guard and observability

• Keep tests 1-8 as the permanent guard.
• Add an invariant assertion usable in CI or doctor: no peer may be reported serving while it has
  neither a connection nor an observed RTT. That single check would have caught this immediately.
• Consider surfacing the death timestamp or a "last verified" age on peer entries so an operator can
  distinguish a live peer from a remembered one without cross-checking pgrep.

Acceptance criteria

• Phase 0 confirms (or refutes) the resurrection path, with the mentioning node identified
• Regression status settled against v0.76.0; PR attributed only if the evidence supports it
• Tests 1-5 exist and initially fail for the stated reasons
• Tests 6-8 pass before and after the change
• Phase 4 table verified on three real hosts, evidence attached
• Zero peer entries with rtt_ms: None reported as serving in any scenario
• Legitimate rejoin and clean eviction both still work

---

Found during an evidence-backed release validation of a12b535d7 against v0.76.0 on three real
hosts. Recorded there as severity 3 with regression status explicitly UNVERIFIED.