fix(bootstrap): age-bound and refresh close-group cache

[dirvine]

Linear issue

V2-884 — Age-bound and refresh the saorsa-core close-group bootstrap cache

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none; no protocol messages or peer validation rules change.
• Storage: existing close_group_cache.json format is unchanged. Old cache files remain readable. Old serialized NodeConfig values that omit the new field receive the one-hour default; explicit null preserves disabled TTL.
• API: adds NodeConfig::close_group_cache_max_age and a builder setter. This is source-breaking for downstream code that constructs the public NodeConfig with a struct literal; builder and deserialization callers remain compatible.

Semver impact

• breaking
• feature
• fix

Test evidence

• cargo check --all-targets --all-features — passed.
• cargo test --lib — 475 passed; unrelated floating-point tolerance test adaptive::trust::tests::test_unit_weight_equivalence failed once at a 1.78e-10 difference, then passed 5/5 isolated reruns.
• Earlier exact-head run before the diagnostics-only follow-up: 476 passed, 0 failed.
• Focused cache/config/task lifecycle tests — passed.
• cargo clippy --all-targets --all-features -- -D warnings — passed.
• cargo fmt --all -- --check and git diff --check — passed.

The proposed T2 classification still requires dev-testnet evidence before release. Detailed matrix: V2-894.

New dependency

none

ADR

ADR-016 — Age-Bounded, Periodically Refreshed Close-Group Cache — Proposed; human acceptance pending.

Mitigation / rollback

Disable age enforcement with close_group_cache_max_age: null, disable cache persistence by omitting close_group_cache_dir, or revert this PR; configured bootstrap peers remain the fallback path.

[jacderida]

Testnet validation — all cache-hygiene cases PASS

Fault-injection run on a dev testnet, 2026-08-10 (V2-894). Binary: ant-node feat/bootstrap-diagnostics @ 3c3e12a (WithAutonomi/ant-node#198), whose [patch.crates-io] pins this PR's head a4faf6f72c1dd4425a2d1f3bdece6789c1f85b83 — verified equal at run start, and confirmed in-band by commit=3c3e12a on every forwarded log line.

Topology: backbone of genesis + 5 bootstraps + 45 WAN nodes, plus a probe tier of 18 single-service VMs (12 probes, 3 untouched controls, 3 spares) joined as an extension. One ant-node service per probe VM so each host-level iptables fault is scoped to exactly one node. Every claim below is a log line with its fields.

Case 1 — a stale snapshot is rejected as a whole

saved_at_epoch_secs edited to now − 3900 (TTL 3600 + 300 s margin) with only that field changed; the pre/post files were diffed with the field deleted from both and asserted identical, so the rejection is attributable to age rather than malformed input. Reachability untouched.

	P01	P02	P03
WARN Close group cache is stale…	cache_age_secs=3945, max_age_secs=3600	3935 / 3600	3924 / 3600
Bootstrap reachability summary	cache_dial_candidates=0, configured 5/5, outbound_reachable=true	same	same

Zero Loaded N peers… lines, zero dials with bootstrap_source=cache, zero Failed to load close group cache, zero non-finite-trust filter lines — on all three probes. Neither cached trust nor Priority-0 peers were used after the stale warning. Each probe then rewrote the snapshot with a fresh save_reason=post_bootstrap save whose saved_at_epoch_secs matches its log line to the second.

Case 2 — fresh-but-dead cached peers fall back to configured

Outbound UDP dropped to only the cached endpoints (38/38/39 of them), leaving configured endpoints reachable.

	P04	P05	P06
Loaded N peers from close group cache	cache_age_secs=260	256	251
Stale warnings	0	0	0
Bootstrap dial failed bootstrap_source=cache	38, all connect_error	38	39
Bootstrap reachability summary	cache 20/0, configured 5/5, true	cache 20/0, configured 5 cand / 3 ok, true	cache 20/0, configured 5/3, true
Healthy, single start	yes	yes	yes

The full chain holds on every probe: fresh cache accepted → every cache dial fails, correctly source-labelled → configured succeeds in the same start, no second restart.

Case 3 — total unreachability is reported explicitly

Full non-loopback UDP partition, both directions, held 18 minutes. The inbound half matters: with zero outbound successes the node adopts inbound peers after 5 s, which would have rescued bootstrap and voided the window.

All three probes: cache_dial_successes=0, configured_dial_successes=0, outbound_bootstrap_successes=0, outbound_reachable=false; 37 cache + 5 configured dial failures with zero outcome=ok; WARN Failed to connect to any bootstrap peers; zero inbound-rescue lines; process stayed up throughout. After removing the rules and restarting: configured 5/5, outbound_reachable=true, healthy. cache_dial_candidates=0 on recovery, since the partition run's shutdown save wrote an empty close group.

The reachability summary is unambiguous and is the signal that correctly identifies this state — see the note below.

Case 5 — periodic refresh survives SIGKILL

Probes untouched from deployment (read-only cache copies only).

	P10	P11	P12
Periodic save cadence	600 s, 600 s	600 s, 600 s	600 s, 600 s
saved_at_epoch_secs strictly increasing	yes	yes	yes
Cache SHA-256 unchanged across SIGKILL	yes	yes	yes
save_reason=shutdown between last periodic and the kill	0	0	0
Loaded saved_at_epoch_secs on restart	1786323685	1786323682	1786323688
== the last periodic value	yes	yes	yes
cache_age_secs	170	174	171
cache_dial_candidates / successes	20 / 20	20 / 18	20 / 18
Periodic saves resumed	+600 s	+600 s	+600 s

systemctl kill was verified from systemd itself (Result=signal, ExecMainStatus=9, code=killed, status=9/KILL, NRestarts=0). The unchanged content hash is what proves no shutdown save ran — without it a graceful stop and a kill look identical from the cache alone. The restart reused the latest periodic snapshot and its peers were genuinely live (18–20 of 20 cache dials succeeded).

---

One correction to a stated assumption

The test plan assumed "the dialler dedups addresses: an endpoint dialled as cache is not re-dialled as configured". The data contradicts this. On P05 and P06 two endpoints were both cached and configured; each appears as cache/connect_error and again as configured/connect_error, with configured_dial_candidates=5 (not 3) and 2 failures + 3 successes = 5. Each blocked endpoint was dialled twice, once per source.

Not a defect — the counters stay internally consistent and the case is still scoreable — but worth knowing, because it means an endpoint present in both sets costs two dial attempts on every bootstrap. If dedup was intended, it isn't happening.

Practical consequence for anyone reproducing this: with only 2 configured bootstrap endpoints, an earlier attempt found both already in the probe's cached close group at :10000 on 2 of 3 probes, so blocking the cache also blocked every configured path and the case degenerated into case 3. Deploying 5 bootstraps made it comfortably reproducible.

Note on interpreting the replication diagnostics

Under total partition (case 3), the Replication bootstrap state snapshot from ant-node#198 reports snapshot_state=healthy at its first tick and then stops. Detail and evidence are on WithAutonomi/ant-node#198 — flagged there rather than here, since it concerns the diagnostic rather than this PR's cache hygiene. It reinforces that the Bootstrap reachability summary added here is the authoritative signal for reachability, and it behaved correctly in every case.

Coverage

All four cases pass on all 3 probes each. Controls healthy at six boundaries with zero restarts; iptables -S archived clean on all 12 probes; backbone transfers 101 uploads ok / 1 failed, 120 downloads ok / 0 failed. The single upload failure occurred inside the case-3 window and is the expected consequence of 3 of 68 nodes being unreachable (23/30 chunks stored); controls stayed healthy, so it is recorded as a deviation rather than fault leakage.

Full per-case evidence: V2-895, V2-896, V2-897, V2-899.