feat(bootstrap): persist and restore the whole routing table across a restart - #152
Conversation
… restart A restart currently keeps the k peers nearest to self and nothing else, then rebuilds the rest of the routing table at two k-buckets per 7.5 to 12.5 minutes — a cadence this crate's own comment describes as approximately once-per-day full-table maintenance. Until that completes the node cannot name anyone closer to a distant key than itself, so every consumer that asks "am I among the w closest to this key" gets yes for most of the keyspace. A close group cannot stand in for a routing table, and the reason is combinatorial rather than a matter of degree. For a key K, write c for the number of leading bits K shares with self. Any peer p sharing exactly c leading bits with self agrees with self on bits [0,c) and differs at bit c; K does the same; so p agrees with K at bit c and is strictly closer to K than self is. Every peer in bucket c therefore answers the question, and a node holding w of them can always answer it correctly — while a node whose bucket c is empty cannot, however many neighbours it has. Preserving peers across every bucket is what makes a restored table answer as the original did, at every width. Simulated on an 891-node network at the two widths a storage consumer uses, against a correct share of 1.01% and 2.24%: a converged table claims 1.05% and 2.01%; today's nearest-20 cache claims 41.4% and 95.6%; a full-table snapshot claims 1.05% and 2.01%. Keeping only nine peers per bucket fixes the narrow width but leaves the wide one at 12.3%, which is why the snapshot carries the whole capped table — about 129 entries and 25 KB for that network. The snapshot is written to its own file so an older binary cannot mistake it for a close group, and the close-group cache continues to be written alongside so a downgrade keeps working. It is accepted only when its schema version, owning node, network fingerprint, integrity checksum and age all check out; the owner binding is load-bearing rather than hygiene, because bucket indices are relative to the owner, so another node's snapshot describes a different partition of the id space. It records no trust scores. The close-group cache imports trust before dialling, which is defensible for k vetted neighbours and is not defensible for a whole table: a file on disk must not decide that hundreds of unverified peers start above neutral. Restored peers are dial candidates only, verified through the ordinary identity-checked path before they can enter the routing table, because routing-table membership carries authority for callers above this crate. Restoration is on by default with a kill switch, dials at bounded concurrency, and abandons the remainder once a wall-clock budget expires, so a snapshot full of departed peers costs a bounded startup delay and then behaves exactly like having had no snapshot at all.
ADR-017 states why a close-group cache cannot answer responsibility questions after a restart, and what the snapshot does instead. The reasoning is combinatorial: every peer in the bucket a key falls into is strictly closer to that key than self is, so a restored table answers correctly wherever those buckets are populated, and cannot where they are empty. That is why the snapshot carries every bucket rather than the nearest k peers, and why a per-bucket floor of nine is not enough at the wider width a storage consumer uses. Records the acceptance bindings and the reasons they are load-bearing, the dial-candidate contract, the deliberate absence of trust restoration, the bounded dial budget that degrades to today's behaviour, the alternatives that were measured and rejected, and the evidence that does not yet exist.
Review found the restore handed its successes to `bootstrap_from_peers`, which issues a serial FIND_NODE per seed and then serially dials every peer those queries return. The phase bounded its own dials at 16 concurrent under a 20 second budget and then fed ~130 peers into a path with no bound at all, so a restore could extend startup by tens of minutes. The queries were also redundant: a dialled, identity-verified peer is already admitted to the routing table, so this was re-discovering the table just restored. Snapshot successes now count towards reachability only. The budget no longer drops dials in flight. It stops new ones and lets the rest finish under the identity timeout, because cancelling a handshake mid-flight leaves the far side holding a half-open connection. Loading is now bounded: regular files only, 512 KB ceiling checked before the read, and the peer and address caps re-applied after parsing so the bound holds for a file this process did not write. Restoration is once per process and skipped for clients. A re-bootstrap would have replayed it to rebuild the table the node already had, and a client never asks whether it is responsible for a key, so it keeps its existing six-peer startup bound. The post-bootstrap save no longer writes the snapshot. At that moment the table holds only what the restore re-dialled, so writing it replaced a complete snapshot with a partial one and every restart shrank it further. The periodic and shutdown saves still write it. Cut, on the same review: the network fingerprint, which hashed mutable bootstrap-address spellings rather than a stable network identity and would have invalidated every snapshot on a seed rotation; the payload checksum, which adds no authenticity for a locally written file that JSON parsing already accepts or rejects whole; and the config kill switch, which was new public API for a behaviour that rolls back by shipping the previous version. The snapshot type is now crate-private. The one-hour close-group cache age no longer governs the snapshot. That bound exists for trust scores on k neighbours; bucket coverage does not rot on the same timescale, and a longer maintenance window must not cost a node its table. Seven days, with the same future-skew rejection. Adds restore-path tests for candidate selection (self excluded, already-queued addresses skipped, undialable addresses dropped) and for the load bounds.
Second review round. Loading opened the file by path, checked its metadata, then read the path again. Those are two different files if anything replaces it in between, so neither the regular-file check nor the size ceiling was binding. It now opens once, checks that handle, and reads through a hard ceiling. Saving keyed off a human-readable save-reason string to decide whether the snapshot was written. The periodic and shutdown paths now call it explicitly and the string comparison is gone. A save is also skipped until the bootstrap phase that restores the previous snapshot has finished, and skipped if that restore recovered less than half of what it tried: a node that could not reach the network must not overwrite a full snapshot with the little it managed to re-dial. Without this a node stopped inside the restore window shrank its own snapshot a little further on every cycle. The dial budget stopped new peers but said nothing about how many addresses each one could try, so the phase's real bound was the budget plus up to eight attempts. A snapshot peer is now tried at no more than two addresses. Candidate selection deduplicated by socket address only, so a peer repeated in the file, or reachable at two addresses, could be dialled twice. It now deduplicates by peer id as well. ADR corrected on three points that overstated the code: the flat 1,024-peer ceiling was not mentioned, the phase bound was described as strictly 20 seconds, and "answers as its converged self did at every width" was stated as fact when it is the simulated expectation and no live test exists yet.
Third review round. The readiness flag from the previous round could stick in either direction: a restore that recovered too little switched saving off for the life of the process, so the file went stale and the next restart was a cold one, and a later re-bootstrap could switch it back on because it reports no candidates when it skips the replay. Replaced with the invariant that was wanted in the first place: a save is skipped while the live table holds fewer peers than the snapshot restored at startup. It needs no lifecycle state, cannot be cleared by a re-bootstrap, and self-heals, because ordinary discovery refills the table past the floor and saving resumes on its own. Also corrects the ADR, which claimed the dial phase never exceeds its budget while the paragraph above it says in-flight attempts finish.
Fourth review round, against the floor introduced in the third. The floor started at zero and was only set once the restore step ran, so a node stopped before that step — or one that failed to start — could write its empty or half-dialled table over a complete snapshot. It is now unresolved until the restore step decides it, and a save before that point does nothing. A client never restores, so it never resolves a floor and never writes a snapshot it would not read back. A permanent floor was also unreachable if the network genuinely shrank, if a bucket lost its last peer, or if k_value dropped: every later save was skipped and the file aged out, so the next restart cold-started anyway. The floor now applies for an hour after it is decided, which is the window it exists to protect. After that the live table is the node's best knowledge and the file keeps being refreshed. An empty table is never written, whatever the floor says. Declined from the same round: comparing per-bucket occupancy rather than a peer count before replacing the file. It defends a case the count misses — an equally sized table whose buckets shifted — at the cost of bucket arithmetic on the save path, and the failure it prevents costs one restart with a slightly worse table. The ADR now claims only what the count actually guarantees.
Both stores were relaxed and the count went first, so a save running concurrently with the restore step could see a resolved floor alongside the zero timestamp it was published with, read that as an expired floor, and write a partial table over a good snapshot. Timestamp first, count released after it, and the save acquires the count before reading the timestamp.
dirvine
left a comment
dirvine
left a comment
There was a problem hiding this comment.
Requesting changes for one startup-blocking special-file case.
[Medium] Reject special files before a potentially blocking open — src/bootstrap/routing_snapshot.rs:223
load_from_dir calls tokio::fs::File::open before checking metadata.is_file(). On Unix, opening a FIFO read-only blocks until a writer appears, so a FIFO (or symlink to one) at the fixed routing_snapshot.json path can stall node bootstrap indefinitely; the intended rejection at line 232 is never reached. I reproduced the underlying failure mode locally with mkfifo and a read-only open.
Please open with non-blocking/no-follow semantics on Unix and validate the opened handle with fstat (with an appropriate portable fallback), then add a regression test covering FIFO/special-file rejection without blocking. A path-level precheck alone still leaves a TOCTOU window.
Verification at exact head af7d75b427202192c1413e6c34e3ccf6abbd0226:
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo test --lib— 523 passedcargo test --all-features— passed- focused routing-snapshot and dial-set tests — passed
- regular GitHub CI checks are green;
claude-reviewis failing because the GitHub App is unavailable/returns 401, not because of this code
Non-blocking follow-up: an end-to-end restart test covering save → restore → authenticated DHT admission and the one-hour no-shrink floor would materially improve rollout confidence.
Review-panel caveat: the delegated reviewers and local independent model all timed out, so I am not claiming panel consensus. The blocking finding above is independently reproduced and source-verified.
…ocking A FIFO placed at routing_snapshot.json stalled bootstrap indefinitely: the read-only open blocked until a writer appeared, before the regular-file check on the handle could reject it. Open with O_NONBLOCK and O_NOFOLLOW on Unix, so the open returns immediately, a symlink is refused outright, and the existing fstat-based rejection is actually reached. A path-level precheck was ruled out as TOCTOU-prone; the handle stays the single source of truth. Adds libc as a Unix-only dependency (already in the tree transitively) for the open flags and the mkfifo regression test. Requested in review on #152. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the blocking review finding in dac96df (pushed on behalf of the author, who is away). Fix for the FIFO/special-file stall at
Regression tests (both
Dependency note: this adds Verification at 🤖 Generated with Claude Code |
dirvine
left a comment
dirvine
left a comment
There was a problem hiding this comment.
Re-reviewed exact head dac96dfe26e531b1f9b4cd96affcac773b851de0 and approving.
The previous startup blocker is resolved correctly:
- Unix opens now use
O_NONBLOCK | O_NOFOLLOW, so a FIFO cannot block before validation and symlinks are refused. - Regular-file and size validation remains on the same opened handle, avoiding a path-level TOCTOU check.
- Regression tests cover prompt FIFO rejection and symlink refusal.
Verification at this head:
cargo fmt --all -- --check— passedcargo clippy --all-targets --all-features -- -D warnings— passedcargo test --lib bootstrap::routing_snapshot::tests -- --nocapture— 10 passedcargo test --lib— 525 passed- Linux, macOS and Windows builds/tests and the remaining regular CI checks are green
The failing claude-review check is the known GitHub App/401 infrastructure failure, not a code failure.
The independent review panel has not returned a usable consensus yet, so I am not presenting one. This approval is based on direct source inspection, reproduction-specific regression coverage, local checks, and the completed CI matrix.
|
Independent panel follow-up: all three reviewers agree the FIFO/special-file fix is technically sound and satisfies the requested change. Two found no blocker. One raised deliberate |
Testnet results — 990 nodes, matched treatment/control cohorts, 2026-08-16Ran this PR on a 990-service testnet (66 VMs across OVH / OVH-3AZ / Vultr / DigitalOcean, 30% NAT-simulated, 7 bootstraps, 10 uploaders + 2 downloaders, ~3.5h). Full test plan and per-criterion verdicts in V2-993; this is the summary for the PR. Tested Design: no separate reference network. 50 treatment services restarted with The restore path works
Snapshot convergence during the 1h soak, sampled across 300 services: median 90 peers at T+15 → 130 at T+30 → 131 at T+40, then flat. That lands right on the ~120–130 your 891-node simulation predicted, and the worst individual service was 106. The NAT caveat we wrote into the plan didn't materialise. We expected NAT'd nodes to convert fewer candidates because relay reservations die with the restart. Measured: NAT median 98 vs public 95, ranges fully overlapping (88–107 vs 87–110). No penalty. Adversarial handling: 14/14Every condition rejected with its own distinct reason, node active and normally bootstrapped afterwards, and no restore logged in any case — no partial restore from a hostile file:
For the oversized case we padded with trailing whitespace so the file stayed valid JSON — truncating to oversize trips the parser too, and since both surface as V2-983 signalAll 363 No harmUploads 734/734 → 265/265 → 1545/1546 (pre / during / post restart window); downloads 134/134 → 59/59 → 309/309. One failure in 2545 uploads. Fleet ended at 990/990 services active, 0 failed units, 0 NRestarts, no panics. One result that limits what this run can prove about over-claimThe plan expected control nodes to sit near close-group scale for a long window ("about a day", from bucket refresh at ~2 buckets per 7.5–12.5 min). They didn't:
Over-claim, measured directly as So the mechanism is confirmed and points the right way, but the thin-table window here was ~10 minutes rather than ~a day, and our "≤20% of control over 2 hours" threshold averages a real 10-minute effect across 110 minutes of parity. That's a limitation of the test environment, not evidence against the change — dial conversion was 91% and the table was fully restored in ~18 seconds, which rules out the failure mode we'd pre-registered ("restored candidates not converting into table entries at scale"). Worth flagging because production showed restarted services ranking 27th and 24th of 806, at a comparable node count, so node count alone doesn't explain why control recovers so much faster here — we haven't identified the cause, and quantifying the over-claim reduction properly would need a network that sustains the thin state for longer. One implementation detail worth knowingRestore logs carry Nothing in the run argues against merging. |
3 participants
A restart keeps the
kpeers nearest to self and nothing else. The rest of the routing table is rebuilt by periodic bucket refresh at two buckets per 7.5 to 12.5 minutes, which this crate's own comment calls approximately once-per-day full-table maintenance. Until that finishes the node cannot name anyone closer to a distant key than itself, so every consumer asking "am I among thewclosest to this key" gets yes for most of the keyspace. It then fetches data it should not hold, accepts data pushed at it, and keeps telling itself it is still responsible for records it should be dropping.This saves the whole routing table when state is saved, and restores it as dial candidates at startup. Nothing else.
Why every bucket rather than the nearest
k: for a keyKwithc = CPL(self, K), any peer in bucketcagrees withKat bitcand is therefore strictly closer toKthan self is. A node holdingwof them always answers correctly; a node whose bucketcis empty cannot, however many neighbours it has.Simulated on an 891-node network against correct shares of 1.01% (width 9) and 2.24% (width 20):
Design points worth review attention:
close_group_cache.json, which is still written, so a downgrade behaves exactly as today.FIND_NODEper peer to rediscover the table just restored, then serially dial the results.kneighbours.NodeMode::Client, which keeps its six-peer startup bound.Not in scope: speeding up bucket refresh for a brand-new node with no snapshot, and the consumer-side prune clock, which is a separate issue in
ant-node.Deliberately cut after review, in the interest of keeping this small: a network fingerprint (hashed mutable bootstrap-address spellings, would have invalidated every snapshot on a seed rotation), a payload checksum (no authenticity for a locally written file that JSON parsing already accepts or rejects whole), and a config kill switch (new public API for something that rolls back by shipping the previous version).
Linear issue
https://linear.app/autonominetwork/issue/V2-883/populate-the-routing-table-before-evaluating-storage-responsibility
Risk tier
Proposed T3 because it changes what a node's routing table contains at startup, and every storage-responsibility decision above this crate reads that table. Happy to be moved to T2 at review if that reading is too conservative.
Compatibility
routing_snapshot.jsonbesideclose_group_cache.jsonin the configured cache directory. Older binaries ignore the new file and keep reading the close-group cache, which is still written. No network storage format is touched.NodeConfigfield is added.Semver impact
Test evidence
cargo fmt --all -- --check: clean.cargo clippy --all-targets --all-features -- -D warnings -D clippy::unwrap_used -D clippy::expect_used: clean.cargo test --lib: 523 passed, 0 failed.fix(bootstrap)commits on this branch.New dependency
libc(Unix-only, already in the dependency tree transitively) — added during review for the non-blocking/no-follow snapshot open and itsmkfiforegression test.ADR
ADR-017: Persist the Whole Routing Table Across a Restart — added in this PR.
Mitigation / rollback
Ship the previous version, or delete
routing_snapshot.jsonfrom the cache directory. Failure modes already degrade to current behaviour: a rejected, missing, oversized or corrupt snapshot is skipped, and a snapshot full of departed peers costs the dial budget and nothing more. Older binaries ignore the file entirely.