diff --git a/COMPAT.md b/COMPAT.md index 83a43cf86..1fc3b75d8 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -82,6 +82,7 @@ Full `/tx/:txid` JSON still has `vin[]`. Electrum has no outspend-vin surface. | IBD empty-headers EOF | Empty `headers` to **our locator** + idle path latches `headers_done` even if a peer advertises a taller less-work height / junk `version.start_height`. A proven header walk at the tip that every taller peer failed to extend also ends IBD | Core header sync follows most-work; advertised `start_height` is not a remaining header count | | BIP94 | Not implemented. There is no testnet4 chain and no `-test=bip94`. A retarget multiplies the parent, and a difficulty-boundary timestamp only has to be after the median of ancestor timestamps. This does not fork mainnet, testnet3, signet, or default regtest | On testnet4, and on regtest only with `-test=bip94`, a retarget multiplies the first block of the period (that parent may be a min-difficulty exception) and a difficulty-boundary timestamp more than 600 seconds before its parent is `time-timewarp-attack` | | Milestone script skip | A height-only `--milestone HEIGHT` skips scripts at and below that height with no anchor hash, including when the header lookup is empty. The default anchor skips once two height-to-hash lookups match: this block, and the anchor hash at the anchor height. It does not walk parents, does not read `min_work_be`, and does not require two weeks of burial. Prevouts, double spends, maturity, and fees still run. An output whose script starts with `OP_RETURN`, or is longer than 10,000 bytes, is still not a coin under the skip | Core runs scripts unless the assumevalid block is an ancestor, the best header is on that chain, chain work meets `nMinimumChainWork`, and the proof-equivalent time from the assumevalid block to the best header is more than two weeks | +| Invalid marks across restart | A block that failed connect on a consensus rule stays invalid across restart (`store/block.invalid`, the newest 4096). An operator `invalidateblock` does not: it lasts until the process exits | Both live in the block index (`BLOCK_FAILED_VALID`) and survive restart | | Equal-work fork choice | A connected tip is replaced only by strictly greater chain work, or by `preciousblock` at equal work. After `invalidateblock`, equal work prefers precious, then the earlier held tip. Header ranking before either chain is connected also stays with the chain already chosen when work is equal ([`docs/architecture.md`](./docs/architecture.md) layer 1, “equal work keeps the first seen”). A block that was stored earlier but was not yet connectable does not take the tip later | When work is equal, Core prefers the block seen first (`nSequenceId`). A block stored earlier but not yet connectable becomes the tip once it can be connected, even if a later equal-work block is already the tip. Neither rule makes a block valid on one side and invalid on the other | Compact reconstruct fills short-ids from the live mempool graph, the diff --git a/SCHEMA.md b/SCHEMA.md index 55ee0c465..1f3107dec 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -222,6 +222,7 @@ itself changed. meta # store magic + schema version header.body / header.head # Class A headers + hash index (overflow: header.head.gN) header.adopt # IBD checkpoint list + milestone hash (not a header-chain copy) + block.invalid # blocks that failed connect, kept across restart txout.body / create.loc / create.off / create.loc.ovf # Class A outs (hot loc) seqsigwit.body / seqsigwit.loc / seqsigwit.off / seqsigwit.loc.ovf # Class A inputs+witness (cold loc) seqsigwit.prune # optional: u32 LE pruneheight sidecar (`--prune-seqsigwit`; missing = off) @@ -392,7 +393,19 @@ then milestone hash [u8; 32] (zeros if none) Each checkpoint is 285 bytes. The tail is 364 bytes. Length is exact: `12 + count × 285 + 364`. An older or shorter file does not parse, and the script skip stays off. -Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. +Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. While `block.invalid` lists a block above the saved base, or one whose height is not found within 64 headers of the best chain, a saved candidate above the confirmed tip is not restored: it may build on that block, so the walk starts again from the tip. A listed block at or below the base is on no chain that walk can take. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. + +### `block.invalid` + +Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (synced temp file, rename, directory sync) for each new failure and when `reconsiderblock` clears one. An IBD failure is written here before the retired walk goes to `header.adopt`. While a failure is not on disk, no walk is saved to `header.adopt` and an existing one is removed, and every later mark tries the write again. A restart while the write still fails has no record of that block and validates it again. + +```text +0..8 magic b"rbtcinv1" +8..12 u32 LE count (at most 4096) +repeat block hash [u8; 32] +``` + +Length is exact: `12 + count × 32`. Oldest first. Past 4096 the oldest hash gives way, so the newest failure survives restart. Mutated bodies, a header timestamp too far in the future, store and engine faults, and operator `invalidateblock` are not written. A descendant is invalid through its ancestor: the startup header scan and the IBD resume seed do not walk through a listed hash. --- diff --git a/TESTING.md b/TESTING.md index e7c1c9004..ceb687330 100644 --- a/TESTING.md +++ b/TESTING.md @@ -200,7 +200,7 @@ absolute path. Keep default-tier fixtures small: on tmpfs, test bytes are RAM. | Remining 100-block maturity pads with `confirm_wire_run` | `pad_empty_from` / `build_mature_regtest_with_spend` **once per binary journey** (not once per skinny test) | | Wall-time multi-round microbenches in default suite | Deterministic structure / chunk-load asserts; demote wall arms to `#[ignore]` | -**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer is not banned for that hash. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. +**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). `end_of_ibd_invalid_heavier_fork` 60s wall (180s under llvm-cov), each sync 20s. A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer stays connected and is not asked for that hash again for 30 seconds or until it announces more. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. **Speed / reliability (default suite):** prefer `pad_empty_from` / `build_mature_regtest_with_spend` **once per journey** (tx_relay live hub, Electrum protocol, core_analogs assumevalid+mempool) over remine pads. `pin_compose_multi_pack_timed` keeps functional + layout/covered short-circuit gates (multi-ms floor); sticky vs cold assemble is log-only (not a hard timing assert). Schema-13 wire rebuild must stamp create identity from `txid.body` — zero batch identity is treated as missing (regression covered by the spend reconstruct in `consensus_mature_chain_spend_reconstruct_and_scripthash` + multi-vout confirm scenarios). Coverage vs speed: prefer **one** scenario at the real entry over N micro-opens that only paint lines; when adding coverage for reduce/materialize, use a **tiny** target, not production stream depth. @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. A sixth, on its own hub, because a probe must say what a peer has: it asks after the path top alone; an empty reply there, or one from genesis that stops at the tip (this node's or Core's form), is asked again from the confirmed tip, where an empty reply is a chain that ends there; a reply on the top shows the path below it; an announcement that crosses the probe leaves its ask open; the reply is not a walk answer, and one with a header the walk has not seen gets the walk's own ask; an empty reply to a walk ask is probed again; and a peer with a walk ask out, or one expired and not answered, is neither picked nor asked; an open probe keeps the lane while the walk has nothing to ask and expires. A seventh, on its own hub, because a rewind replaces the confirmed blocks a peer's record took as its chain: the main-loop rewind walks back from a walk candidate with nothing planted above the tip; a peer on the old chain is not asked for the new one, and a peer whose split or announced headers are on the new chain is. An eighth, because a block is asked only of a peer whose chain showed it: a peer whose full reply ended below the block, or one with no record, is not asked while another peer shows it; with nobody showing a block, a peer that left the path is not asked while one that did not is; and a peer whose record stops short of the next blocks is probed again. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | @@ -425,6 +425,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `end_of_ibd_follow` | P2P (**default**) | Miner plus `--sh-index` syncer. One mature regtest: coinbases pay script A, one spend pays script B. IBD reaches that tip, leaves IBD, and Electrum history matches. The next block tip-follows. Restart does not rewrite the scripthash pack mark; one more block arrives by write-behind. Cancelling IBD once the height is below the miner, then `run_p2p`, finishes the same tip and history. With the syncer caught up, dropping the miner leaves tip follow (not IBD). A partial datadir whose miner is down does not open Electrum and stays short of that tip; the same miner address coming back lets that datadir finish | | `end_of_ibd_sh_interrupt` | P2P (**default**) | Same miner and scripts. The syncer first catches up with `--sh-index` off, then the datadir is frozen after pass 1 (`DONE.keys`, no `DONE.post`). Resume builds the index and Electrum's first answer is the full A/B history; restart does not rewrite the pack mark. A block mined after the freeze is in that history. A copy that already has the block, with `include_hwm` at the new tip and that create appended on the unsealed head, still serves the same history and does not open Electrum early | | `end_of_ibd_work_fork` | P2P (**default**) | Two miners on a regtest with retargeting and min-difficulty. One chain retargets harder and stops shorter. The other forks after that retarget, resets to the pow limit, and grows taller with less work. The syncer IBD-adopts the heavier tip and keeps it across a reopen. A scripted BIP324 peer advertises height 100,000, supplies two consecutive full windows of a lighter fork, then replays the second: the syncer asks to continue after each advancing window but finishes IBD after the replay. It then follows one more heavy block from the miners. This journey exceeds 2s because the replay must reach the protocol’s 2,000-header boundary; 89 harder blocks outweigh both windows without confirming 4,001 easy blocks | +| `end_of_ibd_invalid_heavier_fork` | P2P (**default**) | Three miners on one regtest. The split miner never halves the subsidy, so from height 20 its taller chain is coinbase excess to the syncer while it still carries the most header work. Honest forks at 14 and the third miner forks at the first invalid block. With honest at 17 the syncer keeps the split chain's valid prefix (19) and IBD returns. Once honest reaches 30, a fresh syncer confirms that prefix, retires the split candidate on the invalid block, rewinds 19 → 14, and finishes at honest 30. The syncer that kept the prefix restarts, with both peers and then with honest alone, and finishes at honest 30 without rewinding to chase the stored split headers. A syncer whose only other peer forks at the invalid block finishes on that fork | | `node_run_p2p_short` | Node (**default**) | Product `run_p2p` `--blocks-only` `--connect` to a live seeder (`--max-tip-age` so the 3-block pad is not stale IBD); process `getpeerinfo` / `getconnectioncount` / `getnetworkinfo` / `getnettotals` / `ping` while connected (v2 `manual`, as Core reports `-connect`; handshake `startingheight` equals the seeder tip; `timeoffset` present; `synced_headers`/`synced_blocks` stay `-1` until the peer announces a header hash (empty getheaders at tip does not copy VERSION height); `servicesnames` present; `getnetworkinfo.timeoffset` present); after catch-up `localrelay` / mempool `relay_enabled` stay false and `sendrawtransaction` is not `relay disabled`; Electrum `broadcast` and Esplora `POST /tx` admit decode/consensus errors (not hub-missing / not `relay disabled`); `addconnection inbound` refuses; `disconnectnode` unknown `nodeid` / empty params error then a real addr drops that row from the next `getpeerinfo`, the seeder sees it go, and a second `disconnectnode` of that `nodeid` (or of an address never connected) is `-29`; the `--connect` peer is redialled as `manual` under a new id and the seeder sees the inbound; `addnode onetry` and `add` of that address succeed and open no second session; seeder inbound `tx` then ends that session. Exit via `stop`. `max_run_secs=0` is `node_listen_and_exit`. Mock-clock `timeoffset` is the median (odd N, even N upper-middle, inbound-only 0, peer clock behind). A connecting dummy reports `-1`. Header-only and connected peers report `synced_blocks` differently. A query without a chain errors. `pingwait`, `NETWORK_LIMITED`, and `noban` are on the peer row | Removed (covered by the rows above): `confirm_cross_block_prevout_without_tx_head`, diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md new file mode 100644 index 000000000..3fbf78d27 --- /dev/null +++ b/changelog.d/ibd-invalid-heavier.md @@ -0,0 +1,35 @@ +Fixed + +- **IBD leaves a header chain whose block fails consensus.** When the most + work header chain held a consensus-invalid block, IBD rejected the block + but kept that chain as its candidate. It never downloaded a lighter valid + chain another peer served, and stayed below the invalid block. The + candidate now falls back to the invalid block's parent. Headers that build + on the invalid block are refused, and a heavier valid chain from any peer + takes over, rewinding confirmed blocks of the dead chain when it forks + below them. +- **IBD asks the header peer for a block no other peer has.** The peer that + served the headers was never asked for blocks while another peer was + connected, and a peer that answered `notfound` was asked again for the + same block. A heavier chain only the header peer had stalled IBD. A peer + is not asked again for a block it answered `notfound` for 30 seconds or + until it announces more, or for a block above where its announced headers + left the download path or ended. +- **IBD asks a block peer only for blocks its chain showed.** A peer that + had announced nothing, or whose announced headers ended below a block, + was asked for blocks of any chain, so a Core peer serving a lighter chain + stayed silent on blocks it did not have and was dropped for a 10 minute + cooldown. Each block peer is asked for headers after the download path's + top, and from the confirmed tip when it does not have that top, and again + when its record no longer covers the next blocks. It is asked only for + blocks that record shows. With no peer that shows a block, a peer whose + chain has not left the path is asked first. +- **A block that failed connect stays invalid after restart.** The mark was + in memory only, so a restart adopted the stored header chain through that + block again and rewound a valid tip to chase it. Connect failures are + kept in `store/block.invalid`; `reconsiderblock` clears one. The startup + header scan and the IBD resume seed do not walk through them. +- **IBD does not blacklist a block whose timestamp is too far in the + future.** A block that a clock stepped back left more than 2 hours ahead + waits until the clock catches up, instead of being rejected until IBD + halts. diff --git a/crates/rbitcoin-ibd/src/ibd/assign.rs b/crates/rbitcoin-ibd/src/ibd/assign.rs index 5df539cce..e8239a721 100644 --- a/crates/rbitcoin-ibd/src/ibd/assign.rs +++ b/crates/rbitcoin-ibd/src/ibd/assign.rs @@ -242,10 +242,11 @@ pub fn assign_work_ordered( ) { let t0 = Instant::now(); let mut issued = 0u64; - let alive = super::header_walk::peers_for_blocks(st); + let alive: Vec = st.slots.iter().filter(|s| s.alive).map(|s| s.id).collect(); if alive.is_empty() { return; } + super::header_walk::settle_peer_splits(st, hub); let (bq_bytes, bq_count) = hub.query.block_queue_stats(); let bq_stop = cfg.assign_stop_bytes; @@ -816,7 +817,7 @@ pub fn issue_batch( let batch: Vec = batch .into_iter() .filter(|h| { - if st.slots[idx].in_flight.contains(h) { + if st.slots[idx].in_flight.contains(h) || !super::header_walk::may_serve(st, pid, h) { return false; } if st.inflight.contains_key(h) { @@ -1315,6 +1316,7 @@ pub fn steal_hung_densify( pid != owner && peer_bps(&st.slots, pid) > owner_bps && !st.inflight.get(&h).is_some_and(|r| r.holds(pid)) + && super::header_walk::may_serve(st, pid, &h) }) .collect(); if faster.is_empty() { diff --git a/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs b/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs index 73aa4ad82..ba88da85c 100644 --- a/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs @@ -198,6 +198,9 @@ pub struct ConfirmFeed { pub epoch: AtomicU64, /// Isolate until confirmed tip reaches this height (`u32::MAX` = off). pub force_single_until: AtomicU32, + /// Lookup selects nothing until this IBD mono ms (0: no hold): a block + /// was too far ahead of the clock to connect. + pub hold_until_ms: AtomicU64, } pub struct ConfirmFeedInner { @@ -227,9 +230,21 @@ impl ConfirmFeed { stop: AtomicBool::new(false), epoch: AtomicU64::new(0), force_single_until: AtomicU32::new(u32::MAX), + hold_until_ms: AtomicU64::new(0), } } + /// Lookup waits `for_` before it selects a body again. + pub fn hold_for(&self, for_: Duration) { + let until = super::peer_io::ibd_mono_ms().saturating_add(for_.as_millis() as u64); + self.hold_until_ms.store(until, Ordering::Release); + } + + /// A hold from [`Self::hold_for`] has not passed. + pub fn held(&self) -> bool { + super::peer_io::ibd_mono_ms() < self.hold_until_ms.load(Ordering::Acquire) + } + pub fn epoch(&self) -> u64 { self.epoch.load(Ordering::Acquire) } @@ -375,6 +390,9 @@ impl ConfirmRejectClass { if s.contains("missing retarget first header") { return Self::SoftWire; } + if s.contains("timestamp too far in future") { + return Self::Cascade; + } if s.contains("fk mismatch") || s.contains("plan not committed in order") || s.contains("connect height not tip+1") @@ -415,6 +433,7 @@ impl ConfirmRejectClass { Self::SoftWire } ConsensusError::BadHeader("missing retarget first header") => Self::SoftWire, + ConsensusError::BadHeader("timestamp too far in future") => Self::Cascade, ConsensusError::Internal(m) => Self::from_err_str(m), ConsensusError::BadBlock(_) | ConsensusError::BadTx(_) @@ -897,6 +916,16 @@ pub fn publish_confirm_progress(query: &rbitcoin_query::Query) { publish_confirm_progress_at(query, Instant::now()); } +/// Pause before a block too far ahead of the clock is tried again. +pub const TIME_TOO_NEW_HOLD: Duration = Duration::from_secs(1); + +/// A reject for a block more than 2 hours ahead of the clock: it is valid +/// once the clock catches up. +pub fn is_time_too_new(err: &str) -> bool { + let s = err.to_ascii_lowercase(); + s.contains("timestamp too far in future") || s.contains("time-too-new") +} + #[allow(clippy::too_many_arguments)] // call-site args stay unbundled #[doc(hidden)] pub fn emit_confirm_reject( @@ -911,6 +940,11 @@ pub fn emit_confirm_reject( parent: RejectParent, ) -> Result<(), std::sync::mpsc::SendError> { let class = class.isolate_if_batched(batch_len); + // A clock stepped back can leave a stored header more than 2 hours + // ahead. Try it again after a pause, not at once. + if is_time_too_new(&err) { + feed.hold_for(TIME_TOO_NEW_HOLD); + } if class == ConfirmRejectClass::Cascade && batch_len > 1 { let until = height.saturating_add(batch_len as u32).saturating_sub(1); feed.request_single_block(until); @@ -2819,6 +2853,14 @@ pub fn spawn_confirm_engine( if feed.stopped() { break; } + if feed.held() { + let g = feed.inner.lock().unwrap_or_else(|err| err.into_inner()); + let _ = feed + .cv + .wait_timeout(g, Duration::from_millis(20)) + .unwrap_or_else(|err| err.into_inner()); + continue; + } if hub.query.take_disconnect(&mut disco_seen).is_some() { hub.query.set_lookup_taken_hi(None); } diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index e0f67f16b..002839f45 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -325,6 +325,11 @@ pub fn on_headers_batch( fks: &[rbitcoin_primitives::Fk], work: Option, ) -> usize { + // A block in this batch can fail while its headers are written outside + // the lock. The retire already dropped that path. + if super::header_walk::refuse_invalid_batch(st, peer, &headers) { + return 0; + } let accepted = accepted_headers(&headers, fks); let mut added = 0usize; let mut batch_prev: Option<(BlockHash, u32)> = None; @@ -467,6 +472,7 @@ pub fn apply_peer_event( PeerEvent::NotFound { peer, hashes } => apply_notfound(st, peer, hashes), PeerEvent::BlocksInv { peer, hashes } => { if !hashes.is_empty() { + note_announcement(st, peer); super::header_walk::note_block_inv(st, peer); } } @@ -640,7 +646,18 @@ pub fn plan_header_batch( challenger_work_before: st.header_walk.challenger.as_ref().map(|ch| ch.tip.work), outer_len: headers.len(), }; - if mark.outer_len == 0 { + if answer_probe(st, hub, peer, &headers) { + return (mark, HeaderPlan::Done); + } + if mark.outer_len > 0 { + note_announcement(st, peer); + note_peer_headers(st, hub, peer, &headers); + } else { + if st.header_walk.reserved != Some(peer) { + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.forget_probe(); + } + } super::header_walk::settle_walk_peer( st, hub, @@ -665,7 +682,12 @@ pub fn plan_header_reply( peer: usize, headers: Vec, ) -> HeaderPlan { - let ask = super::header_walk::take_header_ask(st, peer); + // A batch that crossed this peer's open probe leaves that ask in flight. + let ask = if st.header_walk.probe_open == Some(peer) { + super::header_walk::HeaderAsk::Unsolicited + } else { + super::header_walk::take_header_ask(st, peer) + }; let mut inflight = super::header_walk::InFlight::capture(st); if headers.is_empty() { inflight.hold(st); @@ -678,6 +700,9 @@ pub fn plan_header_reply( HeaderPlan::Done }; } + if super::header_walk::refuse_invalid_batch(st, peer, &headers) { + return HeaderPlan::Done; + } let class = super::header_walk::classify(st, hub, ask, &headers); let mut headers = headers; let keep = match class { @@ -882,12 +907,116 @@ pub fn apply_block_decode_failed(st: &mut IbdWorkState, peer: usize, hash: Block } #[doc(hidden)] +fn note_peer_headers( + st: &mut IbdWorkState, + hub: &ChainHub, + peer: usize, + headers: &[bitcoin::block::Header], +) { + let Some(height) = headers + .first() + .and_then(|h| parent_height(&st.hash_height, hub, h.prev_blockhash)) + else { + return; + }; + let tip = hub.tip_height().unwrap_or(0); + let path = &st.height_to_hash; + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.note_headers( + path, + |ht| super::peer_chain::confirmed_at(hub, ht), + tip, + height, + headers, + ); + } +} + +/// The reply to this peer's open probe, read into its chain record and kept +/// from the walk. A probe after the path top is answered empty, on that top, +/// or from genesis when the peer does not have it. A probe from the +/// confirmed chain is answered empty or on a confirmed block. A batch of +/// another shape is an announcement that crossed the probe. True when the +/// reply was consumed. +fn answer_probe( + st: &mut IbdWorkState, + hub: &ChainHub, + peer: usize, + headers: &[bitcoin::block::Header], +) -> bool { + if st.header_walk.probe_open != Some(peer) { + return false; + } + let tip = hub.tip_height().unwrap_or(0); + let top = st.header_walk.probe_top; + if let Some(first) = headers.first() { + let prev = first.prev_blockhash; + let confirmed_h = hub + .query + .height_of_hash(&prev.to_byte_array()) + .ok() + .flatten() + .map(|h| h.0) + .filter(|&h| h <= tip && super::peer_chain::confirmed_at(hub, h) == Some(prev)); + let shaped = match top { + Some((_, hash)) => { + prev == hash || prev.to_byte_array() == [0u8; 32] || confirmed_h == Some(0) + } + None => confirmed_h.is_some(), + }; + if !shaped { + return false; + } + } + st.header_walk.probe_open = None; + st.header_walk.probe_top = None; + let _ = super::header_walk::take_header_ask(st, peer); + let past_tip = headers.first().is_some_and(|first| { + parent_height(&st.hash_height, hub, first.prev_blockhash) + .is_some_and(|h| h.saturating_add(headers.len() as u32) > tip.saturating_add(1)) + }); + if !headers.is_empty() { + note_announcement(st, peer); + note_peer_headers(st, hub, peer, headers); + // A header this node has not seen: the peer's chain leaves this one + // or runs past the path. The walk asks it, as after a block inv. + let last = headers[headers.len() - 1].block_hash(); + let seen = st.hash_height.contains_key(&last) + || matches!(hub.query.height_of_hash(&last.to_byte_array()), Ok(Some(_))); + if !seen { + super::header_walk::note_block_inv(st, peer); + } + } + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + match top { + Some((height, hash)) if headers.first().map(|h| h.prev_blockhash) == Some(hash) => { + s.chain.note_reaches(height, hash); + } + // A reply from genesis that goes past the tip says what a probe + // from the tip would. Empty is also a peer that sends nothing + // below the minimum work. + Some(_) if past_tip => {} + Some(_) => s.chain.note_lacks_top(), + None if headers.is_empty() => s.chain.note_ends_at(tip), + None => {} + } + } + true +} + +fn note_announcement(st: &mut IbdWorkState, peer: usize) { + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.note_announcement(); + } +} + pub fn apply_notfound(st: &mut IbdWorkState, peer: usize, hashes: Vec) { note_block_progress(&mut st.slots, peer); let mut freed = Vec::new(); if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { for h in &hashes { s.track_remove(h); + s.chain.note_notfound(*h, super::peer_io::ibd_mono_ms()); let empty = st .inflight .get_mut(h) @@ -1333,6 +1462,17 @@ pub fn apply_cascade_reject( // Leave the body queue: the plan was stale, and the confirm thread that // rejected offered any body it held back. clear_hash_inflight(&mut st.slots, &mut st.inflight, hash); + // A block too far ahead of the clock is not a stale plan: the engine + // waits before it tries again, and it connects once the clock catches up. + if super::confirm::is_time_too_new(err) { + note_confirm_stuck(st); + static N: AtomicU32 = AtomicU32::new(0); + let n = N.fetch_add(1, Ordering::Relaxed) + 1; + if n <= 3 || n.is_multiple_of(60) { + warn!("ibd: confirm waits @{height} {hash}: {err} (n={n})"); + } + return; + } const CASCADE_HALT_AFTER: u8 = 3; let tip = hub .and_then(|h| h.tip_hash()) @@ -1405,6 +1545,13 @@ pub fn apply_consensus_invalid_reject( } st.body.mark_rejected(hash); st.reorg.invalid.mark(hash.to_byte_array()); + if let Some(h) = hub { + // The failure goes to disk before the retired walk does: a walk saved + // with its failure lost would resume onto the invalid chain, and + // `save_adopt` saves none while a mark is not on disk. + let _ = h.note_failed_connect(hash); + super::header_walk::retire_invalid(st, h, hash, height); + } if st.height_to_hash.get(&height) == Some(&hash) { st.height_to_hash.remove(&height); } diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index a03cb7098..b831acfa4 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -80,6 +80,11 @@ pub struct HeaderWalk { pub origin: bool, /// Look-ahead tips that every peer left empty before the work floor. pub dead_ends: HashSet, + /// The last header of each refused batch that built on a rejected block. + /// A batch that builds on one is refused too. + pub invalid_tips: HashSet, + /// `invalid_tips` in insertion order, so the cap drops the oldest. + pub invalid_tip_order: std::collections::VecDeque, /// Peers that have answered empty at the current tip. pub emptied: HashSet, /// Checkpoint work has reached the milestone floor. @@ -102,6 +107,15 @@ pub struct HeaderWalk { pub challenge_external: bool, /// Challenge ask already sent. Judged when that peer replies or times out. pub challenge_open: Option, + /// Block peer owed one chain probe: a headers ask from the confirmed tip. + /// Not a walk ask. Not written to `header.adopt`. + pub probe: Option, + /// Probe ask already sent. Its reply is read into that peer's chain + /// record, not by the walk. + pub probe_open: Option, + /// The path top the open probe asked headers after; none when it asked + /// from the confirmed tip. + pub probe_top: Option<(u32, BlockHash)>, /// The one in-flight headers ask. Not written to `header.adopt`. pub walk: Lane, /// Expired asks. Not written to `header.adopt`. @@ -135,6 +149,9 @@ pub const LOOKAHEAD_ASK: Duration = Duration::from_secs(5); pub const HEADER_MISS_DISCONNECT: u8 = 2; #[doc(hidden)] pub const DEAD_END_CAP: usize = 1024; +/// Past the cap the oldest refused tip gives way, so the newest is kept: the +/// next batch of a long invalid chain links to it. +const INVALID_TIP_CAP: usize = 1024; impl HeaderWalk { pub fn tip_height(&self) -> u32 { @@ -540,6 +557,55 @@ pub fn note_block_inv(st: &mut IbdWorkState, peer: usize) { } } +/// Heights above the confirmed tip a peer's record should decide before it +/// is probed again. +const PROBE_AHEAD: u32 = 4_096; + +/// Ask one block peer for headers, so block getdata follows what it holds: +/// one that has not announced a chain (Core starts header sync with each +/// peer), or one whose record no longer decides the next blocks. One probe +/// at a time, never the reserved header peer, and not a peer with a walk ask +/// open. +pub fn probe_unprobed_peer(st: &mut IbdWorkState, hub: &ChainHub) { + use super::peer_io::PeerSlot; + let walk = &st.header_walk; + // A record only chooses among peers. With one live peer it would only + // take the header lane, and its reply would not reach the walk. + if walk.probe.is_some() + || walk.probe_open.is_some() + || st.slots.iter().filter(|s| s.alive).count() < 2 + { + return; + } + let tip = hub.tip_height().unwrap_or(0); + let ahead = tip.saturating_add(PROBE_AHEAD).min(st.max_ordered_height); + let now = Instant::now(); + let due = |s: &PeerSlot| { + !s.chain.probed() || (ahead > tip && s.chain.needs_refresh(&st.height_to_hash, ahead, now)) + }; + let Some(slot) = st + .slots + .iter() + .find(|s| s.alive && !walk_busy(walk, s.id) && due(s)) + else { + return; + }; + st.header_walk.probe = Some(slot.id); +} + +/// The header peer, or a peer with a walk ask open or expired and not yet +/// answered, whose late reply could be read as a probe's. +fn walk_busy(walk: &HeaderWalk, peer: usize) -> bool { + [ + walk.reserved, + walk.challenge_open, + walk.walk.peer, + walk.walk.prev_peer, + ] + .contains(&Some(peer)) + || walk.header_misses.contains_key(&peer) +} + /// A follow-session block inv has no IBD slot. Poke one live header peer /// the same way [`note_block_inv`] does. Returns false when nobody is live /// yet, so the caller keeps the flag. @@ -575,6 +641,13 @@ pub fn forget_walk_peer(st: &mut IbdWorkState, peer: usize) { st.header_walk.challenge_open = None; st.header_walk.challenge_external = false; } + if st.header_walk.probe == Some(peer) { + st.header_walk.probe = None; + } + if st.header_walk.probe_open == Some(peer) { + st.header_walk.probe_open = None; + st.header_walk.probe_top = None; + } } /// Chainwork of the candidate, for judging a challenge reply. @@ -612,21 +685,45 @@ pub fn note_reservation( } } -/// Peers that may receive block getdata. The reserved header peer is left -/// out while another peer is connected. A single peer still does both. -pub fn peers_for_blocks(st: &IbdWorkState) -> Vec { - let alive: Vec = st.slots.iter().filter(|s| s.alive).map(|s| s.id).collect(); - let Some(reserved) = st.header_walk.reserved else { - return alive; +/// Whether block `hash` may be asked of `peer`. As in Core, a peer is asked +/// only for a block its chain showed (`PeerChain::shows`), and not for one it +/// answered `notfound`. The reserved header peer is asked only when no other +/// peer shows the block. A single peer still does both. +pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { + use super::peer_io::PeerSlot; + let height = st.hash_height.get(hash).copied(); + let now = super::peer_io::ibd_mono_ms(); + let path = &st.height_to_hash; + let askable = |s: &PeerSlot| s.alive && !s.chain.lacks(hash, now); + let shows = |s: &PeerSlot| askable(s) && height.is_some_and(|ht| s.chain.shows(path, ht)); + let open = + |s: &PeerSlot| askable(s) && !height.is_some_and(|ht| s.chain.left_path_by(path, ht)); + let Some(slot) = st.slots.iter().find(|s| s.id == peer) else { + return false; }; - if alive.len() <= 1 { - return alive; - } - let rest: Vec = alive.iter().copied().filter(|&id| id != reserved).collect(); - if rest.is_empty() { - alive + // With no peer that shows the block, a record can be stale: a peer whose + // chain has not left the path is asked, else any peer that did not answer + // `notfound`, the header peer too. Otherwise the header peer stays on + // headers while another peer can fetch the block. + let tier: &dyn Fn(&PeerSlot) -> bool = if st.slots.iter().any(&shows) { + &shows + } else if st.slots.iter().any(&open) { + &open } else { - rest + return askable(slot); + }; + tier(slot) + && (st.header_walk.reserved != Some(peer) + || !st.slots.iter().any(|s| s.id != peer && tier(s))) +} + +/// Settle each peer's recorded split against this node's confirmed chain once +/// that height has left the path. Runs before block assignment. +pub fn settle_peer_splits(st: &mut IbdWorkState, hub: &ChainHub) { + let path = &st.height_to_hash; + for slot in &mut st.slots { + slot.chain + .settle_split(path, |ht| super::peer_chain::confirmed_at(hub, ht)); } } @@ -649,12 +746,20 @@ pub enum HeaderPoll { Wait, } -/// A block inv is waiting, and the walk lane is not already in flight. +/// A block inv or a probe is waiting, or an open probe is past its window, +/// and the walk lane is not already in flight. /// /// An in-flight ask keeps the challenge until the lane clears. The poll /// must not call `send_getheaders` on every block event during that window. pub fn inv_challenge_due(st: &IbdWorkState) -> bool { - st.header_walk.challenge.is_some() && !lane_fresh(&st.header_walk.walk) + probe_or_challenge_owed(st) && !lane_fresh(&st.header_walk.walk) +} + +/// A challenge or probe waits to be sent, or a probe waits to be answered +/// or expire. Either keeps the lane. +fn probe_or_challenge_owed(st: &IbdWorkState) -> bool { + let walk = &st.header_walk; + walk.challenge.is_some() || walk.probe.is_some() || walk.probe_open.is_some() } /// The queue still has room, or a gap / look-ahead still needs headers. @@ -772,6 +877,137 @@ pub fn note_dead(st: &mut IbdWorkState, hash: BlockHash) { } #[doc(hidden)] +fn note_invalid_tip(st: &mut IbdWorkState, hash: BlockHash) { + let walk = &mut st.header_walk; + if !walk.invalid_tips.insert(hash) { + return; + } + walk.invalid_tip_order.push_back(hash); + if walk.invalid_tip_order.len() > INVALID_TIP_CAP { + if let Some(oldest) = walk.invalid_tip_order.pop_front() { + walk.invalid_tips.remove(&oldest); + } + } +} + +/// Refuse a linked batch that holds or builds on a rejected block, and +/// retire `peer` from the walk. The batch's last header is remembered, so +/// the next batch of that chain is refused too. +pub fn refuse_invalid_batch(st: &mut IbdWorkState, peer: usize, headers: &[Header]) -> bool { + let Some(first) = headers.first() else { + return false; + }; + // An unlinked batch does not build on what it names, so its last + // header is not remembered. The usual checks drop it. + if !pow_linked(headers) { + return false; + } + let rejected = |hash: &BlockHash| { + st.header_walk.invalid_tips.contains(hash) + || st.reorg.invalid.contains(hash.to_byte_array()) + }; + if !rejected(&first.prev_blockhash) && !headers.iter().any(|h| rejected(&h.block_hash())) { + return false; + } + if let Some(last) = headers.last() { + note_invalid_tip(st, last.block_hash()); + } + st.header_walk.walk_quiet.insert(peer); + if st.header_walk.reserved == Some(peer) { + st.header_walk.reserved = None; + } + true +} + +/// The block at `height` failed consensus. When the download path holds it, +/// the path above its parent is dropped and every peer may be asked for +/// headers again. A candidate built on it falls back to that parent. +pub fn retire_invalid(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, height: u32) { + if !st.is_on_path(&hash, height) { + return; + } + let Some(parent) = hub.header_of(&hash).map(|h| h.prev_blockhash) else { + return; + }; + let parent_h = height.saturating_sub(1); + drop_dead_path(st, hub, parent_h); + if !st.header_walk.origin || st.header_walk.tip_height() < height { + return; + } + let tip = cut_back_to(st, hub, parent, parent_h); + if st.header_walk.base_height > parent_h { + st.header_walk.base_hash = tip.hash; + st.header_walk.base_height = parent_h; + st.header_walk.base_work = tip.work.to_be_bytes(); + st.header_walk.base_diff = tip.diff; + } + st.header_walk.tip = tip; + // `header.adopt` restores the tip from the last checkpoint. + if st + .header_walk + .checkpoints + .last() + .is_none_or(|c| c.height < parent_h) + { + let at_parent = checkpoint_at(&st.header_walk.tip); + st.header_walk.checkpoints.push(at_parent); + } + // A challenger's headers are not stored, and its fork can sit below the + // confirmed tip. Peers are asked again from the parent, and a heavier + // chain comes back as headers that are stored and can be rewound onto. + st.header_walk.challenger = None; + rbitcoin_log::warn!( + "ibd: header candidate retired at {hash} height={height}; walk back to {parent} height={parent_h}" + ); + publish_work(st, hub); + save_adopt(st, hub); +} + +/// Forget the download path above `keep`: queued hashes, bodies waiting in +/// the body queue, the confirm plan, and the densify cursor. Every peer may +/// be asked for headers again. +fn drop_dead_path(st: &mut IbdWorkState, hub: &ChainHub, keep: u32) { + st.clear_path_above(keep); + drop_queue_above(st, keep, &HashSet::new()); + for queued in hub.query.block_queue_queued_heights() { + if queued > keep { + let _ = hub.query.block_queue_dequeue_height(queued); + } + } + st.confirm_quiesce = true; + st.max_ready_height = st.max_ready_height.min(keep); + let next = keep.saturating_add(1); + st.densify_scan_lo = st.densify_scan_lo.min(next); + st.densify_ready_through = st.densify_ready_through.min(next); + st.header_walk.reserved = None; + st.header_walk.walk_quiet.clear(); + st.header_walk.emptied.clear(); + clear_off_path(st); +} + +/// A walk tip at stored `hash`, `height` on the candidate. Checkpoints and a +/// milestone above it are dropped. +fn cut_back_to(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, height: u32) -> WalkTip { + let mut tip = WalkTip { + hash: Some(hash), + height, + work: work_at(st, hub, hash).unwrap_or(Work::from_be_bytes([0; 32])), + header: parent_header(st, hub, hash), + times: times_ending_at(st, hub, hash), + diff: diff_at(st, hub, hash), + milestone_hash: st.header_walk.tip.milestone_hash, + milestone_prev: st.header_walk.tip.milestone_prev, + milestone_header: st.header_walk.tip.milestone_header, + }; + forget_milestone_below(st, hub, height); + if height < hub.milestone.height { + tip.milestone_hash = None; + tip.milestone_header = None; + } + st.header_walk.checkpoints.retain(|c| c.height <= height); + tip +} + pub fn rewind_above(st: &mut IbdWorkState, hub: &ChainHub, height: u32) { rewind_to(st, hub, height, None); } @@ -1446,26 +1682,8 @@ pub fn adopt_heavier( return; }; // Capture the fork's period and timestamps before the challenger is dropped. - let mut tip = WalkTip { - hash: Some(prev), - height: fork_height, - work: work_at(st, hub, prev).unwrap_or(Work::from_be_bytes([0; 32])), - header: parent_header(st, hub, prev), - times: times_ending_at(st, hub, prev), - diff: diff_at(st, hub, prev), - milestone_hash: st.header_walk.tip.milestone_hash, - milestone_prev: st.header_walk.tip.milestone_prev, - milestone_header: st.header_walk.tip.milestone_header, - }; + let mut tip = cut_back_to(st, hub, prev, fork_height); st.header_walk.challenger = None; - forget_milestone_below(st, hub, fork_height); - if fork_height < hub.milestone.height { - tip.milestone_hash = None; - tip.milestone_header = None; - } - st.header_walk - .checkpoints - .retain(|c| c.height <= fork_height); let mut link = prev; for hdr in headers { if !push_header(&mut tip, hub, hdr) { @@ -2237,6 +2455,15 @@ pub fn expire_header_ask(st: &mut IbdWorkState) -> Option { *e = e.saturating_add(1); *e }; + if st.header_walk.probe_open == Some(peer) { + st.header_walk.probe_open = None; + st.header_walk.probe_top = None; + if n >= HEADER_MISS_DISCONNECT { + st.header_walk.header_misses.remove(&peer); + punish_header_peer(st, peer); + } + return None; + } if st.header_walk.challenge_open == Some(peer) { st.header_walk.challenge_open = None; st.header_walk.challenge_external = false; @@ -2287,12 +2514,12 @@ pub fn send_getheaders( ) -> Result { ensure_origin(st, hub); super::path::append_stored_headers(st, hub); - if st.header_walk.challenge.is_none() && super::path::walk_successor_indexed(st, hub) { + let owed = probe_or_challenge_owed(st); + if !owed && super::path::walk_successor_indexed(st, hub) { release_lane(&mut st.header_walk.walk); return Ok(false); } - let want = - st.header_walk.challenge.is_some() || wants_lookahead(st) || walk_ahead_of_queue(st, hub); + let want = owed || wants_lookahead(st) || walk_ahead_of_queue(st, hub); if !want { release_lane(&mut st.header_walk.walk); return Ok(false); @@ -2349,6 +2576,45 @@ pub fn arm_header_ask( } st.header_walk.challenge_external = false; } + if let Some(peer) = take_probe(st) { + // After the path top alone: a reply that builds on it shows the + // whole path. A peer that did not show it is asked with the + // confirmed chain only, whose first hash it has is at or below the + // tip, so the reply is what it has above it. + let tip_h = hub.tip_height().unwrap_or(0); + let from_tip = st + .slots + .iter() + .any(|s| s.id == peer && s.chain.probe_from_tip()); + let top = path_top(st, hub).filter(|&(_, h)| h > tip_h && !from_tip); + let sent = match top { + Some((hash, height)) => { + let sent = super::dial::request_headers_from( + &st.slots, + peer, + &mut st.header_req_seq, + vec![hash], + )?; + if sent { + note_lane(st, peer); + } + st.header_walk.probe_top = sent.then_some((height, hash)); + sent + } + None => { + st.header_walk.probe_top = None; + ask_header_peer(st, hub, peer, &[])? + } + }; + if sent { + st.header_walk.probe_open = Some(peer); + let now = Instant::now(); + if let Some(slot) = st.slots.iter_mut().find(|s| s.id == peer) { + slot.chain.note_probe_sent(now, top.is_none()); + } + return Ok(true); + } + } ensure_reserved(st, above, skip); if st.header_walk.reserved.is_none() { // Connect-time start height can sit on the stored tip, so no peer @@ -2450,6 +2716,14 @@ pub fn ensure_reserved(st: &mut IbdWorkState, above: u32, skip: Option) { st.header_walk.reserved = pick_header_peer(st, above, skip); } +/// A probe peer who is still connected and not [`walk_busy`]. A busy peer +/// is dropped unprobed, so it is picked again later. +fn take_probe(st: &mut IbdWorkState) -> Option { + let peer = st.header_walk.probe.take()?; + let alive = st.slots.iter().any(|s| s.alive && s.id == peer); + (alive && !walk_busy(&st.header_walk, peer)).then_some(peer) +} + /// A challenge peer who is still connected and not retired. #[doc(hidden)] pub fn take_challenge(st: &mut IbdWorkState) -> Option { @@ -2747,6 +3021,20 @@ pub const ADOPT_MAGIC: &[u8; 8] = b"rbtchdr1"; pub const DIFF_SNAP_LEN: usize = 4 + 80 + 4 + 4; #[doc(hidden)] +/// Drop `header.adopt`; the next start walks again from the confirmed tip. +pub fn forget_adopt(hub: &ChainHub) { + let path = adopt_path(hub); + match std::fs::remove_file(&path) { + Ok(()) => { + if let Some(dir) = path.parent() { + let _ = rbitcoin_store::file::fsync_parent_dir(dir); + } + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => rbitcoin_log::warn!("ibd: remove {}: {e}", path.display()), + } +} + pub fn adopt_path(hub: &ChainHub) -> std::path::PathBuf { hub.query.store().path().join("header.adopt") } @@ -2829,7 +3117,7 @@ pub fn base_conflicts(st: &IbdWorkState, hub: &ChainHub, walk: &HeaderWalk) -> b pub fn on_confirmed_rewind(st: &mut IbdWorkState, hub: &ChainHub, lca_h: u32) { if st.header_walk.base_hash.is_some() && lca_h < st.header_walk.base_height { st.header_walk = HeaderWalk::default(); - let _ = std::fs::remove_file(adopt_path(hub)); + forget_adopt(hub); return; } if hub.milestone.height == 0 || !st.header_walk.origin { @@ -2867,6 +3155,12 @@ pub fn save_adopt(st: &IbdWorkState, hub: &ChainHub) { if !st.header_walk.origin { return; } + // A walk saved while a failure is not on disk could resume onto the + // invalid chain. No walk is the resume point until the mark is. + if !hub.invalid_marks_on_disk() { + forget_adopt(hub); + return; + } let mut buf = Vec::new(); buf.extend_from_slice(ADOPT_MAGIC); let n = st.header_walk.checkpoints.len() as u32; @@ -2959,12 +3253,35 @@ pub fn retire_adopt_if_confirmed(st: &mut IbdWorkState, hub: &ChainHub) { } hub.query .note_milestone_checkpoint_work(tip_h, work.to_be_bytes()); - let _ = std::fs::remove_file(adopt_path(hub)); + forget_adopt(hub); } /// Read `header.adopt`. A file that does not parse does not restore /// checkpoints. Headers already queued and linked from the confirmed tip are /// still noted on the milestone path. +/// Ancestors walked to find a stored failure's height. +const FAILURE_HEIGHT_WALK: usize = 64; + +/// A stored failure that a walk saved from `base_height` may build on: one +/// above that base, or one whose height is not found within a few headers +/// of the best chain. The chain at or below the base is the confirmed one. +fn failure_above(hub: &ChainHub, base_height: u32) -> bool { + hub.invalid_blocks() + .into_iter() + .any(|hash| failure_height(hub, hash).is_none_or(|h| h > base_height)) +} + +fn failure_height(hub: &ChainHub, hash: BlockHash) -> Option { + let path = + super::reorg::header_hashes_to_best_ancestor_n(hub, hash, FAILURE_HEIGHT_WALK).ok()?; + let join = super::reorg::parent_hash_of(hub, *path.first()?).ok()??; + if !hub.is_connected(&join) { + return None; + } + let join_h = hub.query.height_of_hash(&join.to_byte_array()).ok()??.0; + Some(join_h.saturating_add(path.len() as u32)) +} + pub fn restore_adopt(st: &mut IbdWorkState, hub: &ChainHub) -> bool { retire_adopt_if_confirmed(st, hub); let started = std::time::Instant::now(); @@ -2995,6 +3312,16 @@ pub fn restore_adopt(st: &mut IbdWorkState, hub: &ChainHub) -> bool { renote_stored_path(st, hub); return false; } + // A saved candidate above the confirmed tip may build on a block marked + // invalid after it was saved. The walk starts again from the tip. + if parsed.tip.height > hub.tip_height().unwrap_or(0) && failure_above(hub, parsed.base_height) { + rbitcoin_log::info!( + "ibd: headers resume refused height={}: invalid blocks are stored", + parsed.tip.height + ); + renote_stored_path(st, hub); + return false; + } st.header_walk = parsed; restore_tip_header(st, hub); renote_stored_path(st, hub); @@ -3107,6 +3434,8 @@ pub fn parse_adopt(bytes: &[u8]) -> Option { }, origin: true, dead_ends: HashSet::new(), + invalid_tips: HashSet::new(), + invalid_tip_order: std::collections::VecDeque::new(), emptied: HashSet::new(), proven: false, off_path: std::collections::HashMap::new(), @@ -3117,6 +3446,9 @@ pub fn parse_adopt(bytes: &[u8]) -> Option { challenge: None, challenge_external: false, challenge_open: None, + probe: None, + probe_open: None, + probe_top: None, walk: Lane::default(), header_misses: HashMap::new(), base_diff, diff --git a/crates/rbitcoin-ibd/src/ibd/mod.rs b/crates/rbitcoin-ibd/src/ibd/mod.rs index 4c5327378..022e3d789 100644 --- a/crates/rbitcoin-ibd/src/ibd/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/mod.rs @@ -37,6 +37,8 @@ pub mod header_walk; #[doc(hidden)] pub mod path; #[doc(hidden)] +pub mod peer_chain; +#[doc(hidden)] pub mod peer_io; #[doc(hidden)] pub mod perf_log; @@ -471,6 +473,11 @@ pub async fn ibd_cancellable( let window = cfg.window; let mut st = IbdWorkState::new(initial_slots, hub.tip_hash(), hub.tip_height()); + for hash in hub.invalid_blocks() { + st.reorg + .invalid + .mark(bitcoin::hashes::Hash::to_byte_array(hash)); + } st.addr_cooldown = boot_cooldown; st.addr_strikes = boot_strikes; st.perms = perms; @@ -696,6 +703,7 @@ pub async fn ibd_cancellable( if hub.header_inv_challenge_pending() && header_walk::note_external_block_inv(st) { hub.clear_header_inv_challenge(); } + header_walk::probe_unprobed_peer(st, hub.as_ref()); let challenge_due = header_walk::inv_challenge_due(st); if cadence.headers_due(now_cadence, path_empty, challenge_due) { let live = st.ordered_set.len(); diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs new file mode 100644 index 000000000..4eb6bff69 --- /dev/null +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -0,0 +1,348 @@ +//! What one IBD peer has shown of its chain. +//! +//! Core asks a peer only for blocks on that peer's best known chain +//! (`pindexBestKnownBlock`). A peer whose announced headers leave the +//! download path is not asked for blocks above that point, and a peer that +//! answered `notfound` for a block is not asked for it again for a while. + +use bitcoin::block::Header; +use bitcoin::BlockHash; +use std::collections::{HashMap, VecDeque}; +use std::time::{Duration, Instant}; + +/// `notfound` hashes kept per peer. Past the cap a peer may be asked again. +const LACKS_CAP: usize = 1024; +/// How long a `notfound` keeps a peer off that block: one stall window. A +/// peer that answered `notfound` everywhere is asked again after it. +const LACKS_TTL_MS: u64 = 30_000; + +/// Announced hashes kept per peer: one full `headers` reply. +const ANNOUNCED_CAP: usize = rbitcoin_net::codec::MAX_HEADERS_RESULTS; + +/// How often a peer whose record no longer decides the next blocks is +/// probed again. +const PROBE_REFRESH: Duration = Duration::from_secs(10); + +#[derive(Debug, Default)] +pub struct PeerChain { + /// `notfound` hashes, with the mono ms they were answered. + lacks: HashMap, + /// This peer's announced chain above the confirmed tip, oldest first. + announced: VecDeque, + /// Height of `announced[0]`. + from: u32, + /// Top of this peer's chain: the last reply was shorter than a full + /// `headers` batch and ended here, or a probe found nothing above the + /// confirmed tip. The peer has no block above it. + ends: Option, + /// The peer's header at the first height where its announced chain + /// differed from this node's chain (path or confirmed) when announced. + split: Option<(u32, BlockHash)>, + /// That height has left the path and this node's confirmed block there + /// is a different one. + split_confirmed: bool, + /// A rewind replaced this node's chain from this height, and the record + /// took the old blocks as the peer's chain. The peer is not taken to hold + /// a block at or above it until it answers again. + stale_from: Option, + /// The highest path block this peer showed through a probe that asked + /// for headers after it. It holds every path block at or below it while + /// the path still holds that block there. + reaches: Option<(u32, BlockHash)>, + /// The last probe asked after the path top and the reply did not show + /// it, so the next probe asks from the confirmed tip. + probe_from_tip: bool, + /// The peer announced headers, or was asked for them. + probed: bool, + /// When the last probe was sent. + probed_at: Option, +} + +impl PeerChain { + pub fn note_notfound(&mut self, hash: BlockHash, now_ms: u64) { + if self.lacks.len() >= LACKS_CAP { + self.lacks + .retain(|_, at| now_ms.saturating_sub(*at) < LACKS_TTL_MS); + } + if self.lacks.len() < LACKS_CAP { + self.lacks.insert(hash, now_ms); + } + } + + pub fn lacks(&self, hash: &BlockHash, now_ms: u64) -> bool { + self.lacks + .get(hash) + .is_some_and(|at| now_ms.saturating_sub(*at) < LACKS_TTL_MS) + } + + /// A headers message or a block inv: the peer may have those blocks now. + pub fn note_announcement(&mut self) { + self.lacks.clear(); + } + + pub fn probed(&self) -> bool { + self.probed + } + + /// Whether the next probe asks from the confirmed tip. + pub fn probe_from_tip(&self) -> bool { + self.probe_from_tip + } + + /// A probe was sent `at`, from the tip or after the path top. + pub fn note_probe_sent(&mut self, at: Instant, from_tip: bool) { + self.probed = true; + self.probed_at = Some(at); + if from_tip { + self.probe_from_tip = false; + } + } + + /// The peer's chain holds the path block `hash` at `height`. + pub fn note_reaches(&mut self, height: u32, hash: BlockHash) { + self.reaches = Some((height, hash)); + } + + /// The reply to a probe after the path top did not show it: ask from + /// the confirmed tip next. + pub fn note_lacks_top(&mut self) { + self.probe_from_tip = true; + self.probed = false; + } + + /// The record no longer says whether the peer holds the path block at + /// `height`, and no probe was sent within one refresh. + pub fn needs_refresh(&self, path: &HashMap, height: u32, now: Instant) -> bool { + self.probed + && self + .probed_at + .is_none_or(|at| now.saturating_duration_since(at) >= PROBE_REFRESH) + && !self.shows(path, height) + && !self.left_path_by(path, height) + } + + /// An empty reply to a walk ask: the peer is at one of several locator + /// hashes, the path top or the confirmed tip. Ask it once more from the + /// tip. + pub fn forget_probe(&mut self) { + self.probed = false; + } + + /// An empty reply to a probe from the confirmed `tip`: this peer has + /// nothing above it. + pub fn note_ends_at(&mut self, tip: u32) { + self.probed = true; + self.announced.clear(); + self.from = tip.saturating_add(1); + self.ends = Some(tip); + } + + /// Keep announced `headers`, the first at `height`, against this node's + /// chain: the `path`, or a `confirmed` block at or below `tip`. The first + /// height where they differ is where the peer leaves this chain; a later + /// reply that agrees there clears it. Only heights above `tip` are kept, + /// newest last: a reply that continues the kept run extends it, any other + /// replaces it, and the oldest hashes past one full reply give way. A + /// short reply is where the peer's chain ends, also at or below the tip. + pub fn note_headers( + &mut self, + path: &HashMap, + confirmed: impl Fn(u32) -> Option, + tip: u32, + height: u32, + headers: &[Header], + ) { + self.probed = true; + let hashes: Vec = headers.iter().map(|h| h.block_hash()).collect(); + let mut split_moved = false; + let ours = |ht: u32| { + path.get(&ht) + .copied() + .or_else(|| (ht <= tip).then(|| confirmed(ht)).flatten()) + }; + // A reply on this node's block proves this chain below it, so a split + // at or below that block is gone. Every chain shares genesis. + if let Some(first) = headers.first() { + if height > 1 + && ours(height - 1) == Some(first.prev_blockhash) + && self.split.is_some_and(|(at, _)| at < height) + { + self.split = None; + } + } + for (ht, hash) in (height..).zip(&hashes) { + let on_path = path.get(&ht).copied(); + let Some(our) = ours(ht) else { + continue; + }; + if our == *hash { + if self.split.is_some_and(|(at, _)| at == ht) { + self.split = None; + } + continue; + } + if self.split.is_none_or(|(at, _)| ht <= at) { + self.split = Some((ht, *hash)); + self.split_confirmed = on_path.is_none(); + split_moved = true; + } + break; + } + let short = headers.len() < ANNOUNCED_CAP; + let top = height.saturating_add(hashes.len() as u32).saturating_sub(1); + // A stale record is answered by a reply that reaches its fork. One + // that ends below the fork says nothing there: ask again. + if let Some(stale) = self.stale_from { + if top >= stale { + self.stale_from = None; + } else { + self.probed = false; + } + } + self.ends = short.then_some(top); + let drop = tip.saturating_add(1).saturating_sub(self.from) as usize; + self.announced.drain(..drop.min(self.announced.len())); + self.from = self.from.max(tip.saturating_add(1)); + let skip = tip.saturating_add(1).saturating_sub(height) as usize; + let Some(fresh) = hashes.get(skip..).filter(|h| !h.is_empty()) else { + // Nothing above the tip. A short reply ended the peer's chain at + // or below it; a split that moved leaves the kept run on the + // chain the peer left. + if split_moved || short { + self.announced.clear(); + } + return; + }; + let start = height.saturating_add(skip as u32); + let end = self.from.saturating_add(self.announced.len() as u32); + // The kept run continues only from the parent of `start`. + let parent = match skip { + 0 => headers[0].prev_blockhash, + k => hashes[k - 1], + }; + let keep = (start > self.from && start <= end) + .then(|| (start - self.from) as usize) + .filter(|&k| self.announced[k - 1] == parent); + match keep { + Some(k) => self.announced.truncate(k), + None => { + self.announced.clear(); + self.from = start; + } + } + self.announced.extend(fresh.iter().copied()); + let over = self.announced.len().saturating_sub(ANNOUNCED_CAP); + self.announced.drain(..over); + self.from = self.from.saturating_add(over as u32); + } + + /// This node's confirmed chain above `lca` was rewound away, and `path` + /// is the chain planted in its place. A record that starts past the fork + /// took the old blocks as the peer's chain, so it is stale from the fork + /// up until the peer answers again, and the peer is probed. A split the + /// peer showed at or below the fork already says where it left, and an + /// announced hash on the new path says it is on the new chain. + pub fn note_rewind(&mut self, path: &HashMap, lca: u32) { + let at = lca.saturating_add(1); + if self.from <= at || self.split.is_some_and(|(h, _)| h <= at) { + return; + } + if (self.from..) + .zip(&self.announced) + .any(|(h, hash)| path.get(&h) == Some(hash)) + { + return; + } + self.stale_from = Some(self.stale_from.map_or(at, |s| s.min(at))); + self.probed = false; + } + + /// The recorded split's height has left the path: compare it with this + /// node's `confirmed` block there. The same block means this node took + /// the peer's chain, so the split is gone. + pub fn settle_split( + &mut self, + path: &HashMap, + confirmed: impl Fn(u32) -> Option, + ) { + let Some((at, hash)) = self.split else { + return; + }; + if path.contains_key(&at) { + return; + } + match confirmed(at) { + Some(ours) if ours == hash => self.split = None, + ours => self.split_confirmed = ours.is_some(), + } + } + + /// This peer's announced chain holds the path block at `height`: it + /// announced that block, or a path block above it, and has not left the + /// path at or below it. + pub fn shows(&self, path: &HashMap, height: u32) -> bool { + if self.left_path_by(path, height) { + return false; + } + // A path block the peer holds, from a probe or from where its chain + // left this one before this node took its chain. + let holds = |at: Option<(u32, BlockHash)>| { + at.is_some_and(|(h, hash)| height <= h && path.get(&h) == Some(&hash)) + }; + if holds(self.reaches) || holds(self.split) { + return true; + } + let Some(len) = (self.announced.len() as u32).checked_sub(1) else { + return false; + }; + if height > self.from.saturating_add(len) { + return false; + } + let at = height.max(self.from); + path.get(&at) + .is_some_and(|p| *p == self.announced[(at - self.from) as usize]) + } + + /// The peer does not have the path block at `height`: its chain left this + /// node's chain at or below it, it holds a different block there, its + /// whole chain ends below it, or a rewind left its record stale there. Two + /// chains that differ at one height differ above it. + pub fn left_path_by(&self, path: &HashMap, height: u32) -> bool { + if self.stale_from.is_some_and(|from| height >= from) { + return true; + } + if let Some((at, hash)) = self.split { + let differs = match path.get(&at) { + Some(p) => *p != hash, + None => self.split_confirmed, + }; + if at <= height && differs { + return true; + } + } + if self.ends.is_some_and(|top| height > top) { + return true; + } + let Some(len) = (self.announced.len() as u32).checked_sub(1) else { + return false; + }; + if height < self.from { + return false; + } + let top = self.from.saturating_add(len); + let at = height.min(top); + let peer = self.announced[(at - self.from) as usize]; + path.get(&at).is_some_and(|p| *p != peer) + } +} + +/// This node's confirmed block at `height`. +pub fn confirmed_at(hub: &rbitcoin_net::chain::ChainHub, height: u32) -> Option { + use bitcoin::hashes::Hash; + let (_, rec) = hub + .query + .header_at_height(rbitcoin_primitives::Height(height)) + .ok() + .flatten()?; + Some(BlockHash::from_byte_array(rec.hash)) +} diff --git a/crates/rbitcoin-ibd/src/ibd/peer_io.rs b/crates/rbitcoin-ibd/src/ibd/peer_io.rs index e62c3c2c0..48e303b8d 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_io.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_io.rs @@ -160,6 +160,8 @@ pub struct PeerSlot { /// All streamed wire bytes. Reader-only `fetch_add`. Not the stall clock. pub bytes_rx_total: Arc, pub rate: PeerRate, + /// What this peer has shown of its chain. IBD main thread only. + pub chain: super::peer_chain::PeerChain, pub alive: bool, pub task: JoinHandle<()>, } @@ -1061,6 +1063,7 @@ pub async fn spawn_peer( rate.note_connected(connected_ms); rate }, + chain: Default::default(), alive: true, task, }) diff --git a/crates/rbitcoin-ibd/src/ibd/reorg.rs b/crates/rbitcoin-ibd/src/ibd/reorg.rs index 5eaaa2361..d341f9a49 100644 --- a/crates/rbitcoin-ibd/src/ibd/reorg.rs +++ b/crates/rbitcoin-ibd/src/ibd/reorg.rs @@ -363,7 +363,10 @@ pub fn consider_disconnected_heavier( st: &mut super::state::IbdWorkState, hub: &ChainHub, ) -> Result { - let cands = connecting_search_candidates(st, hub)?; + let mut cands = connecting_search_candidates(st, hub)?; + if cands.is_empty() { + cands.extend(unplanted_walk_tip(st, hub)); + } let invalid: HashSet<[u8; 32]> = st.reorg.invalid.iter().collect(); let Some(path) = select_heavier_path(hub, &cands, &invalid)? else { return Ok(false); @@ -371,6 +374,21 @@ pub fn consider_disconnected_heavier( apply_header_rewind(st, hub, &path) } +/// The walk's candidate while nothing is planted above the confirmed tip. A +/// retire walks back to the parent, and a heavier chain that forked under +/// the tip then takes no path slots, so tip+1 names no fork. The main loop +/// rewinds to it; the header thread's rewind does not see it. A candidate +/// that is not stored has no parent, so no rewind is prepared for it. +fn unplanted_walk_tip(st: &super::state::IbdWorkState, hub: &ChainHub) -> Option { + let tip_h = hub.tip_height()?; + let planted = + !st.ordered.is_empty() || st.height_to_hash.contains_key(&tip_h.saturating_add(1)); + if planted || st.header_walk.tip_height() <= tip_h { + return None; + } + st.header_walk.tip_hash() +} + /// Header tips on known competing forks that do not contain an invalid hash. /// /// After a consensus-invalid reject the planted path may be a dead-end while a @@ -529,6 +547,8 @@ pub struct RewindPrep { pub archived: Vec, pub connected: Vec, pub fks: Vec>, + /// The confirmed tip was above the LCA, so blocks were disconnected. + pub rewound: bool, } /// Store side of a rewind: parent lookup and the confirmed-tip disconnect. @@ -553,6 +573,7 @@ pub fn prepare_header_rewind( return Ok(None); }; let tip_h = hub.tip_height().unwrap_or(0); + let rewound = tip_h > lca_h; if tip_h.saturating_sub(lca_h) > REWIND_MAX_DEPTH { warn!( "ibd: heavier fork rewind depth {} exceeds cap {REWIND_MAX_DEPTH} (lca={lca_h} tip={tip_h})", @@ -599,6 +620,7 @@ pub fn prepare_header_rewind( archived, connected, fks, + rewound, })) } @@ -632,6 +654,11 @@ pub fn commit_header_rewind( } prev = *hash; } + if prep.rewound { + for slot in &mut st.slots { + slot.chain.note_rewind(&st.height_to_hash, prep.lca_h); + } + } for h in hub.query.block_queue_queued_heights() { if h > prep.lca_h { let _ = hub.query.block_queue_dequeue_height(h); diff --git a/crates/rbitcoin-ibd/tests/ibd/assign.rs b/crates/rbitcoin-ibd/tests/ibd/assign.rs index 9fb8a8d79..803e738df 100644 --- a/crates/rbitcoin-ibd/tests/ibd/assign.rs +++ b/crates/rbitcoin-ibd/tests/ibd/assign.rs @@ -70,6 +70,7 @@ pub(crate) fn dummy_slot(id: usize) -> PeerSlot { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/dial.rs b/crates/rbitcoin-ibd/tests/ibd/dial.rs index 1d4ae0e6a..d123d08d3 100644 --- a/crates/rbitcoin-ibd/tests/ibd/dial.rs +++ b/crates/rbitcoin-ibd/tests/ibd/dial.rs @@ -87,6 +87,7 @@ fn dummy_slot(id: usize, a: SocketAddr, alive: bool) -> PeerSlot { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs index 37cebed6f..60a1c636a 100644 --- a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs +++ b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs @@ -99,6 +99,10 @@ fn confirm_reject_class_matches_substring_table() { ConfirmRejectClass::Cascade, ), ("confirm cancelled", ConfirmRejectClass::Cancelled), + ( + "consensus: bad header: timestamp too far in future", + ConfirmRejectClass::Cascade, + ), ]; for (s, want) in cases { assert_eq!(ConfirmRejectClass::from_err_str(s), *want, "{s}"); @@ -145,6 +149,13 @@ fn confirm_reject_class_matches_substring_table() { ConfirmRejectClass::from_consensus(&ConsensusError::Cancelled), ConfirmRejectClass::Cancelled ); + // Core BLOCK_TIME_FUTURE: the block may connect once the clock passes it. + assert_eq!( + ConfirmRejectClass::from_consensus(&ConsensusError::BadHeader( + "timestamp too far in future" + )), + ConfirmRejectClass::Cascade + ); assert_eq!( ConfirmRejectClass::from_confirm(&rbitcoin_query::ConfirmError::from( StoreError::Cancelled("stop"), @@ -717,6 +728,7 @@ fn apply_peer_event_body_and_control_surface() { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } @@ -840,10 +852,13 @@ fn apply_peer_event_body_and_control_surface() { let mut room = 4usize; let mut issued = 0u64; assert!( - rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, notfound_hash, &mut room, &mut issued), + !rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, notfound_hash, &mut room, &mut issued), + "a peer is not asked again for a block it does not have" + ); + assert!( + rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, h(11), &mut room, &mut issued), "notfound does not ban the peer" ); - assert!(st.slots[0].in_flight.contains(¬found_hash)); // Headers: attach height from tip parent and order. let hdr = dummy_header(gen, 1); @@ -862,6 +877,11 @@ fn apply_peer_event_body_and_control_surface() { ); assert!(st.known_headers.contains(&hash)); assert!(st.ordered_set.contains(&hash) || st.hash_height.contains_key(&hash)); + assert!( + rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, notfound_hash, &mut room, &mut issued), + "a peer that announces again may be asked again" + ); + assert!(st.slots[0].in_flight.contains(¬found_hash)); // Empty headers with lag → keep headers_done false. st.max_peer_height = 100; @@ -1037,6 +1057,7 @@ fn apply_peer_event_block_framed_bq_horizon_and_headers_done() { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } @@ -3106,6 +3127,43 @@ fn isolated_soft_reject_does_not_count_toward_cascade_halt() { assert!(st.halt.is_none(), "{:?}", st.halt); } +/// A block too far ahead of the clock is not a stale plan: its rejects at +/// one tip do not halt IBD, while a stale plan at that tip halts on the +/// third. +#[test] +fn a_block_too_far_ahead_does_not_count_toward_the_cascade_halt() { + let mut st = IbdWorkState::new(Vec::new(), Some(h(0)), Some(0)); + let hash = h(1); + st.record_height(hash, 1); + let reject = |st: &mut IbdWorkState, err: &str| { + apply_confirm_reject_class( + st, + 1, + hash, + ConfirmRejectClass::Cascade, + err, + None, + None, + 1, + None, + None, + RejectParent::default(), + ); + }; + for _ in 0..5 { + reject( + &mut st, + "consensus: bad header: timestamp too far in future", + ); + reject(&mut st, "time-too-new"); + } + assert!(st.halt.is_none(), "{:?}", st.halt); + for _ in 0..3 { + reject(&mut st, "fk mismatch"); + } + assert!(st.halt.is_some()); +} + /// tip+1 and tip+2 run scripts in one wave and tip+2's script fails. The /// scripts reject offers the wave back and isolates it: tip+1 connects from /// the offered-back body with no second getdata and no stale-plan retry, @@ -3548,6 +3606,117 @@ fn ibd_flag_zero_tx_block_is_consensus_invalid() { let _ = std::fs::remove_dir_all(dir); } +/// A block whose header a clock stepped back left more than 2 hours ahead +/// waits. The engine holds before it tries again instead of retrying at +/// once, those rejects do not halt IBD, and the block connects once the +/// clock passes its time. +#[test] +fn a_block_too_far_ahead_waits_and_then_connects() { + use super::super::assign::dummy_slot; + use bitcoin::consensus::serialize; + use rbitcoin_ibd::ibd::confirm::{spawn_confirm_engine, ConfirmEvent, ConfirmFeed}; + use rbitcoin_ibd::ibd::events::{apply_confirm_events, apply_peer_event}; + use rbitcoin_ibd::ibd::peer_io::PeerEvent; + use rbitcoin_ibd::ibd::state::InflightReq; + use rbitcoin_ibd::ibd::status::LoopStats; + use rbitcoin_net::seeds::AddrMan; + use std::net::{IpAddr, Ipv4Addr, SocketAddr}; + use std::sync::atomic::AtomicU32; + use std::sync::mpsc::{channel, RecvTimeoutError}; + use std::sync::Arc; + use std::time::{Duration, Instant}; + + let (dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("too-far-ahead"); + hub.query.enter_direct_index_mode().unwrap(); + let hub = Arc::new(hub); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let now = rbitcoin_consensus::clock::wall_now(); + let ahead = now + 2 * 3600 + 3; + let block = mine(gen, (ahead - 1_300_000_000) as u32, 1, vec![]); + let hash = block.block_hash(); + // Stored while the clock was ahead. + let fk = rbitcoin_consensus::with_now(ahead, || hub.ensure_header_fk(&block.header)).unwrap(); + + let mut st = IbdWorkState::new(vec![dummy_slot(1)], Some(gen), Some(0)); + let feed = Arc::new(ConfirmFeed::new()); + st.record_height(hash, 1); + st.header_fks.insert(hash, fk); + st.ordered.push_back(hash); + st.ordered_set.insert(hash); + st.slots[0].in_flight.insert(hash); + st.inflight.insert(hash, InflightReq::new(1)); + apply_peer_event( + &mut st, + &hub, + PeerEvent::BlockFramed { + peer: 1, + hash, + payload: serialize(&block), + header_fk: None, + }, + &AtomicU32::new(1), + &mut AddrMan::new(), + SocketAddr::new(IpAddr::V4(Ipv4Addr::new(10, 0, 0, 1)), 1), + Some(feed.as_ref()), + ); + + let (ev_tx, ev_rx) = channel(); + let (engine, _queues) = spawn_confirm_engine( + Arc::clone(&hub), + Arc::clone(&feed), + ev_tx, + Arc::new(AtomicU32::new(0)), + Arc::new(LoopStats::default()), + ); + let started = Instant::now(); + let deadline = started + Duration::from_secs(20); + let mut rejects = 0u32; + let connected = loop { + match ev_rx.recv_timeout(Duration::from_millis(5)) { + Ok(ev @ ConfirmEvent::Reject { .. }) => { + rejects += 1; + let (tx, rx) = channel(); + tx.send(ev).unwrap(); + drop(tx); + apply_confirm_events( + &mut st, + &hub, + &rx, + &AtomicU32::new(1), + &AtomicU32::new(0), + &mut Instant::now(), + Some(feed.as_ref()), + ); + assert!(st.halt.is_none(), "a block too far ahead does not halt IBD"); + } + Ok(ConfirmEvent::Accepted { .. }) => break true, + Err(RecvTimeoutError::Timeout) => { + if hub.tip_hash() == Some(hash) { + break true; + } + if Instant::now() > deadline { + break false; + } + } + Err(RecvTimeoutError::Disconnected) => break false, + } + }; + feed.request_stop(); + feed.notify(); + let _ = engine.join(); + assert!( + connected, + "the block connects once the clock passes its time" + ); + let waited = started.elapsed().as_secs().max(1) as u32; + assert!( + rejects <= 3 * waited + 2, + "{rejects} rejects in {waited}s: the engine waits between tries" + ); + let _ = std::fs::remove_dir_all(dir); +} + /// tip+1 spends a txid no block created. Its body matches its header, so /// the missing parent is the block's own fault (Core /// `bad-txns-inputs-missingorspent`): the hash is marked invalid, IBD does @@ -4060,6 +4229,7 @@ fn coord_loop_body_and_reject_do_not_read_headers() { first_data_ms: 0, bytes_rx_total: std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 4227e3aa8..a2dd84bbd 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -57,6 +57,7 @@ fn slot(id: usize) -> (PeerSlot, mpsc::UnboundedReceiver) { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, }, @@ -1312,6 +1313,63 @@ fn restart_restores_the_checkpoint_skip_and_garbage_does_not() { !skips(&hub, 1, h1.block_hash().as_byte_array()), "a file that does not parse does not skip scripts by height" ); + + // A candidate retired on a failed block restarts at its parent. + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-retire"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let p1 = mine(gen, 61); + let p2 = mine(p1.block_hash(), 62); + let p3 = mine(p2.block_hash(), 63); + let (s1, _rx1) = slot(1); + let mut st = IbdWorkState::new(vec![s1], Some(gen), Some(0)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + apply(&mut st, &hub, 1, vec![p1, p2, p3]); + assert!(st.is_on_path(&p3.block_hash(), 3)); + retire_invalid(&mut st, &hub, p3.block_hash(), 3); + st.header_walk = HeaderWalk::default(); + assert!(restore_adopt(&mut st, &hub)); + assert_eq!(st.header_walk.tip_hash(), Some(p2.block_hash())); + assert_eq!(st.header_walk.tip_height(), 2); + // Once a failed block is stored, a saved candidate above the confirmed + // tip may build on it and is not restored. + hub.note_failed_connect(p3.block_hash()); + st.header_walk = HeaderWalk::default(); + assert!(!restore_adopt(&mut st, &hub)); +} + +/// A failure stored at or below the saved walk's base is on no chain that +/// walk can take, so the walk is restored. One whose height is not found may +/// be on it. +#[test] +fn a_failure_below_the_saved_base_keeps_the_resume() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-resume-below"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let b1 = mine_block(gen, 1); + hub.accept_block(b1.clone()).unwrap(); + let b2 = mine_block(b1.block_hash(), 2); + hub.accept_block(b2.clone()).unwrap(); + let stale = mine(gen, 64); + hub.ensure_header(&stale).unwrap(); + assert!(hub.note_failed_connect(stale.block_hash())); + let (s2, _rx2) = slot(2); + let mut st = IbdWorkState::new(vec![s2], hub.tip_hash(), hub.tip_height()); + assert!(send_getheaders(&mut st, &hub).unwrap()); + let q3 = mine(b2.block_hash(), 65); + let q4 = mine(q3.block_hash(), 66); + apply(&mut st, &hub, 2, vec![q3, q4]); + save_adopt(&st, &hub); + st.header_walk = HeaderWalk::default(); + assert!( + restore_adopt(&mut st, &hub), + "a failure below the walk's base does not stop the resume" + ); + assert_eq!(st.header_walk.tip_hash(), Some(q4.block_hash())); + // A failure whose height is not found may be on it. + assert!(hub.note_failed_connect(BlockHash::from_byte_array([0x5a; 32]))); + st.header_walk = HeaderWalk::default(); + assert!(!restore_adopt(&mut st, &hub)); } #[test] @@ -1435,6 +1493,35 @@ fn two_peers_reserve_the_header_walk() { assert!(matches!(slow_rx.try_recv(), Ok(PeerCmd::GetData { .. }))); assert!(fast_rx.try_recv().is_err()); + apply_peer_event( + &mut st, + &hub, + PeerEvent::NotFound { + peer: 0, + hashes: vec![hash], + }, + &AtomicU32::new(0), + &mut AddrMan::new(), + SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 1), + None, + ); + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!( + st.slots[1].in_flight.contains(&hash), + "the header peer takes a block the other peer does not have" + ); + assert!(slow_rx.try_recv().is_err(), "notfound is not asked again"); + assert!(matches!(fast_rx.try_recv(), Ok(PeerCmd::GetData { .. }))); + let _ = st.slots[1].track_remove(&hash); + st.inflight.remove(&hash); + st.ordered.clear(); st.ordered_set.clear(); st.height_to_hash.remove(&1); @@ -1558,6 +1645,621 @@ fn two_peers_reserve_the_header_walk() { assert!(!st.slots[0].alive); assert!(st.addr_cooldown.contains_key(&addr)); assert!(matches!(rx.try_recv(), Ok(PeerCmd::Shutdown))); + + // A fourth, because the header peer must hold the path: a block peer + // that never announced a chain is asked for headers once. Its reply + // forks below the path, so it is not asked for path blocks. + let (mut head, mut head_rx) = slot(5); + let (mut other, mut other_rx) = slot(6); + let (mut short, _short_rx) = slot(7); + // Asked once, and answered nothing yet. + let (mut unknown, _unknown_rx) = slot(12); + head.first_data_ms = 50; + other.first_data_ms = 5_000; + short.first_data_ms = 6_000; + unknown.first_data_ms = 7_000; + unknown.chain.note_probe_sent(Instant::now(), true); + let mut st = IbdWorkState::new(vec![head, other, short, unknown], Some(gen), Some(0)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + assert_eq!(drain_getheaders(&mut head_rx), 1); + let path = [mine(gen, 41), mine(mine(gen, 41).block_hash(), 42)]; + apply(&mut st, &hub, 5, path.to_vec()); + assert_eq!(st.header_walk.reserved, Some(5)); + let _ = take_header_ask(&mut st, 5); + probe_unprobed_peer(&mut st, &hub); + assert!(send_getheaders(&mut st, &hub).unwrap()); + assert_eq!( + drain_getheaders(&mut other_rx), + 1, + "the silent peer is asked once" + ); + assert_eq!(drain_getheaders(&mut head_rx), 0); + let fork = mine(gen, 43); + apply(&mut st, &hub, 6, vec![fork, mine(fork.block_hash(), 46)]); + // A short reply is everything that peer has: its chain ends at height 1. + apply(&mut st, &hub, 7, vec![path[0]]); + probe_unprobed_peer(&mut st, &hub); + assert!(st.header_walk.probe.is_none(), "a peer is probed once"); + for h in &path { + st.body.mark_missing(h.block_hash()); + } + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!( + st.slots[1].in_flight.is_empty(), + "a peer whose chain left the path below a block is not asked for it" + ); + assert!( + !st.slots[2].in_flight.contains(&path[1].block_hash()), + "a peer whose whole chain ends below a block is not asked for it" + ); + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!(!st.slots[2].in_flight.contains(&path[1].block_hash())); + assert!( + may_serve(&st, 5, &path[1].block_hash()), + "a peer that announced nothing does not keep the header peer from a block it showed" + ); + assert!( + [0, 3] + .iter() + .any(|&i| st.slots[i].in_flight.contains(&path[1].block_hash())), + "the header peer, or a peer with no record, takes what no other peer holds" + ); + + // With no peer that holds the block by what it announced, a peer that + // did not answer `notfound` is still asked. + st.slots[0].chain.note_notfound( + path[1].block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + st.slots[2].chain.note_notfound( + path[1].block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + st.slots[3].chain.note_notfound( + path[1].block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + assert!(may_serve(&st, 6, &path[1].block_hash())); + + // A batch stored after one of its blocks failed is not planted, and does + // not grow the candidate again. + let late0 = mine(path[1].block_hash(), 44); + let late = [late0, mine(late0.block_hash(), 45)]; + st.reorg.invalid.mark(late[0].block_hash().to_byte_array()); + let tip = st.header_walk.tip_hash(); + let (fks, work) = rbitcoin_ibd::ibd::events::prepare_header_store(&hub, &late); + let added = + rbitcoin_ibd::ibd::events::on_headers_batch(&mut st, &hub, 5, late.to_vec(), &fks, work); + assert_eq!(added, 0); + assert!(!st.ordered_set.contains(&late[1].block_hash())); + assert_eq!(st.header_walk.tip_hash(), tip); + + // A fifth, on its own hub, because this node takes the forked peer's + // chain: once the fork height is confirmed and leaves the path, that peer + // holds the next path block again, also while a peer that announced + // nothing is up. + let (_dir5, hub5) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-taken"); + hub5.ensure_genesis().unwrap(); + let gen5 = hub5.tip_hash().unwrap(); + let (mut head, _head_rx) = slot(8); + let (forked, _forked_rx) = slot(9); + let (silent, _silent_rx) = slot(10); + let (other_fork, _other_rx) = slot(11); + head.first_data_ms = 50; + let mut st = IbdWorkState::new(vec![head, forked, silent, other_fork], Some(gen5), Some(0)); + assert!(send_getheaders(&mut st, &hub5).unwrap()); + let lost0 = mine(gen5, 81); + apply(&mut st, &hub5, 8, vec![lost0, mine(lost0.block_hash(), 82)]); + let taken1 = mine_block(gen5, 1); + let taken2 = mine_block(taken1.block_hash(), 2); + apply(&mut st, &hub5, 9, vec![taken1.header, taken2.header]); + let elsewhere = mine(gen5, 83); + apply( + &mut st, + &hub5, + 11, + vec![elsewhere, mine(elsewhere.block_hash(), 84)], + ); + hub5.accept_block(taken1.clone()).unwrap(); + st.height_to_hash.remove(&1); + st.height_to_hash.insert(2, taken2.block_hash()); + st.record_height(taken2.block_hash(), 2); + st.body.mark_missing(taken2.block_hash()); + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub5, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!( + may_serve(&st, 9, &taken2.block_hash()), + "the peer whose chain this node took is asked again" + ); + assert!( + !may_serve(&st, 11, &taken2.block_hash()), + "a peer that left where this node confirmed a different block has still left" + ); + // With no peer that holds a block by what it announced, the header peer + // is asked too. + st.slots[1].chain.note_notfound( + taken2.block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + st.slots[2].chain.note_notfound( + taken2.block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + assert!(may_serve(&st, 8, &taken2.block_hash())); + + // A sixth, on its own hub, because a probe must say what a peer has. It + // asks after the path top alone, and a reply on that top shows the whole + // path. A peer that does not show it is asked from the confirmed tip, + // where an empty reply is a chain that ends there. The probe is not a + // walk answer, and a reply with a header the walk has not seen gets the + // walk's own ask. An empty reply to a walk ask is a caught-up peer as well + // as one behind, so that peer is probed again. + let (_dir6, hub6) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-probe"); + hub6.ensure_genesis().unwrap(); + let gen6 = hub6.tip_hash().unwrap(); + let (mut head, mut head_rx) = slot(13); + let (mut caught, mut caught_rx) = slot(14); + let (mut behind, mut behind_rx) = slot(15); + let (mut fallback, mut fallback_rx) = slot(26); + let (mut side, mut side_rx) = slot(16); + head.first_data_ms = 50; + caught.first_data_ms = 5_000; + behind.first_data_ms = 6_000; + fallback.first_data_ms = 6_500; + side.first_data_ms = 7_000; + let mut st = IbdWorkState::new( + vec![head, caught, behind, fallback, side], + Some(gen6), + Some(0), + ); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(drain_getheaders(&mut head_rx), 1); + let path = [mine(gen6, 51), mine(mine(gen6, 51).block_hash(), 52)]; + apply(&mut st, &hub6, 13, path.to_vec()); + assert_eq!(st.header_walk.reserved, Some(13)); + let _ = take_header_ask(&mut st, 13); + + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + let locator = getheaders_locator(&mut caught_rx).expect("the caught-up peer is probed"); + assert_eq!( + locator, + vec![path[1].block_hash()], + "the probe asks after the path top alone" + ); + // An announcement that crosses the probe is not its reply. + let stray = mine(BlockHash::from_byte_array([0x77; 32]), 53); + apply(&mut st, &hub6, 14, vec![stray]); + assert_eq!( + st.header_walk.walk.peer, + Some(14), + "the probe ask stays open, so it can still expire" + ); + // That stray batch went to the walk, which retired the peer from it. + st.header_walk.walk_quiet.remove(&14); + // At the top the peer has nothing after it, so it is asked again from + // the tip. + apply(&mut st, &hub6, 14, Vec::new()); + assert_eq!(st.header_walk.probe_open, None); + assert!( + !st.header_walk.walk_quiet.contains(&14), + "a probe reply is not a walk answer" + ); + assert_eq!(st.header_walk.reserved, Some(13)); + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + let locator = getheaders_locator(&mut caught_rx).expect("the peer is asked again"); + assert_eq!(locator.first(), Some(&gen6), "then from the confirmed tip"); + assert!( + !st.slots[1].chain.probe_from_tip(), + "and the probe after that asks after the top again" + ); + apply(&mut st, &hub6, 14, path.to_vec()); + assert!(may_serve(&st, 14, &path[1].block_hash())); + + // A peer without the top answers from genesis: this node sends genesis + // itself, Core the header after it. One that stops at or below the tip + // is asked again from the tip, where an empty reply is a chain that ends + // there. + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(getheaders_locator(&mut behind_rx).map(|l| l.len()), Some(1)); + apply(&mut st, &hub6, 15, vec![hub6.header_of(&gen6).unwrap()]); + assert_eq!(st.header_walk.probe_open, None); + assert!(st.slots[2].chain.probe_from_tip()); + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut behind_rx).is_some()); + apply(&mut st, &hub6, 15, Vec::new()); + assert!( + st.slots[2].chain.left_path_by(&st.height_to_hash, 1), + "a peer with nothing above the tip does not hold a path block" + ); + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut fallback_rx).is_some()); + // A reply from genesis that already goes past the tip says what a probe + // from the tip would. + apply(&mut st, &hub6, 26, vec![mine(gen6, 55)]); + assert_eq!(st.header_walk.probe_open, None); + assert!(!st.slots[3].chain.probe_from_tip()); + assert!(st.slots[3].chain.left_path_by(&st.height_to_hash, 1)); + st.header_walk.challenge = None; + + // A reply on the path top shows the path below it. Its new header is one + // the walk has not seen, so that peer gets the walk's own ask. + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut side_rx).is_some()); + apply(&mut st, &hub6, 16, vec![mine(path[1].block_hash(), 54)]); + assert!(st.slots[4].chain.shows(&st.height_to_hash, 1)); + assert_eq!(st.header_walk.challenge, Some(16)); + st.header_walk.challenge = None; + + // A block inv's ask carries the path top, so an empty reply is a peer at + // that top as well as one behind it. + note_block_inv(&mut st, 14); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut caught_rx).is_some()); + apply(&mut st, &hub6, 14, Vec::new()); + assert!( + may_serve(&st, 14, &path[1].block_hash()), + "an empty walk reply does not take a caught-up peer off the path" + ); + assert!( + !st.slots[1].chain.probed(), + "an empty walk reply is followed by a probe" + ); + + // A late reply to a walk ask could be read as the probe's, so a peer + // with one out is neither picked nor asked. + st.header_walk.walk.peer = None; + st.header_walk.walk.at = None; + st.header_walk.walk.prev_peer = Some(14); + probe_unprobed_peer(&mut st, &hub6); + assert_ne!(st.header_walk.probe, Some(14)); + st.header_walk.probe = Some(14); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(st.header_walk.probe, None); + assert_ne!(st.header_walk.probe_open, Some(14)); + assert!(!st.slots[1].chain.probed(), "it is picked again later"); + // Neither is a peer with an expired walk ask that it has not answered, + // after another peer's expiry took the lane's last slot for it. + st.header_walk.walk.prev_peer = None; + st.header_walk.header_misses.insert(14, 1); + probe_unprobed_peer(&mut st, &hub6); + assert_ne!(st.header_walk.probe, Some(14)); + st.header_walk.header_misses.clear(); + + // An open probe keeps the lane until it is answered or expires, also + // when the walk has nothing left to ask. + for s in &mut st.slots { + s.peer_height = 0; + } + st.header_walk.walk.peer = None; + st.header_walk.walk.at = None; + probe_unprobed_peer(&mut st, &hub6); + assert_eq!(st.header_walk.probe, Some(14)); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(st.header_walk.probe_open, Some(14)); + let _ = send_getheaders(&mut st, &hub6); + assert_eq!( + st.header_walk.walk.peer, + Some(14), + "the probe keeps its lane" + ); + // With header sync otherwise done, the poll still asks once the probe is + // past its window, so the ask that expires it is sent. + st.headers_done = true; + let quiet = || HeaderPollIn { + live: 0, + need_ready_headroom: false, + lag: 0, + min_cache: 0, + fan: 0, + }; + assert!(matches!(header_poll(&st, quiet()), HeaderPoll::Wait)); + age_header_ask(&mut st); + assert!( + matches!(header_poll(&st, quiet()), HeaderPoll::Ask), + "a probe past its window is polled" + ); + let _ = send_getheaders(&mut st, &hub6); + assert_eq!( + st.header_walk.probe_open, None, + "an unanswered probe expires" + ); + + // A seventh, on its own hub, because a rewind replaces the confirmed + // blocks a peer's record took as its chain: that peer is not asked for + // the new chain's blocks, and one that had shown the new chain, all of + // it or only its top, is. + let (_dir7, hub7) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-rewound"); + hub7.ensure_genesis().unwrap(); + let gen7 = hub7.tip_hash().unwrap(); + let l1 = mine_block(gen7, 1); + hub7.accept_block(l1.clone()).unwrap(); + let l2 = mine_block(l1.block_hash(), 2); + hub7.accept_block(l2.clone()).unwrap(); + let w1 = mine(gen7, 71); + let w2 = mine(w1.block_hash(), 72); + let w3 = mine(w2.block_hash(), 73); + for w in [&w1, &w2, &w3] { + hub7.ensure_header(w).unwrap(); + } + let (old, _old_rx) = slot(17); + let (fresh, _fresh_rx) = slot(18); + let (taken, _taken_rx) = slot(19); + let (suffix, _suffix_rx) = slot(20); + let mut st = IbdWorkState::new( + vec![old, fresh, taken, suffix], + hub7.tip_hash(), + hub7.tip_height(), + ); + let confirmed = |h| rbitcoin_ibd::ibd::peer_chain::confirmed_at(&hub7, h); + let l3 = mine(l2.block_hash(), 74); + st.slots[0].chain.note_headers( + &st.height_to_hash, + confirmed, + 2, + 1, + &[l1.header, l2.header, l3], + ); + st.slots[2] + .chain + .note_headers(&st.height_to_hash, confirmed, 2, 1, &[w1, w2]); + // Announced only the new chain's top, above the old tip. + st.slots[3] + .chain + .note_headers(&st.height_to_hash, confirmed, 2, 3, &[w3]); + st.header_walk.tip.hash = Some(w3.block_hash()); + st.header_walk.tip.height = 3; + assert!(rbitcoin_ibd::ibd::reorg::consider_disconnected_heavier(&mut st, &hub7).unwrap()); + assert_eq!(st.height_to_hash.get(&1), Some(&w1.block_hash())); + assert!( + !may_serve(&st, 17, &w1.block_hash()), + "a peer on the rewound chain is not asked for the new one" + ); + assert!(!st.slots[0].chain.probed(), "and is probed again"); + assert!( + !st.slots[1].chain.left_path_by(&st.height_to_hash, 1), + "a peer with no record is not left stale by the rewind" + ); + assert!(!may_serve(&st, 18, &w1.block_hash())); + assert!( + may_serve(&st, 19, &w1.block_hash()), + "a peer whose split is the new chain's block shows it" + ); + assert!( + may_serve(&st, 20, &w3.block_hash()), + "a peer that announced the new chain is asked for it" + ); + + // An eighth, because a block is asked only of a peer whose chain showed + // it: a peer whose full reply ended below the block, or one with no + // record, is not asked for it while another peer shows it. With no peer + // that shows a block, a peer that has not left the path is asked first. + let (_dir8, hub8) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-shown"); + hub8.ensure_genesis().unwrap(); + let gen8 = hub8.tip_hash().unwrap(); + let (head, _head_rx) = slot(22); + let (full, _full_rx) = slot(23); + let (blank, _blank_rx) = slot(24); + let (left, _left_rx) = slot(25); + let mut st = IbdWorkState::new(vec![head, full, blank, left], Some(gen8), Some(0)); + let mut prev = gen8; + let mut chain8 = Vec::new(); + for n in 0..2_004u32 { + let h = mine(prev, 90_000 + n); + prev = h.block_hash(); + chain8.push(h); + } + for (i, h) in chain8.iter().enumerate() { + st.record_height(h.block_hash(), i as u32 + 1); + st.height_to_hash.insert(i as u32 + 1, h.block_hash()); + } + st.header_walk.reserved = Some(22); + let none = |_| None; + let path8 = st.height_to_hash.clone(); + st.slots[0] + .chain + .note_headers(&path8, none, 0, 1, &chain8[..2_003]); + st.slots[1] + .chain + .note_headers(&path8, none, 0, 1, &chain8[..2_000]); + let fork = mine(chain8[0].block_hash(), 99_999); + st.slots[3].chain.note_headers(&path8, none, 0, 2, &[fork]); + let at = |h: u32| chain8[h as usize - 1].block_hash(); + assert!(may_serve(&st, 23, &at(100))); + assert!( + !may_serve(&st, 22, &at(100)), + "the header peer stays on headers while another peer shows the block" + ); + assert!( + !may_serve(&st, 23, &at(2_003)), + "a full reply that ended below a block does not show it" + ); + assert!( + !may_serve(&st, 24, &at(2_003)), + "a peer with no record is not asked for a block another peer shows" + ); + assert!( + may_serve(&st, 22, &at(2_003)), + "the header peer shows it alone" + ); + assert!(may_serve(&st, 24, &at(2_004))); + assert!( + !may_serve(&st, 25, &at(2_004)), + "with nobody showing a block, a peer that left the path is not asked" + ); + // A peer whose record stops short of the next blocks is probed again; one + // probed just now is not. + st.max_ordered_height = 2_004; + probe_unprobed_peer(&mut st, &hub8); + assert_eq!(st.header_walk.probe, Some(23)); + st.header_walk.probe = None; + st.slots[1].chain.note_probe_sent(Instant::now(), true); + probe_unprobed_peer(&mut st, &hub8); + assert_eq!( + st.header_walk.probe, + Some(24), + "the next is the peer with no record" + ); +} + +/// A long chain on an invalid block stays refused past the cap on +/// remembered tips: the newest link is kept. +#[test] +fn a_long_chain_on_an_invalid_block_stays_refused() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-refuse-cap"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let (s0, _rx0) = slot(0); + let mut st = IbdWorkState::new(vec![s0], Some(gen), Some(0)); + let bad = mine(gen, 1); + st.reorg.invalid.mark(bad.block_hash().to_byte_array()); + let mut prev = bad.block_hash(); + let mut branch = None; + for n in 0..1_100u32 { + let next = mine(prev, 2 + n); + assert!( + refuse_invalid_batch(&mut st, 0, &[next]), + "batch {n} builds on the refused chain" + ); + prev = next.block_hash(); + if n == 100 { + // A second branch off the invalid block, refused once. + let side = mine(bad.block_hash(), 5_000); + assert!(refuse_invalid_batch(&mut st, 0, &[side])); + branch = Some(side.block_hash()); + } + } + let side_next = mine(branch.unwrap(), 5_001); + assert!( + refuse_invalid_batch(&mut st, 0, &[side_next]), + "a branch refused after the oldest tips is still refused past the cap" + ); + + // A batch that is not one linked chain does not build on the invalid + // block, so its last header is not remembered: an honest header cannot + // be refused by sending it after one that is. + let honest1 = mine(gen, 7_001); + let honest2 = mine(honest1.block_hash(), 7_002); + let junk = Header { + prev_blockhash: bad.block_hash(), + ..honest1 + }; + assert!(!refuse_invalid_batch(&mut st, 0, &[junk, honest2])); + assert!(!refuse_invalid_batch(&mut st, 0, &[bad, honest1])); + assert!( + !refuse_invalid_batch(&mut st, 0, &[honest1, honest2]), + "the honest chain is not refused" + ); +} + +/// A peer that announced the path and then forks below the confirmed tip no +/// longer shows path blocks, so it does not keep the header peer that +/// announced them from them. +#[test] +fn a_peer_that_forks_below_the_tip_does_not_keep_the_header_peer_off() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-fork-below"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let b1 = mine_block(gen, 1); + hub.accept_block(b1.clone()).unwrap(); + let b2 = mine_block(b1.block_hash(), 2); + hub.accept_block(b2.clone()).unwrap(); + let (mut head, _head_rx) = slot(0); + let (mut other, _other_rx) = slot(1); + head.first_data_ms = 50; + other.first_data_ms = 5_000; + let mut st = IbdWorkState::new(vec![head, other], Some(b2.block_hash()), Some(2)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + let h3 = mine(b2.block_hash(), 3); + let path = [h3, mine(h3.block_hash(), 4)]; + apply(&mut st, &hub, 0, path.to_vec()); + apply(&mut st, &hub, 1, path.to_vec()); + assert_eq!(st.header_walk.reserved, Some(0)); + assert!(st.slots[1].chain.shows(&st.height_to_hash, 3)); + apply(&mut st, &hub, 1, vec![mine(b1.block_hash(), 92)]); + assert!( + !st.slots[1].chain.shows(&st.height_to_hash, 3), + "a peer that left the path does not show a block on it" + ); + assert!( + may_serve(&st, 0, &path[0].block_hash()), + "the header peer that announced the block is still asked" + ); +} + +/// When the failure cannot be written, this process still retires the +/// candidate, but the retired walk is not left on disk as the resume point. +#[test] +fn a_retire_whose_failure_is_not_written_leaves_no_resume_point() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-retire-unsaved"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let store = hub.query.store().path().to_path_buf(); + std::fs::create_dir(store.join("block.invalid")).unwrap(); + let (s1, _rx1) = slot(1); + let mut st = IbdWorkState::new(vec![s1], Some(gen), Some(0)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + let p1 = mine(gen, 61); + apply(&mut st, &hub, 1, vec![p1, mine(p1.block_hash(), 62)]); + assert!(st.is_on_path(&p1.block_hash(), 1)); + rbitcoin_ibd::ibd::events::apply_consensus_invalid_reject( + &mut st, + 1, + p1.block_hash(), + "bad block: coinbase excess value", + Some(hub.query.as_ref()), + Some(&hub), + ); + assert_eq!(st.header_walk.tip_height(), 0, "the candidate is retired"); + assert!( + !store.join("header.adopt").exists(), + "a retired walk is not the resume point while its failure is not on disk" + ); + assert!( + !hub.note_failed_connect(p1.block_hash()), + "a mark whose write failed is still not on disk" + ); + save_adopt(&st, &hub); + assert!( + !store.join("header.adopt").exists(), + "no walk is saved while a failure is not on disk" + ); + std::fs::remove_dir(store.join("block.invalid")).unwrap(); + assert!( + hub.note_failed_connect(p1.block_hash()), + "once the file can be written the pending mark goes to disk" + ); + assert!(std::fs::read(store.join("block.invalid")) + .unwrap() + .windows(32) + .any(|w| w == p1.block_hash().as_byte_array())); + save_adopt(&st, &hub); + assert!(store.join("header.adopt").exists()); } #[test] @@ -1973,6 +2675,16 @@ fn a_heavier_header_chain_keeps_its_context() { } } +fn getheaders_locator(rx: &mut mpsc::UnboundedReceiver) -> Option> { + loop { + match rx.try_recv() { + Ok(PeerCmd::GetHeaders { locator }) => return Some(locator), + Ok(_) => {} + Err(_) => return None, + } + } +} + fn drain_getheaders(rx: &mut mpsc::UnboundedReceiver) -> usize { let mut n = 0usize; loop { diff --git a/crates/rbitcoin-ibd/tests/ibd/mod.rs b/crates/rbitcoin-ibd/tests/ibd/mod.rs index 8dfd33023..8a54d7c91 100644 --- a/crates/rbitcoin-ibd/tests/ibd/mod.rs +++ b/crates/rbitcoin-ibd/tests/ibd/mod.rs @@ -11,6 +11,7 @@ mod exit; mod header_walk; mod path; mod peer_book_and_config_tests; +mod peer_chain; mod peer_io; mod perf_log; mod progress; diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs new file mode 100644 index 000000000..a23a403e6 --- /dev/null +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -0,0 +1,270 @@ +use bitcoin::block::Header; +use bitcoin::hashes::Hash; +use bitcoin::{BlockHash, CompactTarget, TxMerkleNode}; +use rbitcoin_ibd::ibd::peer_chain::PeerChain; +use std::collections::HashMap; + +fn chain(n: u32, salt: u8) -> Vec
{ + extend(BlockHash::from_byte_array([0u8; 32]), n, salt) +} + +fn extend(from: BlockHash, n: u32, salt: u8) -> Vec
{ + let mut prev = from; + (1..=n) + .map(|i| { + let mut h = Header { + version: bitcoin::block::Version::from_consensus(4), + prev_blockhash: prev, + merkle_root: TxMerkleNode::from_byte_array([salt; 32]), + time: 1_600_000_000 + i, + bits: CompactTarget::from_consensus(0x207fffff), + nonce: i, + }; + rbitcoin_consensus::grind_regtest_pow(&mut h); + prev = h.block_hash(); + h + }) + .collect() +} + +/// Two full `headers` replies and then a short one: the peer holds every +/// block of its newest reply, and none past that reply's top. +#[test] +fn an_announced_chain_keeps_the_newest_reply() { + let headers = chain(4_010, 0x5c); + let path: HashMap = (1..).zip(headers.iter().map(|h| h.block_hash())).collect(); + let mut peer = PeerChain::default(); + peer.note_headers(&path, |_| None, 0, 1, &headers[..2_000]); + peer.note_headers(&path, |_| None, 0, 2_001, &headers[2_000..4_000]); + peer.note_headers(&path, |_| None, 0, 4_001, &headers[4_000..]); + assert!( + !peer.left_path_by(&path, 4_005), + "the newest reply is what the peer has" + ); + assert!(!peer.left_path_by(&path, 3_000)); + let mut beyond = path.clone(); + beyond.insert(4_011, BlockHash::from_byte_array([7u8; 32])); + assert!( + peer.left_path_by(&beyond, 4_011), + "a short reply is the end of that peer's chain" + ); +} + +/// A full reply whose split height has since been confirmed and left the +/// path. This node's confirmed block there settles it: a different block +/// means the peer left this chain, the same block means this node took it. +#[test] +fn a_split_is_settled_by_the_confirmed_chain() { + let ours = chain(2_001, 0x5c); + let theirs = chain(2_000, 0x77); + let mut path: HashMap = + (1..).zip(ours.iter().map(|h| h.block_hash())).collect(); + let mut left = PeerChain::default(); + left.note_headers(&path, |_| None, 0, 1, &theirs); + let mut taken = PeerChain::default(); + let mut before = path.clone(); + before.insert(1, theirs[0].block_hash()); + taken.note_headers(&before, |_| None, 0, 1, &ours[..2_000]); + for h in 1..=2_000 { + path.remove(&h); + } + let confirmed = |h: u32| (h <= 2_000).then(|| ours[h as usize - 1].block_hash()); + left.settle_split(&path, confirmed); + taken.settle_split(&path, confirmed); + assert!( + left.left_path_by(&path, 2_001), + "a peer that split where this node confirmed another block has left" + ); + assert!( + !taken.left_path_by(&path, 2_001), + "a peer whose chain this node confirmed has not" + ); +} + +/// A short reply that ends at or below the confirmed tip is all that peer +/// has, so it is not asked above the tip. That holds for a first reply and +/// for one that follows a short run this node has since confirmed. +#[test] +fn a_reply_at_or_below_the_tip_ends_the_chain_there() { + let headers = chain(20, 0x5c); + let above = |tip: u32| -> HashMap { + (tip + 1..=20) + .map(|h| (h, headers[h as usize - 1].block_hash())) + .collect() + }; + let confirmed = |h: u32| Some(headers[h as usize - 1].block_hash()); + let mut behind = PeerChain::default(); + behind.note_headers(&above(10), confirmed, 10, 5, &headers[4..8]); + assert!( + behind.left_path_by(&above(10), 11), + "a first reply that ends below the tip" + ); + let mut caught = PeerChain::default(); + caught.note_headers(&above(10), confirmed, 10, 11, &headers[10..15]); + assert!(caught.left_path_by(&above(10), 16)); + caught.note_headers(&above(15), confirmed, 15, 13, &headers[12..15]); + assert!( + caught.left_path_by(&above(15), 16), + "a reply that ends at the tip after the kept run was confirmed" + ); +} + +/// A rewind keeps a split the peer itself showed at or below the fork. A +/// peer on a third chain stays off both chains when this node rewinds away +/// from its chain and back. +#[test] +fn a_third_chain_split_survives_two_rewinds() { + let ours = chain(3, 0x5c); + let won = chain(3, 0x77); + let third = chain(3, 0x91); + let path = |c: &[Header]| -> HashMap { + (1..).zip(c.iter().map(|h| h.block_hash())).collect() + }; + let confirmed = |h: u32| Some(ours[h as usize - 1].block_hash()); + let mut peer = PeerChain::default(); + peer.note_headers(&HashMap::new(), confirmed, 2, 1, &third); + peer.note_rewind(&path(&won), 0); + assert!(peer.left_path_by(&path(&won), 1)); + peer.note_rewind(&path(&ours), 0); + assert!( + peer.left_path_by(&path(&ours), 1), + "a peer on a third chain is not taken back onto this one" + ); +} + +/// A rewind leaves a record that took the rewound blocks as the peer's chain +/// stale from the fork up, also when the peer's own split is above the fork, +/// across further rewinds, until the peer answers again. It is probed again. +#[test] +fn a_rewind_leaves_a_record_stale_until_the_peer_answers() { + let ours = chain(2_003, 0x5c); + let won = extend(ours[1_996].block_hash(), 3, 0x77); + let third = extend(ours[1_997].block_hash(), 3, 0x91); + let at = |c: &[Header], from: u32| -> HashMap { + (from..).zip(c.iter().map(|h| h.block_hash())).collect() + }; + let confirmed = |h: u32| (h <= 2_001).then(|| ours[h as usize - 1].block_hash()); + // A full reply that leaves this chain at 1999, above the fork at 1998. + let mut reply = ours[1..1_998].to_vec(); + reply.extend_from_slice(&third); + let mut peer = PeerChain::default(); + peer.note_headers(&HashMap::new(), confirmed, 2_001, 2, &reply); + let on_won = at(&won, 1_998); + peer.note_rewind(&on_won, 1_997); + assert!(peer.left_path_by(&on_won, 1_998)); + assert!(!peer.probed(), "the peer is probed again"); + let on_ours = at(&ours[1_997..], 1_998); + peer.note_rewind(&on_ours, 1_997); + assert!( + peer.left_path_by(&on_ours, 2_003), + "it left this chain above the fork" + ); + // A deeper rewind keeps it off from the lower fork. + let deeper = at(&extend(ours[1_988].block_hash(), 2, 0x33), 1_990); + peer.note_rewind(&deeper, 1_989); + assert!(peer.left_path_by(&deeper, 1_990)); + // Its answer is the record again. + peer.note_headers(&on_ours, |_| None, 1_997, 1_998, &ours[1_997..2_000]); + assert!(!peer.left_path_by(&on_ours, 1_999)); + assert!( + peer.left_path_by(&on_ours, 2_001), + "the reply ended at 2000" + ); + + // A full reply that ends below the fork says nothing there: the record + // stays stale and the peer is probed again. + let mut far = PeerChain::default(); + let confirmed_far = |h: u32| (h <= 2_002).then(|| ours[h as usize - 1].block_hash()); + far.note_headers(&HashMap::new(), confirmed_far, 2_002, 2_003, &ours[2_002..]); + let side = at(&extend(ours[2_000].block_hash(), 2, 0x44), 2_002); + far.note_rewind(&side, 2_001); + far.note_headers(&side, confirmed_far, 2_001, 1, &ours[..2_000]); + assert!(far.left_path_by(&side, 2_002)); + assert!(!far.probed()); + // A full reply that reaches the fork on the new chain answers it. + let mut onto = ours[3..2_001].to_vec(); + onto.extend(extend(ours[2_000].block_hash(), 2, 0x44)); + far.note_headers(&side, confirmed_far, 2_001, 4, &onto); + assert!(!far.left_path_by(&side, 2_002)); +} + +/// A peer that showed a stale sibling and then builds on this node's block +/// at or above that height has joined this chain: the split is gone. +#[test] +fn a_reply_on_this_chain_above_a_split_clears_it() { + let ours = chain(3, 0x5c); + let sibling = chain(1, 0x77); + let path: HashMap = (1..).zip(ours.iter().map(|h| h.block_hash())).collect(); + let mut peer = PeerChain::default(); + peer.note_headers(&path, |_| None, 0, 1, &sibling); + assert!(peer.left_path_by(&path, 2)); + peer.note_headers(&path, |_| None, 0, 2, &ours[1..]); + assert!(!peer.left_path_by(&path, 2)); + assert!(!peer.left_path_by(&path, 3)); + assert!( + peer.shows(&path, 3), + "the peer is asked for this chain again" + ); + + // A reply on this node's block that leaves it at the next height: the + // parent clears the old split and the new one is where it left. + let mut moved = PeerChain::default(); + moved.note_headers(&path, |_| None, 0, 1, &sibling); + let other = chain(2, 0x91); + let off = Header { + prev_blockhash: ours[0].block_hash(), + ..other[1] + }; + moved.note_headers(&path, |_| None, 0, 2, &[off]); + assert!( + !moved.left_path_by(&path, 1), + "it has this node's block below" + ); + assert!(moved.left_path_by(&path, 2)); +} + +/// A peer whose record does not decide a height is probed again, but not +/// within 10 s of its last probe. +#[test] +fn a_record_that_stops_short_is_refreshed() { + use std::time::{Duration, Instant}; + let headers = chain(20, 0x5c); + let path: HashMap = (1..).zip(headers.iter().map(|h| h.block_hash())).collect(); + let Some(sent) = Instant::now().checked_sub(Duration::from_secs(30)) else { + return; + }; + let mut ended = PeerChain::default(); + ended.note_probe_sent(sent, true); + ended.note_headers(&path, |_| None, 0, 1, &headers[..10]); + assert!( + !ended.needs_refresh(&path, 5, Instant::now()), + "a height the record shows is decided" + ); + assert!( + !ended.needs_refresh(&path, 15, Instant::now()), + "so is one above where a short reply ended" + ); + let mut reached = PeerChain::default(); + reached.note_probe_sent(sent, false); + reached.note_reaches(10, headers[9].block_hash()); + assert!(!reached.needs_refresh(&path, 5, sent + Duration::from_secs(10))); + assert!(reached.needs_refresh(&path, 15, sent + Duration::from_secs(10))); + assert!( + !reached.needs_refresh(&path, 15, sent + Duration::from_millis(9_999)), + "not within 10 s of the last probe" + ); +} + +/// A `notfound` keeps a peer off that block for one stall window, then the +/// peer may be asked again. +#[test] +fn a_notfound_lasts_one_stall_window() { + let mut peer = PeerChain::default(); + let hash = BlockHash::from_byte_array([3u8; 32]); + peer.note_notfound(hash, 1_000); + assert!(peer.lacks(&hash, 1_000 + 29_999)); + assert!( + !peer.lacks(&hash, 1_000 + 30_000), + "past the stall window the peer is asked again" + ); +} diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_io.rs b/crates/rbitcoin-ibd/tests/ibd/peer_io.rs index e6c4c38cf..d0aff5c79 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_io.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_io.rs @@ -42,6 +42,7 @@ fn dummy_slot(id: usize) -> PeerSlot { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/reorg.rs b/crates/rbitcoin-ibd/tests/ibd/reorg.rs index 33262d069..02d80f1d7 100644 --- a/crates/rbitcoin-ibd/tests/ibd/reorg.rs +++ b/crates/rbitcoin-ibd/tests/ibd/reorg.rs @@ -211,6 +211,55 @@ fn consider_disconnected_from_work_path_without_resume_seed() { let _ = std::fs::remove_dir_all(dir); } +/// The walk's candidate forks below the confirmed tip, as after a retire +/// walks back and a heavier chain arrives that forked under it. None of it +/// is planted, so the path's next height names no fork. Its stored tip is +/// where the rewind walks back from. A walk tip that is not stored is not, +/// and once a path is planted the walk's tip is not read from the store. +#[test] +fn a_walk_candidate_forked_below_the_tip_is_rewound_to() { + let (dir, hub) = tmp_hub(); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let l1 = mine(gen, 1_500_063_100, 1); + hub.accept_block(l1.clone()).unwrap(); + let l2 = mine(l1.block_hash(), 1_500_063_200, 2); + hub.accept_block(l2.clone()).unwrap(); + let w1 = distinct_sib(mine(gen, 1_500_063_101, 1), l1.block_hash()); + let w2 = mine(w1.block_hash(), 1_500_063_201, 2); + let w3 = mine(w2.block_hash(), 1_500_063_301, 3); + for w in [&w1, &w2, &w3] { + hub.ensure_header(&w.header).unwrap(); + } + let w4 = mine(w3.block_hash(), 1_500_063_401, 4); + + let mut st = + rbitcoin_ibd::ibd::state::IbdWorkState::new(Vec::new(), hub.tip_hash(), hub.tip_height()); + st.header_walk.tip.hash = Some(w4.block_hash()); + st.header_walk.tip.height = 4; + assert!(!consider_disconnected_heavier(&mut st, &hub).unwrap()); + assert_eq!( + hub.tip_height(), + Some(2), + "an unstored walk tip is not a rewind" + ); + + st.header_walk.tip.hash = Some(w3.block_hash()); + st.header_walk.tip.height = 3; + assert!(consider_disconnected_heavier(&mut st, &hub).unwrap()); + assert_eq!(hub.tip_height(), Some(0), "rewound to the fork"); + assert_eq!(st.height_to_hash.get(&1), Some(&w1.block_hash())); + assert_eq!(st.height_to_hash.get(&3), Some(&w3.block_hash())); + reset_parent_hash_of_count(); + assert!(!consider_disconnected_heavier(&mut st, &hub).unwrap()); + assert_eq!( + parent_hash_of_count(), + 0, + "a planted path is not a store walk" + ); + let _ = std::fs::remove_dir_all(dir); +} + /// A real block whose prev is not a stored header fails the live confirm /// load as BadPrev. That reject is soft reget: the tip stays, the slot is /// evicted, and the hash is not blacklisted or fetched again. diff --git a/crates/rbitcoin-net/src/chain.rs b/crates/rbitcoin-net/src/chain.rs index f2f1dc3e3..0409796cb 100644 --- a/crates/rbitcoin-net/src/chain.rs +++ b/crates/rbitcoin-net/src/chain.rs @@ -237,6 +237,55 @@ impl ForkTips { pub struct Invalidated { pub set: RwLock>, pub paths: RwLock>, + /// Connect failures kept across restart in [`INVALID_FILE`], oldest first. + /// Operator invalidations are not: their paths are in-process only. + pub failed: RwLock>, + pub file: std::path::PathBuf, + /// `failed` holds a mark the last write did not put on disk. + pub pending: std::sync::atomic::AtomicBool, +} + +/// Refusal of a hash already marked invalid. Not a fresh validation result. +const BLOCK_IS_INVALIDATED: &str = "block is invalidated"; + +/// Sidecar of connect failures next to `header.adopt`. Not a schema bump. +const INVALID_FILE: &str = "block.invalid"; +const INVALID_MAGIC: &[u8; 8] = b"rbtcinv1"; +/// Past the cap the oldest kept failure gives way, so the newest survives restart. +const INVALID_FILE_CAP: usize = 4_096; + +fn parse_invalid_file(bytes: &[u8]) -> Option> { + let rest = bytes.strip_prefix(INVALID_MAGIC)?; + let (count, rest) = rest.split_first_chunk::<4>()?; + let n = u32::from_le_bytes(*count) as usize; + if n > INVALID_FILE_CAP || rest.len() != n.checked_mul(32)? { + return None; + } + Some( + rest.chunks_exact(32) + .map(|c| { + let mut raw = [0u8; 32]; + raw.copy_from_slice(c); + BlockHash::from_byte_array(raw) + }) + .collect(), + ) +} + +fn write_invalid_file(path: &std::path::Path, hashes: &[BlockHash]) -> bool { + let mut buf = Vec::with_capacity(12 + 32 * hashes.len()); + buf.extend_from_slice(INVALID_MAGIC); + buf.extend_from_slice(&(hashes.len() as u32).to_le_bytes()); + for h in hashes { + buf.extend_from_slice(h.as_byte_array()); + } + let wrote = rbitcoin_store::file::write_synced_tmp_file(path, |f| { + std::io::Write::write_all(f, &buf).map_err(|e| rbitcoin_store::StoreError::io(path, e)) + }); + if let Err(e) = &wrote { + warn!("chain: write {}: {e}", path.display()); + } + wrote.is_ok() } /// The hashes one `invalidateblock` marked, in height order. `connected` when @@ -249,11 +298,59 @@ pub struct InvalidPath { } impl Invalidated { + /// Load the connect failures in `store/block.invalid`. A missing file is + /// none; one that does not parse is none and is replaced on the next write. #[doc(hidden)] - pub fn new() -> Self { + pub fn open(store: &std::path::Path) -> Self { + let file = store.join(INVALID_FILE); + let failed = match std::fs::read(&file) { + Ok(bytes) => parse_invalid_file(&bytes).unwrap_or_else(|| { + warn!( + "chain: {} does not parse; no stored invalid blocks", + file.display() + ); + Vec::new() + }), + Err(_) => Vec::new(), + }; Self { - set: RwLock::new(HashSet::new()), + set: RwLock::new(failed.iter().copied().collect()), paths: RwLock::new(Vec::new()), + failed: RwLock::new(failed), + file, + pending: std::sync::atomic::AtomicBool::new(false), + } + } + + /// True once `hash`, and every mark before it, is in the file on disk. A + /// write that failed before is tried again. + fn keep_failed(&self, hash: BlockHash) -> bool { + use std::sync::atomic::Ordering; + let mut failed = self.failed.write().unwrap(); + if !failed.contains(&hash) { + if failed.len() >= INVALID_FILE_CAP { + failed.remove(0); + } + failed.push(hash); + } else if !self.pending.load(Ordering::Acquire) { + return true; + } + let wrote = write_invalid_file(&self.file, &failed); + self.pending.store(!wrote, Ordering::Release); + wrote + } + + /// Drop kept failures that are no longer marked invalid. + fn prune_failed(&self) { + let set = self.set.read().unwrap(); + let mut failed = self.failed.write().unwrap(); + let before = failed.len(); + failed.retain(|h| set.contains(h)); + let pending = self.pending.load(std::sync::atomic::Ordering::Acquire); + if failed.len() != before || pending { + let wrote = write_invalid_file(&self.file, &failed); + self.pending + .store(!wrote, std::sync::atomic::Ordering::Release); } } } @@ -578,6 +675,7 @@ impl ChainHub { // Full confirmed-set fill in background (mainnet-scale tips make a // synchronous walk multi-minute). Tip/genesis are seeded immediately. spawn_confirmed_seed(query.clone(), confirmed.clone()); + let invalidated = Invalidated::open(query.store().path()); Self { query, cache: Arc::new(BlockCache::new()), @@ -591,7 +689,7 @@ impl ChainHub { mempool: std::sync::OnceLock::new(), self_weak: std::sync::OnceLock::new(), clock: rbitcoin_consensus::NodeClock::new(), - invalidated: Invalidated::new(), + invalidated, held_bodies: RwLock::new(HeldBodies::new()), precious: RwLock::new(None), fork_tips: RwLock::new(ForkTips::new()), @@ -1351,10 +1449,41 @@ impl ChainHub { self.drop_held(hash); } + /// A block failed connect on a consensus rule. Unlike + /// [`Self::note_invalid_block`], the mark survives restart; mutated bodies, + /// a future timestamp, and store faults are not this. + /// True when the mark is on disk. + pub fn note_failed_connect(&self, hash: BlockHash) -> bool { + self.note_invalid_block(hash); + self.invalidated.keep_failed(hash) + } + + /// Every kept connect failure is in `block.invalid` on disk. + pub fn invalid_marks_on_disk(&self) -> bool { + !self + .invalidated + .pending + .load(std::sync::atomic::Ordering::Acquire) + } + + /// Every hash marked invalid, for a header scan that must not count them. + pub fn invalid_blocks(&self) -> Vec { + self.invalidated + .set + .read() + .unwrap() + .iter() + .copied() + .collect() + } + /// A reconstructed compact with the right header hash and wrong txs must /// not poison later getdata of that hash. #[doc(hidden)] pub fn remember_failed_accept(&self, offered: BlockHash, e: &NetError) { + if matches!(e, NetError::Consensus(m) if m == BLOCK_IS_INVALIDATED) { + return; + } let hash = e .failing_block_hash() .map(BlockHash::from_byte_array) @@ -1365,12 +1494,12 @@ impl ChainHub { return; } if e.failing_block_hash().is_some() { - self.note_invalid_block(hash); + let _ = self.note_failed_connect(hash); return; } if let NetError::Consensus(s) = e { if !s.to_ascii_lowercase().contains("not found") { - self.note_invalid_block(hash); + let _ = self.note_failed_connect(hash); } } } @@ -2313,6 +2442,7 @@ impl ChainHub { inv.remove(h); } } + self.invalidated.prune_failed(); let restored: HashSet = paths .iter() @@ -2553,7 +2683,7 @@ impl ChainHub { return Ok(AcceptOutcome::AlreadyHave); } if self.invalidated.set.read().unwrap().contains(&hash) { - return Err(NetError::Consensus("block is invalidated".into())); + return Err(NetError::Consensus(BLOCK_IS_INVALIDATED.into())); } if rbitcoin_consensus::block_mutated_without_coinbase(&block) { return Err(NetError::Mutated( @@ -2696,7 +2826,7 @@ impl ChainHub { .iter() .any(|b| self.is_block_invalid(&b.block_hash())) { - return Err(NetError::Consensus("block is invalidated".into())); + return Err(NetError::Consensus(BLOCK_IS_INVALIDATED.into())); } for w in blocks.windows(2) { if w[1].header.prev_blockhash != w[0].block_hash() { diff --git a/crates/rbitcoin-net/tests/chain.rs b/crates/rbitcoin-net/tests/chain.rs index a124437cc..804ca8381 100644 --- a/crates/rbitcoin-net/tests/chain.rs +++ b/crates/rbitcoin-net/tests/chain.rs @@ -53,6 +53,38 @@ fn tmp_hub() -> (rbitcoin_query::testutil::TempDir, ChainHub) { rbitcoin_net::chain::tiny_regtest_hub_labeled("chain") } +/// A full `block.invalid` drops its oldest hash for a new failure, so the +/// newest one survives restart. +#[test] +fn a_full_invalid_file_keeps_the_newest_failure() { + let (dir, hub) = tmp_hub(); + drop(hub); + let mut full = b"rbtcinv1".to_vec(); + full.extend_from_slice(&4_096u32.to_le_bytes()); + for i in 0..4_096u32 { + let mut raw = [0u8; 32]; + raw[..4].copy_from_slice(&i.to_le_bytes()); + raw[31] = 0xaa; + full.extend_from_slice(&raw); + } + let marks = dir.path().join("block.invalid"); + std::fs::write(&marks, &full).unwrap(); + let q = rbitcoin_query::Query::open_or_create_tiny(dir.path()).unwrap(); + let hub = ChainHub::new(q, ChainParams::regtest(), Milestone::NONE); + let newest = BlockHash::from_byte_array([0xbb; 32]); + hub.note_failed_connect(newest); + let kept = std::fs::read(&marks).unwrap(); + assert_eq!(kept.len(), full.len(), "the file stays at its cap"); + assert!( + kept.windows(32).any(|w| w == newest.as_byte_array()), + "the newest failure is kept" + ); + assert!( + !kept.windows(32).any(|w| w == &full[12..44]), + "the oldest one gives way" + ); +} + #[test] fn ancestor_walk_stops_at_cap() { let hashes: Vec = (0..9u8) @@ -4090,12 +4122,36 @@ fn reconsider_skips_a_branch_that_fails() { rbitcoin_consensus::grind_regtest_pow(&mut bad5.header); let bad5_hash = bad5.block_hash(); assert!(hub.accept_received_block(bad5).is_err()); + let marks = dir.path().join("block.invalid"); + let kept = |marks: &std::path::Path| { + std::fs::read(marks) + .unwrap() + .windows(32) + .any(|w| w == bad5_hash.as_byte_array()) + }; + assert!(kept(&marks), "a connect failure is kept across restart"); let a5 = mine_distinct(a[3], t0 + 31, 5, &[bad5_hash]); let a6 = mine(a5.block_hash(), t0 + 32, 6); hub.accept_block(a5).unwrap(); hub.accept_block(a6).unwrap(); hub.invalidate_block(bad5_hash).unwrap(); hub.reconsider_block(bad5_hash).unwrap(); + assert!(!kept(&marks), "reconsiderblock forgets the kept failure"); + let tip_block = hub + .block_at_height(hub.tip_height().unwrap()) + .unwrap() + .unwrap(); + let tip_hash = tip_block.block_hash(); + hub.invalidate_block(tip_hash).unwrap(); + assert!(hub.accept_received_block(tip_block).is_err()); + assert!( + !std::fs::read(&marks) + .unwrap() + .windows(32) + .any(|w| w == tip_hash.as_byte_array()), + "a block refused as already invalidated is not a connect failure" + ); + hub.reconsider_block(tip_hash).unwrap(); assert!( !hub.fork_tips.read().unwrap().iter().any(|h| h == bad5_hash), "a block that never connected is not kept as a fork tip" diff --git a/crates/rbitcoin-net/tests/peer_header_sync_journey.rs b/crates/rbitcoin-net/tests/peer_header_sync_journey.rs index 43df6b8a4..e5e26c578 100644 --- a/crates/rbitcoin-net/tests/peer_header_sync_journey.rs +++ b/crates/rbitcoin-net/tests/peer_header_sync_journey.rs @@ -68,6 +68,7 @@ fn slot_at(id: usize, ip: Ipv4Addr, height: u32) -> (PeerSlot, mpsc::UnboundedRe first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, }, diff --git a/crates/rbitcoin-node/src/run.rs b/crates/rbitcoin-node/src/run.rs index b7a8c4766..d8d6d3ecf 100644 --- a/crates/rbitcoin-node/src/run.rs +++ b/crates/rbitcoin-node/src/run.rs @@ -1752,7 +1752,15 @@ pub fn stored_header_chain_heavier_than_tip(hub: &ChainHub) -> bool { return true; } }; - match hub.query.has_heavier_header_chain_than_tip(rec.hash, tip_h) { + let invalid: Vec<[u8; 32]> = hub + .invalid_blocks() + .into_iter() + .map(bitcoin::hashes::Hash::to_byte_array) + .collect(); + match hub + .query + .has_heavier_header_chain_than_tip(rec.hash, tip_h, &invalid) + { Ok(heavier) => heavier, Err(e) => { warn!("node: startup header scan failed: {e} — catch-up"); diff --git a/crates/rbitcoin-query/src/lib.rs b/crates/rbitcoin-query/src/lib.rs index 67152bb1b..d6b4d607e 100644 --- a/crates/rbitcoin-query/src/lib.rs +++ b/crates/rbitcoin-query/src/lib.rs @@ -2416,13 +2416,15 @@ impl Query { /// /// The sibling search climbs to genesis. IBD seeding keeps the 32-hop cap; /// startup uses this so a deep stored fork still stays in catch-up. - /// A tip hash with no header row is an empty path (`Ok(false)`). + /// A tip hash with no header row is an empty path (`Ok(false)`). A path + /// through an `exclude` hash (a known-invalid block) does not count. pub fn has_heavier_header_chain_than_tip( &self, tip_hash: [u8; 32], tip_height: u32, + exclude: &[[u8; 32]], ) -> Result { - let path = self.resume_work_path_limited(tip_hash, tip_height, 1, &[], u32::MAX)?; + let path = self.resume_work_path_limited(tip_hash, tip_height, 1, exclude, u32::MAX)?; Ok(!path.is_empty()) } diff --git a/crates/rbitcoin-test/tests/end_of_ibd.rs b/crates/rbitcoin-test/tests/end_of_ibd.rs index 985985b20..3398ee0b9 100644 --- a/crates/rbitcoin-test/tests/end_of_ibd.rs +++ b/crates/rbitcoin-test/tests/end_of_ibd.rs @@ -16,7 +16,9 @@ use rbitcoin_node::{run_p2p, NodeConfig}; use rbitcoin_primitives::{Fk, Height, Network}; use rbitcoin_query::Query; use rbitcoin_store::{script_hash, ScriptHashRecord}; -use rbitcoin_test::mine::{mine_regtest_block, mine_regtest_block_at, regtest_genesis}; +use rbitcoin_test::mine::{ + coinbase_tx, mine_regtest_block, mine_regtest_block_at, regtest_genesis, +}; use rbitcoin_test::TestDatadir; use serde_json::{json, Value}; use std::collections::BTreeSet; @@ -1185,6 +1187,145 @@ async fn work_fork_journey() { light.shutdown().await; } +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn end_of_ibd_invalid_heavier_fork() { + let wall = llvm_cov_wall(60, 180); + tokio::time::timeout(wall, invalid_heavier_fork_journey()) + .await + .expect("end_of_ibd_invalid_heavier_fork wall"); +} + +/// The subsidy halves at this height for the syncer and the honest miners. +const SPLIT_HALVING: u32 = 20; + +fn split_params() -> ChainParams { + let mut params = ChainParams::regtest(); + params.overlay_subsidy_halving_interval(SPLIT_HALVING); + params +} + +/// Extend `chain` to `top`. Blocks at or past [`SPLIT_HALVING`] pay 25 BTC +/// when `halved`; a miner that never halves pays 50 BTC there, which the +/// syncer rejects as coinbase excess. `skew` keeps sibling hashes apart. +fn extend_split(chain: &mut Vec, top: u32, halved: bool, skew: u32) { + while (chain.len() as u32) <= top { + let height = chain.len() as u32; + let prev = chain.last().unwrap(); + let time = prev.header.time.saturating_add(1).saturating_add(skew); + let mut block = mine_regtest_block(prev.block_hash(), time, height, vec![]); + if halved && height >= SPLIT_HALVING { + block.txdata[0] = coinbase_tx(height, Amount::from_sat(25_0000_0000)); + block.header.merkle_root = block.compute_merkle_root().unwrap(); + rbitcoin_consensus::grind_regtest_pow(&mut block.header); + } + chain.push(block); + } +} + +/// Sync `dir` from `peers`; IBD must return at `tip`. +async fn sync_to(dir: &Path, peers: &[SocketAddr], tip: &bitcoin::Block, height: u32, beat: &str) { + let node = start_node(dir, split_params()).await; + let addrs: Vec = peers.iter().copied().map(NetAddr::Ip).collect(); + tokio::time::timeout( + Duration::from_secs(20), + rbitcoin_ibd::sync(&node, &addrs, IbdConfig::for_test()), + ) + .await + .unwrap_or_else(|_| { + panic!( + "{beat}: IBD did not return (tip {:?}, want {height})", + node.tip_height() + ) + }) + .unwrap_or_else(|e| panic!("{beat}: sync: {e}")); + assert_eq!(node.tip_height(), Some(height), "{beat}: tip height"); + assert_eq!( + node.hub.tip_hash(), + Some(tip.block_hash()), + "{beat}: tip hash" + ); + node.shutdown().await; +} + +async fn invalid_heavier_fork_journey() { + // `split` never halves, so from SPLIT_HALVING on its chain is invalid to + // the syncer while it carries the most header work. `honest` forks below + // the halving; `at_fork` forks at the first invalid block. + let mut split_blocks = vec![regtest_genesis()]; + extend_split(&mut split_blocks, 40, false, 0); + let mut honest_blocks = split_blocks[..=14].to_vec(); + extend_split(&mut honest_blocks, 17, true, 1_000); + let mut at_fork_blocks = split_blocks[..SPLIT_HALVING as usize].to_vec(); + extend_split(&mut at_fork_blocks, 30, true, 2_000); + + let split_dir = TestDatadir::new().unwrap(); + let honest_dir = TestDatadir::new().unwrap(); + let at_fork_dir = TestDatadir::new().unwrap(); + let split = start_node(split_dir.path().as_path(), ChainParams::regtest()).await; + let honest = start_node(honest_dir.path().as_path(), split_params()).await; + let at_fork = start_node(at_fork_dir.path().as_path(), split_params()).await; + load_chain(&split, &split_blocks); + load_chain(&honest, &honest_blocks); + load_chain(&at_fork, &at_fork_blocks); + + // The split chain's valid prefix (to 19) has more work than honest (17). + let syncer = TestDatadir::new().unwrap(); + let pair = [split.local_addr, honest.local_addr]; + let valid_top = &split_blocks[SPLIT_HALVING as usize - 1]; + sync_to( + syncer.path().as_path(), + &pair, + valid_top, + SPLIT_HALVING - 1, + "valid prefix", + ) + .await; + + // Once honest outgrows that prefix, a syncer that confirmed the prefix + // leaves it for the honest chain. + let from = honest_blocks.len() as u32; + extend_split(&mut honest_blocks, 30, true, 1_000); + for (height, block) in honest_blocks.iter().enumerate().skip(from as usize) { + honest.ingest_block(height as u32, block.clone()).unwrap(); + } + let rewinds = TestDatadir::new().unwrap(); + sync_to( + rewinds.path().as_path(), + &pair, + honest_blocks.last().unwrap(), + 30, + "heavier honest", + ) + .await; + + // The syncer that kept the prefix restarts. The invalid block stays + // invalid, so the split headers stored above it are not a path. + let honest_tip = honest_blocks.last().unwrap(); + sync_to(syncer.path().as_path(), &pair, honest_tip, 30, "restart").await; + sync_to( + syncer.path().as_path(), + &[honest.local_addr], + honest_tip, + 30, + "restart with only honest", + ) + .await; + + // A fork at the first invalid block: nothing on the split side to keep. + let fresh = TestDatadir::new().unwrap(); + sync_to( + fresh.path().as_path(), + &[split.local_addr, at_fork.local_addr], + at_fork_blocks.last().unwrap(), + 30, + "fork at the invalid block", + ) + .await; + split.shutdown().await; + honest.shutdown().await; + at_fork.shutdown().await; +} + fn startup_cfg(dir: &Path, miner: SocketAddr, max_tip_age: u64) -> NodeConfig { let mut cfg = NodeConfig::default() .with_datadir(dir) @@ -1224,7 +1365,7 @@ fn regtest_chain(blocks: u32) -> Vec { } /// One difficulty-1 header under genesis. Its work beats a long easy regtest chain. -fn plant_deep_heavy_header(q: &Query) { +fn plant_deep_heavy_header(q: &Query) -> [u8; 32] { use rbitcoin_store::{block_header_hash, HeaderRecord}; let (gfk, genesis) = q.header_at_height(Height(0)).unwrap().unwrap(); let version = 1i32; @@ -1245,6 +1386,7 @@ fn plant_deep_heavy_header(q: &Query) { weight: 0, }) .unwrap(); + hash } type StartupLogs = Vec<(rbitcoin_log::Level, String)>; @@ -1361,7 +1503,7 @@ async fn startup_tip_follow_journey() { let deep_dir = TestDatadir::new().unwrap(); let deep_node = start_miner(deep_dir.path().as_path()).await; load_chain(&deep_node, &deep); - plant_deep_heavy_header(&deep_node.hub.query); + let heavy = plant_deep_heavy_header(&deep_node.hub.query); deep_node.hub.query.flush().unwrap(); deep_node.shutdown().await; @@ -1375,6 +1517,22 @@ async fn startup_tip_follow_journey() { let logs = stop_logged(rpc, node).await; assert_startup(&logs, false); + // The same heavier header, once a connect failure marked it invalid, is + // not a chain to catch up to. + let mut marks = b"rbtcinv1".to_vec(); + marks.extend_from_slice(&1u32.to_le_bytes()); + marks.extend_from_slice(&heavy); + std::fs::write(deep_dir.path().join("store").join("block.invalid"), marks).unwrap(); + let node = spawn_run_p2p_logs(startup_cfg( + deep_dir.path().as_path(), + deep_miner.local_addr, + u64::MAX, + )); + let rpc = published_listener(deep_dir.path().as_path(), "rpc", Duration::from_secs(30)).await; + wait_height(rpc, 40).await; + let logs = stop_logged(rpc, node).await; + assert_startup(&logs, true); + deep_miner.shutdown().await; miner.shutdown().await; } diff --git a/docs/ibd-memory.md b/docs/ibd-memory.md index 984471d53..216c389f6 100644 --- a/docs/ibd-memory.md +++ b/docs/ibd-memory.md @@ -56,8 +56,9 @@ pres and **not** the raw bytes. Reorg gather that wants wire re-encodes. | **SH catalog runs** | leftover `scripthash.runs` discarded at tip (unsorted collect does not write them) | write-behind / discard; not during Direct confirm | | **SH unsorted collect / pack** | Two `txout` scans. All extract phases share `sh_extract_workers()` = min(CPUs, max(1, free RAM / 1.5 GiB)); `RBITCOIN_SH_MERGE_WORKERS` override. **Collect maps** use that same 1.5 GiB **per worker** as a cap (not a prefault): each worker owns a contiguous create-fk span and `n_shards` unsized identity maps (`HashMap`, `0` = multi; 64 B/key estimate). After each 64 k-fk loc/body batch, spill the largest shard map while `sum(estimate) >= 1.5 GiB`. Status `scanned=` is finished fks (not max fk). Spills, including the end flush, go through one writer and a 1-slot queue so encode+tmp+rename is serial. Fuse8 mmap on pass 2 is process-wide, not subtracted from the cap. `keys/NN/` is a **directory** of `SHKSP01` spill files (`000000`, …; n_multi key16s already `0`, then first-fk-sorted uleb(delta)‖key16 singles; tmp+rename, no `sync_all`). Merge folds those spill files only into one identity map, one walk to `scripthash.head/NN` pack8 and `multi/NN.fuse8` (file, not kept resident for other shards), unlinks `keys/NN/`. The folded map is consumed into the pack8 records and dropped before fuse8 and BDZ. A previous `DONE` / 24 B `NN` layout, or `keys/NN` as a file, with no valid `DONE.keys` is deleted and pass 1 restarts. A spill whose magic is not `SHKSP01` is Corrupt. Pass 2 keeps fuse8 mmap only (no BDZ). Same static fk spans; fuse-hit creates append `fk` to per-worker `HashMap>` (skip `vec.last() == fk`). Estimate `80 × n_keys + 8 × n_fks`; spill-largest while over 1.5 GiB. `post/NN/` is a **directory** of `SHPST01` spill files (`000000`, …; `u32 n_keys` then `key16 ‖ uleb(n) ‖ n × uleb(fk − prev)`; tmp+rename, no `sync_all`). A `post/NN` file, or a spill whose magic is not `SHPST01`, is Corrupt. **Pack fold** (named, after collect maps drop): one shard ~1.2 M multi keys × ~30 fks ≈ 0.3–0.5 GiB; 8 pack workers a few GiB. Fold spills into one map, then `MphfHead::open` + `slot_for_key16` + 2+ body + `rewrite_val_slots`; `len == 1` after fold is `fp_singles` (pass-1 `inline_one` stays). Unlink `post/NN/`. Packed 2-bit `g` ~8 MiB per pack worker + 16 MiB `body_buf`. Inner loc/body batch 65 536 fks inside each worker span (not a steal grain); body IO 16 MiB | Tip finalize. Extract under `scripthash.unsorted/{keys,multi,post}` | | **`tx.head` wipe-rebuild workers** | min(CPUs, host free RAM / 1 GiB, range count); floor 1. Same free-RAM probe as SH. Matches BDZ peel+keys+g peak at default 2²⁵. **Not** the SH extract 1.5 GiB cap. Env `RBITCOIN_TX_HEAD_REBUILD_WORKERS` override (`1` = serial) | Empty/wipe `tx.head` rebuild from `txid.body`. Logs `workers=` `free_GiB=` | +| **Per-peer chain record** | `PeerChain` per IBD slot: the newest announced run above the confirmed tip, at most 2,000 hashes; at most 1,024 `notfound` hashes, each kept 30 s; one split, one chain end, one path block the peer showed after a probe, and the height a rewind left the record stale from. Header-walk `invalid_tips` (tips of chains refused for building on a rejected block): 1,024, oldest first out | Dropped with the slot or IBD state. Not written to disk | | **Ordered work path** | `MAX_ORDERED_HEADERS` | `IbdWorkState::hygiene` | -| **Header-walk checkpoints** | One checkpoint per accepted 2,000-header reply, and those headers are written. The ordered queue is a window of at most 64,000 over headers already stored. One peer, the lowest time to a first block byte among peers that have not failed the walk, is asked only when the next header is not in the store. While the queue has room the locator starts at the stored tip, then the candidate, then a lighter fork still being weighed, so the reply can be enqueued. A full queue asks from the candidate. Block download uses the other peers. Above the work floor a short reply that does not extend the candidate moves the reservation; a full 2,000-header window does not, so a lighter fork can still catch up. A block inv from another peer is one challenge, asked even when that peer's connect-time start height is below the walk: if the reply beats the candidate they become the header peer, and if it does not they leave the walk and the previous peer stays. An inv from a session with no header slot does not retire the reserved peer. One missed ask moves the reservation; a second miss disconnects. Less work does not disconnect. Each checkpoint holds hash, height, work, that reply's last 80-byte header, up to 11 timestamps, and the difficulty period (one period-start header plus the last non-limit `nBits`) so a fork or rewind can retarget. One competing chain is kept the same way until it loses or replaces the candidate. Mainnet stays under about 250 KB. Queue maps stay capped with the queue. Side headers below the work floor are one hash per peer. More than 4,000 disconnects that peer. Abandoned look-ahead tips are cleared at 1,024 and when a heavier chain is adopted. | Dropped with IBD state. `header.adopt` is the sidecar, not a resident copy. The file is deleted once the confirmed chain proves the anchor | +| **Header-walk checkpoints** | One checkpoint per accepted 2,000-header reply, and those headers are written. The ordered queue is a window of at most 64,000 over headers already stored. One peer, the lowest time to a first block byte among peers that have not failed the walk, is asked only when the next header is not in the store. While the queue has room the locator starts at the stored tip, then the candidate, then a lighter fork still being weighed, so the reply can be enqueued. A full queue asks from the candidate. Block download uses the other peers, each asked only for blocks its chain showed (see **Per-peer chain record**); the header peer takes a block no other peer shows. Above the work floor a short reply that does not extend the candidate moves the reservation; a full 2,000-header window does not, so a lighter fork can still catch up. A block inv from another peer is one challenge, asked even when that peer's connect-time start height is below the walk: if the reply beats the candidate they become the header peer, and if it does not they leave the walk and the previous peer stays. An inv from a session with no header slot does not retire the reserved peer. One missed ask moves the reservation; a second miss disconnects. Less work does not disconnect. Each checkpoint holds hash, height, work, that reply's last 80-byte header, up to 11 timestamps, and the difficulty period (one period-start header plus the last non-limit `nBits`) so a fork or rewind can retarget. One competing chain is kept the same way until it loses or replaces the candidate. Mainnet stays under about 250 KB. Queue maps stay capped with the queue. Side headers below the work floor are one hash per peer. More than 4,000 disconnects that peer. Abandoned look-ahead tips are cleared at 1,024 and when a heavier chain is adopted. | Dropped with IBD state. `header.adopt` is the sidecar, not a resident copy. The file is deleted once the confirmed chain proves the anchor | Tests that need a clean process must call these **same** entry points (or drop the owning `Query` / pipeline), not a secret test-only free-all that masks production