From b0d02edaa8b231980e7fc1d7dde8360fcc3ad2ab Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 14:40:21 -0400 Subject: [PATCH 01/17] net: give the header peer a block no other peer has IBD never sent block getdata to the reserved header peer while another peer was connected, and a peer that answered `notfound` was asked again for the same hash. When only the header peer had a heavier chain, every getdata went to a peer that kept answering `notfound`, and IBD stalled. Each IBD slot now remembers the hashes it answered `notfound` until it announces more (a headers message or a block inv). A peer is not asked for such a hash, and the reserved header peer is asked when no other live peer may have the block. One `may_serve` check in `issue_batch` replaces the blanket reserved-peer exclusion. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 4 +-- changelog.d/ibd-invalid-heavier.md | 8 +++++ crates/rbitcoin-ibd/src/ibd/assign.rs | 5 +-- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 12 ++++++- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 23 +++++-------- crates/rbitcoin-ibd/src/ibd/mod.rs | 2 ++ crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 33 +++++++++++++++++++ crates/rbitcoin-ibd/src/ibd/peer_io.rs | 3 ++ crates/rbitcoin-ibd/tests/ibd/assign.rs | 1 + crates/rbitcoin-ibd/tests/ibd/dial.rs | 1 + .../tests/ibd/events/confirm_reject_tests.rs | 15 +++++++-- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 30 +++++++++++++++++ crates/rbitcoin-ibd/tests/ibd/peer_io.rs | 1 + .../tests/peer_header_sync_journey.rs | 1 + 14 files changed, 117 insertions(+), 22 deletions(-) create mode 100644 changelog.d/ibd-invalid-heavier.md create mode 100644 crates/rbitcoin-ibd/src/ibd/peer_chain.rs diff --git a/TESTING.md b/TESTING.md index e7c1c9004..12c228f40 100644 --- a/TESTING.md +++ b/TESTING.md @@ -200,7 +200,7 @@ absolute path. Keep default-tier fixtures small: on tmpfs, test bytes are RAM. | Remining 100-block maturity pads with `confirm_wire_run` | `pad_empty_from` / `build_mature_regtest_with_spend` **once per binary journey** (not once per skinny test) | | Wall-time multi-round microbenches in default suite | Deterministic structure / chunk-load asserts; demote wall arms to `#[ignore]` | -**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer is not banned for that hash. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. +**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer stays connected and is not asked for that hash again until it announces more. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. **Speed / reliability (default suite):** prefer `pad_empty_from` / `build_mature_regtest_with_spend` **once per journey** (tx_relay live hub, Electrum protocol, core_analogs assumevalid+mempool) over remine pads. `pin_compose_multi_pack_timed` keeps functional + layout/covered short-circuit gates (multi-ms floor); sticky vs cold assemble is log-only (not a hard timing assert). Schema-13 wire rebuild must stamp create identity from `txid.body` — zero batch identity is treated as missing (regression covered by the spend reconstruct in `consensus_mature_chain_spend_reconstruct_and_scripthash` + multi-vout confirm scenarios). Coverage vs speed: prefer **one** scenario at the real entry over N micro-opens that only paint lines; when adding coverage for reduce/materialize, use a **tiny** target, not production stream depth. @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md new file mode 100644 index 000000000..d86c115ff --- /dev/null +++ b/changelog.d/ibd-invalid-heavier.md @@ -0,0 +1,8 @@ +Fixed + +- **IBD asks the header peer for a block no other peer has.** The peer that + served the headers was never asked for blocks while another peer was + connected, and a peer that answered `notfound` was asked again for the + same block. A heavier chain only the header peer had stalled IBD. A peer + is not asked again for a block it answered `notfound` until it announces + more. diff --git a/crates/rbitcoin-ibd/src/ibd/assign.rs b/crates/rbitcoin-ibd/src/ibd/assign.rs index 5df539cce..70af07cec 100644 --- a/crates/rbitcoin-ibd/src/ibd/assign.rs +++ b/crates/rbitcoin-ibd/src/ibd/assign.rs @@ -242,7 +242,7 @@ pub fn assign_work_ordered( ) { let t0 = Instant::now(); let mut issued = 0u64; - let alive = super::header_walk::peers_for_blocks(st); + let alive: Vec = st.slots.iter().filter(|s| s.alive).map(|s| s.id).collect(); if alive.is_empty() { return; } @@ -816,7 +816,7 @@ pub fn issue_batch( let batch: Vec = batch .into_iter() .filter(|h| { - if st.slots[idx].in_flight.contains(h) { + if st.slots[idx].in_flight.contains(h) || !super::header_walk::may_serve(st, pid, h) { return false; } if st.inflight.contains_key(h) { @@ -1315,6 +1315,7 @@ pub fn steal_hung_densify( pid != owner && peer_bps(&st.slots, pid) > owner_bps && !st.inflight.get(&h).is_some_and(|r| r.holds(pid)) + && super::header_walk::may_serve(st, pid, &h) }) .collect(); if faster.is_empty() { diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index e0f67f16b..8811eef15 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -467,6 +467,7 @@ pub fn apply_peer_event( PeerEvent::NotFound { peer, hashes } => apply_notfound(st, peer, hashes), PeerEvent::BlocksInv { peer, hashes } => { if !hashes.is_empty() { + note_announcement(st, peer); super::header_walk::note_block_inv(st, peer); } } @@ -640,7 +641,9 @@ pub fn plan_header_batch( challenger_work_before: st.header_walk.challenger.as_ref().map(|ch| ch.tip.work), outer_len: headers.len(), }; - if mark.outer_len == 0 { + if mark.outer_len > 0 { + note_announcement(st, peer); + } else { super::header_walk::settle_walk_peer( st, hub, @@ -882,12 +885,19 @@ pub fn apply_block_decode_failed(st: &mut IbdWorkState, peer: usize, hash: Block } #[doc(hidden)] +fn note_announcement(st: &mut IbdWorkState, peer: usize) { + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.note_announcement(); + } +} + pub fn apply_notfound(st: &mut IbdWorkState, peer: usize, hashes: Vec) { note_block_progress(&mut st.slots, peer); let mut freed = Vec::new(); if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { for h in &hashes { s.track_remove(h); + s.chain.note_notfound(*h); let empty = st .inflight .get_mut(h) diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index a03cb7098..1b4dc04a5 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -612,22 +612,15 @@ pub fn note_reservation( } } -/// Peers that may receive block getdata. The reserved header peer is left -/// out while another peer is connected. A single peer still does both. -pub fn peers_for_blocks(st: &IbdWorkState) -> Vec { - let alive: Vec = st.slots.iter().filter(|s| s.alive).map(|s| s.id).collect(); - let Some(reserved) = st.header_walk.reserved else { - return alive; - }; - if alive.len() <= 1 { - return alive; - } - let rest: Vec = alive.iter().copied().filter(|&id| id != reserved).collect(); - if rest.is_empty() { - alive - } else { - rest +/// Whether block `hash` may be asked of `peer`. A peer is not asked for a +/// block it answered `notfound`. The reserved header peer is asked only when +/// no other live peer may have it, so a single peer still does both. +pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { + let has = |s: &super::peer_io::PeerSlot| s.alive && !s.chain.lacks(hash); + if !st.slots.iter().any(|s| s.id == peer && has(s)) { + return false; } + st.header_walk.reserved != Some(peer) || !st.slots.iter().any(|s| s.id != peer && has(s)) } /// Inputs for one main-loop header poll. The loop owns the cadence clock. diff --git a/crates/rbitcoin-ibd/src/ibd/mod.rs b/crates/rbitcoin-ibd/src/ibd/mod.rs index 4c5327378..0a9c9d555 100644 --- a/crates/rbitcoin-ibd/src/ibd/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/mod.rs @@ -37,6 +37,8 @@ pub mod header_walk; #[doc(hidden)] pub mod path; #[doc(hidden)] +pub mod peer_chain; +#[doc(hidden)] pub mod peer_io; #[doc(hidden)] pub mod perf_log; diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs new file mode 100644 index 000000000..88035cb98 --- /dev/null +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -0,0 +1,33 @@ +//! What one IBD peer has shown of its chain. +//! +//! Core asks a peer only for blocks on that peer's best known chain +//! (`pindexBestKnownBlock`). A peer that answered `notfound` for a block is +//! not asked for it again. + +use bitcoin::BlockHash; +use std::collections::HashSet; + +/// `notfound` hashes kept per peer. Past the cap a peer may be asked again. +const LACKS_CAP: usize = 1024; + +#[derive(Debug, Default)] +pub struct PeerChain { + lacks: HashSet, +} + +impl PeerChain { + pub fn note_notfound(&mut self, hash: BlockHash) { + if self.lacks.len() < LACKS_CAP { + self.lacks.insert(hash); + } + } + + pub fn lacks(&self, hash: &BlockHash) -> bool { + self.lacks.contains(hash) + } + + /// A headers message or a block inv: the peer may have those blocks now. + pub fn note_announcement(&mut self) { + self.lacks.clear(); + } +} diff --git a/crates/rbitcoin-ibd/src/ibd/peer_io.rs b/crates/rbitcoin-ibd/src/ibd/peer_io.rs index e62c3c2c0..48e303b8d 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_io.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_io.rs @@ -160,6 +160,8 @@ pub struct PeerSlot { /// All streamed wire bytes. Reader-only `fetch_add`. Not the stall clock. pub bytes_rx_total: Arc, pub rate: PeerRate, + /// What this peer has shown of its chain. IBD main thread only. + pub chain: super::peer_chain::PeerChain, pub alive: bool, pub task: JoinHandle<()>, } @@ -1061,6 +1063,7 @@ pub async fn spawn_peer( rate.note_connected(connected_ms); rate }, + chain: Default::default(), alive: true, task, }) diff --git a/crates/rbitcoin-ibd/tests/ibd/assign.rs b/crates/rbitcoin-ibd/tests/ibd/assign.rs index 9fb8a8d79..803e738df 100644 --- a/crates/rbitcoin-ibd/tests/ibd/assign.rs +++ b/crates/rbitcoin-ibd/tests/ibd/assign.rs @@ -70,6 +70,7 @@ pub(crate) fn dummy_slot(id: usize) -> PeerSlot { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/dial.rs b/crates/rbitcoin-ibd/tests/ibd/dial.rs index 1d4ae0e6a..d123d08d3 100644 --- a/crates/rbitcoin-ibd/tests/ibd/dial.rs +++ b/crates/rbitcoin-ibd/tests/ibd/dial.rs @@ -87,6 +87,7 @@ fn dummy_slot(id: usize, a: SocketAddr, alive: bool) -> PeerSlot { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs index 37cebed6f..0a7994454 100644 --- a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs +++ b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs @@ -717,6 +717,7 @@ fn apply_peer_event_body_and_control_surface() { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } @@ -840,10 +841,13 @@ fn apply_peer_event_body_and_control_surface() { let mut room = 4usize; let mut issued = 0u64; assert!( - rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, notfound_hash, &mut room, &mut issued), + !rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, notfound_hash, &mut room, &mut issued), + "a peer is not asked again for a block it does not have" + ); + assert!( + rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, h(11), &mut room, &mut issued), "notfound does not ban the peer" ); - assert!(st.slots[0].in_flight.contains(¬found_hash)); // Headers: attach height from tip parent and order. let hdr = dummy_header(gen, 1); @@ -862,6 +866,11 @@ fn apply_peer_event_body_and_control_surface() { ); assert!(st.known_headers.contains(&hash)); assert!(st.ordered_set.contains(&hash) || st.hash_height.contains_key(&hash)); + assert!( + rbitcoin_ibd::ibd::assign::issue_one(&mut st, 1, notfound_hash, &mut room, &mut issued), + "a peer that announces again may be asked again" + ); + assert!(st.slots[0].in_flight.contains(¬found_hash)); // Empty headers with lag → keep headers_done false. st.max_peer_height = 100; @@ -1037,6 +1046,7 @@ fn apply_peer_event_block_framed_bq_horizon_and_headers_done() { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } @@ -4060,6 +4070,7 @@ fn coord_loop_body_and_reject_do_not_read_headers() { first_data_ms: 0, bytes_rx_total: std::sync::Arc::new(std::sync::atomic::AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 4227e3aa8..bca711d8d 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -57,6 +57,7 @@ fn slot(id: usize) -> (PeerSlot, mpsc::UnboundedReceiver) { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, }, @@ -1435,6 +1436,35 @@ fn two_peers_reserve_the_header_walk() { assert!(matches!(slow_rx.try_recv(), Ok(PeerCmd::GetData { .. }))); assert!(fast_rx.try_recv().is_err()); + apply_peer_event( + &mut st, + &hub, + PeerEvent::NotFound { + peer: 0, + hashes: vec![hash], + }, + &AtomicU32::new(0), + &mut AddrMan::new(), + SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 1), + None, + ); + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!( + st.slots[1].in_flight.contains(&hash), + "the header peer takes a block the other peer does not have" + ); + assert!(slow_rx.try_recv().is_err(), "notfound is not asked again"); + assert!(matches!(fast_rx.try_recv(), Ok(PeerCmd::GetData { .. }))); + let _ = st.slots[1].track_remove(&hash); + st.inflight.remove(&hash); + st.ordered.clear(); st.ordered_set.clear(); st.height_to_hash.remove(&1); diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_io.rs b/crates/rbitcoin-ibd/tests/ibd/peer_io.rs index e6c4c38cf..d0aff5c79 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_io.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_io.rs @@ -42,6 +42,7 @@ fn dummy_slot(id: usize) -> PeerSlot { first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, } diff --git a/crates/rbitcoin-net/tests/peer_header_sync_journey.rs b/crates/rbitcoin-net/tests/peer_header_sync_journey.rs index 43df6b8a4..e5e26c578 100644 --- a/crates/rbitcoin-net/tests/peer_header_sync_journey.rs +++ b/crates/rbitcoin-net/tests/peer_header_sync_journey.rs @@ -68,6 +68,7 @@ fn slot_at(id: usize, ip: Ipv4Addr, height: u32) -> (PeerSlot, mpsc::UnboundedRe first_data_ms: 0, bytes_rx_total: Arc::new(AtomicU64::new(0)), rate: Default::default(), + chain: Default::default(), alive: true, task, }, From 5a83f59fd5cac92c4645120b44030edf8d6caa07 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 14:45:48 -0400 Subject: [PATCH 02/17] net: retire an IBD header candidate whose block fails consensus When the most-work header chain held a consensus-invalid block, IBD rejected the block but kept that chain as the walk candidate and the download path. A lighter valid chain another peer served was never fetched, and IBD stayed below the invalid block (issue #1082). On a consensus-invalid body the walk now drops the path above the block's parent (queue, body queue, confirm plan, densify cursor) and moves the candidate back to that parent. A batch that holds or builds on a rejected block is refused and its peer leaves the walk, so peers are asked again from the parent and a heavier valid chain is adopted, rewinding confirmed blocks when it forks below them. A challenger that forks at or below the parent with more work is promoted. Each IBD slot also remembers where its announced headers left the download path, so a peer is not asked for path blocks above that point and the header peer takes a block only it has. `adopt_heavier` and the retire share `cut_back_to`. The live journey `end_of_ibd_invalid_heavier_fork` stages the split with three miners. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 3 +- changelog.d/ibd-invalid-heavier.md | 11 +- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 29 ++++ crates/rbitcoin-ibd/src/ibd/header_walk.rs | 161 ++++++++++++++++++--- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 52 ++++++- crates/rbitcoin-test/tests/end_of_ibd.rs | 130 ++++++++++++++++- 6 files changed, 358 insertions(+), 28 deletions(-) diff --git a/TESTING.md b/TESTING.md index 12c228f40..196475f14 100644 --- a/TESTING.md +++ b/TESTING.md @@ -200,7 +200,7 @@ absolute path. Keep default-tier fixtures small: on tmpfs, test bytes are RAM. | Remining 100-block maturity pads with `confirm_wire_run` | `pad_empty_from` / `build_mature_regtest_with_spend` **once per binary journey** (not once per skinny test) | | Wall-time multi-round microbenches in default suite | Deterministic structure / chunk-load asserts; demote wall arms to `#[ignore]` | -**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer stays connected and is not asked for that hash again until it announces more. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. +**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). `end_of_ibd_invalid_heavier_fork` 60s wall (180s under llvm-cov), each sync 20s. A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer stays connected and is not asked for that hash again until it announces more. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. **Speed / reliability (default suite):** prefer `pad_empty_from` / `build_mature_regtest_with_spend` **once per journey** (tx_relay live hub, Electrum protocol, core_analogs assumevalid+mempool) over remine pads. `pin_compose_multi_pack_timed` keeps functional + layout/covered short-circuit gates (multi-ms floor); sticky vs cold assemble is log-only (not a hard timing assert). Schema-13 wire rebuild must stamp create identity from `txid.body` — zero batch identity is treated as missing (regression covered by the spend reconstruct in `consensus_mature_chain_spend_reconstruct_and_scripthash` + multi-vout confirm scenarios). Coverage vs speed: prefer **one** scenario at the real entry over N micro-opens that only paint lines; when adding coverage for reduce/materialize, use a **tiny** target, not production stream depth. @@ -425,6 +425,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `end_of_ibd_follow` | P2P (**default**) | Miner plus `--sh-index` syncer. One mature regtest: coinbases pay script A, one spend pays script B. IBD reaches that tip, leaves IBD, and Electrum history matches. The next block tip-follows. Restart does not rewrite the scripthash pack mark; one more block arrives by write-behind. Cancelling IBD once the height is below the miner, then `run_p2p`, finishes the same tip and history. With the syncer caught up, dropping the miner leaves tip follow (not IBD). A partial datadir whose miner is down does not open Electrum and stays short of that tip; the same miner address coming back lets that datadir finish | | `end_of_ibd_sh_interrupt` | P2P (**default**) | Same miner and scripts. The syncer first catches up with `--sh-index` off, then the datadir is frozen after pass 1 (`DONE.keys`, no `DONE.post`). Resume builds the index and Electrum's first answer is the full A/B history; restart does not rewrite the pack mark. A block mined after the freeze is in that history. A copy that already has the block, with `include_hwm` at the new tip and that create appended on the unsealed head, still serves the same history and does not open Electrum early | | `end_of_ibd_work_fork` | P2P (**default**) | Two miners on a regtest with retargeting and min-difficulty. One chain retargets harder and stops shorter. The other forks after that retarget, resets to the pow limit, and grows taller with less work. The syncer IBD-adopts the heavier tip and keeps it across a reopen. A scripted BIP324 peer advertises height 100,000, supplies two consecutive full windows of a lighter fork, then replays the second: the syncer asks to continue after each advancing window but finishes IBD after the replay. It then follows one more heavy block from the miners. This journey exceeds 2s because the replay must reach the protocol’s 2,000-header boundary; 89 harder blocks outweigh both windows without confirming 4,001 easy blocks | +| `end_of_ibd_invalid_heavier_fork` | P2P (**default**) | Three miners on one regtest. The split miner never halves the subsidy, so from height 20 its taller chain is coinbase excess to the syncer while it still carries the most header work. Honest forks at 14 and the third miner forks at the first invalid block. With honest at 17 the syncer keeps the split chain's valid prefix (19) and IBD returns. Once honest reaches 30, a fresh syncer confirms that prefix, retires the split candidate on the invalid block, rewinds 19 → 14, and finishes at honest 30. A syncer whose only other peer forks at the invalid block finishes on that fork | | `node_run_p2p_short` | Node (**default**) | Product `run_p2p` `--blocks-only` `--connect` to a live seeder (`--max-tip-age` so the 3-block pad is not stale IBD); process `getpeerinfo` / `getconnectioncount` / `getnetworkinfo` / `getnettotals` / `ping` while connected (v2 `manual`, as Core reports `-connect`; handshake `startingheight` equals the seeder tip; `timeoffset` present; `synced_headers`/`synced_blocks` stay `-1` until the peer announces a header hash (empty getheaders at tip does not copy VERSION height); `servicesnames` present; `getnetworkinfo.timeoffset` present); after catch-up `localrelay` / mempool `relay_enabled` stay false and `sendrawtransaction` is not `relay disabled`; Electrum `broadcast` and Esplora `POST /tx` admit decode/consensus errors (not hub-missing / not `relay disabled`); `addconnection inbound` refuses; `disconnectnode` unknown `nodeid` / empty params error then a real addr drops that row from the next `getpeerinfo`, the seeder sees it go, and a second `disconnectnode` of that `nodeid` (or of an address never connected) is `-29`; the `--connect` peer is redialled as `manual` under a new id and the seeder sees the inbound; `addnode onetry` and `add` of that address succeed and open no second session; seeder inbound `tx` then ends that session. Exit via `stop`. `max_run_secs=0` is `node_listen_and_exit`. Mock-clock `timeoffset` is the median (odd N, even N upper-middle, inbound-only 0, peer clock behind). A connecting dummy reports `-1`. Header-only and connected peers report `synced_blocks` differently. A query without a chain errors. `pingwait`, `NETWORK_LIMITED`, and `noban` are on the peer row | Removed (covered by the rows above): `confirm_cross_block_prevout_without_tx_head`, diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md index d86c115ff..0cb127aaf 100644 --- a/changelog.d/ibd-invalid-heavier.md +++ b/changelog.d/ibd-invalid-heavier.md @@ -1,8 +1,17 @@ Fixed +- **IBD leaves a header chain whose block fails consensus.** When the most + work header chain held a consensus-invalid block, IBD rejected the block + but kept that chain as its candidate. It never downloaded a lighter valid + chain another peer served, and stayed below the invalid block. The + candidate now falls back to the invalid block's parent. Headers that build + on the invalid block are refused, and a heavier valid chain from any peer + takes over, rewinding confirmed blocks of the dead chain when it forks + below them. - **IBD asks the header peer for a block no other peer has.** The peer that served the headers was never asked for blocks while another peer was connected, and a peer that answered `notfound` was asked again for the same block. A heavier chain only the header peer had stalled IBD. A peer is not asked again for a block it answered `notfound` until it announces - more. + more, or for a block above where its announced headers left the download + path. diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 8811eef15..35d5af4b2 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -643,6 +643,7 @@ pub fn plan_header_batch( }; if mark.outer_len > 0 { note_announcement(st, peer); + note_peer_headers(st, hub, peer, &headers); } else { super::header_walk::settle_walk_peer( st, @@ -681,6 +682,9 @@ pub fn plan_header_reply( HeaderPlan::Done }; } + if super::header_walk::refuse_invalid_batch(st, peer, &headers) { + return HeaderPlan::Done; + } let class = super::header_walk::classify(st, hub, ask, &headers); let mut headers = headers; let keep = match class { @@ -885,6 +889,28 @@ pub fn apply_block_decode_failed(st: &mut IbdWorkState, peer: usize, hash: Block } #[doc(hidden)] +fn note_peer_headers( + st: &mut IbdWorkState, + hub: &ChainHub, + peer: usize, + headers: &[bitcoin::block::Header], +) { + let Some(height) = headers + .first() + .and_then(|h| parent_height(&st.hash_height, hub, h.prev_blockhash)) + else { + return; + }; + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.note_headers( + &st.height_to_hash, + hub.tip_height().unwrap_or(0), + height, + headers, + ); + } +} + fn note_announcement(st: &mut IbdWorkState, peer: usize) { if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { s.chain.note_announcement(); @@ -1415,6 +1441,9 @@ pub fn apply_consensus_invalid_reject( } st.body.mark_rejected(hash); st.reorg.invalid.mark(hash.to_byte_array()); + if let Some(h) = hub { + super::header_walk::retire_invalid(st, h, hash, height); + } if st.height_to_hash.get(&height) == Some(&hash) { st.height_to_hash.remove(&height); } diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 1b4dc04a5..27259362c 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -80,6 +80,9 @@ pub struct HeaderWalk { pub origin: bool, /// Look-ahead tips that every peer left empty before the work floor. pub dead_ends: HashSet, + /// Rejected blocks, and the last header of each refused batch that built + /// on one. A batch that holds or builds on either is refused. + pub invalid_tips: HashSet, /// Peers that have answered empty at the current tip. pub emptied: HashSet, /// Checkpoint work has reached the milestone floor. @@ -135,6 +138,8 @@ pub const LOOKAHEAD_ASK: Duration = Duration::from_secs(5); pub const HEADER_MISS_DISCONNECT: u8 = 2; #[doc(hidden)] pub const DEAD_END_CAP: usize = 1024; +/// Past the cap a new invalid tip is not stored; the oldest stay. +const INVALID_TIP_CAP: usize = 1024; impl HeaderWalk { pub fn tip_height(&self) -> u32 { @@ -613,10 +618,16 @@ pub fn note_reservation( } /// Whether block `hash` may be asked of `peer`. A peer is not asked for a -/// block it answered `notfound`. The reserved header peer is asked only when -/// no other live peer may have it, so a single peer still does both. +/// block it answered `notfound`, or for a path block above where its +/// announced chain left the path. The reserved header peer is asked only +/// when no other live peer may have it, so a single peer still does both. pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { - let has = |s: &super::peer_io::PeerSlot| s.alive && !s.chain.lacks(hash); + let height = st.hash_height.get(hash).copied(); + let has = |s: &super::peer_io::PeerSlot| { + s.alive + && !s.chain.lacks(hash) + && !height.is_some_and(|ht| s.chain.left_path_by(&st.height_to_hash, ht)) + }; if !st.slots.iter().any(|s| s.id == peer && has(s)) { return false; } @@ -765,6 +776,129 @@ pub fn note_dead(st: &mut IbdWorkState, hash: BlockHash) { } #[doc(hidden)] +fn note_invalid_tip(st: &mut IbdWorkState, hash: BlockHash) { + if st.header_walk.invalid_tips.len() < INVALID_TIP_CAP { + st.header_walk.invalid_tips.insert(hash); + } +} + +/// Refuse a batch that holds or builds on a rejected block, and retire +/// `peer` from the walk. The batch's last header is remembered, so the next +/// batch of that chain is refused too. +pub fn refuse_invalid_batch(st: &mut IbdWorkState, peer: usize, headers: &[Header]) -> bool { + let Some(first) = headers.first() else { + return false; + }; + let tips = &st.header_walk.invalid_tips; + if !tips.contains(&first.prev_blockhash) + && !headers.iter().any(|h| tips.contains(&h.block_hash())) + { + return false; + } + if let Some(last) = headers.last() { + note_invalid_tip(st, last.block_hash()); + } + st.header_walk.walk_quiet.insert(peer); + if st.header_walk.reserved == Some(peer) { + st.header_walk.reserved = None; + } + true +} + +/// The block at `height` failed consensus. When the download path holds it, +/// the path above its parent is dropped and every peer may be asked for +/// headers again. A candidate built on it falls back to that parent, and a +/// challenger that forks at or below the parent with more work takes over. +pub fn retire_invalid(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, height: u32) { + note_invalid_tip(st, hash); + if !st.is_on_path(&hash, height) { + return; + } + let Some(parent) = hub.header_of(&hash).map(|h| h.prev_blockhash) else { + return; + }; + let parent_h = height.saturating_sub(1); + drop_dead_path(st, hub, parent_h); + if !st.header_walk.origin || st.header_walk.tip_height() < height { + return; + } + let tip = cut_back_to(st, hub, parent, parent_h); + if st.header_walk.base_height > parent_h { + st.header_walk.base_hash = tip.hash; + st.header_walk.base_height = parent_h; + st.header_walk.base_work = tip.work.to_be_bytes(); + st.header_walk.base_diff = tip.diff; + } + st.header_walk.tip = tip; + if st + .header_walk + .challenger + .as_ref() + .is_some_and(|ch| ch.fork_height > parent_h) + { + st.header_walk.challenger = None; + } + rbitcoin_log::warn!( + "ibd: header candidate retired at {hash} height={height}; walk back to {parent} height={parent_h}" + ); + let beat = st + .header_walk + .challenger + .as_ref() + .is_some_and(|ch| ch.tip.work > st.header_walk.tip.work); + if beat { + promote_challenger(st, hub); + } else { + publish_work(st, hub); + save_adopt(st, hub); + } +} + +/// Forget the download path above `keep`: queued hashes, bodies waiting in +/// the body queue, the confirm plan, and the densify cursor. Every peer may +/// be asked for headers again. +fn drop_dead_path(st: &mut IbdWorkState, hub: &ChainHub, keep: u32) { + st.clear_path_above(keep); + drop_queue_above(st, keep, &HashSet::new()); + for queued in hub.query.block_queue_queued_heights() { + if queued > keep { + let _ = hub.query.block_queue_dequeue_height(queued); + } + } + st.confirm_quiesce = true; + st.max_ready_height = st.max_ready_height.min(keep); + let next = keep.saturating_add(1); + st.densify_scan_lo = st.densify_scan_lo.min(next); + st.densify_ready_through = st.densify_ready_through.min(next); + st.header_walk.reserved = None; + st.header_walk.walk_quiet.clear(); + st.header_walk.emptied.clear(); + clear_off_path(st); +} + +/// A walk tip at stored `hash`, `height` on the candidate. Checkpoints and a +/// milestone above it are dropped. +fn cut_back_to(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, height: u32) -> WalkTip { + let mut tip = WalkTip { + hash: Some(hash), + height, + work: work_at(st, hub, hash).unwrap_or(Work::from_be_bytes([0; 32])), + header: parent_header(st, hub, hash), + times: times_ending_at(st, hub, hash), + diff: diff_at(st, hub, hash), + milestone_hash: st.header_walk.tip.milestone_hash, + milestone_prev: st.header_walk.tip.milestone_prev, + milestone_header: st.header_walk.tip.milestone_header, + }; + forget_milestone_below(st, hub, height); + if height < hub.milestone.height { + tip.milestone_hash = None; + tip.milestone_header = None; + } + st.header_walk.checkpoints.retain(|c| c.height <= height); + tip +} + pub fn rewind_above(st: &mut IbdWorkState, hub: &ChainHub, height: u32) { rewind_to(st, hub, height, None); } @@ -1439,26 +1573,8 @@ pub fn adopt_heavier( return; }; // Capture the fork's period and timestamps before the challenger is dropped. - let mut tip = WalkTip { - hash: Some(prev), - height: fork_height, - work: work_at(st, hub, prev).unwrap_or(Work::from_be_bytes([0; 32])), - header: parent_header(st, hub, prev), - times: times_ending_at(st, hub, prev), - diff: diff_at(st, hub, prev), - milestone_hash: st.header_walk.tip.milestone_hash, - milestone_prev: st.header_walk.tip.milestone_prev, - milestone_header: st.header_walk.tip.milestone_header, - }; + let mut tip = cut_back_to(st, hub, prev, fork_height); st.header_walk.challenger = None; - forget_milestone_below(st, hub, fork_height); - if fork_height < hub.milestone.height { - tip.milestone_hash = None; - tip.milestone_header = None; - } - st.header_walk - .checkpoints - .retain(|c| c.height <= fork_height); let mut link = prev; for hdr in headers { if !push_header(&mut tip, hub, hdr) { @@ -3100,6 +3216,7 @@ pub fn parse_adopt(bytes: &[u8]) -> Option { }, origin: true, dead_ends: HashSet::new(), + invalid_tips: HashSet::new(), emptied: HashSet::new(), proven: false, off_path: std::collections::HashMap::new(), diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 88035cb98..c8729e735 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -1,11 +1,13 @@ //! What one IBD peer has shown of its chain. //! //! Core asks a peer only for blocks on that peer's best known chain -//! (`pindexBestKnownBlock`). A peer that answered `notfound` for a block is -//! not asked for it again. +//! (`pindexBestKnownBlock`). A peer whose announced headers leave the +//! download path is not asked for blocks above that point, and a peer that +//! answered `notfound` for a block is not asked for it again. +use bitcoin::block::Header; use bitcoin::BlockHash; -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; /// `notfound` hashes kept per peer. Past the cap a peer may be asked again. const LACKS_CAP: usize = 1024; @@ -13,6 +15,9 @@ const LACKS_CAP: usize = 1024; #[derive(Debug, Default)] pub struct PeerChain { lacks: HashSet, + /// The peer's header at the first height above the confirmed tip that + /// the download path did not hold when it was announced. + forks_at: Option<(BlockHash, u32)>, } impl PeerChain { @@ -30,4 +35,45 @@ impl PeerChain { pub fn note_announcement(&mut self) { self.lacks.clear(); } + + /// Compare announced `headers`, the first at `height`, with the path + /// above the confirmed `tip`. The first header the path does not hold is + /// where this peer may leave it. A later batch does not move that point + /// up, and one that matches the path through it clears it. + pub fn note_headers( + &mut self, + path: &HashMap, + tip: u32, + height: u32, + headers: &[Header], + ) { + let mut matched_to = None; + for (ht, hdr) in (height..).zip(headers) { + if ht <= tip { + continue; + } + let hash = hdr.block_hash(); + if path.get(&ht) == Some(&hash) { + matched_to = Some(ht); + continue; + } + if self.forks_at.is_none_or(|(_, fork)| ht <= fork) { + self.forks_at = Some((hash, ht)); + } + return; + } + if let (Some((_, fork)), Some(matched)) = (self.forks_at, matched_to) { + if matched >= fork { + self.forks_at = None; + } + } + } + + /// The peer's chain holds a different block than the path at or below + /// `height`, so it does not have the path block there. + pub fn left_path_by(&self, path: &HashMap, height: u32) -> bool { + self.forks_at.is_some_and(|(hash, fork)| { + fork <= height && path.get(&fork).is_some_and(|p| *p != hash) + }) + } } diff --git a/crates/rbitcoin-test/tests/end_of_ibd.rs b/crates/rbitcoin-test/tests/end_of_ibd.rs index 985985b20..43d42b901 100644 --- a/crates/rbitcoin-test/tests/end_of_ibd.rs +++ b/crates/rbitcoin-test/tests/end_of_ibd.rs @@ -16,7 +16,9 @@ use rbitcoin_node::{run_p2p, NodeConfig}; use rbitcoin_primitives::{Fk, Height, Network}; use rbitcoin_query::Query; use rbitcoin_store::{script_hash, ScriptHashRecord}; -use rbitcoin_test::mine::{mine_regtest_block, mine_regtest_block_at, regtest_genesis}; +use rbitcoin_test::mine::{ + coinbase_tx, mine_regtest_block, mine_regtest_block_at, regtest_genesis, +}; use rbitcoin_test::TestDatadir; use serde_json::{json, Value}; use std::collections::BTreeSet; @@ -1185,6 +1187,132 @@ async fn work_fork_journey() { light.shutdown().await; } +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn end_of_ibd_invalid_heavier_fork() { + let wall = llvm_cov_wall(60, 180); + tokio::time::timeout(wall, invalid_heavier_fork_journey()) + .await + .expect("end_of_ibd_invalid_heavier_fork wall"); +} + +/// The subsidy halves at this height for the syncer and the honest miners. +const SPLIT_HALVING: u32 = 20; + +fn split_params() -> ChainParams { + let mut params = ChainParams::regtest(); + params.overlay_subsidy_halving_interval(SPLIT_HALVING); + params +} + +/// Extend `chain` to `top`. Blocks at or past [`SPLIT_HALVING`] pay 25 BTC +/// when `halved`; a miner that never halves pays 50 BTC there, which the +/// syncer rejects as coinbase excess. `skew` keeps sibling hashes apart. +fn extend_split(chain: &mut Vec, top: u32, halved: bool, skew: u32) { + while (chain.len() as u32) <= top { + let height = chain.len() as u32; + let prev = chain.last().unwrap(); + let time = prev.header.time.saturating_add(1).saturating_add(skew); + let mut block = mine_regtest_block(prev.block_hash(), time, height, vec![]); + if halved && height >= SPLIT_HALVING { + block.txdata[0] = coinbase_tx(height, Amount::from_sat(25_0000_0000)); + block.header.merkle_root = block.compute_merkle_root().unwrap(); + rbitcoin_consensus::grind_regtest_pow(&mut block.header); + } + chain.push(block); + } +} + +/// Sync `dir` from `peers`; IBD must return at `tip`. +async fn sync_to(dir: &Path, peers: &[SocketAddr], tip: &bitcoin::Block, height: u32, beat: &str) { + let node = start_node(dir, split_params()).await; + let addrs: Vec = peers.iter().copied().map(NetAddr::Ip).collect(); + tokio::time::timeout( + Duration::from_secs(20), + rbitcoin_ibd::sync(&node, &addrs, IbdConfig::for_test()), + ) + .await + .unwrap_or_else(|_| { + panic!( + "{beat}: IBD did not return (tip {:?}, want {height})", + node.tip_height() + ) + }) + .unwrap_or_else(|e| panic!("{beat}: sync: {e}")); + assert_eq!(node.tip_height(), Some(height), "{beat}: tip height"); + assert_eq!( + node.hub.tip_hash(), + Some(tip.block_hash()), + "{beat}: tip hash" + ); + node.shutdown().await; +} + +async fn invalid_heavier_fork_journey() { + // `split` never halves, so from SPLIT_HALVING on its chain is invalid to + // the syncer while it carries the most header work. `honest` forks below + // the halving; `at_fork` forks at the first invalid block. + let mut split_blocks = vec![regtest_genesis()]; + extend_split(&mut split_blocks, 40, false, 0); + let mut honest_blocks = split_blocks[..=14].to_vec(); + extend_split(&mut honest_blocks, 17, true, 1_000); + let mut at_fork_blocks = split_blocks[..SPLIT_HALVING as usize].to_vec(); + extend_split(&mut at_fork_blocks, 30, true, 2_000); + + let split_dir = TestDatadir::new().unwrap(); + let honest_dir = TestDatadir::new().unwrap(); + let at_fork_dir = TestDatadir::new().unwrap(); + let split = start_node(split_dir.path().as_path(), ChainParams::regtest()).await; + let honest = start_node(honest_dir.path().as_path(), split_params()).await; + let at_fork = start_node(at_fork_dir.path().as_path(), split_params()).await; + load_chain(&split, &split_blocks); + load_chain(&honest, &honest_blocks); + load_chain(&at_fork, &at_fork_blocks); + + // The split chain's valid prefix (to 19) has more work than honest (17). + let syncer = TestDatadir::new().unwrap(); + let pair = [split.local_addr, honest.local_addr]; + let valid_top = &split_blocks[SPLIT_HALVING as usize - 1]; + sync_to( + syncer.path().as_path(), + &pair, + valid_top, + SPLIT_HALVING - 1, + "valid prefix", + ) + .await; + + // Once honest outgrows that prefix, a syncer that confirmed the prefix + // leaves it for the honest chain. + let from = honest_blocks.len() as u32; + extend_split(&mut honest_blocks, 30, true, 1_000); + for (height, block) in honest_blocks.iter().enumerate().skip(from as usize) { + honest.ingest_block(height as u32, block.clone()).unwrap(); + } + let rewinds = TestDatadir::new().unwrap(); + sync_to( + rewinds.path().as_path(), + &pair, + honest_blocks.last().unwrap(), + 30, + "heavier honest", + ) + .await; + + // A fork at the first invalid block: nothing on the split side to keep. + let fresh = TestDatadir::new().unwrap(); + sync_to( + fresh.path().as_path(), + &[split.local_addr, at_fork.local_addr], + at_fork_blocks.last().unwrap(), + 30, + "fork at the invalid block", + ) + .await; + split.shutdown().await; + honest.shutdown().await; + at_fork.shutdown().await; +} + fn startup_cfg(dir: &Path, miner: SocketAddr, max_tip_age: u64) -> NodeConfig { let mut cfg = NodeConfig::default() .with_datadir(dir) From 18f00e9d9b590d6660e874004059a2afd56148ba Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 14:47:43 -0400 Subject: [PATCH 03/17] net: keep a failed connect invalid across restart Invalid marks were in memory only. After a restart the node forgot that a stored header chain held a consensus-invalid block: the startup scan saw that chain as more work, and the IBD resume seed planted it again, rewinding a valid tip to chase it (issue #1082). The hub's invalid set stays the one owner. A block that fails connect on a consensus rule (IBD reject, or tip-follow's failed accept) is also written to the `store/block.invalid` sidecar, at most 4096 hashes, and `reconsiderblock` removes it. The hub loads the file at open, IBD seeds its reject set from the hub, so the resume seed and the header walk refuse that chain, and the startup heavier-chain scan excludes those hashes. Mutated bodies, store faults, and operator `invalidateblock` are not written. This is a schema addition without a version bump: a missing or unparseable file is an empty set (SCHEMA.md). IBD also stops treating a header timestamp too far in the future as consensus-invalid. It is requeued, as in tip-follow and Core (BLOCK_TIME_FUTURE), so it is never persisted. Co-Authored-By: Claude Opus 5.5 --- COMPAT.md | 1 + SCHEMA.md | 13 +++ TESTING.md | 2 +- changelog.d/ibd-invalid-heavier.md | 7 ++ crates/rbitcoin-ibd/src/ibd/confirm/mod.rs | 4 + crates/rbitcoin-ibd/src/ibd/events/mod.rs | 1 + crates/rbitcoin-ibd/src/ibd/header_walk.rs | 14 +-- crates/rbitcoin-ibd/src/ibd/mod.rs | 5 + .../tests/ibd/events/confirm_reject_tests.rs | 11 ++ crates/rbitcoin-net/src/chain.rs | 108 +++++++++++++++++- crates/rbitcoin-net/tests/chain.rs | 9 ++ crates/rbitcoin-node/src/run.rs | 10 +- crates/rbitcoin-query/src/lib.rs | 6 +- crates/rbitcoin-test/tests/end_of_ibd.rs | 34 +++++- 14 files changed, 207 insertions(+), 18 deletions(-) diff --git a/COMPAT.md b/COMPAT.md index 83a43cf86..e06e0d507 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -82,6 +82,7 @@ Full `/tx/:txid` JSON still has `vin[]`. Electrum has no outspend-vin surface. | IBD empty-headers EOF | Empty `headers` to **our locator** + idle path latches `headers_done` even if a peer advertises a taller less-work height / junk `version.start_height`. A proven header walk at the tip that every taller peer failed to extend also ends IBD | Core header sync follows most-work; advertised `start_height` is not a remaining header count | | BIP94 | Not implemented. There is no testnet4 chain and no `-test=bip94`. A retarget multiplies the parent, and a difficulty-boundary timestamp only has to be after the median of ancestor timestamps. This does not fork mainnet, testnet3, signet, or default regtest | On testnet4, and on regtest only with `-test=bip94`, a retarget multiplies the first block of the period (that parent may be a min-difficulty exception) and a difficulty-boundary timestamp more than 600 seconds before its parent is `time-timewarp-attack` | | Milestone script skip | A height-only `--milestone HEIGHT` skips scripts at and below that height with no anchor hash, including when the header lookup is empty. The default anchor skips once two height-to-hash lookups match: this block, and the anchor hash at the anchor height. It does not walk parents, does not read `min_work_be`, and does not require two weeks of burial. Prevouts, double spends, maturity, and fees still run. An output whose script starts with `OP_RETURN`, or is longer than 10,000 bytes, is still not a coin under the skip | Core runs scripts unless the assumevalid block is an ancestor, the best header is on that chain, chain work meets `nMinimumChainWork`, and the proof-equivalent time from the assumevalid block to the best header is more than two weeks | +| Invalid marks across restart | A block that failed connect on a consensus rule stays invalid across restart (`store/block.invalid`, at most 4096). An operator `invalidateblock` does not: it lasts until the process exits | Both live in the block index (`BLOCK_FAILED_VALID`) and survive restart | | Equal-work fork choice | A connected tip is replaced only by strictly greater chain work, or by `preciousblock` at equal work. After `invalidateblock`, equal work prefers precious, then the earlier held tip. Header ranking before either chain is connected also stays with the chain already chosen when work is equal ([`docs/architecture.md`](./docs/architecture.md) layer 1, “equal work keeps the first seen”). A block that was stored earlier but was not yet connectable does not take the tip later | When work is equal, Core prefers the block seen first (`nSequenceId`). A block stored earlier but not yet connectable becomes the tip once it can be connected, even if a later equal-work block is already the tip. Neither rule makes a block valid on one side and invalid on the other | Compact reconstruct fills short-ids from the live mempool graph, the diff --git a/SCHEMA.md b/SCHEMA.md index 55ee0c465..da747d070 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -222,6 +222,7 @@ itself changed. meta # store magic + schema version header.body / header.head # Class A headers + hash index (overflow: header.head.gN) header.adopt # IBD checkpoint list + milestone hash (not a header-chain copy) + block.invalid # blocks that failed connect, kept across restart txout.body / create.loc / create.off / create.loc.ovf # Class A outs (hot loc) seqsigwit.body / seqsigwit.loc / seqsigwit.off / seqsigwit.loc.ovf # Class A inputs+witness (cold loc) seqsigwit.prune # optional: u32 LE pruneheight sidecar (`--prune-seqsigwit`; missing = off) @@ -394,6 +395,18 @@ Each checkpoint is 285 bytes. The tail is 364 bytes. Length is exact: `12 + coun Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. +### `block.invalid` + +Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (temp file, then rename) for each new failure and when `reconsiderblock` clears one. + +```text +0..8 magic b"rbtcinv1" +8..12 u32 LE count (at most 4096) +repeat block hash [u8; 32] +``` + +Length is exact: `12 + count × 32`. Oldest first. Past 4096 a new failure is marked in memory only. Mutated bodies, a header timestamp too far in the future, store and engine faults, and operator `invalidateblock` are not written. A descendant is invalid through its ancestor: the startup header scan and the IBD resume seed do not walk through a listed hash. + --- ## Class A — transactions diff --git a/TESTING.md b/TESTING.md index 196475f14..7de6642a1 100644 --- a/TESTING.md +++ b/TESTING.md @@ -425,7 +425,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `end_of_ibd_follow` | P2P (**default**) | Miner plus `--sh-index` syncer. One mature regtest: coinbases pay script A, one spend pays script B. IBD reaches that tip, leaves IBD, and Electrum history matches. The next block tip-follows. Restart does not rewrite the scripthash pack mark; one more block arrives by write-behind. Cancelling IBD once the height is below the miner, then `run_p2p`, finishes the same tip and history. With the syncer caught up, dropping the miner leaves tip follow (not IBD). A partial datadir whose miner is down does not open Electrum and stays short of that tip; the same miner address coming back lets that datadir finish | | `end_of_ibd_sh_interrupt` | P2P (**default**) | Same miner and scripts. The syncer first catches up with `--sh-index` off, then the datadir is frozen after pass 1 (`DONE.keys`, no `DONE.post`). Resume builds the index and Electrum's first answer is the full A/B history; restart does not rewrite the pack mark. A block mined after the freeze is in that history. A copy that already has the block, with `include_hwm` at the new tip and that create appended on the unsealed head, still serves the same history and does not open Electrum early | | `end_of_ibd_work_fork` | P2P (**default**) | Two miners on a regtest with retargeting and min-difficulty. One chain retargets harder and stops shorter. The other forks after that retarget, resets to the pow limit, and grows taller with less work. The syncer IBD-adopts the heavier tip and keeps it across a reopen. A scripted BIP324 peer advertises height 100,000, supplies two consecutive full windows of a lighter fork, then replays the second: the syncer asks to continue after each advancing window but finishes IBD after the replay. It then follows one more heavy block from the miners. This journey exceeds 2s because the replay must reach the protocol’s 2,000-header boundary; 89 harder blocks outweigh both windows without confirming 4,001 easy blocks | -| `end_of_ibd_invalid_heavier_fork` | P2P (**default**) | Three miners on one regtest. The split miner never halves the subsidy, so from height 20 its taller chain is coinbase excess to the syncer while it still carries the most header work. Honest forks at 14 and the third miner forks at the first invalid block. With honest at 17 the syncer keeps the split chain's valid prefix (19) and IBD returns. Once honest reaches 30, a fresh syncer confirms that prefix, retires the split candidate on the invalid block, rewinds 19 → 14, and finishes at honest 30. A syncer whose only other peer forks at the invalid block finishes on that fork | +| `end_of_ibd_invalid_heavier_fork` | P2P (**default**) | Three miners on one regtest. The split miner never halves the subsidy, so from height 20 its taller chain is coinbase excess to the syncer while it still carries the most header work. Honest forks at 14 and the third miner forks at the first invalid block. With honest at 17 the syncer keeps the split chain's valid prefix (19) and IBD returns. Once honest reaches 30, a fresh syncer confirms that prefix, retires the split candidate on the invalid block, rewinds 19 → 14, and finishes at honest 30. The syncer that kept the prefix restarts, with both peers and then with honest alone, and finishes at honest 30 without rewinding to chase the stored split headers. A syncer whose only other peer forks at the invalid block finishes on that fork | | `node_run_p2p_short` | Node (**default**) | Product `run_p2p` `--blocks-only` `--connect` to a live seeder (`--max-tip-age` so the 3-block pad is not stale IBD); process `getpeerinfo` / `getconnectioncount` / `getnetworkinfo` / `getnettotals` / `ping` while connected (v2 `manual`, as Core reports `-connect`; handshake `startingheight` equals the seeder tip; `timeoffset` present; `synced_headers`/`synced_blocks` stay `-1` until the peer announces a header hash (empty getheaders at tip does not copy VERSION height); `servicesnames` present; `getnetworkinfo.timeoffset` present); after catch-up `localrelay` / mempool `relay_enabled` stay false and `sendrawtransaction` is not `relay disabled`; Electrum `broadcast` and Esplora `POST /tx` admit decode/consensus errors (not hub-missing / not `relay disabled`); `addconnection inbound` refuses; `disconnectnode` unknown `nodeid` / empty params error then a real addr drops that row from the next `getpeerinfo`, the seeder sees it go, and a second `disconnectnode` of that `nodeid` (or of an address never connected) is `-29`; the `--connect` peer is redialled as `manual` under a new id and the seeder sees the inbound; `addnode onetry` and `add` of that address succeed and open no second session; seeder inbound `tx` then ends that session. Exit via `stop`. `max_run_secs=0` is `node_listen_and_exit`. Mock-clock `timeoffset` is the median (odd N, even N upper-middle, inbound-only 0, peer clock behind). A connecting dummy reports `-1`. Header-only and connected peers report `synced_blocks` differently. A query without a chain errors. `pingwait`, `NETWORK_LIMITED`, and `noban` are on the peer row | Removed (covered by the rows above): `confirm_cross_block_prevout_without_tx_head`, diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md index 0cb127aaf..332572304 100644 --- a/changelog.d/ibd-invalid-heavier.md +++ b/changelog.d/ibd-invalid-heavier.md @@ -15,3 +15,10 @@ Fixed is not asked again for a block it answered `notfound` until it announces more, or for a block above where its announced headers left the download path. +- **A block that failed connect stays invalid after restart.** The mark was + in memory only, so a restart adopted the stored header chain through that + block again and rewound a valid tip to chase it. Connect failures are + kept in `store/block.invalid`; `reconsiderblock` clears one. The startup + header scan and the IBD resume seed do not walk through them. +- **IBD does not blacklist a block whose timestamp is too far in the + future.** It is requeued, as in tip-follow and Core. diff --git a/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs b/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs index 73aa4ad82..478e4925a 100644 --- a/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs @@ -375,6 +375,9 @@ impl ConfirmRejectClass { if s.contains("missing retarget first header") { return Self::SoftWire; } + if s.contains("timestamp too far in future") { + return Self::Cascade; + } if s.contains("fk mismatch") || s.contains("plan not committed in order") || s.contains("connect height not tip+1") @@ -415,6 +418,7 @@ impl ConfirmRejectClass { Self::SoftWire } ConsensusError::BadHeader("missing retarget first header") => Self::SoftWire, + ConsensusError::BadHeader("timestamp too far in future") => Self::Cascade, ConsensusError::Internal(m) => Self::from_err_str(m), ConsensusError::BadBlock(_) | ConsensusError::BadTx(_) diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 35d5af4b2..18ec9bd3c 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -1442,6 +1442,7 @@ pub fn apply_consensus_invalid_reject( st.body.mark_rejected(hash); st.reorg.invalid.mark(hash.to_byte_array()); if let Some(h) = hub { + h.note_failed_connect(hash); super::header_walk::retire_invalid(st, h, hash, height); } if st.height_to_hash.get(&height) == Some(&hash) { diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 27259362c..8cf480122 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -80,8 +80,8 @@ pub struct HeaderWalk { pub origin: bool, /// Look-ahead tips that every peer left empty before the work floor. pub dead_ends: HashSet, - /// Rejected blocks, and the last header of each refused batch that built - /// on one. A batch that holds or builds on either is refused. + /// The last header of each refused batch that built on a rejected block. + /// A batch that builds on one is refused too. pub invalid_tips: HashSet, /// Peers that have answered empty at the current tip. pub emptied: HashSet, @@ -789,10 +789,11 @@ pub fn refuse_invalid_batch(st: &mut IbdWorkState, peer: usize, headers: &[Heade let Some(first) = headers.first() else { return false; }; - let tips = &st.header_walk.invalid_tips; - if !tips.contains(&first.prev_blockhash) - && !headers.iter().any(|h| tips.contains(&h.block_hash())) - { + let rejected = |hash: &BlockHash| { + st.header_walk.invalid_tips.contains(hash) + || st.reorg.invalid.contains(hash.to_byte_array()) + }; + if !rejected(&first.prev_blockhash) && !headers.iter().any(|h| rejected(&h.block_hash())) { return false; } if let Some(last) = headers.last() { @@ -810,7 +811,6 @@ pub fn refuse_invalid_batch(st: &mut IbdWorkState, peer: usize, headers: &[Heade /// headers again. A candidate built on it falls back to that parent, and a /// challenger that forks at or below the parent with more work takes over. pub fn retire_invalid(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, height: u32) { - note_invalid_tip(st, hash); if !st.is_on_path(&hash, height) { return; } diff --git a/crates/rbitcoin-ibd/src/ibd/mod.rs b/crates/rbitcoin-ibd/src/ibd/mod.rs index 0a9c9d555..630477f20 100644 --- a/crates/rbitcoin-ibd/src/ibd/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/mod.rs @@ -473,6 +473,11 @@ pub async fn ibd_cancellable( let window = cfg.window; let mut st = IbdWorkState::new(initial_slots, hub.tip_hash(), hub.tip_height()); + for hash in hub.invalid_blocks() { + st.reorg + .invalid + .mark(bitcoin::hashes::Hash::to_byte_array(hash)); + } st.addr_cooldown = boot_cooldown; st.addr_strikes = boot_strikes; st.perms = perms; diff --git a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs index 0a7994454..2db43e10d 100644 --- a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs +++ b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs @@ -99,6 +99,10 @@ fn confirm_reject_class_matches_substring_table() { ConfirmRejectClass::Cascade, ), ("confirm cancelled", ConfirmRejectClass::Cancelled), + ( + "consensus: bad header: timestamp too far in future", + ConfirmRejectClass::Cascade, + ), ]; for (s, want) in cases { assert_eq!(ConfirmRejectClass::from_err_str(s), *want, "{s}"); @@ -145,6 +149,13 @@ fn confirm_reject_class_matches_substring_table() { ConfirmRejectClass::from_consensus(&ConsensusError::Cancelled), ConfirmRejectClass::Cancelled ); + // Core BLOCK_TIME_FUTURE: the block may connect once the clock passes it. + assert_eq!( + ConfirmRejectClass::from_consensus(&ConsensusError::BadHeader( + "timestamp too far in future" + )), + ConfirmRejectClass::Cascade + ); assert_eq!( ConfirmRejectClass::from_confirm(&rbitcoin_query::ConfirmError::from( StoreError::Cancelled("stop"), diff --git a/crates/rbitcoin-net/src/chain.rs b/crates/rbitcoin-net/src/chain.rs index f2f1dc3e3..01f3b1a2f 100644 --- a/crates/rbitcoin-net/src/chain.rs +++ b/crates/rbitcoin-net/src/chain.rs @@ -237,6 +237,48 @@ impl ForkTips { pub struct Invalidated { pub set: RwLock>, pub paths: RwLock>, + /// Connect failures kept across restart in [`INVALID_FILE`], oldest first. + /// Operator invalidations are not: their paths are in-process only. + pub failed: RwLock>, + pub file: std::path::PathBuf, +} + +/// Sidecar of connect failures next to `header.adopt`. Not a schema bump. +const INVALID_FILE: &str = "block.invalid"; +const INVALID_MAGIC: &[u8; 8] = b"rbtcinv1"; +/// Past the cap a new failure is not written; the oldest stay. +const INVALID_FILE_CAP: usize = 4_096; + +fn parse_invalid_file(bytes: &[u8]) -> Option> { + let rest = bytes.strip_prefix(INVALID_MAGIC)?; + let (count, rest) = rest.split_first_chunk::<4>()?; + let n = u32::from_le_bytes(*count) as usize; + if n > INVALID_FILE_CAP || rest.len() != n.checked_mul(32)? { + return None; + } + Some( + rest.chunks_exact(32) + .map(|c| { + let mut raw = [0u8; 32]; + raw.copy_from_slice(c); + BlockHash::from_byte_array(raw) + }) + .collect(), + ) +} + +fn write_invalid_file(path: &std::path::Path, hashes: &[BlockHash]) { + let mut buf = Vec::with_capacity(12 + 32 * hashes.len()); + buf.extend_from_slice(INVALID_MAGIC); + buf.extend_from_slice(&(hashes.len() as u32).to_le_bytes()); + for h in hashes { + buf.extend_from_slice(h.as_byte_array()); + } + let tmp = path.with_file_name("block.invalid.tmp"); + let wrote = std::fs::write(&tmp, &buf).and_then(|()| std::fs::rename(&tmp, path)); + if let Err(e) = wrote { + warn!("chain: write {}: {e}", path.display()); + } } /// The hashes one `invalidateblock` marked, in height order. `connected` when @@ -249,11 +291,46 @@ pub struct InvalidPath { } impl Invalidated { + /// Load the connect failures in `store/block.invalid`. A missing file is + /// none; one that does not parse is none and is replaced on the next write. #[doc(hidden)] - pub fn new() -> Self { + pub fn open(store: &std::path::Path) -> Self { + let file = store.join(INVALID_FILE); + let failed = match std::fs::read(&file) { + Ok(bytes) => parse_invalid_file(&bytes).unwrap_or_else(|| { + warn!( + "chain: {} does not parse; no stored invalid blocks", + file.display() + ); + Vec::new() + }), + Err(_) => Vec::new(), + }; Self { - set: RwLock::new(HashSet::new()), + set: RwLock::new(failed.iter().copied().collect()), paths: RwLock::new(Vec::new()), + failed: RwLock::new(failed), + file, + } + } + + fn keep_failed(&self, hash: BlockHash) { + let mut failed = self.failed.write().unwrap(); + if failed.len() >= INVALID_FILE_CAP || failed.contains(&hash) { + return; + } + failed.push(hash); + write_invalid_file(&self.file, &failed); + } + + /// Drop kept failures that are no longer marked invalid. + fn prune_failed(&self) { + let set = self.set.read().unwrap(); + let mut failed = self.failed.write().unwrap(); + let before = failed.len(); + failed.retain(|h| set.contains(h)); + if failed.len() != before { + write_invalid_file(&self.file, &failed); } } } @@ -578,6 +655,7 @@ impl ChainHub { // Full confirmed-set fill in background (mainnet-scale tips make a // synchronous walk multi-minute). Tip/genesis are seeded immediately. spawn_confirmed_seed(query.clone(), confirmed.clone()); + let invalidated = Invalidated::open(query.store().path()); Self { query, cache: Arc::new(BlockCache::new()), @@ -591,7 +669,7 @@ impl ChainHub { mempool: std::sync::OnceLock::new(), self_weak: std::sync::OnceLock::new(), clock: rbitcoin_consensus::NodeClock::new(), - invalidated: Invalidated::new(), + invalidated, held_bodies: RwLock::new(HeldBodies::new()), precious: RwLock::new(None), fork_tips: RwLock::new(ForkTips::new()), @@ -1351,6 +1429,25 @@ impl ChainHub { self.drop_held(hash); } + /// A block failed connect on a consensus rule. Unlike + /// [`Self::note_invalid_block`], the mark survives restart; mutated bodies, + /// a future timestamp, and store faults are not this. + pub fn note_failed_connect(&self, hash: BlockHash) { + self.note_invalid_block(hash); + self.invalidated.keep_failed(hash); + } + + /// Every hash marked invalid, for a header scan that must not count them. + pub fn invalid_blocks(&self) -> Vec { + self.invalidated + .set + .read() + .unwrap() + .iter() + .copied() + .collect() + } + /// A reconstructed compact with the right header hash and wrong txs must /// not poison later getdata of that hash. #[doc(hidden)] @@ -1365,12 +1462,12 @@ impl ChainHub { return; } if e.failing_block_hash().is_some() { - self.note_invalid_block(hash); + self.note_failed_connect(hash); return; } if let NetError::Consensus(s) = e { if !s.to_ascii_lowercase().contains("not found") { - self.note_invalid_block(hash); + self.note_failed_connect(hash); } } } @@ -2313,6 +2410,7 @@ impl ChainHub { inv.remove(h); } } + self.invalidated.prune_failed(); let restored: HashSet = paths .iter() diff --git a/crates/rbitcoin-net/tests/chain.rs b/crates/rbitcoin-net/tests/chain.rs index a124437cc..70fa07575 100644 --- a/crates/rbitcoin-net/tests/chain.rs +++ b/crates/rbitcoin-net/tests/chain.rs @@ -4090,12 +4090,21 @@ fn reconsider_skips_a_branch_that_fails() { rbitcoin_consensus::grind_regtest_pow(&mut bad5.header); let bad5_hash = bad5.block_hash(); assert!(hub.accept_received_block(bad5).is_err()); + let marks = dir.path().join("block.invalid"); + let kept = |marks: &std::path::Path| { + std::fs::read(marks) + .unwrap() + .windows(32) + .any(|w| w == bad5_hash.as_byte_array()) + }; + assert!(kept(&marks), "a connect failure is kept across restart"); let a5 = mine_distinct(a[3], t0 + 31, 5, &[bad5_hash]); let a6 = mine(a5.block_hash(), t0 + 32, 6); hub.accept_block(a5).unwrap(); hub.accept_block(a6).unwrap(); hub.invalidate_block(bad5_hash).unwrap(); hub.reconsider_block(bad5_hash).unwrap(); + assert!(!kept(&marks), "reconsiderblock forgets the kept failure"); assert!( !hub.fork_tips.read().unwrap().iter().any(|h| h == bad5_hash), "a block that never connected is not kept as a fork tip" diff --git a/crates/rbitcoin-node/src/run.rs b/crates/rbitcoin-node/src/run.rs index b7a8c4766..d8d6d3ecf 100644 --- a/crates/rbitcoin-node/src/run.rs +++ b/crates/rbitcoin-node/src/run.rs @@ -1752,7 +1752,15 @@ pub fn stored_header_chain_heavier_than_tip(hub: &ChainHub) -> bool { return true; } }; - match hub.query.has_heavier_header_chain_than_tip(rec.hash, tip_h) { + let invalid: Vec<[u8; 32]> = hub + .invalid_blocks() + .into_iter() + .map(bitcoin::hashes::Hash::to_byte_array) + .collect(); + match hub + .query + .has_heavier_header_chain_than_tip(rec.hash, tip_h, &invalid) + { Ok(heavier) => heavier, Err(e) => { warn!("node: startup header scan failed: {e} — catch-up"); diff --git a/crates/rbitcoin-query/src/lib.rs b/crates/rbitcoin-query/src/lib.rs index 67152bb1b..d6b4d607e 100644 --- a/crates/rbitcoin-query/src/lib.rs +++ b/crates/rbitcoin-query/src/lib.rs @@ -2416,13 +2416,15 @@ impl Query { /// /// The sibling search climbs to genesis. IBD seeding keeps the 32-hop cap; /// startup uses this so a deep stored fork still stays in catch-up. - /// A tip hash with no header row is an empty path (`Ok(false)`). + /// A tip hash with no header row is an empty path (`Ok(false)`). A path + /// through an `exclude` hash (a known-invalid block) does not count. pub fn has_heavier_header_chain_than_tip( &self, tip_hash: [u8; 32], tip_height: u32, + exclude: &[[u8; 32]], ) -> Result { - let path = self.resume_work_path_limited(tip_hash, tip_height, 1, &[], u32::MAX)?; + let path = self.resume_work_path_limited(tip_hash, tip_height, 1, exclude, u32::MAX)?; Ok(!path.is_empty()) } diff --git a/crates/rbitcoin-test/tests/end_of_ibd.rs b/crates/rbitcoin-test/tests/end_of_ibd.rs index 43d42b901..3398ee0b9 100644 --- a/crates/rbitcoin-test/tests/end_of_ibd.rs +++ b/crates/rbitcoin-test/tests/end_of_ibd.rs @@ -1298,6 +1298,19 @@ async fn invalid_heavier_fork_journey() { ) .await; + // The syncer that kept the prefix restarts. The invalid block stays + // invalid, so the split headers stored above it are not a path. + let honest_tip = honest_blocks.last().unwrap(); + sync_to(syncer.path().as_path(), &pair, honest_tip, 30, "restart").await; + sync_to( + syncer.path().as_path(), + &[honest.local_addr], + honest_tip, + 30, + "restart with only honest", + ) + .await; + // A fork at the first invalid block: nothing on the split side to keep. let fresh = TestDatadir::new().unwrap(); sync_to( @@ -1352,7 +1365,7 @@ fn regtest_chain(blocks: u32) -> Vec { } /// One difficulty-1 header under genesis. Its work beats a long easy regtest chain. -fn plant_deep_heavy_header(q: &Query) { +fn plant_deep_heavy_header(q: &Query) -> [u8; 32] { use rbitcoin_store::{block_header_hash, HeaderRecord}; let (gfk, genesis) = q.header_at_height(Height(0)).unwrap().unwrap(); let version = 1i32; @@ -1373,6 +1386,7 @@ fn plant_deep_heavy_header(q: &Query) { weight: 0, }) .unwrap(); + hash } type StartupLogs = Vec<(rbitcoin_log::Level, String)>; @@ -1489,7 +1503,7 @@ async fn startup_tip_follow_journey() { let deep_dir = TestDatadir::new().unwrap(); let deep_node = start_miner(deep_dir.path().as_path()).await; load_chain(&deep_node, &deep); - plant_deep_heavy_header(&deep_node.hub.query); + let heavy = plant_deep_heavy_header(&deep_node.hub.query); deep_node.hub.query.flush().unwrap(); deep_node.shutdown().await; @@ -1503,6 +1517,22 @@ async fn startup_tip_follow_journey() { let logs = stop_logged(rpc, node).await; assert_startup(&logs, false); + // The same heavier header, once a connect failure marked it invalid, is + // not a chain to catch up to. + let mut marks = b"rbtcinv1".to_vec(); + marks.extend_from_slice(&1u32.to_le_bytes()); + marks.extend_from_slice(&heavy); + std::fs::write(deep_dir.path().join("store").join("block.invalid"), marks).unwrap(); + let node = spawn_run_p2p_logs(startup_cfg( + deep_dir.path().as_path(), + deep_miner.local_addr, + u64::MAX, + )); + let rpc = published_listener(deep_dir.path().as_path(), "rpc", Duration::from_secs(30)).await; + wait_height(rpc, 40).await; + let logs = stop_logged(rpc, node).await; + assert_startup(&logs, true); + deep_miner.shutdown().await; miner.shutdown().await; } From 06d5961ee7c0f83e686ecad306147dcfdd2406d6 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 14:48:53 -0400 Subject: [PATCH 04/17] net: ask each IBD block peer for headers once A peer that had announced nothing was asked for blocks of any chain. A Core peer serving a lighter chain ignores getdata for blocks it does not have, so it went quiet, was dropped, and cooled down for 10 minutes before the node needed its chain (issue #1082). Each block peer that has not announced a chain now gets one headers ask, through the block-inv challenge (one peer at a time, never the reserved header peer). Each slot keeps the hashes of its announced chain above the confirmed tip, at most one headers reply. Two chains that differ at one height differ above it, so one comparison at the lower of the asked height and the peer's top tells whether the peer holds a path block. A lighter fork in the reply also becomes the walk's challenger. The retire no longer promotes a challenger. Its headers are not stored and its fork can sit below the confirmed tip; peers are asked again from the parent instead, and a heavier chain comes back as stored headers. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 2 +- changelog.d/ibd-invalid-heavier.md | 5 ++ crates/rbitcoin-ibd/src/ibd/events/mod.rs | 8 +- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 49 ++++++----- crates/rbitcoin-ibd/src/ibd/mod.rs | 1 + crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 91 ++++++++++++-------- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 42 +++++++++ 7 files changed, 132 insertions(+), 66 deletions(-) diff --git a/TESTING.md b/TESTING.md index 7de6642a1..aff554c1f 100644 --- a/TESTING.md +++ b/TESTING.md @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once, and once its reply forks below the path it is not asked for path blocks. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md index 332572304..e3a1d228a 100644 --- a/changelog.d/ibd-invalid-heavier.md +++ b/changelog.d/ibd-invalid-heavier.md @@ -15,6 +15,11 @@ Fixed is not asked again for a block it answered `notfound` until it announces more, or for a block above where its announced headers left the download path. +- **IBD asks each block peer for headers once.** A peer that had announced + nothing was asked for blocks of any chain, so a Core peer serving a + lighter chain stayed silent on blocks it did not have and was dropped for + a 10 minute cooldown. Each block peer now gets one headers ask, so its + chain is known before blocks are asked of it. - **A block that failed connect stays invalid after restart.** The mark was in memory only, so a restart adopted the stored header chain through that block again and rewound a valid tip to chase it. Connect failures are diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 18ec9bd3c..feda936e0 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -902,12 +902,8 @@ fn note_peer_headers( return; }; if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { - s.chain.note_headers( - &st.height_to_hash, - hub.tip_height().unwrap_or(0), - height, - headers, - ); + s.chain + .note_headers(hub.tip_height().unwrap_or(0), height, headers); } } diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 8cf480122..440b579cf 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -545,6 +545,27 @@ pub fn note_block_inv(st: &mut IbdWorkState, peer: usize) { } } +/// Ask one block peer that has not announced a chain for headers, so block +/// getdata can follow what it holds. Core starts header sync with each +/// peer. This is the block-inv challenge: one peer at a time, once per +/// peer, never the reserved header peer. +pub fn challenge_unprobed_peer(st: &mut IbdWorkState) { + if st.header_walk.challenge.is_some() || st.header_walk.challenge_open.is_some() { + return; + } + let reserved = st.header_walk.reserved; + let Some(slot) = st + .slots + .iter_mut() + .find(|s| s.alive && Some(s.id) != reserved && !s.chain.probed()) + else { + return; + }; + slot.chain.note_probed(); + st.header_walk.challenge = Some(slot.id); + st.header_walk.challenge_external = false; +} + /// A follow-session block inv has no IBD slot. Poke one live header peer /// the same way [`note_block_inv`] does. Returns false when nobody is live /// yet, so the caller keeps the flag. @@ -808,8 +829,7 @@ pub fn refuse_invalid_batch(st: &mut IbdWorkState, peer: usize, headers: &[Heade /// The block at `height` failed consensus. When the download path holds it, /// the path above its parent is dropped and every peer may be asked for -/// headers again. A candidate built on it falls back to that parent, and a -/// challenger that forks at or below the parent with more work takes over. +/// headers again. A candidate built on it falls back to that parent. pub fn retire_invalid(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, height: u32) { if !st.is_on_path(&hash, height) { return; @@ -830,28 +850,15 @@ pub fn retire_invalid(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, he st.header_walk.base_diff = tip.diff; } st.header_walk.tip = tip; - if st - .header_walk - .challenger - .as_ref() - .is_some_and(|ch| ch.fork_height > parent_h) - { - st.header_walk.challenger = None; - } + // A challenger's headers are not stored, and its fork can sit below the + // confirmed tip. Peers are asked again from the parent, and a heavier + // chain comes back as headers that are stored and can be rewound onto. + st.header_walk.challenger = None; rbitcoin_log::warn!( "ibd: header candidate retired at {hash} height={height}; walk back to {parent} height={parent_h}" ); - let beat = st - .header_walk - .challenger - .as_ref() - .is_some_and(|ch| ch.tip.work > st.header_walk.tip.work); - if beat { - promote_challenger(st, hub); - } else { - publish_work(st, hub); - save_adopt(st, hub); - } + publish_work(st, hub); + save_adopt(st, hub); } /// Forget the download path above `keep`: queued hashes, bodies waiting in diff --git a/crates/rbitcoin-ibd/src/ibd/mod.rs b/crates/rbitcoin-ibd/src/ibd/mod.rs index 630477f20..0eaf2590c 100644 --- a/crates/rbitcoin-ibd/src/ibd/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/mod.rs @@ -703,6 +703,7 @@ pub async fn ibd_cancellable( if hub.header_inv_challenge_pending() && header_walk::note_external_block_inv(st) { hub.clear_header_inv_challenge(); } + header_walk::challenge_unprobed_peer(st); let challenge_due = header_walk::inv_challenge_due(st); if cadence.headers_due(now_cadence, path_empty, challenge_due) { let live = st.ordered_set.len(); diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index c8729e735..4e325f3b7 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -7,17 +7,23 @@ use bitcoin::block::Header; use bitcoin::BlockHash; -use std::collections::{HashMap, HashSet}; +use std::collections::{HashMap, HashSet, VecDeque}; /// `notfound` hashes kept per peer. Past the cap a peer may be asked again. const LACKS_CAP: usize = 1024; +/// Announced hashes kept per peer: one full `headers` reply. +const ANNOUNCED_CAP: usize = rbitcoin_net::codec::MAX_HEADERS_RESULTS; + #[derive(Debug, Default)] pub struct PeerChain { lacks: HashSet, - /// The peer's header at the first height above the confirmed tip that - /// the download path did not hold when it was announced. - forks_at: Option<(BlockHash, u32)>, + /// This peer's announced chain above the confirmed tip, oldest first. + announced: VecDeque, + /// Height of `announced[0]`. + from: u32, + /// The peer announced headers, or was asked for them once. + probed: bool, } impl PeerChain { @@ -36,44 +42,53 @@ impl PeerChain { self.lacks.clear(); } - /// Compare announced `headers`, the first at `height`, with the path - /// above the confirmed `tip`. The first header the path does not hold is - /// where this peer may leave it. A later batch does not move that point - /// up, and one that matches the path through it clears it. - pub fn note_headers( - &mut self, - path: &HashMap, - tip: u32, - height: u32, - headers: &[Header], - ) { - let mut matched_to = None; - for (ht, hdr) in (height..).zip(headers) { - if ht <= tip { - continue; - } - let hash = hdr.block_hash(); - if path.get(&ht) == Some(&hash) { - matched_to = Some(ht); - continue; - } - if self.forks_at.is_none_or(|(_, fork)| ht <= fork) { - self.forks_at = Some((hash, ht)); - } + pub fn probed(&self) -> bool { + self.probed + } + + pub fn note_probed(&mut self) { + self.probed = true; + } + + /// Keep announced `headers`, the first at `height`, above the confirmed + /// `tip`. A batch that overlaps or continues the kept run replaces it + /// from where it starts; one that starts lower replaces it. One that + /// leaves a gap above it is dropped: the run nearer the tip says more. + pub fn note_headers(&mut self, tip: u32, height: u32, headers: &[Header]) { + self.probed = true; + let drop = tip.saturating_add(1).saturating_sub(self.from) as usize; + self.announced.drain(..drop.min(self.announced.len())); + self.from = self.from.max(tip.saturating_add(1)); + let skip = tip.saturating_add(1).saturating_sub(height) as usize; + let Some(fresh) = headers.get(skip..).filter(|h| !h.is_empty()) else { return; + }; + let start = height.saturating_add(skip as u32); + let end = self.from.saturating_add(self.announced.len() as u32); + if self.announced.is_empty() || start < self.from { + self.announced.clear(); + self.from = start; + } else if start > end { + return; + } else { + self.announced.truncate((start - self.from) as usize); } - if let (Some((_, fork)), Some(matched)) = (self.forks_at, matched_to) { - if matched >= fork { - self.forks_at = None; - } - } + self.announced.extend(fresh.iter().map(|h| h.block_hash())); + self.announced.truncate(ANNOUNCED_CAP); } - /// The peer's chain holds a different block than the path at or below - /// `height`, so it does not have the path block there. + /// The peer's announced chain holds a different block than the path at + /// `height`, or at its own top below it. Two chains that differ at one + /// height differ above it, so the peer does not have the path block. pub fn left_path_by(&self, path: &HashMap, height: u32) -> bool { - self.forks_at.is_some_and(|(hash, fork)| { - fork <= height && path.get(&fork).is_some_and(|p| *p != hash) - }) + let Some(len) = (self.announced.len() as u32).checked_sub(1) else { + return false; + }; + if height < self.from { + return false; + } + let at = height.min(self.from.saturating_add(len)); + let peer = self.announced[(at - self.from) as usize]; + path.get(&at).is_some_and(|p| *p != peer) } } diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index bca711d8d..805e10606 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1588,6 +1588,48 @@ fn two_peers_reserve_the_header_walk() { assert!(!st.slots[0].alive); assert!(st.addr_cooldown.contains_key(&addr)); assert!(matches!(rx.try_recv(), Ok(PeerCmd::Shutdown))); + + // A fourth, because the header peer must hold the path: a block peer + // that never announced a chain is asked for headers once. Its reply + // forks below the path, so it is not asked for path blocks. + let (mut head, mut head_rx) = slot(5); + let (mut other, mut other_rx) = slot(6); + head.first_data_ms = 50; + other.first_data_ms = 5_000; + let mut st = IbdWorkState::new(vec![head, other], Some(gen), Some(0)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + assert_eq!(drain_getheaders(&mut head_rx), 1); + let path = [mine(gen, 41), mine(mine(gen, 41).block_hash(), 42)]; + apply(&mut st, &hub, 5, path.to_vec()); + assert_eq!(st.header_walk.reserved, Some(5)); + let _ = take_header_ask(&mut st, 5); + challenge_unprobed_peer(&mut st); + assert!(send_getheaders(&mut st, &hub).unwrap()); + assert_eq!( + drain_getheaders(&mut other_rx), + 1, + "the silent peer is asked once" + ); + assert_eq!(drain_getheaders(&mut head_rx), 0); + apply(&mut st, &hub, 6, vec![mine(gen, 43)]); + challenge_unprobed_peer(&mut st); + assert!(st.header_walk.challenge.is_none(), "a peer is probed once"); + for h in &path { + st.body.mark_missing(h.block_hash()); + } + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!( + st.slots[1].in_flight.is_empty(), + "a peer whose chain left the path below a block is not asked for it" + ); + assert!(st.slots[0].in_flight.contains(&path[0].block_hash())); } #[test] From 4aaa1e245526026ecc67906089dabd03fa6948f8 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 15:03:07 -0400 Subject: [PATCH 05/17] net: close the gaps review found in the IBD retire A review of this branch found these, each pinned before the fix: - A header batch can be written outside the lock while one of its blocks fails. `on_headers_batch` now refuses a batch that holds or builds on a rejected block at commit too, so a retire is not undone. - `retire_invalid` left no checkpoint at the parent, so `header.adopt` restored an older tip or none. It now records one. - The failure mark was written before the retire saved the walk. The retire runs first, and `restore_adopt` does not bring back a saved candidate above the confirmed tip while invalid blocks are stored. - A peer whose reply was shorter than a full batch has nothing past its top and is not asked above it. Where its chain split from this node's (path or confirmed) is kept when that height leaves the path. When no peer holds a block by what it announced, any peer that did not answer `notfound` may be asked, so a stale record cannot stall a download. - A block refused because it was already marked invalid (an operator `invalidateblock`) is not written to `block.invalid`. - `block.invalid` is written through the store's synced temp-file helper (sync, rename, directory sync). Durability is not observable in-process; the on-disk pins still cover the bytes. Co-Authored-By: Claude Opus 5.5 --- SCHEMA.md | 4 +- TESTING.md | 2 +- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 20 +++++- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 38 ++++++++-- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 62 +++++++++++++--- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 74 +++++++++++++++++++- crates/rbitcoin-net/src/chain.rs | 15 ++-- crates/rbitcoin-net/tests/chain.rs | 15 ++++ 8 files changed, 200 insertions(+), 30 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index da747d070..552416dd1 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -393,11 +393,11 @@ then milestone hash [u8; 32] (zeros if none) Each checkpoint is 285 bytes. The tail is 364 bytes. Length is exact: `12 + count × 285 + 364`. An older or shorter file does not parse, and the script skip stays off. -Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. +Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. While `block.invalid` lists a block, a saved candidate above the confirmed tip is not restored: it may build on that block, so the walk starts again from the tip. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. ### `block.invalid` -Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (temp file, then rename) for each new failure and when `reconsiderblock` clears one. +Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (synced temp file, rename, directory sync) for each new failure and when `reconsiderblock` clears one. ```text 0..8 magic b"rbtcinv1" diff --git a/TESTING.md b/TESTING.md index aff554c1f..a2edeee1a 100644 --- a/TESTING.md +++ b/TESTING.md @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once, and once its reply forks below the path it is not asked for path blocks. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it, also once the fork height has left the path; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index feda936e0..4f55bc1e2 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -325,6 +325,11 @@ pub fn on_headers_batch( fks: &[rbitcoin_primitives::Fk], work: Option, ) -> usize { + // A block in this batch can fail while its headers are written outside + // the lock. The retire already dropped that path. + if super::header_walk::refuse_invalid_batch(st, peer, &headers) { + return 0; + } let accepted = accepted_headers(&headers, fks); let mut added = 0usize; let mut batch_prev: Option<(BlockHash, u32)> = None; @@ -901,9 +906,18 @@ fn note_peer_headers( else { return; }; + let tip = hub.tip_height().unwrap_or(0); + let path = &st.height_to_hash; + let ours = |ht: u32| { + path.get(&ht).copied().or_else(|| { + (ht <= tip) + .then(|| hub.query.header_at_height(rbitcoin_primitives::Height(ht))) + .and_then(|r| r.ok().flatten()) + .map(|(_, rec)| BlockHash::from_byte_array(rec.hash)) + }) + }; if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { - s.chain - .note_headers(hub.tip_height().unwrap_or(0), height, headers); + s.chain.note_headers(ours, tip, height, headers); } } @@ -1438,8 +1452,8 @@ pub fn apply_consensus_invalid_reject( st.body.mark_rejected(hash); st.reorg.invalid.mark(hash.to_byte_array()); if let Some(h) = hub { - h.note_failed_connect(hash); super::header_walk::retire_invalid(st, h, hash, height); + h.note_failed_connect(hash); } if st.height_to_hash.get(&height) == Some(&hash) { st.height_to_hash.remove(&height); diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 440b579cf..e804afe84 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -639,16 +639,20 @@ pub fn note_reservation( } /// Whether block `hash` may be asked of `peer`. A peer is not asked for a -/// block it answered `notfound`, or for a path block above where its -/// announced chain left the path. The reserved header peer is asked only -/// when no other live peer may have it, so a single peer still does both. +/// block it answered `notfound`, or for a path block its announced chain +/// does not hold. The reserved header peer is asked only when no other live +/// peer may have it, so a single peer still does both. pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { + use super::peer_io::PeerSlot; let height = st.hash_height.get(hash).copied(); - let has = |s: &super::peer_io::PeerSlot| { - s.alive - && !s.chain.lacks(hash) - && !height.is_some_and(|ht| s.chain.left_path_by(&st.height_to_hash, ht)) + let askable = |s: &PeerSlot| s.alive && !s.chain.lacks(hash); + let holds = |s: &PeerSlot| { + askable(s) && !height.is_some_and(|ht| s.chain.left_path_by(&st.height_to_hash, ht)) }; + // A chain record can be stale. With no peer that holds the block by what + // it announced, any peer that did not answer `notfound` may be asked. + let by_chain = st.slots.iter().any(&holds); + let has = |s: &PeerSlot| if by_chain { holds(s) } else { askable(s) }; if !st.slots.iter().any(|s| s.id == peer && has(s)) { return false; } @@ -850,6 +854,16 @@ pub fn retire_invalid(st: &mut IbdWorkState, hub: &ChainHub, hash: BlockHash, he st.header_walk.base_diff = tip.diff; } st.header_walk.tip = tip; + // `header.adopt` restores the tip from the last checkpoint. + if st + .header_walk + .checkpoints + .last() + .is_none_or(|c| c.height < parent_h) + { + let at_parent = checkpoint_at(&st.header_walk.tip); + st.header_walk.checkpoints.push(at_parent); + } // A challenger's headers are not stored, and its fork can sit below the // confirmed tip. Peers are asked again from the parent, and a heavier // chain comes back as headers that are stored and can be rewound onto. @@ -3111,6 +3125,16 @@ pub fn restore_adopt(st: &mut IbdWorkState, hub: &ChainHub) -> bool { renote_stored_path(st, hub); return false; } + // A saved candidate above the confirmed tip may build on a block marked + // invalid after it was saved. The walk starts again from the tip. + if parsed.tip.height > hub.tip_height().unwrap_or(0) && !hub.invalid_blocks().is_empty() { + rbitcoin_log::info!( + "ibd: headers resume refused height={}: invalid blocks are stored", + parsed.tip.height + ); + renote_stored_path(st, hub); + return false; + } st.header_walk = parsed; restore_tip_header(st, hub); renote_stored_path(st, hub); diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 4e325f3b7..116a2c6e5 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -22,6 +22,12 @@ pub struct PeerChain { announced: VecDeque, /// Height of `announced[0]`. from: u32, + /// The last reply was shorter than a full `headers` batch: the peer had + /// nothing past its top. + complete: bool, + /// The peer's header at the first height where its announced chain + /// differed from this node's chain (path or confirmed) when announced. + split: Option<(u32, BlockHash)>, /// The peer announced headers, or was asked for them once. probed: bool, } @@ -50,17 +56,43 @@ impl PeerChain { self.probed = true; } - /// Keep announced `headers`, the first at `height`, above the confirmed - /// `tip`. A batch that overlaps or continues the kept run replaces it - /// from where it starts; one that starts lower replaces it. One that + /// Keep announced `headers`, the first at `height`. `ours` is this node's + /// block at a height (path or confirmed), when known. The first height + /// where they differ is where the peer leaves this chain; a later reply + /// that agrees there clears it. Only heights above the confirmed `tip` + /// are kept. A reply that overlaps or continues the kept run replaces it + /// from where it starts, one that starts lower replaces it, and one that /// leaves a gap above it is dropped: the run nearer the tip says more. - pub fn note_headers(&mut self, tip: u32, height: u32, headers: &[Header]) { + pub fn note_headers( + &mut self, + ours: impl Fn(u32) -> Option, + tip: u32, + height: u32, + headers: &[Header], + ) { self.probed = true; + self.complete = headers.len() < ANNOUNCED_CAP; + let hashes: Vec = headers.iter().map(|h| h.block_hash()).collect(); + for (ht, hash) in (height..).zip(&hashes) { + let Some(our) = ours(ht) else { + continue; + }; + if our == *hash { + if self.split.is_some_and(|(at, _)| at == ht) { + self.split = None; + } + continue; + } + if self.split.is_none_or(|(at, _)| ht <= at) { + self.split = Some((ht, *hash)); + } + break; + } let drop = tip.saturating_add(1).saturating_sub(self.from) as usize; self.announced.drain(..drop.min(self.announced.len())); self.from = self.from.max(tip.saturating_add(1)); let skip = tip.saturating_add(1).saturating_sub(height) as usize; - let Some(fresh) = headers.get(skip..).filter(|h| !h.is_empty()) else { + let Some(fresh) = hashes.get(skip..).filter(|h| !h.is_empty()) else { return; }; let start = height.saturating_add(skip as u32); @@ -73,21 +105,31 @@ impl PeerChain { } else { self.announced.truncate((start - self.from) as usize); } - self.announced.extend(fresh.iter().map(|h| h.block_hash())); + self.announced.extend(fresh.iter().copied()); self.announced.truncate(ANNOUNCED_CAP); } - /// The peer's announced chain holds a different block than the path at - /// `height`, or at its own top below it. Two chains that differ at one - /// height differ above it, so the peer does not have the path block. + /// The peer does not have the path block at `height`: its chain left this + /// node's chain at or below it, it holds a different block there, or its + /// whole chain ends below it. Two chains that differ at one height differ + /// above it. A recorded split whose height has left the path still holds. pub fn left_path_by(&self, path: &HashMap, height: u32) -> bool { + if let Some((at, hash)) = self.split { + if at <= height && path.get(&at).is_none_or(|p| *p != hash) { + return true; + } + } let Some(len) = (self.announced.len() as u32).checked_sub(1) else { return false; }; if height < self.from { return false; } - let at = height.min(self.from.saturating_add(len)); + let top = self.from.saturating_add(len); + if height > top && self.complete { + return true; + } + let at = height.min(top); let peer = self.announced[(at - self.from) as usize]; path.get(&at).is_some_and(|p| *p != peer) } diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 805e10606..5bc89400d 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1313,6 +1313,29 @@ fn restart_restores_the_checkpoint_skip_and_garbage_does_not() { !skips(&hub, 1, h1.block_hash().as_byte_array()), "a file that does not parse does not skip scripts by height" ); + + // A candidate retired on a failed block restarts at its parent. + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-retire"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let p1 = mine(gen, 61); + let p2 = mine(p1.block_hash(), 62); + let p3 = mine(p2.block_hash(), 63); + let (s1, _rx1) = slot(1); + let mut st = IbdWorkState::new(vec![s1], Some(gen), Some(0)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + apply(&mut st, &hub, 1, vec![p1, p2, p3]); + assert!(st.is_on_path(&p3.block_hash(), 3)); + retire_invalid(&mut st, &hub, p3.block_hash(), 3); + st.header_walk = HeaderWalk::default(); + assert!(restore_adopt(&mut st, &hub)); + assert_eq!(st.header_walk.tip_hash(), Some(p2.block_hash())); + assert_eq!(st.header_walk.tip_height(), 2); + // Once a failed block is stored, a saved candidate above the confirmed + // tip may build on it and is not restored. + hub.note_failed_connect(p3.block_hash()); + st.header_walk = HeaderWalk::default(); + assert!(!restore_adopt(&mut st, &hub)); } #[test] @@ -1594,9 +1617,11 @@ fn two_peers_reserve_the_header_walk() { // forks below the path, so it is not asked for path blocks. let (mut head, mut head_rx) = slot(5); let (mut other, mut other_rx) = slot(6); + let (mut short, _short_rx) = slot(7); head.first_data_ms = 50; other.first_data_ms = 5_000; - let mut st = IbdWorkState::new(vec![head, other], Some(gen), Some(0)); + short.first_data_ms = 6_000; + let mut st = IbdWorkState::new(vec![head, other, short], Some(gen), Some(0)); assert!(send_getheaders(&mut st, &hub).unwrap()); assert_eq!(drain_getheaders(&mut head_rx), 1); let path = [mine(gen, 41), mine(mine(gen, 41).block_hash(), 42)]; @@ -1611,7 +1636,10 @@ fn two_peers_reserve_the_header_walk() { "the silent peer is asked once" ); assert_eq!(drain_getheaders(&mut head_rx), 0); - apply(&mut st, &hub, 6, vec![mine(gen, 43)]); + let fork = mine(gen, 43); + apply(&mut st, &hub, 6, vec![fork, mine(fork.block_hash(), 46)]); + // A short reply is everything that peer has: its chain ends at height 1. + apply(&mut st, &hub, 7, vec![path[0]]); challenge_unprobed_peer(&mut st); assert!(st.header_walk.challenge.is_none(), "a peer is probed once"); for h in &path { @@ -1629,7 +1657,47 @@ fn two_peers_reserve_the_header_walk() { st.slots[1].in_flight.is_empty(), "a peer whose chain left the path below a block is not asked for it" ); - assert!(st.slots[0].in_flight.contains(&path[0].block_hash())); + assert!( + !st.slots[2].in_flight.contains(&path[1].block_hash()), + "a peer whose whole chain ends below a block is not asked for it" + ); + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!(!st.slots[2].in_flight.contains(&path[1].block_hash())); + assert!( + st.slots[0].in_flight.contains(&path[1].block_hash()), + "the header peer takes what no other peer holds" + ); + + // Once the fork height leaves the path (confirmed and pruned), that peer + // has still left it. + st.height_to_hash.remove(&1); + st.height_to_hash.remove(&2); + assert!(!may_serve(&st, 6, &path[1].block_hash())); + // With no peer that holds the block by what it announced, a peer that + // did not answer `notfound` is still asked. + st.slots[0].chain.note_notfound(path[1].block_hash()); + st.slots[2].chain.note_notfound(path[1].block_hash()); + assert!(may_serve(&st, 6, &path[1].block_hash())); + + // A batch stored after one of its blocks failed is not planted, and does + // not grow the candidate again. + let late0 = mine(path[1].block_hash(), 44); + let late = [late0, mine(late0.block_hash(), 45)]; + st.reorg.invalid.mark(late[0].block_hash().to_byte_array()); + let tip = st.header_walk.tip_hash(); + let (fks, work) = rbitcoin_ibd::ibd::events::prepare_header_store(&hub, &late); + let added = + rbitcoin_ibd::ibd::events::on_headers_batch(&mut st, &hub, 5, late.to_vec(), &fks, work); + assert_eq!(added, 0); + assert!(!st.ordered_set.contains(&late[1].block_hash())); + assert_eq!(st.header_walk.tip_hash(), tip); } #[test] diff --git a/crates/rbitcoin-net/src/chain.rs b/crates/rbitcoin-net/src/chain.rs index 01f3b1a2f..a91e0268f 100644 --- a/crates/rbitcoin-net/src/chain.rs +++ b/crates/rbitcoin-net/src/chain.rs @@ -243,6 +243,9 @@ pub struct Invalidated { pub file: std::path::PathBuf, } +/// Refusal of a hash already marked invalid. Not a fresh validation result. +const BLOCK_IS_INVALIDATED: &str = "block is invalidated"; + /// Sidecar of connect failures next to `header.adopt`. Not a schema bump. const INVALID_FILE: &str = "block.invalid"; const INVALID_MAGIC: &[u8; 8] = b"rbtcinv1"; @@ -274,8 +277,9 @@ fn write_invalid_file(path: &std::path::Path, hashes: &[BlockHash]) { for h in hashes { buf.extend_from_slice(h.as_byte_array()); } - let tmp = path.with_file_name("block.invalid.tmp"); - let wrote = std::fs::write(&tmp, &buf).and_then(|()| std::fs::rename(&tmp, path)); + let wrote = rbitcoin_store::file::write_synced_tmp_file(path, |f| { + std::io::Write::write_all(f, &buf).map_err(|e| rbitcoin_store::StoreError::io(path, e)) + }); if let Err(e) = wrote { warn!("chain: write {}: {e}", path.display()); } @@ -1452,6 +1456,9 @@ impl ChainHub { /// not poison later getdata of that hash. #[doc(hidden)] pub fn remember_failed_accept(&self, offered: BlockHash, e: &NetError) { + if matches!(e, NetError::Consensus(m) if m == BLOCK_IS_INVALIDATED) { + return; + } let hash = e .failing_block_hash() .map(BlockHash::from_byte_array) @@ -2651,7 +2658,7 @@ impl ChainHub { return Ok(AcceptOutcome::AlreadyHave); } if self.invalidated.set.read().unwrap().contains(&hash) { - return Err(NetError::Consensus("block is invalidated".into())); + return Err(NetError::Consensus(BLOCK_IS_INVALIDATED.into())); } if rbitcoin_consensus::block_mutated_without_coinbase(&block) { return Err(NetError::Mutated( @@ -2794,7 +2801,7 @@ impl ChainHub { .iter() .any(|b| self.is_block_invalid(&b.block_hash())) { - return Err(NetError::Consensus("block is invalidated".into())); + return Err(NetError::Consensus(BLOCK_IS_INVALIDATED.into())); } for w in blocks.windows(2) { if w[1].header.prev_blockhash != w[0].block_hash() { diff --git a/crates/rbitcoin-net/tests/chain.rs b/crates/rbitcoin-net/tests/chain.rs index 70fa07575..614243b9d 100644 --- a/crates/rbitcoin-net/tests/chain.rs +++ b/crates/rbitcoin-net/tests/chain.rs @@ -4105,6 +4105,21 @@ fn reconsider_skips_a_branch_that_fails() { hub.invalidate_block(bad5_hash).unwrap(); hub.reconsider_block(bad5_hash).unwrap(); assert!(!kept(&marks), "reconsiderblock forgets the kept failure"); + let tip_block = hub + .block_at_height(hub.tip_height().unwrap()) + .unwrap() + .unwrap(); + let tip_hash = tip_block.block_hash(); + hub.invalidate_block(tip_hash).unwrap(); + assert!(hub.accept_received_block(tip_block).is_err()); + assert!( + !std::fs::read(&marks) + .unwrap() + .windows(32) + .any(|w| w == tip_hash.as_byte_array()), + "a block refused as already invalidated is not a connect failure" + ); + hub.reconsider_block(tip_hash).unwrap(); assert!( !hub.fork_tips.read().unwrap().iter().any(|h| h == bad5_hash), "a block that never connected is not kept as a fork tip" From c6a81220936e132186e4546acef2f440d0bb2d0a Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 16:01:33 -0400 Subject: [PATCH 06/17] net: keep a peer's newest announced chain and settle its split Review on #1112: - A recorded split whose height had been confirmed and pruned from the path still counted, so the peer whose chain IBD had just taken was no longer asked for its blocks. A missing path slot no longer counts by itself: before block assignment the split is compared with this node's confirmed block there; the same block clears it, a different one keeps it. - The announced run kept the oldest reply: a continuation was cut off at one reply, the next full reply landed in a gap and was dropped, and a later short reply still set `complete`, so the peer looked finished at a stale top. The newest run is kept now, and `complete` describes it. - With no peer that holds a block by what it announced, the fallback skipped the header peer while another peer was up. It no longer does. - `invalid_tips` starts again at its cap, so the newest refused tip of a long invalid chain is kept, and a full `block.invalid` drops its oldest hash so the newest failure survives restart. Co-Authored-By: Claude Opus 5.5 --- COMPAT.md | 2 +- SCHEMA.md | 2 +- TESTING.md | 2 +- crates/rbitcoin-ibd/src/ibd/assign.rs | 1 + crates/rbitcoin-ibd/src/ibd/events/mod.rs | 16 ++-- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 33 ++++++-- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 72 ++++++++++++++---- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 80 ++++++++++++++++++-- crates/rbitcoin-ibd/tests/ibd/mod.rs | 1 + crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 78 +++++++++++++++++++ crates/rbitcoin-net/src/chain.rs | 7 +- crates/rbitcoin-net/tests/chain.rs | 32 ++++++++ 12 files changed, 283 insertions(+), 43 deletions(-) create mode 100644 crates/rbitcoin-ibd/tests/ibd/peer_chain.rs diff --git a/COMPAT.md b/COMPAT.md index e06e0d507..1fc3b75d8 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -82,7 +82,7 @@ Full `/tx/:txid` JSON still has `vin[]`. Electrum has no outspend-vin surface. | IBD empty-headers EOF | Empty `headers` to **our locator** + idle path latches `headers_done` even if a peer advertises a taller less-work height / junk `version.start_height`. A proven header walk at the tip that every taller peer failed to extend also ends IBD | Core header sync follows most-work; advertised `start_height` is not a remaining header count | | BIP94 | Not implemented. There is no testnet4 chain and no `-test=bip94`. A retarget multiplies the parent, and a difficulty-boundary timestamp only has to be after the median of ancestor timestamps. This does not fork mainnet, testnet3, signet, or default regtest | On testnet4, and on regtest only with `-test=bip94`, a retarget multiplies the first block of the period (that parent may be a min-difficulty exception) and a difficulty-boundary timestamp more than 600 seconds before its parent is `time-timewarp-attack` | | Milestone script skip | A height-only `--milestone HEIGHT` skips scripts at and below that height with no anchor hash, including when the header lookup is empty. The default anchor skips once two height-to-hash lookups match: this block, and the anchor hash at the anchor height. It does not walk parents, does not read `min_work_be`, and does not require two weeks of burial. Prevouts, double spends, maturity, and fees still run. An output whose script starts with `OP_RETURN`, or is longer than 10,000 bytes, is still not a coin under the skip | Core runs scripts unless the assumevalid block is an ancestor, the best header is on that chain, chain work meets `nMinimumChainWork`, and the proof-equivalent time from the assumevalid block to the best header is more than two weeks | -| Invalid marks across restart | A block that failed connect on a consensus rule stays invalid across restart (`store/block.invalid`, at most 4096). An operator `invalidateblock` does not: it lasts until the process exits | Both live in the block index (`BLOCK_FAILED_VALID`) and survive restart | +| Invalid marks across restart | A block that failed connect on a consensus rule stays invalid across restart (`store/block.invalid`, the newest 4096). An operator `invalidateblock` does not: it lasts until the process exits | Both live in the block index (`BLOCK_FAILED_VALID`) and survive restart | | Equal-work fork choice | A connected tip is replaced only by strictly greater chain work, or by `preciousblock` at equal work. After `invalidateblock`, equal work prefers precious, then the earlier held tip. Header ranking before either chain is connected also stays with the chain already chosen when work is equal ([`docs/architecture.md`](./docs/architecture.md) layer 1, “equal work keeps the first seen”). A block that was stored earlier but was not yet connectable does not take the tip later | When work is equal, Core prefers the block seen first (`nSequenceId`). A block stored earlier but not yet connectable becomes the tip once it can be connected, even if a later equal-work block is already the tip. Neither rule makes a block valid on one side and invalid on the other | Compact reconstruct fills short-ids from the live mempool graph, the diff --git a/SCHEMA.md b/SCHEMA.md index 552416dd1..fcaad9dd2 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -405,7 +405,7 @@ Sidecar of blocks that failed connect on a consensus rule, so a restart does not repeat block hash [u8; 32] ``` -Length is exact: `12 + count × 32`. Oldest first. Past 4096 a new failure is marked in memory only. Mutated bodies, a header timestamp too far in the future, store and engine faults, and operator `invalidateblock` are not written. A descendant is invalid through its ancestor: the startup header scan and the IBD resume seed do not walk through a listed hash. +Length is exact: `12 + count × 32`. Oldest first. Past 4096 the oldest hash gives way, so the newest failure survives restart. Mutated bodies, a header timestamp too far in the future, store and engine faults, and operator `invalidateblock` are not written. A descendant is invalid through its ancestor: the startup header scan and the IBD resume seed do not walk through a listed hash. --- diff --git a/TESTING.md b/TESTING.md index a2edeee1a..b9a71f732 100644 --- a/TESTING.md +++ b/TESTING.md @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it, also once the fork height has left the path; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/crates/rbitcoin-ibd/src/ibd/assign.rs b/crates/rbitcoin-ibd/src/ibd/assign.rs index 70af07cec..e8239a721 100644 --- a/crates/rbitcoin-ibd/src/ibd/assign.rs +++ b/crates/rbitcoin-ibd/src/ibd/assign.rs @@ -246,6 +246,7 @@ pub fn assign_work_ordered( if alive.is_empty() { return; } + super::header_walk::settle_peer_splits(st, hub); let (bq_bytes, bq_count) = hub.query.block_queue_stats(); let bq_stop = cfg.assign_stop_bytes; diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 4f55bc1e2..7d646badb 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -908,16 +908,14 @@ fn note_peer_headers( }; let tip = hub.tip_height().unwrap_or(0); let path = &st.height_to_hash; - let ours = |ht: u32| { - path.get(&ht).copied().or_else(|| { - (ht <= tip) - .then(|| hub.query.header_at_height(rbitcoin_primitives::Height(ht))) - .and_then(|r| r.ok().flatten()) - .map(|(_, rec)| BlockHash::from_byte_array(rec.hash)) - }) - }; if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { - s.chain.note_headers(ours, tip, height, headers); + s.chain.note_headers( + path, + |ht| super::peer_chain::confirmed_at(hub, ht), + tip, + height, + headers, + ); } } diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index e804afe84..ae0603274 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -138,7 +138,8 @@ pub const LOOKAHEAD_ASK: Duration = Duration::from_secs(5); pub const HEADER_MISS_DISCONNECT: u8 = 2; #[doc(hidden)] pub const DEAD_END_CAP: usize = 1024; -/// Past the cap a new invalid tip is not stored; the oldest stay. +/// Past the cap the set starts again, so the newest refused tip is kept: the +/// next batch of a long invalid chain links to it. const INVALID_TIP_CAP: usize = 1024; impl HeaderWalk { @@ -649,14 +650,29 @@ pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { let holds = |s: &PeerSlot| { askable(s) && !height.is_some_and(|ht| s.chain.left_path_by(&st.height_to_hash, ht)) }; + let Some(slot) = st.slots.iter().find(|s| s.id == peer) else { + return false; + }; // A chain record can be stale. With no peer that holds the block by what - // it announced, any peer that did not answer `notfound` may be asked. - let by_chain = st.slots.iter().any(&holds); - let has = |s: &PeerSlot| if by_chain { holds(s) } else { askable(s) }; - if !st.slots.iter().any(|s| s.id == peer && has(s)) { + // it announced, any peer that did not answer `notfound` may be asked, + // the header peer too. + if !st.slots.iter().any(&holds) { + return askable(slot); + } + if !holds(slot) { return false; } - st.header_walk.reserved != Some(peer) || !st.slots.iter().any(|s| s.id != peer && has(s)) + st.header_walk.reserved != Some(peer) || !st.slots.iter().any(|s| s.id != peer && holds(s)) +} + +/// Settle each peer's recorded split against this node's confirmed chain once +/// that height has left the path. Runs before block assignment. +pub fn settle_peer_splits(st: &mut IbdWorkState, hub: &ChainHub) { + let path = &st.height_to_hash; + for slot in &mut st.slots { + slot.chain + .settle_split(path, |ht| super::peer_chain::confirmed_at(hub, ht)); + } } /// Inputs for one main-loop header poll. The loop owns the cadence clock. @@ -802,9 +818,10 @@ pub fn note_dead(st: &mut IbdWorkState, hash: BlockHash) { #[doc(hidden)] fn note_invalid_tip(st: &mut IbdWorkState, hash: BlockHash) { - if st.header_walk.invalid_tips.len() < INVALID_TIP_CAP { - st.header_walk.invalid_tips.insert(hash); + if st.header_walk.invalid_tips.len() >= INVALID_TIP_CAP { + st.header_walk.invalid_tips.clear(); } + st.header_walk.invalid_tips.insert(hash); } /// Refuse a batch that holds or builds on a rejected block, and retire diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 116a2c6e5..465c58451 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -28,6 +28,9 @@ pub struct PeerChain { /// The peer's header at the first height where its announced chain /// differed from this node's chain (path or confirmed) when announced. split: Option<(u32, BlockHash)>, + /// That height has left the path and this node's confirmed block there + /// is a different one. + split_confirmed: bool, /// The peer announced headers, or was asked for them once. probed: bool, } @@ -56,25 +59,25 @@ impl PeerChain { self.probed = true; } - /// Keep announced `headers`, the first at `height`. `ours` is this node's - /// block at a height (path or confirmed), when known. The first height - /// where they differ is where the peer leaves this chain; a later reply - /// that agrees there clears it. Only heights above the confirmed `tip` - /// are kept. A reply that overlaps or continues the kept run replaces it - /// from where it starts, one that starts lower replaces it, and one that - /// leaves a gap above it is dropped: the run nearer the tip says more. + /// Keep announced `headers`, the first at `height`, against this node's + /// chain: the `path`, or a `confirmed` block at or below `tip`. The first + /// height where they differ is where the peer leaves this chain; a later + /// reply that agrees there clears it. Only heights above `tip` are kept, + /// newest last: a reply that continues the kept run extends it, any other + /// replaces it, and the oldest hashes past one full reply give way. pub fn note_headers( &mut self, - ours: impl Fn(u32) -> Option, + path: &HashMap, + confirmed: impl Fn(u32) -> Option, tip: u32, height: u32, headers: &[Header], ) { self.probed = true; - self.complete = headers.len() < ANNOUNCED_CAP; let hashes: Vec = headers.iter().map(|h| h.block_hash()).collect(); for (ht, hash) in (height..).zip(&hashes) { - let Some(our) = ours(ht) else { + let on_path = path.get(&ht).copied(); + let Some(our) = on_path.or_else(|| (ht <= tip).then(|| confirmed(ht)).flatten()) else { continue; }; if our == *hash { @@ -85,6 +88,7 @@ impl PeerChain { } if self.split.is_none_or(|(at, _)| ht <= at) { self.split = Some((ht, *hash)); + self.split_confirmed = on_path.is_none(); } break; } @@ -97,25 +101,50 @@ impl PeerChain { }; let start = height.saturating_add(skip as u32); let end = self.from.saturating_add(self.announced.len() as u32); - if self.announced.is_empty() || start < self.from { + if self.announced.is_empty() || start < self.from || start > end { self.announced.clear(); self.from = start; - } else if start > end { - return; } else { self.announced.truncate((start - self.from) as usize); } self.announced.extend(fresh.iter().copied()); - self.announced.truncate(ANNOUNCED_CAP); + let over = self.announced.len().saturating_sub(ANNOUNCED_CAP); + self.announced.drain(..over); + self.from = self.from.saturating_add(over as u32); + self.complete = headers.len() < ANNOUNCED_CAP; + } + + /// The recorded split's height has left the path: compare it with this + /// node's `confirmed` block there. The same block means this node took + /// the peer's chain, so the split is gone. + pub fn settle_split( + &mut self, + path: &HashMap, + confirmed: impl Fn(u32) -> Option, + ) { + let Some((at, hash)) = self.split else { + return; + }; + if path.contains_key(&at) { + return; + } + match confirmed(at) { + Some(ours) if ours == hash => self.split = None, + ours => self.split_confirmed = ours.is_some(), + } } /// The peer does not have the path block at `height`: its chain left this /// node's chain at or below it, it holds a different block there, or its /// whole chain ends below it. Two chains that differ at one height differ - /// above it. A recorded split whose height has left the path still holds. + /// above it. pub fn left_path_by(&self, path: &HashMap, height: u32) -> bool { if let Some((at, hash)) = self.split { - if at <= height && path.get(&at).is_none_or(|p| *p != hash) { + let differs = match path.get(&at) { + Some(p) => *p != hash, + None => self.split_confirmed, + }; + if at <= height && differs { return true; } } @@ -134,3 +163,14 @@ impl PeerChain { path.get(&at).is_some_and(|p| *p != peer) } } + +/// This node's confirmed block at `height`. +pub fn confirmed_at(hub: &rbitcoin_net::chain::ChainHub, height: u32) -> Option { + use bitcoin::hashes::Hash; + let (_, rec) = hub + .query + .header_at_height(rbitcoin_primitives::Height(height)) + .ok() + .flatten()?; + Some(BlockHash::from_byte_array(rec.hash)) +} diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 5bc89400d..3334c68cf 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1675,11 +1675,6 @@ fn two_peers_reserve_the_header_walk() { "the header peer takes what no other peer holds" ); - // Once the fork height leaves the path (confirmed and pruned), that peer - // has still left it. - st.height_to_hash.remove(&1); - st.height_to_hash.remove(&2); - assert!(!may_serve(&st, 6, &path[1].block_hash())); // With no peer that holds the block by what it announced, a peer that // did not answer `notfound` is still asked. st.slots[0].chain.note_notfound(path[1].block_hash()); @@ -1698,6 +1693,81 @@ fn two_peers_reserve_the_header_walk() { assert_eq!(added, 0); assert!(!st.ordered_set.contains(&late[1].block_hash())); assert_eq!(st.header_walk.tip_hash(), tip); + + // A fifth, on its own hub, because this node takes the forked peer's + // chain: once the fork height is confirmed and leaves the path, that peer + // holds the next path block again, also while a peer that announced + // nothing is up. + let (_dir5, hub5) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-taken"); + hub5.ensure_genesis().unwrap(); + let gen5 = hub5.tip_hash().unwrap(); + let (mut head, _head_rx) = slot(8); + let (forked, _forked_rx) = slot(9); + let (silent, _silent_rx) = slot(10); + let (other_fork, _other_rx) = slot(11); + head.first_data_ms = 50; + let mut st = IbdWorkState::new(vec![head, forked, silent, other_fork], Some(gen5), Some(0)); + assert!(send_getheaders(&mut st, &hub5).unwrap()); + let lost0 = mine(gen5, 81); + apply(&mut st, &hub5, 8, vec![lost0, mine(lost0.block_hash(), 82)]); + let taken1 = mine_block(gen5, 1); + let taken2 = mine_block(taken1.block_hash(), 2); + apply(&mut st, &hub5, 9, vec![taken1.header, taken2.header]); + let elsewhere = mine(gen5, 83); + apply( + &mut st, + &hub5, + 11, + vec![elsewhere, mine(elsewhere.block_hash(), 84)], + ); + hub5.accept_block(taken1.clone()).unwrap(); + st.height_to_hash.remove(&1); + st.height_to_hash.insert(2, taken2.block_hash()); + st.record_height(taken2.block_hash(), 2); + st.body.mark_missing(taken2.block_hash()); + rbitcoin_ibd::ibd::assign::assign_work_ordered( + &mut st, + &hub5, + &cfg, + &stats, + rbitcoin_ibd::ibd::assign::AssignDepth::Full, + None, + ); + assert!( + may_serve(&st, 9, &taken2.block_hash()), + "the peer whose chain this node took is asked again" + ); + assert!( + !may_serve(&st, 11, &taken2.block_hash()), + "a peer that left where this node confirmed a different block has still left" + ); + // With no peer that holds a block by what it announced, the header peer + // is asked too. + st.slots[1].chain.note_notfound(taken2.block_hash()); + st.slots[2].chain.note_notfound(taken2.block_hash()); + assert!(may_serve(&st, 8, &taken2.block_hash())); +} + +/// A long chain on an invalid block stays refused past the cap on +/// remembered tips: the newest link is kept. +#[test] +fn a_long_chain_on_an_invalid_block_stays_refused() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-refuse-cap"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let (s0, _rx0) = slot(0); + let mut st = IbdWorkState::new(vec![s0], Some(gen), Some(0)); + let bad = mine(gen, 1); + st.reorg.invalid.mark(bad.block_hash().to_byte_array()); + let mut prev = bad.block_hash(); + for n in 0..1_100u32 { + let next = mine(prev, 2 + n); + assert!( + refuse_invalid_batch(&mut st, 0, &[next]), + "batch {n} builds on the refused chain" + ); + prev = next.block_hash(); + } } #[test] diff --git a/crates/rbitcoin-ibd/tests/ibd/mod.rs b/crates/rbitcoin-ibd/tests/ibd/mod.rs index 8dfd33023..8a54d7c91 100644 --- a/crates/rbitcoin-ibd/tests/ibd/mod.rs +++ b/crates/rbitcoin-ibd/tests/ibd/mod.rs @@ -11,6 +11,7 @@ mod exit; mod header_walk; mod path; mod peer_book_and_config_tests; +mod peer_chain; mod peer_io; mod perf_log; mod progress; diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs new file mode 100644 index 000000000..7d9a74ecb --- /dev/null +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -0,0 +1,78 @@ +use bitcoin::block::Header; +use bitcoin::hashes::Hash; +use bitcoin::{BlockHash, CompactTarget, TxMerkleNode}; +use rbitcoin_ibd::ibd::peer_chain::PeerChain; +use std::collections::HashMap; + +fn chain(n: u32, salt: u8) -> Vec
{ + let mut prev = BlockHash::from_byte_array([0u8; 32]); + (1..=n) + .map(|i| { + let mut h = Header { + version: bitcoin::block::Version::from_consensus(4), + prev_blockhash: prev, + merkle_root: TxMerkleNode::from_byte_array([salt; 32]), + time: 1_600_000_000 + i, + bits: CompactTarget::from_consensus(0x207fffff), + nonce: i, + }; + rbitcoin_consensus::grind_regtest_pow(&mut h); + prev = h.block_hash(); + h + }) + .collect() +} + +/// Two full `headers` replies and then a short one: the peer holds every +/// block of its newest reply, and none past that reply's top. +#[test] +fn an_announced_chain_keeps_the_newest_reply() { + let headers = chain(4_010, 0x5c); + let path: HashMap = (1..).zip(headers.iter().map(|h| h.block_hash())).collect(); + let mut peer = PeerChain::default(); + peer.note_headers(&path, |_| None, 0, 1, &headers[..2_000]); + peer.note_headers(&path, |_| None, 0, 2_001, &headers[2_000..4_000]); + peer.note_headers(&path, |_| None, 0, 4_001, &headers[4_000..]); + assert!( + !peer.left_path_by(&path, 4_005), + "the newest reply is what the peer has" + ); + assert!(!peer.left_path_by(&path, 3_000)); + let mut beyond = path.clone(); + beyond.insert(4_011, BlockHash::from_byte_array([7u8; 32])); + assert!( + peer.left_path_by(&beyond, 4_011), + "a short reply is the end of that peer's chain" + ); +} + +/// A full reply whose split height has since been confirmed and left the +/// path. This node's confirmed block there settles it: a different block +/// means the peer left this chain, the same block means this node took it. +#[test] +fn a_split_is_settled_by_the_confirmed_chain() { + let ours = chain(2_001, 0x5c); + let theirs = chain(2_000, 0x77); + let mut path: HashMap = + (1..).zip(ours.iter().map(|h| h.block_hash())).collect(); + let mut left = PeerChain::default(); + left.note_headers(&path, |_| None, 0, 1, &theirs); + let mut taken = PeerChain::default(); + let mut before = path.clone(); + before.insert(1, theirs[0].block_hash()); + taken.note_headers(&before, |_| None, 0, 1, &ours[..2_000]); + for h in 1..=2_000 { + path.remove(&h); + } + let confirmed = |h: u32| (h <= 2_000).then(|| ours[h as usize - 1].block_hash()); + left.settle_split(&path, confirmed); + taken.settle_split(&path, confirmed); + assert!( + left.left_path_by(&path, 2_001), + "a peer that split where this node confirmed another block has left" + ); + assert!( + !taken.left_path_by(&path, 2_001), + "a peer whose chain this node confirmed has not" + ); +} diff --git a/crates/rbitcoin-net/src/chain.rs b/crates/rbitcoin-net/src/chain.rs index a91e0268f..90a3dc334 100644 --- a/crates/rbitcoin-net/src/chain.rs +++ b/crates/rbitcoin-net/src/chain.rs @@ -249,7 +249,7 @@ const BLOCK_IS_INVALIDATED: &str = "block is invalidated"; /// Sidecar of connect failures next to `header.adopt`. Not a schema bump. const INVALID_FILE: &str = "block.invalid"; const INVALID_MAGIC: &[u8; 8] = b"rbtcinv1"; -/// Past the cap a new failure is not written; the oldest stay. +/// Past the cap the oldest kept failure gives way, so the newest survives restart. const INVALID_FILE_CAP: usize = 4_096; fn parse_invalid_file(bytes: &[u8]) -> Option> { @@ -320,9 +320,12 @@ impl Invalidated { fn keep_failed(&self, hash: BlockHash) { let mut failed = self.failed.write().unwrap(); - if failed.len() >= INVALID_FILE_CAP || failed.contains(&hash) { + if failed.contains(&hash) { return; } + if failed.len() >= INVALID_FILE_CAP { + failed.remove(0); + } failed.push(hash); write_invalid_file(&self.file, &failed); } diff --git a/crates/rbitcoin-net/tests/chain.rs b/crates/rbitcoin-net/tests/chain.rs index 614243b9d..804ca8381 100644 --- a/crates/rbitcoin-net/tests/chain.rs +++ b/crates/rbitcoin-net/tests/chain.rs @@ -53,6 +53,38 @@ fn tmp_hub() -> (rbitcoin_query::testutil::TempDir, ChainHub) { rbitcoin_net::chain::tiny_regtest_hub_labeled("chain") } +/// A full `block.invalid` drops its oldest hash for a new failure, so the +/// newest one survives restart. +#[test] +fn a_full_invalid_file_keeps_the_newest_failure() { + let (dir, hub) = tmp_hub(); + drop(hub); + let mut full = b"rbtcinv1".to_vec(); + full.extend_from_slice(&4_096u32.to_le_bytes()); + for i in 0..4_096u32 { + let mut raw = [0u8; 32]; + raw[..4].copy_from_slice(&i.to_le_bytes()); + raw[31] = 0xaa; + full.extend_from_slice(&raw); + } + let marks = dir.path().join("block.invalid"); + std::fs::write(&marks, &full).unwrap(); + let q = rbitcoin_query::Query::open_or_create_tiny(dir.path()).unwrap(); + let hub = ChainHub::new(q, ChainParams::regtest(), Milestone::NONE); + let newest = BlockHash::from_byte_array([0xbb; 32]); + hub.note_failed_connect(newest); + let kept = std::fs::read(&marks).unwrap(); + assert_eq!(kept.len(), full.len(), "the file stays at its cap"); + assert!( + kept.windows(32).any(|w| w == newest.as_byte_array()), + "the newest failure is kept" + ); + assert!( + !kept.windows(32).any(|w| w == &full[12..44]), + "the oldest one gives way" + ); +} + #[test] fn ancestor_walk_stops_at_cap() { let hashes: Vec = (0..9u8) From 8a6eaa2925b59ea925b7d4fd9e60aea98d1d1613 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 16:09:03 -0400 Subject: [PATCH 07/17] net: a header peer that shows a block is kept from it only by one that does Second review pass on #1112 found three ways a peer with no record either way for a block could still exclude the reserved header peer that announced it: an announced window that no longer reaches that height after the candidate changed, a split cleared before a later rewind, and a reply that never recorded where the peer's chain ends. Each let a silent peer take the block while the peer that has it waited. Once the header peer's announced chain shows a block (that block, or a path block above it), only another peer whose announced chain shows it keeps the header peer from it. While the header peer has shown nothing, it still stays on headers when another peer may hold the block. A peer with no record may still be asked, as before this branch. `invalid_tips` now drops its oldest tip at the cap instead of starting over, so a branch refused after the oldest tips is still refused. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 2 +- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 33 ++++++++++++++++---- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 14 +++++++++ crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 29 +++++++++++++++-- 4 files changed, 68 insertions(+), 10 deletions(-) diff --git a/TESTING.md b/TESTING.md index b9a71f732..e83044e2d 100644 --- a/TESTING.md +++ b/TESTING.md @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index ae0603274..3d0f03096 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -83,6 +83,8 @@ pub struct HeaderWalk { /// The last header of each refused batch that built on a rejected block. /// A batch that builds on one is refused too. pub invalid_tips: HashSet, + /// `invalid_tips` in insertion order, so the cap drops the oldest. + pub invalid_tip_order: std::collections::VecDeque, /// Peers that have answered empty at the current tip. pub emptied: HashSet, /// Checkpoint work has reached the milestone floor. @@ -138,7 +140,7 @@ pub const LOOKAHEAD_ASK: Duration = Duration::from_secs(5); pub const HEADER_MISS_DISCONNECT: u8 = 2; #[doc(hidden)] pub const DEAD_END_CAP: usize = 1024; -/// Past the cap the set starts again, so the newest refused tip is kept: the +/// Past the cap the oldest refused tip gives way, so the newest is kept: the /// next batch of a long invalid chain links to it. const INVALID_TIP_CAP: usize = 1024; @@ -642,7 +644,8 @@ pub fn note_reservation( /// Whether block `hash` may be asked of `peer`. A peer is not asked for a /// block it answered `notfound`, or for a path block its announced chain /// does not hold. The reserved header peer is asked only when no other live -/// peer may have it, so a single peer still does both. +/// peer may have it, or, once its own announced chain shows the block, when no +/// other peer's shows it. A single peer still does both. pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { use super::peer_io::PeerSlot; let height = st.hash_height.get(hash).copied(); @@ -662,7 +665,18 @@ pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { if !holds(slot) { return false; } - st.header_walk.reserved != Some(peer) || !st.slots.iter().any(|s| s.id != peer && holds(s)) + if st.header_walk.reserved != Some(peer) { + return true; + } + // A header peer whose announced chain shows the block is kept from it only + // by another peer that shows it too. Otherwise any peer that may hold it + // does: the header peer stays on headers while others fetch blocks. + let shows = + |s: &PeerSlot| askable(s) && height.is_some_and(|ht| s.chain.shows(&st.height_to_hash, ht)); + let shown = shows(slot); + !st.slots + .iter() + .any(|s| s.id != peer && if shown { shows(s) } else { holds(s) }) } /// Settle each peer's recorded split against this node's confirmed chain once @@ -818,10 +832,16 @@ pub fn note_dead(st: &mut IbdWorkState, hash: BlockHash) { #[doc(hidden)] fn note_invalid_tip(st: &mut IbdWorkState, hash: BlockHash) { - if st.header_walk.invalid_tips.len() >= INVALID_TIP_CAP { - st.header_walk.invalid_tips.clear(); + let walk = &mut st.header_walk; + if !walk.invalid_tips.insert(hash) { + return; + } + walk.invalid_tip_order.push_back(hash); + if walk.invalid_tip_order.len() > INVALID_TIP_CAP { + if let Some(oldest) = walk.invalid_tip_order.pop_front() { + walk.invalid_tips.remove(&oldest); + } } - st.header_walk.invalid_tips.insert(hash); } /// Refuse a batch that holds or builds on a rejected block, and retire @@ -3265,6 +3285,7 @@ pub fn parse_adopt(bytes: &[u8]) -> Option { origin: true, dead_ends: HashSet::new(), invalid_tips: HashSet::new(), + invalid_tip_order: std::collections::VecDeque::new(), emptied: HashSet::new(), proven: false, off_path: std::collections::HashMap::new(), diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 465c58451..b25131259 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -134,6 +134,20 @@ impl PeerChain { } } + /// This peer's announced chain holds the path block at `height`: it + /// announced that block, or a path block above it. + pub fn shows(&self, path: &HashMap, height: u32) -> bool { + let Some(len) = (self.announced.len() as u32).checked_sub(1) else { + return false; + }; + if height > self.from.saturating_add(len) { + return false; + } + let at = height.max(self.from); + path.get(&at) + .is_some_and(|p| *p == self.announced[(at - self.from) as usize]) + } + /// The peer does not have the path block at `height`: its chain left this /// node's chain at or below it, it holds a different block there, or its /// whole chain ends below it. Two chains that differ at one height differ diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 3334c68cf..a70831a33 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1618,10 +1618,14 @@ fn two_peers_reserve_the_header_walk() { let (mut head, mut head_rx) = slot(5); let (mut other, mut other_rx) = slot(6); let (mut short, _short_rx) = slot(7); + // Asked once, and answered nothing yet. + let (mut unknown, _unknown_rx) = slot(12); head.first_data_ms = 50; other.first_data_ms = 5_000; short.first_data_ms = 6_000; - let mut st = IbdWorkState::new(vec![head, other, short], Some(gen), Some(0)); + unknown.first_data_ms = 7_000; + unknown.chain.note_probed(); + let mut st = IbdWorkState::new(vec![head, other, short, unknown], Some(gen), Some(0)); assert!(send_getheaders(&mut st, &hub).unwrap()); assert_eq!(drain_getheaders(&mut head_rx), 1); let path = [mine(gen, 41), mine(mine(gen, 41).block_hash(), 42)]; @@ -1671,14 +1675,21 @@ fn two_peers_reserve_the_header_walk() { ); assert!(!st.slots[2].in_flight.contains(&path[1].block_hash())); assert!( - st.slots[0].in_flight.contains(&path[1].block_hash()), - "the header peer takes what no other peer holds" + may_serve(&st, 5, &path[1].block_hash()), + "a peer that announced nothing does not keep the header peer from a block it showed" + ); + assert!( + [0, 3] + .iter() + .any(|&i| st.slots[i].in_flight.contains(&path[1].block_hash())), + "the header peer, or a peer with no record, takes what no other peer holds" ); // With no peer that holds the block by what it announced, a peer that // did not answer `notfound` is still asked. st.slots[0].chain.note_notfound(path[1].block_hash()); st.slots[2].chain.note_notfound(path[1].block_hash()); + st.slots[3].chain.note_notfound(path[1].block_hash()); assert!(may_serve(&st, 6, &path[1].block_hash())); // A batch stored after one of its blocks failed is not planted, and does @@ -1760,6 +1771,7 @@ fn a_long_chain_on_an_invalid_block_stays_refused() { let bad = mine(gen, 1); st.reorg.invalid.mark(bad.block_hash().to_byte_array()); let mut prev = bad.block_hash(); + let mut branch = None; for n in 0..1_100u32 { let next = mine(prev, 2 + n); assert!( @@ -1767,7 +1779,18 @@ fn a_long_chain_on_an_invalid_block_stays_refused() { "batch {n} builds on the refused chain" ); prev = next.block_hash(); + if n == 100 { + // A second branch off the invalid block, refused once. + let side = mine(bad.block_hash(), 5_000); + assert!(refuse_invalid_batch(&mut st, 0, &[side])); + branch = Some(side.block_hash()); + } } + let side_next = mine(branch.unwrap(), 5_001); + assert!( + refuse_invalid_batch(&mut st, 0, &[side_next]), + "a branch refused after the oldest tips is still refused past the cap" + ); } #[test] From 96c81a0738f85bbc4092e49d43a0d765cec6bbd4 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 16:54:34 -0400 Subject: [PATCH 08/17] net: write a failure before the retired walk, and never show a left path Review on #1112: - `shows` could be true where `left_path_by` was also true: a peer that announced the path above the tip and then sent a batch forking at or below the tip kept its old run. If the header peer also showed the block, that stale `shows` kept the header peer off it while the stale peer was not asked either, and no getdata went out. `shows` is now false wherever the peer left the path, and a reply that moves the split without landing above the tip drops the kept run. - The retire saved `header.adopt` before `block.invalid` was written. A crash or a failed write in between left the retired walk on disk with no mark, so the next start chased the invalid chain again. The failure is written first now; `restore_adopt` already refuses a saved candidate above the tip while the file lists a block. If the write fails, the retire still holds in this process and `header.adopt` is removed. Co-Authored-By: Claude Opus 5.5 --- SCHEMA.md | 2 +- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 8 ++- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 9 ++- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 13 +++- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 65 ++++++++++++++++++++ crates/rbitcoin-net/src/chain.rs | 23 ++++--- 6 files changed, 105 insertions(+), 15 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index fcaad9dd2..b7d6a81c2 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -397,7 +397,7 @@ Each checkpoint is the hash, height, and total work at the end of one look-ahead ### `block.invalid` -Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (synced temp file, rename, directory sync) for each new failure and when `reconsiderblock` clears one. +Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (synced temp file, rename, directory sync) for each new failure and when `reconsiderblock` clears one. An IBD failure is written here before the retired walk goes to `header.adopt`; when this write fails, `header.adopt` is removed, so the next start walks again from the confirmed tip. ```text 0..8 magic b"rbtcinv1" diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 7d646badb..331539a19 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -1450,8 +1450,14 @@ pub fn apply_consensus_invalid_reject( st.body.mark_rejected(hash); st.reorg.invalid.mark(hash.to_byte_array()); if let Some(h) = hub { + // The failure goes to disk before the retired walk does: a walk saved + // with its failure lost would resume onto the invalid chain. Without + // the mark on disk, the retired walk is not left as the resume point. + let kept = h.note_failed_connect(hash); super::header_walk::retire_invalid(st, h, hash, height); - h.note_failed_connect(hash); + if !kept { + super::header_walk::forget_adopt(h); + } } if st.height_to_hash.get(&height) == Some(&hash) { st.height_to_hash.remove(&height); diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 3d0f03096..e614475d4 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -2914,6 +2914,11 @@ pub const ADOPT_MAGIC: &[u8; 8] = b"rbtchdr1"; pub const DIFF_SNAP_LEN: usize = 4 + 80 + 4 + 4; #[doc(hidden)] +/// Drop `header.adopt`; the next start walks again from the confirmed tip. +pub fn forget_adopt(hub: &ChainHub) { + let _ = std::fs::remove_file(adopt_path(hub)); +} + pub fn adopt_path(hub: &ChainHub) -> std::path::PathBuf { hub.query.store().path().join("header.adopt") } @@ -2996,7 +3001,7 @@ pub fn base_conflicts(st: &IbdWorkState, hub: &ChainHub, walk: &HeaderWalk) -> b pub fn on_confirmed_rewind(st: &mut IbdWorkState, hub: &ChainHub, lca_h: u32) { if st.header_walk.base_hash.is_some() && lca_h < st.header_walk.base_height { st.header_walk = HeaderWalk::default(); - let _ = std::fs::remove_file(adopt_path(hub)); + forget_adopt(hub); return; } if hub.milestone.height == 0 || !st.header_walk.origin { @@ -3126,7 +3131,7 @@ pub fn retire_adopt_if_confirmed(st: &mut IbdWorkState, hub: &ChainHub) { } hub.query .note_milestone_checkpoint_work(tip_h, work.to_be_bytes()); - let _ = std::fs::remove_file(adopt_path(hub)); + forget_adopt(hub); } /// Read `header.adopt`. A file that does not parse does not restore diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index b25131259..c926ed151 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -75,6 +75,7 @@ impl PeerChain { ) { self.probed = true; let hashes: Vec = headers.iter().map(|h| h.block_hash()).collect(); + let mut split_moved = false; for (ht, hash) in (height..).zip(&hashes) { let on_path = path.get(&ht).copied(); let Some(our) = on_path.or_else(|| (ht <= tip).then(|| confirmed(ht)).flatten()) else { @@ -89,6 +90,7 @@ impl PeerChain { if self.split.is_none_or(|(at, _)| ht <= at) { self.split = Some((ht, *hash)); self.split_confirmed = on_path.is_none(); + split_moved = true; } break; } @@ -97,6 +99,11 @@ impl PeerChain { self.from = self.from.max(tip.saturating_add(1)); let skip = tip.saturating_add(1).saturating_sub(height) as usize; let Some(fresh) = hashes.get(skip..).filter(|h| !h.is_empty()) else { + // The peer's chain now leaves this one at or below the tip; the + // kept run is from the chain it left. + if split_moved { + self.announced.clear(); + } return; }; let start = height.saturating_add(skip as u32); @@ -135,8 +142,12 @@ impl PeerChain { } /// This peer's announced chain holds the path block at `height`: it - /// announced that block, or a path block above it. + /// announced that block, or a path block above it, and has not left the + /// path at or below it. pub fn shows(&self, path: &HashMap, height: u32) -> bool { + if self.left_path_by(path, height) { + return false; + } let Some(len) = (self.announced.len() as u32).checked_sub(1) else { return false; }; diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index a70831a33..f5f6fe1dc 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1793,6 +1793,71 @@ fn a_long_chain_on_an_invalid_block_stays_refused() { ); } +/// A peer that announced the path and then forks below the confirmed tip no +/// longer shows path blocks, so it does not keep the header peer that +/// announced them from them. +#[test] +fn a_peer_that_forks_below_the_tip_does_not_keep_the_header_peer_off() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-fork-below"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let b1 = mine_block(gen, 1); + hub.accept_block(b1.clone()).unwrap(); + let b2 = mine_block(b1.block_hash(), 2); + hub.accept_block(b2.clone()).unwrap(); + let (mut head, _head_rx) = slot(0); + let (mut other, _other_rx) = slot(1); + head.first_data_ms = 50; + other.first_data_ms = 5_000; + let mut st = IbdWorkState::new(vec![head, other], Some(b2.block_hash()), Some(2)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + let h3 = mine(b2.block_hash(), 3); + let path = [h3, mine(h3.block_hash(), 4)]; + apply(&mut st, &hub, 0, path.to_vec()); + apply(&mut st, &hub, 1, path.to_vec()); + assert_eq!(st.header_walk.reserved, Some(0)); + assert!(st.slots[1].chain.shows(&st.height_to_hash, 3)); + apply(&mut st, &hub, 1, vec![mine(b1.block_hash(), 92)]); + assert!( + !st.slots[1].chain.shows(&st.height_to_hash, 3), + "a peer that left the path does not show a block on it" + ); + assert!( + may_serve(&st, 0, &path[0].block_hash()), + "the header peer that announced the block is still asked" + ); +} + +/// When the failure cannot be written, this process still retires the +/// candidate, but the retired walk is not left on disk as the resume point. +#[test] +fn a_retire_whose_failure_is_not_written_leaves_no_resume_point() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-retire-unsaved"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let store = hub.query.store().path().to_path_buf(); + std::fs::create_dir(store.join("block.invalid")).unwrap(); + let (s1, _rx1) = slot(1); + let mut st = IbdWorkState::new(vec![s1], Some(gen), Some(0)); + assert!(send_getheaders(&mut st, &hub).unwrap()); + let p1 = mine(gen, 61); + apply(&mut st, &hub, 1, vec![p1, mine(p1.block_hash(), 62)]); + assert!(st.is_on_path(&p1.block_hash(), 1)); + rbitcoin_ibd::ibd::events::apply_consensus_invalid_reject( + &mut st, + 1, + p1.block_hash(), + "bad block: coinbase excess value", + Some(hub.query.as_ref()), + Some(&hub), + ); + assert_eq!(st.header_walk.tip_height(), 0, "the candidate is retired"); + assert!( + !store.join("header.adopt").exists(), + "a retired walk is not the resume point while its failure is not on disk" + ); +} + #[test] fn idle_header_requests_rotate_and_skip_a_short_peer() { let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-rotate"); diff --git a/crates/rbitcoin-net/src/chain.rs b/crates/rbitcoin-net/src/chain.rs index 90a3dc334..21798eec6 100644 --- a/crates/rbitcoin-net/src/chain.rs +++ b/crates/rbitcoin-net/src/chain.rs @@ -270,7 +270,7 @@ fn parse_invalid_file(bytes: &[u8]) -> Option> { ) } -fn write_invalid_file(path: &std::path::Path, hashes: &[BlockHash]) { +fn write_invalid_file(path: &std::path::Path, hashes: &[BlockHash]) -> bool { let mut buf = Vec::with_capacity(12 + 32 * hashes.len()); buf.extend_from_slice(INVALID_MAGIC); buf.extend_from_slice(&(hashes.len() as u32).to_le_bytes()); @@ -280,9 +280,10 @@ fn write_invalid_file(path: &std::path::Path, hashes: &[BlockHash]) { let wrote = rbitcoin_store::file::write_synced_tmp_file(path, |f| { std::io::Write::write_all(f, &buf).map_err(|e| rbitcoin_store::StoreError::io(path, e)) }); - if let Err(e) = wrote { + if let Err(e) = &wrote { warn!("chain: write {}: {e}", path.display()); } + wrote.is_ok() } /// The hashes one `invalidateblock` marked, in height order. `connected` when @@ -318,16 +319,17 @@ impl Invalidated { } } - fn keep_failed(&self, hash: BlockHash) { + /// True once `hash` is in the file on disk. + fn keep_failed(&self, hash: BlockHash) -> bool { let mut failed = self.failed.write().unwrap(); if failed.contains(&hash) { - return; + return true; } if failed.len() >= INVALID_FILE_CAP { failed.remove(0); } failed.push(hash); - write_invalid_file(&self.file, &failed); + write_invalid_file(&self.file, &failed) } /// Drop kept failures that are no longer marked invalid. @@ -337,7 +339,7 @@ impl Invalidated { let before = failed.len(); failed.retain(|h| set.contains(h)); if failed.len() != before { - write_invalid_file(&self.file, &failed); + let _ = write_invalid_file(&self.file, &failed); } } } @@ -1439,9 +1441,10 @@ impl ChainHub { /// A block failed connect on a consensus rule. Unlike /// [`Self::note_invalid_block`], the mark survives restart; mutated bodies, /// a future timestamp, and store faults are not this. - pub fn note_failed_connect(&self, hash: BlockHash) { + /// True when the mark is on disk. + pub fn note_failed_connect(&self, hash: BlockHash) -> bool { self.note_invalid_block(hash); - self.invalidated.keep_failed(hash); + self.invalidated.keep_failed(hash) } /// Every hash marked invalid, for a header scan that must not count them. @@ -1472,12 +1475,12 @@ impl ChainHub { return; } if e.failing_block_hash().is_some() { - self.note_failed_connect(hash); + let _ = self.note_failed_connect(hash); return; } if let NetError::Consensus(s) = e { if !s.to_ascii_lowercase().contains("not found") { - self.note_failed_connect(hash); + let _ = self.note_failed_connect(hash); } } } From 7e385004d2edbbdf21c646cafb7f2277de27e601 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Tue, 20 Oct 2026 17:00:38 -0400 Subject: [PATCH 09/17] net: save no walk while a failure is not on disk, and let a notfound expire Third review pass on #1112: - When `block.invalid` could not be written, the retire still saved `header.adopt` before removing it, and a later walk save could bring it back. A failure that was not on disk was also reported as kept the second time and never written again. The hub now tracks a pending mark: every later mark (and a reconsider prune) tries the write again, and `save_adopt` saves no walk, removing an existing one, until the mark is on disk. The removal syncs the directory. - A `notfound` kept a peer off a block until it announced again, so a block every peer had answered `notfound` was never asked for again. It now lasts one stall window (30 s). Co-Authored-By: Claude Opus 5.5 --- SCHEMA.md | 2 +- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 11 ++--- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 20 ++++++++- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 24 ++++++++--- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 45 +++++++++++++++++--- crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 14 ++++++ crates/rbitcoin-net/src/chain.rs | 37 ++++++++++++---- 7 files changed, 122 insertions(+), 31 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index b7d6a81c2..5661ab7cb 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -397,7 +397,7 @@ Each checkpoint is the hash, height, and total work at the end of one look-ahead ### `block.invalid` -Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (synced temp file, rename, directory sync) for each new failure and when `reconsiderblock` clears one. An IBD failure is written here before the retired walk goes to `header.adopt`; when this write fails, `header.adopt` is removed, so the next start walks again from the confirmed tip. +Sidecar of blocks that failed connect on a consensus rule, so a restart does not adopt a stored header chain through them again. Not a schema bump: missing is none. A file that does not parse is none and is replaced at the next mark. The whole file is written again (synced temp file, rename, directory sync) for each new failure and when `reconsiderblock` clears one. An IBD failure is written here before the retired walk goes to `header.adopt`. While a failure is not on disk, no walk is saved to `header.adopt` and an existing one is removed, and every later mark tries the write again. A restart while the write still fails has no record of that block and validates it again. ```text 0..8 magic b"rbtcinv1" diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 331539a19..d56fdf086 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -931,7 +931,7 @@ pub fn apply_notfound(st: &mut IbdWorkState, peer: usize, hashes: Vec if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { for h in &hashes { s.track_remove(h); - s.chain.note_notfound(*h); + s.chain.note_notfound(*h, super::peer_io::ibd_mono_ms()); let empty = st .inflight .get_mut(h) @@ -1451,13 +1451,10 @@ pub fn apply_consensus_invalid_reject( st.reorg.invalid.mark(hash.to_byte_array()); if let Some(h) = hub { // The failure goes to disk before the retired walk does: a walk saved - // with its failure lost would resume onto the invalid chain. Without - // the mark on disk, the retired walk is not left as the resume point. - let kept = h.note_failed_connect(hash); + // with its failure lost would resume onto the invalid chain, and + // `save_adopt` saves none while a mark is not on disk. + let _ = h.note_failed_connect(hash); super::header_walk::retire_invalid(st, h, hash, height); - if !kept { - super::header_walk::forget_adopt(h); - } } if st.height_to_hash.get(&height) == Some(&hash) { st.height_to_hash.remove(&height); diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index e614475d4..000c2ccba 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -649,7 +649,8 @@ pub fn note_reservation( pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { use super::peer_io::PeerSlot; let height = st.hash_height.get(hash).copied(); - let askable = |s: &PeerSlot| s.alive && !s.chain.lacks(hash); + let now = super::peer_io::ibd_mono_ms(); + let askable = |s: &PeerSlot| s.alive && !s.chain.lacks(hash, now); let holds = |s: &PeerSlot| { askable(s) && !height.is_some_and(|ht| s.chain.left_path_by(&st.height_to_hash, ht)) }; @@ -2916,7 +2917,16 @@ pub const DIFF_SNAP_LEN: usize = 4 + 80 + 4 + 4; #[doc(hidden)] /// Drop `header.adopt`; the next start walks again from the confirmed tip. pub fn forget_adopt(hub: &ChainHub) { - let _ = std::fs::remove_file(adopt_path(hub)); + let path = adopt_path(hub); + match std::fs::remove_file(&path) { + Ok(()) => { + if let Some(dir) = path.parent() { + let _ = rbitcoin_store::file::fsync_parent_dir(dir); + } + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => rbitcoin_log::warn!("ibd: remove {}: {e}", path.display()), + } } pub fn adopt_path(hub: &ChainHub) -> std::path::PathBuf { @@ -3039,6 +3049,12 @@ pub fn save_adopt(st: &IbdWorkState, hub: &ChainHub) { if !st.header_walk.origin { return; } + // A walk saved while a failure is not on disk could resume onto the + // invalid chain. No walk is the resume point until the mark is. + if !hub.invalid_marks_on_disk() { + forget_adopt(hub); + return; + } let mut buf = Vec::new(); buf.extend_from_slice(ADOPT_MAGIC); let n = st.header_walk.checkpoints.len() as u32; diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index c926ed151..6f04edbbd 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -3,21 +3,25 @@ //! Core asks a peer only for blocks on that peer's best known chain //! (`pindexBestKnownBlock`). A peer whose announced headers leave the //! download path is not asked for blocks above that point, and a peer that -//! answered `notfound` for a block is not asked for it again. +//! answered `notfound` for a block is not asked for it again for a while. use bitcoin::block::Header; use bitcoin::BlockHash; -use std::collections::{HashMap, HashSet, VecDeque}; +use std::collections::{HashMap, VecDeque}; /// `notfound` hashes kept per peer. Past the cap a peer may be asked again. const LACKS_CAP: usize = 1024; +/// How long a `notfound` keeps a peer off that block: one stall window. A +/// peer that answered `notfound` everywhere is asked again after it. +const LACKS_TTL_MS: u64 = 30_000; /// Announced hashes kept per peer: one full `headers` reply. const ANNOUNCED_CAP: usize = rbitcoin_net::codec::MAX_HEADERS_RESULTS; #[derive(Debug, Default)] pub struct PeerChain { - lacks: HashSet, + /// `notfound` hashes, with the mono ms they were answered. + lacks: HashMap, /// This peer's announced chain above the confirmed tip, oldest first. announced: VecDeque, /// Height of `announced[0]`. @@ -36,14 +40,20 @@ pub struct PeerChain { } impl PeerChain { - pub fn note_notfound(&mut self, hash: BlockHash) { + pub fn note_notfound(&mut self, hash: BlockHash, now_ms: u64) { + if self.lacks.len() >= LACKS_CAP { + self.lacks + .retain(|_, at| now_ms.saturating_sub(*at) < LACKS_TTL_MS); + } if self.lacks.len() < LACKS_CAP { - self.lacks.insert(hash); + self.lacks.insert(hash, now_ms); } } - pub fn lacks(&self, hash: &BlockHash) -> bool { - self.lacks.contains(hash) + pub fn lacks(&self, hash: &BlockHash, now_ms: u64) -> bool { + self.lacks + .get(hash) + .is_some_and(|at| now_ms.saturating_sub(*at) < LACKS_TTL_MS) } /// A headers message or a block inv: the peer may have those blocks now. diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index f5f6fe1dc..78e9d0261 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1687,9 +1687,18 @@ fn two_peers_reserve_the_header_walk() { // With no peer that holds the block by what it announced, a peer that // did not answer `notfound` is still asked. - st.slots[0].chain.note_notfound(path[1].block_hash()); - st.slots[2].chain.note_notfound(path[1].block_hash()); - st.slots[3].chain.note_notfound(path[1].block_hash()); + st.slots[0].chain.note_notfound( + path[1].block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + st.slots[2].chain.note_notfound( + path[1].block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + st.slots[3].chain.note_notfound( + path[1].block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); assert!(may_serve(&st, 6, &path[1].block_hash())); // A batch stored after one of its blocks failed is not planted, and does @@ -1754,8 +1763,14 @@ fn two_peers_reserve_the_header_walk() { ); // With no peer that holds a block by what it announced, the header peer // is asked too. - st.slots[1].chain.note_notfound(taken2.block_hash()); - st.slots[2].chain.note_notfound(taken2.block_hash()); + st.slots[1].chain.note_notfound( + taken2.block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); + st.slots[2].chain.note_notfound( + taken2.block_hash(), + rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), + ); assert!(may_serve(&st, 8, &taken2.block_hash())); } @@ -1856,6 +1871,26 @@ fn a_retire_whose_failure_is_not_written_leaves_no_resume_point() { !store.join("header.adopt").exists(), "a retired walk is not the resume point while its failure is not on disk" ); + assert!( + !hub.note_failed_connect(p1.block_hash()), + "a mark whose write failed is still not on disk" + ); + save_adopt(&st, &hub); + assert!( + !store.join("header.adopt").exists(), + "no walk is saved while a failure is not on disk" + ); + std::fs::remove_dir(store.join("block.invalid")).unwrap(); + assert!( + hub.note_failed_connect(p1.block_hash()), + "once the file can be written the pending mark goes to disk" + ); + assert!(std::fs::read(store.join("block.invalid")) + .unwrap() + .windows(32) + .any(|w| w == p1.block_hash().as_byte_array())); + save_adopt(&st, &hub); + assert!(store.join("header.adopt").exists()); } #[test] diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs index 7d9a74ecb..994b003f1 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -76,3 +76,17 @@ fn a_split_is_settled_by_the_confirmed_chain() { "a peer whose chain this node confirmed has not" ); } + +/// A `notfound` keeps a peer off that block for one stall window, then the +/// peer may be asked again. +#[test] +fn a_notfound_lasts_one_stall_window() { + let mut peer = PeerChain::default(); + let hash = BlockHash::from_byte_array([3u8; 32]); + peer.note_notfound(hash, 1_000); + assert!(peer.lacks(&hash, 1_000 + 29_999)); + assert!( + !peer.lacks(&hash, 1_000 + 30_000), + "past the stall window the peer is asked again" + ); +} diff --git a/crates/rbitcoin-net/src/chain.rs b/crates/rbitcoin-net/src/chain.rs index 21798eec6..0409796cb 100644 --- a/crates/rbitcoin-net/src/chain.rs +++ b/crates/rbitcoin-net/src/chain.rs @@ -241,6 +241,8 @@ pub struct Invalidated { /// Operator invalidations are not: their paths are in-process only. pub failed: RwLock>, pub file: std::path::PathBuf, + /// `failed` holds a mark the last write did not put on disk. + pub pending: std::sync::atomic::AtomicBool, } /// Refusal of a hash already marked invalid. Not a fresh validation result. @@ -316,20 +318,26 @@ impl Invalidated { paths: RwLock::new(Vec::new()), failed: RwLock::new(failed), file, + pending: std::sync::atomic::AtomicBool::new(false), } } - /// True once `hash` is in the file on disk. + /// True once `hash`, and every mark before it, is in the file on disk. A + /// write that failed before is tried again. fn keep_failed(&self, hash: BlockHash) -> bool { + use std::sync::atomic::Ordering; let mut failed = self.failed.write().unwrap(); - if failed.contains(&hash) { + if !failed.contains(&hash) { + if failed.len() >= INVALID_FILE_CAP { + failed.remove(0); + } + failed.push(hash); + } else if !self.pending.load(Ordering::Acquire) { return true; } - if failed.len() >= INVALID_FILE_CAP { - failed.remove(0); - } - failed.push(hash); - write_invalid_file(&self.file, &failed) + let wrote = write_invalid_file(&self.file, &failed); + self.pending.store(!wrote, Ordering::Release); + wrote } /// Drop kept failures that are no longer marked invalid. @@ -338,8 +346,11 @@ impl Invalidated { let mut failed = self.failed.write().unwrap(); let before = failed.len(); failed.retain(|h| set.contains(h)); - if failed.len() != before { - let _ = write_invalid_file(&self.file, &failed); + let pending = self.pending.load(std::sync::atomic::Ordering::Acquire); + if failed.len() != before || pending { + let wrote = write_invalid_file(&self.file, &failed); + self.pending + .store(!wrote, std::sync::atomic::Ordering::Release); } } } @@ -1447,6 +1458,14 @@ impl ChainHub { self.invalidated.keep_failed(hash) } + /// Every kept connect failure is in `block.invalid` on disk. + pub fn invalid_marks_on_disk(&self) -> bool { + !self + .invalidated + .pending + .load(std::sync::atomic::Ordering::Acquire) + } + /// Every hash marked invalid, for a header scan that must not count them. pub fn invalid_blocks(&self) -> Vec { self.invalidated From 812f44c64cf059f8fbf08eb1b5a4ea2e31c0c13b Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 11:42:56 -0400 Subject: [PATCH 10/17] net: probe a peer from the confirmed tip, and rewind onto a fork below it Maintainer review: an empty headers reply, or one that ends at or below the confirmed tip, left the peer looking as if it had announced nothing, so it was still asked for path blocks and could be stall-disconnected. - A short reply's top is kept as where the peer's chain ends, also when it is at or below the tip and the kept run is gone. `left_path_by` reads that end with an empty run. - The probe asks from the confirmed chain only. An empty reply then means nothing above the tip, and that peer is not asked for path blocks. A walk ask carries the path top first, so an empty reply to it is a caught-up peer as well as one behind. That peer is probed again rather than marked. - The probe reply goes into the peer's chain record, not the walk. One that shows a header the walk has not seen gives that peer the walk's own ask. A batch that crosses the probe leaves its ask open, and a peer with a walk ask out is not probed. - A probe needs a second live peer. A record only chooses among peers; with one, the probe would take the header lane and its reply would not reach the walk, so the walk would not continue a full fork window. Running the #1082 repro against Core found two more: - After a retire, a heavier chain that forked below the confirmed tip could become the walk's candidate with nothing planted above the tip. IBD neither rewound nor exited. The main-loop rewind now walks back from that candidate. - A rewind replaced the confirmed blocks a peer's record took as its chain, so a peer on the old chain was asked for the new one. A peer whose record starts past the fork splits there at the rewind. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 2 +- changelog.d/ibd-invalid-heavier.md | 13 +- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 62 +++++++- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 89 +++++++++-- crates/rbitcoin-ibd/src/ibd/mod.rs | 2 +- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 54 +++++-- crates/rbitcoin-ibd/src/ibd/reorg.rs | 31 +++- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 156 ++++++++++++++++++- crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 28 ++++ crates/rbitcoin-ibd/tests/ibd/reorg.rs | 49 ++++++ 10 files changed, 448 insertions(+), 38 deletions(-) diff --git a/TESTING.md b/TESTING.md index e83044e2d..16979197d 100644 --- a/TESTING.md +++ b/TESTING.md @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. A sixth, on its own hub, because the probe must say what a peer has above the tip: it asks from the confirmed tip; a peer whose reply is empty is not asked for path blocks and a caught-up peer is; an announcement that crosses the probe leaves its ask open; the reply is not a walk answer, and one that leaves the path gets the walk's own ask; an empty reply to a walk ask is probed again; and a peer with a walk ask out is neither picked nor asked. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md index e3a1d228a..bf226e72c 100644 --- a/changelog.d/ibd-invalid-heavier.md +++ b/changelog.d/ibd-invalid-heavier.md @@ -12,14 +12,17 @@ Fixed served the headers was never asked for blocks while another peer was connected, and a peer that answered `notfound` was asked again for the same block. A heavier chain only the header peer had stalled IBD. A peer - is not asked again for a block it answered `notfound` until it announces - more, or for a block above where its announced headers left the download - path. + is not asked again for a block it answered `notfound` for 30 seconds or + until it announces more, or for a block above where its announced headers + left the download path or ended. - **IBD asks each block peer for headers once.** A peer that had announced nothing was asked for blocks of any chain, so a Core peer serving a lighter chain stayed silent on blocks it did not have and was dropped for - a 10 minute cooldown. Each block peer now gets one headers ask, so its - chain is known before blocks are asked of it. + a 10 minute cooldown. Each block peer now gets one headers ask from the + confirmed tip, so its chain is known before blocks are asked of it. A + peer with nothing above the tip is not asked for blocks. An empty reply + to a header-sync ask does not say whether the peer is caught up or + behind, so that peer is asked again from the tip. - **A block that failed connect stays invalid after restart.** The mark was in memory only, so a restart adopted the stored header chain through that block again and rewound a valid tip to chase it. Connect failures are diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index d56fdf086..3ccb514b9 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -646,10 +646,18 @@ pub fn plan_header_batch( challenger_work_before: st.header_walk.challenger.as_ref().map(|ch| ch.tip.work), outer_len: headers.len(), }; + if answer_probe(st, hub, peer, &headers) { + return (mark, HeaderPlan::Done); + } if mark.outer_len > 0 { note_announcement(st, peer); note_peer_headers(st, hub, peer, &headers); } else { + if st.header_walk.reserved != Some(peer) { + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.forget_probe(); + } + } super::header_walk::settle_walk_peer( st, hub, @@ -674,7 +682,12 @@ pub fn plan_header_reply( peer: usize, headers: Vec, ) -> HeaderPlan { - let ask = super::header_walk::take_header_ask(st, peer); + // A batch that crossed this peer's open probe leaves that ask in flight. + let ask = if st.header_walk.probe_open == Some(peer) { + super::header_walk::HeaderAsk::Unsolicited + } else { + super::header_walk::take_header_ask(st, peer) + }; let mut inflight = super::header_walk::InFlight::capture(st); if headers.is_empty() { inflight.hold(st); @@ -919,6 +932,53 @@ fn note_peer_headers( } } +/// The reply to this peer's open probe, read into its chain record and kept +/// from the walk. The probe asked from the confirmed chain, so its reply is +/// empty or builds on a confirmed block; a batch that does not is an +/// announcement that crossed the probe. True when the reply was consumed. +fn answer_probe( + st: &mut IbdWorkState, + hub: &ChainHub, + peer: usize, + headers: &[bitcoin::block::Header], +) -> bool { + if st.header_walk.probe_open != Some(peer) { + return false; + } + let tip = hub.tip_height().unwrap_or(0); + if let Some(first) = headers.first() { + let prev = first.prev_blockhash; + let confirmed = hub + .query + .height_of_hash(&prev.to_byte_array()) + .ok() + .flatten() + .is_some_and(|h| h.0 <= tip && super::peer_chain::confirmed_at(hub, h.0) == Some(prev)); + if !confirmed { + return false; + } + } + st.header_walk.probe_open = None; + let _ = super::header_walk::take_header_ask(st, peer); + if headers.is_empty() { + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + s.chain.note_ends_at(tip); + } + } else { + note_announcement(st, peer); + note_peer_headers(st, hub, peer, headers); + // A header this node has not seen: the peer's chain leaves this one + // or runs past the path. The walk asks it, as after a block inv. + let last = headers[headers.len() - 1].block_hash(); + let seen = st.hash_height.contains_key(&last) + || matches!(hub.query.height_of_hash(&last.to_byte_array()), Ok(Some(_))); + if !seen { + super::header_walk::note_block_inv(st, peer); + } + } + true +} + fn note_announcement(st: &mut IbdWorkState, peer: usize) { if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { s.chain.note_announcement(); diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 000c2ccba..73c52c29e 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -107,6 +107,12 @@ pub struct HeaderWalk { pub challenge_external: bool, /// Challenge ask already sent. Judged when that peer replies or times out. pub challenge_open: Option, + /// Block peer owed one chain probe: a headers ask from the confirmed tip. + /// Not a walk ask. Not written to `header.adopt`. + pub probe: Option, + /// Probe ask already sent. Its reply is read into that peer's chain + /// record, not by the walk. + pub probe_open: Option, /// The one in-flight headers ask. Not written to `header.adopt`. pub walk: Lane, /// Expired asks. Not written to `header.adopt`. @@ -550,23 +556,39 @@ pub fn note_block_inv(st: &mut IbdWorkState, peer: usize) { /// Ask one block peer that has not announced a chain for headers, so block /// getdata can follow what it holds. Core starts header sync with each -/// peer. This is the block-inv challenge: one peer at a time, once per -/// peer, never the reserved header peer. -pub fn challenge_unprobed_peer(st: &mut IbdWorkState) { - if st.header_walk.challenge.is_some() || st.header_walk.challenge_open.is_some() { +/// peer. One probe at a time, never the reserved header peer, and not a +/// peer with a walk ask open. The probe asks from the confirmed tip, so an +/// empty reply means nothing above it. +pub fn probe_unprobed_peer(st: &mut IbdWorkState) { + let walk = &st.header_walk; + // A record only chooses among peers. With one live peer it would only + // take the header lane, and its reply would not reach the walk. + if walk.probe.is_some() + || walk.probe_open.is_some() + || st.slots.iter().filter(|s| s.alive).count() < 2 + { return; } - let reserved = st.header_walk.reserved; let Some(slot) = st .slots - .iter_mut() - .find(|s| s.alive && Some(s.id) != reserved && !s.chain.probed()) + .iter() + .find(|s| s.alive && !walk_busy(walk, s.id) && !s.chain.probed()) else { return; }; - slot.chain.note_probed(); - st.header_walk.challenge = Some(slot.id); - st.header_walk.challenge_external = false; + st.header_walk.probe = Some(slot.id); +} + +/// The header peer, or a peer with a walk ask open, whose late reply could +/// be read as a probe's. +fn walk_busy(walk: &HeaderWalk, peer: usize) -> bool { + [ + walk.reserved, + walk.challenge_open, + walk.walk.peer, + walk.walk.prev_peer, + ] + .contains(&Some(peer)) } /// A follow-session block inv has no IBD slot. Poke one live header peer @@ -604,6 +626,12 @@ pub fn forget_walk_peer(st: &mut IbdWorkState, peer: usize) { st.header_walk.challenge_open = None; st.header_walk.challenge_external = false; } + if st.header_walk.probe == Some(peer) { + st.header_walk.probe = None; + } + if st.header_walk.probe_open == Some(peer) { + st.header_walk.probe_open = None; + } } /// Chainwork of the candidate, for judging a challenge reply. @@ -709,12 +737,14 @@ pub enum HeaderPoll { Wait, } -/// A block inv is waiting, and the walk lane is not already in flight. +/// A block inv or a probe is waiting, and the walk lane is not already in +/// flight. /// /// An in-flight ask keeps the challenge until the lane clears. The poll /// must not call `send_getheaders` on every block event during that window. pub fn inv_challenge_due(st: &IbdWorkState) -> bool { - st.header_walk.challenge.is_some() && !lane_fresh(&st.header_walk.walk) + (st.header_walk.challenge.is_some() || st.header_walk.probe.is_some()) + && !lane_fresh(&st.header_walk.walk) } /// The queue still has room, or a gap / look-ahead still needs headers. @@ -2405,6 +2435,14 @@ pub fn expire_header_ask(st: &mut IbdWorkState) -> Option { *e = e.saturating_add(1); *e }; + if st.header_walk.probe_open == Some(peer) { + st.header_walk.probe_open = None; + if n >= HEADER_MISS_DISCONNECT { + st.header_walk.header_misses.remove(&peer); + punish_header_peer(st, peer); + } + return None; + } if st.header_walk.challenge_open == Some(peer) { st.header_walk.challenge_open = None; st.header_walk.challenge_external = false; @@ -2455,12 +2493,12 @@ pub fn send_getheaders( ) -> Result { ensure_origin(st, hub); super::path::append_stored_headers(st, hub); - if st.header_walk.challenge.is_none() && super::path::walk_successor_indexed(st, hub) { + let owed = st.header_walk.challenge.is_some() || st.header_walk.probe.is_some(); + if !owed && super::path::walk_successor_indexed(st, hub) { release_lane(&mut st.header_walk.walk); return Ok(false); } - let want = - st.header_walk.challenge.is_some() || wants_lookahead(st) || walk_ahead_of_queue(st, hub); + let want = owed || wants_lookahead(st) || walk_ahead_of_queue(st, hub); if !want { release_lane(&mut st.header_walk.walk); return Ok(false); @@ -2517,6 +2555,17 @@ pub fn arm_header_ask( } st.header_walk.challenge_external = false; } + if let Some(peer) = take_probe(st) { + // Only the confirmed chain: the first of its hashes the peer has is + // at or below the tip, so the reply is what it has above it. + if ask_header_peer(st, hub, peer, &[])? { + st.header_walk.probe_open = Some(peer); + if let Some(slot) = st.slots.iter_mut().find(|s| s.id == peer) { + slot.chain.note_probed(); + } + return Ok(true); + } + } ensure_reserved(st, above, skip); if st.header_walk.reserved.is_none() { // Connect-time start height can sit on the stored tip, so no peer @@ -2618,6 +2667,14 @@ pub fn ensure_reserved(st: &mut IbdWorkState, above: u32, skip: Option) { st.header_walk.reserved = pick_header_peer(st, above, skip); } +/// A probe peer who is still connected and not [`walk_busy`]. A busy peer +/// is dropped unprobed, so it is picked again later. +fn take_probe(st: &mut IbdWorkState) -> Option { + let peer = st.header_walk.probe.take()?; + let alive = st.slots.iter().any(|s| s.alive && s.id == peer); + (alive && !walk_busy(&st.header_walk, peer)).then_some(peer) +} + /// A challenge peer who is still connected and not retired. #[doc(hidden)] pub fn take_challenge(st: &mut IbdWorkState) -> Option { @@ -3317,6 +3374,8 @@ pub fn parse_adopt(bytes: &[u8]) -> Option { challenge: None, challenge_external: false, challenge_open: None, + probe: None, + probe_open: None, walk: Lane::default(), header_misses: HashMap::new(), base_diff, diff --git a/crates/rbitcoin-ibd/src/ibd/mod.rs b/crates/rbitcoin-ibd/src/ibd/mod.rs index 0eaf2590c..950d1c689 100644 --- a/crates/rbitcoin-ibd/src/ibd/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/mod.rs @@ -703,7 +703,7 @@ pub async fn ibd_cancellable( if hub.header_inv_challenge_pending() && header_walk::note_external_block_inv(st) { hub.clear_header_inv_challenge(); } - header_walk::challenge_unprobed_peer(st); + header_walk::probe_unprobed_peer(st); let challenge_due = header_walk::inv_challenge_due(st); if cadence.headers_due(now_cadence, path_empty, challenge_due) { let live = st.ordered_set.len(); diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 6f04edbbd..16ff064c8 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -26,9 +26,10 @@ pub struct PeerChain { announced: VecDeque, /// Height of `announced[0]`. from: u32, - /// The last reply was shorter than a full `headers` batch: the peer had - /// nothing past its top. - complete: bool, + /// Top of this peer's chain: the last reply was shorter than a full + /// `headers` batch and ended here, or a probe found nothing above the + /// confirmed tip. The peer has no block above it. + ends: Option, /// The peer's header at the first height where its announced chain /// differed from this node's chain (path or confirmed) when announced. split: Option<(u32, BlockHash)>, @@ -69,12 +70,29 @@ impl PeerChain { self.probed = true; } + /// An empty reply to a walk ask: the peer is at one of several locator + /// hashes, the path top or the confirmed tip. Ask it once more from the + /// tip. + pub fn forget_probe(&mut self) { + self.probed = false; + } + + /// An empty reply to a probe from the confirmed `tip`: this peer has + /// nothing above it. + pub fn note_ends_at(&mut self, tip: u32) { + self.probed = true; + self.announced.clear(); + self.from = tip.saturating_add(1); + self.ends = Some(tip); + } + /// Keep announced `headers`, the first at `height`, against this node's /// chain: the `path`, or a `confirmed` block at or below `tip`. The first /// height where they differ is where the peer leaves this chain; a later /// reply that agrees there clears it. Only heights above `tip` are kept, /// newest last: a reply that continues the kept run extends it, any other - /// replaces it, and the oldest hashes past one full reply give way. + /// replaces it, and the oldest hashes past one full reply give way. A + /// short reply is where the peer's chain ends, also at or below the tip. pub fn note_headers( &mut self, path: &HashMap, @@ -104,14 +122,17 @@ impl PeerChain { } break; } + let short = headers.len() < ANNOUNCED_CAP; + self.ends = short.then(|| height.saturating_add(hashes.len() as u32).saturating_sub(1)); let drop = tip.saturating_add(1).saturating_sub(self.from) as usize; self.announced.drain(..drop.min(self.announced.len())); self.from = self.from.max(tip.saturating_add(1)); let skip = tip.saturating_add(1).saturating_sub(height) as usize; let Some(fresh) = hashes.get(skip..).filter(|h| !h.is_empty()) else { - // The peer's chain now leaves this one at or below the tip; the - // kept run is from the chain it left. - if split_moved { + // Nothing above the tip. A short reply ended the peer's chain at + // or below it; a split that moved leaves the kept run on the + // chain the peer left. + if split_moved || short { self.announced.clear(); } return; @@ -128,7 +149,18 @@ impl PeerChain { let over = self.announced.len().saturating_sub(ANNOUNCED_CAP); self.announced.drain(..over); self.from = self.from.saturating_add(over as u32); - self.complete = headers.len() < ANNOUNCED_CAP; + } + + /// This node's confirmed chain above `lca` was rewound away, and `left` + /// was its block at `lca + 1`. A record that starts past that height took + /// those blocks as the peer's chain, so the peer split from the new one + /// there. A lower split already says where it left. + pub fn note_rewind(&mut self, lca: u32, left: BlockHash) { + let at = lca.saturating_add(1); + if self.from > at && self.split.is_none_or(|(h, _)| h > at) { + self.split = Some((at, left)); + self.split_confirmed = false; + } } /// The recorded split's height has left the path: compare it with this @@ -183,6 +215,9 @@ impl PeerChain { return true; } } + if self.ends.is_some_and(|top| height > top) { + return true; + } let Some(len) = (self.announced.len() as u32).checked_sub(1) else { return false; }; @@ -190,9 +225,6 @@ impl PeerChain { return false; } let top = self.from.saturating_add(len); - if height > top && self.complete { - return true; - } let at = height.min(top); let peer = self.announced[(at - self.from) as usize]; path.get(&at).is_some_and(|p| *p != peer) diff --git a/crates/rbitcoin-ibd/src/ibd/reorg.rs b/crates/rbitcoin-ibd/src/ibd/reorg.rs index 5eaaa2361..902f7e0f1 100644 --- a/crates/rbitcoin-ibd/src/ibd/reorg.rs +++ b/crates/rbitcoin-ibd/src/ibd/reorg.rs @@ -363,7 +363,10 @@ pub fn consider_disconnected_heavier( st: &mut super::state::IbdWorkState, hub: &ChainHub, ) -> Result { - let cands = connecting_search_candidates(st, hub)?; + let mut cands = connecting_search_candidates(st, hub)?; + if cands.is_empty() { + cands.extend(unplanted_walk_tip(st, hub)); + } let invalid: HashSet<[u8; 32]> = st.reorg.invalid.iter().collect(); let Some(path) = select_heavier_path(hub, &cands, &invalid)? else { return Ok(false); @@ -371,6 +374,21 @@ pub fn consider_disconnected_heavier( apply_header_rewind(st, hub, &path) } +/// The walk's candidate while nothing is planted above the confirmed tip. A +/// retire walks back to the parent, and a heavier chain that forked under +/// the tip then takes no path slots, so tip+1 names no fork. The main loop +/// rewinds to it; the header thread's rewind does not see it. A candidate +/// that is not stored has no parent, so no rewind is prepared for it. +fn unplanted_walk_tip(st: &super::state::IbdWorkState, hub: &ChainHub) -> Option { + let tip_h = hub.tip_height()?; + let planted = + !st.ordered.is_empty() || st.height_to_hash.contains_key(&tip_h.saturating_add(1)); + if planted || st.header_walk.tip_height() <= tip_h { + return None; + } + st.header_walk.tip_hash() +} + /// Header tips on known competing forks that do not contain an invalid hash. /// /// After a consensus-invalid reject the planted path may be a dead-end while a @@ -529,6 +547,8 @@ pub struct RewindPrep { pub archived: Vec, pub connected: Vec, pub fks: Vec>, + /// The confirmed block above the LCA before the disconnect. + pub left: Option, } /// Store side of a rewind: parent lookup and the confirmed-tip disconnect. @@ -553,6 +573,9 @@ pub fn prepare_header_rewind( return Ok(None); }; let tip_h = hub.tip_height().unwrap_or(0); + let left = (tip_h > lca_h) + .then(|| super::peer_chain::confirmed_at(hub, lca_h.saturating_add(1))) + .flatten(); if tip_h.saturating_sub(lca_h) > REWIND_MAX_DEPTH { warn!( "ibd: heavier fork rewind depth {} exceeds cap {REWIND_MAX_DEPTH} (lca={lca_h} tip={tip_h})", @@ -599,6 +622,7 @@ pub fn prepare_header_rewind( archived, connected, fks, + left, })) } @@ -610,6 +634,11 @@ pub fn commit_header_rewind( prep: &RewindPrep, ) { st.clear_path_above(prep.lca_h); + if let Some(left) = prep.left { + for slot in &mut st.slots { + slot.chain.note_rewind(prep.lca_h, left); + } + } hub.query.clear_milestone_path_above(prep.lca_h); super::header_walk::on_confirmed_rewind(st, hub, prep.lca_h); hub.query.set_lookup_taken_hi(Some(prep.lca_h)); diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 78e9d0261..2fd3ae341 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1632,7 +1632,7 @@ fn two_peers_reserve_the_header_walk() { apply(&mut st, &hub, 5, path.to_vec()); assert_eq!(st.header_walk.reserved, Some(5)); let _ = take_header_ask(&mut st, 5); - challenge_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st); assert!(send_getheaders(&mut st, &hub).unwrap()); assert_eq!( drain_getheaders(&mut other_rx), @@ -1644,8 +1644,8 @@ fn two_peers_reserve_the_header_walk() { apply(&mut st, &hub, 6, vec![fork, mine(fork.block_hash(), 46)]); // A short reply is everything that peer has: its chain ends at height 1. apply(&mut st, &hub, 7, vec![path[0]]); - challenge_unprobed_peer(&mut st); - assert!(st.header_walk.challenge.is_none(), "a peer is probed once"); + probe_unprobed_peer(&mut st); + assert!(st.header_walk.probe.is_none(), "a peer is probed once"); for h in &path { st.body.mark_missing(h.block_hash()); } @@ -1772,6 +1772,146 @@ fn two_peers_reserve_the_header_walk() { rbitcoin_ibd::ibd::peer_io::ibd_mono_ms(), ); assert!(may_serve(&st, 8, &taken2.block_hash())); + + // A sixth, on its own hub, because the probe must say what a peer has + // above the tip: it asks from the confirmed tip, so an empty reply is a + // chain that ends there and that peer is not asked for path blocks. A + // caught-up peer sends the path and is. The probe is not a walk answer, + // and a reply with a chain the walk has not seen gets the walk's own ask. + // An empty reply to a walk ask is a caught-up peer as well as one behind, + // so that peer is probed again. + let (_dir6, hub6) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-probe"); + hub6.ensure_genesis().unwrap(); + let gen6 = hub6.tip_hash().unwrap(); + let (mut head, mut head_rx) = slot(13); + let (mut caught, mut caught_rx) = slot(14); + let (mut behind, mut behind_rx) = slot(15); + let (mut side, mut side_rx) = slot(16); + head.first_data_ms = 50; + caught.first_data_ms = 5_000; + behind.first_data_ms = 6_000; + side.first_data_ms = 7_000; + let mut st = IbdWorkState::new(vec![head, caught, behind, side], Some(gen6), Some(0)); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(drain_getheaders(&mut head_rx), 1); + let path = [mine(gen6, 51), mine(mine(gen6, 51).block_hash(), 52)]; + apply(&mut st, &hub6, 13, path.to_vec()); + assert_eq!(st.header_walk.reserved, Some(13)); + let _ = take_header_ask(&mut st, 13); + + probe_unprobed_peer(&mut st); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + let locator = getheaders_locator(&mut caught_rx).expect("the caught-up peer is probed"); + assert_eq!( + locator.first(), + Some(&gen6), + "the probe asks from the confirmed tip" + ); + assert!(!locator.contains(&path[1].block_hash())); + // An announcement that crosses the probe is not its reply. + let next = mine(path[1].block_hash(), 53); + apply(&mut st, &hub6, 14, vec![next]); + assert_eq!( + st.header_walk.walk.peer, + Some(14), + "the probe ask stays open, so it can still expire" + ); + apply(&mut st, &hub6, 14, vec![path[0], path[1], next]); + assert_eq!(st.header_walk.probe_open, None); + assert!( + !st.header_walk.walk_quiet.contains(&14), + "a probe reply is not a walk answer" + ); + assert_eq!(st.header_walk.reserved, Some(13)); + + probe_unprobed_peer(&mut st); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut behind_rx).is_some()); + apply(&mut st, &hub6, 15, Vec::new()); + assert!( + !may_serve(&st, 15, &path[0].block_hash()), + "a peer with nothing above the tip is not asked for a path block" + ); + assert!(may_serve(&st, 14, &path[1].block_hash())); + + // A probe reply that leaves the path is a chain the walk has not seen, + // so that peer gets the walk's own ask. + probe_unprobed_peer(&mut st); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut side_rx).is_some()); + apply(&mut st, &hub6, 16, vec![mine(gen6, 54)]); + assert_eq!(st.header_walk.challenge, Some(16)); + + // A block inv's ask carries the path top, so an empty reply is a peer at + // that top as well as one behind it. + note_block_inv(&mut st, 14); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut caught_rx).is_some()); + apply(&mut st, &hub6, 14, Vec::new()); + assert!( + may_serve(&st, 14, &path[1].block_hash()), + "an empty walk reply does not take a caught-up peer off the path" + ); + assert!( + !st.slots[1].chain.probed(), + "an empty walk reply is followed by a probe" + ); + + // A late reply to a walk ask could be read as the probe's, so a peer + // with one out is neither picked nor asked. + st.header_walk.walk.peer = None; + st.header_walk.walk.at = None; + st.header_walk.walk.prev_peer = Some(14); + probe_unprobed_peer(&mut st); + assert_ne!(st.header_walk.probe, Some(14)); + st.header_walk.probe = Some(14); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(st.header_walk.probe, None); + assert_ne!(st.header_walk.probe_open, Some(14)); + assert!(!st.slots[1].chain.probed(), "it is picked again later"); + + // A seventh, on its own hub, because a rewind replaces the confirmed + // blocks a peer's record took as its chain: that peer is not asked for + // the new chain's blocks, and one that had shown the new chain is. + let (_dir7, hub7) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-rewound"); + hub7.ensure_genesis().unwrap(); + let gen7 = hub7.tip_hash().unwrap(); + let l1 = mine_block(gen7, 1); + hub7.accept_block(l1.clone()).unwrap(); + let l2 = mine_block(l1.block_hash(), 2); + hub7.accept_block(l2.clone()).unwrap(); + let w1 = mine(gen7, 71); + let w2 = mine(w1.block_hash(), 72); + let w3 = mine(w2.block_hash(), 73); + for w in [&w1, &w2, &w3] { + hub7.ensure_header(w).unwrap(); + } + let (old, _old_rx) = slot(17); + let (fresh, _fresh_rx) = slot(18); + let (taken, _taken_rx) = slot(19); + let mut st = IbdWorkState::new(vec![old, fresh, taken], hub7.tip_hash(), hub7.tip_height()); + let confirmed = |h| rbitcoin_ibd::ibd::peer_chain::confirmed_at(&hub7, h); + let l3 = mine(l2.block_hash(), 74); + st.slots[0].chain.note_headers( + &st.height_to_hash, + confirmed, + 2, + 1, + &[l1.header, l2.header, l3], + ); + st.slots[2] + .chain + .note_headers(&st.height_to_hash, confirmed, 2, 1, &[w1, w2]); + st.header_walk.tip.hash = Some(w3.block_hash()); + st.header_walk.tip.height = 3; + assert!(rbitcoin_ibd::ibd::reorg::consider_disconnected_heavier(&mut st, &hub7).unwrap()); + assert_eq!(st.height_to_hash.get(&1), Some(&w1.block_hash())); + assert!( + !may_serve(&st, 17, &w1.block_hash()), + "a peer on the rewound chain is not asked for the new one" + ); + assert!(may_serve(&st, 18, &w1.block_hash())); + assert!(may_serve(&st, 19, &w2.block_hash())); } /// A long chain on an invalid block stays refused past the cap on @@ -2306,6 +2446,16 @@ fn a_heavier_header_chain_keeps_its_context() { } } +fn getheaders_locator(rx: &mut mpsc::UnboundedReceiver) -> Option> { + loop { + match rx.try_recv() { + Ok(PeerCmd::GetHeaders { locator }) => return Some(locator), + Ok(_) => {} + Err(_) => return None, + } + } +} + fn drain_getheaders(rx: &mut mpsc::UnboundedReceiver) -> usize { let mut n = 0usize; loop { diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs index 994b003f1..8b67884eb 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -77,6 +77,34 @@ fn a_split_is_settled_by_the_confirmed_chain() { ); } +/// A short reply that ends at or below the confirmed tip is all that peer +/// has, so it is not asked above the tip. That holds for a first reply and +/// for one that follows a short run this node has since confirmed. +#[test] +fn a_reply_at_or_below_the_tip_ends_the_chain_there() { + let headers = chain(20, 0x5c); + let above = |tip: u32| -> HashMap { + (tip + 1..=20) + .map(|h| (h, headers[h as usize - 1].block_hash())) + .collect() + }; + let confirmed = |h: u32| Some(headers[h as usize - 1].block_hash()); + let mut behind = PeerChain::default(); + behind.note_headers(&above(10), confirmed, 10, 5, &headers[4..8]); + assert!( + behind.left_path_by(&above(10), 11), + "a first reply that ends below the tip" + ); + let mut caught = PeerChain::default(); + caught.note_headers(&above(10), confirmed, 10, 11, &headers[10..15]); + assert!(caught.left_path_by(&above(10), 16)); + caught.note_headers(&above(15), confirmed, 15, 13, &headers[12..15]); + assert!( + caught.left_path_by(&above(15), 16), + "a reply that ends at the tip after the kept run was confirmed" + ); +} + /// A `notfound` keeps a peer off that block for one stall window, then the /// peer may be asked again. #[test] diff --git a/crates/rbitcoin-ibd/tests/ibd/reorg.rs b/crates/rbitcoin-ibd/tests/ibd/reorg.rs index 33262d069..02d80f1d7 100644 --- a/crates/rbitcoin-ibd/tests/ibd/reorg.rs +++ b/crates/rbitcoin-ibd/tests/ibd/reorg.rs @@ -211,6 +211,55 @@ fn consider_disconnected_from_work_path_without_resume_seed() { let _ = std::fs::remove_dir_all(dir); } +/// The walk's candidate forks below the confirmed tip, as after a retire +/// walks back and a heavier chain arrives that forked under it. None of it +/// is planted, so the path's next height names no fork. Its stored tip is +/// where the rewind walks back from. A walk tip that is not stored is not, +/// and once a path is planted the walk's tip is not read from the store. +#[test] +fn a_walk_candidate_forked_below_the_tip_is_rewound_to() { + let (dir, hub) = tmp_hub(); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let l1 = mine(gen, 1_500_063_100, 1); + hub.accept_block(l1.clone()).unwrap(); + let l2 = mine(l1.block_hash(), 1_500_063_200, 2); + hub.accept_block(l2.clone()).unwrap(); + let w1 = distinct_sib(mine(gen, 1_500_063_101, 1), l1.block_hash()); + let w2 = mine(w1.block_hash(), 1_500_063_201, 2); + let w3 = mine(w2.block_hash(), 1_500_063_301, 3); + for w in [&w1, &w2, &w3] { + hub.ensure_header(&w.header).unwrap(); + } + let w4 = mine(w3.block_hash(), 1_500_063_401, 4); + + let mut st = + rbitcoin_ibd::ibd::state::IbdWorkState::new(Vec::new(), hub.tip_hash(), hub.tip_height()); + st.header_walk.tip.hash = Some(w4.block_hash()); + st.header_walk.tip.height = 4; + assert!(!consider_disconnected_heavier(&mut st, &hub).unwrap()); + assert_eq!( + hub.tip_height(), + Some(2), + "an unstored walk tip is not a rewind" + ); + + st.header_walk.tip.hash = Some(w3.block_hash()); + st.header_walk.tip.height = 3; + assert!(consider_disconnected_heavier(&mut st, &hub).unwrap()); + assert_eq!(hub.tip_height(), Some(0), "rewound to the fork"); + assert_eq!(st.height_to_hash.get(&1), Some(&w1.block_hash())); + assert_eq!(st.height_to_hash.get(&3), Some(&w3.block_hash())); + reset_parent_hash_of_count(); + assert!(!consider_disconnected_heavier(&mut st, &hub).unwrap()); + assert_eq!( + parent_hash_of_count(), + 0, + "a planted path is not a store walk" + ); + let _ = std::fs::remove_dir_all(dir); +} + /// A real block whose prev is not a stored header fails the live confirm /// load as BadPrev. That reject is soft reget: the tip stays, the slot is /// evicted, and the hash is not blacklisted or fetched again. From 490c4468fad71d132278e6e32964ddb94071cbdb Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 11:57:51 -0400 Subject: [PATCH 11/17] net: keep an open probe's lane, and keep a rewound peer on the chain it showed A second review of the probe and rewind change found three gaps, each pinned before the fix: - With nothing left for the walk to ask, the lane was released under an open probe, so it never expired and no other peer was probed. An open probe now keeps the lane, and the header poll asks once its window passes, also with header sync otherwise done, so it can expire. - A peer with an expired walk ask that the lane no longer tracked could be probed, and its late walk reply read as the probe's. A peer with an unanswered expired ask is not probed. - At a rewind, a peer whose record started past the fork was split onto the old chain even when its announced headers were on the new one. The split is now inferred after the new chain is planted, and not for a peer that announced a header on it. A split the peer itself showed at or below the fork is kept, so a peer on a third chain stays off both chains across a rewind and back. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 2 +- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 21 ++++--- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 28 ++++++--- crates/rbitcoin-ibd/src/ibd/reorg.rs | 10 +-- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 64 +++++++++++++++++++- crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 23 +++++++ 6 files changed, 125 insertions(+), 23 deletions(-) diff --git a/TESTING.md b/TESTING.md index 16979197d..059afaa14 100644 --- a/TESTING.md +++ b/TESTING.md @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. A sixth, on its own hub, because the probe must say what a peer has above the tip: it asks from the confirmed tip; a peer whose reply is empty is not asked for path blocks and a caught-up peer is; an announcement that crosses the probe leaves its ask open; the reply is not a walk answer, and one that leaves the path gets the walk's own ask; an empty reply to a walk ask is probed again; and a peer with a walk ask out is neither picked nor asked. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. A sixth, on its own hub, because the probe must say what a peer has above the tip: it asks from the confirmed tip; a peer whose reply is empty is not asked for path blocks and a caught-up peer is; an announcement that crosses the probe leaves its ask open; the reply is not a walk answer, and one that leaves the path gets the walk's own ask; an empty reply to a walk ask is probed again; and a peer with a walk ask out, or one expired and not answered, is neither picked nor asked; an open probe keeps the lane while the walk has nothing to ask and expires. A seventh, on its own hub, because a rewind replaces the confirmed blocks a peer's record took as its chain: the main-loop rewind walks back from a walk candidate with nothing planted above the tip; a peer on the old chain is not asked for the new one, and a peer whose split or announced headers are on the new chain is. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 73c52c29e..b1ecd5ff4 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -579,8 +579,8 @@ pub fn probe_unprobed_peer(st: &mut IbdWorkState) { st.header_walk.probe = Some(slot.id); } -/// The header peer, or a peer with a walk ask open, whose late reply could -/// be read as a probe's. +/// The header peer, or a peer with a walk ask open or expired and not yet +/// answered, whose late reply could be read as a probe's. fn walk_busy(walk: &HeaderWalk, peer: usize) -> bool { [ walk.reserved, @@ -589,6 +589,7 @@ fn walk_busy(walk: &HeaderWalk, peer: usize) -> bool { walk.walk.prev_peer, ] .contains(&Some(peer)) + || walk.header_misses.contains_key(&peer) } /// A follow-session block inv has no IBD slot. Poke one live header peer @@ -737,14 +738,20 @@ pub enum HeaderPoll { Wait, } -/// A block inv or a probe is waiting, and the walk lane is not already in -/// flight. +/// A block inv or a probe is waiting, or an open probe is past its window, +/// and the walk lane is not already in flight. /// /// An in-flight ask keeps the challenge until the lane clears. The poll /// must not call `send_getheaders` on every block event during that window. pub fn inv_challenge_due(st: &IbdWorkState) -> bool { - (st.header_walk.challenge.is_some() || st.header_walk.probe.is_some()) - && !lane_fresh(&st.header_walk.walk) + probe_or_challenge_owed(st) && !lane_fresh(&st.header_walk.walk) +} + +/// A challenge or probe waits to be sent, or a probe waits to be answered +/// or expire. Either keeps the lane. +fn probe_or_challenge_owed(st: &IbdWorkState) -> bool { + let walk = &st.header_walk; + walk.challenge.is_some() || walk.probe.is_some() || walk.probe_open.is_some() } /// The queue still has room, or a gap / look-ahead still needs headers. @@ -2493,7 +2500,7 @@ pub fn send_getheaders( ) -> Result { ensure_origin(st, hub); super::path::append_stored_headers(st, hub); - let owed = st.header_walk.challenge.is_some() || st.header_walk.probe.is_some(); + let owed = probe_or_challenge_owed(st); if !owed && super::path::walk_successor_indexed(st, hub) { release_lane(&mut st.header_walk.walk); return Ok(false); diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 16ff064c8..16517f647 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -151,16 +151,28 @@ impl PeerChain { self.from = self.from.saturating_add(over as u32); } - /// This node's confirmed chain above `lca` was rewound away, and `left` - /// was its block at `lca + 1`. A record that starts past that height took - /// those blocks as the peer's chain, so the peer split from the new one - /// there. A lower split already says where it left. - pub fn note_rewind(&mut self, lca: u32, left: BlockHash) { + /// This node's confirmed chain above `lca` was rewound away, `left` was + /// its block at `lca + 1`, and `path` is the chain planted in its place. + /// A record that starts past that height took the old blocks as the + /// peer's chain, so the peer split from the new one there. A split the + /// peer showed at or below that height already says where it left, and + /// an announced hash on the new path says the peer is on it. + pub fn note_rewind(&mut self, path: &HashMap, lca: u32, left: BlockHash) { let at = lca.saturating_add(1); - if self.from > at && self.split.is_none_or(|(h, _)| h > at) { - self.split = Some((at, left)); - self.split_confirmed = false; + if self.from <= at { + return; + } + if self.split.is_some_and(|(h, _)| h <= at) { + return; + } + if (self.from..) + .zip(&self.announced) + .any(|(h, hash)| path.get(&h) == Some(hash)) + { + return; } + self.split = Some((at, left)); + self.split_confirmed = false; } /// The recorded split's height has left the path: compare it with this diff --git a/crates/rbitcoin-ibd/src/ibd/reorg.rs b/crates/rbitcoin-ibd/src/ibd/reorg.rs index 902f7e0f1..b798449f8 100644 --- a/crates/rbitcoin-ibd/src/ibd/reorg.rs +++ b/crates/rbitcoin-ibd/src/ibd/reorg.rs @@ -634,11 +634,6 @@ pub fn commit_header_rewind( prep: &RewindPrep, ) { st.clear_path_above(prep.lca_h); - if let Some(left) = prep.left { - for slot in &mut st.slots { - slot.chain.note_rewind(prep.lca_h, left); - } - } hub.query.clear_milestone_path_above(prep.lca_h); super::header_walk::on_confirmed_rewind(st, hub, prep.lca_h); hub.query.set_lookup_taken_hi(Some(prep.lca_h)); @@ -661,6 +656,11 @@ pub fn commit_header_rewind( } prev = *hash; } + if let Some(left) = prep.left { + for slot in &mut st.slots { + slot.chain.note_rewind(&st.height_to_hash, prep.lca_h, left); + } + } for h in hub.query.block_queue_queued_heights() { if h > prep.lca_h { let _ = hub.query.block_queue_dequeue_height(h); diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 2fd3ae341..59da3471c 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1869,10 +1869,57 @@ fn two_peers_reserve_the_header_walk() { assert_eq!(st.header_walk.probe, None); assert_ne!(st.header_walk.probe_open, Some(14)); assert!(!st.slots[1].chain.probed(), "it is picked again later"); + // Neither is a peer with an expired walk ask that it has not answered, + // after another peer's expiry took the lane's last slot for it. + st.header_walk.walk.prev_peer = None; + st.header_walk.header_misses.insert(14, 1); + probe_unprobed_peer(&mut st); + assert_ne!(st.header_walk.probe, Some(14)); + st.header_walk.header_misses.clear(); + + // An open probe keeps the lane until it is answered or expires, also + // when the walk has nothing left to ask. + for s in &mut st.slots { + s.peer_height = 0; + } + st.header_walk.walk.peer = None; + st.header_walk.walk.at = None; + probe_unprobed_peer(&mut st); + assert_eq!(st.header_walk.probe, Some(14)); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(st.header_walk.probe_open, Some(14)); + let _ = send_getheaders(&mut st, &hub6); + assert_eq!( + st.header_walk.walk.peer, + Some(14), + "the probe keeps its lane" + ); + // With header sync otherwise done, the poll still asks once the probe is + // past its window, so the ask that expires it is sent. + st.headers_done = true; + let quiet = || HeaderPollIn { + live: 0, + need_ready_headroom: false, + lag: 0, + min_cache: 0, + fan: 0, + }; + assert!(matches!(header_poll(&st, quiet()), HeaderPoll::Wait)); + age_header_ask(&mut st); + assert!( + matches!(header_poll(&st, quiet()), HeaderPoll::Ask), + "a probe past its window is polled" + ); + let _ = send_getheaders(&mut st, &hub6); + assert_eq!( + st.header_walk.probe_open, None, + "an unanswered probe expires" + ); // A seventh, on its own hub, because a rewind replaces the confirmed // blocks a peer's record took as its chain: that peer is not asked for - // the new chain's blocks, and one that had shown the new chain is. + // the new chain's blocks, and one that had shown the new chain, all of + // it or only its top, is. let (_dir7, hub7) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-rewound"); hub7.ensure_genesis().unwrap(); let gen7 = hub7.tip_hash().unwrap(); @@ -1889,7 +1936,12 @@ fn two_peers_reserve_the_header_walk() { let (old, _old_rx) = slot(17); let (fresh, _fresh_rx) = slot(18); let (taken, _taken_rx) = slot(19); - let mut st = IbdWorkState::new(vec![old, fresh, taken], hub7.tip_hash(), hub7.tip_height()); + let (suffix, _suffix_rx) = slot(20); + let mut st = IbdWorkState::new( + vec![old, fresh, taken, suffix], + hub7.tip_hash(), + hub7.tip_height(), + ); let confirmed = |h| rbitcoin_ibd::ibd::peer_chain::confirmed_at(&hub7, h); let l3 = mine(l2.block_hash(), 74); st.slots[0].chain.note_headers( @@ -1902,6 +1954,10 @@ fn two_peers_reserve_the_header_walk() { st.slots[2] .chain .note_headers(&st.height_to_hash, confirmed, 2, 1, &[w1, w2]); + // Announced only the new chain's top, above the old tip. + st.slots[3] + .chain + .note_headers(&st.height_to_hash, confirmed, 2, 3, &[w3]); st.header_walk.tip.hash = Some(w3.block_hash()); st.header_walk.tip.height = 3; assert!(rbitcoin_ibd::ibd::reorg::consider_disconnected_heavier(&mut st, &hub7).unwrap()); @@ -1912,6 +1968,10 @@ fn two_peers_reserve_the_header_walk() { ); assert!(may_serve(&st, 18, &w1.block_hash())); assert!(may_serve(&st, 19, &w2.block_hash())); + assert!( + may_serve(&st, 20, &w3.block_hash()), + "a peer that announced the new chain is asked for it" + ); } /// A long chain on an invalid block stays refused past the cap on diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs index 8b67884eb..1b72c2590 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -105,6 +105,29 @@ fn a_reply_at_or_below_the_tip_ends_the_chain_there() { ); } +/// A rewind keeps a split the peer itself showed at or below the fork. A +/// peer on a third chain stays off both chains when this node rewinds away +/// from its chain and back. +#[test] +fn a_third_chain_split_survives_two_rewinds() { + let ours = chain(3, 0x5c); + let won = chain(3, 0x77); + let third = chain(3, 0x91); + let path = |c: &[Header]| -> HashMap { + (1..).zip(c.iter().map(|h| h.block_hash())).collect() + }; + let confirmed = |h: u32| Some(ours[h as usize - 1].block_hash()); + let mut peer = PeerChain::default(); + peer.note_headers(&HashMap::new(), confirmed, 2, 1, &third); + peer.note_rewind(&path(&won), 0, ours[0].block_hash()); + assert!(peer.left_path_by(&path(&won), 1)); + peer.note_rewind(&path(&ours), 0, won[0].block_hash()); + assert!( + peer.left_path_by(&path(&ours), 1), + "a peer on a third chain is not taken back onto this one" + ); +} + /// A `notfound` keeps a peer off that block for one stall window, then the /// peer may be asked again. #[test] From a372775c07ad9a2b122c2ec9ed6d2dd6720eaae5 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 12:13:21 -0400 Subject: [PATCH 12/17] net: refuse only a linked header batch on a rejected block Review: `refuse_invalid_batch` ran before any linkage or proof-of-work check and remembered the last header of every batch it refused. One `headers` message with a header naming a rejected block, followed by an honest header, put that honest header in `invalid_tips`. After that, every batch holding it was refused and its peer left the walk, so in the #1082 case IBD stayed at the fork as before this branch. A batch is refused, and its last header remembered, only when it is one linked chain that meets its own proof of work. An unlinked batch goes to the usual checks, which drop it. A linked batch that builds on or holds a rejected block cannot end in an honest header. Co-Authored-By: Claude Opus 5.5 --- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 11 ++++++++--- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 16 ++++++++++++++++ 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index b1ecd5ff4..4b0a46086 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -882,13 +882,18 @@ fn note_invalid_tip(st: &mut IbdWorkState, hash: BlockHash) { } } -/// Refuse a batch that holds or builds on a rejected block, and retire -/// `peer` from the walk. The batch's last header is remembered, so the next -/// batch of that chain is refused too. +/// Refuse a linked batch that holds or builds on a rejected block, and +/// retire `peer` from the walk. The batch's last header is remembered, so +/// the next batch of that chain is refused too. pub fn refuse_invalid_batch(st: &mut IbdWorkState, peer: usize, headers: &[Header]) -> bool { let Some(first) = headers.first() else { return false; }; + // An unlinked batch does not build on what it names, so its last + // header is not remembered. The usual checks drop it. + if !pow_linked(headers) { + return false; + } let rejected = |hash: &BlockHash| { st.header_walk.invalid_tips.contains(hash) || st.reorg.invalid.contains(hash.to_byte_array()) diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 59da3471c..0c06cc609 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -2006,6 +2006,22 @@ fn a_long_chain_on_an_invalid_block_stays_refused() { refuse_invalid_batch(&mut st, 0, &[side_next]), "a branch refused after the oldest tips is still refused past the cap" ); + + // A batch that is not one linked chain does not build on the invalid + // block, so its last header is not remembered: an honest header cannot + // be refused by sending it after one that is. + let honest1 = mine(gen, 7_001); + let honest2 = mine(honest1.block_hash(), 7_002); + let junk = Header { + prev_blockhash: bad.block_hash(), + ..honest1 + }; + assert!(!refuse_invalid_batch(&mut st, 0, &[junk, honest2])); + assert!(!refuse_invalid_batch(&mut st, 0, &[bad, honest1])); + assert!( + !refuse_invalid_batch(&mut st, 0, &[honest1, honest2]), + "the honest chain is not refused" + ); } /// A peer that announced the path and then forks below the confirmed tip no From 552a4b87a9dcf327d444c48b6ac7070d5a1a93e7 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 12:18:08 -0400 Subject: [PATCH 13/17] net: clear a peer's split once it builds on this chain above it Review: a recorded split was cleared only by a later reply holding a header at that exact height. A reply that builds on this node's block above the split proves the peer has this chain there, but the split stayed, and once that height was confirmed the peer counted as having left for good. A one-block race near the end of IBD was enough. - A reply whose first header's parent is this node's block clears a split at or below that block. - The kept run is continued only from the parent of the reply's first new header. A reply at the next height that built on another block was appended to the run, so the peer still looked as if it held the stale sibling. Co-Authored-By: Claude Opus 5.5 --- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 36 +++++++++++++++++---- crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 35 ++++++++++++++++++++ 2 files changed, 65 insertions(+), 6 deletions(-) diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 16517f647..1aba71a59 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -104,9 +104,24 @@ impl PeerChain { self.probed = true; let hashes: Vec = headers.iter().map(|h| h.block_hash()).collect(); let mut split_moved = false; + let ours = |ht: u32| { + path.get(&ht) + .copied() + .or_else(|| (ht <= tip).then(|| confirmed(ht)).flatten()) + }; + // A reply on this node's block proves this chain below it, so a split + // at or below that block is gone. Every chain shares genesis. + if let Some(first) = headers.first() { + if height > 1 + && ours(height - 1) == Some(first.prev_blockhash) + && self.split.is_some_and(|(at, _)| at < height) + { + self.split = None; + } + } for (ht, hash) in (height..).zip(&hashes) { let on_path = path.get(&ht).copied(); - let Some(our) = on_path.or_else(|| (ht <= tip).then(|| confirmed(ht)).flatten()) else { + let Some(our) = ours(ht) else { continue; }; if our == *hash { @@ -139,11 +154,20 @@ impl PeerChain { }; let start = height.saturating_add(skip as u32); let end = self.from.saturating_add(self.announced.len() as u32); - if self.announced.is_empty() || start < self.from || start > end { - self.announced.clear(); - self.from = start; - } else { - self.announced.truncate((start - self.from) as usize); + // The kept run continues only from the parent of `start`. + let parent = match skip { + 0 => headers[0].prev_blockhash, + k => hashes[k - 1], + }; + let keep = (start > self.from && start <= end) + .then(|| (start - self.from) as usize) + .filter(|&k| self.announced[k - 1] == parent); + match keep { + Some(k) => self.announced.truncate(k), + None => { + self.announced.clear(); + self.from = start; + } } self.announced.extend(fresh.iter().copied()); let over = self.announced.len().saturating_sub(ANNOUNCED_CAP); diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs index 1b72c2590..68b35e760 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -128,6 +128,41 @@ fn a_third_chain_split_survives_two_rewinds() { ); } +/// A peer that showed a stale sibling and then builds on this node's block +/// at or above that height has joined this chain: the split is gone. +#[test] +fn a_reply_on_this_chain_above_a_split_clears_it() { + let ours = chain(3, 0x5c); + let sibling = chain(1, 0x77); + let path: HashMap = (1..).zip(ours.iter().map(|h| h.block_hash())).collect(); + let mut peer = PeerChain::default(); + peer.note_headers(&path, |_| None, 0, 1, &sibling); + assert!(peer.left_path_by(&path, 2)); + peer.note_headers(&path, |_| None, 0, 2, &ours[1..]); + assert!(!peer.left_path_by(&path, 2)); + assert!(!peer.left_path_by(&path, 3)); + assert!( + peer.shows(&path, 3), + "the peer is asked for this chain again" + ); + + // A reply on this node's block that leaves it at the next height: the + // parent clears the old split and the new one is where it left. + let mut moved = PeerChain::default(); + moved.note_headers(&path, |_| None, 0, 1, &sibling); + let other = chain(2, 0x91); + let off = Header { + prev_blockhash: ours[0].block_hash(), + ..other[1] + }; + moved.note_headers(&path, |_| None, 0, 2, &[off]); + assert!( + !moved.left_path_by(&path, 1), + "it has this node's block below" + ); + assert!(moved.left_path_by(&path, 2)); +} + /// A `notfound` keeps a peer off that block for one stall window, then the /// peer may be asked again. #[test] From 07e3124a5cda01b5b1e3fb0c7cc774ab9afcbd74 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 12:20:51 -0400 Subject: [PATCH 14/17] net: resume a saved walk past stored failures below its base Review: `restore_adopt` refused a saved candidate above the tip whenever `block.invalid` listed any block, also one far below the confirmed tip that cannot be in a candidate whose base matches the confirmed chain. Once one failure was stored, every later restart during IBD skipped the resume until `reconsiderblock`. The resume is refused only when a listed block is above the saved base, or when its height is not found within 64 headers of the best chain. Co-Authored-By: Claude Opus 5.5 --- SCHEMA.md | 2 +- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 25 +++++++++++++- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 34 ++++++++++++++++++++ 3 files changed, 59 insertions(+), 2 deletions(-) diff --git a/SCHEMA.md b/SCHEMA.md index 5661ab7cb..1f3107dec 100644 --- a/SCHEMA.md +++ b/SCHEMA.md @@ -393,7 +393,7 @@ then milestone hash [u8; 32] (zeros if none) Each checkpoint is 285 bytes. The tail is 364 bytes. Length is exact: `12 + count × 285 + 364`. An older or shorter file does not parse, and the script skip stays off. -Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. While `block.invalid` lists a block, a saved candidate above the confirmed tip is not restored: it may build on that block, so the walk starts again from the tip. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. +Each checkpoint is the hash, height, and total work at the end of one look-ahead reply past the download queue, plus that reply's last header, up to 11 timestamps ending there, and the difficulty period after that header. `ntimes` is how many of those 11 slots are live, oldest first. The milestone hash is the header at the anchored milestone height when the walk passed it. The base is the stored header the checkpoints were built on. On restart, if that height already has a different hash, the file is ignored. A reorg below the base deletes the file. While `block.invalid` lists a block above the saved base, or one whose height is not found within 64 headers of the best chain, a saved candidate above the confirmed tip is not restored: it may build on that block, so the walk starts again from the tip. A listed block at or below the base is on no chain that walk can take. The per-checkpoint header, timestamps, and period-start header let the next look-ahead, a fork from that hash, or a rewind onto it check `nBits` and median time without a row in `header.body`. Full-difficulty height and bits are the last header in the period whose `nBits` are not the minimum-difficulty limit, so a restart can walk testnet difficulty back. The same pair is stored again for the base, for when every checkpoint has been rewound. The tail's period fields are the walk tip. Headers between the queue and that tip are not in this file. ### `block.invalid` diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 4b0a46086..9ad69aa03 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -3222,6 +3222,29 @@ pub fn retire_adopt_if_confirmed(st: &mut IbdWorkState, hub: &ChainHub) { /// Read `header.adopt`. A file that does not parse does not restore /// checkpoints. Headers already queued and linked from the confirmed tip are /// still noted on the milestone path. +/// Ancestors walked to find a stored failure's height. +const FAILURE_HEIGHT_WALK: usize = 64; + +/// A stored failure that a walk saved from `base_height` may build on: one +/// above that base, or one whose height is not found within a few headers +/// of the best chain. The chain at or below the base is the confirmed one. +fn failure_above(hub: &ChainHub, base_height: u32) -> bool { + hub.invalid_blocks() + .into_iter() + .any(|hash| failure_height(hub, hash).is_none_or(|h| h > base_height)) +} + +fn failure_height(hub: &ChainHub, hash: BlockHash) -> Option { + let path = + super::reorg::header_hashes_to_best_ancestor_n(hub, hash, FAILURE_HEIGHT_WALK).ok()?; + let join = super::reorg::parent_hash_of(hub, *path.first()?).ok()??; + if !hub.is_connected(&join) { + return None; + } + let join_h = hub.query.height_of_hash(&join.to_byte_array()).ok()??.0; + Some(join_h.saturating_add(path.len() as u32)) +} + pub fn restore_adopt(st: &mut IbdWorkState, hub: &ChainHub) -> bool { retire_adopt_if_confirmed(st, hub); let started = std::time::Instant::now(); @@ -3254,7 +3277,7 @@ pub fn restore_adopt(st: &mut IbdWorkState, hub: &ChainHub) -> bool { } // A saved candidate above the confirmed tip may build on a block marked // invalid after it was saved. The walk starts again from the tip. - if parsed.tip.height > hub.tip_height().unwrap_or(0) && !hub.invalid_blocks().is_empty() { + if parsed.tip.height > hub.tip_height().unwrap_or(0) && failure_above(hub, parsed.base_height) { rbitcoin_log::info!( "ibd: headers resume refused height={}: invalid blocks are stored", parsed.tip.height diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 0c06cc609..9692b3243 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1338,6 +1338,40 @@ fn restart_restores_the_checkpoint_skip_and_garbage_does_not() { assert!(!restore_adopt(&mut st, &hub)); } +/// A failure stored at or below the saved walk's base is on no chain that +/// walk can take, so the walk is restored. One whose height is not found may +/// be on it. +#[test] +fn a_failure_below_the_saved_base_keeps_the_resume() { + let (_dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-resume-below"); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let b1 = mine_block(gen, 1); + hub.accept_block(b1.clone()).unwrap(); + let b2 = mine_block(b1.block_hash(), 2); + hub.accept_block(b2.clone()).unwrap(); + let stale = mine(gen, 64); + hub.ensure_header(&stale).unwrap(); + assert!(hub.note_failed_connect(stale.block_hash())); + let (s2, _rx2) = slot(2); + let mut st = IbdWorkState::new(vec![s2], hub.tip_hash(), hub.tip_height()); + assert!(send_getheaders(&mut st, &hub).unwrap()); + let q3 = mine(b2.block_hash(), 65); + let q4 = mine(q3.block_hash(), 66); + apply(&mut st, &hub, 2, vec![q3, q4]); + save_adopt(&st, &hub); + st.header_walk = HeaderWalk::default(); + assert!( + restore_adopt(&mut st, &hub), + "a failure below the walk's base does not stop the resume" + ); + assert_eq!(st.header_walk.tip_hash(), Some(q4.block_hash())); + // A failure whose height is not found may be on it. + assert!(hub.note_failed_connect(BlockHash::from_byte_array([0x5a; 32]))); + st.header_walk = HeaderWalk::default(); + assert!(!restore_adopt(&mut st, &hub)); +} + #[test] fn a_failed_adopt_still_notes_the_queued_path() { let (_dir, mut hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-renote"); From 39ec677973f8da7f3ffa370d3f32cf3e868ccd36 Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 13:05:48 -0400 Subject: [PATCH 15/17] net: wait on a block whose time is too far ahead instead of halting IBD Review: this branch made a "timestamp too far in future" reject a requeue, as in tip-follow and Core. But the confirm lookup selects the body again at once, and `apply_cascade_reject` halts IBD after three rejects at the same tip. A stored header that a backward clock correction left more than 2 hours ahead halted IBD within milliseconds. - Such a reject puts a 1 s hold on the confirm feed, and the lookup selects nothing while it holds, so the block is tried about once a second instead of in a tight loop. - It does not count toward the cascade halt. The block connects once the clock catches up. Header acceptance already refuses such a header, so only a clock stepped back can leave one stored. Co-Authored-By: Claude Opus 5.5 --- changelog.d/ibd-invalid-heavier.md | 4 +- crates/rbitcoin-ibd/src/ibd/confirm/mod.rs | 38 +++++ crates/rbitcoin-ibd/src/ibd/events/mod.rs | 11 ++ .../tests/ibd/events/confirm_reject_tests.rs | 148 ++++++++++++++++++ 4 files changed, 200 insertions(+), 1 deletion(-) diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md index bf226e72c..360412cfa 100644 --- a/changelog.d/ibd-invalid-heavier.md +++ b/changelog.d/ibd-invalid-heavier.md @@ -29,4 +29,6 @@ Fixed kept in `store/block.invalid`; `reconsiderblock` clears one. The startup header scan and the IBD resume seed do not walk through them. - **IBD does not blacklist a block whose timestamp is too far in the - future.** It is requeued, as in tip-follow and Core. + future.** A block that a clock stepped back left more than 2 hours ahead + waits until the clock catches up, instead of being rejected until IBD + halts. diff --git a/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs b/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs index 478e4925a..ba88da85c 100644 --- a/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/confirm/mod.rs @@ -198,6 +198,9 @@ pub struct ConfirmFeed { pub epoch: AtomicU64, /// Isolate until confirmed tip reaches this height (`u32::MAX` = off). pub force_single_until: AtomicU32, + /// Lookup selects nothing until this IBD mono ms (0: no hold): a block + /// was too far ahead of the clock to connect. + pub hold_until_ms: AtomicU64, } pub struct ConfirmFeedInner { @@ -227,9 +230,21 @@ impl ConfirmFeed { stop: AtomicBool::new(false), epoch: AtomicU64::new(0), force_single_until: AtomicU32::new(u32::MAX), + hold_until_ms: AtomicU64::new(0), } } + /// Lookup waits `for_` before it selects a body again. + pub fn hold_for(&self, for_: Duration) { + let until = super::peer_io::ibd_mono_ms().saturating_add(for_.as_millis() as u64); + self.hold_until_ms.store(until, Ordering::Release); + } + + /// A hold from [`Self::hold_for`] has not passed. + pub fn held(&self) -> bool { + super::peer_io::ibd_mono_ms() < self.hold_until_ms.load(Ordering::Acquire) + } + pub fn epoch(&self) -> u64 { self.epoch.load(Ordering::Acquire) } @@ -901,6 +916,16 @@ pub fn publish_confirm_progress(query: &rbitcoin_query::Query) { publish_confirm_progress_at(query, Instant::now()); } +/// Pause before a block too far ahead of the clock is tried again. +pub const TIME_TOO_NEW_HOLD: Duration = Duration::from_secs(1); + +/// A reject for a block more than 2 hours ahead of the clock: it is valid +/// once the clock catches up. +pub fn is_time_too_new(err: &str) -> bool { + let s = err.to_ascii_lowercase(); + s.contains("timestamp too far in future") || s.contains("time-too-new") +} + #[allow(clippy::too_many_arguments)] // call-site args stay unbundled #[doc(hidden)] pub fn emit_confirm_reject( @@ -915,6 +940,11 @@ pub fn emit_confirm_reject( parent: RejectParent, ) -> Result<(), std::sync::mpsc::SendError> { let class = class.isolate_if_batched(batch_len); + // A clock stepped back can leave a stored header more than 2 hours + // ahead. Try it again after a pause, not at once. + if is_time_too_new(&err) { + feed.hold_for(TIME_TOO_NEW_HOLD); + } if class == ConfirmRejectClass::Cascade && batch_len > 1 { let until = height.saturating_add(batch_len as u32).saturating_sub(1); feed.request_single_block(until); @@ -2823,6 +2853,14 @@ pub fn spawn_confirm_engine( if feed.stopped() { break; } + if feed.held() { + let g = feed.inner.lock().unwrap_or_else(|err| err.into_inner()); + let _ = feed + .cv + .wait_timeout(g, Duration::from_millis(20)) + .unwrap_or_else(|err| err.into_inner()); + continue; + } if hub.query.take_disconnect(&mut disco_seen).is_some() { hub.query.set_lookup_taken_hi(None); } diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index 3ccb514b9..cdb348b95 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -1437,6 +1437,17 @@ pub fn apply_cascade_reject( // Leave the body queue: the plan was stale, and the confirm thread that // rejected offered any body it held back. clear_hash_inflight(&mut st.slots, &mut st.inflight, hash); + // A block too far ahead of the clock is not a stale plan: the engine + // waits before it tries again, and it connects once the clock catches up. + if super::confirm::is_time_too_new(err) { + note_confirm_stuck(st); + static N: AtomicU32 = AtomicU32::new(0); + let n = N.fetch_add(1, Ordering::Relaxed) + 1; + if n <= 3 || n.is_multiple_of(60) { + warn!("ibd: confirm waits @{height} {hash}: {err} (n={n})"); + } + return; + } const CASCADE_HALT_AFTER: u8 = 3; let tip = hub .and_then(|h| h.tip_hash()) diff --git a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs index 2db43e10d..60a1c636a 100644 --- a/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs +++ b/crates/rbitcoin-ibd/tests/ibd/events/confirm_reject_tests.rs @@ -3127,6 +3127,43 @@ fn isolated_soft_reject_does_not_count_toward_cascade_halt() { assert!(st.halt.is_none(), "{:?}", st.halt); } +/// A block too far ahead of the clock is not a stale plan: its rejects at +/// one tip do not halt IBD, while a stale plan at that tip halts on the +/// third. +#[test] +fn a_block_too_far_ahead_does_not_count_toward_the_cascade_halt() { + let mut st = IbdWorkState::new(Vec::new(), Some(h(0)), Some(0)); + let hash = h(1); + st.record_height(hash, 1); + let reject = |st: &mut IbdWorkState, err: &str| { + apply_confirm_reject_class( + st, + 1, + hash, + ConfirmRejectClass::Cascade, + err, + None, + None, + 1, + None, + None, + RejectParent::default(), + ); + }; + for _ in 0..5 { + reject( + &mut st, + "consensus: bad header: timestamp too far in future", + ); + reject(&mut st, "time-too-new"); + } + assert!(st.halt.is_none(), "{:?}", st.halt); + for _ in 0..3 { + reject(&mut st, "fk mismatch"); + } + assert!(st.halt.is_some()); +} + /// tip+1 and tip+2 run scripts in one wave and tip+2's script fails. The /// scripts reject offers the wave back and isolates it: tip+1 connects from /// the offered-back body with no second getdata and no stale-plan retry, @@ -3569,6 +3606,117 @@ fn ibd_flag_zero_tx_block_is_consensus_invalid() { let _ = std::fs::remove_dir_all(dir); } +/// A block whose header a clock stepped back left more than 2 hours ahead +/// waits. The engine holds before it tries again instead of retrying at +/// once, those rejects do not halt IBD, and the block connects once the +/// clock passes its time. +#[test] +fn a_block_too_far_ahead_waits_and_then_connects() { + use super::super::assign::dummy_slot; + use bitcoin::consensus::serialize; + use rbitcoin_ibd::ibd::confirm::{spawn_confirm_engine, ConfirmEvent, ConfirmFeed}; + use rbitcoin_ibd::ibd::events::{apply_confirm_events, apply_peer_event}; + use rbitcoin_ibd::ibd::peer_io::PeerEvent; + use rbitcoin_ibd::ibd::state::InflightReq; + use rbitcoin_ibd::ibd::status::LoopStats; + use rbitcoin_net::seeds::AddrMan; + use std::net::{IpAddr, Ipv4Addr, SocketAddr}; + use std::sync::atomic::AtomicU32; + use std::sync::mpsc::{channel, RecvTimeoutError}; + use std::sync::Arc; + use std::time::{Duration, Instant}; + + let (dir, hub) = rbitcoin_net::chain::tiny_regtest_hub_labeled("too-far-ahead"); + hub.query.enter_direct_index_mode().unwrap(); + let hub = Arc::new(hub); + hub.ensure_genesis().unwrap(); + let gen = hub.tip_hash().unwrap(); + let now = rbitcoin_consensus::clock::wall_now(); + let ahead = now + 2 * 3600 + 3; + let block = mine(gen, (ahead - 1_300_000_000) as u32, 1, vec![]); + let hash = block.block_hash(); + // Stored while the clock was ahead. + let fk = rbitcoin_consensus::with_now(ahead, || hub.ensure_header_fk(&block.header)).unwrap(); + + let mut st = IbdWorkState::new(vec![dummy_slot(1)], Some(gen), Some(0)); + let feed = Arc::new(ConfirmFeed::new()); + st.record_height(hash, 1); + st.header_fks.insert(hash, fk); + st.ordered.push_back(hash); + st.ordered_set.insert(hash); + st.slots[0].in_flight.insert(hash); + st.inflight.insert(hash, InflightReq::new(1)); + apply_peer_event( + &mut st, + &hub, + PeerEvent::BlockFramed { + peer: 1, + hash, + payload: serialize(&block), + header_fk: None, + }, + &AtomicU32::new(1), + &mut AddrMan::new(), + SocketAddr::new(IpAddr::V4(Ipv4Addr::new(10, 0, 0, 1)), 1), + Some(feed.as_ref()), + ); + + let (ev_tx, ev_rx) = channel(); + let (engine, _queues) = spawn_confirm_engine( + Arc::clone(&hub), + Arc::clone(&feed), + ev_tx, + Arc::new(AtomicU32::new(0)), + Arc::new(LoopStats::default()), + ); + let started = Instant::now(); + let deadline = started + Duration::from_secs(20); + let mut rejects = 0u32; + let connected = loop { + match ev_rx.recv_timeout(Duration::from_millis(5)) { + Ok(ev @ ConfirmEvent::Reject { .. }) => { + rejects += 1; + let (tx, rx) = channel(); + tx.send(ev).unwrap(); + drop(tx); + apply_confirm_events( + &mut st, + &hub, + &rx, + &AtomicU32::new(1), + &AtomicU32::new(0), + &mut Instant::now(), + Some(feed.as_ref()), + ); + assert!(st.halt.is_none(), "a block too far ahead does not halt IBD"); + } + Ok(ConfirmEvent::Accepted { .. }) => break true, + Err(RecvTimeoutError::Timeout) => { + if hub.tip_hash() == Some(hash) { + break true; + } + if Instant::now() > deadline { + break false; + } + } + Err(RecvTimeoutError::Disconnected) => break false, + } + }; + feed.request_stop(); + feed.notify(); + let _ = engine.join(); + assert!( + connected, + "the block connects once the clock passes its time" + ); + let waited = started.elapsed().as_secs().max(1) as u32; + assert!( + rejects <= 3 * waited + 2, + "{rejects} rejects in {waited}s: the engine waits between tries" + ); + let _ = std::fs::remove_dir_all(dir); +} + /// tip+1 spends a txid no block created. Its body matches its header, so /// the missing parent is the block's own fault (Core /// `bad-txns-inputs-missingorspent`): the hash is marked invalid, IBD does From 8bebe6d2df34e45462f13d618273491480d5c94d Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 12:42:11 -0400 Subject: [PATCH 16/17] net: ask an IBD block peer only for blocks its chain showed Review: a block peer was eligible for any block it had not ruled out. A peer with no record counted as holding every block, and one whose full probe reply ended below a block stayed eligible above it, so an honest Core peer could still be asked for a fork's blocks, stay silent, and take the 10 minute cooldown in #1082. The issue's direction is Core's rule: ask a peer only for a block on the chain it announced. - `may_serve` asks a peer only for a block its record shows. With no peer that shows the block, a peer whose chain has not left the path is asked, else any peer outside its `notfound` window, so a stale record cannot stop a download. The header peer stays on headers while another peer can fetch the block, except in that last case. - A probe asks for headers after the download path's top alone. A reply on that top shows every path block below it, up to the 64,000-header queue, so one probe covers a caught-up peer. An empty reply (also a peer below the minimum work), or one from genesis that stops at or below the tip, is asked again from the confirmed tip. - A recorded split whose hash is now on the path shows that block too. - A peer whose record no longer decides the next 4,096 heights is probed again, at most every 10 s. docs/ibd-memory.md lists the per-peer record and `invalid_tips`, and says how block peers are chosen. Co-Authored-By: Claude Opus 5.5 --- TESTING.md | 4 +- changelog.d/ibd-invalid-heavier.md | 17 +- crates/rbitcoin-ibd/src/ibd/events/mod.rs | 47 +++-- crates/rbitcoin-ibd/src/ibd/header_walk.rs | 116 ++++++++---- crates/rbitcoin-ibd/src/ibd/mod.rs | 2 +- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 59 +++++- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 182 +++++++++++++++---- crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 32 ++++ docs/ibd-memory.md | 3 +- 9 files changed, 366 insertions(+), 96 deletions(-) diff --git a/TESTING.md b/TESTING.md index 059afaa14..ceb687330 100644 --- a/TESTING.md +++ b/TESTING.md @@ -200,7 +200,7 @@ absolute path. Keep default-tier fixtures small: on tmpfs, test bytes are RAM. | Remining 100-block maturity pads with `confirm_wire_run` | `pad_empty_from` / `build_mature_regtest_with_spend` **once per binary journey** (not once per skinny test) | | Wall-time multi-round microbenches in default suite | Deterministic structure / chunk-load asserts; demote wall arms to `#[ignore]` | -**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). `end_of_ibd_invalid_heavier_fork` 60s wall (180s under llvm-cov), each sync 20s. A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer stays connected and is not asked for that hash again until it announces more. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. +**P2P walls:** `two_node_header_and_block_sync`, `three_node_relay_path`, `ibd_two_peers`, `tip_follow_after_ibd`, `tip_follow_getheaders_catches_missed_blocks`, and `node_run_p2p_short` 60s wall (180s under `coverage.sh` / llvm-cov). `serve_after_restart_via_reconstruct` and `end_of_ibd_follow` 90s wall (180s under llvm-cov). `end_of_ibd_sh_interrupt` 120s wall (240s under llvm-cov). `end_of_ibd_work_fork` 30s wall (90s under llvm-cov). `end_of_ibd_invalid_heavier_fork` 60s wall (180s under llvm-cov), each sync 20s. A warm run is about 5s once an unknown-block getdata is answered `notfound`; the old 150s cap hid two 30s stall waits. The peer stays connected and is not asked for that hash again for 30 seconds or until it announces more. `p2p_compact_hb_getblocktxn_and_orphan` 30s wall (90s under llvm-cov). `p2p_timeout_getaddr_and_keepalive_ping`, `p2p_feeler_completes_and_closes`, and `p2p_inbound_full_rejects_extra` 20s wall. Live `P2PNode` tests in one process may overlap. Each script stage registers its thread; hub-only reorgs do not run a `P2PNode`. **Speed / reliability (default suite):** prefer `pad_empty_from` / `build_mature_regtest_with_spend` **once per journey** (tx_relay live hub, Electrum protocol, core_analogs assumevalid+mempool) over remine pads. `pin_compose_multi_pack_timed` keeps functional + layout/covered short-circuit gates (multi-ms floor); sticky vs cold assemble is log-only (not a hard timing assert). Schema-13 wire rebuild must stamp create identity from `txid.body` — zero batch identity is treated as missing (regression covered by the spend reconstruct in `consensus_mature_chain_spend_reconstruct_and_scripthash` + multi-vout confirm scenarios). Coverage vs speed: prefer **one** scenario at the real entry over N micro-opens that only paint lines; when adding coverage for reduce/materialize, use a **tiny** target, not production stream depth. @@ -403,7 +403,7 @@ A cell that names a crate-local test describes the witness that exists today. It | `hostile_peer_session` | P2P (**default**, crate) | One lib entry in `peer::tests`, one in `chain::tests`, one in `assign::tests`. Header cap, send budget, one-shot `getaddr`, addr relay to one or two peers, zero-prev not held, witness padding and time-too-new not cached invalid. A getdata past the byte budget pauses and is served once the writer drains (`getdata_over_send_budget_waits_for_writer`). `tip_script_pres_skips_only_matching_wtxid` is the mempool-graph result, not this peer session. | | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. | | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. | -| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. A sixth, on its own hub, because the probe must say what a peer has above the tip: it asks from the confirmed tip; a peer whose reply is empty is not asked for path blocks and a caught-up peer is; an announcement that crosses the probe leaves its ask open; the reply is not a walk answer, and one that leaves the path gets the walk's own ask; an empty reply to a walk ask is probed again; and a peer with a walk ask out, or one expired and not answered, is neither picked nor asked; an open probe keeps the lane while the walk has nothing to ask and expires. A seventh, on its own hub, because a rewind replaces the confirmed blocks a peer's record took as its chain: the main-loop rewind walks back from a walk candidate with nothing planted above the tip; a peer on the old chain is not asked for the new one, and a peer whose split or announced headers are on the new chain is. | +| `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a block the other peer answered `notfound` goes to the header peer; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. A fourth, because the header peer must hold the path: a block peer that never announced a chain is asked for headers once; a peer whose reply forks below the path, or whose short reply ends below a block, is not asked for it; a peer that announced nothing does not keep the header peer from a block it showed; with no peer that holds a block by what it announced, a peer that did not answer `notfound` still is; and a batch stored after one of its blocks failed is not planted and does not grow the candidate. A fifth, on its own hub, because this node takes the forked peer's chain: once the fork height is confirmed and leaves the path, that peer is asked again, a peer that split where this node confirmed another block is not, and with no peer that holds a block by what it announced the header peer is asked too. A sixth, on its own hub, because a probe must say what a peer has: it asks after the path top alone; an empty reply there, or one from genesis that stops at the tip (this node's or Core's form), is asked again from the confirmed tip, where an empty reply is a chain that ends there; a reply on the top shows the path below it; an announcement that crosses the probe leaves its ask open; the reply is not a walk answer, and one with a header the walk has not seen gets the walk's own ask; an empty reply to a walk ask is probed again; and a peer with a walk ask out, or one expired and not answered, is neither picked nor asked; an open probe keeps the lane while the walk has nothing to ask and expires. A seventh, on its own hub, because a rewind replaces the confirmed blocks a peer's record took as its chain: the main-loop rewind walks back from a walk candidate with nothing planted above the tip; a peer on the old chain is not asked for the new one, and a peer whose split or announced headers are on the new chain is. An eighth, because a block is asked only of a peer whose chain showed it: a peer whose full reply ended below the block, or one with no record, is not asked while another peer shows it; with nobody showing a block, a peer that left the path is not asked while one that did not is; and a peer whose record stops short of the next blocks is probed again. | | `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. | | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. | | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. | diff --git a/changelog.d/ibd-invalid-heavier.md b/changelog.d/ibd-invalid-heavier.md index 360412cfa..3fbf78d27 100644 --- a/changelog.d/ibd-invalid-heavier.md +++ b/changelog.d/ibd-invalid-heavier.md @@ -15,14 +15,15 @@ Fixed is not asked again for a block it answered `notfound` for 30 seconds or until it announces more, or for a block above where its announced headers left the download path or ended. -- **IBD asks each block peer for headers once.** A peer that had announced - nothing was asked for blocks of any chain, so a Core peer serving a - lighter chain stayed silent on blocks it did not have and was dropped for - a 10 minute cooldown. Each block peer now gets one headers ask from the - confirmed tip, so its chain is known before blocks are asked of it. A - peer with nothing above the tip is not asked for blocks. An empty reply - to a header-sync ask does not say whether the peer is caught up or - behind, so that peer is asked again from the tip. +- **IBD asks a block peer only for blocks its chain showed.** A peer that + had announced nothing, or whose announced headers ended below a block, + was asked for blocks of any chain, so a Core peer serving a lighter chain + stayed silent on blocks it did not have and was dropped for a 10 minute + cooldown. Each block peer is asked for headers after the download path's + top, and from the confirmed tip when it does not have that top, and again + when its record no longer covers the next blocks. It is asked only for + blocks that record shows. With no peer that shows a block, a peer whose + chain has not left the path is asked first. - **A block that failed connect stays invalid after restart.** The mark was in memory only, so a restart adopted the stored header chain through that block again and rewound a valid tip to chase it. Connect failures are diff --git a/crates/rbitcoin-ibd/src/ibd/events/mod.rs b/crates/rbitcoin-ibd/src/ibd/events/mod.rs index cdb348b95..002839f45 100644 --- a/crates/rbitcoin-ibd/src/ibd/events/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/events/mod.rs @@ -933,9 +933,11 @@ fn note_peer_headers( } /// The reply to this peer's open probe, read into its chain record and kept -/// from the walk. The probe asked from the confirmed chain, so its reply is -/// empty or builds on a confirmed block; a batch that does not is an -/// announcement that crossed the probe. True when the reply was consumed. +/// from the walk. A probe after the path top is answered empty, on that top, +/// or from genesis when the peer does not have it. A probe from the +/// confirmed chain is answered empty or on a confirmed block. A batch of +/// another shape is an announcement that crossed the probe. True when the +/// reply was consumed. fn answer_probe( st: &mut IbdWorkState, hub: &ChainHub, @@ -946,25 +948,34 @@ fn answer_probe( return false; } let tip = hub.tip_height().unwrap_or(0); + let top = st.header_walk.probe_top; if let Some(first) = headers.first() { let prev = first.prev_blockhash; - let confirmed = hub + let confirmed_h = hub .query .height_of_hash(&prev.to_byte_array()) .ok() .flatten() - .is_some_and(|h| h.0 <= tip && super::peer_chain::confirmed_at(hub, h.0) == Some(prev)); - if !confirmed { + .map(|h| h.0) + .filter(|&h| h <= tip && super::peer_chain::confirmed_at(hub, h) == Some(prev)); + let shaped = match top { + Some((_, hash)) => { + prev == hash || prev.to_byte_array() == [0u8; 32] || confirmed_h == Some(0) + } + None => confirmed_h.is_some(), + }; + if !shaped { return false; } } st.header_walk.probe_open = None; + st.header_walk.probe_top = None; let _ = super::header_walk::take_header_ask(st, peer); - if headers.is_empty() { - if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { - s.chain.note_ends_at(tip); - } - } else { + let past_tip = headers.first().is_some_and(|first| { + parent_height(&st.hash_height, hub, first.prev_blockhash) + .is_some_and(|h| h.saturating_add(headers.len() as u32) > tip.saturating_add(1)) + }); + if !headers.is_empty() { note_announcement(st, peer); note_peer_headers(st, hub, peer, headers); // A header this node has not seen: the peer's chain leaves this one @@ -976,6 +987,20 @@ fn answer_probe( super::header_walk::note_block_inv(st, peer); } } + if let Some(s) = st.slots.iter_mut().find(|s| s.id == peer) { + match top { + Some((height, hash)) if headers.first().map(|h| h.prev_blockhash) == Some(hash) => { + s.chain.note_reaches(height, hash); + } + // A reply from genesis that goes past the tip says what a probe + // from the tip would. Empty is also a peer that sends nothing + // below the minimum work. + Some(_) if past_tip => {} + Some(_) => s.chain.note_lacks_top(), + None if headers.is_empty() => s.chain.note_ends_at(tip), + None => {} + } + } true } diff --git a/crates/rbitcoin-ibd/src/ibd/header_walk.rs b/crates/rbitcoin-ibd/src/ibd/header_walk.rs index 9ad69aa03..b831acfa4 100644 --- a/crates/rbitcoin-ibd/src/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/src/ibd/header_walk.rs @@ -113,6 +113,9 @@ pub struct HeaderWalk { /// Probe ask already sent. Its reply is read into that peer's chain /// record, not by the walk. pub probe_open: Option, + /// The path top the open probe asked headers after; none when it asked + /// from the confirmed tip. + pub probe_top: Option<(u32, BlockHash)>, /// The one in-flight headers ask. Not written to `header.adopt`. pub walk: Lane, /// Expired asks. Not written to `header.adopt`. @@ -554,12 +557,17 @@ pub fn note_block_inv(st: &mut IbdWorkState, peer: usize) { } } -/// Ask one block peer that has not announced a chain for headers, so block -/// getdata can follow what it holds. Core starts header sync with each -/// peer. One probe at a time, never the reserved header peer, and not a -/// peer with a walk ask open. The probe asks from the confirmed tip, so an -/// empty reply means nothing above it. -pub fn probe_unprobed_peer(st: &mut IbdWorkState) { +/// Heights above the confirmed tip a peer's record should decide before it +/// is probed again. +const PROBE_AHEAD: u32 = 4_096; + +/// Ask one block peer for headers, so block getdata follows what it holds: +/// one that has not announced a chain (Core starts header sync with each +/// peer), or one whose record no longer decides the next blocks. One probe +/// at a time, never the reserved header peer, and not a peer with a walk ask +/// open. +pub fn probe_unprobed_peer(st: &mut IbdWorkState, hub: &ChainHub) { + use super::peer_io::PeerSlot; let walk = &st.header_walk; // A record only chooses among peers. With one live peer it would only // take the header lane, and its reply would not reach the walk. @@ -569,10 +577,16 @@ pub fn probe_unprobed_peer(st: &mut IbdWorkState) { { return; } + let tip = hub.tip_height().unwrap_or(0); + let ahead = tip.saturating_add(PROBE_AHEAD).min(st.max_ordered_height); + let now = Instant::now(); + let due = |s: &PeerSlot| { + !s.chain.probed() || (ahead > tip && s.chain.needs_refresh(&st.height_to_hash, ahead, now)) + }; let Some(slot) = st .slots .iter() - .find(|s| s.alive && !walk_busy(walk, s.id) && !s.chain.probed()) + .find(|s| s.alive && !walk_busy(walk, s.id) && due(s)) else { return; }; @@ -632,6 +646,7 @@ pub fn forget_walk_peer(st: &mut IbdWorkState, peer: usize) { } if st.header_walk.probe_open == Some(peer) { st.header_walk.probe_open = None; + st.header_walk.probe_top = None; } } @@ -670,43 +685,36 @@ pub fn note_reservation( } } -/// Whether block `hash` may be asked of `peer`. A peer is not asked for a -/// block it answered `notfound`, or for a path block its announced chain -/// does not hold. The reserved header peer is asked only when no other live -/// peer may have it, or, once its own announced chain shows the block, when no -/// other peer's shows it. A single peer still does both. +/// Whether block `hash` may be asked of `peer`. As in Core, a peer is asked +/// only for a block its chain showed (`PeerChain::shows`), and not for one it +/// answered `notfound`. The reserved header peer is asked only when no other +/// peer shows the block. A single peer still does both. pub fn may_serve(st: &IbdWorkState, peer: usize, hash: &BlockHash) -> bool { use super::peer_io::PeerSlot; let height = st.hash_height.get(hash).copied(); let now = super::peer_io::ibd_mono_ms(); + let path = &st.height_to_hash; let askable = |s: &PeerSlot| s.alive && !s.chain.lacks(hash, now); - let holds = |s: &PeerSlot| { - askable(s) && !height.is_some_and(|ht| s.chain.left_path_by(&st.height_to_hash, ht)) - }; + let shows = |s: &PeerSlot| askable(s) && height.is_some_and(|ht| s.chain.shows(path, ht)); + let open = + |s: &PeerSlot| askable(s) && !height.is_some_and(|ht| s.chain.left_path_by(path, ht)); let Some(slot) = st.slots.iter().find(|s| s.id == peer) else { return false; }; - // A chain record can be stale. With no peer that holds the block by what - // it announced, any peer that did not answer `notfound` may be asked, - // the header peer too. - if !st.slots.iter().any(&holds) { + // With no peer that shows the block, a record can be stale: a peer whose + // chain has not left the path is asked, else any peer that did not answer + // `notfound`, the header peer too. Otherwise the header peer stays on + // headers while another peer can fetch the block. + let tier: &dyn Fn(&PeerSlot) -> bool = if st.slots.iter().any(&shows) { + &shows + } else if st.slots.iter().any(&open) { + &open + } else { return askable(slot); - } - if !holds(slot) { - return false; - } - if st.header_walk.reserved != Some(peer) { - return true; - } - // A header peer whose announced chain shows the block is kept from it only - // by another peer that shows it too. Otherwise any peer that may hold it - // does: the header peer stays on headers while others fetch blocks. - let shows = - |s: &PeerSlot| askable(s) && height.is_some_and(|ht| s.chain.shows(&st.height_to_hash, ht)); - let shown = shows(slot); - !st.slots - .iter() - .any(|s| s.id != peer && if shown { shows(s) } else { holds(s) }) + }; + tier(slot) + && (st.header_walk.reserved != Some(peer) + || !st.slots.iter().any(|s| s.id != peer && tier(s))) } /// Settle each peer's recorded split against this node's confirmed chain once @@ -2449,6 +2457,7 @@ pub fn expire_header_ask(st: &mut IbdWorkState) -> Option { }; if st.header_walk.probe_open == Some(peer) { st.header_walk.probe_open = None; + st.header_walk.probe_top = None; if n >= HEADER_MISS_DISCONNECT { st.header_walk.header_misses.remove(&peer); punish_header_peer(st, peer); @@ -2568,12 +2577,40 @@ pub fn arm_header_ask( st.header_walk.challenge_external = false; } if let Some(peer) = take_probe(st) { - // Only the confirmed chain: the first of its hashes the peer has is - // at or below the tip, so the reply is what it has above it. - if ask_header_peer(st, hub, peer, &[])? { + // After the path top alone: a reply that builds on it shows the + // whole path. A peer that did not show it is asked with the + // confirmed chain only, whose first hash it has is at or below the + // tip, so the reply is what it has above it. + let tip_h = hub.tip_height().unwrap_or(0); + let from_tip = st + .slots + .iter() + .any(|s| s.id == peer && s.chain.probe_from_tip()); + let top = path_top(st, hub).filter(|&(_, h)| h > tip_h && !from_tip); + let sent = match top { + Some((hash, height)) => { + let sent = super::dial::request_headers_from( + &st.slots, + peer, + &mut st.header_req_seq, + vec![hash], + )?; + if sent { + note_lane(st, peer); + } + st.header_walk.probe_top = sent.then_some((height, hash)); + sent + } + None => { + st.header_walk.probe_top = None; + ask_header_peer(st, hub, peer, &[])? + } + }; + if sent { st.header_walk.probe_open = Some(peer); + let now = Instant::now(); if let Some(slot) = st.slots.iter_mut().find(|s| s.id == peer) { - slot.chain.note_probed(); + slot.chain.note_probe_sent(now, top.is_none()); } return Ok(true); } @@ -3411,6 +3448,7 @@ pub fn parse_adopt(bytes: &[u8]) -> Option { challenge_open: None, probe: None, probe_open: None, + probe_top: None, walk: Lane::default(), header_misses: HashMap::new(), base_diff, diff --git a/crates/rbitcoin-ibd/src/ibd/mod.rs b/crates/rbitcoin-ibd/src/ibd/mod.rs index 950d1c689..022e3d789 100644 --- a/crates/rbitcoin-ibd/src/ibd/mod.rs +++ b/crates/rbitcoin-ibd/src/ibd/mod.rs @@ -703,7 +703,7 @@ pub async fn ibd_cancellable( if hub.header_inv_challenge_pending() && header_walk::note_external_block_inv(st) { hub.clear_header_inv_challenge(); } - header_walk::probe_unprobed_peer(st); + header_walk::probe_unprobed_peer(st, hub.as_ref()); let challenge_due = header_walk::inv_challenge_due(st); if cadence.headers_due(now_cadence, path_empty, challenge_due) { let live = st.ordered_set.len(); diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 1aba71a59..49f828a5c 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -8,6 +8,7 @@ use bitcoin::block::Header; use bitcoin::BlockHash; use std::collections::{HashMap, VecDeque}; +use std::time::{Duration, Instant}; /// `notfound` hashes kept per peer. Past the cap a peer may be asked again. const LACKS_CAP: usize = 1024; @@ -18,6 +19,10 @@ const LACKS_TTL_MS: u64 = 30_000; /// Announced hashes kept per peer: one full `headers` reply. const ANNOUNCED_CAP: usize = rbitcoin_net::codec::MAX_HEADERS_RESULTS; +/// How often a peer whose record no longer decides the next blocks is +/// probed again. +const PROBE_REFRESH: Duration = Duration::from_secs(10); + #[derive(Debug, Default)] pub struct PeerChain { /// `notfound` hashes, with the mono ms they were answered. @@ -36,8 +41,17 @@ pub struct PeerChain { /// That height has left the path and this node's confirmed block there /// is a different one. split_confirmed: bool, - /// The peer announced headers, or was asked for them once. + /// The highest path block this peer showed through a probe that asked + /// for headers after it. It holds every path block at or below it while + /// the path still holds that block there. + reaches: Option<(u32, BlockHash)>, + /// The last probe asked after the path top and the reply did not show + /// it, so the next probe asks from the confirmed tip. + probe_from_tip: bool, + /// The peer announced headers, or was asked for them. probed: bool, + /// When the last probe was sent. + probed_at: Option, } impl PeerChain { @@ -66,8 +80,41 @@ impl PeerChain { self.probed } - pub fn note_probed(&mut self) { + /// Whether the next probe asks from the confirmed tip. + pub fn probe_from_tip(&self) -> bool { + self.probe_from_tip + } + + /// A probe was sent `at`, from the tip or after the path top. + pub fn note_probe_sent(&mut self, at: Instant, from_tip: bool) { self.probed = true; + self.probed_at = Some(at); + if from_tip { + self.probe_from_tip = false; + } + } + + /// The peer's chain holds the path block `hash` at `height`. + pub fn note_reaches(&mut self, height: u32, hash: BlockHash) { + self.reaches = Some((height, hash)); + } + + /// The reply to a probe after the path top did not show it: ask from + /// the confirmed tip next. + pub fn note_lacks_top(&mut self) { + self.probe_from_tip = true; + self.probed = false; + } + + /// The record no longer says whether the peer holds the path block at + /// `height`, and no probe was sent within one refresh. + pub fn needs_refresh(&self, path: &HashMap, height: u32, now: Instant) -> bool { + self.probed + && self + .probed_at + .is_none_or(|at| now.saturating_duration_since(at) >= PROBE_REFRESH) + && !self.shows(path, height) + && !self.left_path_by(path, height) } /// An empty reply to a walk ask: the peer is at one of several locator @@ -226,6 +273,14 @@ impl PeerChain { if self.left_path_by(path, height) { return false; } + // A path block the peer holds, from a probe or from where its chain + // left this one before this node took its chain. + let holds = |at: Option<(u32, BlockHash)>| { + at.is_some_and(|(h, hash)| height <= h && path.get(&h) == Some(&hash)) + }; + if holds(self.reaches) || holds(self.split) { + return true; + } let Some(len) = (self.announced.len() as u32).checked_sub(1) else { return false; }; diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 9692b3243..1f161ee3c 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -1658,7 +1658,7 @@ fn two_peers_reserve_the_header_walk() { other.first_data_ms = 5_000; short.first_data_ms = 6_000; unknown.first_data_ms = 7_000; - unknown.chain.note_probed(); + unknown.chain.note_probe_sent(Instant::now(), true); let mut st = IbdWorkState::new(vec![head, other, short, unknown], Some(gen), Some(0)); assert!(send_getheaders(&mut st, &hub).unwrap()); assert_eq!(drain_getheaders(&mut head_rx), 1); @@ -1666,7 +1666,7 @@ fn two_peers_reserve_the_header_walk() { apply(&mut st, &hub, 5, path.to_vec()); assert_eq!(st.header_walk.reserved, Some(5)); let _ = take_header_ask(&mut st, 5); - probe_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st, &hub); assert!(send_getheaders(&mut st, &hub).unwrap()); assert_eq!( drain_getheaders(&mut other_rx), @@ -1678,7 +1678,7 @@ fn two_peers_reserve_the_header_walk() { apply(&mut st, &hub, 6, vec![fork, mine(fork.block_hash(), 46)]); // A short reply is everything that peer has: its chain ends at height 1. apply(&mut st, &hub, 7, vec![path[0]]); - probe_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st, &hub); assert!(st.header_walk.probe.is_none(), "a peer is probed once"); for h in &path { st.body.mark_missing(h.block_hash()); @@ -1807,25 +1807,31 @@ fn two_peers_reserve_the_header_walk() { ); assert!(may_serve(&st, 8, &taken2.block_hash())); - // A sixth, on its own hub, because the probe must say what a peer has - // above the tip: it asks from the confirmed tip, so an empty reply is a - // chain that ends there and that peer is not asked for path blocks. A - // caught-up peer sends the path and is. The probe is not a walk answer, - // and a reply with a chain the walk has not seen gets the walk's own ask. - // An empty reply to a walk ask is a caught-up peer as well as one behind, - // so that peer is probed again. + // A sixth, on its own hub, because a probe must say what a peer has. It + // asks after the path top alone, and a reply on that top shows the whole + // path. A peer that does not show it is asked from the confirmed tip, + // where an empty reply is a chain that ends there. The probe is not a + // walk answer, and a reply with a header the walk has not seen gets the + // walk's own ask. An empty reply to a walk ask is a caught-up peer as well + // as one behind, so that peer is probed again. let (_dir6, hub6) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-probe"); hub6.ensure_genesis().unwrap(); let gen6 = hub6.tip_hash().unwrap(); let (mut head, mut head_rx) = slot(13); let (mut caught, mut caught_rx) = slot(14); let (mut behind, mut behind_rx) = slot(15); + let (mut fallback, mut fallback_rx) = slot(26); let (mut side, mut side_rx) = slot(16); head.first_data_ms = 50; caught.first_data_ms = 5_000; behind.first_data_ms = 6_000; + fallback.first_data_ms = 6_500; side.first_data_ms = 7_000; - let mut st = IbdWorkState::new(vec![head, caught, behind, side], Some(gen6), Some(0)); + let mut st = IbdWorkState::new( + vec![head, caught, behind, fallback, side], + Some(gen6), + Some(0), + ); assert!(send_getheaders(&mut st, &hub6).unwrap()); assert_eq!(drain_getheaders(&mut head_rx), 1); let path = [mine(gen6, 51), mine(mine(gen6, 51).block_hash(), 52)]; @@ -1833,48 +1839,82 @@ fn two_peers_reserve_the_header_walk() { assert_eq!(st.header_walk.reserved, Some(13)); let _ = take_header_ask(&mut st, 13); - probe_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st, &hub6); assert!(send_getheaders(&mut st, &hub6).unwrap()); let locator = getheaders_locator(&mut caught_rx).expect("the caught-up peer is probed"); assert_eq!( - locator.first(), - Some(&gen6), - "the probe asks from the confirmed tip" + locator, + vec![path[1].block_hash()], + "the probe asks after the path top alone" ); - assert!(!locator.contains(&path[1].block_hash())); // An announcement that crosses the probe is not its reply. - let next = mine(path[1].block_hash(), 53); - apply(&mut st, &hub6, 14, vec![next]); + let stray = mine(BlockHash::from_byte_array([0x77; 32]), 53); + apply(&mut st, &hub6, 14, vec![stray]); assert_eq!( st.header_walk.walk.peer, Some(14), "the probe ask stays open, so it can still expire" ); - apply(&mut st, &hub6, 14, vec![path[0], path[1], next]); + // That stray batch went to the walk, which retired the peer from it. + st.header_walk.walk_quiet.remove(&14); + // At the top the peer has nothing after it, so it is asked again from + // the tip. + apply(&mut st, &hub6, 14, Vec::new()); assert_eq!(st.header_walk.probe_open, None); assert!( !st.header_walk.walk_quiet.contains(&14), "a probe reply is not a walk answer" ); assert_eq!(st.header_walk.reserved, Some(13)); + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + let locator = getheaders_locator(&mut caught_rx).expect("the peer is asked again"); + assert_eq!(locator.first(), Some(&gen6), "then from the confirmed tip"); + assert!( + !st.slots[1].chain.probe_from_tip(), + "and the probe after that asks after the top again" + ); + apply(&mut st, &hub6, 14, path.to_vec()); + assert!(may_serve(&st, 14, &path[1].block_hash())); - probe_unprobed_peer(&mut st); + // A peer without the top answers from genesis: this node sends genesis + // itself, Core the header after it. One that stops at or below the tip + // is asked again from the tip, where an empty reply is a chain that ends + // there. + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert_eq!(getheaders_locator(&mut behind_rx).map(|l| l.len()), Some(1)); + apply(&mut st, &hub6, 15, vec![hub6.header_of(&gen6).unwrap()]); + assert_eq!(st.header_walk.probe_open, None); + assert!(st.slots[2].chain.probe_from_tip()); + probe_unprobed_peer(&mut st, &hub6); assert!(send_getheaders(&mut st, &hub6).unwrap()); assert!(getheaders_locator(&mut behind_rx).is_some()); apply(&mut st, &hub6, 15, Vec::new()); assert!( - !may_serve(&st, 15, &path[0].block_hash()), - "a peer with nothing above the tip is not asked for a path block" + st.slots[2].chain.left_path_by(&st.height_to_hash, 1), + "a peer with nothing above the tip does not hold a path block" ); - assert!(may_serve(&st, 14, &path[1].block_hash())); + probe_unprobed_peer(&mut st, &hub6); + assert!(send_getheaders(&mut st, &hub6).unwrap()); + assert!(getheaders_locator(&mut fallback_rx).is_some()); + // A reply from genesis that already goes past the tip says what a probe + // from the tip would. + apply(&mut st, &hub6, 26, vec![mine(gen6, 55)]); + assert_eq!(st.header_walk.probe_open, None); + assert!(!st.slots[3].chain.probe_from_tip()); + assert!(st.slots[3].chain.left_path_by(&st.height_to_hash, 1)); + st.header_walk.challenge = None; - // A probe reply that leaves the path is a chain the walk has not seen, - // so that peer gets the walk's own ask. - probe_unprobed_peer(&mut st); + // A reply on the path top shows the path below it. Its new header is one + // the walk has not seen, so that peer gets the walk's own ask. + probe_unprobed_peer(&mut st, &hub6); assert!(send_getheaders(&mut st, &hub6).unwrap()); assert!(getheaders_locator(&mut side_rx).is_some()); - apply(&mut st, &hub6, 16, vec![mine(gen6, 54)]); + apply(&mut st, &hub6, 16, vec![mine(path[1].block_hash(), 54)]); + assert!(st.slots[4].chain.shows(&st.height_to_hash, 1)); assert_eq!(st.header_walk.challenge, Some(16)); + st.header_walk.challenge = None; // A block inv's ask carries the path top, so an empty reply is a peer at // that top as well as one behind it. @@ -1896,7 +1936,7 @@ fn two_peers_reserve_the_header_walk() { st.header_walk.walk.peer = None; st.header_walk.walk.at = None; st.header_walk.walk.prev_peer = Some(14); - probe_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st, &hub6); assert_ne!(st.header_walk.probe, Some(14)); st.header_walk.probe = Some(14); assert!(send_getheaders(&mut st, &hub6).unwrap()); @@ -1907,7 +1947,7 @@ fn two_peers_reserve_the_header_walk() { // after another peer's expiry took the lane's last slot for it. st.header_walk.walk.prev_peer = None; st.header_walk.header_misses.insert(14, 1); - probe_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st, &hub6); assert_ne!(st.header_walk.probe, Some(14)); st.header_walk.header_misses.clear(); @@ -1918,7 +1958,7 @@ fn two_peers_reserve_the_header_walk() { } st.header_walk.walk.peer = None; st.header_walk.walk.at = None; - probe_unprobed_peer(&mut st); + probe_unprobed_peer(&mut st, &hub6); assert_eq!(st.header_walk.probe, Some(14)); assert!(send_getheaders(&mut st, &hub6).unwrap()); assert_eq!(st.header_walk.probe_open, Some(14)); @@ -2000,12 +2040,90 @@ fn two_peers_reserve_the_header_walk() { !may_serve(&st, 17, &w1.block_hash()), "a peer on the rewound chain is not asked for the new one" ); - assert!(may_serve(&st, 18, &w1.block_hash())); - assert!(may_serve(&st, 19, &w2.block_hash())); + assert!( + !st.slots[1].chain.left_path_by(&st.height_to_hash, 1), + "a peer with no record gets no split at the rewind" + ); + assert!(!may_serve(&st, 18, &w1.block_hash())); + assert!( + may_serve(&st, 19, &w1.block_hash()), + "a peer whose split is the new chain's block shows it" + ); assert!( may_serve(&st, 20, &w3.block_hash()), "a peer that announced the new chain is asked for it" ); + + // An eighth, because a block is asked only of a peer whose chain showed + // it: a peer whose full reply ended below the block, or one with no + // record, is not asked for it while another peer shows it. With no peer + // that shows a block, a peer that has not left the path is asked first. + let (_dir8, hub8) = rbitcoin_net::chain::tiny_regtest_hub_labeled("header-walk-shown"); + hub8.ensure_genesis().unwrap(); + let gen8 = hub8.tip_hash().unwrap(); + let (head, _head_rx) = slot(22); + let (full, _full_rx) = slot(23); + let (blank, _blank_rx) = slot(24); + let (left, _left_rx) = slot(25); + let mut st = IbdWorkState::new(vec![head, full, blank, left], Some(gen8), Some(0)); + let mut prev = gen8; + let mut chain8 = Vec::new(); + for n in 0..2_004u32 { + let h = mine(prev, 90_000 + n); + prev = h.block_hash(); + chain8.push(h); + } + for (i, h) in chain8.iter().enumerate() { + st.record_height(h.block_hash(), i as u32 + 1); + st.height_to_hash.insert(i as u32 + 1, h.block_hash()); + } + st.header_walk.reserved = Some(22); + let none = |_| None; + let path8 = st.height_to_hash.clone(); + st.slots[0] + .chain + .note_headers(&path8, none, 0, 1, &chain8[..2_003]); + st.slots[1] + .chain + .note_headers(&path8, none, 0, 1, &chain8[..2_000]); + let fork = mine(chain8[0].block_hash(), 99_999); + st.slots[3].chain.note_headers(&path8, none, 0, 2, &[fork]); + let at = |h: u32| chain8[h as usize - 1].block_hash(); + assert!(may_serve(&st, 23, &at(100))); + assert!( + !may_serve(&st, 22, &at(100)), + "the header peer stays on headers while another peer shows the block" + ); + assert!( + !may_serve(&st, 23, &at(2_003)), + "a full reply that ended below a block does not show it" + ); + assert!( + !may_serve(&st, 24, &at(2_003)), + "a peer with no record is not asked for a block another peer shows" + ); + assert!( + may_serve(&st, 22, &at(2_003)), + "the header peer shows it alone" + ); + assert!(may_serve(&st, 24, &at(2_004))); + assert!( + !may_serve(&st, 25, &at(2_004)), + "with nobody showing a block, a peer that left the path is not asked" + ); + // A peer whose record stops short of the next blocks is probed again; one + // probed just now is not. + st.max_ordered_height = 2_004; + probe_unprobed_peer(&mut st, &hub8); + assert_eq!(st.header_walk.probe, Some(23)); + st.header_walk.probe = None; + st.slots[1].chain.note_probe_sent(Instant::now(), true); + probe_unprobed_peer(&mut st, &hub8); + assert_eq!( + st.header_walk.probe, + Some(24), + "the next is the peer with no record" + ); } /// A long chain on an invalid block stays refused past the cap on diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs index 68b35e760..dba220943 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -163,6 +163,38 @@ fn a_reply_on_this_chain_above_a_split_clears_it() { assert!(moved.left_path_by(&path, 2)); } +/// A peer whose record does not decide a height is probed again, but not +/// within 10 s of its last probe. +#[test] +fn a_record_that_stops_short_is_refreshed() { + use std::time::{Duration, Instant}; + let headers = chain(20, 0x5c); + let path: HashMap = (1..).zip(headers.iter().map(|h| h.block_hash())).collect(); + let Some(sent) = Instant::now().checked_sub(Duration::from_secs(30)) else { + return; + }; + let mut ended = PeerChain::default(); + ended.note_probe_sent(sent, true); + ended.note_headers(&path, |_| None, 0, 1, &headers[..10]); + assert!( + !ended.needs_refresh(&path, 5, Instant::now()), + "a height the record shows is decided" + ); + assert!( + !ended.needs_refresh(&path, 15, Instant::now()), + "so is one above where a short reply ended" + ); + let mut reached = PeerChain::default(); + reached.note_probe_sent(sent, false); + reached.note_reaches(10, headers[9].block_hash()); + assert!(!reached.needs_refresh(&path, 5, sent + Duration::from_secs(10))); + assert!(reached.needs_refresh(&path, 15, sent + Duration::from_secs(10))); + assert!( + !reached.needs_refresh(&path, 15, sent + Duration::from_millis(9_999)), + "not within 10 s of the last probe" + ); +} + /// A `notfound` keeps a peer off that block for one stall window, then the /// peer may be asked again. #[test] diff --git a/docs/ibd-memory.md b/docs/ibd-memory.md index 984471d53..3aaf61f7d 100644 --- a/docs/ibd-memory.md +++ b/docs/ibd-memory.md @@ -56,8 +56,9 @@ pres and **not** the raw bytes. Reorg gather that wants wire re-encodes. | **SH catalog runs** | leftover `scripthash.runs` discarded at tip (unsorted collect does not write them) | write-behind / discard; not during Direct confirm | | **SH unsorted collect / pack** | Two `txout` scans. All extract phases share `sh_extract_workers()` = min(CPUs, max(1, free RAM / 1.5 GiB)); `RBITCOIN_SH_MERGE_WORKERS` override. **Collect maps** use that same 1.5 GiB **per worker** as a cap (not a prefault): each worker owns a contiguous create-fk span and `n_shards` unsized identity maps (`HashMap`, `0` = multi; 64 B/key estimate). After each 64 k-fk loc/body batch, spill the largest shard map while `sum(estimate) >= 1.5 GiB`. Status `scanned=` is finished fks (not max fk). Spills, including the end flush, go through one writer and a 1-slot queue so encode+tmp+rename is serial. Fuse8 mmap on pass 2 is process-wide, not subtracted from the cap. `keys/NN/` is a **directory** of `SHKSP01` spill files (`000000`, …; n_multi key16s already `0`, then first-fk-sorted uleb(delta)‖key16 singles; tmp+rename, no `sync_all`). Merge folds those spill files only into one identity map, one walk to `scripthash.head/NN` pack8 and `multi/NN.fuse8` (file, not kept resident for other shards), unlinks `keys/NN/`. The folded map is consumed into the pack8 records and dropped before fuse8 and BDZ. A previous `DONE` / 24 B `NN` layout, or `keys/NN` as a file, with no valid `DONE.keys` is deleted and pass 1 restarts. A spill whose magic is not `SHKSP01` is Corrupt. Pass 2 keeps fuse8 mmap only (no BDZ). Same static fk spans; fuse-hit creates append `fk` to per-worker `HashMap>` (skip `vec.last() == fk`). Estimate `80 × n_keys + 8 × n_fks`; spill-largest while over 1.5 GiB. `post/NN/` is a **directory** of `SHPST01` spill files (`000000`, …; `u32 n_keys` then `key16 ‖ uleb(n) ‖ n × uleb(fk − prev)`; tmp+rename, no `sync_all`). A `post/NN` file, or a spill whose magic is not `SHPST01`, is Corrupt. **Pack fold** (named, after collect maps drop): one shard ~1.2 M multi keys × ~30 fks ≈ 0.3–0.5 GiB; 8 pack workers a few GiB. Fold spills into one map, then `MphfHead::open` + `slot_for_key16` + 2+ body + `rewrite_val_slots`; `len == 1` after fold is `fp_singles` (pass-1 `inline_one` stays). Unlink `post/NN/`. Packed 2-bit `g` ~8 MiB per pack worker + 16 MiB `body_buf`. Inner loc/body batch 65 536 fks inside each worker span (not a steal grain); body IO 16 MiB | Tip finalize. Extract under `scripthash.unsorted/{keys,multi,post}` | | **`tx.head` wipe-rebuild workers** | min(CPUs, host free RAM / 1 GiB, range count); floor 1. Same free-RAM probe as SH. Matches BDZ peel+keys+g peak at default 2²⁵. **Not** the SH extract 1.5 GiB cap. Env `RBITCOIN_TX_HEAD_REBUILD_WORKERS` override (`1` = serial) | Empty/wipe `tx.head` rebuild from `txid.body`. Logs `workers=` `free_GiB=` | +| **Per-peer chain record** | `PeerChain` per IBD slot: the newest announced run above the confirmed tip, at most 2,000 hashes; at most 1,024 `notfound` hashes, each kept 30 s; one split, one chain end, and one path block the peer showed after a probe. Header-walk `invalid_tips` (tips of chains refused for building on a rejected block): 1,024, oldest first out | Dropped with the slot or IBD state. Not written to disk | | **Ordered work path** | `MAX_ORDERED_HEADERS` | `IbdWorkState::hygiene` | -| **Header-walk checkpoints** | One checkpoint per accepted 2,000-header reply, and those headers are written. The ordered queue is a window of at most 64,000 over headers already stored. One peer, the lowest time to a first block byte among peers that have not failed the walk, is asked only when the next header is not in the store. While the queue has room the locator starts at the stored tip, then the candidate, then a lighter fork still being weighed, so the reply can be enqueued. A full queue asks from the candidate. Block download uses the other peers. Above the work floor a short reply that does not extend the candidate moves the reservation; a full 2,000-header window does not, so a lighter fork can still catch up. A block inv from another peer is one challenge, asked even when that peer's connect-time start height is below the walk: if the reply beats the candidate they become the header peer, and if it does not they leave the walk and the previous peer stays. An inv from a session with no header slot does not retire the reserved peer. One missed ask moves the reservation; a second miss disconnects. Less work does not disconnect. Each checkpoint holds hash, height, work, that reply's last 80-byte header, up to 11 timestamps, and the difficulty period (one period-start header plus the last non-limit `nBits`) so a fork or rewind can retarget. One competing chain is kept the same way until it loses or replaces the candidate. Mainnet stays under about 250 KB. Queue maps stay capped with the queue. Side headers below the work floor are one hash per peer. More than 4,000 disconnects that peer. Abandoned look-ahead tips are cleared at 1,024 and when a heavier chain is adopted. | Dropped with IBD state. `header.adopt` is the sidecar, not a resident copy. The file is deleted once the confirmed chain proves the anchor | +| **Header-walk checkpoints** | One checkpoint per accepted 2,000-header reply, and those headers are written. The ordered queue is a window of at most 64,000 over headers already stored. One peer, the lowest time to a first block byte among peers that have not failed the walk, is asked only when the next header is not in the store. While the queue has room the locator starts at the stored tip, then the candidate, then a lighter fork still being weighed, so the reply can be enqueued. A full queue asks from the candidate. Block download uses the other peers, each asked only for blocks its chain showed (see **Per-peer chain record**); the header peer takes a block no other peer shows. Above the work floor a short reply that does not extend the candidate moves the reservation; a full 2,000-header window does not, so a lighter fork can still catch up. A block inv from another peer is one challenge, asked even when that peer's connect-time start height is below the walk: if the reply beats the candidate they become the header peer, and if it does not they leave the walk and the previous peer stays. An inv from a session with no header slot does not retire the reserved peer. One missed ask moves the reservation; a second miss disconnects. Less work does not disconnect. Each checkpoint holds hash, height, work, that reply's last 80-byte header, up to 11 timestamps, and the difficulty period (one period-start header plus the last non-limit `nBits`) so a fork or rewind can retarget. One competing chain is kept the same way until it loses or replaces the candidate. Mainnet stays under about 250 KB. Queue maps stay capped with the queue. Side headers below the work floor are one hash per peer. More than 4,000 disconnects that peer. Abandoned look-ahead tips are cleared at 1,024 and when a heavier chain is adopted. | Dropped with IBD state. `header.adopt` is the sidecar, not a resident copy. The file is deleted once the confirmed chain proves the anchor | Tests that need a clean process must call these **same** entry points (or drop the owning `Query` / pipeline), not a secret test-only free-all that masks production From 7be776ad00d6d9b510a7eeeda26e3ff20513e68c Mon Sep 17 00:00:00 2001 From: Nitesh Balusu Date: Wed, 21 Oct 2026 13:00:49 -0400 Subject: [PATCH 17/17] net: re-probe a peer whose record a rewind left stale Review of d00f2c50: at a rewind, a record that took the rewound blocks as the peer's chain was given an inferred split at the fork, in place of a split the peer had shown. A peer on a third chain that left this chain at 1999, then a rewind to 1997 and one back, looked eligible for blocks above 1999 it did not hold. Further review found that any inferred block can be lost or contradicted across later rewinds and replies. A rewind no longer infers a block. Such a record is stale from the fork up until the peer answers again: the peer is not taken to hold a block there, and it is probed at once. Only a reply that reaches the fork clears it; a full reply that ends below the fork says nothing there, so the peer is asked again. A later rewind keeps the lower fork. The peer's own split is kept as it was. A record whose split is at or below the fork, or whose announced headers are on the new chain, is not stale. Co-Authored-By: Claude Opus 5.5 --- crates/rbitcoin-ibd/src/ibd/peer_chain.rs | 48 +++++++++----- crates/rbitcoin-ibd/src/ibd/reorg.rs | 14 ++--- crates/rbitcoin-ibd/tests/ibd/header_walk.rs | 3 +- crates/rbitcoin-ibd/tests/ibd/peer_chain.rs | 66 +++++++++++++++++++- docs/ibd-memory.md | 2 +- 5 files changed, 103 insertions(+), 30 deletions(-) diff --git a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs index 49f828a5c..4eb6bff69 100644 --- a/crates/rbitcoin-ibd/src/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/src/ibd/peer_chain.rs @@ -41,6 +41,10 @@ pub struct PeerChain { /// That height has left the path and this node's confirmed block there /// is a different one. split_confirmed: bool, + /// A rewind replaced this node's chain from this height, and the record + /// took the old blocks as the peer's chain. The peer is not taken to hold + /// a block at or above it until it answers again. + stale_from: Option, /// The highest path block this peer showed through a probe that asked /// for headers after it. It holds every path block at or below it while /// the path still holds that block there. @@ -185,7 +189,17 @@ impl PeerChain { break; } let short = headers.len() < ANNOUNCED_CAP; - self.ends = short.then(|| height.saturating_add(hashes.len() as u32).saturating_sub(1)); + let top = height.saturating_add(hashes.len() as u32).saturating_sub(1); + // A stale record is answered by a reply that reaches its fork. One + // that ends below the fork says nothing there: ask again. + if let Some(stale) = self.stale_from { + if top >= stale { + self.stale_from = None; + } else { + self.probed = false; + } + } + self.ends = short.then_some(top); let drop = tip.saturating_add(1).saturating_sub(self.from) as usize; self.announced.drain(..drop.min(self.announced.len())); self.from = self.from.max(tip.saturating_add(1)); @@ -222,18 +236,15 @@ impl PeerChain { self.from = self.from.saturating_add(over as u32); } - /// This node's confirmed chain above `lca` was rewound away, `left` was - /// its block at `lca + 1`, and `path` is the chain planted in its place. - /// A record that starts past that height took the old blocks as the - /// peer's chain, so the peer split from the new one there. A split the - /// peer showed at or below that height already says where it left, and - /// an announced hash on the new path says the peer is on it. - pub fn note_rewind(&mut self, path: &HashMap, lca: u32, left: BlockHash) { + /// This node's confirmed chain above `lca` was rewound away, and `path` + /// is the chain planted in its place. A record that starts past the fork + /// took the old blocks as the peer's chain, so it is stale from the fork + /// up until the peer answers again, and the peer is probed. A split the + /// peer showed at or below the fork already says where it left, and an + /// announced hash on the new path says it is on the new chain. + pub fn note_rewind(&mut self, path: &HashMap, lca: u32) { let at = lca.saturating_add(1); - if self.from <= at { - return; - } - if self.split.is_some_and(|(h, _)| h <= at) { + if self.from <= at || self.split.is_some_and(|(h, _)| h <= at) { return; } if (self.from..) @@ -242,8 +253,8 @@ impl PeerChain { { return; } - self.split = Some((at, left)); - self.split_confirmed = false; + self.stale_from = Some(self.stale_from.map_or(at, |s| s.min(at))); + self.probed = false; } /// The recorded split's height has left the path: compare it with this @@ -293,10 +304,13 @@ impl PeerChain { } /// The peer does not have the path block at `height`: its chain left this - /// node's chain at or below it, it holds a different block there, or its - /// whole chain ends below it. Two chains that differ at one height differ - /// above it. + /// node's chain at or below it, it holds a different block there, its + /// whole chain ends below it, or a rewind left its record stale there. Two + /// chains that differ at one height differ above it. pub fn left_path_by(&self, path: &HashMap, height: u32) -> bool { + if self.stale_from.is_some_and(|from| height >= from) { + return true; + } if let Some((at, hash)) = self.split { let differs = match path.get(&at) { Some(p) => *p != hash, diff --git a/crates/rbitcoin-ibd/src/ibd/reorg.rs b/crates/rbitcoin-ibd/src/ibd/reorg.rs index b798449f8..d341f9a49 100644 --- a/crates/rbitcoin-ibd/src/ibd/reorg.rs +++ b/crates/rbitcoin-ibd/src/ibd/reorg.rs @@ -547,8 +547,8 @@ pub struct RewindPrep { pub archived: Vec, pub connected: Vec, pub fks: Vec>, - /// The confirmed block above the LCA before the disconnect. - pub left: Option, + /// The confirmed tip was above the LCA, so blocks were disconnected. + pub rewound: bool, } /// Store side of a rewind: parent lookup and the confirmed-tip disconnect. @@ -573,9 +573,7 @@ pub fn prepare_header_rewind( return Ok(None); }; let tip_h = hub.tip_height().unwrap_or(0); - let left = (tip_h > lca_h) - .then(|| super::peer_chain::confirmed_at(hub, lca_h.saturating_add(1))) - .flatten(); + let rewound = tip_h > lca_h; if tip_h.saturating_sub(lca_h) > REWIND_MAX_DEPTH { warn!( "ibd: heavier fork rewind depth {} exceeds cap {REWIND_MAX_DEPTH} (lca={lca_h} tip={tip_h})", @@ -622,7 +620,7 @@ pub fn prepare_header_rewind( archived, connected, fks, - left, + rewound, })) } @@ -656,9 +654,9 @@ pub fn commit_header_rewind( } prev = *hash; } - if let Some(left) = prep.left { + if prep.rewound { for slot in &mut st.slots { - slot.chain.note_rewind(&st.height_to_hash, prep.lca_h, left); + slot.chain.note_rewind(&st.height_to_hash, prep.lca_h); } } for h in hub.query.block_queue_queued_heights() { diff --git a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs index 1f161ee3c..a2dd84bbd 100644 --- a/crates/rbitcoin-ibd/tests/ibd/header_walk.rs +++ b/crates/rbitcoin-ibd/tests/ibd/header_walk.rs @@ -2040,9 +2040,10 @@ fn two_peers_reserve_the_header_walk() { !may_serve(&st, 17, &w1.block_hash()), "a peer on the rewound chain is not asked for the new one" ); + assert!(!st.slots[0].chain.probed(), "and is probed again"); assert!( !st.slots[1].chain.left_path_by(&st.height_to_hash, 1), - "a peer with no record gets no split at the rewind" + "a peer with no record is not left stale by the rewind" ); assert!(!may_serve(&st, 18, &w1.block_hash())); assert!( diff --git a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs index dba220943..a23a403e6 100644 --- a/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs +++ b/crates/rbitcoin-ibd/tests/ibd/peer_chain.rs @@ -5,7 +5,11 @@ use rbitcoin_ibd::ibd::peer_chain::PeerChain; use std::collections::HashMap; fn chain(n: u32, salt: u8) -> Vec
{ - let mut prev = BlockHash::from_byte_array([0u8; 32]); + extend(BlockHash::from_byte_array([0u8; 32]), n, salt) +} + +fn extend(from: BlockHash, n: u32, salt: u8) -> Vec
{ + let mut prev = from; (1..=n) .map(|i| { let mut h = Header { @@ -119,15 +123,71 @@ fn a_third_chain_split_survives_two_rewinds() { let confirmed = |h: u32| Some(ours[h as usize - 1].block_hash()); let mut peer = PeerChain::default(); peer.note_headers(&HashMap::new(), confirmed, 2, 1, &third); - peer.note_rewind(&path(&won), 0, ours[0].block_hash()); + peer.note_rewind(&path(&won), 0); assert!(peer.left_path_by(&path(&won), 1)); - peer.note_rewind(&path(&ours), 0, won[0].block_hash()); + peer.note_rewind(&path(&ours), 0); assert!( peer.left_path_by(&path(&ours), 1), "a peer on a third chain is not taken back onto this one" ); } +/// A rewind leaves a record that took the rewound blocks as the peer's chain +/// stale from the fork up, also when the peer's own split is above the fork, +/// across further rewinds, until the peer answers again. It is probed again. +#[test] +fn a_rewind_leaves_a_record_stale_until_the_peer_answers() { + let ours = chain(2_003, 0x5c); + let won = extend(ours[1_996].block_hash(), 3, 0x77); + let third = extend(ours[1_997].block_hash(), 3, 0x91); + let at = |c: &[Header], from: u32| -> HashMap { + (from..).zip(c.iter().map(|h| h.block_hash())).collect() + }; + let confirmed = |h: u32| (h <= 2_001).then(|| ours[h as usize - 1].block_hash()); + // A full reply that leaves this chain at 1999, above the fork at 1998. + let mut reply = ours[1..1_998].to_vec(); + reply.extend_from_slice(&third); + let mut peer = PeerChain::default(); + peer.note_headers(&HashMap::new(), confirmed, 2_001, 2, &reply); + let on_won = at(&won, 1_998); + peer.note_rewind(&on_won, 1_997); + assert!(peer.left_path_by(&on_won, 1_998)); + assert!(!peer.probed(), "the peer is probed again"); + let on_ours = at(&ours[1_997..], 1_998); + peer.note_rewind(&on_ours, 1_997); + assert!( + peer.left_path_by(&on_ours, 2_003), + "it left this chain above the fork" + ); + // A deeper rewind keeps it off from the lower fork. + let deeper = at(&extend(ours[1_988].block_hash(), 2, 0x33), 1_990); + peer.note_rewind(&deeper, 1_989); + assert!(peer.left_path_by(&deeper, 1_990)); + // Its answer is the record again. + peer.note_headers(&on_ours, |_| None, 1_997, 1_998, &ours[1_997..2_000]); + assert!(!peer.left_path_by(&on_ours, 1_999)); + assert!( + peer.left_path_by(&on_ours, 2_001), + "the reply ended at 2000" + ); + + // A full reply that ends below the fork says nothing there: the record + // stays stale and the peer is probed again. + let mut far = PeerChain::default(); + let confirmed_far = |h: u32| (h <= 2_002).then(|| ours[h as usize - 1].block_hash()); + far.note_headers(&HashMap::new(), confirmed_far, 2_002, 2_003, &ours[2_002..]); + let side = at(&extend(ours[2_000].block_hash(), 2, 0x44), 2_002); + far.note_rewind(&side, 2_001); + far.note_headers(&side, confirmed_far, 2_001, 1, &ours[..2_000]); + assert!(far.left_path_by(&side, 2_002)); + assert!(!far.probed()); + // A full reply that reaches the fork on the new chain answers it. + let mut onto = ours[3..2_001].to_vec(); + onto.extend(extend(ours[2_000].block_hash(), 2, 0x44)); + far.note_headers(&side, confirmed_far, 2_001, 4, &onto); + assert!(!far.left_path_by(&side, 2_002)); +} + /// A peer that showed a stale sibling and then builds on this node's block /// at or above that height has joined this chain: the split is gone. #[test] diff --git a/docs/ibd-memory.md b/docs/ibd-memory.md index 3aaf61f7d..216c389f6 100644 --- a/docs/ibd-memory.md +++ b/docs/ibd-memory.md @@ -56,7 +56,7 @@ pres and **not** the raw bytes. Reorg gather that wants wire re-encodes. | **SH catalog runs** | leftover `scripthash.runs` discarded at tip (unsorted collect does not write them) | write-behind / discard; not during Direct confirm | | **SH unsorted collect / pack** | Two `txout` scans. All extract phases share `sh_extract_workers()` = min(CPUs, max(1, free RAM / 1.5 GiB)); `RBITCOIN_SH_MERGE_WORKERS` override. **Collect maps** use that same 1.5 GiB **per worker** as a cap (not a prefault): each worker owns a contiguous create-fk span and `n_shards` unsized identity maps (`HashMap`, `0` = multi; 64 B/key estimate). After each 64 k-fk loc/body batch, spill the largest shard map while `sum(estimate) >= 1.5 GiB`. Status `scanned=` is finished fks (not max fk). Spills, including the end flush, go through one writer and a 1-slot queue so encode+tmp+rename is serial. Fuse8 mmap on pass 2 is process-wide, not subtracted from the cap. `keys/NN/` is a **directory** of `SHKSP01` spill files (`000000`, …; n_multi key16s already `0`, then first-fk-sorted uleb(delta)‖key16 singles; tmp+rename, no `sync_all`). Merge folds those spill files only into one identity map, one walk to `scripthash.head/NN` pack8 and `multi/NN.fuse8` (file, not kept resident for other shards), unlinks `keys/NN/`. The folded map is consumed into the pack8 records and dropped before fuse8 and BDZ. A previous `DONE` / 24 B `NN` layout, or `keys/NN` as a file, with no valid `DONE.keys` is deleted and pass 1 restarts. A spill whose magic is not `SHKSP01` is Corrupt. Pass 2 keeps fuse8 mmap only (no BDZ). Same static fk spans; fuse-hit creates append `fk` to per-worker `HashMap>` (skip `vec.last() == fk`). Estimate `80 × n_keys + 8 × n_fks`; spill-largest while over 1.5 GiB. `post/NN/` is a **directory** of `SHPST01` spill files (`000000`, …; `u32 n_keys` then `key16 ‖ uleb(n) ‖ n × uleb(fk − prev)`; tmp+rename, no `sync_all`). A `post/NN` file, or a spill whose magic is not `SHPST01`, is Corrupt. **Pack fold** (named, after collect maps drop): one shard ~1.2 M multi keys × ~30 fks ≈ 0.3–0.5 GiB; 8 pack workers a few GiB. Fold spills into one map, then `MphfHead::open` + `slot_for_key16` + 2+ body + `rewrite_val_slots`; `len == 1` after fold is `fp_singles` (pass-1 `inline_one` stays). Unlink `post/NN/`. Packed 2-bit `g` ~8 MiB per pack worker + 16 MiB `body_buf`. Inner loc/body batch 65 536 fks inside each worker span (not a steal grain); body IO 16 MiB | Tip finalize. Extract under `scripthash.unsorted/{keys,multi,post}` | | **`tx.head` wipe-rebuild workers** | min(CPUs, host free RAM / 1 GiB, range count); floor 1. Same free-RAM probe as SH. Matches BDZ peel+keys+g peak at default 2²⁵. **Not** the SH extract 1.5 GiB cap. Env `RBITCOIN_TX_HEAD_REBUILD_WORKERS` override (`1` = serial) | Empty/wipe `tx.head` rebuild from `txid.body`. Logs `workers=` `free_GiB=` | -| **Per-peer chain record** | `PeerChain` per IBD slot: the newest announced run above the confirmed tip, at most 2,000 hashes; at most 1,024 `notfound` hashes, each kept 30 s; one split, one chain end, and one path block the peer showed after a probe. Header-walk `invalid_tips` (tips of chains refused for building on a rejected block): 1,024, oldest first out | Dropped with the slot or IBD state. Not written to disk | +| **Per-peer chain record** | `PeerChain` per IBD slot: the newest announced run above the confirmed tip, at most 2,000 hashes; at most 1,024 `notfound` hashes, each kept 30 s; one split, one chain end, one path block the peer showed after a probe, and the height a rewind left the record stale from. Header-walk `invalid_tips` (tips of chains refused for building on a rejected block): 1,024, oldest first out | Dropped with the slot or IBD state. Not written to disk | | **Ordered work path** | `MAX_ORDERED_HEADERS` | `IbdWorkState::hygiene` | | **Header-walk checkpoints** | One checkpoint per accepted 2,000-header reply, and those headers are written. The ordered queue is a window of at most 64,000 over headers already stored. One peer, the lowest time to a first block byte among peers that have not failed the walk, is asked only when the next header is not in the store. While the queue has room the locator starts at the stored tip, then the candidate, then a lighter fork still being weighed, so the reply can be enqueued. A full queue asks from the candidate. Block download uses the other peers, each asked only for blocks its chain showed (see **Per-peer chain record**); the header peer takes a block no other peer shows. Above the work floor a short reply that does not extend the candidate moves the reservation; a full 2,000-header window does not, so a lighter fork can still catch up. A block inv from another peer is one challenge, asked even when that peer's connect-time start height is below the walk: if the reply beats the candidate they become the header peer, and if it does not they leave the walk and the previous peer stays. An inv from a session with no header slot does not retire the reserved peer. One missed ask moves the reservation; a second miss disconnects. Less work does not disconnect. Each checkpoint holds hash, height, work, that reply's last 80-byte header, up to 11 timestamps, and the difficulty period (one period-start header plus the last non-limit `nBits`) so a fork or rewind can retarget. One competing chain is kept the same way until it loses or replaces the candidate. Mainnet stays under about 250 KB. Queue maps stay capped with the queue. Side headers below the work floor are one hash per peer. More than 4,000 disconnects that peer. Abandoned look-ahead tips are cleared at 1,024 and when a heavier chain is adopted. | Dropped with IBD state. `header.adopt` is the sidecar, not a resident copy. The file is deleted once the confirmed chain proves the anchor |