ibd: reuse the confirmed getheaders tail until the tip changes

TESTING.md

@@ -388,7 +388,7 @@ A cell that names a crate-local test describes the witness that exists today. It
 | `peer_blocksonly_and_orphan_tx` | P2P (**default**, crate) | One hub, relay off and then on. Blocks-only: a tx or wtx inv from an ordinary peer disconnects, a sendraw INVs once it is unbroadcast (never to block-relay) and serves, a `relay@` whitelisted peer's tx is kept and INV'd to the other inbound, and a type-0 getdata is ignored while the tip block still serves. A block confirms those txs and relay-on purges them. Then forced INVs skip block-relay, seen, feefilter, and isolated local-origin peers. Inbound waits 30s even for unbroadcast, never gets a tx older than its connection, and idle ticks do not clone or rescan. GetData serves only an announced or reorged-back tx, and a sendraw after a +300s jump neither INVs nor serves. An orphan parks on a tokio worker without a reject log and asks only the still-missing parent after NONPREF+TXID. |
 | `stored_header_resends_walk_once` | IBD (crate) | One 240-header genesis chain. A re-sent stored run walks no ancestors, an unmapped stored run walks once, an unknown parent is not stored, a rejected tail keeps the stored prefix under a lowered walk cap, and a run past that cap walks once and is still accepted. |
 | `two_peers_reserve_the_header_walk` | IBD (crate) | One hub. The lowest time-to-first-byte peer takes headers and the other takes blocks; a short miss moves the reservation without disconnecting; a heavier inv takes it and a lighter inv restores it; refill under the low-water mark stays on the reserved peer. A second chapter, because that lighter challenger is retired: a quiet header peer is skipped, asked again, then disconnected. A third, because that survivor already missed once: the only header peer is disconnected after two misses. |
-| `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. |
+| `a_seeded_header_walk_stores_drains_and_refills` | IBD (crate) | One walk from the stored tip. The first locator and the status line start there; the first batch is a checkpoint and the empty queue refills from that tip; with room in the queue the reserved peer keeps walking and a second peer is not asked; an empty queue still checkpoints; a refill stores without moving the walk; an empty queue refills from the confirmed tip; a full batch stays stored while a walk ask inside its window is not stirred. On that hub, two catch-up asks at one published tip share one confirmed-chain tail and the locator still starts at a stored path tip ahead of the confirmed tip; a same-height tip hash and a taller tip each rebuild the tail. |
 | `a_heavier_header_chain_keeps_its_context` | IBD (crate) | One hub: a challenger keeps median time while the tip is genesis (invalid at the median, valid one second later); that hub then confirms a stale ancestor, a heavier chain from below it replaces the candidate and drops its script skip, and a rewind uses the confirmed base work and height. Retarget is a second hub (period length is not regtest difficulty): a heavier fork across a gap retarget is adopted, rewind restores difficulty before the next retarget, and a challenger keeps its retarget snapshot. Those three checkpoint layouts cannot be one candidate. |
 | `analog_selector_keeps_each_boundary_rate` | Mempool (crate) | One selector table: the exact band edge, absolute log distance, the requested quantile, and the two-hundredth nearest window each keep their own rate. |
 | `fee_history_gaps_cache_and_reorg_share_one_history` | IBD (crate) | One history: heights without a hurdle are not observations, rates stay cached until a new height, only the newest hashes are kept, and a reorg drops those above. The RPC backfill stays `fee_history_backfills_from_the_chain_when_relay_starts`. |

crates/rbitcoin-net/src/ibd/dial.rs

@@ -415,14 +415,12 @@ pub(crate) fn replacement_available(
 pub(crate) fn request_headers_from(
     slots: &[PeerSlot],
     peer: usize,
-    hub: &ChainHub,
     _seq: &mut u32,
-    work_tips: &[BlockHash],
+    locator: Vec<BlockHash>,
 ) -> Result<bool, NetError> {
     let Some(s) = slots.iter().find(|s| s.id == peer && s.alive) else {
         return Ok(false);
     };
-    let locator = ibd_header_locator(hub, work_tips)?;
     Ok(s.cmd_tx.send(PeerCmd::GetHeaders { locator }).is_ok())
 }
 
@@ -436,7 +434,8 @@ pub(crate) fn request_headers_from(
 pub(crate) fn ibd_header_locator(
     hub: &ChainHub,
     work_tips: &[BlockHash],
-) -> Result<Vec<BlockHash>, NetError> {
+    confirmed_tail: &[BlockHash],
+) -> Vec<BlockHash> {
     let mut locator = Vec::with_capacity(32);
     // Eight hashes is the walk's own stop. This only enforces the protocol
     // maximum, so a stored tip prepended in front does not drop the oldest
@@ -449,27 +448,24 @@ pub(crate) fn ibd_header_locator(
             locator.push(*h);
         }
     }
+    // In-memory snapshot. Not a header-store walk.
     if let Some(t) = hub.tip_hash() {
-        if !locator.contains(&t) {
+        if locator.len() < crate::codec::MAX_LOCATOR_SZ && !locator.contains(&t) {
             locator.push(t);
         }
     }
-    let rest = hub
-        .query
-        .locator_hashes()
-        .map_err(|e| NetError::Consensus(e.to_string()))?;
-    for h in rest {
-        if !locator.contains(&h) {
-            locator.push(h);
-        }
+    for h in confirmed_tail {
         if locator.len() >= crate::codec::MAX_LOCATOR_SZ {
             break;
         }
+        if !locator.contains(h) {
+            locator.push(*h);
+        }
     }
     if locator.is_empty() {
         locator.push(BlockHash::from_byte_array([0u8; 32]));
     }
-    Ok(locator)
+    locator
 }
 
 /// Mark `peer` dead and drop it from every hash it holds. Returns the hashes
@@ -1836,9 +1832,9 @@ mod tests {
         let mut seq = 0u32;
         let mut dead = dummy_slot(1, addr(1), false);
         dead.alive = false;
-        assert!(!request_headers_from(&[dead], 1, &hub, &mut seq, &[]).unwrap());
+        assert!(!request_headers_from(&[dead], 1, &mut seq, vec![]).unwrap());
         // Locator alone (empty work tips + no tip) still returns genesis zero.
-        let loc = ibd_header_locator(&hub, &[]).unwrap();
+        let loc = ibd_header_locator(&hub, &[], &[]);
         assert!(!loc.is_empty());
         let _ = std::fs::remove_dir_all(dir);
     }

crates/rbitcoin-net/src/ibd/header_walk.rs

@@ -18,7 +18,7 @@
 //! checkpoints is the previous checkpoint plus those stored headers. That
 //! replay runs only when a competing headers batch forks there.
 
-use super::state::IbdWorkState;
+use super::state::{ConfirmedLocatorTail, IbdWorkState};
 use super::ORDERED_HEADERS_SOFT_CAP;
 use crate::chain::ChainHub;
 use bitcoin::block::Header;
@@ -2368,13 +2368,43 @@ fn ask_header_peer(
     peer: usize,
     tips: &[BlockHash],
 ) -> Result<bool, crate::error::NetError> {
-    if !super::dial::request_headers_from(&st.slots, peer, hub, &mut st.header_req_seq, tips)? {
+    let tail = confirmed_chain_tail(st, hub)?;
+    let locator = super::dial::ibd_header_locator(hub, tips, &tail);
+    if !super::dial::request_headers_from(&st.slots, peer, &mut st.header_req_seq, locator)? {
         return Ok(false);
     }
     note_lane(st, peer);
     Ok(true)
 }
 
+/// Confirmed-chain tail for `getheaders`.
+///
+/// Rebuilt when `published_tip` height or hash changes. The work-path prefix
+/// is not part of this vec.
+fn confirmed_chain_tail(
+    st: &mut IbdWorkState,
+    hub: &ChainHub,
+) -> Result<Vec<BlockHash>, crate::error::NetError> {
+    let published = hub.query.published_tip();
+    let height = published.map(|tip| tip.height);
+    let hash = published.map(|tip| tip.hash);
+    if let Some(cached) = &st.confirmed_locator_tail {
+        if cached.height == height && cached.hash == hash {
+            return Ok(cached.hashes.clone());
+        }
+    }
+    let tail = hub
+        .query
+        .locator_hashes()
+        .map_err(|e| crate::error::NetError::Consensus(e.to_string()))?;
+    st.confirmed_locator_tail = Some(ConfirmedLocatorTail {
+        height,
+        hash,
+        hashes: tail.clone(),
+    });
+    Ok(tail)
+}
+
 fn parent_header(st: &IbdWorkState, hub: &ChainHub, hash: BlockHash) -> Option<Header> {
     known(st, hash).header.or_else(|| hub.header_of(&hash))
 }
@@ -3574,6 +3604,9 @@ mod tests {
 
     /// One ask from the stored tip. An accepted batch is written and
     /// checkpointed. A header already indexed is appended and not asked for.
+    /// On that same hub, a second ask at one confirmed tip reuses the
+    /// confirmed-chain tail. A same-height tip hash, then a taller tip, each
+    /// rebuild it.
     #[test]
     fn a_seeded_header_walk_stores_drains_and_refills() {
         let (_dir, hub) = crate::chain::tiny_regtest_hub_labeled("header-walk-seeded");
@@ -3655,6 +3688,135 @@ mod tests {
         assert!(st.ordered_set.contains(&second.block_hash()));
         assert!(rx0.try_recv().is_err());
         assert!(rx1.try_recv().is_err());
+        confirmed_tail_follows_the_published_tip(&hub);
+    }
+
+    /// Same hub as the seeded walk. Two asks share one confirmed tail. A
+    /// same-height hash, then a taller tip, each rebuild it.
+    fn confirmed_tail_follows_the_published_tip(hub: &ChainHub) {
+        let script = bitcoin::ScriptBuf::from_bytes(vec![0x51]);
+        hub.generate_to_script(2, script.clone(), vec![]).unwrap();
+        let tip = hub.tip_hash().unwrap();
+        let tip_h = hub.tip_height().unwrap();
+        let genesis = hub.params.genesis_hash;
+        let ancestor = hub
+            .query
+            .locator_hashes()
+            .unwrap()
+            .into_iter()
+            .find(|hash| *hash != tip && *hash != genesis)
+            .expect("the confirmed chain locator has an ancestor between tip and genesis");
+        let (peer, mut rx) = slot(0);
+        let mut st = IbdWorkState::new(vec![peer], Some(tip), Some(tip_h));
+        let ahead = BlockHash::from_byte_array([0xab; 32]);
+        st.ordered.push_back(ahead);
+        st.ordered_set.insert(ahead);
+        st.hash_height.insert(ahead, tip_h.saturating_add(100));
+        assert!(st.ordered.len() < ORDERED_HEADERS_SOFT_CAP);
+        assert!(send_getheaders(&mut st, hub).unwrap());
+        let first = take_locator(&mut rx);
+        assert_eq!(
+            first.first().copied(),
+            Some(ahead),
+            "a queue with room asks from the stored path tip"
+        );
+        assert!(
+            first.contains(&ancestor),
+            "the confirmed ancestor stays in the locator"
+        );
+        assert!(first.contains(&tip));
+        same_tip_reuses_the_tail(&mut st, hub, &mut rx, &first);
+        let same_h_tip = same_height_rebuilds_the_tail(&mut st, hub, &mut rx, tip, tip_h, ahead);
+        taller_tip_rebuilds_the_tail(&mut st, hub, &mut rx, &script, same_h_tip, ahead);
+    }
+
+    fn same_tip_reuses_the_tail(
+        st: &mut IbdWorkState,
+        hub: &ChainHub,
+        rx: &mut mpsc::UnboundedReceiver<PeerCmd>,
+        first: &[BlockHash],
+    ) {
+        assert_ne!(take_header_ask(st, 0), HeaderAsk::Unsolicited);
+        let before = rbitcoin_query::header_lookup_count();
+        assert!(send_getheaders(st, hub).unwrap());
+        let second_ask = take_locator(rx);
+        assert_eq!(
+            rbitcoin_query::header_lookup_count().saturating_sub(before),
+            0,
+            "a second ask at the same confirmed tip does not walk the header store"
+        );
+        assert_eq!(second_ask, first);
+    }
+
+    fn same_height_rebuilds_the_tail(
+        st: &mut IbdWorkState,
+        hub: &ChainHub,
+        rx: &mut mpsc::UnboundedReceiver<PeerCmd>,
+        tip: BlockHash,
+        tip_h: u32,
+        ahead: BlockHash,
+    ) -> BlockHash {
+        hub.invalidate_block(tip).unwrap();
+        assert_eq!(hub.tip_height(), Some(tip_h - 1));
+        let alt_script = bitcoin::ScriptBuf::from_bytes(vec![0x52]);
+        hub.generate_to_script(1, alt_script, vec![]).unwrap();
+        let same_h_tip = hub.tip_hash().unwrap();
+        let published = hub.query.published_tip().expect("published tip");
+        assert_eq!(hub.tip_height(), Some(tip_h));
+        assert_eq!(published.height, tip_h);
+        assert_eq!(published.hash, same_h_tip.to_byte_array());
+        assert_ne!(same_h_tip, tip);
+        assert_ne!(take_header_ask(st, 0), HeaderAsk::Unsolicited);
+        let before = rbitcoin_query::header_lookup_count();
+        assert!(send_getheaders(st, hub).unwrap());
+        let reorged = take_locator(rx);
+        assert!(
+            rbitcoin_query::header_lookup_count() > before,
+            "a same-height tip hash rebuilds the tail"
+        );
+        assert_eq!(reorged.first().copied(), Some(ahead));
+        assert!(
+            reorged.contains(&same_h_tip),
+            "the rebuilt tail names the replacement tip"
+        );
+        let tail = st
+            .confirmed_locator_tail
+            .as_ref()
+            .expect("the same-height ask stored a tail");
+        assert_eq!(tail.height, Some(tip_h));
+        assert_eq!(tail.hash, Some(*same_h_tip.as_byte_array()));
+        assert!(tail.hashes.contains(&same_h_tip));
+        assert!(
+            !tail.hashes.contains(&tip),
+            "the replaced tip hash leaves the confirmed tail"
+        );
+        same_h_tip
+    }
+
+    fn taller_tip_rebuilds_the_tail(
+        st: &mut IbdWorkState,
+        hub: &ChainHub,
+        rx: &mut mpsc::UnboundedReceiver<PeerCmd>,
+        script: &bitcoin::ScriptBuf,
+        same_h_tip: BlockHash,
+        ahead: BlockHash,
+    ) {
+        hub.generate_to_script(1, script.clone(), vec![]).unwrap();
+        let new_tip = hub.tip_hash().unwrap();
+        assert_ne!(new_tip, same_h_tip);
+        assert_ne!(take_header_ask(st, 0), HeaderAsk::Unsolicited);
+        let before = rbitcoin_query::header_lookup_count();
+        assert!(send_getheaders(st, hub).unwrap());
+        let taller = take_locator(rx);
+        assert!(
+            rbitcoin_query::header_lookup_count() > before,
+            "a changed confirmed tip rebuilds the tail"
+        );
+        assert_eq!(taller.first().copied(), Some(ahead));
+        assert!(
+            taller.contains(&new_tip),
+            "the rebuilt tail names the new confirmed tip"
+        );
     }
 
     fn locator_front(rx: &mut mpsc::UnboundedReceiver<PeerCmd>) -> Option<BlockHash> {
@@ -3671,6 +3833,13 @@ mod tests {
         }
     }
 
+    fn take_locator(rx: &mut mpsc::UnboundedReceiver<PeerCmd>) -> Vec<BlockHash> {
+        match rx.try_recv() {
+            Ok(PeerCmd::GetHeaders { locator }) => locator,
+            _ => panic!("expected one getheaders"),
+        }
+    }
+
     /// Walk tip is `first` (height 1). Queue tail height is 0, so the walk is ahead.
     fn walk_ahead_of_a_short_queue(
         label: &str,

crates/rbitcoin-net/src/ibd/state.rs

@@ -125,6 +125,17 @@ impl InflightReq {
     }
 }
 
+/// Confirmed-chain `getheaders` tail for one published tip.
+///
+/// Absent until the first catch-up ask. The next ask reuses `hashes` while
+/// both `height` and `hash` still match `published_tip`. The work-path
+/// prefix is not stored here.
+pub(crate) struct ConfirmedLocatorTail {
+    pub(crate) height: Option<u32>,
+    pub(crate) hash: Option<[u8; 32]>,
+    pub(crate) hashes: Vec<BlockHash>,
+}
+
 /// Core mutable state for the IBD event loop.
 pub(crate) struct IbdWorkState {
     pub slots: Vec<PeerSlot>,
@@ -198,6 +209,8 @@ pub(crate) struct IbdWorkState {
     pub(crate) header_cmd: Option<HeaderCmdTx>,
     /// Heights below this are already claim-ready. Rewind starts here.
     pub(crate) densify_ready_through: u32,
+    /// Confirmed-chain `getheaders` tail for one published tip.
+    pub(crate) confirmed_locator_tail: Option<ConfirmedLocatorTail>,
 }
 
 pub(crate) enum HeaderCmd {
@@ -289,6 +302,7 @@ impl IbdWorkState {
             confirm_feed: None,
             header_cmd: None,
             densify_ready_through: 0,
+            confirmed_locator_tail: None,
         }
     }