ibd: fetch archived bodies above the tip after a crash

changelog.d/archived-retry-same-batch-parent.md

@@ -0,0 +1,8 @@
+Fixed
+
+- **IBD connects archived blocks that spend an archived create.** A retry
+  from archived bodies found no range for a parent created in the same batch,
+  or in an earlier chunk of the same wave. It logged `lookup stage miss` and
+  halted IBD below valid blocks. The retry follows an invalid block in a
+  batch, or a restart after a crash between the Class A write and the tip
+  advance.

changelog.d/ibd-archived-tip-hole.md

@@ -0,0 +1,6 @@
+Fixed
+
+- **IBD fetches archived bodies above the tip again after a crash.** A crash
+  between the Class A write and the tip advance leaves bodies above the tip.
+  The restart marked them archived, densify skipped them, and only a
+  quarter-full queue raced the tip hole, so IBD could sit at the old tip.

crates/rbitcoin-net/src/ibd/assign.rs

@@ -294,6 +294,15 @@ pub(crate) fn assign_work_ordered(
         );
     if race_tip {
         issued += cover_tip_batch_holes(st, hub, cfg, &alive, &tip_holes);
+    } else {
+        // Densify skips a Class A seed (`need_hash_at`). Without a race,
+        // only this cover gets an archived tip hole back into the queue.
+        let archived: Vec<BlockHash> = tip_holes
+            .iter()
+            .copied()
+            .filter(|h| st.body.is_known_archived(h))
+            .collect();
+        issued += cover_tip_holes(st, hub, cfg, &alive, &archived, 1);
     }
     if tip_holes.is_empty() {
         issued += cover_first_pre_hole(st, hub, cfg, &alive);

crates/rbitcoin-net/src/ibd/events/confirm_reject_tests.rs

@@ -2678,6 +2678,145 @@ fn batched_spend_of_a_later_block_is_rejected_alone() {
     rig.finish();
 }
 
+/// A wave whose first block spends a create from its own block and whose
+/// second block is invalid at write. The cascade retries the first block from
+/// the archive, and that retry must still pin the same-block parent.
+#[test]
+fn archived_retry_pins_a_same_block_parent() {
+    use super::super::assign::tests::lock_default_assign_stop;
+    use super::super::state::InflightReq;
+    use bitcoin::{Amount, ScriptBuf};
+    use rbitcoin_consensus::{grind_regtest_pow, mine_regtest_paying};
+
+    let _env = lock_default_assign_stop();
+    let mut rig = WireRig::new("archived-same-block", 2);
+    let (t, cbs) = (rig.t, rig.cbs.clone());
+    let spk = ScriptBuf::from_bytes(vec![0x51]);
+    let parent = WireRig::spend(cbs[0]);
+    let child = WireRig::spend(parent.compute_txid());
+    let b1 = mine_regtest_paying(
+        rig.tip,
+        rig.tip_time + 600,
+        t + 1,
+        spk.clone(),
+        vec![parent, child],
+    );
+    let mut b2 = mine_regtest_paying(b1.block_hash(), rig.tip_time + 1200, t + 2, spk, vec![]);
+    b2.txdata[0].output[0].value += Amount::from_sat(1);
+    b2.header.merkle_root = b2.compute_merkle_root().unwrap();
+    grind_regtest_pow(&mut b2.header);
+    let (h1, h2) = (b1.block_hash(), b2.block_hash());
+    rig.plant(&[&b1, &b2]);
+    for (peer, body) in [(1, &b1), (2, &b2)] {
+        let hash = body.block_hash();
+        rig.st.slots[peer - 1].in_flight.insert(hash);
+        rig.st.inflight.insert(hash, InflightReq::new(peer));
+        rig.deliver(peer, body);
+    }
+    rig.start_engine();
+
+    let rejects = rig.pump(
+        |_, hash| if hash == h1 { b1.clone() } else { b2.clone() },
+        |st, _, _| st.halt.is_some() || st.reorg.invalid.contains(h2.to_byte_array()),
+    );
+    assert_eq!(
+        rig.st.halt, None,
+        "rejects={rejects:?} last={:?}",
+        rig.last_reject_err
+    );
+    assert!(
+        rejects.contains(&(h1, ConfirmRejectClass::Cascade, 2)),
+        "both blocks in one wave: {rejects:?}"
+    );
+    assert!(!rig.st.engine_fault_seen.contains(&h1), "{rejects:?}");
+    assert!(rig.st.reorg.invalid.contains(h2.to_byte_array()));
+    assert_eq!(rig.hub.tip_height(), Some(t + 1), "{rejects:?}");
+    rig.finish();
+}
+
+/// Archived blocks above the tip, as a crash between the Class A write and
+/// the tip advance leaves them. The wave is longer than one load batch, so
+/// the last block loads in the next chunk and spends a create from the
+/// first, noted in flight with no outs.
+#[test]
+fn archived_wave_pins_a_parent_from_the_previous_chunk() {
+    use super::super::assign::tests::lock_default_assign_stop;
+    use super::super::state::InflightReq;
+    use bitcoin::ScriptBuf;
+    use rbitcoin_consensus::mine_regtest_paying;
+
+    let _env = lock_default_assign_stop();
+    let mut rig = WireRig::new("archived-cross-chunk", 1);
+    let (t, cbs) = (rig.t, rig.cbs.clone());
+    let op_true = ScriptBuf::from_bytes(vec![0x51]);
+    let n = super::super::confirm::CONFIRM_RUN_MAX_BLOCKS as u32 + 1;
+    let parent = WireRig::spend(cbs[0]);
+    let mut child = Some(WireRig::spend(parent.compute_txid()));
+    let mut extra = vec![parent];
+    let mut blocks: Vec<bitcoin::Block> = Vec::new();
+    for i in 1..=n {
+        let prev = blocks.last().map_or(rig.tip, |b| b.block_hash());
+        if i == n {
+            extra = child.take().into_iter().collect();
+        }
+        let b = mine_regtest_paying(
+            prev,
+            rig.tip_time + 600 * i,
+            t + i,
+            op_true.clone(),
+            std::mem::take(&mut extra),
+        );
+        blocks.push(b);
+    }
+    // One Class A write for the run, as the write stage does, with no tip advance.
+    let bodies: Vec<_> = blocks.iter().cloned().map(std::sync::Arc::new).collect();
+    let fks: Vec<_> = bodies
+        .iter()
+        .map(|b| rig.hub.ensure_header_fk(&b.header).unwrap())
+        .collect();
+    let txids: Vec<Vec<[u8; 32]>> = bodies
+        .iter()
+        .map(|b| {
+            b.txdata
+                .iter()
+                .map(|t| t.compute_txid().to_byte_array())
+                .collect()
+        })
+        .collect();
+    let items: Vec<_> = bodies
+        .iter()
+        .zip(&fks)
+        .zip(&txids)
+        .map(|((b, fk), ids)| (*fk, b, ids.as_slice()))
+        .collect();
+    rig.hub.query.archive_class_a_from_wire(&items).unwrap();
+    assert_eq!(rig.hub.tip_height(), Some(t));
+    let refs: Vec<&bitcoin::Block> = blocks.iter().collect();
+    rig.plant(&refs);
+    for body in &blocks {
+        let hash = body.block_hash();
+        rig.st.slots[0].in_flight.insert(hash);
+        rig.st.inflight.insert(hash, InflightReq::new(1));
+        rig.deliver(1, body);
+    }
+    rig.start_engine();
+
+    let last = blocks.last().unwrap().block_hash();
+    let by_hash: std::collections::HashMap<_, _> =
+        blocks.iter().map(|b| (b.block_hash(), b.clone())).collect();
+    let rejects = rig.pump(
+        |_, hash| by_hash[&hash].clone(),
+        |st, hub, _| st.halt.is_some() || hub.tip_hash() == Some(last),
+    );
+    assert_eq!(
+        rig.st.halt, None,
+        "rejects={rejects:?} last={:?}",
+        rig.last_reject_err
+    );
+    assert_eq!(rig.hub.tip_height(), Some(t + n), "{rejects:?}");
+    rig.finish();
+}
+
 /// tip+1 is the last block on the path, and load cannot read its header
 /// row when it stamps the wave: a local store fault, not a block verdict.
 /// The wave goes back to lookup, so once the row reads again tip+1

crates/rbitcoin-query/src/confirm_run/lookup.rs

@@ -366,6 +366,16 @@ pub(super) fn stamp_parent_pin_archived(
             query.confirm_stats(),
         )
         .map_err(ConfirmError::from)?;
+    } else {
+        // The IBD skeleton carries external ranges. A parent with no body is
+        // an archived create: this batch's own, or an earlier chunk's of the
+        // same wave (noted in flight with no outs). Only the store has its
+        // range. In-flight pins are skipped inside the fill.
+        let mut archived: U64Map<crate::ParentIdent> =
+            stamp.idents.extract_if(|_, e| e.body.is_none()).collect();
+        crate::fill_missing_parent_ranges(query.store(), ifo, &mut archived, query.confirm_stats())
+            .map_err(ConfirmError::from)?;
+        stamp.idents.extend(archived);
     }
     for ident in stamp.idents.values() {
         if ident.txid == [0u8; 32] {

crates/rbitcoin-query/src/stamp.rs

@@ -6,8 +6,10 @@
 //! rehydrate. In-flight holds CreatePins until load drops map rows below a
 //! lookup-wave drain+fence snapshot taken before TipOnly. Same-wave creates are
 //! omitted from that skeleton. Class A `CreatePin::set_loc` after append;
-//! later-wave stamp reads pin loc. IBD (`skeleton = Some`) never loc-by-fk:
-//! spent is pin loc, skeleton TipOnly loc, or unset for write fill.
+//! later-wave stamp reads pin loc. IBD (`skeleton = Some`) never loc-by-fk
+//! here: spent is pin loc, skeleton TipOnly loc, or unset for write fill. The
+//! all-archived retry (`stamp_parent_pin_archived`) loc-fills its archived
+//! parents that have no body. External IBD parents still must not.
 
 use crate::{CreatePin, InFlight, QueryError, U64Map};
 use crate::{IdMap, TxidHasher};

crates/rbitcoin-test/tests/end_of_ibd.rs

@@ -1,6 +1,7 @@
 //! End of IBD: tip follow, scripthash history, and restart on a live miner.
 
 use bitcoin::absolute::LockTime;
+use bitcoin::hashes::Hash;
 use bitcoin::script::ScriptBuf;
 use bitcoin::transaction::Version as TxVersion;
 use bitcoin::{
@@ -500,7 +501,7 @@ async fn finish_partial(dir: &Path, miner: SocketAddr, synced: &Synced) {
     stop_run_p2p(rpc, node).await;
 }
 
-async fn partial_with_miner_down(miner_dir: &Path, synced: &Synced) {
+async fn partial_with_miner_down(miner_dir: &Path, synced: &mut Synced) {
     let miner = start_miner(miner_dir).await;
     let miner_addr = miner.local_addr;
     assert_eq!(miner.tip_height().unwrap(), synced.height);
@@ -525,9 +526,97 @@ async fn partial_with_miner_down(miner_dir: &Path, synced: &Synced) {
     let miner = start_miner(miner_dir).await;
     finish_partial(partial_dir.path().as_path(), miner.local_addr, synced).await;
     finish_partial(held_dir.path().as_path(), miner.local_addr, synced).await;
+    crash_with_class_a_above_tip(&miner, synced).await;
     miner.shutdown().await;
 }
 
+/// A crash between the Class A write and the tip advance leaves bodies above
+/// the tip with no connect. The second block spends a create of the first and
+/// one of its own, so the next start must fetch both and pin those parents
+/// from the archive. The store holds catch-up indexes, as a crash in IBD does,
+/// so a halted IBD has no tip-follow to fall back on.
+async fn crash_with_class_a_above_tip(miner: &P2PNode, synced: &mut Synced) {
+    let dir = TestDatadir::new().unwrap();
+    let query = Query::open_or_create_tiny(dir.store_path()).unwrap();
+    let node = P2PNode::start(any_port(), query, ChainParams::regtest(), Milestone::NONE)
+        .await
+        .expect("crash syncer");
+    let peers = [NetAddr::Ip(miner.local_addr)];
+    node.sync(&peers, IbdConfig::for_test())
+        .await
+        .expect("crash syncer ibd");
+    node.shutdown().await;
+
+    let op_true = ScriptBuf::from_bytes(vec![0x51]);
+    let matured = miner
+        .query
+        .reconstruct_block_at_height(Height(2))
+        .unwrap()
+        .txdata[0]
+        .compute_txid();
+    let parent = spend_to(matured, Amount::from_sat(49_0000_0000), vec![0x51]);
+    let child = spend_to(
+        parent.compute_txid(),
+        Amount::from_sat(48_0000_0000),
+        vec![0x51],
+    );
+    let grandchild = spend_to(
+        child.compute_txid(),
+        Amount::from_sat(47_0000_0000),
+        vec![0x51],
+    );
+    let mut blocks = Vec::new();
+    for txs in [vec![parent], vec![child, grandchild]] {
+        miner
+            .hub
+            .generate_to_script(1, op_true.clone(), txs)
+            .unwrap();
+        let block = miner
+            .query
+            .reconstruct_block_at_height(Height(miner.tip_height().unwrap()));
+        blocks.push(Arc::new(block.unwrap()));
+    }
+
+    let query = Query::open_or_create_tiny(dir.store_path()).unwrap();
+    let tip = query.tip_height();
+    assert_eq!(tip, Some(Height(synced.height)));
+    let mut prev_fk = query
+        .get_header_by_hash(blocks[0].header.prev_blockhash.as_byte_array())
+        .unwrap()
+        .unwrap()
+        .0;
+    let mut fks = Vec::new();
+    let mut txids = Vec::new();
+    for block in &blocks {
+        let hash = block.block_hash().to_byte_array();
+        let rec = rbitcoin_query::header_to_record(prev_fk, &block.header, hash);
+        prev_fk = query.ensure_header(&rec).unwrap();
+        fks.push(prev_fk);
+        let ids: Vec<[u8; 32]> = block
+            .txdata
+            .iter()
+            .map(|t| t.compute_txid().to_byte_array())
+            .collect();
+        synced
+            .script_a
+            .extend(block.txdata.iter().map(|t| t.compute_txid().to_string()));
+        txids.push(ids);
+    }
+    let items: Vec<_> = blocks
+        .iter()
+        .zip(&fks)
+        .zip(&txids)
+        .map(|((b, fk), ids)| (*fk, b, ids.as_slice()))
+        .collect();
+    query.archive_class_a_from_wire(&items).unwrap();
+    query.flush().unwrap();
+    assert_eq!(query.tip_height(), tip);
+    synced.height = miner.tip_height().unwrap();
+    synced.hash = miner.hub.tip_hash().unwrap().to_string();
+    drop(query);
+    finish_partial(dir.path().as_path(), miner.local_addr, synced).await;
+}
+
 async fn follow_journey() {
     let (blocks, script_a, script_b) = fixture_chain();
     let miner_dir = TestDatadir::new().unwrap();
@@ -543,7 +632,7 @@ async fn follow_journey() {
     assert_eq!(synced.height as usize, blocks.len() - 1);
 
     ibd_then_restart(miner, miner_addr, &mut synced).await;
-    partial_with_miner_down(miner_dir.path().as_path(), &synced).await;
+    partial_with_miner_down(miner_dir.path().as_path(), &mut synced).await;
 }
 
 #[tokio::test(flavor = "multi_thread", worker_threads = 4)]

docs/architecture.md

@@ -36,9 +36,11 @@ appender. Roles: [`concurrency.md`](./concurrency.md). Stage IO:
 
 **Invariant — Class A never leads tip:** there is no dual-track “archive Class A
 far ahead of confirmed tip.” Wire plan + Class A append + Class C tip advance
-are one confirm-write era. Do not reintroduce plan-time “archive lead” heuristics
-(e.g. body `posix_fadvise(DONTNEED)` when body count ≫ tip) — under this path
-just-written body pages stay tip-hot. **No** ContigPark / archive-job fallback
+are one confirm-write era on a clean run. A crash inside that era is the
+exception ([crash-recovery.md](./crash-recovery.md#class-a-archive)). Do not
+reintroduce plan-time “archive lead” heuristics (e.g. body
+`posix_fadvise(DONTNEED)` when body count ≫ tip) — under this path just-written
+body pages stay tip-hot. **No** ContigPark / archive-job fallback
 for unknown-height bodies (mark missing → re-getdata).
 
 - **Storage center** is a **transaction-relational archive** on **map-free

docs/concurrency.md

@@ -10,7 +10,7 @@ Short map of who may write which tables. **Format is unstable until 1.0.**
 | Confirm **lookup** | 1 OS thread | pack/hold from BQ stamped Σ inputs; emit: decode BQ raw + TipOnly `head_fk` + **`take_raw` onto loadq** (`Arc<Block>` + `TxPrecompute`). Does **not** plan_batch / structure |
 | Confirm **load** | 1 OS thread | `confirm_wire_lookup_stamp` (structure with **loadq `pres`**, plan_batch binds carried BQ keys, skeleton bind) + pin `txout` + assemble |
 | Confirm **scripts** | 1 OS thread (`ibd-confirm`) publishes waves + `rbtc-scripts` steal (lock-free claim). The interpreter runs here, not on a peer IO task (`verify_tx_scripts_detached`) | **none** — pure CPU |
-| Confirm **write** | 1 OS thread (`ibd-confirm-write`) + 1 process-wide `ibd-confirm-head` | **sole Class A appender** (`txout`+`seqsigwit`+`spent`; IBD encodes ins from `Arc<Block>` + SpendEdges) + structural + Class C + spend annotate on **`spent.body`** + tip GC; **`block_queue_dequeue_height`**. `tx.head` write-behind insert runs on **`ibd-confirm-head`** (not a per-batch spawn). `--datadir-hdd` starts that insert after spend annotate; otherwise it overlaps structural and Class C. Class A **never leads tip** (same commit era; no archive-ahead DONTNEED). When `index_live` is set, this thread appends that index after `confirmed[h]` is set (one put per confirm batch) and does not read Class A for it. Startup `prepare_live_indexes` seals a restart gap of at most one write drain (14 × 144 heights) before confirm; a larger gap is left for `rbtc-idx-wb` |
+| Confirm **write** | 1 OS thread (`ibd-confirm-write`) + 1 process-wide `ibd-confirm-head` | **sole Class A appender** (`txout`+`seqsigwit`+`spent`; IBD encodes ins from `Arc<Block>` + SpendEdges) + structural + Class C + spend annotate on **`spent.body`** + tip GC; **`block_queue_dequeue_height`**. `tx.head` write-behind insert runs on **`ibd-confirm-head`** (not a per-batch spawn). `--datadir-hdd` starts that insert after spend annotate; otherwise it overlaps structural and Class C. Class A **never leads tip** (same commit era on a clean run; no archive-ahead DONTNEED). A crash inside the era can leave it above the tip ([crash-recovery.md](./crash-recovery.md#class-a-archive)). When `index_live` is set, this thread appends that index after `confirmed[h]` is set (one put per confirm batch) and does not read Class A for it. Startup `prepare_live_indexes` seals a restart gap of at most one write drain (14 × 144 heights) before confirm; a larger gap is left for `rbtc-idx-wb` |
 | IBD main loop | 1 tokio task | none (orchestration only). Event drain + confirm-offer every turn; getdata assign ≤50 ms (immediate if inflight empty); main-loop `getheaders`/`locator_hashes` ≤500 ms (empty path immediate); peer stall/relative-slow and work-path hygiene ≤1 s. Full-batch header continuation stays on the Headers event. Tip hash is the snapshot confirm published with `confirmed[]` (`published_tip`); each admitted header's parent stays on the work state. Those two reads are not store IO. The write holds `tip_commit` from the tip-height check through `flush_class_c_tip`. IBD rewind tries that lock and retries on the next pass. |
 
 **IoSession TLS:** one completion session per OS thread (`with_thread_local`). Harvest / poison / drain / do-not-flatten: [`io-modality.md`](./io-modality.md). `RBITCOIN_IO=pread` disables the session. SH k-way merge submits 256 KiB ahead preads on that TLS session and waits only when promote needs a page that has not completed.

docs/crash-recovery.md

@@ -53,7 +53,7 @@ Open revalidation runs in `Query::open_or_create` **before** P2P can extend tip.
 ## Class A (archive)
 
 - Append-oriented; re-archive is **idempotent** when `header_txs` already present.
-- **Never leads tip:** Class A is published only on the confirm-write path with tip advance in the same era (no dual-track archive-ahead).
+- **Never leads tip** on a clean run: Class A is published only on the confirm-write path with tip advance in the same era (no dual-track archive-ahead). A crash between the Class A write and the tip advance can leave bodies above the tip. IBD gets them again through the tip-hole cover.
 - Kill mid-archive without a complete body association ⇒ not treated as archived; re-getdata.
 
 ## Spends (v5: annotation on create outputs)