Cut suite idle time and keep block templates on a stripped pool

.github/workflows/ci.yml

@@ -99,6 +99,8 @@ jobs:
         run: ./scripts/tmpfs-test-runner.test.sh
       - name: overlay-functional runner self-test
         run: ./scripts/overlay-functional/run.sh.test.sh
+      - name: journeys manifest
+        run: ./scripts/journeys-manifest.test.sh
       - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
         with:
           shared-key: ci-clippy

Cargo.toml

@@ -100,4 +100,8 @@ debug = "line-tables-only"
 debug = false
 
 [profile.test.package."*"]
+# `*` is dependencies only (workspace crates keep the profile above).
+# Optimized crypto and codec crates make block hashing in tests cheap.
+# rust-cache holds the dep build across CI runs.
+opt-level = 2
 debug = false

TESTING.md

@@ -181,7 +181,7 @@ absolute path. Keep default-tier fixtures small: on tmpfs, test bytes are RAM.
 | `rbitcoin-store --lib` | **<45 s** | Catalog-run fixtures stay tens of tiny files, not thousands |
 | `rbitcoin-consensus --lib` | **<30 s** | Pure-result rule: [True journeys](#true-journeys). Mainnet 866342 (~1.6 s) is the historical prevout pin — one zstd decode, overweight on a clone. |
 | `rbitcoin-query --lib` | **<20 s** | |
-| `rbitcoin-test --test scenarios` | **<15 s** | Prefer `pad_empty_from` / shared mature helpers |
+| `rbitcoin-test --test journeys` | **<15 s** for the old scenarios module | Prefer `pad_empty_from` / shared mature helpers. Default journeys are one binary (`scenarios::` and the rest). |
 | **Full** `cargo test --workspace` | **≤3 min** warm | Stretch **<2 min** |
 
 **New default-suite test rule:** if a new or expanded default test routinely takes **>2 s wall** on a warm tree, the PR must **justify** it (what contract needs that cost, why a smaller N / unit cannot hit the branch). Prefer `#[ignore]` + reason string for true microbenches / host-only forensics.

changelog.d/suite-under-150s.md

@@ -0,0 +1,15 @@
+Changed
+
+- **P2P shutdown wakes dial, accept, and connect-retry.** Those tasks were blocked on a channel or a tick, so every shutdown sat out the full 750 ms grace and then aborted them. The grace remains for a session that is still unwinding.
+- **`--max-run-secs` exits when the deadline is reached.** The tip loop used to notice it only on the 5 s perf tick.
+- **IBD densify assign-stop is a field on the IBD config.** `RBITCOIN_BLOCK_QUEUE_BYTES` / `_GB` are still read once when a production config is built. Tests pass the cap in, so they no longer serialize on a process-wide env lock.
+- **Test dependencies compile at opt-level 2.** Workspace crates stay unoptimized. Dev and test builds still keep line tables.
+- **The hub `OP_TRUE` signet generate test uses regtest difficulty.** Accept still requires the BIP325 solution. The real signet target stays covered by the consensus miner test.
+- **`NODE_COMPACT_FILTERS` lives on the peer hub.** Version, feeler, addr, and `getnetworkinfo` read that node's bit. There is no process-wide flag.
+- **`getblocktemplate`, `getrawmempool`, and `generate` stay off while the pool is not maintained.** Relay off (still catching up) returns `-10`. A node with no mempool still serves `getblocktemplate` on the chain tip and an empty `getrawmempool`. `-blocksonly` returns a permanent error so a miner does not retry. `generate` during catch-up mines the coinbase only. Once relay is on, `getblocktemplate` and Stratum v2 build on the tip recorded after the pool strip. A Stratum v2 session that is still holding tries again on the template interval, so the tip that ends catch-up is not missed when relay turns on after that event. A block connect and an initial-download confirm batch hold that mark from before the chain tip moves until the drop is stored. A disconnect, including a header rewind, moves it to the fork point the pool was put back on. There is no chain scan under the mempool lock.
+- **A node built with the journey `test-hooks` feature warns at startup.** That build uses the short reconnect and SV2 cooldown. Release, nix, and `cargo build -p rbitcoin-node` do not.
+- **`cargo llvm-cov` no longer builds the workspace twice.** The coverage script was compiling plain node and cli bins before the instrumented test build. The suite already builds the bins it execs.
+
+Fixed
+
+- **Default journeys link and run as one test binary.** Doctests are off on library crates that have none. The node, cli, and journey-node bins do not build an empty test harness. Journeys compile a 50ms SAM retry, a 250ms connect redial, and a 100ms SV2 budget cooldown. The node binary keeps 500ms, 2s, and 1s.

crates/rbitcoin-bench/Cargo.toml

@@ -18,6 +18,9 @@ mutants = []
 default = []
 cli = []
 
+[lib]
+doctest = false
+
 [dependencies]
 rbitcoin-primitives = { workspace = true }
 bitcoin = { workspace = true }

crates/rbitcoin-cli/Cargo.toml

@@ -10,6 +10,10 @@ description = "rbitcoin RPC CLI client"
 [[bin]]
 name = "rbitcoin-cli"
 path = "src/main.rs"
+test = false
+
+[lib]
+doctest = false
 
 [features]
 mutants = []

crates/rbitcoin-consensus/Cargo.toml

@@ -13,6 +13,9 @@ corpus = ["dep:serde_json"]
 # Nightly cargo-mutants only. Off for the node build.
 mutants = ["dep:mutants"]
 
+[lib]
+doctest = false
+
 [dependencies]
 rbitcoin-primitives = { workspace = true }
 arc-swap = { workspace = true }

crates/rbitcoin-electrum/Cargo.toml

@@ -10,6 +10,9 @@ description = "Electrum protocol server (scripthash JSON-RPC over TCP)"
 [features]
 mutants = []
 
+[lib]
+doctest = false
+
 [dependencies]
 rbitcoin-primitives = { workspace = true }
 rbitcoin-log = { workspace = true }

crates/rbitcoin-esplora/Cargo.toml

@@ -10,6 +10,9 @@ description = "Esplora-compatible REST (electrs HTTP drop-in except address-pref
 [features]
 mutants = []
 
+[lib]
+doctest = false
+
 [dependencies]
 rbitcoin-primitives = { workspace = true }
 rbitcoin-log = { workspace = true }

crates/rbitcoin-log/Cargo.toml

@@ -7,6 +7,9 @@ rust-version.workspace = true
 repository.workspace = true
 description = "Minimal leveled stderr logger and long-stage progress registry (no external deps)"
 
+[lib]
+doctest = false
+
 [dependencies]
 
 [lints]

crates/rbitcoin-mempool/Cargo.toml

@@ -7,6 +7,9 @@ rust-version.workspace = true
 repository.workspace = true
 description = "Cluster mempool with InRam buffers + private sidecar files under datadir/mempool/"
 
+[lib]
+doctest = false
+
 [dependencies]
 rbitcoin-log = { workspace = true }
 rbitcoin-consensus = { workspace = true }

crates/rbitcoin-net/Cargo.toml

@@ -9,6 +9,12 @@ description = "Bitcoin P2P (BIP324, IBD, tip follow, compact v2, tx relay)"
 
 [features]
 mutants = []
+# Journeys turn this on. The node binary does not, so its redial pass stays
+# 2s and a SAM retry stays 500ms.
+test-hooks = []
+
+[lib]
+doctest = false
 
 [dependencies]
 rbitcoin-primitives = { workspace = true }

crates/rbitcoin-net/src/chain.rs

@@ -1817,19 +1817,33 @@ impl ChainHub {
             .into_iter()
             .map(|(h, raw)| (h, BlockHash::from_byte_array(raw)))
             .collect();
+        // Held across the write so the chain tip cannot move while the
+        // relay-on purge is deciding which tip it stripped.
+        let mp_hold = self.mempool().cloned();
+        let mut template_hold = mp_hold.as_ref().map(|mp| mp.template_hold());
         confirm_write_phase(&self.query, &self.params, self.milestone, batch)
             .map_err(NetError::from_confirm)?;
-        self.note_confirmed_tip(&meta, &headers)?;
+        self.note_confirmed_tip_while(&meta, &headers, template_hold.as_mut())?;
         Ok(meta
             .iter()
             .map(|&(height, _)| AcceptOutcome::Accepted { height })
             .collect())
     }
 
+    #[cfg(test)]
     pub(crate) fn note_confirmed_tip(
         &self,
         need_meta: &[(u32, BlockHash)],
         headers: &[Header],
+    ) -> Result<(), NetError> {
+        self.note_confirmed_tip_while(need_meta, headers, None)
+    }
+
+    pub(crate) fn note_confirmed_tip_while(
+        &self,
+        need_meta: &[(u32, BlockHash)],
+        headers: &[Header],
+        mut hold: Option<&mut crate::tx_relay::TemplateHold<'_>>,
     ) -> Result<(), NetError> {
         if headers.len() != need_meta.len() {
             return Err(NetError::store(rbitcoin_store::StoreError::Corrupt(
@@ -1838,18 +1852,28 @@ impl ChainHub {
         }
         if let Some(mp) = self.mempool() {
             mp.clear_recent_rejects();
+            let mut stripped: Option<[u8; 32]> = None;
             if mp.relay_enabled() {
                 for &(height, hash) in need_meta {
                     mp.note_block_fee_history(Height(height));
-                    if let Ok(Some(block)) =
+                    let Ok(Some(block)) =
                         self.query.reconstruct_block_by_hash(&hash.to_byte_array())
-                    {
-                        mp.note_recent_confirmed(&block.txdata);
-                        let ids: Vec<_> = block.txdata.iter().map(|t| t.compute_txid()).collect();
-                        let spent = spent_outpoints(&block.txdata);
-                        let n = mp.remove_for_block_spent(&ids, &spent);
-                        rbitcoin_log::debug!("mempool: removed {n} confirmed tx(s) @ {hash}");
-                    }
+                    else {
+                        break;
+                    };
+                    mp.note_recent_confirmed(&block.txdata);
+                    let ids: Vec<_> = block.txdata.iter().map(|t| t.compute_txid()).collect();
+                    let spent = spent_outpoints(&block.txdata);
+                    let n = mp.remove_for_block_spent(&ids, &spent);
+                    rbitcoin_log::debug!("mempool: removed {n} confirmed tx(s) @ {hash}");
+                    stripped = Some(hash.to_byte_array());
+                }
+            }
+            if let Some(hash) = stripped {
+                if let Some(hold) = hold.as_mut() {
+                    hold.commit(hash);
+                } else {
+                    mp.note_template_prev(hash);
                 }
             }
         }
@@ -2718,6 +2742,11 @@ impl ChainHub {
         if keep_height > tip {
             return Err(NetError::Protocol("rewind above tip"));
         }
+        // Before the commit lock, same order as `disconnect_to`. This rewind
+        // already holds that lock, so it cannot call `disconnect_to`, and the
+        // mark would stay on the block that was just disconnected.
+        let mp_hold = self.mempool().cloned();
+        let mut template_hold = mp_hold.as_ref().map(|mp| mp.template_hold());
         let _commit = if wait {
             self.query
                 .lock_tip_commit()
@@ -2728,6 +2757,12 @@ impl ChainHub {
             return Ok(false);
         };
         self.disconnect_to_in_commit(keep_height)?;
+        if let Some(hold) = template_hold.as_mut() {
+            if let Ok(Some((_, rec))) = self.query.header_at_height(Height(keep_height)) {
+                hold.commit(rec.hash);
+            }
+        }
+        drop(template_hold);
         Ok(true)
     }
 
@@ -3073,6 +3108,10 @@ impl ChainHub {
             crate::tip_accept::on_tip_accept_thread(),
             "connect_at must run on tip-accept"
         );
+        // Held until this connect returns, so a template cannot observe the
+        // new query tip before the pool strip is recorded.
+        let mp_hold = self.mempool().cloned();
+        let mut template_hold = mp_hold.as_ref().map(|mp| mp.template_hold());
         let hash = block.block_hash();
         let header = block.header;
         tip_accept_stats_reset(&self.query);
@@ -3142,6 +3181,9 @@ impl ChainHub {
             rbitcoin_log::debug!("mempool: removed {n} confirmed tx(s) @ height {height}");
             if mp.relay_enabled() {
                 mp.note_block_fee_history(Height(height));
+                if let Some(hold) = template_hold.as_mut() {
+                    hold.commit(hash.to_byte_array());
+                }
             }
         }
         let mp_strip_ns = t_mp.elapsed().as_nanos() as u64;
@@ -3154,6 +3196,9 @@ impl ChainHub {
         // uses note_confirmed_tip without this line; IBD retains periodic status.
         info!("{}", log_update_tip_line(height, &hash, &header, tx_count));
         log_tip_accept_sh(&self.query, height, tx_count, wall_ns, mp_strip_ns, pres_ns);
+        // Record the stripped tip before the announcement, so a builder that
+        // wakes on this event already sees the new mark.
+        drop(template_hold);
         let event = TipEvent {
             height,
             hash,
@@ -3168,11 +3213,23 @@ impl ChainHub {
     }
 
     fn disconnect_to(&self, keep_height: u32) -> Result<(), NetError> {
+        // Before the commit lock: the relay-on purge takes this same mutex
+        // and then reads the query. The pool matches `keep_height` once the
+        // disconnected transactions are back in it.
+        let mp_hold = self.mempool().cloned();
+        let mut template_hold = mp_hold.as_ref().map(|mp| mp.template_hold());
         let _commit = self
             .query
             .lock_tip_commit()
             .map_err(|e| NetError::Store(e.to_string()))?;
-        self.disconnect_to_in_commit(keep_height)
+        self.disconnect_to_in_commit(keep_height)?;
+        if let Some(hold) = template_hold.as_mut() {
+            if let Ok(Some((_, rec))) = self.query.header_at_height(Height(keep_height)) {
+                hold.commit(rec.hash);
+            }
+        }
+        drop(template_hold);
+        Ok(())
     }
 
     fn disconnect_to_in_commit(&self, keep_height: u32) -> Result<(), NetError> {
@@ -6246,7 +6303,7 @@ mod tests {
         {
             let hub = hub.clone();
             thread::spawn(move || {
-                let until = Instant::now() + Duration::from_secs(2);
+                let until = Instant::now() + Duration::from_millis(200);
                 while Instant::now() < until {
                     assert_eq!(hub.best_header_height(), 2);
                     assert!(hub.chaintips().iter().any(|t| t.hash == tip));

crates/rbitcoin-net/src/i2p_sam.rs

@@ -9,6 +9,14 @@ use std::sync::Mutex;
 use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::net::TcpStream;
 
+/// Pause before a SAM retry. Thirty-two session attempts are about 16s, so a
+/// router that is not accepting yet still comes up.
+#[cfg(not(any(test, feature = "test-hooks")))]
+const SAM_RETRY: std::time::Duration = std::time::Duration::from_millis(500);
+/// Crate tests and journeys. The node binary keeps 500ms.
+#[cfg(any(test, feature = "test-hooks"))]
+const SAM_RETRY: std::time::Duration = std::time::Duration::from_millis(50);
+
 static INSTALLED: Mutex<Option<Installed>> = Mutex::new(None);
 
 struct Installed {
@@ -193,7 +201,7 @@ impl I2pSam {
                 Ok(v) => return Ok(v),
                 Err(e) if sam_retry(&e) => {
                     last = Some(e);
-                    tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                    tokio::time::sleep(SAM_RETRY).await;
                 }
                 Err(e) => return Err(e),
             }
@@ -274,7 +282,7 @@ impl I2pSam {
                 }
                 Err(e) if stream_retry(&e) => {
                     last = Some(e);
-                    tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                    tokio::time::sleep(SAM_RETRY).await;
                 }
                 Err(e) => return Err(e),
             }
@@ -312,7 +320,7 @@ impl I2pDialer {
                 Err(e) if stream_socket_dead(&e) => return Err(e),
                 Err(e) if stream_retry(&e) => {
                     last = Some(e);
-                    tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                    tokio::time::sleep(SAM_RETRY).await;
                 }
                 Err(e) => return Err(e),
             }