Short map of who may write which tables. Format is unstable until 1.0.
| Role | Threading | Store writes |
|---|---|---|
| Peer IO (N tasks) | tokio multi-thread | none; decoded blocks offer body queue only; note height/hash readiness on confirm feed |
| Confirm lookup | 1 OS thread | pack/hold from BQ stamped Σ inputs; emit: decode BQ raw + TipOnly head_fk + take_raw onto loadq (Arc<Block> + TxPrecompute). Does not plan_batch / structure |
| Confirm load | 1 OS thread | confirm_wire_lookup_stamp (structure with loadq pres, plan_batch binds carried BQ keys, skeleton bind) + pin txout + assemble |
| Confirm scripts | 1 OS thread (ibd-confirm) publishes waves + rbtc-scripts steal (lock-free claim). The interpreter runs here, not on a peer IO task (verify_tx_scripts_detached) |
none — pure CPU |
| Confirm write | 1 OS thread (ibd-confirm-write) + 1 process-wide ibd-confirm-head |
sole Class A appender (txout+seqsigwit+spent; IBD encodes ins from Arc<Block> + SpendEdges) + structural + Class C + spend annotate on spent.body + tip GC; block_queue_dequeue_height. tx.head write-behind insert runs on ibd-confirm-head overlapping structural + Class C (not a per-batch spawn). Class A never leads tip (same commit era; no archive-ahead DONTNEED). When index_live is set, this thread appends that index after confirmed[h] is set (one put per confirm batch) and does not read Class A for it. Startup prepare_live_indexes seals a restart gap of at most one write drain (14 × 144 heights) before confirm; a larger gap is left for rbtc-idx-wb |
| IBD main loop | 1 tokio task | none (orchestration only). Event drain + confirm-offer every turn; getdata assign ≤50 ms (immediate if inflight empty); main-loop getheaders/locator_hashes ≤500 ms (empty path immediate); peer stall/relative-slow and work-path hygiene ≤1 s. Full-batch header continuation stays on the Headers event. Tip hash is the snapshot confirm published with confirmed[] (published_tip); each admitted header's parent stays on the work state. Those two reads are not store IO. The write holds tip_commit from the tip-height check through flush_class_c_tip. IBD rewind tries that lock and retries on the next pass. |
IoSession TLS: one completion session per OS thread (with_thread_local). Harvest / poison / drain / do-not-flatten: io-modality.md. RBITCOIN_IO=pread disables the session. SH k-way merge submits 256 KiB ahead preads on that TLS session and waits only when promote needs a page that has not completed.
Height-ordered unified pipeline (current): peer → body queue (raw only) → lookup (in-order from max(path_lo, lookup_taken_hi+1); decode + TipOnly head_fk; dequeue raw into loadq=14). Hole/densify/receive: in-hand = confirmed ∨ BQ hash ∨ H ≤ lookup_taken_hi (Cancelled leaves taken_hi; every other post-lookup reject re-arms it at the confirmed tip; a load stamp, scripts, or write EngineFault re-offers the full batch including the first height after that re-arm and does not request_single_block; a load pin EngineFault retries one block at a time, as a cascade, since it can be a stale-fk plan race; a load reject bumps the feed epoch, so valid lower batches already in scripts or write are dropped and their bodies fetched again from peers — the price of a consistent restamp; epoch-stale loadq chunks re-offer on drop; BadPrev most-work rewind then sets the LCA) → load (recv load-sized batch; stamp binds carried BQ keys + BQ header_fk (skip header ensure/put when the store row matches) + pin + assemble) → scripts → write. Stage IO: invariants.md. No peer→confirm-feed wire retain. No hash-only / Class-A-only confirm (bq wire required). Load bind order is same-batch → in-flight → load-batch skeleton → Corrupt (plan=None leftover TipOnly). Lookup does not own a published identity chain. Class A sets loc once on the same CreatePin Arc load cloned into InFlight (OnceLock); later-wave stamp reads it (no disk loc-by-fk on that path). Header records are 88 B consensus fields. Block size and weight are stamped on txstat, not rewritten into header.body. Write drain inserts tx.head in parallel with Class C on the process-wide ibd-confirm-head thread. Drain complete is max inserted fk (not tip/fence — those advance during drain). Header-cache GC polls store tip every load pack. In-flight prune is after the last load batch of a lookup wave finishes its in-flight read, using the drain+fence height snapshotted before that wave's TipOnly (docs/invariants.md). No leftover pending map. Bodies without a known height are marked missing and re-getdata after the height map is ready — there is no dual-track archive-job / ContigPark fallback. Load pack waits on feed.cv when tip+1 is in ready but the BQ is not resolve-complete (no retain/BQ spin). Pack takes the feed mutex to collect candidates and again to mark inflight; one BQ pack_snapshot in between.
Load claim pack size: soft Σ tx.input budget (hardcoded 8000; include overshoot block) or hard 144 blocks. Also stop a LoadBatch before the next height when header_txs.has_body differs from the part's first height (crash some→none). Lookup may still decode a mixed resolve wave; loadq chunks are one kind. Dense mainnet blocks hit the input soft stop after typically a few blocks (often 1–3); early tiny blocks may pack many until the hard cap. Do not treat ~32 as pack size (that was 8000/250 mid-chain, not fat-era).
IBD lookup resolve wave: TipOnly head_fk bounded by remaining loadq slots × load pack (safety cap 1080 heights / soft 64000 inputs; include-overshoot; 256 k unique-key safety cap). Decode parks as resolved BQ rows; taken_hi bumps per load-batch send. Unsent tail stays on the BQ (no re-decode). Hard min 8000 inputs per wave when more unresolved heights can still join — including ready=0 / load-frontier / unknown window. Last available thin wave still emits. Each load chunk carries a BatchParentIds skeleton (shared wave ids/spent + per-chunk need-vouts). Same-wave creates are omitted from TipOnly need. When ready > half the 1-min BQ window, lookup waits for a full wave instead of minting a 1-block layer — unless the first unresolved height is within path_lo + win/2 and the collect is already ≥8000 inputs (load is about to claim it; O(1) from the already-sorted unresolved list). Pack/hold uses enqueue-stamped n_inputs (peer CompactSize walk). wave_intake does not clone raw payloads; decode clones raw_payload(height) only on emit. Hold is decode=0 / precompute=0. lookup_thr wave=(decode= precompute= collect= head= loc=): precompute= is from_tx_wire on payload slices (sighash midstates only when scripts run); head= is TipOnly get_fk_by_txid_batch, not load stamp; loc= is create.loc fill inside that TipOnly.
ibd: perf: one DEBUG line, ibd: perf plus a JSON object of IbdPerfSample (ts is unix milliseconds; zeros are included). A change on lookup, load, scripts, or write, or on a sidecar the write thread joins, adds a field on that struct in perf_log.rs in the same commit. The field shows up because the struct serializes. JSON keys are the field names (load_ms, not a hand-written load= token). load= is pin+assemble only. Load OS-thread stamp= nests pack= (plan HashMap) vs head= (leftover TipOnly prep_head_fk_ns). IBD skeleton keeps head= ~0. stamp_sub struct_txid= must be 0 on IBD (loadq pres). Non-zero means load dropped lookup hashes and from_tx'd again. Post-stamp in-flight drop on the last load batch of a wave is prune= (height-index remove; IBD has no pstore Weak walk). A lone block's missing-parent check after a stamp miss (the wait for its parent to be the tip, the TipOnly read, and the unsynced-head scan) is reject=, not prune= (089). Lookup-wave decode= / precompute= / collect= / head= / spent= nest under lookup_thr wave=. Pin names thin= for the vout-map prefix. script= is per-batch verify wall on ibd-confirm (jobs=/skip=), the script-wave completion offset. idx_asm= is filter and tweak execution on the index wave (first index job through that wave's completion). It is not time queued behind scripts and it is not inside script=. thr script work is the stage wall: the later of those two completion offsets from batch start, not their sum. idx_put= is the write-thread index append and is inside write=. Steal claim is an ArcSwap snapshot + fetch_add(32) (no WAVES mutex per job; in_wave counts in-flight chunks). The scripts thread publishes another scriptq batch when both waves have nothing left to claim (up to 4 in-flight, matching scriptq) and parks until a worker unparks it on wave complete (load also unparks on scriptq send). script= stamps when script verify completes, not at write-queue pop. Write snapshot-drains the whole writeq (cap 14) so one flush_class_c_tip covers the queued run. append_contiguous stops when archive_plan polarity differs (Some vs None); leftover is the next meta-batch. ready>0 + scriptq=1 + high stamp= head= means leftover on load, not a hungry script pool. High stamp= pack= with head=0 is HashMap CPU on load (leave it there; lookup is already the wave). Do not retune steal or borrow rbtc-scripts-* for decode. Process restart leftover is empty RAM identity (horizon does not survive).
Tip follow / reorg: peer reconstructs on the session task, then offers the body to the process-wide tip-accept OS thread (bounded queue 8, oneshot wait). That thread is the sole production owner of connect_lock / accept_and_connect_block_preverified / confirm_wire_run_preverified at tip (1-block, or one accept_branch run). Scripts still publish via start_for_each_owned → rbtc-scripts-* steal — same as IBD’s publisher, not a second interpreter. Disconnect keeps Class A archive; re-extension always supplies wire from the peer, not hash-only load. IBD most-work reorg rewinds the tip to the LCA (rewind_to_height) from the IBD orchestration task only — never from confirm lookup/load/scripts/write threads — then the confirm pipeline connects the winner as a linear extension. accept_branch remains for tip-follow / held-body apply (also on tip-accept). Selector / rewind / invalid-heavy: architecture.md.
Wire retained on the pipeline batch only: lookup/load pull bitcoin::Block from the body queue; that wire rides through scripts; no Class-A wire rebuild. Script jobs carry lookup TxPrecompute (no job from_tx). Split Class A (txout / seqsigwit / spent) is planned once and committed in the write stage.
Body queue: process-local in-RAM FIFO (id / height / hash / header_fk / raw charge) plus a first-wins height → id map (one row per height). has_hash is a hash→height map on that same mutex (no full-queue scan). Lookup's decoded stash is a second mutex: getdata enqueue and has_hash never take it, and row reads never take it. Height APIs (is_resolve_complete, get_by_height, hash_at_height, pack snapshot) do not walk index.values(). Peer copies wire then locks and enqueues raw; lookup decode borrows that queued Arc (it does not memcpy the frame) and parks resolved, then dequeues on load-batch send into loadq (LoadBatch). Load stamp consumes that pres — it does not read block_queue_resolved. Never both raw and decoded. Offer on peer Block → RAM (wire the input walk cannot decode is refused, never queued); lookup takes; write dequeue is a no-op if the row is already gone. A raw row that does not decode stops the lookup wave at that height; lookup dequeues it and emits a SoftWire reject so the hash is fetched again (never blacklisted). Restart starts empty (legacy store/block_queue/ is best-effort removed). Soft densify assign, never-refuse-enqueue, why RAM: ibd-memory.md.
Pipeline pins: plan batch_pin / BatchParents only (no process create FIFO). freeze_after_pin runs before pin and drops staging maps pin does not read (external_parents, batch_creates). Pin reads edges, batch_pin, planned_fks, and per_header_ranges. Write concatenates batch_pin / planned_fks / SpendEdges. Wire packed ins stay empty; write encodes seqsigwit from the wire transaction. A records row with empty ins is Corrupt. Outs live in BatchParents — not a third plan-local outs map. SharedParentPin vacant insert stores Frozen outs/layout halves; first real compose promotes that half to ArcSwap (RCU; no-op cover stays Frozen; no in-place mutation). BatchParents sticky-caches the last outs Arc for multi-input assemble. get_parent_txout_parts holds that Arc and yields &[u8]; IBD assemble below milestone counts sigops from the borrow (no ScriptBuf::from_bytes). Script jobs borrow script bytes (same-block from the wire block on the job, historical from the pin Arc) and do not clone scriptPubKey. Taproot sighash hashes those borrows inside script= and does not build TxOut. Pack pending_spent is OutPointSet (folds every hasher write — TxidHasher would drop the txid when vout arrives). IBD pin(... range_fill= contract=) names residual pin wall. ConfirmParentCache holds tip-ahead Arc header plans only (insert/replace/drop under tip GC). In-flight lifetime / prune: invariants.md. Not a coins cache / spend FIFO.
tx.head (segmented): see heads.md. Lookup: live pin by
txid → hot (open + ages ≤3) → ID/idx → cold (ages ≥4) if needed.
The 2-wave split is sealed age, not an IO flag.
Datadir secret (schema 12): store/store.secret CSPRNG at create. XOR scripts/witness at rest; keyed TXID mix for heads.
Three thread kinds only. Tokio workers must not wait on a std mutex/rwlock, store IO, or script verify (assert_not_reactor / BlockingRegion on the blocking pool). Tokio names both workers and spawn_blocking threads tokio-rt-worker; blocking work enters BlockingRegion.
| Kind | Who | Notes |
|---|---|---|
| Reactor | tokio-rt-worker × nCPU |
epoll, timers, framing. Node caps spawn_blocking at nCPU (min 4), not tokio’s 512. |
| Role | tip-accept, ibd-confirm-*, rbtc-sh-wb, rbtc-idx-wb |
Sole writer for a resource. Not a job pool. |
| Work pool | rbtc-scripts-* steal; tokio blocking pool |
Independent jobs. Mempool accept / Electrum / Esplora / RPC run here. |
| Role | Notes |
|---|---|
peer_session (split read/write) |
Serve + reconstruct compact/body. Offers reconstructed blocks to tip-accept (does not take connect_lock or run confirm on the tokio worker). A reconstructed/received body whose header has valid PoW and extends the current tip is announced as cmpctblock to peers who sent sendcmpct announce=1 and already have the parent before connect (Core NewPoWValidBlock). That is not a connected tip. P2P tx is accept_tx_async (blocking pool). INV / getdata / compact first-pass fill use mempool try_read only (busy write → skip that item; never park). A pending compact owns those clones; blocktxn apply does not try_read again. Handshake FeeFilter reads the atomic published fee floor (no inner). 50 ms tick: age-INV is a due-log cursor (newly due only); clock_due / unbroadcast may full-walk ≤1/30 s. any_tx_inv_due is min live accept_at, not a map walk. PeerHub::on_session_heartbeat (headers-sync stall timeout). While the writer is past the 4 MiB send budget, or a paused getdata tail waits for serve room, the reader reads no frame; the heartbeat still runs, so the ping timeout still fires. Inbound accept is inbound_connect_and_handshake (60 s VERSION/VERACK); timeout drops the max_inbound permit. |
tip-accept |
One process-wide OS thread. Queue depth 8. Sole production thread for accept_block / accept_branch / accept_received_block / generate_to_script / connect_lock at tip. Confirm is confirm_wire_run_preverified (lookup stamp, load pin/assemble, confirm_scripts_phase → rbtc-scripts-*, write + ibd-confirm-head drain). TLS uring is this thread’s with_thread_local session. SIGINT stays on tokio; the current job finishes, then the session sees shutdown. Dropping the session’s join future detaches (does not condvar-wait on the worker). |
rbtc-sh-wb |
One Class B scripthash appender. Used for tip follow and short catch-up when a durable SH head already exists. Confirm enqueues RAM records; connect_at / note_confirmed_tip release after tip_tx. This thread put_create_batch_append only for released heights, then advances sh_indexed_through. Apply errors re-queue and halt. Post-IBD Class A collect uses pack sessions (not a second appender); it runs while still Direct so this thread no-ops. |
rbtc-idx-wb / rbtc-idx-cpu |
One block index builder for BIP158 basic filters (--block-filter-index) and BIP-352 tweaks (--sp-tweaks), spawned at tip entry only when an enabled watermark is still behind the tip after startup repair. Live batches append on the confirm write thread; this reader does not walk a chain that was indexed as it connected. Waits on the same release gate as rbtc-sh-wb (index_released_through). The IO thread plans windows (≤64 heights / ≤50k creates) from the lower index watermark to the released tip and reads each with read_index_window on one completion session (locators, block spans, parent locators + P2TR-output seqsigwit + parent txids, parent records). One CPU worker, at most two windows behind, publishes one job per height to rbtc-scripts-* (filter GCS and tweak EC). That wave claims one height, so a short window still spreads across the pool. The CPU worker commits each index once per window under the index write-behind mutex, only if its watermark and every confirmed[h] are unchanged; disconnect_tip_with takes that mutex before truncating both. A failed commit (reorg) makes the IO thread re-plan from the watermarks. Apply errors halt the node. |
| Electrum / Esplora | Confirmed SH reads join durable index plus a RAM SH head (pending jobs keyed by scripthash) and pin that visible height (live tip while jobs sit, never above published tip). A tx is in mempool overlay or SH (pending/durable), not both and not neither. Reorg reaccepts then drops pending. Headers subscribe is live tip. |
| Epoch finalize | Single-threaded control path; flushes table maps / fd durability |
| Mode | When | Spentness | Durable tx.head / spends |
SH |
|---|---|---|---|---|
| Direct | IBD (enter_direct_index_mode) |
confirmed-strong annotations | commit-stage head insert; spend annotate in same stage | Class A collect → unsorted shards → seal at tip |
| Tip | after IBD (enter_tip_mode) |
confirmed-strong annotations | live heads + confirm spends | write-behind after a pack-complete head (every scripthash.head/NN.packed; may lag live tip by 1+ blocks) |
Do not enter Tip until IBD catch-up complete: no best-chain remainder
(ordered / height_to_hash above tip / BQ ready ahead / awaiting reorg /
on-path getdata) and path high water at or within 1 of max peer height
(one-block version chatter only when headers_done), or a proven header
walk at the tip that no connected peer can extend. Competing
hash_height and leftover explore getdata are not remainder. Tip entry
bulk-materializes SH (two Class A txout scans → identity-map key spills
then fuse-hit postings → BDZ scripthash.head/NN; extract workers capped at
one per 1.5 GiB host free RAM). Shared file scripthash.body
is one writer. Overflow body
is one writer (ingest / compact). It does not rebuild tx.head or
spend annotations.
Default is lock-free on table hot paths:
| Mechanism | What it replaces |
|---|---|
Capacity grow (TableFile) |
Grow with fallocate/set_len and a published HWM. Do not introduce remap-epoch schemes (retired example: map epochs). Readers use the published HWM (Acquire) |
Atomic count / HWM |
Publish barrier (Acquire readers / Release appender) |
| Role exclusivity | One appender, one annotator — not a global store mutex |
tx.head insert |
Sole writer: page-coalesced pwrite + published_len Release (no CAS, no CPU fence). Role exclusivity — not multi-inserter safe |
tx.head segment seal |
Roll opens the next OA immediately; BDZ+fuse8 runs on a sidecar. Lookup probes every unsealed OA until publish. Write joins the sidecar only on the next roll, flush, or Drop (not on the rolling insert). |
header.head overflow |
Insert past 7/8 rolls header.head.gN (new empty file). Occupied rewrite is open-only: undersized single gen writes header.head.grow then rename. |
ChainHub::confirmed |
RwLock<HashSet> for O(1) has_block (IBD assign path) |
MempoolHub::inner |
One RwLock<ActiveMempool>. Write is graph mutation + durable slot append only (microseconds at ~12k live). No Query / script verify / compact reconstruct under write. The commit recheck of a confirmed coin runs before the write lock unless a confirm or disconnect holds the UTXO view. Prepare may hold a read across Query. Blocking read/write is assert_not_reactor. Session INV/getdata/compact use try_read; miss = skip that item (re-getdata / notfound / reconstruct without those txs). Compact fill siphashes live txid/wtxid and clones matching bodies only — never list_live(). Handshake FeeFilter reads the published fee-floor atomic, not inner. Age-INV log and min live accept_at are not inner (same class as accept_at). Accept caches {tip_header_fk, height, mtp} until the header fk changes. get_coin resolves the parent on the connected chain, decodes its packed outputs once, reads coinbase-ness from the Class C coinbase table (a failed read leaves the coin unavailable), and computes create MTP only for BIP68 time locks. index_txid hashes the admitted tx + prepare prevouts (no second Query). Do not CoW a 12k-txid presence set on every accept. Lock order: inner → sh_index / unbroadcast / relay_*. |
There is no global “pause queries during confirm write.” Tip-as-commit +
is_confirmed_strong define query visibility (crash-recovery.md).
Wallet APIs (Electrum / Esplora) must not mix two chain prefixes in one
response, and they must tell the client which tip hash the body belongs
to. Yuval brought the A-B-A hole to our attention (same height, different
block, silent status). The shape we ship follows
mempool/mempool#6584
(HTTP tip-hash header on every response) and the Electrum 1.7
chaintip discussion in
spesmilo/electrum-protocol#2
(added, then reverted in
#17 because ElectrumX
cannot pin bitcoind RPC — we can).
| Rule | Detail |
|---|---|
| Pin | Query::pin_chain_view captures {height, hash, header_fk} of published tip |
| Buried / as-of | pin_chain_view_at(hash) for a still-live ancestor. Esplora ?asof=; Electrum trailing asof:<hash> after server.version dialect 1.4.2-asof. Stamp is that hash. If it leaves the tip chain: 404 / asof not on chain — do not retry onto another block at the same height |
| Filter | SH join uses is_confirmed_strong_at(fk, view.height); slot keys on hash |
| Live-check | ChainView::still_live ⇔ confirmed[height] == header_fk |
| Extension | Prefix pin stays live; creates above the pin are filtered |
| Disconnect / same-height replace | Live pin dies; run_at_chain_view retries (bound 8) then StoreError::Stale |
| Not OK | Pause queries during write, MVCC Class C, serving a disconnected hash |
API tokens: COMPAT.md (Esplora headers, Electrum JSON-RPC extra members, status preimage).
- Do not spawn a second Class A writer while IBD confirm write is running.
- Pipeline depth: lookup(N+1) ∥ load(N) ∥ scripts(N−1) ∥ write(N−2) via BQ
ready=+ bounded scriptq/writeq. - Scripts for batch N may run while load does N+1 and write does N−1. Scripts never touch disk.
- Load ahead of store tip: lookup may stamp batch N+1 while write has not advanced tip.
The load thread holds a reserved create-fk HWM and the in-flight map from
uncommitted plans (
WireLoadPipeline/archive_plan_batch_from_wire). First height of a batch is the pipeline path_lo (tip+1 or last-loaded+1), not only store tip. Write still applies batches in height order; on permanent reject, lookup clears reserved state and re-syncs fromtxs.count(). - On SIGINT, IBD cancels cooperatively — do not drop nested runtimes mid-await.
Single Class A writer is intentional. Multi‑GiB FdOnly grow is fallocate-only
(no remap). Hash heads do not rewrite occupied tables while serving; a leftover
undersized header.head may be rewritten via header.head.grow then rename once on open. Class C tip tables use L2
write-behind (flush_class_c_tip before BQ dequeue); large tables stay L0.
See io-modality.md for operator IO levers.
Cold parent create.loc / txout.body on the load thread uses
FdOnly loc + bulk body (idx_body_pipeline → bulk_io uring/pread). Batch
creates come from wire, not a second Class A full-decode pass.