Repository navigation
ibd: prefetch next-batch parent body ranges (needs Linux measurement) - #1299
Closed
Hero-Gamer wants to merge 5 commits into
Closed
Hero-Gamer wants to merge 5 commits into
Hero-Gamer wants to merge 5 commits into
Conversation
- PeerHub::new(clock: Arc<NodeClock>) where NodeClock::new()->Arc<Self> - Remove mock_now/set_mock_now second battery from PeerHub - Add on_clock_jump() that requests tx INVs, runs heartbeat, queues self-announce - service.rs: share hub.clock.clone() so setmocktime reaches peers via same Arc - mine.rs: setmocktime writes ChainHub clock once, then peers.on_clock_jump() - Migrate journey tests: hub.clock.set_mock(x) + on_clock_jump() Fixes second mock battery noted in reardencode#821 Step 2. No prod behavior change: clock jump now triggers same INV + heartbeat as before, but via single NodeClock source. Tested: 660 rbitcoin-net journey tests pass, fmt + clippy clean.
Add WILLNEED wrapper next to existing DONTNEED, for prefetching next-batch parent body ranges during IBD. Linux only, no-op elsewhere, trace on failure. No behavior change yet.
- store: add advise_will_need (WILLNEED) next to DONTNEED - query: add ParentPinStamp::body_ranges_iter() for next-batch prefetch No behavior change yet, helpers for gated prefetch.
- query ConfirmWindow: prefetch_bytes, prefetch_ns via confirm_window! macro - ibd perf_log: prefetch_bytes, prefetch_ns fields + mapping from window No behavior yet, just counters to measure Grok's prefetch idea. Gated prefetch will populate these when pin_new !=0 and batch >1MiB.
…ripts/write - store: TableFile::advise_will_need (WILLNEED wrapper), VarTable/TxTable/Store wrappers - query: ParentPinStamp::body_ranges_iter helper + prefetch_parent_body_ranges() Gated: total_bytes > 1MiB to avoid fadvise overhead on small batches Records prefetch_bytes/ns into ConfirmStats, exposed as ibd:perf prefetch_bytes/ns - ibd: perf_log mapping for new counters - No behavior change on non-Linux (no-op), trace on fadvise failure - Expected win: 5-10% less cold_range_body stall when pin_new!=0, overlaps kernel readahead of N+1 parent bodies with scripts(N) / write(N-1) Implements Grok's prefetch idea, 1st behavior commit. Keep under 150 lines.
Contributor
Author
|
Superseded by #1300 clean scoped PR (7 files 106 ins, coalesced, gated). Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Idea (from Grok)
Prefetch next-batch parent body ranges while current batch is in scripts/write stage, to reduce
load_cold_range_body_msstall.Overlaps kernel readahead of N+1 batch with
load(N) || scripts(N-1) || write(N-2).Impl (85 insertions, 42 behavior)
file.rs:advise_will_need- Linuxposix_fadvise(WILLNEED)wrapper, trace on failure, no-op on Maclookup.rs:body_ranges_iter()+prefetch_parent_body_ranges()advise_tx_body_will_needfor each range in parent_pinprefetch_bytes,prefetch_nsconfirm_stats.rs+perf_log.rs: counters wired toibd:perfJSONvar_table.rs/tx_table/store.rs: plumbingLinux-only, no behavior change on Mac.
Status
cargo clippy -p rbitcoin-query -p rbitcoin-store -p rbitcoin-ibd --lib -- -D warningsNeeds
Linux signet IBD measurement, two runs with
RUST_LOG=ibd:perf=debug:ee66131a(fix/peerhub-single-clock-v2)Compare
load_cold_range_body_mswhenload_pin_new>0vsprefetch_bytes/prefetch_ns. Expect 5-10% win.Can someone with Linux box run this and share
ibd:perflogs?Diff
git diff ee66131a..HEAD --stat= 7 files, 85 insertionsBehavior commit: 42 lines