Repository navigation
perf: make prov fast without a persisted cache - #24
Merged
Merged
Conversation
The walk now reads each document's spanning children before resolving them, and parses their frontmatter and body links in parallel into the read memo. The walk itself is unchanged and meets them as memo hits. A body's links are remembered beside the document they were found in, so the census and the inbound index stop parsing the same prose twice.
… shares its reads The prefetch reads and parses each child in one task on rayon's pool when ReadStorage::shared_reader hands over a reader, and the tree descent prefetches its children too, now that it can read them together. Needs fs-transaction with shared_reader (diaryx-org/fs-transaction agent/shared-reads); the workspace takes it from a local patch until that is released.
find_front parses only what locating the frontmatter takes, which is what prov's fenced_prefix did on its side, so the workaround goes. The body scan asks nodes_of_kind for the six kinds it sorts rather than copying out every node. Needs twig 7.1; the workspace takes it from a local patch until that is released.
Graph::read hands back an Arc<Loaded>, the document's text and parse as one read found them, and the memo stores that Arc: a second read in one operation is a reference count, where load copied both. A prefetch moves what it built into the memo instead of copying it in. load keeps its signature, as an owned copy of read, and the walk, the tree descent, the check passes, the inbound index and the views' documents and corpus read shared.
Off by default: mimalloc is C, and building prov from source needs a Rust toolchain and nothing else. The release passes cargo-features: mimalloc, which needs diaryx-org/.github's reusable-homebrew to take that input.
The prefetch read one parent's children per batch, which on a tree three children wide is a batch of three: rayon woke every worker for it, and a deep journal walked slower than with no prefetch at all. The walk and the tree descent now prefetch once, before they start, a level per batch, so the batches are large whatever the tree's shape. A batch under sixteen is read in place, and the prefetch stops when the memo's budget is spent.
…ets in bulk missing_containment and orphans asked an ordered set whether each document on disk was reached, a dozen component-by-component path comparisons each; they ask a hash view of it now. reachable_set and documents_among build their ordered sets from a vector sorted by bytes, which is nearly Path's order, instead of inserting each path.
The read memo, the walk's queue and visited set, the shadow probe, the
reachable views check asks, and the views' per-document maps are keyed by
path and asked once per link per document. prov_graph::hash::{FastMap,
FastSet} are std's collections with foldhash's seeded hasher.
stdout flushes at every newline, so tree, docs and a grouped view or query were a write call per document. They write through a BufWriter now, and a grouped view flushes before its failures go to stderr, so a terminal shows the answer first as it did.
check listed most of the tree four times over: documents_among and orphans each listed the reached directories, missing_containment and nested_findings each walked the whole folder. The read memo now keeps a directory's entries beside the name index exact_name already kept, and the read-only scans list through it. A write forgets every listing above it, since it may have created the directories a tree scan lists. A mutation's own listing stays fresh.
Path's Hash walks the path a component at a time, a hasher call per component, and these are asked once per link per document. The paths are normalized, so their bytes agree with Path's equality; a path spelled two ways costs at worst a second read.
…k's thread The census walk runs on one thread because its findings depend on its order, but most of its per-document work did not: extracting the frontmatter links, parsing each, classifying it (external, same-document, foreign, malformed, local id, nominal, path) and resolving and normalizing its path. The prefetch now does that on rayon's pool and the memo keeps it beside the document, as it keeps the body's links; the walk is left the registry and the shared directory listing. The inverse check and the tree prefetch's next level take the same precomputed links. Nominal links still go to the resolvers whole, so the title index is consulted as before.
The walk asks the directory listing of every link's target; the tree prefetch now lists the directories its next level lives in through the storage's shared reader, on rayon's pool, so the walk finds them in the memo. Needs SharedRead::read_dir (fs-transaction agent/shared-reads).
twig 7.1 for meta::find_front and Document::nodes_of_kind; fs-transaction 0.4.2 for ReadStorage::shared_reader. The workspace no longer patches either from a local checkout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
prov keeps no persisted cache, and this makes it fast without one (DESIGN §5 now says so, and lists how). Every speedup comes from reading and parsing less, once, and in parallel. Nothing derived is written to disk.
Measured on two synthetic ~10k-document workspaces: flat is 100 sections × 100 documents, deep is three children per node, eight levels. Each figure is an average of 10–20 interleaved runs, warm cache, 11 cores. Under "now", the first figure is the default build and the one in brackets has
--features mimalloc:treedocschecksearchmvtreechecksearchParse less
find_front.[[or](isn't parsed for links.Parse and read once, per operation
Arc<Loaded>values instead of copying them.Graph::readis new, andloadkeeps its signature.graph::prepare), are remembered beside the document.prov_graph::hash).Parallel, with the
parallelfeatureReadStorage::shared_reader(new in fs-transaction 0.4.2). Batches under 16 documents are read in place.Smaller
check.tree,docsand grouped views use buffered output.Release-facing
prov-clihas amimallocfeature, off by default, because mimalloc is C. The release turns it on withcargo-features: mimalloc, which feat(homebrew): a rust release can pass cargo features to its native build .github#1 (merged) added to the shared workflow.Diaryx doesn't enable
paralleltoday, so it gets the single-threaded wins only.Local CI, all seven jobs: fmt, clippy, test, guide transcripts, prov check, package isolation and MSRV. clippy and tests also pass with
paralleloff.