cpp_wam store backend: in-memory index + shared cache + mmap .data (indexed matches lmdb) + crossover benchmark - #4270
Merged
Merged
Conversation
Add a raw-lookup benchmark that drives the C++ WAM SeekFactSource read path
directly (indexed on-disk UWFI/UWIX seek vs lmdb with the L1 direct-mapped +
L2 FIFO app caches) over the 256,225-row ABI symprov/2 store. Not the JS lmdb.
build.pl codegen a minimal wam_cpp project (sym_lookup/symprov) so
wam_runtime.h/.cpp expose SeekFactSource for each backend
bench_main.cpp lookup harness: encode_store_key -> src.rows(), report the
deterministic D43 I/O + L1/L2 cache counters + wall time;
optional posix_fadvise(DONTNEED) cold-cache eviction
gen_workload.mjs skewed / uniform / miss workloads from real store keys
bench_crossover.sh one-shot: build v1 lmdb store, workloads, both binaries,
sweep backend x workload x R x {warm,cold}, min-of-N wall
RESULTS.md finding: no crossover on this hardware -- lmdb wins at every
cell (13-72x). indexed is syscall-bound (~37 read()/lookup,
no cache, linear in R); lmdb amortizes to ~1 mmap op/distinct
key then serves reuse from L1/L2 (reads flat in R). Hard
memory caps unavailable (no systemd bus / cgroup deleg / root
on WSL2); store << RAM so cold~=warm; disk-bound regime
unreachable. Deterministic I/O attribution is the primary
signal and stands independent of memory.
Store artifacts stay under the gitignored abi/.out/. Frozen files untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe
…ebench The "fair fight" for the ABI store crossover. Optimize the REAL indexed backend so lmdb is tested on a level field, at symbol AND package scale. Runtime (shared cpp_wam seek path, templates/targets/cpp_wam/runtime.h.mustache): the indexed SeekFactSource now slurps the whole .idx key table into RAM once at open (idx_blob_) and binary-searches it IN MEMORY (idx_key_compare + rewritten lookup_offsets) plus one positioned .data record read -- eliminating the ~37 per-probe seek+read syscalls per lookup. Answer-identical. Correctness: 503-case store differential + 51-case corpus + 122-check ABI verify all 0 divergences/failures with the optimized runtime; bench cross-check confirms optimized-indexed rows_found == lmdb rows_found in every cell. Byte-frozen header goldens re-baselined (plain 90019->91891, lmdb 90260->92132; +1872 chars each, outside the LMDB gate). Frozen resolver files / debian/ untouched. Finding (RESULTS.md, OLD vs OPTIMIZED vs lmdb, both scales): the optimization gives indexed a uniform ~8-9.5x speedup. lmdb STILL wins on a level field at both scales, but the margin collapses -- to ~1.6-2.2x on zero-reuse (the fairest) and growing with key reuse (up to 7.6x symbol / 21.5x package at R=10). The residual edge is lmdb's L1/L2 ROW cache (reuse) + one mmap read/record vs two positioned reads -- caching, not the storage engine. Recommendation: optimized-indexed as the dependency-free universal default (within ~2x, 3x smaller, no liblmdb/format dance); a follow-up row cache on indexed would close the reuse gap; keep lmdb opt-in for high-volume high-reuse symbol resolution. bench updates: gen_workload.mjs random-5pct hot-set fallback for package scale; bench_main.cpp already drives SeekFactSource directly; bench_crossover.sh is now a two-scale driver (store rebuild + optional BUILD_OLD comparison binary). EXPERIMENT branch: the shared-runtime change would need its own PR/review if kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe
…hes too) Neutralize caching -- the only remaining differentiator -- so the store crossover becomes a true engine-vs-engine comparison. The L1 direct-mapped + L2 FIFO row cache (key -> decoded rows) was lmdb-only; it is orthogonal to storage, so lift it out of the WAM_CPP_ENABLE_LMDB gate into an engine-agnostic cache used by rows() for BOTH backends. Shared ensure_cache_config() + env UW_WAM_FACT_L1_SLOTS / UW_WAM_FACT_L2_CAP (UW_WAM_LMDB_* still honored for back-compat). Full (unbound-arg1) scans stay uncached; lmdb behavior unchanged. Correctness: 503 store differential + 51 corpus + 122 ABI verify = 0 divergences/failures (gates built at -O0 under memory pressure, one at a time); bench cross-check: indexed+cache rows_found == lmdb == nocache in every cell, and indexed+cache now reports IDENTICAL L1/L2/miss counts to lmdb. Header goldens re-baselined (plain 91891->92370, lmdb 92132->92611; +479 each, gate-independent). Frozen resolver/store/debian untouched. Finding (RESULTS.md, three-way idx-nocache vs idx+cache vs lmdb, both scales): the cache was the WHOLE reuse differentiator. With it, indexed+cache matches lmdb within ~1.1-1.4x on reuse (was 7-21x) -- read count now flat in R, same L1/L2/miss as lmdb. On zero-reuse (pure miss) lmdb keeps a ~1.9-2.7x engine edge (mmap single fetch vs indexed's two positioned reads/record), and the cache is a slight tax there. Recommendation: optimized+cached indexed as the dependency-free universal default (matches lmdb on the reuse-heavy workloads the resolver actually runs, 3x smaller, no external dep); lmdb opt-in only for pure-miss high-volume symbol scans. EXPERIMENT branch: the shared-runtime cache lift would need its own PR/review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe
Step 1 -- close the resident pure-miss gap. The indexed miss path did TWO
positioned reads per record (length prefix, then payload). mmap the .data file
and read each record IN PLACE (one page access, no read syscall, exact bytes) --
like lmdb's mmap value fetch; the ifstream two-read path stays as a POSIX-guarded
fallback. Result: symbol unique R=1 (zero-reuse) indexed vs lmdb goes 2.2x -> 1.00x
warm (0.92x cold); deterministic read count falls to ~1/record = lmdb parity.
Answer-identical: 503 differential + 51 corpus + 122 ABI verify = 0 (built -O0,
one at a time under memory pressure); bench cross-check indexed rows_found == lmdb.
Header goldens re-baselined (plain 92370->96529, lmdb 92611->96770; +4159 each,
gate-independent). Frozen resolver/store/debian untouched.
Step 2 -- calibrated IO cost model (cost_model.{sh,mjs}, cost_model_probe.c).
Real memory pressure is not creatable on this WSL2 box, so measure primitives
and extrapolate the disk-bound regime (labeled as estimate). Measured: t_seek
(cold 4KB page) ~173-180us, t_mem (warm) ~0.5us -> ~340x; t_hit/t_miss_resident
per backend (indexed ~= lmdb after mmap); and the SCATTER factor exactly from
the .idx -- indexed cold-reads-per-miss = rows_per_key (source-order scatter;
measured 1/2/4/8/16), lmdb ~= 1 (key-clustered leaf). Model T = h*t_hit +
(1-h)*cold_reads*[(1/r)*t_mem + (1-1/r)*t_seek]; solving T_lmdb<T_indexed gives
K(store/RAM) ~= 1 as a THRESHOLD (benefit only once store>RAM) with MAGNITUDE
~= rows_per_key. Practical rule: use lmdb only when store>RAM AND rows_per_key>=2
with key-interleaved data (speedup ~= rows_per_key); for the ABI ~1-row/key store
lmdb is never worth it at any ratio, and sorting the indexed .data by key clusters
it (~1 page/key, measured) to erase even the multi-row edge.
EXPERIMENT branch: the shared-runtime mmap change would need its own PR/review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe
…+ doc Add an `auto` backend policy and make it the default when UW_STORE_BACKEND is unset (explicit indexed|lmdb still override). POLICY layer only -- it picks a backend, never changes answers (both return identical rows). Rule (examples/pkg_resolver/store/ensure_lmdb.sh:uw_resolve_store_backend): choose LMDB iff store_size_bytes > UW_STORE_LMDB_RAM_FACTOR(=2) * available_RAM AND lmdb usable; else INDEXED. If lmdb is wanted but unusable (ensure_lmdb fails / MDB_INVALID) it WARNs loudly and falls back to indexed (answer- identical, safe). Prints the chosen backend + the size-vs-2xRAM numbers. available_RAM = /proc/meminfo MemAvailable, override UW_STORE_AVAIL_RAM_BYTES. store_size = built indexed .data+.idx if present, else source P/2 JSONL (excludes cases.jsonl). Wired into cpp_store/build.sh (default BACKEND_REQ=auto, resolved to a concrete backend before the build; C++ lane opts into v1 lmdb). Proof it doesn't change answers: 503-case store differential 0 divergences and 51-case corpus 0 divergences (corpus verified through the auto path -> indexed); store/test_auto_select.sh checks the rule returns lmdb above 2x and indexed below via the RAM override, explicit modes pass through, and the factor is tunable. Build-side only -- the runtime template is untouched, so byte-frozen goldens are untouched (confirmed). Frozen resolver/store/debian untouched. Docs: examples/pkg_resolver/abi/bench/BACKEND_SELECTION.md -- theory (cost model; caching+read-path not the engine were the story; onset ~1x RAM, ramps by ~2x, asymptote = rows_per_key; skew pushes onset past 1x so 2x is a conservative floor), benchmarks (idx-nocache vs idx+cache vs +mmap vs lmdb; measured t_seek/ t_mem; K(rows_per_key) table), the size-only default policy, and DEFERRED refinements (fold in rows_per_key -- ABI is 1.03 rows/key so lmdb never wins; key-sort indexed .data to erase the multi-row edge without the dependency). Honesty: disk-bound regime is modeled from measured primitives, not stress-tested (no fair memory cap on this WSL2 box). cpp_store/README points to the doc; RESULTS is the data appendix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe
s243a
added a commit
that referenced
this pull request
Sep 16, 2026
…_key gate (#4271) Fix-forward on the cpp_wam store backend (#4270): make the .idx mmap'd (evictable, not a non-evictable heap slurp that bad_alloc'd under the memory pressure the auto policy routes to indexed), a real system-liblmdb + MDB_INVALID probe with loud fallback to indexed, a rows_per_key gate in the auto rule, plus tellg guards, fd hygiene, a corrupt-.idx compare guard, a real cold-cache bench, and a content-level (row_digest) benchmark answer-identity assert. Reviewed independently by Fable (base) and kimi k3 (fixes, final VERDICT: APPROVE, verified against the tree incl. a link-order repro and a SHA-256 goldens check). kimi caught a linker-order bug (-llmdb before the stdin source) that made the MDB_INVALID probe dead code -> silent under-answering; fixed (source before lib + real mdb_version symbol) and hand-exercised end-to-end. Verified: store differential 503/0, corpus 51/0, ABI 122/0, test_auto_select all pass, byte-frozen goldens intact. Frozen resolver/store/debian untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cpp_wam store backend: in-memory index + shared cache + mmap
.data, with a crossover benchmark & cost modelImproves the D43 store-backed seek fact source (
SeekFactSource) in the sharedcpp_wamruntime so the dependency-free indexed backend matches or beats lmdb across scales, and adds an auto-selection default plus the benchmark/cost-model that motivated it. Grew out of measuring the lmdb-vs-indexed crossover on a real, large store (the ABI symbol store: 256k rows, 42 MB indexed / 128 MB lmdb).Runtime improvements (
templates/targets/cpp_wam/runtime.h.mustache).idx: load the sorted key table into RAM once at open; a lookup is an in-memory binary search + one positioned.dataread (was ~37 per-proberead()syscalls). ~7–9× faster.SeekFactSource, so both backends cache identically (envUW_WAM_FACT_L1_SLOTS/UW_WAM_FACT_L2_CAP; lmdb names still honored)..data: read each record in place (one page access, like lmdb), closing the pure-miss gap to ~1×; POSIX-guarded, with the two-readifstreampath kept as fallback.Auto-selection default (
store/ensure_lmdb.sh,cpp_store/build.sh)UW_STORE_BACKENDnow defaults toauto: choose lmdb iff store >UW_STORE_LMDB_RAM_FACTOR(=2) × available RAM and lmdb is usable, else indexed (loud, answer-identical fallback to indexed if lmdb is unusable). Explicitindexed/lmdboverride. Conservative and size-only for now;rows_per_keyrefinement documented for later.Finding (see
examples/pkg_resolver/abi/bench/BACKEND_SELECTION.md)There is no engine crossover — the apparent lmdb advantage was caching + read-path, both fixed here. lmdb's only residual edge is multi-row-per-key stores once store > RAM (magnitude ≈
rows_per_key), and even that is erasable by key-sorting the indexed.data. The disk-bound regime is modeled/extrapolated from measured primitives (resident costs + calibrated cold-seek latency), not stress-tested (no fair memory cap available).Verification
Store differential 503 / 0, corpus 51 / 0, ABI verify 122 / 0,
test_wam_cpp_templatesgreen (goldens re-baselined for the runtime change),store/test_auto_select.shall pass; the benchmark cross-checksrows_foundidentical across old/optimized-indexed/lmdb over 256k keys. Frozenresolver.pl/resolver_store.pl/debian/untouched.🤖 Generated with Claude Code
https://claude.ai/code/session_01RoXjhStCqoig6944pVNBGe