Skip to content

pkg_resolver: LMDB catalog backend + fleet materialisation tiers - #4216

Open
s243a wants to merge 4 commits into
claude/peerhailer-exploratory-docs-aodas5from
grok/pkg-resolver-lmdb-cache
Open

s243a wants to merge 4 commits into
claude/peerhailer-exploratory-docs-aodas5from
grok/pkg-resolver-lmdb-cache

Conversation

@s243a

@s243a s243a commented Sep 3, 2026

Copy link
Copy Markdown
Owner

LMDB catalog backend for store-backed uw-resolve, plus a fact-source cache that uses the fleet graph-kernel materialisation vocabulary (docs/WAM_GO_STATUS.md, docs/WAM_FSHARP_STATUS.md, Rust LookupSource).

Declaration switch

UW_STORE_BACKEND=indexed|lmdb (default indexed) compiles the same adapter (store_pkg/2store_revdep/2) against D43 indexed(Prefix) or four lmdb(Dir) environments (StoreDir/lmdb/{pkg,dep,conflict,revdep}). Same 10 queries, same JSON shim. No silent fallback: missing lmdb npm is a loud error (D43 /tmp/uw-lmdb-pkg install, no repo package.json).

Materialisation tiers (fleet names)

tier JS meaning
eager scan once at first use; serve from an A1 index
lazy fetch per lookup (2×2 "cache off"; D48 bytes-read proof)
cached bounded L1 LRU. Default, matching Go/F#.
auto residual: no core/cost_model.pl hookup; falls back to cached (stderr once)
  • Env: UW_LMDB_MATERIALISATION=eager|lazy|cached|auto. Measurement alias unchanged: UW_FACT_CACHE=0 → lazy, =1 → cached.
  • Cap: UW_LMDB_L2_CAPACITY / UW_FACT_CACHE_CAP (default 4096) sizes the L1. L2 is a residual (Go L1 = L2/8).
  • Runtime.configure_lmdb_materialisation(mode, cap); configure_fact_cache kept as alias.
  • Equivalence: lazy / cached / eager identical corpus results (39 cases, both backends).
  • D48 scale demo forces lazy so bytes ∝ query (8242 B / 650 reads). Default cached still seek-on-miss: D43 bytes-read proof is 506 / 992872 / 29 reads.
  • Rust graph kernels measure cached vs lazy at 2.25–3.17× (docs/reports/wam_rust_cached_scaling_sweep_2026-06-14.md) when lookups dominate. This resolver is interpreter-bound; cache value is repeated queries (~111× fewer indexed reads).

Gates (this revision, this VM)

Gate Result
Store corpus indexed 39/39
Store corpus lmdb 39/39 (gated arm ran, /tmp/uw-lmdb-pkg)
Store diff indexed default cached 500/0 (4.875s)
Store diff lmdb default cached 500/0 (4.701s)
Cache equiv lazy/cached/eager indexed + lmdb green (off_hits=0 on_hits=79 replay_hits=270 eager=eager)
D48 term plunit + dump/wamjs + term diff 38 / 39/39 / 2400/0
D43 fact_sources + cut + builtins + lowered green; bytes-read 506/992872 at materialisation=cached
CONFORMANCE_TARGETS=javascript green
cli_args 17/17, 5067/0/0

2×2 + 100× (seed 0xc0ffee01; off=lazy, on=cached)

Same 10-package resolve_layered in every cell. WAM loop dominates store I/O.

backend cache 1× wall 1× I/O 100× wall 100× I/O
indexed off 90.9 ms 8242 B / 650 reads 6073 ms 64406 reads
indexed on 89.1 ms 7321 B / 578 reads 5900 ms 578 reads
lmdb off 103.6 ms 114 ops 6026 ms 11400 ops
lmdb on 108.7 ms 31 ops 5850 ms 31 ops

This-revision 5k probe (same query, same result): lazy 8242 B / 650; cached 7321 B / 578 (83 hits); eager 742819 B / 67548 (full scan — why the scale demo forces lazy).

Honest reading: LMDB does not beat indexed here. The cache's value is repeated queries in one process, not first-query wall time.

Residuals

  • L2 two-level cache (lmdb_l2_capacity as a real L2)
  • auto + core/cost_model.pl (JS warns and uses cached)
  • compile-time lmdb_materialisation(...) / lmdb_l2_capacity(...) on write_wam_javascript_project (wam_javascript_target.pl frozen this round; runtime env + optional source.materialisation until then)
  • secondary indexes, incremental updates, write paths (must Runtime.fact_cache_reset()), CSR
Open in Web Open in Cursor 

cursoragent and others added 3 commits September 3, 2026 17:49
Add a bounded read-through cache (UW_FACT_CACHE, default off, cap 4096)
around indexed/lmdb lookups, plus lmdb_ops counters on UW_FACT_IO_STATS.
uw_fact_lmdb.js gains build-all for the four resolver stores. Catalogs
are read-only: no invalidation; a write path must fact_cache_reset().

Co-authored-by: johns243a <johns243a@gmail.com>
Declaration switch UW_STORE_BACKEND=indexed|lmdb compiles the same
store adapter against D43 indexed prefixes or four LMDB envs. Cache
on/off is asserted identical on the dump corpus. lmdb stays opt-in
via /tmp/uw-lmdb-pkg with a loud missing-package error.

Co-authored-by: johns243a <johns243a@gmail.com>
Include js-cache plunit in test_resolver_store/0. Rebuild the store
differential binary when wam_runtime.js lacks configure_fact_cache.

Co-authored-by: johns243a <johns243a@gmail.com>
@s243a
s243a marked this pull request as ready for review September 3, 2026 17:53
…on tiers

Name the JS store cache after Go/F#/Rust lmdb_materialisation
(eager|lazy|cached|auto). Default cached; UW_FACT_CACHE=0|1 remains the
lazy|cached measurement alias. L2 and cost-model auto are residuals.

Co-authored-by: johns243a <johns243a@gmail.com>
@cursor cursor Bot changed the title pkg_resolver: LMDB catalog backend + L1 fact-source cache pkg_resolver: LMDB catalog backend + fleet materialisation tiers Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants