cpp_store fix-forward: mmap .idx (no heap slurp), real lmdb probe, rows_per_key gate

examples/pkg_resolver/abi/bench/BACKEND_SELECTION.md

@@ -18,9 +18,10 @@ the decision doc: what we learned, the policy, and how to refine it.
 - lmdb only pulls ahead in one regime: **disk-bound (store > RAM) AND
   multi-row-per-key**, where its key-clustered leaves save cold seeks. The
   asymptotic win there is **≈ rows_per_key**.
-- **Default policy (`auto`): pick lmdb iff `store_size > 2 × available_RAM` and
-  lmdb is usable, else indexed.** Deliberately conservative and **size-only** for
-  now (a rows-per-key refinement is deferred; see below).
+- **Default policy (`auto`): pick lmdb iff `store_size > 2 × available_RAM`
+  AND `rows_per_key ≥ 2` AND lmdb is usable for the C++ lane, else indexed.**
+  Deliberately conservative; the `rows_per_key ≥ 2` gate keeps ~1-row/key stores
+  (ABI symprov = 1.03) on indexed even when huge, because lmdb buys nothing there.
 
 ## Theory: the cost model
 
@@ -99,41 +100,60 @@ Implemented in `examples/pkg_resolver/store/ensure_lmdb.sh`
 
 ```
 choose LMDB  iff  store_size_bytes > UW_STORE_LMDB_RAM_FACTOR × available_RAM_bytes
-             AND  lmdb is usable (uw_ensure_lmdb succeeds)
+             AND  rows_per_key >= UW_STORE_LMDB_MIN_ROWS_PER_KEY
+             AND  lmdb is usable for the C++ lane
 else INDEXED
 ```
 
 - `UW_STORE_LMDB_RAM_FACTOR` — the headroom factor, **default 2** (named constant,
-  tunable). 2× is deliberately conservative: the model puts the *onset* at ~1×
-  RAM and skew pushes it higher, so 2× only trips lmdb once the store clearly
-  exceeds RAM and disk-bound misses are unavoidable.
+  tunable; a non-integer value is rejected with a warning and treated as 2). 2× is
+  deliberately conservative: the model puts the *onset* at ~1× RAM and skew pushes
+  it higher, so 2× only trips lmdb once the store clearly exceeds RAM and
+  disk-bound misses are unavoidable.
+- `UW_STORE_LMDB_MIN_ROWS_PER_KEY` — **default 2**. `rows_per_key` is aggregated
+  cheaply from the UWIX `.idx` headers (`n_records / n_keys`, no scan) across all
+  indexes in the store dir. Because the asymptotic lmdb win is ≈ rows_per_key, a
+  ~1-row/key store (ABI symprov = 1.03) never benefits — this gate keeps it on
+  indexed even above 2× RAM. When no `.idx` exists yet (pre-build), rows_per_key
+  is *unknown* and the gate is skipped (size-only for that first build).
 - `available_RAM` — `/proc/meminfo` `MemAvailable`, overridable with
   `UW_STORE_AVAIL_RAM_BYTES` (WSL2's `MemAvailable` balloons, so the override
   matters for tests/reproducibility).
 - `store_size` — the **built** indexed store (`*.data` + `*.idx`) when present,
   else the source **P/2 JSONL** (`*.jsonl`, excluding `cases.jsonl`) as a
   pre-build estimate. One consistent measure; documented here.
-- **Fallback:** if the size rule wants lmdb but lmdb is not usable
-  (`uw_ensure_lmdb` fails / `MDB_INVALID`), it **WARNs loudly and uses indexed** —
-  safe because indexed is answer-identical and ~as fast up to the disk-bound
-  multi-row regime. It prints the chosen backend and the size-vs-`2×RAM` numbers.
+- **`lmdb usable` for the C++ lane** is a real probe, not just "the npm module
+  loads": (1) `uw_ensure_lmdb` (the v1-format module used to *build* the store),
+  (2) system `liblmdb` links (`#include <lmdb.h>` + `-llmdb`) — the C++ reader
+  needs it, and (3) best-effort: an already-built lmdb store under `DIR/lmdb/*`
+  actually `mdb_env_open`s (catches `MDB_INVALID` from a wrong page format).
+- **Fallback:** if the rule wants lmdb but it is not usable, it **WARNs loudly and
+  uses indexed** — safe because indexed is answer-identical and ~as fast up to the
+  disk-bound multi-row regime. It prints the chosen backend and the numbers.
 - **Policy only, never answers.** Whichever backend is chosen returns identical
   rows. Proven: 503-case store differential + 51-case corpus stay **0
-  divergences** (corpus verified through the auto path → indexed), and
-  `test_auto_select.sh` checks the rule returns lmdb above 2× and indexed below
-  (via `UW_STORE_AVAIL_RAM_BYTES`).
+  divergences** (corpus verified through the auto path → indexed);
+  `bench_crossover.sh` asserts `rows_found` identical across backends in every
+  cell; and `test_auto_select.sh` checks the size rule and the rows_per_key gate
+  (lmdb above 2× with rpk≥2, indexed below 2× or at rpk<2) via the RAM override.
+
+## Storage note: the index is mmap'd, not slurped
+
+The indexed backend **mmaps** both `.idx` and `.data` (`PROT_READ`,
+`MAP_PRIVATE`), page-cache-backed and **evictable**. An earlier version slurped
+the whole `.idx` into a heap `std::string` — but the `.idx` is **39-97% of the
+store**, so that allocated ~that much *non-evictable anonymous* memory. Under the
+exact pressure that routes a `> 2× RAM` store to indexed, the slurp risked
+`bad_alloc` → caught as a goal failure → *silent under-answering*. mmap restores
+the O(1)-heap, page-cache-friendly behavior (with an `ifstream` fallback for
+non-POSIX). It also fixes the D43 counters (the mmap path charges nothing at open;
+only actual record reads count).
 
 ## Future refinement (deferred, per the owner)
 
-The size-only rule is conservative but coarse. Two cheap improvements, explicitly
-deferred:
+One cheap improvement remains, explicitly deferred:
 
-1. **Fold in rows_per_key.** The asymptotic benefit is ≈ rows_per_key, so for
-   ~1-row-per-key stores lmdb is *never* worth it even above 2× RAM. The ABI
-   `symprov` store is **1.03 rows/key** (249,097 keys / 256,225 rows) — cheaply
-   computable from the `.idx` header (keys vs records) with no scan. A refined
-   selector should require `rows_per_key ≳ 2` in addition to the size trigger.
-2. **Key-sort the indexed `.data`.** Indexed's only structural disadvantage is
+1. **Key-sort the indexed `.data`.** Indexed's only structural disadvantage is
    source-order scatter (M scattered pages per multi-row key). Building `.data`
    in **key order** clusters a key's rows into ~1 page (measured: an 8-row/key
    store drops from 8 to **1.05** pages/key), erasing lmdb's disk-bound edge

examples/pkg_resolver/abi/bench/bench_crossover.sh

@@ -147,6 +147,22 @@ for SCALE in $SCALES; do
     run_cell "$RES" lmdb        lmdb    "$OUT/bench_lmdb"        "$LMDB" "$WLDIR" "$wl" "$R" "$ev"
   done; done; done
   echo "== wrote $RES =="
+  # ASSERT answer-identity: for each (workload,R,evict) cell every backend must
+  # return the same rows_found (a backend choice must never change answers).
+  node -e '
+const fs=require("fs");
+const rows=fs.readFileSync(process.argv[1],"utf8").split("\n").filter(Boolean).map(JSON.parse);
+const by={}; let bad=0;
+for(const o of rows){const k=`${o.workload}|R${o.R}|ev${o.evict}`;(by[k]=by[k]||[]).push(o);}
+for(const k of Object.keys(by)){
+  const g=by[k], want=g[0].rows_found;
+  for(const o of g) if(o.rows_found!==want){
+    console.error(`  ROWS MISMATCH ${k}: ${o.label}=${o.rows_found} vs ${g[0].label}=${want}`); bad=1;
+  }
+}
+if(bad){console.error("== ROWS_FOUND NOT IDENTICAL ACROSS BACKENDS =="); process.exit(1);}
+console.log("== rows_found identical across backends for every cell ("+Object.keys(by).length+" cells) ==");
+' "$RES"
 done
 echo "store sizes:"; du -h "$OUT/idx/symprov.data" "$OUT/idx/symprov.idx" "$OUT/lmdb_v1/symprov/data.mdb" \
   "$SCALE_DIR/pkg.data" "$SCALE_DIR/pkg.idx" "$PB/lmdb_pkg/data.mdb" 2>/dev/null | sed 's/^/  /'

examples/pkg_resolver/abi/bench/bench_main.cpp

@@ -51,8 +51,14 @@ static void evict_file(const std::string& p) {
   if (fd < 0) return;
   struct stat st{};
   if (::fstat(fd, &st) == 0 && st.st_size > 0) {
-    // Read through it once to ensure it is cached, then drop it, so DONTNEED
-    // has resident pages to evict (DONTNEED is a no-op on non-resident pages).
+    // Pre-fault the file so its pages are actually RESIDENT, THEN drop them:
+    // POSIX_FADV_DONTNEED is a no-op on non-resident pages, so without the
+    // streaming read below the "cold" run would be cold-in-name-only. The
+    // sequential read forces the pages in; DONTNEED then evicts them, so the
+    // subsequent lookups genuinely fault from disk.
+    char buf[1 << 16];
+    ssize_t n;
+    while ((n = ::read(fd, buf, sizeof(buf))) > 0) { /* force pages resident */ }
     ::posix_fadvise(fd, 0, st.st_size, POSIX_FADV_DONTNEED);
   }
   ::close(fd);

examples/pkg_resolver/store/ensure_lmdb.sh

@@ -79,10 +79,15 @@ uw_require_lmdb() {
 # backends return identical rows). Documented in
 # examples/pkg_resolver/abi/bench/BACKEND_SELECTION.md.
 #
-#   choose LMDB iff  store_size_bytes > FACTOR * available_RAM_bytes  AND lmdb
-#   is usable; else INDEXED. FACTOR (UW_STORE_LMDB_RAM_FACTOR, default 2) is a
-#   conservative headroom over the ~1x-RAM crossover onset. Size-only for now
-#   (a future refinement folds in rows_per_key -- see the doc).
+#   choose LMDB iff  store_size_bytes > FACTOR * available_RAM_bytes
+#              AND   rows_per_key >= MIN_RPK   (lmdb buys ~nothing at ~1 row/key)
+#              AND   lmdb is usable for the C++ lane (system liblmdb links, and a
+#                    built lmdb store, if present, opens)
+#   else INDEXED. FACTOR (UW_STORE_LMDB_RAM_FACTOR, default 2) is conservative
+#   headroom over the ~1x-RAM crossover onset; MIN_RPK
+#   (UW_STORE_LMDB_MIN_ROWS_PER_KEY, default 2) reflects that the asymptotic lmdb
+#   win is ~= rows_per_key, so a 1-row/key store (e.g. ABI symprov, 1.03) never
+#   benefits. NOTE: stat/od are GNU/coreutils (Linux); fine for this lane.
 # ---------------------------------------------------------------------------
 
 # Sum the store bytes under DIR: the BUILT indexed store (*.data + *.idx) when
@@ -103,6 +108,26 @@ uw_store_size_bytes() {  # DIR -> bytes
   echo "$total"
 }
 
+# Aggregate rows_per_key across the store's UWIX indexes (cheap: read the header
+# of each *.idx -- n_keys at byte 8, n_records at byte 20, both u32 LE). Echoes a
+# 2-dp float, or "" when no index exists yet (pre-build -> caller treats as
+# unknown and does not gate on it).
+uw_store_rows_per_key() {  # DIR -> float | ""
+  local dir="${1:?usage: uw_store_rows_per_key DIR}" recs=0 keys=0 f had=0 nk nr
+  shopt -s nullglob
+  for f in "$dir"/*.idx; do
+    nk=$(od -An -tu4 -j8  -N4 "$f" 2>/dev/null | tr -d ' ')
+    nr=$(od -An -tu4 -j20 -N4 "$f" 2>/dev/null | tr -d ' ')
+    if [ -n "$nk" ] && [ -n "$nr" ]; then keys=$(( keys + nk )); recs=$(( recs + nr )); had=1; fi
+  done
+  shopt -u nullglob
+  if [ "$had" -eq 1 ] && [ "$keys" -gt 0 ]; then
+    awk "BEGIN{printf \"%.2f\", $recs/$keys}"
+  else
+    echo ""
+  fi
+}
+
 # Available RAM in bytes: UW_STORE_AVAIL_RAM_BYTES override (WSL2 MemAvailable
 # balloons, so an override matters for testing/reproducibility) else
 # /proc/meminfo MemAvailable.
@@ -113,6 +138,36 @@ uw_available_ram_bytes() {
   echo $(( kb * 1024 ))
 }
 
+# Is lmdb actually usable for the C++ store lane? Checks what the C++ build/read
+# needs -- NOT just that the npm module loads:
+#   1. uw_ensure_lmdb (the v1-format npm module, used to BUILD the store),
+#   2. system liblmdb links (<lmdb.h> + -llmdb) -- the C++ reader needs it,
+#   3. best-effort: an already-built lmdb store under DIR/lmdb/* OPENS with
+#      vanilla liblmdb (catches MDB_INVALID from a wrong page format).
+# Returns 0 if usable, non-zero otherwise. All output suppressed.
+uw_lmdb_cpp_usable() {  # [DIR]
+  local dir="${1:-}" cxx="${CXX:-g++}"
+  uw_ensure_lmdb >/dev/null 2>&1 || return 1
+  printf '#include <lmdb.h>\nint main(){return 0;}\n' \
+    | "$cxx" -x c++ -std=c++17 -O0 -o /dev/null -llmdb - >/dev/null 2>&1 || return 1
+  # best-effort smoke-open of a built store (first lmdb sub-env under DIR/lmdb)
+  if [ -n "$dir" ]; then
+    local envdir=""
+    shopt -s nullglob
+    local d; for d in "$dir"/lmdb/*/ "$dir"/lmdb_v1/*/; do [ -f "$d/data.mdb" ] && { envdir="${d%/}"; break; }; done
+    shopt -u nullglob
+    if [ -n "$envdir" ]; then
+      local probe; probe="$(mktemp -d)/mdbprobe"
+      if printf '#include <lmdb.h>\nint main(int c,char**v){MDB_env*e;if(mdb_env_create(&e))return 1;int rc=mdb_env_open(e,v[1],MDB_RDONLY|MDB_NOTLS,0664);int ok=(rc==0);if(ok){MDB_txn*t;MDB_dbi d;if(mdb_txn_begin(e,0,MDB_RDONLY,&t)==0){if(mdb_dbi_open(t,0,0,&d)!=0)ok=0;mdb_txn_abort(t);}else ok=0;}mdb_env_close(e);return ok?0:2;}\n' \
+           | "$cxx" -x c++ -std=c++17 -O0 -o "$probe" -llmdb - >/dev/null 2>&1; then
+        "$probe" "$envdir" >/dev/null 2>&1 || { rm -rf "$(dirname "$probe")"; return 1; }
+      fi
+      rm -rf "$(dirname "$probe")"
+    fi
+  fi
+  return 0
+}
+
 # Resolve a requested backend (auto|indexed|lmdb) to a CONCRETE one.
 # Echoes indexed|lmdb on stdout; all diagnostics go to stderr so callers can
 # capture the choice with $(...). auto that wants lmdb but finds it unusable
@@ -125,18 +180,27 @@ uw_resolve_store_backend() {  # MODE DIR -> echoes indexed|lmdb
     *) echo "uw_resolve_store_backend: unknown backend '$mode' (auto|indexed|lmdb)" >&2; return 2 ;;
   esac
   local factor="${UW_STORE_LMDB_RAM_FACTOR:-2}"
-  local store ram threshold
+  local min_rpk="${UW_STORE_LMDB_MIN_ROWS_PER_KEY:-2}"
+  case "$factor" in ''|*[!0-9]*) echo "uw_store auto: UW_STORE_LMDB_RAM_FACTOR='$factor' is not a non-negative integer -> using 2" >&2; factor=2 ;; esac
+  local store ram threshold rpk
   store=$(uw_store_size_bytes "$dir")
   ram=$(uw_available_ram_bytes)
   threshold=$(( ram * factor ))
-  if [ "$store" -gt "$threshold" ]; then
-    if uw_ensure_lmdb >/dev/null 2>&1; then
-      echo "uw_store auto: store=${store}B > ${factor}x avail_RAM(${ram}B)=${threshold}B -> lmdb" >&2
-      echo lmdb; return 0
-    fi
-    echo "uw_store auto: store=${store}B > ${factor}x avail_RAM(${ram}B)=${threshold}B would pick lmdb, but lmdb is NOT usable (uw_ensure_lmdb failed / MDB_INVALID) -- FALLING BACK to indexed (answer-identical)" >&2
+  if [ "$store" -le "$threshold" ]; then
+    echo "uw_store auto: store=${store}B <= ${factor}x avail_RAM(${ram}B)=${threshold}B -> indexed" >&2
     echo indexed; return 0
   fi
-  echo "uw_store auto: store=${store}B <= ${factor}x avail_RAM(${ram}B)=${threshold}B -> indexed" >&2
+  # Above the size threshold: gate on rows_per_key (lmdb's asymptotic win ~= rpk).
+  rpk=$(uw_store_rows_per_key "$dir")
+  if [ -n "$rpk" ] && awk "BEGIN{exit !($rpk < $min_rpk)}"; then
+    echo "uw_store auto: store=${store}B > ${factor}x avail_RAM(${ram}B)=${threshold}B BUT rows_per_key=${rpk} < ${min_rpk} -> indexed (lmdb buys ~nothing at ~1 row/key)" >&2
+    echo indexed; return 0
+  fi
+  # Size + rows_per_key (or unknown rpk) favor lmdb; require it to be usable.
+  if uw_lmdb_cpp_usable "$dir"; then
+    echo "uw_store auto: store=${store}B > ${factor}x avail_RAM(${ram}B)=${threshold}B, rows_per_key=${rpk:-unknown} -> lmdb" >&2
+    echo lmdb; return 0
+  fi
+  echo "uw_store auto: store=${store}B > ${factor}x avail_RAM(${ram}B)=${threshold}B, rows_per_key=${rpk:-unknown} would pick lmdb, but lmdb is NOT usable for the C++ lane (missing liblmdb / npm module / MDB_INVALID store) -- FALLING BACK to indexed (answer-identical)" >&2
   echo indexed; return 0
 }