Bench/cross system webbery

bench/cross-system/README.md

@@ -1,20 +1,27 @@
-# Cross-system IC2 benchmark
+# Cross-system benchmark
 
-Side-by-side latency for **LDBC SNB Interactive Complex Read 2** on
+Side-by-side latency for LDBC SNB Interactive Complex queries on
 gqlite + a set of external graph systems, against the same SF0.1
-dataset and the same 15 substitution-parameter rows.
+dataset and the same substitution-parameter rows. Currently only IC2
+is wired up (it's the only IC whose `bench/ldbc-queries/ic<n>.toml`
+has `status = "implemented"`); more come online as the parser gains
+features. The runner accepts `--ic <n>`; each invocation runs one IC
+across all (selected) systems.
 
 ## What gets compared
 
 | System | Subdir | Status |
 |---|---|---|
 | gqlite (lazy backend) | [`gqlite/`](gqlite/) | ✅ implemented |
 | GraphQLite — colliery-io/graphqlite (Cypher, SQLite-backed) | [`graphqlite/`](graphqlite/) | ✅ implemented |
+| Kuzu — kuzudb (vectorized columnar engine, CIDR 2023; pinned to v0.11.3 since [upstream archived 2025-10-10](https://github.com/kuzudb/kuzu)) | [`kuzu/`](kuzu/) | ✅ implemented; see [`kuzu/DIVERGENCES.md`](kuzu/DIVERGENCES.md) for the archival-status framing and the `UNION ALL` query-shape divergence |
 | GraphLite — GraphLite-AI/GraphLite (ISO GQL, Sled-backed) | — | not yet integrated |
-| GQLite — webbery/gqlite (custom DSL, dead since April 2023) | — | not yet integrated |
+| GQLite — webbery/gqlite (custom DSL, dead since April 2023) | [`webbery_gqlite/`](webbery_gqlite/) | ❌ discarded — see [`webbery_gqlite/SKIPPED.md`](webbery_gqlite/SKIPPED.md) for blockers (no LIMIT in grammar, no label-disjunction, per-row load idiom) |
 
 ## Setup
 
+### 1. Shared dataset
+
 The bench depends on the same LDBC SF0.1 dataset our regular
 `ldbc_bench` uses. From the repo root:
 
@@ -30,16 +37,54 @@ That produces:
 - `bench/data/substitution_parameters-sf0.1/.../interactive_2_param.txt`
   — 15 IC2 param rows (`personId|maxDate`)
 
-Each external system additionally needs its own one-time setup
-(install the system's CLI/library, load the LDBC CSVs into the
-system's native format). See each subdir's README/script.
+### 2. Per-system prerequisites
+
+Each external system has its own install line. To install all of
+them in one go:
+
+```bash
+bash bench/cross-system/install_python_deps.sh
+```
+
+That script just runs `pip install -r requirements.txt` in each
+implemented per-system subdir. If you'd rather install them
+piecemeal, they're:
+
+| System | Install command | Other prerequisites |
+|---|---|---|
+| `gqlite` | `cargo build --release` (covered by step 1) | none |
+| `graphqlite` | `pip install -r bench/cross-system/graphqlite/requirements.txt` | none |
+| `kuzu` | `pip install -r bench/cross-system/kuzu/requirements.txt` | none |
+
+Each per-system subdir has its own `README.md` documenting prereqs,
+CLI, and any per-system gotchas. Failing-but-scaffolded systems
+(e.g. on the `bench/cross-system-failed-attempts` branch) ship a
+`DIVERGENCES.md` that explains why their integration didn't pan out.
+
+### 3. Per-system data load
+
+The first time `run_all.sh` runs, each implemented system's runner
+will auto-invoke its own `setup.py` (or equivalent) to load the LDBC
+CSVs into that system's native format. This is a one-time cost
+amortized over every subsequent bench run. To pre-load explicitly:
+
+```bash
+python bench/cross-system/graphqlite/setup.py --ic 2
+python bench/cross-system/kuzu/setup.py --ic 2
+```
+
+(`gqlite` doesn't need a separate per-system setup — `bench_setup`
+in step 1 already produced its `.gdb`.)
 
 ## Running
 
 ```bash
-# Everything that's implemented:
+# Default: IC2 against every implemented system.
 bench/cross-system/run_all.sh
 
+# Pick a different IC (must be 'implemented' in its toml):
+bench/cross-system/run_all.sh --ic 2
+
 # Just one system (useful while iterating on a per-system runner):
 bench/cross-system/run_all.sh --only gqlite
 bench/cross-system/run_all.sh --only gqlite,graphqlite
@@ -48,10 +93,13 @@ bench/cross-system/run_all.sh --only gqlite,graphqlite
 bench/cross-system/run_all.sh --iters 30 --warmup 3
 ```
 
+For a multi-IC sweep, run the script once per IC — each invocation
+lands in its own timestamped results dir.
+
 Output lands in `bench/cross-system/results/<timestamp>/`:
 - `<system>.csv` per system — raw per-iter rows (same schema as
   `ldbc_bench`:
-  `query;backend;params;row;iter;result_count;result_shape;elapsed_ns`)
+  `query;backend;params;row;iter;result_count;elapsed_ns`)
 - `cross_system.csv` — concatenation of all the above
 - `comparison.txt` — `compare_results.py` output (latency table +
   count/shape consistency check + side-by-side comparison)
@@ -100,3 +148,4 @@ the comment-link explains.
 - LDBC-driver-mediated audited compliance — that's a different
   deliverable (~3 weeks more work). This bench is research-paper-tier.
 - CI integration — bench machines vary too much to threshold on.
+

bench/cross-system/compare_results.py

@@ -1,14 +1,16 @@
 #!/usr/bin/env python3
-"""Compare per-system IC2 latencies from a unified CSV.
+"""Compare per-system latencies from a unified CSV.
 
 Input: a single CSV in the schema emitted by `src/bin/ldbc_bench.rs`:
 
     query;backend;params;row;iter;result_count;elapsed_ns
 
-The `backend` column is the system label (`lazy` for gqlite,
-`graphqlite-cypher` for graphqlite, etc.). `row` is the params-row
-index (0..14 for IC2). `params` is the raw pipe-joined param row from
-the LDBC param file, used as the join key across systems.
+The `query` column identifies the IC (e.g. `IC2`); the unified CSV
+typically holds a single IC's results from a single run_all.sh
+invocation. The `backend` column is the system label (`lazy` for
+gqlite, `graphqlite-cypher` for graphqlite, etc.). `row` is the
+params-row index. `params` is the raw pipe-joined param row from the
+LDBC param file, used as the join key across systems.
 
 Result-shape verification is logged by each runner to stderr (search
 for `SHAPE` lines in the per-system stderr output) and cross-checked
@@ -62,6 +64,7 @@ def main() -> int:
 
     by_cell: dict[tuple[int, str], list[tuple[int, int]]] = defaultdict(list)
     raw_params_by_row: dict[int, str] = {}
+    queries_seen: set[str] = set()
 
     with path.open() as f:
         reader = csv.DictReader(f, delimiter=";")
@@ -76,16 +79,20 @@ def main() -> int:
             backend = r["backend"]
             by_cell[(row_idx, backend)].append((elapsed, rc))
             raw_params_by_row.setdefault(row_idx, r["params"])
+            queries_seen.add(r.get("query", ""))
 
     if not by_cell:
         print("no rows found in input — nothing to compare.", file=sys.stderr)
         return 1
 
     systems = sorted({b for (_, b) in by_cell.keys()})
     rows = sorted({r for (r, _) in by_cell.keys()})
+    query_label = sorted(queries_seen).pop() if len(queries_seen) == 1 else (
+        ",".join(sorted(queries_seen)) if queries_seen else "?"
+    )
 
     # ---- 1. Per-cell summary ----
-    print("=== Per-cell summary (latency, ms; result_count) ===")
+    print(f"=== Per-cell summary [{query_label}] (latency, ms; result_count) ===")
     print()
     header = ["params_row", "system", "iters", "median_ms", "p95_ms", "rc"]
     widths = [10, 18, 6, 10, 10, 4]
@@ -110,7 +117,7 @@ def main() -> int:
 
     # ---- 2. Result-count consistency check ----
     print()
-    print("=== Result-count consistency (per params_row across systems) ===")
+    print(f"=== Result-count consistency [{query_label}] (per params_row across systems) ===")
     print()
     mismatches: list[int] = []
     for rk in rows:
@@ -139,7 +146,7 @@ def main() -> int:
 
     # ---- 3. Side-by-side latency table ----
     print()
-    print("=== Side-by-side latency (median ms) ===")
+    print(f"=== Side-by-side latency [{query_label}] (median ms) ===")
     print()
     header = ["params_row"] + systems
     print("  " + "  ".join(f"{h:>14}" for h in header))

bench/cross-system/gqlite/run.sh

@@ -1,43 +1,40 @@
 #!/usr/bin/env bash
-# Run our IC2 (gqlite, three GraphAccess backends) and emit per-iter
-# CSV in the cross-system schema.
+# Run a chosen IC against gqlite (lazy backend) and emit per-iter CSV
+# in the cross-system schema.
 #
-# Output schema (matches src/bin/ldbc_bench.rs at line 665):
-#   query;backend;params;row;iter;result_count;elapsed_ns
-#
-# Each backend produces the same 15 params × N iters rows; backend
-# column distinguishes them. compare_results.py uses (params, row)
-# as the join key across systems.
+# Output schema (matches src/bin/ldbc_bench.rs):
+#   query;backend;params;row;iter;result_count;result_shape;elapsed_ns
 #
 # Usage:
-#   bench/cross-system/gqlite/run.sh <out_csv> [--iters N] [--warmup N]
+#   bench/cross-system/gqlite/run.sh <out_csv> [--ic <n>] [--iters N] [--warmup N]
+#
+# Default --ic is 2. ldbc_bench reads bench/ldbc-queries/ic<n>.toml and
+# skips ICs whose status is not "implemented".
 #
 # Prereq: ./target/release/bench_setup has been run, so
 # bench/data/ldbc-sf0.1.gdb and bench/data/ldbc-sf0.1/...CSVs exist.
 
 set -euo pipefail
 
 if [[ $# -lt 1 ]]; then
-    echo "usage: $0 <out_csv> [--iters N] [--warmup N]" >&2
+    echo "usage: $0 <out_csv> [--ic <n>] [--iters N] [--warmup N]" >&2
     exit 1
 fi
 
 OUT_CSV="$1"; shift
 
-# Forwarded flags. The row cap is baked into ic2.toml's query
-# (LIMIT 20, spec-faithful) so we don't pass one here.
+IC=2
 ITERS=10
 WARMUP=2
 while [[ $# -gt 0 ]]; do
     case "$1" in
+        --ic)     IC="$2"; shift 2 ;;
         --iters)  ITERS="$2"; shift 2 ;;
         --warmup) WARMUP="$2"; shift 2 ;;
         *) echo "unknown arg: $1" >&2; exit 1 ;;
     esac
 done
 
-# Resolve repo root from this script's location, so the script works
-# regardless of where the orchestrator runs it from.
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
 cd "$REPO_ROOT"
@@ -49,19 +46,13 @@ if [[ ! -f "$GDB" ]]; then
     exit 1
 fi
 
-# Build the binary if it isn't already.
 if [[ ! -x ./target/release/ldbc_bench ]]; then
     cargo build --release --bin ldbc_bench >&2
 fi
 
-# One backend: `lazy` (LazyGraphStore — topology in RAM, labels/props
-# from disk via LRU page cache). Memory and Disk are dropped: memory
-# rebuilds from CSV every startup (not a persistence comparison), and
-# disk loads more into RAM than lazy. Lazy is gqlite's default mode
-# and the only one the cross-system claim needs.
-echo "  --- gqlite/lazy ---" >&2
+echo "  --- gqlite/lazy ic$IC ---" >&2
 ./target/release/ldbc_bench "$GDB" \
-    --ic 2 --backend lazy \
+    --ic "$IC" --backend lazy \
     --iters "$ITERS" --warmup "$WARMUP" \
     > "$OUT_CSV"