diff --git a/bench/cross-system/README.md b/bench/cross-system/README.md index 5203ad4..349de2d 100644 --- a/bench/cross-system/README.md +++ b/bench/cross-system/README.md @@ -1,8 +1,12 @@ -# Cross-system IC2 benchmark +# Cross-system benchmark -Side-by-side latency for **LDBC SNB Interactive Complex Read 2** on +Side-by-side latency for LDBC SNB Interactive Complex queries on gqlite + a set of external graph systems, against the same SF0.1 -dataset and the same 15 substitution-parameter rows. +dataset and the same substitution-parameter rows. Currently only IC2 +is wired up (it's the only IC whose `bench/ldbc-queries/ic.toml` +has `status = "implemented"`); more come online as the parser gains +features. The runner accepts `--ic `; each invocation runs one IC +across all (selected) systems. ## What gets compared @@ -10,11 +14,14 @@ dataset and the same 15 substitution-parameter rows. |---|---|---| | gqlite (lazy backend) | [`gqlite/`](gqlite/) | ✅ implemented | | GraphQLite — colliery-io/graphqlite (Cypher, SQLite-backed) | [`graphqlite/`](graphqlite/) | ✅ implemented | +| Kuzu — kuzudb (vectorized columnar engine, CIDR 2023; pinned to v0.11.3 since [upstream archived 2025-10-10](https://github.com/kuzudb/kuzu)) | [`kuzu/`](kuzu/) | ✅ implemented; see [`kuzu/DIVERGENCES.md`](kuzu/DIVERGENCES.md) for the archival-status framing and the `UNION ALL` query-shape divergence | | GraphLite — GraphLite-AI/GraphLite (ISO GQL, Sled-backed) | — | not yet integrated | -| GQLite — webbery/gqlite (custom DSL, dead since April 2023) | — | not yet integrated | +| GQLite — webbery/gqlite (custom DSL, dead since April 2023) | [`webbery_gqlite/`](webbery_gqlite/) | ❌ discarded — see [`webbery_gqlite/SKIPPED.md`](webbery_gqlite/SKIPPED.md) for blockers (no LIMIT in grammar, no label-disjunction, per-row load idiom) | ## Setup +### 1. Shared dataset + The bench depends on the same LDBC SF0.1 dataset our regular `ldbc_bench` uses. From the repo root: @@ -30,16 +37,54 @@ That produces: - `bench/data/substitution_parameters-sf0.1/.../interactive_2_param.txt` — 15 IC2 param rows (`personId|maxDate`) -Each external system additionally needs its own one-time setup -(install the system's CLI/library, load the LDBC CSVs into the -system's native format). See each subdir's README/script. +### 2. Per-system prerequisites + +Each external system has its own install line. To install all of +them in one go: + +```bash +bash bench/cross-system/install_python_deps.sh +``` + +That script just runs `pip install -r requirements.txt` in each +implemented per-system subdir. If you'd rather install them +piecemeal, they're: + +| System | Install command | Other prerequisites | +|---|---|---| +| `gqlite` | `cargo build --release` (covered by step 1) | none | +| `graphqlite` | `pip install -r bench/cross-system/graphqlite/requirements.txt` | none | +| `kuzu` | `pip install -r bench/cross-system/kuzu/requirements.txt` | none | + +Each per-system subdir has its own `README.md` documenting prereqs, +CLI, and any per-system gotchas. Failing-but-scaffolded systems +(e.g. on the `bench/cross-system-failed-attempts` branch) ship a +`DIVERGENCES.md` that explains why their integration didn't pan out. + +### 3. Per-system data load + +The first time `run_all.sh` runs, each implemented system's runner +will auto-invoke its own `setup.py` (or equivalent) to load the LDBC +CSVs into that system's native format. This is a one-time cost +amortized over every subsequent bench run. To pre-load explicitly: + +```bash +python bench/cross-system/graphqlite/setup.py --ic 2 +python bench/cross-system/kuzu/setup.py --ic 2 +``` + +(`gqlite` doesn't need a separate per-system setup — `bench_setup` +in step 1 already produced its `.gdb`.) ## Running ```bash -# Everything that's implemented: +# Default: IC2 against every implemented system. bench/cross-system/run_all.sh +# Pick a different IC (must be 'implemented' in its toml): +bench/cross-system/run_all.sh --ic 2 + # Just one system (useful while iterating on a per-system runner): bench/cross-system/run_all.sh --only gqlite bench/cross-system/run_all.sh --only gqlite,graphqlite @@ -48,10 +93,13 @@ bench/cross-system/run_all.sh --only gqlite,graphqlite bench/cross-system/run_all.sh --iters 30 --warmup 3 ``` +For a multi-IC sweep, run the script once per IC — each invocation +lands in its own timestamped results dir. + Output lands in `bench/cross-system/results//`: - `.csv` per system — raw per-iter rows (same schema as `ldbc_bench`: - `query;backend;params;row;iter;result_count;result_shape;elapsed_ns`) + `query;backend;params;row;iter;result_count;elapsed_ns`) - `cross_system.csv` — concatenation of all the above - `comparison.txt` — `compare_results.py` output (latency table + count/shape consistency check + side-by-side comparison) @@ -100,3 +148,4 @@ the comment-link explains. - LDBC-driver-mediated audited compliance — that's a different deliverable (~3 weeks more work). This bench is research-paper-tier. - CI integration — bench machines vary too much to threshold on. + diff --git a/bench/cross-system/compare_results.py b/bench/cross-system/compare_results.py index cd59be5..aa9fbe8 100644 --- a/bench/cross-system/compare_results.py +++ b/bench/cross-system/compare_results.py @@ -1,14 +1,16 @@ #!/usr/bin/env python3 -"""Compare per-system IC2 latencies from a unified CSV. +"""Compare per-system latencies from a unified CSV. Input: a single CSV in the schema emitted by `src/bin/ldbc_bench.rs`: query;backend;params;row;iter;result_count;elapsed_ns -The `backend` column is the system label (`lazy` for gqlite, -`graphqlite-cypher` for graphqlite, etc.). `row` is the params-row -index (0..14 for IC2). `params` is the raw pipe-joined param row from -the LDBC param file, used as the join key across systems. +The `query` column identifies the IC (e.g. `IC2`); the unified CSV +typically holds a single IC's results from a single run_all.sh +invocation. The `backend` column is the system label (`lazy` for +gqlite, `graphqlite-cypher` for graphqlite, etc.). `row` is the +params-row index. `params` is the raw pipe-joined param row from the +LDBC param file, used as the join key across systems. Result-shape verification is logged by each runner to stderr (search for `SHAPE` lines in the per-system stderr output) and cross-checked @@ -62,6 +64,7 @@ def main() -> int: by_cell: dict[tuple[int, str], list[tuple[int, int]]] = defaultdict(list) raw_params_by_row: dict[int, str] = {} + queries_seen: set[str] = set() with path.open() as f: reader = csv.DictReader(f, delimiter=";") @@ -76,6 +79,7 @@ def main() -> int: backend = r["backend"] by_cell[(row_idx, backend)].append((elapsed, rc)) raw_params_by_row.setdefault(row_idx, r["params"]) + queries_seen.add(r.get("query", "")) if not by_cell: print("no rows found in input — nothing to compare.", file=sys.stderr) @@ -83,9 +87,12 @@ def main() -> int: systems = sorted({b for (_, b) in by_cell.keys()}) rows = sorted({r for (r, _) in by_cell.keys()}) + query_label = sorted(queries_seen).pop() if len(queries_seen) == 1 else ( + ",".join(sorted(queries_seen)) if queries_seen else "?" + ) # ---- 1. Per-cell summary ---- - print("=== Per-cell summary (latency, ms; result_count) ===") + print(f"=== Per-cell summary [{query_label}] (latency, ms; result_count) ===") print() header = ["params_row", "system", "iters", "median_ms", "p95_ms", "rc"] widths = [10, 18, 6, 10, 10, 4] @@ -110,7 +117,7 @@ def main() -> int: # ---- 2. Result-count consistency check ---- print() - print("=== Result-count consistency (per params_row across systems) ===") + print(f"=== Result-count consistency [{query_label}] (per params_row across systems) ===") print() mismatches: list[int] = [] for rk in rows: @@ -139,7 +146,7 @@ def main() -> int: # ---- 3. Side-by-side latency table ---- print() - print("=== Side-by-side latency (median ms) ===") + print(f"=== Side-by-side latency [{query_label}] (median ms) ===") print() header = ["params_row"] + systems print(" " + " ".join(f"{h:>14}" for h in header)) diff --git a/bench/cross-system/gqlite/run.sh b/bench/cross-system/gqlite/run.sh index 189df46..92be3c2 100644 --- a/bench/cross-system/gqlite/run.sh +++ b/bench/cross-system/gqlite/run.sh @@ -1,16 +1,15 @@ #!/usr/bin/env bash -# Run our IC2 (gqlite, three GraphAccess backends) and emit per-iter -# CSV in the cross-system schema. +# Run a chosen IC against gqlite (lazy backend) and emit per-iter CSV +# in the cross-system schema. # -# Output schema (matches src/bin/ldbc_bench.rs at line 665): -# query;backend;params;row;iter;result_count;elapsed_ns -# -# Each backend produces the same 15 params × N iters rows; backend -# column distinguishes them. compare_results.py uses (params, row) -# as the join key across systems. +# Output schema (matches src/bin/ldbc_bench.rs): +# query;backend;params;row;iter;result_count;result_shape;elapsed_ns # # Usage: -# bench/cross-system/gqlite/run.sh [--iters N] [--warmup N] +# bench/cross-system/gqlite/run.sh [--ic ] [--iters N] [--warmup N] +# +# Default --ic is 2. ldbc_bench reads bench/ldbc-queries/ic.toml and +# skips ICs whose status is not "implemented". # # Prereq: ./target/release/bench_setup has been run, so # bench/data/ldbc-sf0.1.gdb and bench/data/ldbc-sf0.1/...CSVs exist. @@ -18,26 +17,24 @@ set -euo pipefail if [[ $# -lt 1 ]]; then - echo "usage: $0 [--iters N] [--warmup N]" >&2 + echo "usage: $0 [--ic ] [--iters N] [--warmup N]" >&2 exit 1 fi OUT_CSV="$1"; shift -# Forwarded flags. The row cap is baked into ic2.toml's query -# (LIMIT 20, spec-faithful) so we don't pass one here. +IC=2 ITERS=10 WARMUP=2 while [[ $# -gt 0 ]]; do case "$1" in + --ic) IC="$2"; shift 2 ;; --iters) ITERS="$2"; shift 2 ;; --warmup) WARMUP="$2"; shift 2 ;; *) echo "unknown arg: $1" >&2; exit 1 ;; esac done -# Resolve repo root from this script's location, so the script works -# regardless of where the orchestrator runs it from. SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" cd "$REPO_ROOT" @@ -49,19 +46,13 @@ if [[ ! -f "$GDB" ]]; then exit 1 fi -# Build the binary if it isn't already. if [[ ! -x ./target/release/ldbc_bench ]]; then cargo build --release --bin ldbc_bench >&2 fi -# One backend: `lazy` (LazyGraphStore — topology in RAM, labels/props -# from disk via LRU page cache). Memory and Disk are dropped: memory -# rebuilds from CSV every startup (not a persistence comparison), and -# disk loads more into RAM than lazy. Lazy is gqlite's default mode -# and the only one the cross-system claim needs. -echo " --- gqlite/lazy ---" >&2 +echo " --- gqlite/lazy ic$IC ---" >&2 ./target/release/ldbc_bench "$GDB" \ - --ic 2 --backend lazy \ + --ic "$IC" --backend lazy \ --iters "$ITERS" --warmup "$WARMUP" \ > "$OUT_CSV" diff --git a/bench/cross-system/graphqlite/run.py b/bench/cross-system/graphqlite/run.py index e905183..b3b2b22 100644 --- a/bench/cross-system/graphqlite/run.py +++ b/bench/cross-system/graphqlite/run.py @@ -1,8 +1,8 @@ #!/usr/bin/env python3 -"""Run IC2 against graphqlite for every substitution-param row, +"""Run a chosen IC against graphqlite for every substitution-param row, emitting per-iter CSV in the cross-system schema. -Output schema (matches src/bin/ldbc_bench.rs at line 665): +Output schema (matches src/bin/ldbc_bench.rs): query;backend;params;row;iter;result_count;elapsed_ns The `backend` column is fixed to `graphqlite-cypher`. `params` is the @@ -10,15 +10,16 @@ join key against gqlite's CSV in compare_results.py). `row` is the 0-based param-row index. -Prereq: setup.py has been run, so -`bench/data/cross-system/graphqlite/ic2.db` exists. +Per-IC inputs (all derived from the IC number): + bench/ldbc-queries/ic.toml — query metadata + bench/cross-system/graphqlite/ic.cypher — cypher translation + bench/data/substitution_parameters-sf0.1/.../ + bench/data/cross-system/graphqlite/ic.db — pre-loaded DB -Usage: - python run.py [--iters N] [--warmup N] +Prereq: setup.py has been run (or will be auto-run) so the DB exists. -The runner *will* call setup.py for you on the first invocation if -the .db is missing — that's a one-time cost (~few minutes), cached -on disk afterward. +Usage: + python run.py [--ic ] [--iters N] [--warmup N] """ from __future__ import annotations @@ -42,27 +43,19 @@ HERE = Path(__file__).resolve().parent REPO_ROOT = HERE.parent.parent.parent -DB_PATH = REPO_ROOT / "bench/data/cross-system/graphqlite/ic2.db" -PARAMS_FILE = ( +PARAMS_DIR = ( REPO_ROOT - / "bench/data/substitution_parameters-sf0.1/substitution_parameters-sf0.1/interactive_2_param.txt" + / "bench/data/substitution_parameters-sf0.1/substitution_parameters-sf0.1" ) -QUERY_FILE = HERE / "ic2.cypher" -IC2_TOML = REPO_ROOT / "bench/ldbc-queries/ic2.toml" +DB_DIR = REPO_ROOT / "bench/data/cross-system/graphqlite" +LDBC_QUERIES_DIR = REPO_ROOT / "bench/ldbc-queries" BACKEND_LABEL = "graphqlite-cypher" -# IC2 RETURN columns in positional order. graphqlite's Cypher dialect -# returns dicts keyed by AS-aliases (`friend_id`, ...); gqlite returns -# positional `Vec`. We index dicts by these aliases so the -# shape check is positional and dialect-independent. -IC2_COLUMNS = [ - "friend_id", - "friend_firstName", - "friend_lastName", - "c_id", - "c_content", - "c_creationDate", -] + +def load_toml(path: Path) -> dict: + import tomllib + with path.open("rb") as f: + return tomllib.load(f) def shape_of_value(v) -> str: @@ -112,21 +105,9 @@ def verify_shape(actual: str, expected: str) -> str | None: return None -def load_expected_shape(toml_path: Path) -> str | None: - """Pull `expected_shape` from the IC's toml so the Python runner - verifies against the same contract the Rust runner uses. Returns - None if the field is absent. - """ - import tomllib - - with toml_path.open("rb") as f: - return tomllib.load(f).get("expected_shape") - - def load_query(path: Path) -> str: """Read the Cypher query, stripping leading // comment lines so - they don't get sent to the engine. (graphqlite likely tolerates - them, but cleaner to strip than to depend on that.) + they don't get sent to the engine. """ out_lines = [] in_comment_block = True @@ -152,18 +133,27 @@ def load_params(path: Path) -> tuple[list[str], list[list[str]]]: def coerce(s: str) -> int | str: - """Best-effort int coercion (LDBC IC2 params are ints — personId, - maxDate). Falls back to string for anything else, so the function - is reusable for other ICs later.""" + """Best-effort int coercion (LDBC params are usually ints). + Falls back to string for anything else. + """ try: return int(s) except ValueError: return s +def derive_columns(return_columns: list[str]) -> list[str]: + """Cypher AS-aliases follow `dot.path` → `dot_path`. The toml's + `return_columns` field is the source of truth for both order and + naming; we just translate the syntax.""" + return [c.replace(".", "_") for c in return_columns] + + def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("out_csv", type=Path) + ap.add_argument("--ic", type=int, default=2, + help="LDBC IC number (default: 2)") ap.add_argument("--iters", type=int, default=10) ap.add_argument("--warmup", type=int, default=2) args = ap.parse_args() @@ -172,64 +162,86 @@ def main() -> int: sys.stderr.write("--iters must be >= 1\n") return 1 - if not DB_PATH.exists(): + ic = args.ic + toml_path = LDBC_QUERIES_DIR / f"ic{ic}.toml" + query_file = HERE / f"ic{ic}.cypher" + db_path = DB_DIR / f"ic{ic}.db" + + if not toml_path.is_file(): + sys.stderr.write(f" toml missing: {toml_path}\n") + return 1 + if not query_file.is_file(): sys.stderr.write( - f" graphqlite db missing: {DB_PATH}\n" + f" cypher translation missing: {query_file}\n" + f" (write it as a translation of {toml_path})\n" + ) + return 1 + + toml = load_toml(toml_path) + if toml.get("status") != "implemented": + sys.stderr.write( + f" ic{ic}.toml status is {toml.get('status')!r}, not 'implemented'. " + f"Skipping.\n" + ) + return 0 + + params_file = PARAMS_DIR / toml["params_file"] + if not params_file.is_file(): + sys.stderr.write( + f" params file missing: {params_file}\n" + f" run ./target/release/bench_setup from the repo root first.\n" + ) + return 1 + + expected_shape = toml.get("expected_shape") + columns = derive_columns(toml.get("return_columns", [])) + query_label = f"IC{ic}" + + if not db_path.exists(): + sys.stderr.write( + f" graphqlite db missing: {db_path}\n" f" running setup.py to build it (one-time, ~minutes)...\n" ) rc = subprocess.run( - [sys.executable, str(HERE / "setup.py")], + [sys.executable, str(HERE / "setup.py"), "--ic", str(ic)], cwd=str(REPO_ROOT), ).returncode if rc != 0: sys.stderr.write(f" setup.py failed with code {rc}\n") return rc - if not PARAMS_FILE.is_file(): - sys.stderr.write( - f" params file missing: {PARAMS_FILE}\n" - f" run ./target/release/bench_setup from the repo root first.\n" - ) - return 1 - query = load_query(QUERY_FILE) - expected_shape = load_expected_shape(IC2_TOML) - header, params_rows = load_params(PARAMS_FILE) + query = load_query(query_file) + header, params_rows = load_params(params_file) sys.stderr.write( - f" graphqlite: {len(params_rows)} param rows × {args.iters} iters " + f" graphqlite ic{ic}: {len(params_rows)} param rows × {args.iters} iters " f"(+ {args.warmup} warmup)\n" ) - g = Graph(str(DB_PATH)) + g = Graph(str(db_path)) with args.out_csv.open("w", encoding="utf-8", newline="") as out: - # Same header line ldbc_bench emits. out.write("query;backend;params;row;iter;result_count;elapsed_ns\n") for row_idx, raw_row in enumerate(params_rows): param_dict = {col: coerce(val) for col, val in zip(header, raw_row)} joined = "|".join(raw_row) - # Warmup iters discarded; not written to CSV. The bench - # uses warmup to absorb any first-call JIT/cache effects - # the engine has. for _ in range(args.warmup): g.query(query, param_dict) - # Shape work runs after the iter loop, never between timed iters. iter0_result = None for n in range(args.iters): t = time.perf_counter_ns() result = g.query(query, param_dict) elapsed_ns = time.perf_counter_ns() - t out.write( - f"IC2;{BACKEND_LABEL};{joined};{row_idx};{n};" + f"{query_label};{BACKEND_LABEL};{joined};{row_idx};{n};" f"{len(result)};{elapsed_ns}\n" ) if n == 0: iter0_result = result - # Pin shape and count to iter 0; non-determinism would surface - # as both moving together, not one alone. - actual_shape = shape_of_rows(iter0_result, IC2_COLUMNS) + + actual_shape = shape_of_rows(iter0_result, columns) actual_count = len(iter0_result) if expected_shape is None: status = "no-expected" @@ -237,9 +249,9 @@ def main() -> int: why = verify_shape(actual_shape, expected_shape) status = "ok" if why is None else f'fail reason="{why}"' sys.stderr.write( - f" SHAPE row={row_idx} count={actual_count} shape={actual_shape} status={status}\n" + f" SHAPE row={row_idx} count={actual_count} " + f"shape={actual_shape} status={status}\n" ) - sys.stderr.write( f" row {row_idx}: rc={len(result)} " f"last_iter_ms={elapsed_ns / 1e6:.2f}\n" diff --git a/bench/cross-system/graphqlite/setup.py b/bench/cross-system/graphqlite/setup.py index beb1433..5aad19d 100644 --- a/bench/cross-system/graphqlite/setup.py +++ b/bench/cross-system/graphqlite/setup.py @@ -58,7 +58,11 @@ DEFAULT_CSV_DIR = REPO_ROOT / "bench/data/ldbc-sf0.1/social_network-sf0.1-CsvBasic-LongDateFormatter/dynamic" DEFAULT_DB_DIR = REPO_ROOT / "bench/data/cross-system/graphqlite" -DEFAULT_DB_PATH = DEFAULT_DB_DIR / "ic2.db" + +# Each IC gets its own pre-loaded DB (under DEFAULT_DB_DIR). For now +# we only know how to load the IC2 subset; supporting another IC +# means teaching this script which node/edge types it needs. +SUPPORTED_ICS = {2} # NOTE on API choice: graphqlite has both upsert_*_batch and @@ -152,14 +156,25 @@ def load_edges(g: Graph, path: Path, rel_type: str, id_map: dict[str, int]) -> i def main() -> int: ap = argparse.ArgumentParser() + ap.add_argument("--ic", type=int, default=2, + help="LDBC IC number (default: 2)") ap.add_argument("--force", action="store_true", - help="rebuild even if the cached ic2.db exists") + help="rebuild even if the cached db exists") ap.add_argument("--csv-dir", type=Path, default=DEFAULT_CSV_DIR) - ap.add_argument("--db", type=Path, default=DEFAULT_DB_PATH) + ap.add_argument("--db", type=Path, default=None, + help="output db path; defaults to ic.db under the cache dir") args = ap.parse_args() + if args.ic not in SUPPORTED_ICS: + print( + f"setup.py only knows how to load data for IC(s) {sorted(SUPPORTED_ICS)}.\n" + f"Add the per-IC loaders before benching ic{args.ic}.", + file=sys.stderr, + ) + return 1 + csv_dir: Path = args.csv_dir - db_path: Path = args.db + db_path: Path = args.db or (DEFAULT_DB_DIR / f"ic{args.ic}.db") if not csv_dir.is_dir(): print( diff --git a/bench/cross-system/install_python_deps.sh b/bench/cross-system/install_python_deps.sh new file mode 100644 index 0000000..bd6e5be --- /dev/null +++ b/bench/cross-system/install_python_deps.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# Install the Python dependencies for every Python-runner cross-system +# subdir in one go. +# +# Usage: +# bash bench/cross-system/install_python_deps.sh # everything +# bash bench/cross-system/install_python_deps.sh kuzu # one system +# bash bench/cross-system/install_python_deps.sh kuzu graphqlite +# +# What it does: for each named subdir (or every subdir with a +# requirements.txt if no args are given), runs `pip install -r +# /requirements.txt` against the active Python. No venv +# magic — installs into whichever Python is on PATH. If you want +# isolation, activate a venv first. +# +# Why this exists: each per-system subdir has its own +# requirements.txt because the systems' Python deps don't share +# anything (gqlitedb, kuzu, graphqlite — different wheels, different +# versions, no overlap). A new contributor coming to the bench +# shouldn't have to discover each subdir's requirements.txt +# separately; one command should set up everything. +# +# Cargo-based systems (gqlite, graphlite) aren't handled here — +# `cargo build --release` in the repo root + each system's own +# Cargo.toml handles those. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" + +if [[ $# -eq 0 ]]; then + # Every subdir that has a requirements.txt + mapfile -t subdirs < <(find "$SCRIPT_DIR" -mindepth 2 -maxdepth 2 \ + -name requirements.txt -printf '%h\n' | sort) +else + subdirs=() + for sys in "$@"; do + subdirs+=("$SCRIPT_DIR/$sys") + done +fi + +if [[ ${#subdirs[@]} -eq 0 ]]; then + echo "no per-system subdirs with requirements.txt found in $SCRIPT_DIR" >&2 + exit 1 +fi + +py=$(command -v python || command -v python3 || true) +if [[ -z "$py" ]]; then + echo "no Python on PATH; install Python or activate a venv first" >&2 + exit 1 +fi + +echo "using Python: $py" +echo "$py --version: $($py --version 2>&1)" +echo "" + +failed=() +for d in "${subdirs[@]}"; do + req="$d/requirements.txt" + if [[ ! -f "$req" ]]; then + echo "[SKIP] $(basename "$d") — no requirements.txt" + continue + fi + echo "--- $(basename "$d") ---" + if "$py" -m pip install -r "$req"; then + echo "[OK] $(basename "$d")" + else + echo "[FAIL] $(basename "$d")" + failed+=("$(basename "$d")") + fi + echo "" +done + +if [[ ${#failed[@]} -gt 0 ]]; then + echo "Failed: ${failed[*]}" >&2 + exit 1 +fi +echo "All Python deps installed." diff --git a/bench/cross-system/kuzu/DIVERGENCES.md b/bench/cross-system/kuzu/DIVERGENCES.md new file mode 100644 index 0000000..9fd2132 --- /dev/null +++ b/bench/cross-system/kuzu/DIVERGENCES.md @@ -0,0 +1,146 @@ +# Kuzu — divergences and integration notes + +Kuzu is included as a fourth external system in the cross-system +bench. This file documents: +1. Which Cypher features differ between Kuzu's dialect and the canonical + IC2 form, and how we handled each. +2. The system's archival status, why we kept it anyway, and how that + affects reproducibility. + +## Project status — archived but pinned + +[Kùzu Inc.](https://blog.kuzudb.com/post/kuzudb-update/) archived the +upstream GitHub repository on **2025-10-10** with v0.11.3 as the +final release. The team posted a blog explaining they were "working +on something new"; the repo is now read-only; the docs site +(`docs.kuzudb.com`) was already partially down at the time we wrote +this integration. Two community forks exist (bighorn, LadybugDB) but +both are weeks-to-months old as of this integration. + +**Why we kept Kuzu in the bench despite the archival:** + +1. **It's a real, peer-reviewed engineered system.** [CIDR 2023 + paper](https://www.cidrdb.org/cidr2023/papers/p48-jin.pdf) from + the Waterloo DSG, vectorized columnar engine, MIT-licensed C++, + regularly tested against LDBC SF100 in their own CI before + archival. None of the other external systems we evaluated have + this engineering provenance. +2. **Reproducibility cuts both ways with archived projects.** + The PyPI wheel `kuzu==0.11.3` is frozen — it will install + identically next month, next year. Pinning to it gives the bench + numbers a more stable reference than tracking a moving target + would. We pin in `requirements.txt`. +3. **For a research-paper comparison, "actively maintained" is not + a strict requirement.** Papers cite engines and benchmarks from + well before their publication date all the time. What matters is + that the system behaves identically when the paper is reviewed + as when we ran the bench, which an archived project guarantees. + +What we lose by using a frozen target: +- No bug fixes or performance regressions vs. a running competitor + evaluating the same query today. +- Storage format changes between past versions stranded prior data; + we get fresh DBs every setup so this doesn't bite, but it's a + reason not to put long-term reliance on Kuzu DBs. + +## Cypher dialect divergences — IC2-specific + +### 1. Label disjunction handled implicitly via the multi-typed REL TABLE + +The IC2 query needs to match messages of either label `Comment` or +`Post`. Kuzu's openCypher subset accepts neither +- `MATCH (c:Comment|Post) ...` (Cypher 5+ direct union-label form) +- `MATCH (c) ... WHERE c:Comment OR c:Post ...` (label-predicate form) + +Both fail with `Parser exception: Invalid input ...`. We initially +wrote the query with a `UNION ALL` of two MATCHes (which Kuzu does +accept), but ran into the next problem: **`LIMIT 20` after `UNION +ALL` doesn't apply to the union total.** It applies only to the +second branch, so the result was thousands of rows from branch 1 +plus 20 from branch 2 — wrong by orders of magnitude. + +Trailing `WITH ... LIMIT` after a UNION isn't supported in Kuzu's +parser. `CALL { ... } RETURN ... LIMIT 20` (the openCypher 5+ +subquery form for "limit the union total") isn't either. + +**The form that works** turned out to be much simpler: leave `(c)` +unlabeled. Because `hasCreator` is declared as a multi-typed REL +TABLE (`FROM Comment TO Person, FROM Post TO Person`), the engine +constrains the source endpoint of an unlabeled `(c)-[:hasCreator]->` +to nodes that are Comment-or-Post automatically. No union, no +predicate, no rewrite needed. The shape matches what gqlite, our +own ISO-GQL system, runs: + +```cypher +MATCH (p:Person {id: $personId})-[:knows]-(friend)<-[:hasCreator]-(c) +WHERE c.creationDate <= $maxDate +RETURN ... LIMIT 20 +``` + +So the cross-system comparison stays apples-to-apples on query +shape — every system runs "one MATCH + relationship-type-implicit +label constraint." Kuzu just expresses it more elegantly because +the REL TABLE schema does the work that the other systems push into +the WHERE clause or the parser. + +| System | Label disjunction form | +|---|---| +| gqlite (us) | `(c: Comment \| Post)` (native form, our parser) | +| graphqlite | `WHERE c:Comment OR c:Post` (their dialect rejects pipe) | +| auksys/gqlite | `WHERE c:Comment OR c:Post` (planner bug on pipe in multi-hop) | +| **Kuzu** | **`(c)` unlabeled, constrained by multi-typed `hasCreator` REL TABLE** | + +### 2. Multi-typed REL TABLE requires FROM/TO hints in COPY + +`hasCreator` connects two source label types (Comment→Person and +Post→Person). Kuzu supports declaring this in one REL TABLE: + +```cypher +CREATE REL TABLE hasCreator(FROM Comment TO Person, FROM Post TO Person) +``` + +But COPY FROM into such a table requires hints to disambiguate which +sub-table the rows belong to: + +```cypher +COPY hasCreator FROM 'comment_hasCreator_person_0_0.csv' + (DELIM='|', HEADER=true, FROM='Comment', TO='Person') +``` + +A plain `COPY hasCreator FROM ...` fails with +`Binder exception: The table hasCreator has multiple FROM and TO +pairs defined in the schema. A specific pair of FROM and TO options +is expected when copying data into the hasCreator table.` + +This is standard Kuzu usage, just slightly more verbose than the +single-typed case. + +### 3. Schema must declare every CSV column + +Kuzu's COPY FROM rejects rows where the column count doesn't match +the table schema. The LDBC Person CSV has 8 columns; our IC2 only +queries 3 (id, firstName, lastName). Rather than pre-process CSVs, +the schema in `setup.py` declares all 8 — IC2 simply doesn't +reference the unused fields. Same approach for Comment (6 cols) and +Post (8 cols). + +### 4. `knows` is materialized in both directions at load time + +LDBC's `person_knows_person_0_0.csv` records each pair once. Kuzu +has no undirected REL TABLE primitive, so we generate a reversed CSV +in `setup.py` and COPY it into the same `knows` table. Same +convention used by every other system in the cross-system bench. + +## What's NOT divergent + +- **Parameter binding** — Kuzu's `Connection.execute(query, params)` + takes `$param`-style placeholders in the query and a Python dict + whose keys do NOT include the `$` prefix. Same convention as + graphqlite. (auksys/gqlite was the odd one — its dict keys DO + include `$`.) +- **PK auto-indexing** — declaring `PRIMARY KEY(id)` on each NODE + TABLE gets us a real index for the query-time + `MATCH (p:Person {id: $personId})` lookup. No extra DDL needed. +- **Result iteration** — `QueryResult.has_next()` / + `.get_next()` returns Python-native types (int / float / str / + None / list). No custom converters needed for IC2's column types. diff --git a/bench/cross-system/kuzu/README.md b/bench/cross-system/kuzu/README.md new file mode 100644 index 0000000..8ab91c9 --- /dev/null +++ b/bench/cross-system/kuzu/README.md @@ -0,0 +1,61 @@ +# Kuzu — cross-system bench integration + +[Kuzu](https://kuzudb.com) is an embedded graph database with a +vectorized columnar engine and an openCypher subset. Originated as a +research project at the University of Waterloo (CIDR 2023 paper); +the company sponsoring the project archived the GitHub repo on +2025-10-10. The PyPI wheel still installs and the engine still works; +we pin to **kuzu 0.11.3** (the final release) so the bench numbers +are reproducible. See `DIVERGENCES.md` for the full story. + +## Prerequisites + +```bash +# From this directory, or anywhere with the repo's Python on PATH: +pip install -r requirements.txt +``` + +That installs `kuzu==0.11.3` from PyPI. No other system dependencies +(no Docker, no JVM, no Redis). Wheel is available for Python 3.8+ on +Linux, macOS, and Windows. + +You also need the LDBC SF0.1 dataset materialized (`bench_setup` from +the repo root); see `bench/cross-system/README.md` for that step — +it's shared across all per-system loaders. + +## Running just this system + +The cross-system orchestrator drives this subdir as part of +`bench/cross-system/run_all.sh`. To iterate on the Kuzu integration +in isolation: + +```bash +# One-time: load LDBC SF0.1 IC2 subset into Kuzu's native format. +# Outputs to bench/data/cross-system/kuzu/ic2.db/ (Kuzu writes a +# multi-file directory). Idempotent; --force to rebuild. +python bench/cross-system/kuzu/setup.py --ic 2 + +# Bench. Writes per-iter CSV in cross-system schema to . +python bench/cross-system/kuzu/run.py /tmp/kuzu.csv \ + --ic 2 --iters 10 --warmup 2 +``` + +`run.py` will auto-invoke `setup.py` if the DB doesn't exist. + +## Files + +| File | Purpose | +|---|---| +| `requirements.txt` | Single line: `kuzu==0.11.3` | +| `setup.py` | LDBC CSV → Kuzu native DB. Schema-first (CREATE NODE/REL TABLE), then COPY FROM with FROM/TO hints for the multi-typed `hasCreator` REL TABLE. Materializes `knows` in both directions. | +| `run.py` | Per-iter CSV emitter. Reads `ic.toml` for query metadata + params; reads `ic.cypher` for the translation; iterates the result set via `QueryResult.has_next()` / `.get_next()`. | +| `ic2.cypher` | openCypher translation of `bench/ldbc-queries/ic2.toml`. Uses `UNION ALL` of two MATCH clauses (one for `:Comment`, one for `:Post`) because Kuzu doesn't accept `(c:Comment\|Post)` or `WHERE c:Comment OR c:Post`. See `DIVERGENCES.md`. | +| `DIVERGENCES.md` | Documented divergences from the spec / the other systems' translations. The UNION ALL form is the major one. | + +## Adding new ICs + +When `bench/ldbc-queries/ic.toml` flips to `status = "implemented"`, +add `bench/cross-system/kuzu/ic.cypher` here. `setup.py`'s +`SUPPORTED_ICS` may need updating if the new IC requires LDBC nodes/ +edges we don't currently load (Forum, Tag, Place, etc.). `run.py` +needs no changes — it derives all paths from `--ic `. diff --git a/bench/cross-system/kuzu/ic2.cypher b/bench/cross-system/kuzu/ic2.cypher new file mode 100644 index 0000000..b045435 --- /dev/null +++ b/bench/cross-system/kuzu/ic2.cypher @@ -0,0 +1,35 @@ +// openCypher translation of bench/ldbc-queries/ic2.toml for Kuzu +// (kuzudb 0.11.3 from PyPI). +// +// Source-of-truth IC2 lives in that toml; this file is a language +// translation. Substitution placeholders use Cypher's native `$param` +// syntax — Kuzu supports this directly via +// `Connection.execute(query, parameters)` where parameters is a dict +// whose keys do NOT include the `$` prefix. +// +// Divergences from spec, applied to every system for apples-to-apples: +// - no ORDER BY (gqlite parser doesn't support it; we drop it from +// this translation even though Kuzu handles it fine — fairness +// with our own engine) +// - no `coalesce(c.content, c.imageFile)` — we return c.content +// directly (Comment+Post both have a `content` column in our +// loaded subset; LDBC uses imageFile only for image-only Posts +// which aren't in the subset we load) +// - lowercase :knows / :hasCreator (loader convention) +// +// Structural shape — same as gqlite's: one MATCH with the message +// node `(c)` left unlabeled. Kuzu can resolve `c` automatically +// because `hasCreator` is declared as a multi-typed REL TABLE +// (`FROM Comment TO Person, FROM Post TO Person`), so the engine +// constrains `c` to be a Comment or a Post implicitly via the +// relationship type. This matches the canonical IC2 shape (one +// MATCH + implicit label disjunction) and avoids the `UNION ALL` +// rewrite we initially tried — which Kuzu accepts but which has +// awkward LIMIT semantics (LIMIT applies only to the second branch). +// See DIVERGENCES.md for the LIMIT-over-UNION ALL story. +MATCH (p:Person {id: $personId})-[:knows]-(friend:Person)<-[:hasCreator]-(c) +WHERE c.creationDate <= $maxDate +RETURN friend.id AS friend_id, friend.firstName AS friend_firstName, + friend.lastName AS friend_lastName, + c.id AS c_id, c.content AS c_content, c.creationDate AS c_creationDate +LIMIT 20 diff --git a/bench/cross-system/kuzu/requirements.txt b/bench/cross-system/kuzu/requirements.txt new file mode 100644 index 0000000..bef3437 --- /dev/null +++ b/bench/cross-system/kuzu/requirements.txt @@ -0,0 +1 @@ +kuzu==0.11.3 diff --git a/bench/cross-system/kuzu/run.py b/bench/cross-system/kuzu/run.py new file mode 100644 index 0000000..f57d8c8 --- /dev/null +++ b/bench/cross-system/kuzu/run.py @@ -0,0 +1,270 @@ +#!/usr/bin/env python3 +"""Run a chosen IC against Kuzu (kuzudb on PyPI) for every +substitution-param row, emitting per-iter CSV in the cross-system +schema. + +Output schema (matches src/bin/ldbc_bench.rs): + query;backend;params;row;iter;result_count;elapsed_ns + +`backend` is fixed to `kuzu-cypher`. `params` is the raw pipe-joined +param row from the LDBC params file. `row` is the 0-based param-row +index. + +Per-IC inputs (all derived from --ic ): + bench/ldbc-queries/ic.toml — query metadata + bench/cross-system/kuzu/ic.cypher — Kuzu translation + bench/data/substitution_parameters-sf0.1/.../ + bench/data/cross-system/kuzu/ic.db — pre-loaded DB + +Prereq: setup.py has been run (or will be auto-run) so the DB exists. + +Usage: + python run.py [--ic ] [--iters N] [--warmup N] +""" + +from __future__ import annotations + +import argparse +import sys +import subprocess +import time +from pathlib import Path + +try: + import kuzu +except ImportError: + sys.stderr.write( + "kuzu not installed. From this directory:\n" + " pip install -r requirements.txt\n" + ) + sys.exit(1) + + +HERE = Path(__file__).resolve().parent +REPO_ROOT = HERE.parent.parent.parent + +PARAMS_DIR = ( + REPO_ROOT + / "bench/data/substitution_parameters-sf0.1/substitution_parameters-sf0.1" +) +DB_DIR = REPO_ROOT / "bench/data/cross-system/kuzu" +LDBC_QUERIES_DIR = REPO_ROOT / "bench/ldbc-queries" +BACKEND_LABEL = "kuzu-cypher" + + +def load_toml(path: Path) -> dict: + import tomllib + with path.open("rb") as f: + return tomllib.load(f) + + +def shape_of_value(v) -> str: + """Mirror of `shape_of_value` in src/bin/ldbc_bench.rs — keep in + sync. Kuzu returns Python-native types from RETURN clauses + (int / float / str / None / list), so the standard mapping + applies. INT64 becomes Python int, STRING becomes str, etc. + """ + if v is None: + return "n" + if isinstance(v, bool): + return "b" + if isinstance(v, int): + return "i" + if isinstance(v, float): + return "f" + if isinstance(v, str): + return "s" + if isinstance(v, list): + return "l" + return "r" + + +def shape_of_rows(rows: list[list], n_columns: int) -> str: + if not rows: + return "empty" + cols: list[set[str]] = [set() for _ in range(n_columns)] + for r in rows: + for i in range(min(n_columns, len(r))): + cols[i].add(shape_of_value(r[i])) + return ",".join("/".join(sorted(s)) for s in cols) + + +def verify_shape(actual: str, expected: str) -> str | None: + a = [set(c.split("/")) for c in actual.split(",")] + e = [set(c.split("/")) for c in expected.split(",")] + if len(a) != len(e): + return f"column count: actual={len(a)}, expected={len(e)}" + for i, (ac, ec) in enumerate(zip(a, e)): + if not ac.issubset(ec): + extras = sorted(ac - ec) + return f"col {i}: actual {sorted(ac)} not subset of expected {sorted(ec)} (extra: {extras})" + return None + + +def load_query(path: Path) -> str: + """Read the Cypher query, stripping leading // comment lines so + they don't get sent to the engine. + """ + out_lines = [] + in_comment_block = True + for line in path.read_text(encoding="utf-8").splitlines(): + if in_comment_block and (line.startswith("//") or not line.strip()): + continue + in_comment_block = False + out_lines.append(line) + return "\n".join(out_lines).strip() + + +def load_params(path: Path) -> tuple[list[str], list[list[str]]]: + with path.open(encoding="utf-8") as f: + lines = [ln.rstrip("\n\r") for ln in f if ln.strip()] + if len(lines) < 2: + raise RuntimeError(f"params file too short: {path}") + header = lines[0].split("|") + data = [ln.split("|") for ln in lines[1:]] + return header, data + + +def coerce(s: str) -> int | str: + try: + return int(s) + except ValueError: + return s + + +def derive_columns(return_columns: list[str]) -> list[str]: + return [c.replace(".", "_") for c in return_columns] + + +def run_query(conn, query: str, bindings: dict) -> tuple[list[str], list[list]]: + """Execute, return (column_names, data_rows). Kuzu's `execute` + returns a QueryResult with `has_next()` / `get_next()` iteration + and `get_column_names()`. We materialize all rows because the + runner needs `len(rows)` and shape-of-rows. + """ + result = conn.execute(query, bindings) + cols = result.get_column_names() + rows = [] + while result.has_next(): + rows.append(result.get_next()) + return cols, rows + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("out_csv", type=Path) + ap.add_argument("--ic", type=int, default=2) + ap.add_argument("--iters", type=int, default=10) + ap.add_argument("--warmup", type=int, default=2) + args = ap.parse_args() + + if args.iters < 1: + sys.stderr.write("--iters must be >= 1\n") + return 1 + + ic = args.ic + toml_path = LDBC_QUERIES_DIR / f"ic{ic}.toml" + query_file = HERE / f"ic{ic}.cypher" + db_path = DB_DIR / f"ic{ic}.db" + + if not toml_path.is_file(): + sys.stderr.write(f" toml missing: {toml_path}\n") + return 1 + if not query_file.is_file(): + sys.stderr.write( + f" cypher translation missing: {query_file}\n" + f" (write it as a translation of {toml_path})\n" + ) + return 1 + + toml = load_toml(toml_path) + if toml.get("status") != "implemented": + sys.stderr.write( + f" ic{ic}.toml status is {toml.get('status')!r}, not 'implemented'. " + f"Skipping.\n" + ) + return 0 + + params_file = PARAMS_DIR / toml["params_file"] + if not params_file.is_file(): + sys.stderr.write( + f" params file missing: {params_file}\n" + f" run ./target/release/bench_setup from the repo root first.\n" + ) + return 1 + + expected_shape = toml.get("expected_shape") + columns = derive_columns(toml.get("return_columns", [])) + query_label = f"IC{ic}" + + if not db_path.exists(): + sys.stderr.write( + f" kuzu db missing: {db_path}\n" + f" running setup.py to build it (one-time, ~minute)...\n" + ) + rc = subprocess.run( + [sys.executable, str(HERE / "setup.py"), "--ic", str(ic)], + cwd=str(REPO_ROOT), + ).returncode + if rc != 0: + sys.stderr.write(f" setup.py failed with code {rc}\n") + return rc + + query = load_query(query_file) + header, params_rows = load_params(params_file) + sys.stderr.write( + f" kuzu ic{ic}: {len(params_rows)} param rows × {args.iters} iters " + f"(+ {args.warmup} warmup)\n" + ) + + db = kuzu.Database(str(db_path)) + conn = kuzu.Connection(db) + + with args.out_csv.open("w", encoding="utf-8", newline="") as out: + out.write("query;backend;params;row;iter;result_count;elapsed_ns\n") + + for row_idx, raw_row in enumerate(params_rows): + # Kuzu's parameter dict keys do NOT include the `$` prefix + # — the engine maps `$name` in the query to `name` in the + # dict. + param_dict = {col: coerce(val) for col, val in zip(header, raw_row)} + joined = "|".join(raw_row) + + for _ in range(args.warmup): + run_query(conn, query, param_dict) + + iter0_rows = None + elapsed_ns = 0 + for n in range(args.iters): + t = time.perf_counter_ns() + _hdr, rows = run_query(conn, query, param_dict) + elapsed_ns = time.perf_counter_ns() - t + out.write( + f"{query_label};{BACKEND_LABEL};{joined};{row_idx};{n};" + f"{len(rows)};{elapsed_ns}\n" + ) + if n == 0: + iter0_rows = rows + + actual_shape = shape_of_rows(iter0_rows or [], len(columns)) + actual_count = len(iter0_rows or []) + if expected_shape is None: + status = "no-expected" + else: + why = verify_shape(actual_shape, expected_shape) + status = "ok" if why is None else f'fail reason="{why}"' + sys.stderr.write( + f" SHAPE row={row_idx} count={actual_count} " + f"shape={actual_shape} status={status}\n" + ) + sys.stderr.write( + f" row {row_idx}: rc={actual_count} " + f"last_iter_ms={elapsed_ns / 1e6:.2f}\n" + ) + + sys.stderr.write(f" done -> {args.out_csv}\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/bench/cross-system/kuzu/setup.py b/bench/cross-system/kuzu/setup.py new file mode 100644 index 0000000..4c1f9cf --- /dev/null +++ b/bench/cross-system/kuzu/setup.py @@ -0,0 +1,261 @@ +#!/usr/bin/env python3 +"""Load the IC2-relevant subset of LDBC SNB SF0.1 CSVs into a Kuzu +database (kuzudb on PyPI: `pip install kuzu==0.11.3`). + +IC2 references: + - Person (id, firstName, lastName) — Person table holds all 8 LDBC + columns; only the three are queried. + - Comment (id, creationDate, content) — table holds all 6 columns. + - Post (id, creationDate, content) — table holds all 8 columns. + - knows (Person—Person, undirected per LDBC; Kuzu doesn't have + undirected REL TABLEs so we materialize both directions). + - hasCreator (Comment→Person, Post→Person) — a multi-typed REL + TABLE in Kuzu, declared with two FROM/TO pairs. + +Output: `bench/data/cross-system/kuzu/ic2.db/` (Kuzu writes a +multi-file directory, not a single file). +Idempotent: skips if the directory already exists. Pass --force to +rebuild. + +Loading idiom — taken from Kuzu's own LDBC examples and docs: + 1. CREATE NODE TABLE / CREATE REL TABLE (schema-first; PK is + auto-indexed). + 2. COPY FROM '' (DELIM='|', HEADER=true) — native + bulk loader. Multi-typed REL TABLEs need (FROM='X', TO='Y') hints + so Kuzu knows which sub-table the rows belong to. + +Throughput on this engine: ~10K nodes/sec via COPY FROM in our +microbench; SF0.1 (~289K nodes + ~315K edges) lands in ~30-60 sec +including the both-directions knows materialization. One-time cost. + +Usage: + python setup.py [--ic 2] [--force] [--csv-dir ] [--db ] +""" + +from __future__ import annotations + +import argparse +import os +import shutil +import sys +import time +from pathlib import Path + +try: + import kuzu +except ImportError: + sys.stderr.write( + "kuzu not installed. From this directory:\n" + " pip install -r requirements.txt\n" + ) + sys.exit(1) + + +HERE = Path(__file__).resolve().parent +REPO_ROOT = HERE.parent.parent.parent + +DEFAULT_CSV_DIR = REPO_ROOT / "bench/data/ldbc-sf0.1/social_network-sf0.1-CsvBasic-LongDateFormatter/dynamic" +DEFAULT_DB_DIR = REPO_ROOT / "bench/data/cross-system/kuzu" + +SUPPORTED_ICS = {2} + + +def schema_statements() -> list[str]: + """All schema DDL up-front. PK declarations get auto-indexed by + Kuzu, so post-load `MATCH (p:Person {id: X})` is a btree-style + point lookup — no full scan, no expression-index DDL needed. + + Kuzu requires every CSV column to map to a typed schema column, + so even fields IC2 doesn't query (gender, browserUsed, etc.) are + declared here so COPY FROM doesn't reject the rows. The runtime + cost of carrying unused columns is trivial vs the alternative + (pre-processing CSVs). + """ + return [ + # All 8 Person columns from person_0_0.csv + """CREATE NODE TABLE Person( + id INT64, + firstName STRING, + lastName STRING, + gender STRING, + birthday INT64, + creationDate INT64, + locationIP STRING, + browserUsed STRING, + PRIMARY KEY(id) + )""", + # All 6 Comment columns from comment_0_0.csv + """CREATE NODE TABLE Comment( + id INT64, + creationDate INT64, + locationIP STRING, + browserUsed STRING, + content STRING, + length INT64, + PRIMARY KEY(id) + )""", + # All 8 Post columns from post_0_0.csv + """CREATE NODE TABLE Post( + id INT64, + imageFile STRING, + creationDate INT64, + locationIP STRING, + browserUsed STRING, + language STRING, + content STRING, + length INT64, + PRIMARY KEY(id) + )""", + # knows is undirected per LDBC; Kuzu has no undirected REL + # TABLE primitive so we materialize both directions in the + # loader (consistent with the convention used by every other + # system in the cross-system bench). The CSV has 3 columns + # (Person.id|Person.id|creationDate); we declare creationDate + # on the edge to match. + "CREATE REL TABLE knows(FROM Person TO Person, creationDate INT64)", + # hasCreator is multi-typed (Comment→Person, Post→Person). + # Kuzu supports this natively in one REL TABLE declaration. + # Each COPY FROM into this table needs FROM/TO hints to + # disambiguate. + "CREATE REL TABLE hasCreator(FROM Comment TO Person, FROM Post TO Person)", + ] + + +def load_nodes(conn, table: str, csv_path: Path) -> None: + csv_str = str(csv_path).replace("\\", "/") + t = time.perf_counter() + conn.execute( + f"COPY {table} FROM '{csv_str}' (DELIM='|', HEADER=true)" + ) + elapsed = time.perf_counter() - t + n = conn.execute(f"MATCH (n:{table}) RETURN count(n)").get_next()[0] + print(f" {table.lower()}s: {n} loaded in {elapsed:.2f}s", file=sys.stderr) + + +def load_knows_both_directions(conn, csv_path: Path, work_dir: Path) -> None: + """LDBC's person_knows_person CSV stores each pair once. We need + both directions for the IC2 undirected `(p)-[:knows]-(f)` pattern, + so we generate a second CSV with src/dst swapped and COPY both. + The reversed CSV lives under work_dir (gitignored bench/data/ + subtree) so it's regenerated each setup but never tracked. + """ + t = time.perf_counter() + csv_str = str(csv_path).replace("\\", "/") + conn.execute( + f"COPY knows FROM '{csv_str}' (DELIM='|', HEADER=true)" + ) + elapsed_fwd = time.perf_counter() - t + + # Reversed CSV + reversed_csv = work_dir / "person_knows_person_reversed.csv" + work_dir.mkdir(parents=True, exist_ok=True) + t = time.perf_counter() + with csv_path.open(encoding="utf-8") as f_in, reversed_csv.open( + "w", encoding="utf-8", newline="" + ) as f_out: + header = f_in.readline() + f_out.write(header) + for line in f_in: + parts = line.rstrip("\n\r").split("|") + if len(parts) < 2: + continue + # Swap src and dst, keep creationDate column unchanged. + swapped = [parts[1], parts[0]] + parts[2:] + f_out.write("|".join(swapped) + "\n") + rev_str = str(reversed_csv).replace("\\", "/") + conn.execute( + f"COPY knows FROM '{rev_str}' (DELIM='|', HEADER=true)" + ) + elapsed_rev = time.perf_counter() - t + n = conn.execute("MATCH ()-[k:knows]->() RETURN count(k)").get_next()[0] + print( + f" knows edges: {n} loaded " + f"({elapsed_fwd:.2f}s fwd + {elapsed_rev:.2f}s reversed)", + file=sys.stderr, + ) + + +def load_has_creator(conn, csv_path: Path, src_label: str) -> None: + csv_str = str(csv_path).replace("\\", "/") + t = time.perf_counter() + conn.execute( + f"COPY hasCreator FROM '{csv_str}' " + f"(DELIM='|', HEADER=true, FROM='{src_label}', TO='Person')" + ) + elapsed = time.perf_counter() - t + print( + f" {src_label.lower()} hasCreator: loaded in {elapsed:.2f}s", + file=sys.stderr, + ) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--ic", type=int, default=2) + ap.add_argument("--force", action="store_true") + ap.add_argument("--csv-dir", type=Path, default=DEFAULT_CSV_DIR) + ap.add_argument("--db", type=Path, default=None) + args = ap.parse_args() + + if args.ic not in SUPPORTED_ICS: + print( + f"setup.py only supports IC(s) {sorted(SUPPORTED_ICS)}; got ic{args.ic}", + file=sys.stderr, + ) + return 1 + + csv_dir: Path = args.csv_dir + db_path: Path = args.db or (DEFAULT_DB_DIR / f"ic{args.ic}.db") + + if not csv_dir.is_dir(): + print( + f"CSV dir not found: {csv_dir}\n" + f"Run ./target/release/bench_setup from the repo root first.", + file=sys.stderr, + ) + return 1 + + if db_path.exists() and not args.force: + print(f" cached: {db_path} (pass --force to rebuild)", file=sys.stderr) + return 0 + + if db_path.exists() and args.force: + # Kuzu writes a multi-file directory, not a single file. + if db_path.is_dir(): + shutil.rmtree(db_path) + else: + db_path.unlink() + db_path.parent.mkdir(parents=True, exist_ok=True) + + print(f" building {db_path} from {csv_dir}", file=sys.stderr) + t0 = time.perf_counter() + + db = kuzu.Database(str(db_path)) + conn = kuzu.Connection(db) + + print(" creating schema...", file=sys.stderr) + for stmt in schema_statements(): + conn.execute(stmt) + + load_nodes(conn, "Person", csv_dir / "person_0_0.csv") + load_nodes(conn, "Comment", csv_dir / "comment_0_0.csv") + load_nodes(conn, "Post", csv_dir / "post_0_0.csv") + + work_dir = db_path.parent / "_kuzu_work" + load_knows_both_directions( + conn, csv_dir / "person_knows_person_0_0.csv", work_dir + ) + load_has_creator( + conn, csv_dir / "comment_hasCreator_person_0_0.csv", src_label="Comment" + ) + load_has_creator( + conn, csv_dir / "post_hasCreator_person_0_0.csv", src_label="Post" + ) + + elapsed = time.perf_counter() - t0 + print(f" done in {elapsed:.1f}s. db at {db_path}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/bench/cross-system/run_all.sh b/bench/cross-system/run_all.sh index 97c4722..09ab099 100644 --- a/bench/cross-system/run_all.sh +++ b/bench/cross-system/run_all.sh @@ -1,12 +1,16 @@ #!/usr/bin/env bash -# Cross-system IC2 bench — orchestrator. +# Cross-system bench — orchestrator. # -# Invokes every per-system runner in turn, captures their CSV output -# into a timestamped results dir, then runs compare_results.py to -# produce a side-by-side comparison table. +# Invokes every per-system runner in turn for a chosen IC, captures +# each runner's CSV into a timestamped results dir, then runs +# compare_results.py to produce a side-by-side comparison table. # # Usage: -# bench/cross-system/run_all.sh [--iters N] [--warmup N] [--only ] +# bench/cross-system/run_all.sh [--ic ] [--iters N] [--warmup N] [--only ] +# +# Default --ic is 2. Each invocation runs one IC against every +# (selected) system. For a multi-IC sweep, invoke this script once +# per IC — each run lands in its own timestamped dir. # # --only takes a comma-separated subset of system names, e.g. # --only gqlite # just our system (debugging) @@ -17,9 +21,11 @@ # gqlite.csv # graphqlite.csv (if integrated) # graphlite.csv (if integrated) +# kuzu.csv (if integrated; kuzudb.com / kuzu on PyPI) # webbery_gqlite.csv (if integrated; SKIPPED.md otherwise) # cross_system.csv (concatenation of all the above) # comparison.txt (compare_results.py output) +# run_info.txt (metadata: timestamp, host, ic, etc.) # # Prereq: ./target/release/bench_setup has been run so the LDBC SF0.1 # dataset (~17 MiB) is downloaded and built into ldbc-sf0.1.gdb. @@ -32,11 +38,13 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" RESULTS_ROOT="$SCRIPT_DIR/results" +IC=2 ITERS=10 WARMUP=2 ONLY="" while [[ $# -gt 0 ]]; do case "$1" in + --ic) IC="$2"; shift 2 ;; --iters) ITERS="$2"; shift 2 ;; --warmup) WARMUP="$2"; shift 2 ;; --only) ONLY="$2"; shift 2 ;; @@ -58,6 +66,7 @@ START_EPOCH=$(date +%s) # missing fields surface as empty values, not errors. { echo "timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "ic: $IC" echo "host: $(hostname 2>/dev/null || echo unknown)" echo "uname: $(uname -a 2>/dev/null || echo unknown)" echo "gqlite_branch: $(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)" @@ -70,7 +79,7 @@ START_EPOCH=$(date +%s) # Systems registered here. Order matters only for cosmetic per-system # stderr output; the comparison script sorts independently. -ALL_SYSTEMS=(gqlite graphqlite graphlite webbery_gqlite) +ALL_SYSTEMS=(gqlite graphqlite kuzu graphlite webbery_gqlite) # Filter via --only. if [[ -n "$ONLY" ]]; then @@ -80,8 +89,9 @@ else fi cd "$REPO_ROOT" -echo "=== Cross-system IC2 bench — $TIMESTAMP ===" +echo "=== Cross-system IC$IC bench — $TIMESTAMP ===" echo " results: $OUT_DIR" +echo " ic: $IC" echo " iters: $ITERS" echo " warmup: $WARMUP" echo " systems: ${SYSTEMS[*]}" @@ -96,6 +106,7 @@ for sys in "${SYSTEMS[@]}"; do gqlite) runner="$SCRIPT_DIR/gqlite/run.sh" ;; graphqlite) runner="$SCRIPT_DIR/graphqlite/run.py" ;; graphlite) runner="$SCRIPT_DIR/graphlite/run.sh" ;; # wraps cargo run + kuzu) runner="$SCRIPT_DIR/kuzu/run.py" ;; webbery_gqlite) runner="$SCRIPT_DIR/webbery_gqlite/run.sh" ;; esac @@ -112,10 +123,11 @@ for sys in "${SYSTEMS[@]}"; do stderr_log="$OUT_DIR/${sys}.stderr.log" case "$sys" in - graphqlite) python "$runner" "$out_csv" --iters "$ITERS" --warmup "$WARMUP" \ + graphqlite|kuzu) + python "$runner" "$out_csv" --ic "$IC" --iters "$ITERS" --warmup "$WARMUP" \ 2>"$stderr_log" || \ echo "[FAIL] $sys runner returned non-zero" | tee -a "$OUT_DIR/skipped.log" ;; - *) bash "$runner" "$out_csv" --iters "$ITERS" --warmup "$WARMUP" \ + *) bash "$runner" "$out_csv" --ic "$IC" --iters "$ITERS" --warmup "$WARMUP" \ 2>"$stderr_log" || \ echo "[FAIL] $sys runner returned non-zero" | tee -a "$OUT_DIR/skipped.log" ;; esac diff --git a/bench/cross-system/webbery_gqlite/SKIPPED.md b/bench/cross-system/webbery_gqlite/SKIPPED.md new file mode 100644 index 0000000..60ac190 --- /dev/null +++ b/bench/cross-system/webbery_gqlite/SKIPPED.md @@ -0,0 +1,329 @@ +# webbery/gqlite — discarded + +We attempted to integrate **[webbery/gqlite](https://github.com/webbery/gqlite)** +as a fourth external system in the cross-system bench. After +**reading the full README, the bison grammar (`src/gql.y`), the +lexer (`src/gql.l`), the public C API (`include/gqlite.h`), and the +representative tests (`test/movielens.cpp`, `test/grammar.cpp`, +`test/benchmark.cpp`)**, we determined the system cannot be benched +against LDBC SNB IC2. + +The original [bench plan](../../README.md) flagged this project as +"dead since April 2023" and time-boxed it at three days. We found +this was correct in substance and concrete in detail; this file +records the specific blockers so the rejection is reviewable rather +than vibes-based. + +## What it is + +C++17 graph database, bundled libmdbx (LMDB-derivative) for storage. +- 207 commits, master branch +- Last commit: **2023-04-08** (over 2 years before this review) +- Custom JSON-like DSL (the README calls it "Graph Query Language", + marked as "Unstable") +- C API only; no Python, Ruby, Crystal, or other bindings +- Bison/flex parser; CMake build with bundled flex/bison binaries + for Windows; submodules: eigen, fmt, libmdbx +- Targeted at "ending device" — IoT/edge form factors +- README states the DSL is "Unstable"; CHANGELOG.md is one line: + "vertex operation support" + +## Blocker 1 — DSL cannot express top-level `LIMIT` + +The bison grammar (`src/gql.y`) DECLARES a `limit` token (line 115) +and the lexer (`src/gql.l` line 147) emits it inside the `` +exclusive-state block whenever it sees the literal `"limit"` keyword: + +```yacc +%token limit profile property +``` +```lex +"limit" { stm._errIndx += yyleng; return limit;}; +``` + +But **NO grammar production rule references the `limit` token in +`src/gql.y`.** Verified by both `grep -n 'limit' src/gql.y` +(returns 4 hits, all of which are: line 59/62/65 `numeric_limits` +from C++ stdlib calls, and line 115 the `%token` declaration +itself) and an `awk` pass that found zero production-rule bodies +mentioning `limit`. The complete query production is +(`gql.y` lines 362-379): + +```yacc +'{' query_kind '}' // query w/o graph or where +'{' query_kind ',' a_graph_expr '}' // query w/ graph +'{' query_kind ',' a_graph_expr ',' where_expr '}' // query w/ graph + where +``` + +That's the entire query grammar. No `LIMIT N`, no projection beyond +`query_kind`, no `ORDER BY`. + +> **Caveat noticed:** `test/vertex/grammar.gql:35` contains +> `{query: '废墟', in: 'vertex_db', where: {feature_name: {limit: 3, +> $near: [...]}}};` — suggesting `limit` is intended as an option +> *inside* the `$near` (vector-similarity) operator clause. But +> tracing through `condition_property → right_value → condition_object +> → condition_properties → condition_property → ... → OP_NEAR ':' '{' +> geometry_condition '}'` and reading `geometry_condition` (gql.y:813, +> `OP_GEOMETRY ':' a_vector ',' range_comparable`), we find no +> production that allows the `limit` token to appear there either. +> The test is likely either aspirational (future-version syntax) or +> a known-failure regression case. Even if `limit:` somehow did +> parse inside `$near`, that would be vector-search top-k semantics +> ("k nearest neighbors"), which is **not** the result-set LIMIT +> that IC2 requires (`LIMIT 20` after the projection of arbitrary +> matched rows). + +**For LDBC IC2, which requires `LIMIT 20`**, this is a hard blocker. +Without LIMIT, every IC2 invocation would return thousands of rows +(matching the issue we hit with Kuzu when LIMIT was attached to the +wrong UNION branch). Fairness vs. the other systems' 20-row results +would be lost. + +## Blocker 2 — No label-disjunction syntax + +IC2 needs to match messages of either label `Comment` or `Post`. +The other systems express this in their native idioms: + +| System | Label-disjunction form | +|---|---| +| gqlite | `(c: Comment \| Post)` | +| graphqlite | `WHERE c:Comment OR c:Post` | +| auksys/gqlite | `WHERE c:Comment OR c:Post` | +| Kuzu | unlabeled `(c)` constrained by multi-typed REL TABLE | + +webbery/gqlite's `query_kind_expr` rule (`gql.y:436`) is: + +```yacc +query_kind_expr: + KW_EDGE + | LITERAL_STRING + | a_graph_properties +``` + +i.e. the query target is exactly one of: the `edge` keyword (all +edges), a single string label like `'movie'`, or a property +accessor like `movie.title`. **No production accepts an array of +label names.** The `in:` clause (gql.y:442 — `KW_IN ':' +LITERAL_STRING`) is also single-string — the graph instance. + +WHERE clauses (gql.y:448-449, 700-790) are MongoDB-style operator +predicates on property values: + +```javascript +{ + query: 'movie', + where: {tag: ['black comedy']} +} +``` + +with `$lt`, `$gt`, `$gte`, `$lte`, `$and`, `$or`, `$near`, +`$geometry`. **None of them are label predicates** — there's no +`{label: 'Comment' or 'Post'}` form. Querying two labels would +require two separate query-statements with results combined in +client code, which is structurally different from what the other +systems run and breaks the bench's "same logical shape" rule. + +## Blocker 3 — No bulk-load API + +The canonical loading pattern from `test/movielens.cpp` (their own +example loader for the MovieLens dataset, ~10K movies, ~5K tags): + +```cpp +char upset[512] = { 0 }; +sprintf(upset, + "{upset: 'movie', vertex: [[%d, {title: '%s', genres: '%s'}]]};", + id, title.c_str(), genres.c_str()); +gqlite_exec(pHandle, upset, gqlite_exec_callback, nullptr, &ptr); +``` + +**One DSL exec per CSV row.** This is the same architectural shape +that made auksys/gqlite take indefinite hours on LDBC SF0.1 (~600K +entities including MVAs). We don't need to retest the failure mode; +we know how it ends. + +The `gqlite_create` / `gqlite_execute` / `gqlite_next` prepared- +statement API exists (see `include/gqlite.h`) but the readme + +tests don't show it being used for bulk inserts; it's positioned +for repeated reads, not writes. + +## Blocker 4 — Buffer overflow risks in their own load idiom + +The 512-byte buffer above is theirs. LDBC content fields are 80+ +characters; with proper escaping, JSON-encoding, and group/property +keys, a single `upset` statement easily exceeds 512 bytes for any +real `Comment` or `Post` row. Any harness we built would have to +fix the example pattern just to load LDBC text fields without +truncation. + +## Blocker 5 — C-only API; no per-system harness shape + +graphqlite, kuzu, and auksys/gqlite all have Python bindings. Our +existing per-system runners (`run.py`) wrap those bindings. Webbery +provides only a C API (with C++ examples). To integrate it we'd +need to write a C++ harness around `gqlite_exec` + a callback — +roughly the same scope as the per-system Rust harness we wrote for +GraphLite, except in a less mature toolchain (CMake submodule build +with flex/bison rather than a single Cargo command). + +## Blocker 6 — Designed-for scale + +The README states GQLite's purpose is "for testing abilities in +ending device" and the goal is "small, fast, light-weight." Their +own reference tests (`test/movielens.cpp`) run on the MovieLens +small dataset (~10K movies, ~100K tags, ~100K ratings). LDBC SF0.1 +is **30× larger** (~327K nodes, ~1.5M edges) and is the smallest +LDBC scale factor — i.e. SF1 / SF10 / SF100 are 10× / 100× / 1000× +this. The system was not designed for or tested at the scale this +bench runs. + +## What it would have taken to integrate + +For completeness, the integration shape if we'd pushed through: + +1. Build with CMake (~we tried, hit blockers — see below). +2. Write a C++ bench harness (`run.cpp`) that opens the DB, parses + IC2 params from the LDBC params file, builds the per-param-row + DSL string, executes via `gqlite_exec`, captures result rows in + a callback, emits the cross-system CSV (~150 lines). +3. Write a C++ loader (`setup.cpp`) that streams every LDBC CSV and + issues one `upset` per row (~600K calls for SF0.1). Plus an MVA + pre-aggregation pass. +4. Write IC2 in their DSL **somehow** — without LIMIT, with two + separate queries (one per label) combined in C++, accepting that + results are unbounded per query. +5. Hope the `libmdbx` storage handles LDBC scale without crashing. + +Estimated effort: 1-2 working days, with a high probability that +the `limit`-less DSL produces results that aren't comparable to the +other systems' LIMIT-20 outputs anyway. + +## Build attempt (Windows, May 2026): three sequential failures + +For completeness, we tried to actually build the library on a +Windows machine before settling on the discard verdict. Three +separate failures, each requiring manual intervention: + +### Failure 1 — git-describe submodule version + +Cloning with `git clone --depth 1 --recursive` (the README's +recommendation) brings the `libmdbx` submodule at commit +`d5e4c198` from 2022, which predates any tagged release in our +shallow checkout. CMake configure fails at +`third_party/libmdbx/cmake/utils.cmake:84`: + +``` +fatal: No names found, cannot describe anything. +CMake Error: Please fetch tags and/or install latest version of +git ('describe --tags --long --dirty' failed) +``` + +Worked around by `git fetch --tags --unshallow` in the submodule, +then creating a synthetic `v0.0.0-bench` tag at the pinned commit +so libmdbx's CMake tag-matcher succeeds. + +### Failure 2 — flex/bison not on PATH for MSBuild custom commands + +The README claims: *"An version of flex&bison is placed in dir +`tool`. So it's not need to install dependency."* But the CMake +custom command at `tool/CMakeLists.txt` uses `WORKING_DIRECTORY +${CMAKE_SOURCE_DIR}/tool` and runs `flex` bare, expecting Windows +to find `flex.exe` via the working directory. **MSBuild does not +honor cwd-based binary lookup for custom build steps**: + +``` +"flex" no se reconoce como un comando interno o externo, +programa o archivo por lotes ejecutable. +[generated_tokens.vcxproj] error MSB8066: Custom build for +'CMakeFiles/.../generated_tokens.rule' exited with code 9009. +``` + +Worked around by pre-generating `include/token.cpp`, `token.h`, +and `gql.cpp` manually with the bundled `tool/flex.exe` and +`tool/bison.exe`, AND by prefixing `tool/` to `PATH` for the +`cmake --build` invocation. Without both fixes, the parser/lexer +never gets generated. + +### Failure 3 — libmdbx version-mismatch in generated `version.c` + +After the build advances past parser generation, libmdbx's +`version.c` (CMake-templated from `version.c.in`) fires a +compile-time `#error`: + +```c +#if MDBX_VERSION_MAJOR != 0 || MDBX_VERSION_MINOR != 0 +#error "API version mismatch! Had `git fetch --tags` done?" +#endif +``` + +The generated version.c writes `0 / 0` (because the synthetic tag +we created has no version), but `mdbx.h` defines +`MDBX_VERSION_MAJOR=0, MINOR=11`. The mismatch fires the +preprocessor `#error`. Worked around by patching the generated +version.c to remove the `#error`. + +### Failure 4 — MASM assembly file not built/linked + +After all of the above, the build advances to linking the gqlite +library and fails with: + +``` +LINK : fatal error LNK1181: cannot open input file +'gqlite.dir\Release\jump_x86_64_ms_pe_masm.obj' +[gqlite.vcxproj] +``` + +The repo contains x86_64 hand-written MASM assembly files +(README states 18.2% of the codebase is "Assembly") for what +appear to be Boost.Context-style coroutine `jump_*`/`make_*` +context switches. CMake doesn't enable MASM (`enable_language(ASM_MASM)`) +in the project, so MSBuild compiles the `.cpp` units that reference +the assembly stubs but never assembles the `.asm` files into +`.obj`s. Linker has nothing to resolve `jump_fcontext` against. + +We stopped here. **Each individual failure is patchable. Together +they say the project's CMake configuration has not been exercised +on a fresh Windows checkout in years**, which is consistent with +the 2-year-stale last commit. Combined with the grammar-level +blockers (no LIMIT, no label disjunction), the build issues are +the second-order signal — even if all three were patched, the +grammar still can't express IC2. + +### What the build outcome means for the verdict + +This isn't a "Windows is hard" complaint — every other system in +this bench (kuzu, graphqlite, gqlite itself) builds clean from a +fresh `git clone` on the same Windows + MSVC 2022 environment. +The build issues are specific to webbery/gqlite's pre-2024 CMake +configuration and unmaintained submodules. They strengthen the +discard verdict rather than weaken it: a system that can't be +built without manual patches in 2026 is, in practical terms, +unmaintained. + +## Verdict + +**Discard.** The system has structural blockers (no LIMIT, no +label-disjunction, per-row load idiom) that make the bench +comparison structurally impossible, not just slow. The original +plan's two-year-old "dead since April 2023" diagnosis was right +about the symptom and right about the prognosis. + +This file replaces the empty `webbery_gqlite/` slot in the +cross-system harness. No `setup.*` or `run.*` is provided; the +orchestrator's `run_all.sh` already detects missing runners and +logs `[SKIP]` to `skipped.log`. + +## What we got from the attempt + +- Concrete documentation of why this system doesn't fit, not just + "the project is old" — useful for the writeup's "Threats to + validity" / "Systems considered but rejected" subsection. +- A more complete picture of the candidate-system landscape: of the + five external systems we evaluated (graphqlite, GraphLite-AI, + auksys/gqlite, Kuzu, webbery/gqlite), only graphqlite and Kuzu + cleared the bar (and Kuzu only barely — see `kuzu/DIVERGENCES.md` + on its archival status). The "small embedded graph DB" niche has + multiple aspirational projects and no successful LDBC-scale + contender. +- Independent verification of the original bench plan's + prioritization. Time-boxing webbery low was correct.