From 3b48639f181361eb1ccddf2fc80214324b2ee6c8 Mon Sep 17 00:00:00 2001 From: kridaydave Date: Wed, 2 Sep 2026 23:25:07 +0530 Subject: [PATCH 1/2] docs: add v5.0.0 vs v3.5.0 benchmark results Preview/organize 12-35x faster in v5 (content-analysis stack removed from the handler path), duplicate finding 2.2x faster, organizer no longer fails on large directories (v3.5 exhausted its EEXIST retry loop on mixed-size batches), path-validation correctness and cost unchanged. Undo is slower by design: v5 verifies rollback-manifest HMAC integrity before restoring. Results, method, and root causes in docs/benchmarks-v5-vs-v3.5.md; linked from the new Performance section in ARCHITECTURE.md. --- ARCHITECTURE.md | 4 ++ docs/benchmarks-v5-vs-v3.5.md | 132 ++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+) create mode 100644 docs/benchmarks-v5-vs-v3.5.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 9420b98..f7b9ed2 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -118,3 +118,7 @@ If a tool adds a way in, add the way out and the way to see it: organize ships w - `scripts/security-gates/` — path-traversal fuzzing, TOCTOU races, sensitive-file patterns, static analysis Tests derive all paths from `os.tmpdir()` or `tests/sandbox/`. A test that needs `sleep()` to pass is wrong; wait on receipts instead. + +## Performance + +The lean-v5 shape was benchmarked against v3.5.0 — preview/organize 12–35× faster with the content-analysis stack removed, organizer no longer fails on large directories, path-validation cost unchanged. Numbers and method: `docs/benchmarks-v5-vs-v3.5.md`. diff --git a/docs/benchmarks-v5-vs-v3.5.md b/docs/benchmarks-v5-vs-v3.5.md new file mode 100644 index 0000000..a1c60ec --- /dev/null +++ b/docs/benchmarks-v5-vs-v3.5.md @@ -0,0 +1,132 @@ +# Benchmarks: v5.0.0 vs v3.5.0 + +Measured comparison of the lean v5 rewrite against v3.5.0 (the last release +before it). Run on 2026-09-02, single Linux machine, both versions built and +exercised at the **tool-handler level** — the same code an MCP client hits. + +- **v3.5.0** = commit `faa0f99` (merge of `release/v3.5.0`) +- **v5.0.0** = commit `3fb0e4e` (main) + +Identical seeded datasets per scenario. Everything ran sandboxed: +`NODE_ENV=test` with each repo root as cwd (the allowed root in both +versions) and `XDG_CONFIG_HOME` pointed at a temp dir, so no real user +config, history, or rollback state was touched. Datasets lived in the +gitignored `tests/sandbox/bench-data/` and were removed afterwards. + +## Results at a glance + +| Area | v3.5.0 | v5.0.0 | +|---|---|---| +| Preview 400 files | 5,936 ms | **170 ms (35× faster)** | +| Organize 400 files | 5,481 ms | **448 ms (12× faster)** | +| Find duplicates, 3,000 files | 2,456 ms | **1,115 ms (2.2× faster)** | +| Organize 3,000 files (stress) | 35,554 ms, 2,800 failures | **2,225 ms, 0 failures** | +| Scan (happy 400 / stress 9,500) | 26 / 607 ms | 37 / 645 ms (parity) | +| Path-validation correctness, 7 attack cases | 7/7 correct | 7/7 correct | +| Path-validation throughput | ~2.4k ops/s | ~2.2k ops/s (parity) | +| `npm run test:security` | 22 tests pass | 78 tests pass | +| Peak RSS (happy path) | 390 MB | **203 MB** | + +The one number v3.5 "wins" is undo (147 ms vs 515 ms for 300 restores). +That is a deliberate v5 trade: v5 verifies rollback-manifest HMAC integrity +before restoring (`src/core/organize/rollback.ts`); v3.5's undo does not. + +## Happy path — 400 files (300 flat + 20 subdirs), median of 5 iterations + +| Operation | v3.5.0 | v5.0.0 | +|---|---|---| +| scan (json / markdown response) | 26 / 23 ms | 37 / 34 ms | +| categorize_by_type | 24 ms | 34 ms | +| preview_organization | 5,936 ms | 170 ms | +| organize_files | 5,481 ms | 448 ms | +| undo_last_operation | 147 ms | 515 ms | + +### Why v3.5 preview/organize were slow + +The core pipeline is not the difference. Calling v3.5's services directly +(scan → plan → move on the same dataset) took ~930 ms. The cost sat in the +handler layer: + +- v3.5's `organize_files`/`preview_organization` handlers route through + `globalOrganizerService`, wired to `ContentAnalyzerService` + + `MetadataCacheService`, doing per-file content analysis on every run. +- Per-move rollback-manifest HMAC signing: CPU profiling showed + `createManifest` + `computeSignature` + `computeHash` at ~26% of + application CPU time. + +v5 deleted the content-analysis/metadata stack (handlers categorize by +extension and sniff only where needed) and signs the rollback manifest once +per operation instead of per move. + +## Stress — 9,500-file tree + 3,000-file flat set (600 duplicate-content copies) + +| Operation | v3.5.0 | v5.0.0 | +|---|---|---| +| Full recursive scan (10 paginated requests, limit 1000) | 607 ms | 645 ms | +| find_duplicate_files (3,000 flat files) | 2,456 ms | 1,115 ms | +| Duplicate groups found | 600 (1,200 files) | 600 (1,200 files) | +| organize_files (3,000 files) | 35,554 ms — moved 200, **2,800 errors** | 2,225 ms — moved 3,000, 0 errors | +| undo_last_operation | 241 ms (restores 200) | 2,564 ms (restores 3,000) | + +Per restored file, undo cost is comparable (~1.2 ms/file in v3.5, ~0.9 ms/file +in v5) — v5 simply had 15× more files to restore. + +### v3.5 organize failures on large directories + +v3.5's organize reproduced the failure in isolation, independent of +duplicates or the find-duplicates-first sequence: + +- 3,000 files, mixed sizes: moved 200, 2,800 errors +- 2,400 unique-content files, mixed sizes: moved 100, 2,300 errors +- 3,000 tiny 4 KB files: moved 3,000, 0 errors + +The failing moves die with +`Failed to move ... after 100 retries due to race conditions`: v3.5 moves +with `COPYFILE_EXCL` and retries on `EEXIST` with `_1`/`_2` suffixes. The +trigger is concurrency-dependent — larger files widen the copy window. v5's +rewritten organizer does not have this code path and moved 3,000/3,000. + +## Security + +**Correctness is identical.** 2,000 calls per case through +`validateStrictPath`: + +| Case | v3.5.0 | v5.0.0 | +|---|---|---| +| Benign existing file | accepted (correct) | accepted (correct) | +| Traversal escaping the allowed root | rejected | rejected | +| Null byte | rejected | rejected | +| Absolute path outside allowed roots | rejected | rejected | +| Symlink to file outside sandbox | rejected | rejected | +| Symlinked directory traversal | rejected | rejected | +| 30-level deep traversal | rejected | rejected | + +**Throughput is parity.** Benign validation costs ~430–465 µs/call in both +versions (~2.2–2.4k ops/s through the full 8-layer gate). Hostile paths fail +fast in both (25–98 µs, 10–39k ops/s); v5 is marginally faster on symlink +cases, v3.5 marginally faster on the trivial reject. Nothing about the v5 +rewrite regressed the security gate's cost. + +As a gate, `npm run test:security` passes on both: v3.5 runs 22 tests in ~4 s; +v5 runs 78 in ~10 s (56 extra adversarial/regression tests added in v5). + +## Caveats + +- Single machine (Linux, tmpfs /tmp, NVMe-class disk); absolute numbers vary, + ratios held across repeated runs. +- Scan numbers include the pagination protocol both versions share + (limit caps at 1,000 files/request); clients scanning 10k+ file trees + should expect ~10 requests per full enumeration on either version. +- Happy-path organize for v5 varied between ~190–450 ms across runs; + medians reported. The v3.5 gap was stable in every run. + +## Reproducing + +Both versions must be built side by side, run with `NODE_ENV=test`, the +version's repo root as cwd, and `XDG_CONFIG_HOME` redirected to a temp dir. +Datasets go under the repo's `tests/sandbox/`. The harness called the same +handler entry points in both versions: +`handleScanDirectory`, `handleCategorizeByType`, `handlePreviewOrganization`, +`handleOrganizeFiles`, `handleUndoLastOperation`, `handleFindDuplicateFiles` +(from `dist/src/tools/`), and `validateStrictPath` (from +`dist/src/services/path-validator.service.js`). From b2b55316daae4198a61c180c0f87d8a1b44799dd Mon Sep 17 00:00:00 2001 From: kridaydave Date: Wed, 2 Sep 2026 23:37:01 +0530 Subject: [PATCH 2/2] docs: address CodeRabbit review on benchmark doc - state that path-validation was measured via validateStrictPath directly, not through the MCP handler layer (operations used handler entry points) - correct the v5 move-path description: COPYFILE_EXCL/EEXIST semantics are kept in atomic-move.ts; v5 dropped the retry-with-suffix loop - scope correctness/robustness conclusions to the tested cases - document dataset seeds, sizes, and iteration counts for reproduction --- ARCHITECTURE.md | 2 +- docs/benchmarks-v5-vs-v3.5.md | 40 +++++++++++++++++++++++++++-------- 2 files changed, 32 insertions(+), 10 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f7b9ed2..bc6b4da 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -121,4 +121,4 @@ Tests derive all paths from `os.tmpdir()` or `tests/sandbox/`. A test that needs ## Performance -The lean-v5 shape was benchmarked against v3.5.0 — preview/organize 12–35× faster with the content-analysis stack removed, organizer no longer fails on large directories, path-validation cost unchanged. Numbers and method: `docs/benchmarks-v5-vs-v3.5.md`. +The lean-v5 shape was benchmarked against v3.5.0 — preview/organize 12–35× faster with the content-analysis stack removed, the tested large-directory organize scenarios completing without failures where v3.5 failed them, path-validation cost unchanged. Numbers and method: `docs/benchmarks-v5-vs-v3.5.md`. diff --git a/docs/benchmarks-v5-vs-v3.5.md b/docs/benchmarks-v5-vs-v3.5.md index a1c60ec..f8309fc 100644 --- a/docs/benchmarks-v5-vs-v3.5.md +++ b/docs/benchmarks-v5-vs-v3.5.md @@ -3,6 +3,9 @@ Measured comparison of the lean v5 rewrite against v3.5.0 (the last release before it). Run on 2026-09-02, single Linux machine, both versions built and exercised at the **tool-handler level** — the same code an MCP client hits. +One exception: the path-validation micro-benchmark called `validateStrictPath` +directly, below the Zod parsing and response formatting that MCP handlers add +on top. - **v3.5.0** = commit `faa0f99` (merge of `release/v3.5.0`) - **v5.0.0** = commit `3fb0e4e` (main) @@ -83,12 +86,16 @@ duplicates or the find-duplicates-first sequence: The failing moves die with `Failed to move ... after 100 retries due to race conditions`: v3.5 moves with `COPYFILE_EXCL` and retries on `EEXIST` with `_1`/`_2` suffixes. The -trigger is concurrency-dependent — larger files widen the copy window. v5's -rewritten organizer does not have this code path and moved 3,000/3,000. +trigger is concurrency-dependent — larger files widen the copy window. +v5 keeps the same `COPYFILE_EXCL` non-destructive semantics +(`src/core/io/atomic-move.ts` still maps destination collisions to `EEXIST`) +but drops the retry-with-suffix loop: a colliding move fails fast with one +structured error instead of being retried 100 times. In the tested +scenarios v5 moved 3,000/3,000 with 0 errors. ## Security -**Correctness is identical.** 2,000 calls per case through +**Correctness matched for the seven tested cases.** 2,000 calls per case through `validateStrictPath`: | Case | v3.5.0 | v5.0.0 | @@ -124,9 +131,24 @@ v5 runs 78 in ~10 s (56 extra adversarial/regression tests added in v5). Both versions must be built side by side, run with `NODE_ENV=test`, the version's repo root as cwd, and `XDG_CONFIG_HOME` redirected to a temp dir. -Datasets go under the repo's `tests/sandbox/`. The harness called the same -handler entry points in both versions: -`handleScanDirectory`, `handleCategorizeByType`, `handlePreviewOrganization`, -`handleOrganizeFiles`, `handleUndoLastOperation`, `handleFindDuplicateFiles` -(from `dist/src/tools/`), and `validateStrictPath` (from -`dist/src/services/path-validator.service.js`). +Datasets go under the repo's `tests/sandbox/`. + +Datasets came from a seeded PRNG (mulberry32), deterministic across +versions: + +- **Happy path** — seed 42: 400 files (300 flat + 20 subdirs × 5 files) + across 16 extensions, 1 KB–200 KB +- **Stress tree** — seed 1337: 10×10×3 directories × 30 files + (9,500 files, 30 KB–1 MB, plus 500 exact-content duplicate copies) +- **Stress flat set** — seed 777: 3,000 files, 4 KB–512 KB, of which 600 + are exact-content copies of 500 originals +- Happy-path numbers are medians of 5 iterations; stress and security + numbers are single runs. Security cases ran 2,000 `validateStrictPath` + calls each. + +The harness was a scratch script calling the same entry points in both +versions — `handleScanDirectory`, `handleCategorizeByType`, +`handlePreviewOrganization`, `handleOrganizeFiles`, +`handleUndoLastOperation`, `handleFindDuplicateFiles` (from +`dist/src/tools/`), and `validateStrictPath` (from +`dist/src/services/path-validator.service.js`) — and was not committed.