Skip to content

Add a Git-baseline JSONSchex before/after benchmark suite #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xinz merged 2 commits into from
Sep 14, 2026
Merged

Add a Git-baseline JSONSchex before/after benchmark suite #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bd7ec0f
Add jsonschex self before and after comparsion
xinz Sep 14, 2026
de74cba
Update bench
xinz Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions bench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,40 @@ mix deps.get

### Library Comparison

Compare JSONSchex against other Elixir JSON Schema libraries:
Compare JSONSchex against other Elixir JSON Schema libraries. The suite contains 52 broad validation cases plus four user-facing scale cases derived from the focused optimization work:

```bash
# Run the full comparison suite in the bench directory
# Run the full 56-case comparison suite in the bench directory
mix run libs_comparison.exs

# Run a specific section only (e.g., ref, type, format, composition, etc.)
# Measure a specific section only (e.g., ref, format, or usage-scale workloads)
BENCH=ref mix run libs_comparison.exs
BENCH=format mix run libs_comparison.exs
BENCH=scale mix run libs_comparison.exs
```

Results are printed to stdout via [Benchee](https://github.com/bencheeorg/benchee), and some benchmark results from my local are [here](./results.txt).

### JSONSchex Version Comparison

Compare the current JSONSchex working tree with a complete source snapshot from a Git revision. The benchmark reports only `Before` and `After` JSONSchex jobs. It retains the original 52-case `libs_comparison.exs` snapshot plus 12 focused bundle, compile, scan, and validation cases for the optimizations from `9ab9748` through the current `main`; the four newer cross-library scale cases are intentionally tracked only by `libs_comparison.exs`:

```bash
mix run jsonschex_before_after.exs \
--baseline 0e3e2f40603fa5e57eff20ba8913861b10776572
```

Use `BENCH=inherited` for only the original 52 validation cases, `BENCH=optimization` for only the 12 focused cases, or another substring such as `array`, `p8`, or `scope_reuse` for a narrower selection. The default `BENCH=all` runs all 64 cases. `BENCH_ORDER` controls the version order inside each Benchee suite, allowing two opposite-order runs to expose and reduce ordering bias:

```bash
BENCH=optimization BENCH_ORDER=after_first \
mix run jsonschex_before_after.exs --baseline <revision>

BENCH=optimization BENCH_ORDER=before_first \
mix run jsonschex_before_after.exs --baseline <revision>
```

Set `VERSION_BENCH_OUTPUT` to write raw per-case statistics as TSV. The output includes each case's origin, operation, and optimization focus, together with revision/source fingerprints, dirty state, measurement durations, filter, and runtime versions needed to audit the run.


See [`jsonschex_before_after_overview.md`](./jsonschex_before_after_overview.md) for the current methodology and results.
Loading