Skip to content

docs(perf): compare main engines and set next optimization targets - #21

Merged
huacnlee merged 1 commit into
mainfrom
docs/main-engine-performance-baseline
Sep 6, 2026
Merged

docs(perf): compare main engines and set next optimization targets#21
huacnlee merged 1 commit into
mainfrom
docs/main-engine-performance-baseline

Conversation

@huacnlee

@huacnlee huacnlee commented Sep 6, 2026

Copy link
Copy Markdown
Member

Issue # (if available)

N/A

Description of changes

Archive a fresh comparison of repository-pinned QuickJS-ng, quickjs-jit automatic mode, and Bun 1.4.0 default configuration after #19 and #20 merged into clean main 47aeb11. Include forced Tier 1/2 diagnostics, raw samples, methodology/provenance, CSV, confidence intervals, charts, and an evidence-based next-stage optimization roadmap. No runtime behavior changes.

Automatic JIT is faster than the interpreter in 8 workloads, statistically tied in 1, and slower in 13, based on per-workload paired 95% intervals. Selected results below express speed relative to the QuickJS-ng interpreter (1x = parity):

Workload JIT automatic speed [95% CI] Bun default speed
scalar-loop 24.024x [23.382, 24.316] 47.26x
float64-dense 7.819x [7.708, 7.895] 6.27x
property-heavy 0.268x [0.267, 0.271] 3.87x
generic-call-entry 0.135x [0.134, 0.136] 57.85x
arrays-typed 0.656x [0.654, 0.658] 10.41x

These are current-protocol results, not equally warmed peak-engine measurements: QuickJS and Bun differ in process-internal warmup and host timing boundaries. Native counters include warmup. Bun uses a recorded launcher that removes the runner's --smol flag. The unmerged property candidate 9ccda01 is excluded.

The roadmap prioritizes aligned measurement and steady-state counters, property and generic-call costs, automatic tiering/fallback overhead, then cold start and library hotspots. Property and generic-call stages propose a paired speed lower bound of 1.5x versus the same-harness main baseline, with eventual interpreter parity or better.

Evidence: full comparison, raw samples, methodology, next targets.

Validation:

  • Completed 22 workloads × 5 modes × 30 latency samples, with 5 discarded warmup processes and 10 one-second throughput windows per workload/mode.
  • Audited all 3,300 latency samples, checksums, entry/exit counters, source/binary hashes, derived results, links, and chart rendering; independent report review found no substantive issue.
  • Staged whitespace check passes; CSV/SVG formatting normalization preserves data/path tokens. No runtime tests rerun for this documentation/data-only commit.
  • Performance acceptance remains failing (runner exit 2 after complete output): compute speed lower bound is below 5x, automatic decision evidence is missing, and startup/definition-eval proxy budgets fail. The gpui-shell gate is inconclusive because external host evidence is absent. Raw gate output is preserved.

Checklist

  • Changelog assessed: not applicable to benchmark evidence and planning documents.
  • Unit tests assessed: no runtime changes; artifact and data validation completed.

@huacnlee
huacnlee merged commit 41916b7 into main Sep 6, 2026
42 checks passed
@huacnlee
huacnlee deleted the docs/main-engine-performance-baseline branch September 6, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant