Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 41 additions & 11 deletions TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,22 @@ boxes are historical hypotheses, not current commitments. New work starts here,
is pre-registered in the experiment record, and is removed or checked off when a
decision is reached.

### P0: complete trustworthy canonical baselines
### P0: complete trustworthy canonical Linux baseline

- [x] Land the corrected Linux system-allocator matrix, shared iteration kernel,
explicit allocator provenance, and regenerated visuals in merge commit
`55ea7c3` (PR #4).
- [ ] After the four-element append decision lands, select that canonical `main`
commit and capture both a clean Ryzen 7950X3D Linux matrix and a clean M4 macOS
matrix at the same Git commit, Rust 1.97.0 compiler identity, schema, workload
matrix, and explicit `system` allocator policy. The current Linux report records
the pre-squash experiment commit and remains valid platform evidence, but is not
the final canonical pair anchor.
- [ ] Validate that Linux/macOS pair and extend the existing `LATEST.md` and SVGs
in one reporting change. Preserve each platform's absolute and relative values;
never pool platforms.
- [x] Capture and validate a clean Ryzen 7950X3D Linux matrix from canonical
`main` merge commit `2dc82e0`, using Rust 1.97.0 and the explicit `system`
allocator policy, then regenerate the Linux-only `LATEST.md` and SVGs.
- [ ] Deferred: capture the matching macOS matrix only when dedicated,
controllable Apple Silicon hardware is available. The local M4 host is not an
acceptable canonical source while interactive iTerm2/WindowServer load prevents
the runner from satisfying its idle gate. Never bypass that gate with
`--allow-host-noise`.
- [ ] Deferred with macOS: validate the Linux/macOS pair and extend `LATEST.md` and
the SVGs without pooling either platform's absolute or relative values. The
absence of a macOS result does not block publishing the canonical Linux result.
- [ ] Enable the already-configured CodSpeed ARM64 macro runner only after public
repository runner-group access is available; keep it a trend lane, not a
substitute for physical-host baselines.
Expand Down Expand Up @@ -137,7 +139,8 @@ header alignment without a new real-workload counterexample.
active roadmap; configuration is ready but organization runner-group access and
`CODSPEED_MACRO_ENABLED=true` remain external prerequisites.
- [x] Move the matching clean M4 macOS capture and pair validation into the active
roadmap; retain full raw rounds outside Git.
roadmap; subsequently defer it until dedicated hardware is available, retaining
full raw rounds outside Git and publishing no rejected/noisy local result.
- [x] Move practical-equivalence-band review into the active roadmap with its
evidence threshold unchanged.

Expand Down Expand Up @@ -176,6 +179,33 @@ header alignment without a new real-workload counterexample.

## Experiment record

### Canonical Linux comparison baseline (`bench/canonical-linux-baseline`)

- Status: complete on Linux; macOS deliberately deferred
- Source: canonical `main` merge commit
`2dc82e0656ba2c3997c4dc2dd26b905ce621bb2f`; Rust 1.97.0
(`2d8144b7880597b6e6d3dfd63a9a9efae3f533d3`); explicit system allocator.
- Linux quality: five balanced rotations pinned to CPU 0 of the Ryzen 7950X3D,
all twelve runtime audits at 100% idle, maximum audited one-minute load 1.13,
no busy processes or host issues, and the performance governor restored to
`schedutil` afterward. The runner cleared the builder's inherited tcmalloc
`LD_PRELOAD`; the recorded effective injection is null.
- Integrity: 110 CPU rows and 60 allocation rows; every allocation has zero
requested/usable bytes after drop and every reallocation count equals its moved
plus in-place counts. Reporting tests and deterministic regeneration pass.
- Result: JackVec has seven confidence-qualified wins, five equivalents, four
inconclusive results, and six losses versus Vec. Empty and sparse nested
requested heap are 0.333x and 0.526x Vec. This remains a situational density
design with targeted CPU wins, not an across-the-board faster Vec.
- Rejected macOS capture: the local M4 runner correctly refused to proceed when
iTerm2 and WindowServer each sustained roughly 28% CPU and two consecutive
>=90% idle samples could not be obtained. No local macOS result is authoritative
or published. Retry only on dedicated controllable hardware; never use
`--allow-host-noise` to manufacture a pair.
- Decision: publish the canonical Linux result and Linux-only graphics now.
Cross-platform validation and graphics remain deferred and do not block Linux
performance work; the next credible implementation audit is `retain<u64>`.

### Preallocated four-element append (`perf/append-small-audit`)

- Status: rejected; implementation and temporary diagnostic reverted
Expand Down
22 changes: 11 additions & 11 deletions comparisons/benchmark-results/LATEST.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,18 @@ implementation and scenario is retained, and platforms are never pooled.
## What this baseline says

- JackVec is not an across-the-board faster `Vec`: it has
8 confidence-qualified wins and
7 confidence-qualified wins and
6 losses versus `Vec` in this matrix.
- Its intended nested-density advantage is substantial: requested memory for the
empty and sparse nested workloads is
0.333× and
0.526× Vec,
respectively, while each collection owner remains one machine word.
- The optimized large append path reaches
1.020× Vec and
0.313× upstream
1.025× Vec and
0.311× upstream
ThinVec. This is a large targeted improvement, not a universal CPU claim.
- JackVec's three largest median CPU gaps versus Vec are `append_preallocated/4` (1.334×), `retain_mixed/u64` (1.245×), `retain_mixed/64_byte` (1.101×). They are
- JackVec's three largest median CPU gaps versus Vec are `append_preallocated/4` (1.323×), `retain_mixed/u64` (1.238×), `retain_mixed/64_byte` (1.105×). They are
retained here as investigation targets; confidence-aware classifications remain
authoritative over point-estimate ordering.
- Against the inline candidates, JackVec wins most measured CPU medians, while
Expand All @@ -38,18 +38,18 @@ silently counted as equality.

| Implementation | Wins | Equivalent | Inconclusive | Losses |
|---|---:|---:|---:|---:|
| JackVec | 8 | 5 | 3 | 6 |
| ThinVec | 8 | 6 | 1 | 7 |
| JackVec | 7 | 5 | 4 | 6 |
| ThinVec | 8 | 5 | 2 | 7 |
| SmallVec4 | 5 | 3 | 1 | 13 |
| SmallVec8 | 5 | 2 | 1 | 14 |
| SmallVec8 | 4 | 2 | 2 | 14 |

For direct context, this simpler head-to-head table compares median CPU times
using the same ±3% practical band. It does not replace the confidence-aware table.

| JackVec compared with | JackVec faster | Within ±3% | JackVec slower |
|---|---:|---:|---:|
| Vec | 9 | 6 | 7 |
| ThinVec | 8 | 10 | 4 |
| Vec | 9 | 7 | 6 |
| ThinVec | 7 | 13 | 2 |
| SmallVec4 | 13 | 4 | 5 |
| SmallVec8 | 13 | 4 | 5 |

Expand All @@ -73,14 +73,14 @@ and spill counts.

## Run provenance

- Commit: `2da08eea786c9755fa6ac883026ebfc8eeeb904e`
- Commit: `2dc82e0656ba2c3997c4dc2dd26b905ce621bb2f`
- Compiler: `1.97.0` (`2d8144b7880597b6e6d3dfd63a9a9efae3f533d3`)
- Allocator policy: `system`; inherited injection:
`{'DYLD_INSERT_LIBRARIES': None, 'LD_PRELOAD': '/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4'}`; effective injection:
`{'DYLD_INSERT_LIBRARIES': None, 'LD_PRELOAD': None}`
- CPU rounds: 5; CPU rows: 110; allocation rows: 60
- Minimum pinned-core idle audit: 100.0%
- Maximum audited one-minute load: 1.18
- Maximum audited one-minute load: 1.13

The performance profile reports the fraction of workloads within each factor of
the fastest implementation for that workload. It is an aggregate view, not a
Expand Down
Loading
Loading