ci(ci): cache baseline lib instead of baseline JSON - #15
Closed
abmcar wants to merge 4 commits into
Closed
Conversation
Change the Performance Regression Check job's cache from
`/tmp/perf_baseline_${mode}.json` (the benchmark output JSON, which encoded
*measurements* made on whatever runner first populated the cache for a given
base.sha) to `/tmp/baseline_lib` (the built baseline `libdtvmapi.so`, a
deterministic function of base.sha alone).
Effect: every PR perf-check job now re-runs baseline benchmarks on the same
ubuntu-latest runner that runs the PR benchmarks, eliminating cross-runner
cache-stale noise. The cache-miss path was already same-runner A-B
(`.ci/run_test_suite.sh:312-334` `elif [ -n "$BENCHMARK_BASELINE_LIB" ]`
branch); this change makes cache-hit follow the same dispatch by dropping
the BENCHMARK_BASELINE_CACHE export.
Empirical motivation (PR DTVMStack#493 commit 5c11550 interpreter perf-check, CI
cache hit on base.sha=c644fbe): CI reported +13-18% regressions on 7 synth
benchmarks; local same-machine A-B-A reported all deltas inside the
baseline self-drift band (max +0.6%). The PR's diff was 100% compiler-only,
no `src/evm/` files, so the CI deltas were entirely cross-runner noise.
Also fixes a separate bug in the baseline-build step: `cp build/lib/*`
dereferences the SONAME symlink chain (`libdtvmapi.so` -> `.0.1` -> `.0.1.0`)
producing 3 full copies (~105 MB) plus any libgtest*.a present, instead of
the single ~35 MB real file. Replaced with `cp "$(readlink -f
build/lib/libdtvmapi.so)" /tmp/baseline_lib/libdtvmapi.so`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
No benchmark summary available. |
|
✅ Performance Check Passed (interpreter) Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
⚡ Performance Regression Check Results✅ Performance Check Passed (interpreter)Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions ✅ Performance Check Passed (multipass)Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
|
✅ Performance Check Passed (multipass) Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
Add an in-source comment block in front of the perf-check Restore step so the rationale is discoverable from the workflow file directly. The bench JSON encodes runner-specific measurements; caching it across runners is what produced PR DTVMStack#493's spurious +15% interpreter "regressions". Caching the deterministic build output instead forces same-runner A-B comparison. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
✅ Performance Check Passed (interpreter) Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
|
✅ Performance Check Passed (multipass) Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
Promotes ~/changes/2026-05-15-perf-check-baseline-lib-cache/README.md into the project's docs/changes/ tree as part of opening the upstream PR. Empirical verification across two fork CI runs: - 25897744713 (cache miss, after re-running 3 spdlog-503 transient failures): both perf-check matrix jobs build baseline lib, then run baseline + PR bench on the same runner. 11/11 jobs green. RESULT: PASS for both modes. - 25900131271 (cache hit, same base.sha, no-op head bump): cache restored, Build-baseline-library step skipped, bench step still runs baseline + PR bench on the same runner via the dispatcher's elif branch. 11/11 jobs green on first try, no spdlog-503 retries. RESULT: PASS for both modes. Cross-runner noise reduction observed on synth/ADDRESS|BYTE|CALLER|EQ|GT|LT| ISZERO benches (PR DTVMStack#493 reported +13-18% all turned into <1% on same-runner). Only 1 micro-benchmark filtered by the baseline<5us gate versus 25 in the cross-runner cache-stale comparison. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
✅ Performance Check Passed (interpreter) Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
|
✅ Performance Check Passed (multipass) Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
Two wording fixes flagged by github-copilot[bot] review: - L16 "The current job..." -> "The pre-PR job (on main):" so the YAML snippet reads correctly after merge/archive. - L65 "Touched files: ... only" -> "Behavior-affecting files: ... only" + note that README.md itself ships in the same commit. No content change beyond wording. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Owner
Author
|
Superseded by upstream DTVMStack#506 (merged). Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork-internal test PR — not for upstream merge. Trigger the modified perf-check workflow in our own fork to verify Option A before opening upstream PR.
What
Switch
Performance Regression Checkjob to cache/tmp/baseline_lib/(builtlibdtvmapi.so) instead of/tmp/perf_baseline_${mode}.json(bench output JSON). Forces every PR to re-run baseline benchmarks on the same runner as the PR benchmarks.Why
PR DTVMStack#493 (cache-hit) reported +13-18% interpreter regressions on synth benchmarks that local same-machine A-B-A could not reproduce (≤0.6%). Root cause: cache JSON was generated on a different runner. Diff is compiler-only —
src/evm/untouched.Full spec:
~/changes/2026-05-15-perf-check-baseline-lib-cache/README.md(not yet promoted to upstreamdocs/changes/).Verification plan
max(|Δ|) < 3%elif [ -n "\$BENCHMARK_BASELINE_LIB" ]dispatch path in.ci/run_test_suite.sh:312-334)~/changes/.../intodocs/changes/, then open upstream PR