fix(compare): prefer newest per-generation dir; skip baseline gen output in warm-cache - #11
Merged
Merged
Conversation
…put in warm-cache Regression introduced by the warm-cache step in #10. `shutil.copytree` merged the baseline's *entire* per-behavior tree into the current run's `artifacts_root`, including the baseline's own per-generation output subdirectory `results/<suite>/<YYYYMMDDTHHMMSS>/scores.jsonl`. When `assert-ai run` then produced its own generation dir, `compare_runs.py::_find_scores_jsonl` saw two `scores.jsonl` matches under `current` and used `sorted()[0]` (oldest by ISO-8601 timestamp = the baseline's copy). Paired McNemar therefore compared the baseline against itself and returned zero discordant pairs -- verdict WARN via TooFewSamples on identical data. Observed on responsibleai/assert-ci-banking-demo run 31806201202 (PR #11 demo FAIL-path): current run scored coercion at 40% policy_violation vs baseline 45%, but the report said 45% == 45% and 0 discordant. Two coupled fixes: - `_find_scores_jsonl` (compare_runs.py) and `_find_file` (detect_test_set_drift.py, write_firstrun_report.py) now return `sorted()[-1]` -- the newest match. assert-ai timestamps per-generation subdirs so lexicographic order is chronological. Semantically correct in every case: when several runs coexist under the same suite root, the current run's data is always the latest. - `Warm the artifact cache from the baseline` (action.yml) skips subdirectories matching `^\d{8}T\d{6}$` when copying. The stage cache proper lives at `results/<suite>/artifacts/<stage>/<version>/` and is copied normally; only the baseline's per-generation output dirs (which the current run doesn't need for cache hydration) are excluded. Belt-and-suspenders: even without the `_find_file` fix, the current run's scores.jsonl is now the only one under its tree. Regression tests: - `test_find_scores_jsonl_prefers_latest_generation`: writes two generation dirs matching the exact banking-demo repro layout, asserts the picker returns the newer one. - `test_find_scores_jsonl_prefers_top_level_when_present`: documents that legacy flat layouts (no generation subdirs) keep working. All 54 tests pass locally.
tangym
added a commit
that referenced
this pull request
Aug 15, 2026
* fix(version): stop hardcoding v1.0.0-rc1 in write_firstrun_report write_firstrun_report.py had its own "ACTION_VERSION = v1.0.0-rc1" that never got bumped, so every FirstRun gate_report.json shipped with a version string that was never released. Same bug class that scripts/_version.py was introduced to fix -- but the guard test only covered compare_runs and detect_test_set_drift, so the drift here slipped through v1.0.0..v1.0.3. Route write_firstrun_report through scripts/_version like the other two writers, and extend tests/test_action_version.py to enforce the invariant on all three so it cannot regress. * chore(release): bump ACTION_VERSION to v1.0.4 Cutting v1.0.4 (patch). Since v1.0.3 all changes are fixes or additive input options -- no breaking changes: #11 fix(compare): prefer newest per-generation dir; skip baseline gen output in warm-cache #10 fix(cache): warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS #9 fix(baseline): anchor _find_run_dir on suite.json for nested layouts #8 feat(baseline): event-aware default for baseline-branch #7 feat(gate): add run-timeout-minutes input to cap eval-step wall clock #6 feat(install): add assert-ai-ref and assert-ai-repo inputs for git installs Plus the v1.0.0-rc1 hardcode fix in the previous commit. tests/test_action_version.py will fail against local tags until v1.0.4 is tagged from this commit; that is expected per docs/release-procedure.md (bump then tag).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Regression introduced by #10's warm-cache step.
shutil.copytreemerged the baseline's entire per-behavior tree into the current run'sartifacts_root— including the baseline's own per-generation output subdirectoryresults/<suite>/<YYYYMMDDTHHMMSS>/scores.jsonl. Whenassert-ai runthen wrote its own generation dir,compare_runs.py::_find_scores_jsonlsaw twoscores.jsonlmatches undercurrentand usedsorted()[0](oldest by ISO-8601 timestamp = the baseline's copy). Paired McNemar therefore compared the baseline against itself and returned zero discordant pairs — verdictWARNviaTooFewSampleson identical data.Observed
assert-ci-banking-demorun 31806201202 (PR #11 demo FAIL-path): current run scored coercion at 40% policy_violation vs baseline 45%, but the report said 45% == 45% and 0 discordant. Verified by pulling the rawscores.jsonlfrom both generation dirs and re-computing rates directly.Fix
Two coupled changes:
1.
_find_scores_jsonl/_find_fileprefer newestsorted()[0]→sorted()[-1]inscripts/compare_runs.py,scripts/detect_test_set_drift.py,scripts/write_firstrun_report.py.assert-aitimestamps per-generation subdirs (YYYYMMDDTHHMMSS) so lexicographic order is chronological. Semantically correct in every case: when multiple generations coexist under one suite root, the current run's data is always the latest.2. Warm-cache skips per-generation output dirs
New
ignorecallback onshutil.copytreeinaction.ymlexcludes subdirectories matching^\d{8}T\d{6}$. The stage cache proper lives atresults/<suite>/artifacts/<stage>/<version>/and is copied normally — only the baseline's per-generation output dirs (which the current run doesn't need for cache hydration) are excluded.Belt-and-suspenders: even without the
_find_filefix, the current run'sscores.jsonlis now the only one under its tree.Regression tests
test_find_scores_jsonl_prefers_latest_generation: writes two generation dirs matching the exact banking-demo repro layout, asserts the picker returns the newer one.test_find_scores_jsonl_prefers_top_level_when_present: documents that legacy flat layouts (no generation subdirs) keep working.All 54 tests pass locally.