fix(baseline): anchor _find_run_dir on suite.json so nested layouts pair correctly - #9
Merged
Merged
Conversation
Recent `assert-ai` versions nest a per-generation subdirectory under
each eval-<timestamp>/ run root:
eval-XXX/ <- run root: suite.json, test_set.jsonl
eval-XXX/YYY/scores.jsonl <- per-generation subdir
`_find_run_dir` anchored on `scores.jsonl`, which returned the inner
generation dir. `detect_test_set_drift.py` then called
`rglob('test_set.jsonl')` from that inner dir and never found the
outer file, so every PR reported FirstRun with
"baseline is missing test_set.jsonl"
even though the baseline artifact was correctly downloaded and both
behaviors ran. The paired gate was quietly disabled for every consumer
running against a modern assert-ai layout. Observed on responsibleai/
assert-ci-banking-demo runs 31789619791 and 31790967103: baseline
uploaded on the first workflow_dispatch, found on the second, but the
gate still showed FirstRun and the resolved baseline path pointed at
eval-XXX/YYY/ where test_set.jsonl doesn't live.
Anchor on `suite.json` first, which lives at the outer eval-XXX/
level and reliably exists. Fall back to `scores.jsonl` so existing
flat-layout fixtures (all the plan_behaviors / compare_runs tests)
keep passing without a fixture rewrite.
Add `test_resolve_returns_outer_run_dir_for_nested_layouts`, which
writes both files at their real levels and asserts `resolved.json`
points at the outer dir where test_set.jsonl actually is.
tangym
added a commit
that referenced
this pull request
Aug 14, 2026
…ed McNemar can reach PASS (#10) The paired regression gate was silently unreachable on any consumer whose behavior YAMLs don't set an explicit `suite:` field (responsibleai/assert-ci-banking-demo and responsibleai/azure-doc-qa-eval are both in this shape). Two independent misses had to line up: 1. `assert-ai` defaults `suite_id` to `eval-<current-timestamp>` when the config leaves `suite:` unset, so every fresh `assert-ai run` writes into its own `results/eval-XXX/` and never sees the prior run's artifacts under `results/eval-YYY/`. 2. Even if suite_id had been stable, the action downloaded the baseline artifact to a sibling directory (`assert-ai-baseline/<slug>/`) that `assert-ai`'s cache lookup never inspects (`assert-ai-artifacts/<slug>/`). The net effect: the test_set stage regenerated on every dispatch with a non-deterministic LLM call, so `detect_test_set_drift.py` correctly observed a SHA mismatch and reported `TestSetChanged`. Observed on responsibleai/assert-ci-banking-demo runs 31795690406 and 31796356987 after PR #9 landed: both used the fixed action, both found and downloaded the previous baseline, both regenerated the test_set from scratch, both reported `TestSetChanged`. No PR could ever reach PASS. Fix both misses together: - `scripts/plan_behaviors.py`: pin the frozen config's `suite` to the behavior slug when the user hasn't set one. A user-supplied `suite:` is preserved as-is; only the empty default flips. Every dispatch of the same behavior now writes into `<artifacts_root>/results/<slug>/` deterministically. - `action.yml`: after `Download baseline artifact`, copy `assert-ai-baseline/<slug>/*` into each behavior's `artifacts_root`. `assert-ai run` then hydrates its context from the baseline's `latest.json`, cache-hits on test_set (identical fingerprint = identical SHA), and skips regeneration. `shutil.copytree(..., dirs_exist_ok=True)` merges non-destructively. Downstream stages behave correctly: - No-code-change PR: all stages cache-hit -> identical scores.jsonl -> paired McNemar on identical data -> PASS. - Real target-code PR: inference-stage fingerprint changes because the target callable changed -> inference cache miss -> fresh inference runs and judging -> paired McNemar compares baseline scores against the new agent's scores case-by-case -> PASS/FAIL/WARN as the design intends. One-time transition: existing baseline artifacts uploaded under the old `results/eval-<timestamp>/` layout are ignored by the new suite-slug layout, so the first dispatch after upgrading a consumer will report FirstRun. The next dispatch establishes and finds the new baseline. Only affects the two-dispatch onboarding window, not steady state. Two regression tests for the plan_behaviors change: - `test_plan_pins_suite_to_slug_when_config_omits_it`: writes a behavior config without `suite:` and asserts the frozen config and the manifest both carry `suite = slug`. - `test_plan_preserves_user_supplied_suite`: writes a behavior config with an explicit `suite:` and asserts we do not overwrite it. All 52 tests pass locally.
tangym
added a commit
that referenced
this pull request
Aug 14, 2026
…ed McNemar can reach PASS The paired regression gate was silently unreachable on any consumer whose behavior YAMLs don't set an explicit `suite:` field (responsibleai/assert-ci-banking-demo and responsibleai/azure-doc-qa-eval are both in this shape). Two independent misses had to line up: 1. `assert-ai` defaults `suite_id` to `eval-<current-timestamp>` when the config leaves `suite:` unset, so every fresh `assert-ai run` writes into its own `results/eval-XXX/` and never sees the prior run's artifacts under `results/eval-YYY/`. 2. Even if suite_id had been stable, the action downloaded the baseline artifact to a sibling directory (`assert-ai-baseline/<slug>/`) that `assert-ai`'s cache lookup never inspects (`assert-ai-artifacts/<slug>/`). The net effect: the test_set stage regenerated on every dispatch with a non-deterministic LLM call, so `detect_test_set_drift.py` correctly observed a SHA mismatch and reported `TestSetChanged`. Observed on responsibleai/assert-ci-banking-demo runs 31795690406 and 31796356987 after PR #9 landed: both used the fixed action, both found and downloaded the previous baseline, both regenerated the test_set from scratch, both reported `TestSetChanged`. No PR could ever reach PASS. Fix both misses together: - `scripts/plan_behaviors.py`: pin the frozen config's `suite` to the behavior slug when the user hasn't set one. A user-supplied `suite:` is preserved as-is; only the empty default flips. Every dispatch of the same behavior now writes into `<artifacts_root>/results/<slug>/` deterministically. - `action.yml`: after `Download baseline artifact`, copy `assert-ai-baseline/<slug>/*` into each behavior's `artifacts_root`. `assert-ai run` then hydrates its context from the baseline's `latest.json`, cache-hits on test_set (identical fingerprint = identical SHA), and skips regeneration. `shutil.copytree(..., dirs_exist_ok=True)` merges non-destructively. Downstream stages behave correctly: - No-code-change PR: all stages cache-hit -> identical scores.jsonl -> paired McNemar on identical data -> PASS. - Real target-code PR: inference-stage fingerprint changes because the target callable changed -> inference cache miss -> fresh inference runs and judging -> paired McNemar compares baseline scores against the new agent's scores case-by-case -> PASS/FAIL/WARN as the design intends. One-time transition: existing baseline artifacts uploaded under the old `results/eval-<timestamp>/` layout are ignored by the new suite-slug layout, so the first dispatch after upgrading a consumer will report FirstRun. The next dispatch establishes and finds the new baseline. Only affects the two-dispatch onboarding window, not steady state. Two regression tests for the plan_behaviors change: - `test_plan_pins_suite_to_slug_when_config_omits_it`: writes a behavior config without `suite:` and asserts the frozen config and the manifest both carry `suite = slug`. - `test_plan_preserves_user_supplied_suite`: writes a behavior config with an explicit `suite:` and asserts we do not overwrite it. All 52 tests pass locally.
tangym
added a commit
that referenced
this pull request
Aug 15, 2026
* fix(version): stop hardcoding v1.0.0-rc1 in write_firstrun_report write_firstrun_report.py had its own "ACTION_VERSION = v1.0.0-rc1" that never got bumped, so every FirstRun gate_report.json shipped with a version string that was never released. Same bug class that scripts/_version.py was introduced to fix -- but the guard test only covered compare_runs and detect_test_set_drift, so the drift here slipped through v1.0.0..v1.0.3. Route write_firstrun_report through scripts/_version like the other two writers, and extend tests/test_action_version.py to enforce the invariant on all three so it cannot regress. * chore(release): bump ACTION_VERSION to v1.0.4 Cutting v1.0.4 (patch). Since v1.0.3 all changes are fixes or additive input options -- no breaking changes: #11 fix(compare): prefer newest per-generation dir; skip baseline gen output in warm-cache #10 fix(cache): warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS #9 fix(baseline): anchor _find_run_dir on suite.json for nested layouts #8 feat(baseline): event-aware default for baseline-branch #7 feat(gate): add run-timeout-minutes input to cap eval-step wall clock #6 feat(install): add assert-ai-ref and assert-ai-repo inputs for git installs Plus the v1.0.0-rc1 hardcode fix in the previous commit. tests/test_action_version.py will fail against local tags until v1.0.4 is tagged from this commit; that is expected per docs/release-procedure.md (bump then tag).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recent
assert-aioutput layouts nest a per-generation subdirectory under eacheval-<timestamp>/run root:plan_behaviors.py::_find_run_diranchored onscores.jsonl, so it returned the inner generation dir. Downstream,detect_test_set_drift.py::_find_filecalledrglob('test_set.jsonl')from that inner path and never found the outer file (rglobonly descends). Result: every gated PR reported FirstRun with the warningeven though the baseline artifact was correctly downloaded and both behaviors' evaluations completed. The paired gate has been silently disabled for every consumer running against a modern assert-ai layout.
How I found it
On
responsibleai/assert-ci-banking-demo:31789619791— dispatched on the PR branch, uploadedassert-ai-baseline(correct).31790967103— dispatched again on same branch, found the artifact viafind_baseline_run.py, downloaded it, ran both behaviors' evals. But verdict = FirstRun.The gate report from run 2:
{ "decision": "FirstRun", "n_paired_cases": 0, "warnings": ["baseline is missing test_set.jsonl"] }Digging into
assert-ai-resolved.json, the resolved baseline path was.../eval-20260814T095043/20260814T095043/— the inner generation dir, wheretest_set.jsonldoesn't live. It lives one level up at.../eval-20260814T095043/.Fix
Anchor
_find_run_dironsuite.json, which lives at the outereval-<timestamp>/level and reliably exists on every completed run. Fall back toscores.jsonlso existing tests (which use a flat legacy layout in their fixtures) keep passing without a fixture rewrite.Test
test_resolve_returns_outer_run_dir_for_nested_layoutsreproduces the exact nested layout observed in production and asserts the resolved paths point at the outer dir wheretest_set.jsonlactually is. Would have caught this bug on the first PR that shipped the layout change.All 50 existing tests still pass — the fallback preserves legacy-layout behavior for the flat-fixture tests.
Follow-ups (out of scope, worth tracking)
action_version: v1.0.3in gate reports is stale._version.py::ACTION_VERSIONneeds bumping every time we cut a real tag;tests/test_action_version.pyis supposed to guard this but currently passes atv1.0.3even though we've since merged severalv1.xfeatures.write_firstrun_report.py::_current_statsunder-counts multi-behavior suites. It reports the first behavior'sn_casesascurrent_n_cases, not the sum. Cosmetic on FirstRun (which shouldn't happen in the fixed flow anyway) but wrong.baseline_missingis non-empty despite the baseline artifact being present.