fix(cache): warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS - #10
Merged
Merged
Conversation
…ed McNemar can reach PASS The paired regression gate was silently unreachable on any consumer whose behavior YAMLs don't set an explicit `suite:` field (responsibleai/assert-ci-banking-demo and responsibleai/azure-doc-qa-eval are both in this shape). Two independent misses had to line up: 1. `assert-ai` defaults `suite_id` to `eval-<current-timestamp>` when the config leaves `suite:` unset, so every fresh `assert-ai run` writes into its own `results/eval-XXX/` and never sees the prior run's artifacts under `results/eval-YYY/`. 2. Even if suite_id had been stable, the action downloaded the baseline artifact to a sibling directory (`assert-ai-baseline/<slug>/`) that `assert-ai`'s cache lookup never inspects (`assert-ai-artifacts/<slug>/`). The net effect: the test_set stage regenerated on every dispatch with a non-deterministic LLM call, so `detect_test_set_drift.py` correctly observed a SHA mismatch and reported `TestSetChanged`. Observed on responsibleai/assert-ci-banking-demo runs 31795690406 and 31796356987 after PR #9 landed: both used the fixed action, both found and downloaded the previous baseline, both regenerated the test_set from scratch, both reported `TestSetChanged`. No PR could ever reach PASS. Fix both misses together: - `scripts/plan_behaviors.py`: pin the frozen config's `suite` to the behavior slug when the user hasn't set one. A user-supplied `suite:` is preserved as-is; only the empty default flips. Every dispatch of the same behavior now writes into `<artifacts_root>/results/<slug>/` deterministically. - `action.yml`: after `Download baseline artifact`, copy `assert-ai-baseline/<slug>/*` into each behavior's `artifacts_root`. `assert-ai run` then hydrates its context from the baseline's `latest.json`, cache-hits on test_set (identical fingerprint = identical SHA), and skips regeneration. `shutil.copytree(..., dirs_exist_ok=True)` merges non-destructively. Downstream stages behave correctly: - No-code-change PR: all stages cache-hit -> identical scores.jsonl -> paired McNemar on identical data -> PASS. - Real target-code PR: inference-stage fingerprint changes because the target callable changed -> inference cache miss -> fresh inference runs and judging -> paired McNemar compares baseline scores against the new agent's scores case-by-case -> PASS/FAIL/WARN as the design intends. One-time transition: existing baseline artifacts uploaded under the old `results/eval-<timestamp>/` layout are ignored by the new suite-slug layout, so the first dispatch after upgrading a consumer will report FirstRun. The next dispatch establishes and finds the new baseline. Only affects the two-dispatch onboarding window, not steady state. Two regression tests for the plan_behaviors change: - `test_plan_pins_suite_to_slug_when_config_omits_it`: writes a behavior config without `suite:` and asserts the frozen config and the manifest both carry `suite = slug`. - `test_plan_preserves_user_supplied_suite`: writes a behavior config with an explicit `suite:` and asserts we do not overwrite it. All 52 tests pass locally.
tangym
added a commit
that referenced
this pull request
Aug 14, 2026
…put in warm-cache (#11) Regression introduced by the warm-cache step in #10. `shutil.copytree` merged the baseline's *entire* per-behavior tree into the current run's `artifacts_root`, including the baseline's own per-generation output subdirectory `results/<suite>/<YYYYMMDDTHHMMSS>/scores.jsonl`. When `assert-ai run` then produced its own generation dir, `compare_runs.py::_find_scores_jsonl` saw two `scores.jsonl` matches under `current` and used `sorted()[0]` (oldest by ISO-8601 timestamp = the baseline's copy). Paired McNemar therefore compared the baseline against itself and returned zero discordant pairs -- verdict WARN via TooFewSamples on identical data. Observed on responsibleai/assert-ci-banking-demo run 31806201202 (PR #11 demo FAIL-path): current run scored coercion at 40% policy_violation vs baseline 45%, but the report said 45% == 45% and 0 discordant. Two coupled fixes: - `_find_scores_jsonl` (compare_runs.py) and `_find_file` (detect_test_set_drift.py, write_firstrun_report.py) now return `sorted()[-1]` -- the newest match. assert-ai timestamps per-generation subdirs so lexicographic order is chronological. Semantically correct in every case: when several runs coexist under the same suite root, the current run's data is always the latest. - `Warm the artifact cache from the baseline` (action.yml) skips subdirectories matching `^\d{8}T\d{6}$` when copying. The stage cache proper lives at `results/<suite>/artifacts/<stage>/<version>/` and is copied normally; only the baseline's per-generation output dirs (which the current run doesn't need for cache hydration) are excluded. Belt-and-suspenders: even without the `_find_file` fix, the current run's scores.jsonl is now the only one under its tree. Regression tests: - `test_find_scores_jsonl_prefers_latest_generation`: writes two generation dirs matching the exact banking-demo repro layout, asserts the picker returns the newer one. - `test_find_scores_jsonl_prefers_top_level_when_present`: documents that legacy flat layouts (no generation subdirs) keep working. All 54 tests pass locally.
tangym
added a commit
that referenced
this pull request
Aug 15, 2026
* fix(version): stop hardcoding v1.0.0-rc1 in write_firstrun_report write_firstrun_report.py had its own "ACTION_VERSION = v1.0.0-rc1" that never got bumped, so every FirstRun gate_report.json shipped with a version string that was never released. Same bug class that scripts/_version.py was introduced to fix -- but the guard test only covered compare_runs and detect_test_set_drift, so the drift here slipped through v1.0.0..v1.0.3. Route write_firstrun_report through scripts/_version like the other two writers, and extend tests/test_action_version.py to enforce the invariant on all three so it cannot regress. * chore(release): bump ACTION_VERSION to v1.0.4 Cutting v1.0.4 (patch). Since v1.0.3 all changes are fixes or additive input options -- no breaking changes: #11 fix(compare): prefer newest per-generation dir; skip baseline gen output in warm-cache #10 fix(cache): warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS #9 fix(baseline): anchor _find_run_dir on suite.json for nested layouts #8 feat(baseline): event-aware default for baseline-branch #7 feat(gate): add run-timeout-minutes input to cap eval-step wall clock #6 feat(install): add assert-ai-ref and assert-ai-repo inputs for git installs Plus the v1.0.0-rc1 hardcode fix in the previous commit. tests/test_action_version.py will fail against local tags until v1.0.4 is tagged from this commit; that is expected per docs/release-procedure.md (bump then tag).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The paired regression gate was silently unreachable on any consumer whose behavior YAMLs don't set an explicit
suite:field (bothassert-ci-banking-demoandazure-doc-qa-evalare in this shape). Two independent misses had to line up:assert-aidefaultssuite_idtoeval-<current-timestamp>when the config leavessuite:unset (seeassert_ai/config.py:219). Every freshassert-ai runwrites into its ownresults/eval-XXX/and never sees the prior run's artifacts underresults/eval-YYY/.assert-ai-baseline/<slug>/) thatassert-ai's cache lookup never inspects — the cache lives under each behavior'sartifacts_root, which isassert-ai-artifacts/<slug>/.Net effect:
test_setregenerated on every dispatch (LLM call, non-deterministic), sodetect_test_set_drift.pycorrectly observed a SHA mismatch and reportedTestSetChanged. No PR could ever reachPASS.Observed on
assert-ci-banking-demoruns31795690406and31796356987after #9 landed: both used the fixed action, both found and downloaded the previous baseline, both regenerated the test_set from scratch, both reportedTestSetChanged.Fix
Two coupled changes:
scripts/plan_behaviors.py— pinsuitein the frozen configEvery dispatch of the same behavior now writes into
<artifacts_root>/results/<slug>/deterministically. Consumer configs that already setsuite:are preserved as-is.action.yml— warm the artifact cache from the baselineNew composite step after
Download baseline artifact:assert-ai runthen hydrates its context from the baseline'slatest.jsonviaactivate_latest_artifacts, cache-hits ontest_set(identical fingerprint = identical SHA), and skips regeneration.Downstream stages behave correctly
scores.jsonl→ paired McNemar on identical data → PASS.inferencefingerprint changes because the target callable changed → inference cache misses → fresh inference and judging → paired McNemar compares baseline scores against the new agent's scores case-by-case → PASS/FAIL/WARN as the design intends.One-time transition
Existing baseline artifacts uploaded under the old
results/eval-<timestamp>/layout are ignored by the newresults/<slug>/layout. The first dispatch of any consumer after upgrading will reportFirstRun; the next dispatch establishes the new baseline and reachesPASS. Two-dispatch onboarding hiccup, not a steady-state cost.Tests
Two regressions on
test_plan_behaviors.py:test_plan_pins_suite_to_slug_when_config_omits_it— behavior config withoutsuite:producesfrozen.suite == slugandmanifest.suite == slug.test_plan_preserves_user_supplied_suite— behavior config with explicitsuite:keeps it verbatim.52/52 tests pass locally.
What consumers need to do
Just bump the action pin to include this commit. No workflow change needed.