Skip to content

fix(cache): warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS - #10

Merged
tangym merged 1 commit into
mainfrom
tangym/warm-artifact-cache
Aug 14, 2026
Merged

fix(cache): warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS#10
tangym merged 1 commit into
mainfrom
tangym/warm-artifact-cache

Conversation

@tangym

@tangym tangym commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

The paired regression gate was silently unreachable on any consumer whose behavior YAMLs don't set an explicit suite: field (both assert-ci-banking-demo and azure-doc-qa-eval are in this shape). Two independent misses had to line up:

  1. assert-ai defaults suite_id to eval-<current-timestamp> when the config leaves suite: unset (see assert_ai/config.py:219). Every fresh assert-ai run writes into its own results/eval-XXX/ and never sees the prior run's artifacts under results/eval-YYY/.
  2. The action downloaded the baseline artifact to a sibling directory (assert-ai-baseline/<slug>/) that assert-ai's cache lookup never inspects — the cache lives under each behavior's artifacts_root, which is assert-ai-artifacts/<slug>/.

Net effect: test_set regenerated on every dispatch (LLM call, non-deterministic), so detect_test_set_drift.py correctly observed a SHA mismatch and reported TestSetChanged. No PR could ever reach PASS.

Observed on assert-ci-banking-demo runs 31795690406 and 31796356987 after #9 landed: both used the fixed action, both found and downloaded the previous baseline, both regenerated the test_set from scratch, both reported TestSetChanged.

Fix

Two coupled changes:

scripts/plan_behaviors.py — pin suite in the frozen config

suite = cfg.get("suite") or slug
cfg["suite"] = suite

Every dispatch of the same behavior now writes into <artifacts_root>/results/<slug>/ deterministically. Consumer configs that already set suite: are preserved as-is.

action.yml — warm the artifact cache from the baseline

New composite step after Download baseline artifact:

- name: Warm the artifact cache from the baseline
  if: ${{ steps.resolve-baseline.outputs.needs-download == 'true' && steps.baseline-run.outputs.found == 'true' }}
  shell: bash
  run: |
    python <<'PY'
    ...
    shutil.copytree(baseline_slug_dir, artifacts_root, dirs_exist_ok=True)
    PY

assert-ai run then hydrates its context from the baseline's latest.json via activate_latest_artifacts, cache-hits on test_set (identical fingerprint = identical SHA), and skips regeneration.

Downstream stages behave correctly

  • No-code-change PR: all stages cache-hit → identical scores.jsonl → paired McNemar on identical data → PASS.
  • Real target-code PR: inference fingerprint changes because the target callable changed → inference cache misses → fresh inference and judging → paired McNemar compares baseline scores against the new agent's scores case-by-case → PASS/FAIL/WARN as the design intends.

One-time transition

Existing baseline artifacts uploaded under the old results/eval-<timestamp>/ layout are ignored by the new results/<slug>/ layout. The first dispatch of any consumer after upgrading will report FirstRun; the next dispatch establishes the new baseline and reaches PASS. Two-dispatch onboarding hiccup, not a steady-state cost.

Tests

Two regressions on test_plan_behaviors.py:

  • test_plan_pins_suite_to_slug_when_config_omits_it — behavior config without suite: produces frozen.suite == slug and manifest.suite == slug.
  • test_plan_preserves_user_supplied_suite — behavior config with explicit suite: keeps it verbatim.

52/52 tests pass locally.

What consumers need to do

Just bump the action pin to include this commit. No workflow change needed.

…ed McNemar can reach PASS

The paired regression gate was silently unreachable on any consumer
whose behavior YAMLs don't set an explicit `suite:` field
(responsibleai/assert-ci-banking-demo and responsibleai/azure-doc-qa-eval
are both in this shape). Two independent misses had to line up:

1. `assert-ai` defaults `suite_id` to `eval-<current-timestamp>`
   when the config leaves `suite:` unset, so every fresh
   `assert-ai run` writes into its own `results/eval-XXX/` and never
   sees the prior run's artifacts under `results/eval-YYY/`.

2. Even if suite_id had been stable, the action downloaded the baseline
   artifact to a sibling directory (`assert-ai-baseline/<slug>/`) that
   `assert-ai`'s cache lookup never inspects (`assert-ai-artifacts/<slug>/`).

The net effect: the test_set stage regenerated on every dispatch with
a non-deterministic LLM call, so `detect_test_set_drift.py` correctly
observed a SHA mismatch and reported `TestSetChanged`. Observed on
responsibleai/assert-ci-banking-demo runs 31795690406 and 31796356987
after PR #9 landed: both used the fixed action, both found and
downloaded the previous baseline, both regenerated the test_set from
scratch, both reported `TestSetChanged`. No PR could ever reach PASS.

Fix both misses together:

- `scripts/plan_behaviors.py`: pin the frozen config's `suite` to
  the behavior slug when the user hasn't set one. A user-supplied
  `suite:` is preserved as-is; only the empty default flips. Every
  dispatch of the same behavior now writes into
  `<artifacts_root>/results/<slug>/` deterministically.

- `action.yml`: after `Download baseline artifact`, copy
  `assert-ai-baseline/<slug>/*` into each behavior's
  `artifacts_root`. `assert-ai run` then hydrates its context from
  the baseline's `latest.json`, cache-hits on test_set (identical
  fingerprint = identical SHA), and skips regeneration.
  `shutil.copytree(..., dirs_exist_ok=True)` merges non-destructively.

Downstream stages behave correctly:

- No-code-change PR: all stages cache-hit -> identical scores.jsonl ->
  paired McNemar on identical data -> PASS.
- Real target-code PR: inference-stage fingerprint changes because the
  target callable changed -> inference cache miss -> fresh inference
  runs and judging -> paired McNemar compares baseline scores against
  the new agent's scores case-by-case -> PASS/FAIL/WARN as the design
  intends.

One-time transition: existing baseline artifacts uploaded under the
old `results/eval-<timestamp>/` layout are ignored by the new
suite-slug layout, so the first dispatch after upgrading a consumer
will report FirstRun. The next dispatch establishes and finds the new
baseline. Only affects the two-dispatch onboarding window, not steady
state.

Two regression tests for the plan_behaviors change:

- `test_plan_pins_suite_to_slug_when_config_omits_it`: writes a
  behavior config without `suite:` and asserts the frozen config and
  the manifest both carry `suite = slug`.
- `test_plan_preserves_user_supplied_suite`: writes a behavior config
  with an explicit `suite:` and asserts we do not overwrite it.

All 52 tests pass locally.
@tangym
tangym merged commit b872dbb into main Aug 14, 2026
7 checks passed
tangym added a commit that referenced this pull request Aug 14, 2026
…put in warm-cache (#11)

Regression introduced by the warm-cache step in #10. `shutil.copytree`
merged the baseline's *entire* per-behavior tree into the current run's
`artifacts_root`, including the baseline's own per-generation output
subdirectory
`results/<suite>/<YYYYMMDDTHHMMSS>/scores.jsonl`. When `assert-ai run`
then produced its own generation dir, `compare_runs.py::_find_scores_jsonl`
saw two `scores.jsonl` matches under `current` and used `sorted()[0]`
(oldest by ISO-8601 timestamp = the baseline's copy). Paired McNemar
therefore compared the baseline against itself and returned zero
discordant pairs -- verdict WARN via TooFewSamples on identical data.

Observed on responsibleai/assert-ci-banking-demo run 31806201202 (PR #11
demo FAIL-path): current run scored coercion at 40% policy_violation vs
baseline 45%, but the report said 45% == 45% and 0 discordant.

Two coupled fixes:

- `_find_scores_jsonl` (compare_runs.py) and `_find_file`
  (detect_test_set_drift.py, write_firstrun_report.py) now return
  `sorted()[-1]` -- the newest match. assert-ai timestamps
  per-generation subdirs so lexicographic order is chronological.
  Semantically correct in every case: when several runs coexist under
  the same suite root, the current run's data is always the latest.

- `Warm the artifact cache from the baseline` (action.yml) skips
  subdirectories matching `^\d{8}T\d{6}$` when copying. The stage
  cache proper lives at `results/<suite>/artifacts/<stage>/<version>/`
  and is copied normally; only the baseline's per-generation output
  dirs (which the current run doesn't need for cache hydration) are
  excluded. Belt-and-suspenders: even without the `_find_file` fix,
  the current run's scores.jsonl is now the only one under its tree.

Regression tests:

- `test_find_scores_jsonl_prefers_latest_generation`: writes two
  generation dirs matching the exact banking-demo repro layout,
  asserts the picker returns the newer one.
- `test_find_scores_jsonl_prefers_top_level_when_present`: documents
  that legacy flat layouts (no generation subdirs) keep working.

All 54 tests pass locally.
tangym added a commit that referenced this pull request Aug 15, 2026
* fix(version): stop hardcoding v1.0.0-rc1 in write_firstrun_report

write_firstrun_report.py had its own "ACTION_VERSION = v1.0.0-rc1" that
never got bumped, so every FirstRun gate_report.json shipped with a version
string that was never released. Same bug class that scripts/_version.py was
introduced to fix -- but the guard test only covered compare_runs and
detect_test_set_drift, so the drift here slipped through v1.0.0..v1.0.3.

Route write_firstrun_report through scripts/_version like the other two
writers, and extend tests/test_action_version.py to enforce the invariant
on all three so it cannot regress.

* chore(release): bump ACTION_VERSION to v1.0.4

Cutting v1.0.4 (patch). Since v1.0.3 all changes are fixes or additive input
options -- no breaking changes:

  #11 fix(compare): prefer newest per-generation dir; skip baseline gen output in warm-cache
  #10 fix(cache):   warm assert-ai's artifact cache from the baseline so paired McNemar can reach PASS
   #9 fix(baseline): anchor _find_run_dir on suite.json for nested layouts
   #8 feat(baseline): event-aware default for baseline-branch
   #7 feat(gate):     add run-timeout-minutes input to cap eval-step wall clock
   #6 feat(install):  add assert-ai-ref and assert-ai-repo inputs for git installs

Plus the v1.0.0-rc1 hardcode fix in the previous commit.

tests/test_action_version.py will fail against local tags until v1.0.4 is
tagged from this commit; that is expected per docs/release-procedure.md
(bump then tag).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant