feat(tests): take the eval model from env vars, bump coder_eval to 0.9.2

[bai-uipath]

What

Stop pinning the agent model in the experiment YAMLs, and take it from a repo variable at every entry point that runs an eval. No model id is hardcoded in any of them. Also bumps the coder_eval pin to 0.9.2.

Companion to UiPath/coder_eval_uipath#74, which gives the nightly runner one model variable per harness.

The model pins

Six experiment files pinned model: claude-sonnet-5: default, nightly, smoke, smoke-windows, activation, skill-comparison-template. All six are removed.

In the nightly and in run-coder-eval.yml those lines were already dead. --model is the last layer of coder_eval's config merge and outranks the experiment, so the pin never applied. They were also wrong in principle: the same experiment file runs under codex and antigravity, which cannot use a Claude model.

They are removed rather than templated because coder_eval expands environment variables only in path fields, so a ${VAR} in agent.model would be passed through as a literal model name.

The gates that had no --model

Three entry points relied on the pin, so removing it would have silently repointed them at $BEDROCK_MODEL. Each now passes a model explicitly, resolved from vars.CLAUDE_CODE_MODEL:

• smoke-skills.yml
• smoke-rpa-skills.yml
• activation-gate.yml, via activation_gate.py

There is no literal fallback behind the variable. An unset AGENT_MODEL fails the step, guarded before coder-eval is invoked, and activation_gate.py exits 2. Gating on a model nobody configured produces a verdict about the wrong thing, and for the activation gate specifically the recall baselines are model-specific, so a silent substitution makes the number meaningless rather than merely wrong. Passing the model at the call site keeps that visible where the baselines live.

$BEDROCK_MODEL is deliberately not used for this. It is the evaluation-side model, feeding the llm_judge grader and the simulated user, and reusing it for the agent would move the grader whenever the agent under test changes.

run-coder-eval.yml picks up CLAUDE_CODE_MODEL / CODEX_MODEL / ANTIGRAVITY_MODEL the same way, replacing three hardcoded model names per job, each guarded. Its agent_model input still overrides all of them for a one-off run. Antigravity's model moves to gemini-3.6-flash.

Required repo variables

The three gates and the dispatch workflow read repo variables, which are now set on this repo:

CLAUDE_CODE_MODEL=claude-sonnet-5
CODEX_MODEL=gpt-5.6-terra
ANTIGRAVITY_MODEL=gemini-3.6-flash

These are the exact values the deleted literals carried, so CI behavior is unchanged. vars, not secrets: nothing here is sensitive, and a secret cannot be read back for an audit.

coder_eval 0.9.2

Three upstream changes since 0.9.1:

• fix(reports): errors count as misses, and cost is no longer lost on error paths. This lowers reported pass rates, by up to ~10 points on a run with a normal error share, with no underlying regression. Compare against error_share before reading a drop as a real change.
• feat(evalboard): harness comparison on the overview, and per-run identity.
• feat(litellm): per-call cost and cache accounting for the open-weight backend.

Kept in this PR rather than split out. It is independent of the model change, so read the next nightly's drop against error_share before attributing any of it here.

The gate went green on an empty run

The model guard started as the first line of the Run smoke tests step. That step carries continue-on-error: true, and both threshold steps below it read zero tasks as a pass:

• Enforce LLM reviewer score threshold: if not run_dirs: sys.exit(0)
• Enforce pass-rate threshold (>= 95%): if n == 0: print('No smoke-tagged tasks matched — skipping rate check'); sys.exit(0)

So when run 30672478895 resolved AGENT_MODEL empty (queued 90 seconds before the repo variables were created), the guard fired correctly, the step exited in under a second having run nothing, and the job still reported success:

AGENT_MODEL:
...sh: line 1: AGENT_MODEL: unset — set the CLAUDE_CODE_MODEL repo variable
step "Run smoke tests"   23:20:02Z → 23:20:02Z   (0s)
job conclusion: success

The RPA gate does not have this hole (if not task_jsons: print('::error::No task.json files found — nothing to gate'); sys.exit(1)), which is why RPA went red off the same empty variable while Linux went green. That asymmetry was the bug, not the guard.

Two changes:

• The guard moves into its own Require an agent model step ahead of the smoke step in both workflows, so an unresolved model fails before any task runs, in a step that names what is missing.
• The Linux pass-rate gate fails closed when zero tasks land and the smoke step's outcome is failure. A legitimately empty selection still passes, since that path leaves the smoke step successful, and flake tolerance is untouched: once any task runs, the rate check governs exactly as before.

Validation

Live runs on coder_eval 0.9.2, from this branch, one task per harness (uipath-admin/audit_events_basic_smoke, tempdir driver, experiments/default.yaml with its pin removed):

run	agent model	evaluation model	result	cost
claude-code, --model given	eu.anthropic.claude-sonnet-5	eu.anthropic.claude-sonnet-4-6	SUCCESS 1.0	$0.375
claude-code, no --model	claude-sonnet-4-6	eu.anthropic.claude-sonnet-4-6	SUCCESS 1.0	$0.503
codex	gpt-5.6-terra	eu.anthropic.claude-sonnet-4-6	SUCCESS 1.0	$0.195
antigravity	gemini-3.6-flash	eu.anthropic.claude-sonnet-4-6	SUCCESS 1.0	$0.182

All four load the experiment files with no model: key and pass. The runs record framework_version: 0.9.2, confirming the pin resolves.

Row 1 versus row 2 is the point of the change. With --model, the agent under test and the evaluation route are two different models in the same run. Without it, they collapse onto $BEDROCK_MODEL.

Known consequence for bare local runs. Row 2 is a real change: coder-eval run with no --model used to get claude-sonnet-5 from the experiment file and now gets $BEDROCK_MODEL from .env. Intended, since that is the environment-variable mechanism, but anyone with an unusual local BEDROCK_MODEL will see their model move.

Linux smoke gate. 40 tasks under the docker driver on coder_eval 0.9.2, with experiments/smoke.yaml carrying no model: key and --model "$AGENT_MODEL" supplying it. Pass rate 97.5% (39/40), above the 95% threshold. Editing tests/.coder-eval-version is itself one of the infra triggers for that gate, so the pin bump got a full run rather than the changed-skills subset.

That run measured e629efeea, while the workflows still carried a || 'claude-sonnet-5' literal behind the variable. The commit that removed the fallback got a smoke run that executed zero tasks and still reported green (below).

Windows RPA gate. skill-rpa-coded-test-case failed once with MAX_TURNS_EXHAUSTED at 998s. The same task passed 3/3 on other branches the same day at 1128 to 1301s, vars.CLAUDE_CODE_MODEL was not yet set so the model was byte-identical to the deleted pin, and the 0.9.1 to 0.9.2 diff moves no dependency or CLI pin and touches the Claude agent only through LiteLLM paths that are inert on Bedrock. Re-ran: passed.

activation_gate.py. Verified the constructed command carries --model <AGENT_MODEL>, and that an unset variable exits 2 before spawning coder-eval. The activation gate does not fire on this PR, since no skill frontmatter changed.

Companion. UiPath/coder_eval_uipath#74 is green, with the per-harness resolution matrix unit-tested against the shipped daily.sh block, a denylist test that fails if a literal model id reappears in any of the three mirrors, and pwsh parity verified. Its three ADO dry runs then resolved eu.anthropic.claude-sonnet-5 / gpt-5.6-terra / gemini-3.6-flash, one per harness, off the variable group, and the antigravity build carried both the Linux and Windows slices to green including the codex and antigravity pre-flight smokes, which read CODEX_MODEL / ANTIGRAVITY_MODEL with no fallback.

One thing the smoke run exposed: the gate logs the command as --model ***. GitHub masks a value only when it matches a registered secret, so secrets.BEDROCK_MODEL holds a plain model name. That is the concrete version of the argument for moving model config to vars: nothing is protected (the model name is in run.json and on the evalboard either way), the value cannot be read back for an audit, and the smoke log can no longer tell you which model it graded. Out of scope here, but it is why the new variables use vars rather than secrets.

[github-actions]

Claude finished @bai-uipath's task in 2m 14s —— View job

---

PR Review: feat(tests): take the eval model from env vars, bump coder_eval to 0.9.2

• Read review criteria and project conventions
• Read full diff
• Read all changed files in full
• Check for security issues (secrets, tokens, paths)
• Verify correctness of workflow changes
• Verify experiment YAML changes
• Verify activation_gate.py changes
• Post review findings

---

Summary

Removes hardcoded model: claude-sonnet-5 from all six experiment YAMLs and instead passes the agent model via --model at every CI entry point, resolved from per-harness repo variables (CLAUDE_CODE_MODEL, CODEX_MODEL, ANTIGRAVITY_MODEL). Each entry point guards against an unset variable with a fail-fast error rather than a silent fallback. Also bumps coder_eval from 0.9.1 to 0.9.2.

Change-by-Change Review

1. tests/experiments/*.yaml (6 files)

Severity: OK

All six experiment files (default, nightly, smoke, smoke-windows, activation, skill-comparison-template) have the same mechanical change: remove model: claude-sonnet-5 and update the comment to explain the new resolution path. The replacement comment is identical across all six files and correctly describes the precedence: CLI --model > env var > $BEDROCK_MODEL for bare local runs. YAML remains valid. No other agent config fields were disturbed.

2. .github/workflows/smoke-skills.yml

Severity: OK

Adds AGENT_MODEL: ${{ vars.CLAUDE_CODE_MODEL }} to the env block, a bash guard (: "${AGENT_MODEL:?...}"), and passes --model "$AGENT_MODEL" to coder-eval run. The guard fires before any coder-eval invocation, producing a clear error message if the variable is unset. The echo line was updated to show the model in the log. Clean and correct.

3. .github/workflows/smoke-rpa-skills.yml

Severity: OK

Same pattern as smoke-skills.yml: adds AGENT_MODEL, guard, and --model "$AGENT_MODEL" on the coder-eval run invocation inside the retry loop. The --model is placed correctly before --tags. Consistent with the Linux smoke gate.

4. .github/workflows/activation-gate.yml

Severity: OK

Adds AGENT_MODEL: ${{ vars.CLAUDE_CODE_MODEL }} to the env block. The guard and --model usage are handled in activation_gate.py (see below). The comment explains the model-specificity of recall baselines, which is the right place for that context.

5. .github/workflows/run-coder-eval.yml

Severity: OK

The most substantial change. Both Linux and Windows jobs now:

• Map all three per-harness repo variables into the env block.
• Use a consistent model="${AGENT_MODEL:-${VAR:?...}}" pattern for each harness branch.
• The else (claude) branch now always sets agent_flags="--model $model" instead of conditionally omitting it when AGENT_MODEL was empty. This is the key behavioral change: the experiment YAML no longer carries the model, so it must always be passed on the command line.

The Linux and Windows blocks are structurally identical (correctly mirrored). The input comment at line 58 was updated to reference the new variable names. The agent_model input still overrides everything, preserving the one-off dispatch workflow.

One minor note: $agent_flags is used unquoted at the coder-eval run call site (line 285), but this is intentional word-splitting for multi-flag expansion and was preexisting behavior — model identifiers won't contain spaces.

6. tests/scripts/activation_gate.py

Severity: OK

Adds AGENT_MODEL = os.environ.get("AGENT_MODEL", "").strip() at module level, a guard in main() that returns exit code 2 when unset/empty, and "--model", AGENT_MODEL in the coder-eval subprocess invocation. The exit code 2 is distinct from the existing exit code 1 (threshold failure) and 0 (pass/skip), giving clear signal. The guard runs after argument parsing but before any work, which is the right position.

7. tests/.coder-eval-version

Severity: OK

0.9.1 → 0.9.2. Single-line, matches the PR description.

What's Missing

Nothing identified. The PR is scoped to CI infrastructure (workflows, experiment configs, gate scripts). No skill files, CODEOWNERS, or test tasks were changed — none needed to be.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill files changed
E2E Tests	N/A	No test task YAMLs changed
Skill Body	N/A	No skill files changed
References & Assets	N/A	No reference files changed
Repo Hygiene	OK	No secrets, no personal paths, changes scoped to CI infra

Issues for Manual Review

• Repo variables are set. The three gates and the dispatch workflow depend on vars.CLAUDE_CODE_MODEL, vars.CODEX_MODEL, and vars.ANTIGRAVITY_MODEL being set on the repository. The PR description states they are set, and guard failures were observed when they weren't. Verify the variables are present in Settings → Variables.
• Antigravity model upgrade. The PR description notes the ANTIGRAVITY_MODEL variable is set to gemini-3.6-flash, while the old hardcoded default was gemini-3.5-flash. This is a model upgrade, not just a refactor. Confirm this is intentional and that antigravity tasks pass on the new model (the validation table in the PR description shows SUCCESS).
• Local dev experience. Bare coder-eval run without --model now gets $BEDROCK_MODEL instead of the experiment's claude-sonnet-5. This is documented as intentional but may surprise contributors with unusual local .env configs. Consider noting this in the test README if it comes up.

Conclusion

Clean, well-structured infrastructure change. The model resolution is now consistent across all four entry points (smoke, smoke-rpa, activation, dispatch), with no hardcoded fallbacks and clear fail-fast guards. The experiment YAMLs are properly decoupled from any specific model, making them genuinely harness-agnostic. The coder_eval version bump is straightforward. No issues found that would block merge.

---