[christso]

Summary

AgentV now has a release-gate contract eval that exercises a real github-models-contract target and verifies the raw stdin payload sent to code-grader scripts.

PR #1364 in the 4.38 line changed that payload from output: [{ role, content }] to output: string, with the full transcript moved to messages. That drift was not covered by a contract eval, so downstream code graders could silently score every case as zero. This adds the missing guard and removes the producer/schema-side answer alias so the contract is enforced in both the live release gate and SDK types.

What This Catches

• output reverting to an array, object, or null instead of the final-answer string.
• Empty/missing final-answer text in output.
• Missing transcript messages or missing assistant messages in the transcript.
• Missing input and criteria passthrough to code graders.
• CamelCase wire leakage such as inputFiles, outputPath, or traceSummary instead of snake_case boundary keys.
• Reintroduction of the deprecated answer alias.

Validation

Red/base evidence:

• origin/main has no examples/contract/evals/code-grader-contract.eval.yaml.
• origin/main still has answer: context.candidate in packages/core/src/evaluation/graders/code-grader.ts.
• origin/main contract runner lists only release-gate and repo-materialization contract evals.

Green/branch evidence:

• GH_MODELS_TOKEN="$(gh auth token)" CONTRACT_EVAL_MODEL="openai/gpt-4.1-mini" bun apps/cli/src/cli.ts eval examples/contract/evals/code-grader-contract.eval.yaml --target github-models-contract --threshold 1 -> PASS, code-grader-stdin-payload score 1, 8 payload assertions passed.
• bun test packages/core/test/evaluation/code-grader-multimodal.test.ts packages/core/test/evaluation/graders.test.ts packages/core/test/evaluation/graders/prompt-resolution.test.ts packages/eval/test/define-code-grader.test.ts packages/eval/test/define-prompt-template.test.ts packages/eval/test/deprecation.test.ts packages/eval/test/file-backed-output.test.ts -> 128 pass.
• bun run validate:examples -> 61 valid, 0 invalid.
• bun run verify -> pass.

Post-Deploy Monitoring & Validation

• Watch the CI contract-eval step for code-grader-payload-contract failures.
• Search CI logs for deprecated answer alias is absent, output is the final answer string, and grader payload uses snake_case wire keys if the contract eval fails.
• Healthy signal: the new contract eval remains at score 1 and downstream code-grader-heavy suites keep their expected pass rates.
• Failure signal: contract eval score below 1, missing messages, output not string, or answer present in the payload. Roll back the payload producer change or fix the boundary serializer before promoting a release.
• Validation window: next release candidate and first post-merge CI run; owner: AgentV maintainers.

---

[cloudflare-workers-and-pages]

Deploying agentv with    Cloudflare Pages

Latest commit:	e94df54
Status:	✅  Deploy successful!
Preview URL:	https://b5ce8d2f.agentv.pages.dev
Branch Preview URL:	https://code-grader-contract.agentv.pages.dev

View logs