diff --git a/docs/adr/2026-06-18-sdk-surface-decision.md b/docs/adr/2026-06-18-sdk-surface-decision.md new file mode 100644 index 000000000..e1dc9a2c1 --- /dev/null +++ b/docs/adr/2026-06-18-sdk-surface-decision.md @@ -0,0 +1,230 @@ +# ADR: Align existing eval authoring surfaces before adding a separate SDK package + +Date: 2026-06-18 + +Status: Proposed + +## Context + +AgentV currently exposes eval authoring through three overlapping TypeScript +surfaces: + +- YAML eval files use `assertions` at suite, test, and turn level in the schema + and docs (`packages/core/src/evaluation/validation/eval-file.schema.ts` and + `apps/web/src/content/docs/docs/evaluation/*.mdx`). +- The programmatic `evaluate()` API uses `assert` in `EvalConfig`, + `EvalTestInput`, and `ConversationTurnInput`, then lowers those entries into + runtime `assertions` during materialization + (`packages/core/src/evaluation/evaluate.ts`). +- `@agentv/eval` is already a small SDK package, but it is for custom + assertion/code-grader runtimes rather than eval authoring. + +The result is vocabulary drift: + +- the runtime model is already `assertions`; +- YAML authoring is already `assertions`; +- programmatic examples in `examples/features/sdk-programmatic-api/*.ts` still + teach `assert`; +- one docs page already drifted the other way and shows `assertions` inside + `evaluate()` even though the current `EvalConfig` type does not accept it + (`apps/web/src/content/docs/docs/evaluation/sdk.mdx`). + +The question is whether that drift justifies a separate `@agentv/sdk` facade, +or whether the current surfaces can be aligned with less code. + +## Current evidence + +1. YAML is already the clearest canonical authoring contract. + - The schema accepts `assertions` on suites, tests, turns, and execution + blocks. + - The YAML parser still carries deprecated `assert` aliases, which means the + canonical name has already effectively been chosen. + +2. Programmatic `evaluate()` already converts authoring input into the YAML + runtime shape. + - `materializeEvalConfig()` maps programmatic `assert` fields into runtime + `assertions` arrays. + - This means a large part of the needed convergence is naming cleanup, not a + missing execution primitive. + +3. A separate `@agentv/sdk` package would add a third authoring vocabulary + unless it simply re-exported existing shapes. + - If it invents Braintrust-style `data/task/scores`, it diverges from AgentV + YAML and from Promptfoo-like `assert`/`assertions` concepts. + - If it mirrors existing YAML concepts, it can live in `@agentv/core` + without a new package. + +4. `@agentv/eval` already occupies the obvious "SDK" name in user mental + models. + - Adding `@agentv/sdk` now creates package-boundary confusion: + `@agentv/core` runs evals, `@agentv/eval` writes graders, and + `@agentv/sdk` would author eval definitions. + - That is more terminology, not less. + +## Decision + +Do not add a separate `@agentv/sdk` package now. + +Converge the existing YAML, programmatic, docs, and Python-facing authoring +surfaces first, using YAML-shaped definitions and small helpers in existing +packages. + +The canonical authoring vocabulary should be: + +- `tests` +- `input` +- `expected_output` on wire/YAML boundaries +- `expectedOutput` in TypeScript authoring helpers +- `assertions` for YAML and for stable programmatic authoring objects + +For programmatic TypeScript, keep camelCase where it is an ordinary TypeScript +boundary, but mirror the YAML structure rather than inventing a second model. + +## Package boundary recommendation + +Use the current package split: + +- `@agentv/core`: eval authoring and execution APIs +- `@agentv/eval`: custom assertion/code-grader runtime SDK + +If ergonomic authoring helpers are needed, add them to `@agentv/core` as a +small surface such as: + +- YAML-shaped TypeScript types for eval definitions +- a typed `defineEval()` or equivalent identity helper +- narrow assertion-builder helpers if repeated object literals prove noisy + +Do not create a new package boundary unless one of these becomes true after the +cleanup ships: + +- authoring helpers pull in dependencies that materially bloat `@agentv/core`; +- non-execution consumers need eval-definition types without the rest of core; +- a real external audience asks for a standalone authoring package often enough + that the boundary removes confusion instead of adding it. + +No current evidence in the repo proves any of those conditions. + +## Naming decision: `assertions` vs `assert` + +Choose `assertions` as the stable public authoring name across YAML and future +programmatic eval-definition APIs. + +Rationale: + +- YAML already uses `assertions`. +- the runtime model already uses `assertions`. +- Promptfoo similarity is good enough here: plural assertion entries under a + test are legible, conventional, and do not require Braintrust-style + vocabulary replacement. +- keeping programmatic `assert` while YAML uses `assertions` preserves a split + that already caused doc drift. + +Migration guidance: + +- YAML keeps `assertions` as canonical and may continue to read `assert` only as + a compatibility alias if needed. +- TypeScript `evaluate()` should converge toward accepting `assertions` on + authoring objects that mirror YAML. +- Existing programmatic `assert` can remain temporarily as a deprecated alias + during the convergence window, but it should not be the documented primary + name. + +## Options considered + +### 1. Current surfaces plus docs cleanup only + +Pros: + +- smallest immediate code change +- no new package + +Cons: + +- preserves the actual `assert` vs `assertions` type split +- leaves TypeScript authoring less YAML-shaped than claimed +- keeps docs/examples fighting the types + +Decision: insufficient. + +### 2. YAML-shaped TypeScript definitions in existing packages + +Pros: + +- aligns YAML, loader, runtime, and docs around one model +- smallest change that actually fixes the vocabulary split +- works for TypeScript and Python-facing generated shapes +- avoids adding a third package and a third naming layer + +Cons: + +- requires a small migration/deprecation pass in `evaluate()` types and docs + +Decision: recommended. + +### 3. Small helpers in existing packages + +Pros: + +- can reduce literal noise without adding new product vocabulary +- composes naturally with option 2 + +Cons: + +- helpers alone do not solve the naming split +- risks premature API flourish if added before the base model is aligned + +Decision: acceptable only after or alongside option 2, and only if the helpers +stay narrow. + +### 4. Separate `@agentv/sdk` package + +Pros: + +- could provide a clean slate for authoring ergonomics + +Cons: + +- adds package-boundary and vocabulary confusion next to `@agentv/core` and + `@agentv/eval` +- risks drifting into Braintrust-shaped `data/task/scores` APIs that the current + user decision explicitly does not want to assume are better +- duplicates types/helpers that can live in current packages with less code +- is not justified by a demonstrated dependency or publishing boundary yet + +Decision: rejected for now as overengineering. + +## Smallest path forward + +1. Treat YAML shape as canonical for authoring concepts. +2. Update programmatic eval-definition types in `@agentv/core` to center + `assertions`, not `assert`. +3. Keep temporary alias support only where needed for compatibility, and mark it + deprecated. +4. Fix docs and examples so every public eval-authoring example uses the same + terminology. +5. Leave `@agentv/eval` focused on custom grader runtime contracts. +6. Revisit a separate package only after real usage shows a boundary that cannot + be met by `@agentv/core` types/helpers. + +## Consequences + +Positive: + +- one authoring vocabulary across YAML, TS, and future Python-facing shapes +- less code than introducing and maintaining `@agentv/sdk` +- better fit with the repo principle of lightweight core plus extension points + +Negative: + +- requires a short deprecation/migration pass for current programmatic `assert` +- `@agentv/core` remains the home for both execution and authoring helpers for + now + +## Tracker impact + +- `av-bv4.2`: resolved in favor of no separate `@agentv/sdk` package at this + stage. +- `av-bv4.9`: package boundary remains `@agentv/core` for eval authoring and + `@agentv/eval` for custom grader runtime code. +- `av-bv4.10`: converge on `assertions` as the stable public authoring term; + keep `assert` only as a temporary deprecated compatibility alias where needed.