feat(cli): bundle evals into portable self-contained artifacts #1140

Objective

Add an eval portability/bundling workflow, not an eval validation command.

Proposed command surface:

agentv eval bundle my-eval.yaml --out ./portable-eval

The output should be a self-contained directory that contains the resolved eval files, local target/config data needed to run them, workspace templates, scripts, test data, and a small manifest. The goal is that the bundle can be moved to another checkout or machine and run without depending on parent .agentv/ config or shared example directories.

Rationale

validate is the wrong framing here. An eval can be schema-valid and behaviorally valid while still not being portable. Portability is a packaging property, not a validity property.

AgentV already creates a self-contained run directory when running evals, including an evals/ folder with the information needed to execute the run. This feature should reuse that existing materialization/bundling logic instead of building a separate validator/fixer or mutating source evals in place.

Design Direction

Bundle mode

agentv eval bundle my-eval.yaml --out ./portable-eval

Expected output shape should follow the existing eval-run self-contained directory structure where possible, rather than inventing a second artifact model:

portable-eval/
  evals/
  agentv_bundle.json
  ...other existing run/bundle support files as needed

agentv_bundle.json is a process-boundary artifact and must use snake_case keys.

The bundle should include or rewrite only what is needed to make the selected eval portable:

1. Resolved eval file(s) and datasets.
2. Workspace template files and local scripts referenced by the eval.
3. Resolved target configuration needed to run the bundled eval without inherited parent .agentv/targets.yaml files.
4. Relative paths rewritten inside the bundle only, not in the source files.
5. Provenance metadata such as source path, created_at, git commit when available, and included targets/config.

Reuse Existing Logic

Prefer extracting/reusing the same code path that eval runs already use to create the self-contained run directory and evals/ folder. This should be a packaging path over existing primitives, not a new eval resolver.

Non-goals

• Do not add agentv eval validate for this feature.
• Do not mutate eval files in place.
• Do not add --fix that rewrites source evals.
• Do not make schema validity and portability the same concept.
• Do not enforce portability conventions on all evals.
• Do not create a second eval execution engine.

Related

• docs: benchmarking best practices guide #1137 — benchmarking best practices
• validate: detect missing local paths in workspace hook commands #1056 — detect missing local paths in workspace hook commands

Acceptance Signals

• agentv eval bundle <eval> --out <dir> creates a portable output directory.
• The bundle reuses the existing eval-run self-contained materialization logic where practical.
• Bundled evals can run without depending on parent .agentv/targets.yaml or shared example directories.
• Source eval files are not modified.
• Bundle manifest uses snake_case keys.
• Clear errors identify non-portable references that cannot be bundled.
• Tests cover inherited targets, relative file dependencies, workspace templates/scripts, and running from the bundled directory.