feat(eval): support project-authored task templates in comet/eval.yaml

[benym]

🔎 Existing issue check

• I have searched the existing issues.

🧭 Problem

comet eval can consume a generated comet/eval.yaml, but that manifest can only select task names already bundled under Comet's eval source tree. Defining a meaningful project-specific task still requires editing or copying the internal task.toml, instruction, Docker environment, validation script, and task-registration layout. This is too much machinery for common Skill evaluations and prevents users from keeping task definitions with their own Skill package.

✨ Proposed solution

Extend the existing comet/eval.yaml format; do not add a command, subcommand, or CLI option. Users continue to run:

comet eval ./comet/eval.yaml

Support two task authoring levels that normalize into the current eval task contract.

1. Inline tasks for the common case

Allow a user to define the prompt, deterministic expectations, and qualitative rubric directly in the manifest:

apiVersion: comet.eval/v1alpha1
kind: SkillEvalManifest

skill:
  name: my-skill
  source: ..

evaluation:
  tasks:
    - name: create-summary
      prompt: |
        Use this Skill to create result.md.
        The result must explain execution and error handling.

      # Optional. Omit for a clean empty workspace.
      workspace: ./fixtures/sample-project

      expect:
        files:
          - result.md

        contains:
          result.md:
            - "# Result"
            - "Error handling"

        json:
          - file: summary.json
            path: $.status
            equals: complete

        commands:
          - run: pnpm test
            timeout: 120

      rubric:
        - The output is complete and easy to understand
        - Boundary cases are handled correctly

The simple form should require only these concepts:

• prompt: the task given to the Agent;
• workspace: an optional project fixture copied into the isolated workspace;
• expect.files: required files or globs;
• expect.contains: required file content;
• expect.json: JSON-path value checks;
• expect.commands: commands that must succeed inside the eval container;
• rubric: optional qualitative criteria for the existing LLM Judge.

Inline tasks default to an empty workspace, the bundled generic eval environment, the generic profile, the manifest interaction settings, and the current Skill's dynamic treatment.

2. Existing task packages for advanced cases

Allow complex tasks to keep using the full existing structure:

evaluation:
  tasks:
    - source: ./tasks/complex-task

tasks/complex-task/
├── task.toml
├── instruction.md
├── environment/
├── validation/
└── data/

This path retains custom Dockerfiles, data, Python validators, setup handlers, and the complete task contract.

Shared runtime behavior

Both forms compile into one internal TaskSpec and then reuse the current Docker/workspace runner, Agent execution adapter, deterministic validators, Rubric profiles, Pass@k/Pass^k aggregation, quality gates, evidence schema, and Local/LangSmith report generation.

For inline tasks:

• generate stable internal check identifiers automatically;
• compile files, contains, json, and commands into the existing boolean validator result model;
• require at least one deterministic expect; rubric alone is not sufficient to define task success;
• run commands only inside the isolated task container with a required/default timeout;
• map rubric to the current rubric_criteria; when Judge is disabled, report it as skipped without changing deterministic results;
• record the manifest, task definition, workspace, and validation-rule hashes for reproducibility.

Existing evaluation.recommendedTasks remains valid for bundled tasks. recommendedTasks and the new tasks list may be used together and are merged for execution.

Validation and safety

• Require inline tasks to provide name, prompt, and at least one deterministic expectation.
• Reject duplicate names across inline, sourced, and bundled tasks instead of silently shadowing one.
• Resolve workspace, source, and checked artifact paths relative to the manifest/Skill package and reject real-path or symlink escapes.
• Report schema, file, JSON, command, timeout, and path failures with the task name and exact YAML field.
• Make --collect validate schemas, paths, task discovery, and compiled checks without starting Docker or an Agent.

Acceptance coverage

Add focused coverage for:

• minimal inline task parsing and execution;
• files, contains, json, and commands compilation and failure messages;
• empty and fixture-backed workspaces;
• inline, sourced, and bundled tasks in one manifest;
• duplicate names, missing prompt/expect, path escape, invalid JSON, command failure, and timeout;
• Judge-enabled and Judge-disabled rubric behavior;
• --collect validation without execution;
• unchanged legacy comet/eval.yaml and full task-package behavior;
• Local/LangSmith suites and selectable Agent adapters consuming the same normalized TaskSpec;
• reproducibility hashes in reports.

🎯 Primary area

Eval (domains/eval)

🪐 Workflow phase

Not phase-specific

🔀 Alternatives considered

1. Add comet eval init or another scaffolding command. This adds CLI surface and still makes simple users edit a multi-file package.
2. Require every user task to use the full task directory. This maximizes literal reuse but keeps the current authoring burden.
3. Add arbitrary custom metrics, formulas, and weights. This creates a second aggregation engine and makes results difficult to compare.

The inline form is a small declarative front end that compiles into the existing task and metric pipeline; the advanced source form preserves the full extension point.

🧰 Compatibility notes

This is an additive manifest extension with no new CLI surface. Existing recommendedTasks, generated manifests, bundled task packages, profiles, deterministic validators, Rubric behavior, Pass@k/Pass^k aggregation, Local/LangSmith suites, and reports remain supported.

🧩 Additional context

Today comet/eval.yaml describes the Skill, selected bundled tasks, expected evidence/artifacts, quality gates, and interaction settings, but it cannot author a task prompt, workspace, validator, or rubric criteria. This feature closes that authoring gap while keeping the current eval engine authoritative.