Skip to content

feat(evals): retire fixture-only execution - #293

Merged
slowdini merged 3 commits into
devfrom
issue-266-retire-fixture-only
Aug 26, 2026
Merged

feat(evals): retire fixture-only execution#293
slowdini merged 3 commits into
devfrom
issue-266-retire-fixture-only

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Closes #266.

What and why

This removes the second, fixture-only execution architecture after codebase-backed evals became the
canonical path. Every selected eval resolves an effective project, every dispatch receives a private
environment built from that project, and task-authored files are overlays on the project tree.

The cleanup also removes completion paths that depended on agent-authored framework files. The
runner owns conversation completion state, and each harness's native transcript owns assistant
messages, tool evidence, timing, and the final response.

Implementation

  • Require either a top-level codebase or an override on every selected eval. Materialize each
    source once, build a private repository for every condition and repetition, apply files and
    files_root as overlays, and mark the starting tree with the baseline ref.
  • Remove isolation from the schema and Rust type. Validation gives old configs a targeted error
    that names the field and explains why it is unnecessary.
  • Assemble run.json from dispatch.json, runner-owned conversation.json, and per-round native
    transcripts. Remove fill-transcripts, outputs/final-message.md, and flat-transcript fallbacks.
  • Make dispatch and transcript recovery part of the runner-ready harness baseline. Static
    validation requires a usable transcript reader and tool vocabulary; harness lint --probe
    executes and parses the real capture before accepting the descriptor.
  • Carry codebase source and resolved revision through conditions.json, every run.json,
    benchmark.json, and promoted BASELINE.md output in both new-skill and revision modes.
  • Update init output, CLI help, schemas, shipped guides, internal harness notes, integration
    coverage, and generated dispatch goldens for the single workflow.

Breaking changes and compatibility

Before, an eval could omit codebase, carry a no-op isolation hint, and treat files as the entire
task environment:

{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": "update-config",
      "prompt": "Update the project configuration.",
      "expected_output": "The configuration is updated.",
      "isolation": "shared",
      "files": ["config.toml"]
    }
  ]
}

After, the project source is explicit and the same files declaration overlays that source:

{
  "skill_name": "example-skill",
  "codebase": {
    "url": "https://github.com/example/project.git",
    "ref": "0123456789abcdef0123456789abcdef01234567"
  },
  "evals": [
    {
      "id": "update-config",
      "prompt": "Update the project configuration.",
      "expected_output": "The configuration is updated.",
      "files": ["config.toml"]
    }
  ]
}

Configs that still contain isolation fail with migration guidance. BYOH descriptors must declare
[dispatch], [transcript], and [tools]; dispatch.capture_prefix is rejected. The retired
fill-transcripts command and agent-authored final-message contract are unavailable. Historical run
records that predate codebase provenance remain readable.

Safety and provenance boundaries

  • Each task runs in a private Git repository with no remotes, disabled hooks, and a runner-owned
    baseline ref. Overlay files and staged skills are committed into that baseline even when ignored
    by the source project.
  • The agent is not asked to author framework completion artifacts. Runner-owned completion state and
    persisted native transcripts determine whether a task completed and what response was returned.
  • Transcript contents remain evidence rather than policy. Descriptor-declared tool roles drive
    deterministic auditing, while runner-owned command checks and diff capture remain outside the
    harness process.
  • Resolved source identity is frozen into the iteration artifacts and reused by reporting and
    promotion instead of being re-resolved later.

Review guide

  • 927d758 implements the environment, harness, schema, and pipeline contracts.
  • f4a43d4 migrates integration coverage and dispatch goldens.
  • a34bed8 updates shipped guidance and internal harness documentation.

Verification

  • cargo fmt --check
  • cargo build
  • cargo test — 909 library tests, 210 CLI tests, and 199 run tests passed
  • cargo clippy --all-targets -- -D warnings
  • git diff --check dev...HEAD

Commit hooks reran formatting and clippy for each commit. The pre-push hook reran the complete test
suite before publishing the branch.

Remove the isolation and fixture-only execution paths so every eval resolves into a private codebase environment. Treat authored files as overlays and preserve resolved source provenance through run artifacts.

Use runner-owned conversations and harness transcripts as the single completion contract, and retire the fill-transcripts, final-message, and capture-prefix fallbacks.
Migrate integration fixtures and dispatch goldens to mandatory codebase-backed environments and runner-owned completion artifacts.

Cover revision-mode provenance and promotion, runner-ready harness probing, transcript recovery, overlays, and the explicit rejection of retired configuration.
Document codebases as mandatory, authored files as overlays, and native transcripts as the completion source across shipped guides and harness notes.

Update the BYOH baseline to require dispatch and transcript contracts so operators have one runner-ready workflow.
@slowdini
slowdini merged commit 715b454 into dev Aug 26, 2026
7 checks passed
@slowdini
slowdini deleted the issue-266-retire-fixture-only branch August 26, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retire the fixture-only eval model

1 participant