Skip to content

feat(init): scaffold pinned codebase fixtures - #292

Merged
slowdini merged 1 commit into
devfrom
feat/default-codebase-fixtures
Aug 25, 2026
Merged

feat(init): scaffold pinned codebase fixtures#292
slowdini merged 1 commit into
devfrom
feat/default-codebase-fixtures

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Closes #265

Why

Full-codebase evals need a useful starting point before an author has chosen or
published a project fixture. A pinned public example makes the first scaffold
runnable, while explicit source options let established suites record the project
they intend to measure.

What changed

  • Make Weeknight the default
    eval-magic init codebase, pinned to
    b6d269c1cdedf7cadb53bacc41acaf5f2cdbe03f.
  • Add --codebase-url with required --codebase-ref, --codebase-path, and
    --codebase-cwd source selection.
  • Resolve relative local paths from the invocation directory and record them
    relative to the generated evals/ directory. Preserve canonical absolute paths.
  • Extend CLI help, the README, and the shipped eval-magic docs codebase guide with
    source selection, fixture sizing, pinning, and provenance guidance.
  • Document eval-magic itself as the complex fixture for tasks that need its Rust,
    schema, integration-test, and repository-instruction surface.
  • Add CLI integration tests for the default, explicit remote, relative path,
    absolute path, current-directory, conflict, validation, and help contracts.

Behavior and boundaries

With no source option, init writes:

{
  "codebase": {
    "url": "https://github.com/slowdini/eval-magic-fixture",
    "ref": "b6d269c1cdedf7cadb53bacc41acaf5f2cdbe03f"
  }
}

Remote URL/ref values are recorded without contacting the remote; run remains the
point that resolves and validates them. URL and ref must be supplied together, and
the remote, path, and current-directory modes conflict with one another. Local
sources must exist and be directories before init prompts for seed fields.

This extends the established singular codebase guide instead of adding a parallel
codebases topic. Existing revision-mode provisioning, snapshot/promotion behavior,
and source plus resolved-SHA propagation remain shared with the new-skill path and
are covered by the codebase run suites.

Examples

# Use the pinned Weeknight default.
eval-magic init

# Record another Git source without resolving it yet.
eval-magic init \
  --codebase-url https://github.com/slowdini/eval-magic \
  --codebase-ref e30a5091c844e07f3aa664413aa7735a11b0a52a

# Use the project in which init is invoked.
eval-magic init --codebase-cwd

Verification

  • cargo fmt --check
  • cargo build
  • cargo test — 926 unit, 209 CLI, and 199 run tests passed
  • cargo clippy --all-targets -- -D warnings
  • git diff --check
  • Resolved the public fixture anonymously at the pinned SHA.
  • Ran default eval-magic init and model-free eval-magic run preparation against
    the published URL; both conditions were clean, remote-free repositories with
    matching baseline and HEAD refs.
  • Fixture CI
    passed at the pinned SHA.

Let eval authors choose a remote URL and ref, a local path, or the invocation directory while keeping Weeknight as the pinned public default.

Document fixture selection and eval-magic as the complex target in the shipped codebase guide.
@slowdini
slowdini merged commit b892346 into dev Aug 25, 2026
7 checks passed
@slowdini
slowdini deleted the feat/default-codebase-fixtures branch August 25, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default codebase fixtures and shipped docs

1 participant