feat(init): scaffold pinned codebase fixtures - #292
Merged
Conversation
Let eval authors choose a remote URL and ref, a local path, or the invocation directory while keeping Weeknight as the pinned public default. Document fixture selection and eval-magic as the complex target in the shipped codebase guide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #265
Why
Full-codebase evals need a useful starting point before an author has chosen or
published a project fixture. A pinned public example makes the first scaffold
runnable, while explicit source options let established suites record the project
they intend to measure.
What changed
eval-magic initcodebase, pinned tob6d269c1cdedf7cadb53bacc41acaf5f2cdbe03f.--codebase-urlwith required--codebase-ref,--codebase-path, and--codebase-cwdsource selection.relative to the generated
evals/directory. Preserve canonical absolute paths.eval-magic docs codebaseguide withsource selection, fixture sizing, pinning, and provenance guidance.
schema, integration-test, and repository-instruction surface.
absolute path, current-directory, conflict, validation, and help contracts.
Behavior and boundaries
With no source option,
initwrites:{ "codebase": { "url": "https://github.com/slowdini/eval-magic-fixture", "ref": "b6d269c1cdedf7cadb53bacc41acaf5f2cdbe03f" } }Remote URL/ref values are recorded without contacting the remote;
runremains thepoint that resolves and validates them. URL and ref must be supplied together, and
the remote, path, and current-directory modes conflict with one another. Local
sources must exist and be directories before
initprompts for seed fields.This extends the established singular
codebaseguide instead of adding a parallelcodebasestopic. Existing revision-mode provisioning, snapshot/promotion behavior,and source plus resolved-SHA propagation remain shared with the new-skill path and
are covered by the codebase run suites.
Examples
Verification
cargo fmt --checkcargo buildcargo test— 926 unit, 209 CLI, and 199 run tests passedcargo clippy --all-targets -- -D warningsgit diff --checkeval-magic initand model-freeeval-magic runpreparation againstthe published URL; both conditions were clean, remote-free repositories with
matching baseline and HEAD refs.
passed at the pinned SHA.