feat: add remote grader refs - #494
Draft
Shayne Boyer (spboyer) wants to merge 1 commit into
Draft
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: c823d89f-d737-4983-b78a-5d4757a3b42e
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces Go-module-style remote grader references (graders[].ref) and a lockfile-driven workflow that resolves, caches, and verifies remote grader presets for offline/locked waza run execution. It adds a new waza get command to populate the local module cache and write waza.lock, and updates documentation to describe the new ref/lock behavior.
Changes:
- Add
graders[].refsupport in the eval spec model and merge/expansion logic to turn locked remote presets into concrete built-in grader configs at runtime. - Add
waza.lock(schema + read/write/validation) plus a resolver that fetches remote modules into~/.waza/cache/..., computes content digests, and enforces digest/lock strictness onwaza run. - Add
waza get [eval.yaml | ref]CLI command and docs covering refs, cache layout, locking, and merge semantics.
Show a summary per file
| File | Description |
|---|---|
| site/src/content/docs/reference/cli.mdx | Documents new waza get command usage and strict lock requirements. |
| site/src/content/docs/guides/graders.mdx | Adds guidance/example for remote grader preset refs in global graders. |
| site/src/content/docs/guides/eval-yaml.mdx | Documents graders[].ref syntax, merge semantics, and get/run flow. |
| README.md | Updates CLI examples and eval YAML docs to include remote refs + waza get. |
| docs/GUIDE.md | Adds guide mention and usage snippet for resolving remote grader presets. |
| cmd/waza/root.go | Registers the new get subcommand. |
| cmd/waza/cmd_run.go | Expands locked remote graders during waza run before execution. |
| cmd/waza/cmd_run_remote_refs_test.go | Integration-style test asserting run expands locked refs from cache + lockfile. |
| cmd/waza/cmd_get.go | Implements waza get command and routing between eval-vs-ref targets. |
| cmd/waza/cmd_get_test.go | Adds tests for get subcommand presence and lockfile writing behavior. |
| internal/registry/resolver.go | Implements ref parsing, module fetch/cache, manifest export resolution, digesting, and merge logic. |
| internal/registry/resolver_test.go | Covers resolve + expand flow and traversal-path hardening. |
| internal/registry/ref.go | Adds Go-module-style Ref parsing model. |
| internal/registry/ref_test.go | Tests ref parsing and required-version behavior. |
| internal/models/spec.go | Extends grader model to include ref and allows ref-only graders (no type) pre-expansion. |
| internal/models/spec_test.go | Adds regression coverage for loading eval specs with ref-only graders. |
| internal/models/lockfile.go | Introduces waza.lock schema, validation, and lookup helpers. |
| internal/models/lockfile_test.go | Adds lockfile round-trip and validation tests (including duplicate ref rejection). |
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 4
- Review effort level: Low
Comment on lines
+81
to
+87
| func isEvalYAMLTarget(target string) bool { | ||
| if info, err := os.Stat(target); err == nil && !info.IsDir() { | ||
| return true | ||
| } | ||
| ext := strings.ToLower(filepath.Ext(target)) | ||
| return ext == ".yaml" || ext == ".yml" | ||
| } |
Comment on lines
+247
to
+253
| if err := extractTar(bytes.NewReader(archive), extractDir); err != nil { | ||
| return "", "", err | ||
| } | ||
| if err := os.Rename(extractDir, cacheDir); err != nil { | ||
| return "", "", err | ||
| } | ||
| return commit, cacheDir, nil |
Comment on lines
+335
to
+339
| func loadGraderFile(path string, fallbackName string) (models.GraderConfig, error) { | ||
| resolved, err := resolveYAMLPath(path) | ||
| if err != nil { | ||
| return models.GraderConfig{}, err | ||
| } |
Comment on lines
+243
to
+247
| archive, err := gitOutput(ctx, "", "--git-dir", mirror, "archive", "--format=tar", commit) | ||
| if err != nil { | ||
| return "", "", fmt.Errorf("archiving commit %s: %w", commit, err) | ||
| } | ||
| if err := extractTar(bytes.NewReader(archive), extractDir); err != nil { |
Member
Author
|
High-signal review notes before merge:
I’m keeping this as the canonical #15 branch once these blockers are addressed. |
3 tasks
Member
Author
|
Correction (formatting): High-signal review notes before merge:
Keeping this as the canonical #15 branch once these blockers are addressed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
graders[].refschema support for Go-module-style remote grader references.waza get [eval.yaml | ref]to resolve refs, populate~/.waza/cache/{host}/{org}/{repo}/{sha}/, and writewaza.lockwith commit/digest/url pins.waza runexpansion with digest verification, plus resolver, lockfile, and integration coverage.waza get, remote grader refs, merge behavior, and strict lockfile requirements.Closes #15
Validation
go fmt ./... && go test ./... && golangci-lint runNotes
site/npm cicould not complete in this environment because dependency restore failed against npm package access (playwright-corereturned package feed 404, and public registry retry hitENOTCONN). Docs changes are included but the docs site build remains unverified here.