fix(grade): skip incomplete command checks - #312
Merged
Conversation
Gate held-out command checks on the runner-owned run record so partial ingest leaves undispatched environments untouched. Bind reusable results to both the assertion and run record, and warn when legacy artifacts imply possible contamination.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run.jsonbefore setup injection, cache handling, command execution, or result writes.Why
Partial ingest previously iterated every task in
dispatch.json, even when only one task had completed. That could copy held-out setup files into an undispatched environment and execute the held-out command there, mutating the environment and exposing evaluation material before a retry.Closes #305.
Behavior
Before: ingesting one completed task could execute command checks in every task environment.
After: only tasks with a runner-owned
run.jsonare eligible. When another task completes, its next ingest executes the check exactly once; subsequent ingest reuses the result.Safety and compatibility
run_record_digestis optional in the schema, so existing result artifacts remain readable. A legacy result cannot be reused and is replaced after one execution for a completed task.--overwritecontinues to force execution even when both cache digests match.Verification
cargo test command_check— 32 unit, 3 CLI, and 8 run tests passedcargo test— 952 unit, 216 CLI, and 208 run tests passedcargo fmt --checkcargo clippy --all-targets -- -D warningsgit diff --checkThe pre-push hook also repeated the complete test suite successfully.