gaia: stop __pycache__ from dirtying the dataset checkout - #1619
Merged
Conversation
Importing scorer.py compiled bytecode into <GAIA_DIR>/__pycache__/, so the first score() dirtied the tree and every later grade refused with 'local modifications'. Run the driver with python -B so bytecode is never written, and have verifyDataset delete any existing __pycache__/ — it's derived state, and a doctored .pyc there could shadow the sha-pinned scorer.py source, so removal (not tolerance) is the right call. Heals already-dirty prod checkouts on the next grade. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On prod, GAIA grading failed with:
The scorer driver does
sys.path.insert(0, gaia_dir)+from scorer import question_scorer, and CPython writes bytecode to<GAIA_DIR>/__pycache__/— so the firstscore()succeeds and dirties the tree, and every later grade refuses.Fix:
python -Bso bytecode is never written into the checkoutverifyDatasetdeletes any existing__pycache__/before the clean-tree check — it's derived state, and a doctored.pycwith matching mtime/size could shadow the sha-pinnedscorer.pysource, so deleting (rather than tolerating it untracked) also closes that pin-bypass vectorAlready-dirty prod checkouts heal themselves on the next grade. Smoke gains a regression case: a planted
__pycache__/is removed, andscore()is asserted not to create one.🤖 Generated with Claude Code