Repository navigation
Add the real-host smoke path and record, the six-pair task bank and the evaluator repair - #2
Merged
Merged
Conversation
…he evaluator repair Adds the Claude Code real-host path (pinned host, per-launch allowlist proxy with host attribution, credential exception, live checks, stop rules and a go/no-go gate) and the record of its first 8-run engineering smoke on synthetic development tasks (docs/development/evaluation-study/smoke-record.md). Adds the WP12 development task bank (six valid/fault pairs across five families, contracts v2, census and calc stages, per-family scoring profiles), the task-bank oracles and their kernel cross-checks, and an evaluator repair driven by the smoke's failure classes with held-out tests and a read-only rejudge command. Engineering evidence only: no treatment effect, no reliability claim; the scoring rules and the development bank await the consolidated review. Source commit e67f54f.
ammarphp
force-pushed
the
evaluation-study-2
branch
from
September 28, 2026 15:32
ac0fa4c to
87f6bc1
Compare
On a host without Seatbelt every run carries the sandbox integrity flag, which nulls only unsupported_claim; a valid refusal stays True. The task-bank tests expected refusal_valid to follow that flag, which failed on ubuntu-24.04 for the kx refusal control. The host-root test used /var/empty, which exists only on macOS; it now takes the first root-owned directory of /var/empty and /usr. Source commit fa77bce.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
docs/development/evaluation-study/smoke-record.md.cli.py rejudgecommand re-scores sealed evidence into a separate directory.What it is not
Checks
mainmoves only after this PR's Linuxtest suitejob is green, by fast-forward (not a web merge).