Skip to content

Add the real-host smoke path and record, the six-pair task bank and the evaluator repair - #2

Merged
ammarphp merged 2 commits into
mainfrom
evaluation-study-2
Sep 29, 2026
Merged

ammarphp merged 2 commits into
mainfrom
evaluation-study-2

Conversation

@ammarphp

@ammarphp ammarphp commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

What this adds

  • Real-host path for Claude Code. It covers:
    • a harness-owned, sha-pinned host copy;
    • a deny-default Seatbelt profile, with keychain services removed;
    • a per-launch allowlist proxy that serves only the host process;
    • a single-name credential exception that never reaches sealed records;
    • live checks and automatic stop rules;
    • a go/no-go gate after the first paid run.
  • First real-host engineering smoke. 8 assignments on two synthetic development tasks, one model, one seed. All 8 were sealed, scored and verified. Costs were reconciled against the caps; CLI-computed usage totalled about $2.59 against a $16 envelope. The record, including its deviations from procedure, is docs/development/evaluation-study/smoke-record.md.
  • WP12 development task bank. Six valid/fault pairs across five families. It adds task and claim contracts v2, new census and calc stages, and per-family scoring profiles. The independent oracles are cross-checked against the RAVEL kernel and stock pyhf.
  • Evaluator repair. Driven by the failure classes the smoke exposed. Held-out cases were committed before each change, and the 8 smoke runs are kept as replay regressions. A read-only cli.py rejudge command re-scores sealed evidence into a separate directory.

What it is not

  • Engineering evidence only: no treatment effect, no reliability claim.
  • The scoring rules, the development bank and its oracles await a consolidated human review.
  • No paid pilot has run.

Checks

  • The staged export's full suite ran on macOS: 6707 collected, 6652 passed, 53 skipped (optional dependencies and dev-only state), 0 failed.
  • The exporter's evidence, agent-surface and publication checks passed.
  • The task bank builds from the staged tree.
  • main moves only after this PR's Linux test suite job is green, by fast-forward (not a web merge).

…he evaluator repair

Adds the Claude Code real-host path (pinned host, per-launch allowlist proxy
with host attribution, credential exception, live checks, stop rules and a
go/no-go gate) and the record of its first 8-run engineering smoke on
synthetic development tasks (docs/development/evaluation-study/smoke-record.md).
Adds the WP12 development task bank (six valid/fault pairs across five
families, contracts v2, census and calc stages, per-family scoring profiles),
the task-bank oracles and their kernel cross-checks, and an evaluator repair
driven by the smoke's failure classes with held-out tests and a read-only
rejudge command.

Engineering evidence only: no treatment effect, no reliability claim; the
scoring rules and the development bank await the consolidated review.
Source commit e67f54f.
On a host without Seatbelt every run carries the sandbox integrity flag,
which nulls only unsupported_claim; a valid refusal stays True. The
task-bank tests expected refusal_valid to follow that flag, which failed
on ubuntu-24.04 for the kx refusal control. The host-root test used
/var/empty, which exists only on macOS; it now takes the first root-owned
directory of /var/empty and /usr. Source commit fa77bce.
@ammarphp
ammarphp merged commit 63b598a into main Sep 29, 2026
@ammarphp
ammarphp deleted the evaluation-study-2 branch October 6, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant