Skip to content

S13: real adapter probes for conformanceCheck; harness identity record (SHIP_HARNESS_RECORD) - #50

Merged
im-tyler merged 1 commit into
mainfrom
claude/w4-harness-probes
Oct 4, 2026
Merged

im-tyler merged 1 commit into
mainfrom
claude/w4-harness-probes

Conversation

@im-tyler

@im-tyler im-tyler commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Changes

  • src/harness-probes.ts: probes drive native (scripted model), claude-code and opencode (fake vendor binary) through durableAgent + the real externalAdapters() + LocalExecutor, for plan-review, steer and investigate. Observed accepted/refused/started/honoured/side-effects feed conformanceCheck. Other operations are reported as unprobed (not a pass).
  • src/harness-identity.ts + 5-line edit in src/runtime.ts: under SHIP_HARNESS_RECORD=on (default off) run-started gets a SIBLING harnessIdentity (model, harness + adapter version, expected vendor revision, configuration digest, forwarded env NAMES only). Not in input, so the step fingerprint and step sequence are unchanged. Off writes no key.
  • Run page: collapsible "Ran on" block (web/src/lib/harness-identity-view.ts) joining the identity with the revision the existing harness-preflight step observed; flags drift/missing binary. Absent when not recorded.
  • Credential lifecycle tests: credential reaches the binary's env, the env file is gone before the binary starts, nothing under the sandbox root or the executor env holds it after exit (also on crash), and the log has no value.

Checks

  • Root: lint clean; 1998 tests pass, 0 fail (main was 1983; +9 probes, +6 identity), run as node --test --experimental-test-isolation=none "dist/**/*.test.js" after a clean build. scripts/grader-sensitivity and pj-b grader were NOT run separately (fixed port 8901; CI runs them).
  • Web: 169 pass, pnpm run build OK.
  • Negative controls (dist edited, then rebuilt): removing the adapter's plan-review refusal fails 3 probe tests; removing the env-file cleanup fails 3 credential tests; forcing the identity on fails the flag-off byte-identity tests. Silent-drop is shown by a declaration claiming steering supported plus a gate that admits the note: silently-dropped for claude-code and opencode, clean for native.

Not verified / still open

  • Fake binaries only: no real claude/opencode, no model. Probes show what Ship's side did, not what the vendor binary does.
  • Steer gate is the real harnessSupports, mirroring the web route's gate-before-store order; the route itself is not driven.
  • approval-park, recovery, browser, interrupt, use-tools have no accept/refuse point to probe.
  • Probes are a library + tests; nothing runs them at worker startup.
  • Observed binary revision comes from the existing preflight step; identity records the expected one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VqsBNqvaWezf1DwQrAnVgX


Generated by Claude Code

…d (SHIP_HARNESS_RECORD)

Probes drive native, claude-code and opencode through durableAgent with a
scripted model / fake vendor binary over LocalExecutor and feed what was
accepted, refused, dropped or started into conformanceCheck. Unprobed
operations are listed, not counted as passing. A credential-lifecycle test
checks the forwarded credential is gone from the sandbox after exit.

Under SHIP_HARNESS_RECORD=on (default off, off writes no key) run-started
gains a sibling harnessIdentity (model, harness, expected revision,
configuration digest; names only), shown on the run page next to the
revision the preflight step observed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VqsBNqvaWezf1DwQrAnVgX
@im-tyler
im-tyler merged commit 0ba8141 into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants