Repository navigation
test: S7 plan and the Layer B harness - #23
Conversation
Seven tasks for the release slice under the gate narrowed in #22: isolate the Layer B host, build the Layer B fixture, stage markers and file oracles, write the scenario scripts, run Layer B with the owner, write the release docs, and cut the release. Task 1.1 is done: a claude -p host loaded both the 0.4.0 plugin and the direct v0.5 copies, so the owner moved Claude to the beta plugin channel pinned to release/v0.5.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/harness/layer-b/ runs the gate scenarios of #22 (F1, F2, F11, F14a, F14b, F18, F27) against a real `claude -p` host with the v0.5 skills: - fixture.sh builds each scenario's workspace: a plan, an AGENTS.md with an owner routing policy, and an executor card for the test double in the scenario's mode. F27 routes to an executor that has no card. - run.sh kills the host at the scenario's crash marker (after-arm, after-result, after-verdict), applies the scenario's edit, and opens a new session for pickup. It relays the owner's answers verbatim and never answers or attests for them. - collect.sh and oracle.sh judge a run from files only: contract, acceptance by ground truth, final-tree digests, stop evidence, no second writer, no overlapping worker lifetimes. A pilot F1 run showed the host stopping to ask for attestation, and the oracle failed that unfinished state as it should. A smoke F27 run passed all checks. Neither counts as evidence; plan task 3.1 runs the gate with the owner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The glob `tests/harness/*` now also matches the `layer-b/` directory, which shellcheck rejects. List the harness scripts, the Layer B scripts and the evidence scripts explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vunm-dev
left a comment
vunm-dev
left a comment
There was a problem hiding this comment.
Recommendation: address both P1 findings before merging.
Two issues prevent this harness from providing reliable Layer B release evidence.
Validation: on the current PR head, I ran fixture.sh -> collect.sh -> oracle.sh for F1, F2, F11, F14a, F14b, F18 and F27 without launching a host or worker. All seven returned pass. The crash-monitoring finding comes from tracing the initial and resumed host calls against the required owner-attestation flow. CI is green; I did not run a live owner-attested Layer B session or change repository files.
| IFS= read -r answer </dev/tty || answer="" | ||
| [ -n "$answer" ] || break | ||
| printf '[%s] %s\n' "$operator" "$answer" >>"$out/owner-answers.txt" | ||
| host "$session" "$answer" |
There was a problem hiding this comment.
[P1] Preserve crash monitoring across owner replies
F14a/F14b require owner attestation before the host can persist an accepted verdict. The initial host call therefore returns before reaching the marker, but this resumed call omits the watcher argument. Crash handling also exists only before this loop, so the scenario never applies its mutation or starts pickup. Keep monitoring subsequent turns until the marker fires, and handle the crash and recovery from that turn.
There was a problem hiding this comment.
Fixed in 8b6a332. Every host turn is now watched for the marker until it fires once, including the turns resumed with the owner's answers. The crash is handled in the turn where it happens: the scenario's after_crash edit, then a new pickup session. run.sh records turn, crash, after_crash and pickup events in run-events.txt. Tested with a stub host under a pseudo-terminal (no answer back door): F14a's marker fired in resumed turn 02, the task edit landed, and pickup ran in turn 03.
| case "$GROUND_TRUTH" in | ||
| must-not-accept) | ||
| [ "${#accepted[@]}" = 0 ] && ok "no receipt accepted" || no "a receipt was accepted" | ||
| [ "$ticked" = 0 ] && ok "task 1.1 is not ticked" || no "task 1.1 is ticked: $task_line" | ||
| ;; | ||
| must-not-project) | ||
| [ "${#projected[@]}" = 0 ] && ok "no verdict projected" || no "a verdict was projected" | ||
| [ "$ticked" = 0 ] && ok "task 1.1 is not ticked" || no "task 1.1 is ticked: $task_line" |
There was a problem hiding this comment.
[P1] Require evidence that each scenario actually ran
Running fixture.sh, collect.sh and oracle.sh without launching any host or worker returns pass for all seven scenarios. Empty receipts and journals satisfy these negative assertions, so an aborted or never-exercised run can qualify as release evidence. Require scenario-specific execution evidence before judging safety: worker activity where expected, the crash marker and recovery for crash scenarios, and an accepted verdict plus the intended digest change for F14. Missing evidence must be incomplete or failure.
There was a problem hiding this comment.
Fixed in 8b6a332. Each scenario now lists the evidence it requires (REQUIRE):
- F1:
host worker - F2:
host worker crash pickup - F11:
host crash pickup(the worker may not have launched before the crash) - F14a:
host worker crash after_crash pickup accepted task-changed - F14b:
host worker crash after_crash pickup accepted artifact-changed - F18:
host worker detached-write - F27:
host
Missing evidence makes the result INCOMPLETE (exit 2), never a pass. Your fixture → collect → oracle run now gives INCOMPLETE for all seven scenarios, and the earlier real-host F27 smoke run still passes. The oracle also now reports an unexpected error as FAIL instead of exiting without a result; a stub run with a malformed receipt exposed that.
…l run Review on #23: - run.sh watched for the crash marker only in the first host turn. F14's accepted verdict comes after the owner's attestation, in a resumed turn, so the marker never fired and neither the scenario edit nor pickup ran. Every turn is now watched until the marker fires once, and the crash is handled in that turn. run.sh also records turn, crash, after_crash and pickup events in run-events.txt. - oracle.sh passed all seven scenarios when no host or worker had run, because empty receipts and journals satisfy every negative check. Each scenario now lists the evidence it requires (REQUIRE): a completed host turn, the worker, the crash and pickup, an accepted verdict, a changed task or artifact, the detached write. Missing evidence makes the run INCOMPLETE (exit 2), never a pass. The oracle also no longer dies silently on a malformed receipt: an unexpected error is reported as FAIL. Tested without a live host: - the reviewer's fixture → collect → oracle run now gives INCOMPLETE for all seven scenarios; - the earlier real-host F27 smoke run still passes; - a stub host under a pseudo-terminal showed F14a's crash caught in the resumed turn, the scenario edit applied, and a pickup session started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l run Review on #23: - run.sh watched for the crash marker only in the first host turn. F14's accepted verdict comes after the owner's attestation, in a resumed turn, so the marker never fired and neither the scenario edit nor pickup ran. Every turn is now watched until the marker fires once, and the crash is handled in that turn. run.sh also records turn, crash, after_crash and pickup events in run-events.txt. - oracle.sh passed all seven scenarios when no host or worker had run, because empty receipts and journals satisfy every negative check. Each scenario now lists the evidence it requires (REQUIRE): a completed host turn, the worker, the crash and pickup, an accepted verdict, a changed task or artifact, the detached write. Missing evidence makes the run INCOMPLETE (exit 2), never a pass. The oracle also no longer dies silently on a malformed receipt: an unexpected error is reported as FAIL. Tested without a live host: - the reviewer's fixture → collect → oracle run now gives INCOMPLETE for all seven scenarios; - the earlier real-host F27 smoke run still passes; - a stub host under a pseudo-terminal showed F14a's crash caught in the resumed turn, the scenario edit applied, and a pickup session started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 participant
PDN-0004, slice S7, first part. Plan:
docs/superpowers/plans/2026-09-25-pdn-0004-s7-release.md. Gate: §20.2 as narrowed in #22. Issue #12.What changes
S7 plan (7 tasks). Tasks 1.1 and 2.1–2.3 are done here. Task 3.1 (running Layer B) needs the owner at the keyboard; 4.x is the release docs and the release itself.
Task 1.1, host isolation. Measured first: a
claude -phost on the owner's machine loaded both the 0.4.0 plugin (passdown:passdown-*) and the direct v0.5 copies (passdown-*), and when asked to list them it left one out. The owner moved Claude to the beta plugin channel pinned torelease/v0.5.0and dropped the direct copies. Nowdoctor.shreports "install channels are clean", and the host sees exactly one set, byte-identical to this branch.tests/harness/layer-b/:fixture.shAGENTS.mdwith an owner routing policy, and a card for the test double in the scenario's modescenarios/<S>.shrun.shcollect.sh,oracle.shREADME.mdCI: shellcheck now names the harness files; the
tests/harness/*glob started matching the new directory.Pilots (not evidence)
new→arm→running→unknown --parent-exited→result, then stopped and asked the owner to attest instead of recording it. The oracle correctly fails that unfinished state (the worker's[x]is still in the plan).Dispatched:line, did not do the task, and reported the missingghostcard with remedies. The oracle passed all six checks.Review focus
scenarios/*.sh). F1 and F2 aremust-not-accept, which also forbids a leftover[x], following §20.1's "any plan[x]" rule.run.sh, especiallyafter-arm: armed, with no handle yet.run.shthat could answer or attest for the owner.🤖 Generated with Claude Code