Repository navigation
feat: measured executor cards for kiro-cli, claude and codex - #21
Conversation
`passdown-attempt validate --file <card> --as card` checks an executor card against design section 18 and the dispatch skill's eligibility rule: - only known sections, capability names and capability values; - a card name that matches its file, a positive card_version and a YYYY-MM-DD measurement date; - recorded evidence behind every verified or unsupported capability; - invocation.headless, output_capture and result_extraction, which a launch needs; - settle_seconds above zero unless descendants_may_outlive is unsupported; - invocation.resume only with resume_session verified. It reuses the helper's own card parser, so hosts, the card lint and the attempt snapshot read a card the same way. tests/attempt.sh covers a complete card and one broken variant per rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
E-KIRO-1 (design section 19) ran kiro-cli 2.24.0 through the real v0.5 attempt lifecycle eight times on a disposable fixture: a read, a bounded write without and with trust, a continuation after `blocked`, an ambiguous task, a resumed continuation without the task text, and SIGINT/SIGKILL interruptions of a long task. The runner, every prompt, transport, receipt and process tree are in docs/evidence/e-kiro-1/ (local paths replaced). The card marks as verified: headless, stream-json structured output, session ID, resume, meaningful exit code, cancel by SIGINT to the process group, and a read-only mode (no trust flags). Native schema enforcement is unsupported. descendants_may_outlive stays unverified: kiro-cli's own processes never left the process group, but a trusted shell can start a detached process, so attempts still end with owner attestation. E-KIRO-1 also found that a worker follows a hand-written description of the result schema exactly, mistakes included. The dispatch skill now tells the host to put the schema itself into the prompt, or pass it through the executor's schema flag. docs/EXECUTOR_SETUP.md becomes the guide to measuring and writing a card, and tests/skills.sh runs the card check on every shipped card and requires its evidence to exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
E-CLAUDE-1 and E-CODEX-1 apply the E-KIRO-1 method to Claude Code 2.1.193 and codex-cli 0.157.0, with a shared fixture (docs/evidence/fixture.sh) and a runner generalized from the E-KIRO-1 one (docs/evidence/executor-run.sh). Every run went through the real attempt lifecycle. The records sit in docs/evidence/e-claude-1/ and e-codex-1/; local paths are replaced, and Claude's init and hook events are reduced so that no tool, MCP server or plugin name is published. Both cards mark as verified: headless structured output, session resume, a read-only mode (Claude's default permission mode, Codex's OS-enforced read-only sandbox) and cancel by SIGINT to the process group. Codex's workspace-write sandbox refused a write outside the workspace. Both run tool shells in their own process groups. After SIGKILL of the worker, those shells survived and kept writing while the launch group was already empty, so descendants_may_outlive is verified for both. The runner held the attempt unknown until the operator stopped the survivor. Neither CLI can enforce passdown's result schema: Claude's --json-schema produced no structured output for it, and Codex's --output-schema was rejected by the API. Claude exits 0 after SIGINT, so its exit code is marked unsupported. Operator gotchas are recorded in the cards: Claude's variadic --allowedTools must use `=`, `codex exec` needs stdin closed, and `codex exec resume` takes -c sandbox_mode instead of --sandbox. docs/EXECUTOR_SETUP.md points at the shared runner and adds the interrupt-after-first-write and KILL steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The design gains the S6 implementation notes: card lint in the helper, the CLI versions that moved before measuring, the live F18 case on Claude Code and Codex, the absence of native schema enforcement for passdown's result schema, differing exit codes, and the operator lessons. CHANGELOG records the slice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vunm-dev
left a comment
There was a problem hiding this comment.
Two findings need adjustment before merging. Validation: the card validation tests, all 150 skill checks, and the bundled helper/schema synchronization check passed. I also verified that the shipped Claude card permits a read attempt with readonly:default to have claim: null, without launching an executor. GitHub CI is green; live executor measurements were not rerun. This is a COMMENT review because the authenticated account is the PR author.
| exit_code_meaningful: unsupported | ||
| descendants_may_outlive: verified | ||
| cancel_signal_honored: verified | ||
| read_only_mode: verified |
There was a problem hiding this comment.
[P1] Do not enable claim-free reads without an enforced read-only profile
The evidence only establishes denial of writes with a user settings file that grants nothing, while the documented read invocation still loads the user's settings, hooks, plugins and MCP servers. The card itself says permissive settings widen this purported read-only mode. Marking read_only_mode verified nevertheless makes the helper accept --mutation-guard readonly:default and create an attempt with claim: null (confirmed locally without launching a worker). On a machine with preapproved writing tools, that read worker can therefore write alongside another claimed writer. Measure and specify an invocation that actually removes write capability independently of inherited permissions, or keep this capability unverified so the read takes a writer claim. The Kiro card's acknowledged auto-approved MCP exception needs the same treatment.
There was a problem hiding this comment.
Fixed in 7ca5bb4.
- Claude Code and kiro-cli now mark
read_only_mode: unverified. Their cards say that a read attempt takes the writer claim, and why: the invocation inherits the user's permissions, hooks and MCP servers, and auto-approved MCP servers were not measured. - For Codex I measured an invocation that does not depend on inherited configuration:
--sandbox read-only --ignore-user-config.- Run R: a shell write failed with
operation not permitted, with no plugin or MCP activity. - Run R2:
apply_patchwas refused with "writing is blocked by read-only sandbox". - The Codex card keeps
verifiedand names that invocation and the matching--mutation-guard. Transports are indocs/evidence/e-codex-1/runs/R*.transport.jsonl.
- Run R: a shell write failed with
- Found while fixing this: a card file named
claude.mdisCLAUDE.mdon case-insensitive file systems, and Claude Code loaded it as project instructions. The card is nowclaude-code, andvalidate --as cardrefuses card names that clash with agent instruction files.
| if [ "$(jq -r .pgroup_members <<<"$facts")" = 0 ] && [ -z "$survivors" ] && [ -z "${new_kiro// /}" ]; then | ||
| H observe "$id" stopped --rev "$(rev "$id")" --evidence owner-attested --attested-by "$operator" --exit "$code" >/dev/null | ||
| note "- stop: owner-attested by $operator (process group empty, no descendant and no new executor process alive)" |
There was a problem hiding this comment.
[P2] Wait for explicit confirmation before recording owner-attested
This branch automatically records owner-attested from empty process listings and then proceeds to inspection and acceptance; it never obtains an operator confirmation. Sampling the descendant tree every 0.3 seconds can miss a quickly detached/reparented process, and the executor-name scan does not establish that no detached tool process remains. This converts the absence of safe machine evidence into an attestation, contrary to dispatch steps 10–11, which require a person to confirm and prohibit attesting on their behalf. Leave the attempt unknown with its evidence saved, and require an explicit confirmation step before recording stopped and continuing. The same automatic branch exists in docs/evidence/e-kiro-1/run.sh; distinguish its historical measurements from protocol-compliant acceptance evidence.
There was a problem hiding this comment.
Fixed in 7ca5bb4.
executor-run.shnow stops atunknownwhen there is no safe machine evidence. It saves the listings and a state file, and recordsowner-attestedonly after an explicit--finish <part> --attested-by <who>. Smoke-tested with one kiro-cli run through both phases.- The historical
e-kiro-1/run.shstays as it ran. Each evidence README gains a "How stops were attested" section: those records were attested automatically from 0.3 s process scans, which is not the confirmation dispatch steps 10–11 require, and theiracceptedverdicts are measurement artifacts, not protocol-compliant acceptance.
Review round 1 on #21, plus one finding made while fixing it: - Claim-free reads need an enforced read-only mode. The Claude Code and kiro-cli read-only behavior was measured only against the operator's own inherited settings: Claude inherits the user's permissions, hooks and MCP servers, and kiro-cli's auto-approved MCP servers were not measured. Both cards now mark read_only_mode unverified, so a read attempt takes the writer claim. For Codex, two new runs (R, R2) measured `--sandbox read-only --ignore-user-config` refusing a shell write and an apply_patch with no plugin or MCP activity. Its card keeps verified and names that invocation. - The measurement runner no longer attests by itself. executor-run.sh stops at unknown and records owner-attested only after `--finish <part> --attested-by <who>`. The three evidence READMEs say that their records were attested automatically, and that their accepted verdicts are measurement artifacts. - The Claude Code card is now claude-code. A card file named claude.md is CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as project instructions while this change was being made. `validate --as card` now refuses a card named like an agent instruction file (claude, agents, gemini), and tests/attempt.sh covers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The S6 notes gain the claude-code naming and review round 1 on #21: claim-free reads only with an enforced read-only mode, and the runner's explicit attestation. CHANGELOG follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vunm-dev
left a comment
There was a problem hiding this comment.
Follow-up review: the Claude/Kiro read-only capability downgrade and the explicit --finish attestation step address the previous findings. One remaining claim-free path needs the same isolation treatment. Card validation, all 150 skill checks, and bundled helper/schema synchronization passed; GitHub CI is green. No live executor was launched during this review.
| descendants_may_outlive: verified | ||
| cancel_signal_honored: verified | ||
| read_only_mode: verified | ||
| sandbox_confined_writes: verified |
There was a problem hiding this comment.
[P1] Close the inherited-permissions gap for snapshot+sandbox reads too
The read-only invocation now requires --ignore-user-config, but sandbox_confined_writes remains verified and the documented workspace-write invocation does not require that isolation. Dispatch also permits claim-free reads via --mutation-guard snapshot+sandbox when this capability is verified. I confirmed that the helper still creates claim: null for this guard, without launching an executor. As the card now explicitly notes, inherited MCP servers/plugins operate outside the sandbox; running the worker in a snapshot does not prevent those tools from writing to the original repository alongside its claimed writer. Mark sandbox_confined_writes unverified until an isolated profile is measured, or measure and explicitly require the isolated invocation for snapshot+sandbox as well, with coverage for this second claim-free path.
There was a problem hiding this comment.
Fixed in c146016: the Codex card now marks sandbox_confined_writes: unverified, so a snapshot+sandbox read on Codex takes the writer claim.
- W refused a shell write outside the workspace, but it ran with the user's MCP servers and plugins loaded, which act outside the sandbox.
apply_patchoutside the workspace and writes to/tmpwere not measured either. - Rather than measure an isolated workspace-write profile now, v0.5 ships one claim-free read path only: Codex's measured read-only invocation,
--sandbox read-only --ignore-user-config(runs R and R2). - The card, the E-CODEX-1 README, the S6 design notes and CHANGELOG say so. The PR description's "writes confined" row is superseded by this.
Review round 2 on #21: the Codex card still marked sandbox_confined_writes verified, which lets `--mutation-guard snapshot+sandbox` create a claim-free read. The measuring run (W) refused a shell write outside the workspace, but it ran with the user's MCP servers and plugins loaded, and those act outside the sandbox; apply_patch outside the workspace and writes to /tmp were not measured. The capability is now unverified. The only claim-free read path shipped in v0.5 is Codex's measured read-only invocation (`--sandbox read-only --ignore-user-config`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 on #21, plus one finding made while fixing it: - Claim-free reads need an enforced read-only mode. The Claude Code and kiro-cli read-only behavior was measured only against the operator's own inherited settings: Claude inherits the user's permissions, hooks and MCP servers, and kiro-cli's auto-approved MCP servers were not measured. Both cards now mark read_only_mode unverified, so a read attempt takes the writer claim. For Codex, two new runs (R, R2) measured `--sandbox read-only --ignore-user-config` refusing a shell write and an apply_patch with no plugin or MCP activity. Its card keeps verified and names that invocation. - The measurement runner no longer attests by itself. executor-run.sh stops at unknown and records owner-attested only after `--finish <part> --attested-by <who>`. The three evidence READMEs say that their records were attested automatically, and that their accepted verdicts are measurement artifacts. - The Claude Code card is now claude-code. A card file named claude.md is CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as project instructions while this change was being made. `validate --as card` now refuses a card named like an agent instruction file (claude, agents, gemini), and tests/attempt.sh covers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The S6 notes gain the claude-code naming and review round 1 on #21: claim-free reads only with an enforced read-only mode, and the runner's explicit attestation. CHANGELOG follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 on #21, plus one finding made while fixing it: - Claim-free reads need an enforced read-only mode. The Claude Code and kiro-cli read-only behavior was measured only against the operator's own inherited settings: Claude inherits the user's permissions, hooks and MCP servers, and kiro-cli's auto-approved MCP servers were not measured. Both cards now mark read_only_mode unverified, so a read attempt takes the writer claim. For Codex, two new runs (R, R2) measured `--sandbox read-only --ignore-user-config` refusing a shell write and an apply_patch with no plugin or MCP activity. Its card keeps verified and names that invocation. - The measurement runner no longer attests by itself. executor-run.sh stops at unknown and records owner-attested only after `--finish <part> --attested-by <who>`. The three evidence READMEs say that their records were attested automatically, and that their accepted verdicts are measurement artifacts. - The Claude Code card is now claude-code. A card file named claude.md is CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as project instructions while this change was being made. `validate --as card` now refuses a card named like an agent instruction file (claude, agents, gemini), and tests/attempt.sh covers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The S6 notes gain the claude-code naming and review round 1 on #21: claim-free reads only with an enforced read-only mode, and the runner's explicit attestation. CHANGELOG follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 on #21: the Codex card still marked sandbox_confined_writes verified, which lets `--mutation-guard snapshot+sandbox` create a claim-free read. The measuring run (W) refused a shell write outside the workspace, but it ran with the user's MCP servers and plugins loaded, and those act outside the sandbox; apply_patch outside the workspace and writes to /tmp were not measured. The capability is now unverified. The only claim-free read path shipped in v0.5 is Codex's measured read-only invocation (`--sandbox read-only --ignore-user-config`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PDN-0004, slice S6 (executor reference cards and E-KIRO-1). Design:
docs/design/PDN-0004-v05-acceptance-recovery.md§18, §19, §26.2 S6; S4 eligibility rule (#18, #19). Issue #12.What changes
Card lint in the helper:
passdown-attempt validate --file <card> --as card.verifiedorunsupportedvalue, and the S4 eligibility minimum (invocation.headless,output_capture,result_extraction;settle_seconds > 0unless no-detach was measured).tests/attempt.shhas one broken variant per rule, andtests/skills.shruns the check on every shipped card and requires its evidence to exist.Three measured cards, each from runs through the real attempt lifecycle on a disposable fixture:
kiro-clidocs/evidence/e-kiro-1/(E-KIRO-1, parts A–E)claude-codedocs/evidence/e-claude-1/codexdocs/evidence/e-codex-1/Shared tooling:
docs/evidence/fixture.shanddocs/evidence/executor-run.sh. The E-KIRO-1 runner is kept as it ran.Skill fix found by measuring: a worker copied a hand-written description of the result schema, mistakes included. Dispatch step 4 now puts the schema itself into the prompt.
Docs:
docs/EXECUTOR_SETUP.mdbecomes "measuring and writing a card". Also updated: the executor README (shipped-cards table; onlyagyleft as an unmeasured hint), CHANGELOG, and the S6 implementation notes.What the measurements found
--sandbox read-only --ignore-user-config--json-schemaproduced none--output-schemarejected by the APISIGINTSIGKILLdescendants_may_outliveunverifiedverifiedverifiedThe
SIGKILLruns on Claude and Codex are a live case of F18. The launch process group was empty while a tool shell kept appending to the file. The runner did not attest: the attempt stayedunknownand held its claim until the operator stopped the survivor. For all three executors on macOS, every attempt ends with owner attestation unless it ran in a containment scope.Review focus
kiro-clidescendants_may_outlive: unverified. Its own processes never left the launch group in 8 runs, but a trusted shell could start a detached process. Isunverifiedthe right call, rather thanunsupported?read_only_mode: verified. It was measured with a user settings file that grants nothing. A permissive settings file widens it; the card says so. Enough?init/hook events reduced (no tool, MCP or plugin names). Anything else that should not be in a public repo?owner-attestedfor measurement runs?Tests (local, macOS, bash 3.2)
tests/skills.sh: 150.tests/attempt.sh: 41.tests/docs.sh: 32.tests/install.sh: 7.tests/version.sh: 4.tests/openspec-apply.sh: 6.sync-bundled --check,validate-plugin,release-notesandshellcheck(including the evidence scripts) are clean.🤖 Generated with Claude Code