Skip to content

feat: measured executor cards for kiro-cli, claude and codex - #21

Merged
vunm-dev merged 7 commits into
release/v0.5.0from
feat/pdn-0004-s6-cards
Sep 25, 2026
Merged

vunm-dev merged 7 commits into
release/v0.5.0from
feat/pdn-0004-s6-cards

Conversation

@vunm-dev

@vunm-dev vunm-dev commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

PDN-0004, slice S6 (executor reference cards and E-KIRO-1). Design: docs/design/PDN-0004-v05-acceptance-recovery.md §18, §19, §26.2 S6; S4 eligibility rule (#18, #19). Issue #12.

What changes

  • Card lint in the helper: passdown-attempt validate --file <card> --as card.

    • It checks known keys and capability values, recorded evidence behind every verified or unsupported value, and the S4 eligibility minimum (invocation.headless, output_capture, result_extraction; settle_seconds > 0 unless no-detach was measured).
    • It uses the helper's own card parser. tests/attempt.sh has one broken variant per rule, and tests/skills.sh runs the check on every shipped card and requires its evidence to exist.
  • Three measured cards, each from runs through the real attempt lifecycle on a disposable fixture:

    Card Version Records
    kiro-cli 2.24.0 docs/evidence/e-kiro-1/ (E-KIRO-1, parts A–E)
    claude-code Claude Code 2.1.193 docs/evidence/e-claude-1/
    codex codex-cli 0.157.0 docs/evidence/e-codex-1/
  • Shared tooling: docs/evidence/fixture.sh and docs/evidence/executor-run.sh. The E-KIRO-1 runner is kept as it ran.

  • Skill fix found by measuring: a worker copied a hand-written description of the result schema, mistakes included. Dispatch step 4 now puts the schema itself into the prompt.

  • Docs: docs/EXECUTOR_SETUP.md becomes "measuring and writing a card". Also updated: the executor README (shipped-cards table; only agy left as an unmeasured hint), CHANGELOG, and the S6 implementation notes.

What the measurements found

kiro-cli claude codex
Structured output, session ID, resume ✅ ✅ ✅
Read-only mode (claim-free read) not established (auto-approved MCP) not established (inherits user settings) ✅ --sandbox read-only --ignore-user-config
Writes confined to the workspace not measured not measured not established (user MCP/plugins act outside the sandbox; review round 2)
Native enforcement of passdown's result schema no option --json-schema produced none --output-schema rejected by the API
Exit code after SIGINT 1 0 1
Tool shells launch process group own process groups own process groups
After SIGKILL nothing survived shell survived and kept writing shell survived and kept writing
descendants_may_outlive unverified verified verified

The SIGKILL runs on Claude and Codex are a live case of F18. The launch process group was empty while a tool shell kept appending to the file. The runner did not attest: the attempt stayed unknown and held its claim until the operator stopped the survivor. For all three executors on macOS, every attempt ends with owner attestation unless it ran in a containment scope.

Review focus

  1. kiro-cli descendants_may_outlive: unverified. Its own processes never left the launch group in 8 runs, but a trusted shell could start a detached process. Is unverified the right call, rather than unsupported?
  2. Claude read_only_mode: verified. It was measured with a user settings file that grants nothing. A permissive settings file widens it; the card says so. Enough?
  3. Evidence publication. Transports are published with local paths replaced, and Claude's init/hook events reduced (no tool, MCP or plugin names). Anything else that should not be in a public repo?
  4. Operator attestation in the evidence. Stops were attested by the measuring session from recorded process listings, and the two survivors were killed before attesting. Is that an acceptable reading of owner-attested for measurement runs?

Tests (local, macOS, bash 3.2)

  • tests/skills.sh: 150. tests/attempt.sh: 41. tests/docs.sh: 32. tests/install.sh: 7. tests/version.sh: 4. tests/openspec-apply.sh: 6.
  • sync-bundled --check, validate-plugin, release-notes and shellcheck (including the evidence scripts) are clean.

🤖 Generated with Claude Code

vunm-dev and others added 4 commits September 25, 2026 11:44
`passdown-attempt validate --file <card> --as card` checks an executor card
against design section 18 and the dispatch skill's eligibility rule:

- only known sections, capability names and capability values;
- a card name that matches its file, a positive card_version and a
  YYYY-MM-DD measurement date;
- recorded evidence behind every verified or unsupported capability;
- invocation.headless, output_capture and result_extraction, which a
  launch needs;
- settle_seconds above zero unless descendants_may_outlive is unsupported;
- invocation.resume only with resume_session verified.

It reuses the helper's own card parser, so hosts, the card lint and the
attempt snapshot read a card the same way. tests/attempt.sh covers a
complete card and one broken variant per rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
E-KIRO-1 (design section 19) ran kiro-cli 2.24.0 through the real v0.5
attempt lifecycle eight times on a disposable fixture: a read, a bounded
write without and with trust, a continuation after `blocked`, an ambiguous
task, a resumed continuation without the task text, and SIGINT/SIGKILL
interruptions of a long task. The runner, every prompt, transport, receipt
and process tree are in docs/evidence/e-kiro-1/ (local paths replaced).

The card marks as verified: headless, stream-json structured output,
session ID, resume, meaningful exit code, cancel by SIGINT to the process
group, and a read-only mode (no trust flags). Native schema enforcement is
unsupported. descendants_may_outlive stays unverified: kiro-cli's own
processes never left the process group, but a trusted shell can start a
detached process, so attempts still end with owner attestation.

E-KIRO-1 also found that a worker follows a hand-written description of the
result schema exactly, mistakes included. The dispatch skill now tells the
host to put the schema itself into the prompt, or pass it through the
executor's schema flag.

docs/EXECUTOR_SETUP.md becomes the guide to measuring and writing a card,
and tests/skills.sh runs the card check on every shipped card and requires
its evidence to exist.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
E-CLAUDE-1 and E-CODEX-1 apply the E-KIRO-1 method to Claude Code 2.1.193
and codex-cli 0.157.0, with a shared fixture (docs/evidence/fixture.sh) and
a runner generalized from the E-KIRO-1 one (docs/evidence/executor-run.sh).
Every run went through the real attempt lifecycle. The records sit in
docs/evidence/e-claude-1/ and e-codex-1/; local paths are replaced, and
Claude's init and hook events are reduced so that no tool, MCP server or
plugin name is published.

Both cards mark as verified: headless structured output, session resume, a
read-only mode (Claude's default permission mode, Codex's OS-enforced
read-only sandbox) and cancel by SIGINT to the process group. Codex's
workspace-write sandbox refused a write outside the workspace.

Both run tool shells in their own process groups. After SIGKILL of the
worker, those shells survived and kept writing while the launch group was
already empty, so descendants_may_outlive is verified for both. The runner
held the attempt unknown until the operator stopped the survivor.

Neither CLI can enforce passdown's result schema: Claude's --json-schema
produced no structured output for it, and Codex's --output-schema was
rejected by the API. Claude exits 0 after SIGINT, so its exit code is
marked unsupported.

Operator gotchas are recorded in the cards: Claude's variadic
--allowedTools must use `=`, `codex exec` needs stdin closed, and
`codex exec resume` takes -c sandbox_mode instead of --sandbox.
docs/EXECUTOR_SETUP.md points at the shared runner and adds the
interrupt-after-first-write and KILL steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The design gains the S6 implementation notes: card lint in the helper, the
CLI versions that moved before measuring, the live F18 case on Claude Code
and Codex, the absence of native schema enforcement for passdown's result
schema, differing exit codes, and the operator lessons. CHANGELOG records
the slice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@vunm-dev vunm-dev left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings need adjustment before merging. Validation: the card validation tests, all 150 skill checks, and the bundled helper/schema synchronization check passed. I also verified that the shipped Claude card permits a read attempt with readonly:default to have claim: null, without launching an executor. GitHub CI is green; live executor measurements were not rerun. This is a COMMENT review because the authenticated account is the PR author.

exit_code_meaningful: unsupported
descendants_may_outlive: verified
cancel_signal_honored: verified
read_only_mode: verified

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not enable claim-free reads without an enforced read-only profile

The evidence only establishes denial of writes with a user settings file that grants nothing, while the documented read invocation still loads the user's settings, hooks, plugins and MCP servers. The card itself says permissive settings widen this purported read-only mode. Marking read_only_mode verified nevertheless makes the helper accept --mutation-guard readonly:default and create an attempt with claim: null (confirmed locally without launching a worker). On a machine with preapproved writing tools, that read worker can therefore write alongside another claimed writer. Measure and specify an invocation that actually removes write capability independently of inherited permissions, or keep this capability unverified so the read takes a writer claim. The Kiro card's acknowledged auto-approved MCP exception needs the same treatment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7ca5bb4.

  • Claude Code and kiro-cli now mark read_only_mode: unverified. Their cards say that a read attempt takes the writer claim, and why: the invocation inherits the user's permissions, hooks and MCP servers, and auto-approved MCP servers were not measured.
  • For Codex I measured an invocation that does not depend on inherited configuration: --sandbox read-only --ignore-user-config.
    • Run R: a shell write failed with operation not permitted, with no plugin or MCP activity.
    • Run R2: apply_patch was refused with "writing is blocked by read-only sandbox".
    • The Codex card keeps verified and names that invocation and the matching --mutation-guard. Transports are in docs/evidence/e-codex-1/runs/R*.transport.jsonl.
  • Found while fixing this: a card file named claude.md is CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as project instructions. The card is now claude-code, and validate --as card refuses card names that clash with agent instruction files.

Comment thread docs/evidence/executor-run.sh Outdated
Comment on lines +202 to +204
if [ "$(jq -r .pgroup_members <<<"$facts")" = 0 ] && [ -z "$survivors" ] && [ -z "${new_kiro// /}" ]; then
H observe "$id" stopped --rev "$(rev "$id")" --evidence owner-attested --attested-by "$operator" --exit "$code" >/dev/null
note "- stop: owner-attested by $operator (process group empty, no descendant and no new executor process alive)"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Wait for explicit confirmation before recording owner-attested

This branch automatically records owner-attested from empty process listings and then proceeds to inspection and acceptance; it never obtains an operator confirmation. Sampling the descendant tree every 0.3 seconds can miss a quickly detached/reparented process, and the executor-name scan does not establish that no detached tool process remains. This converts the absence of safe machine evidence into an attestation, contrary to dispatch steps 10–11, which require a person to confirm and prohibit attesting on their behalf. Leave the attempt unknown with its evidence saved, and require an explicit confirmation step before recording stopped and continuing. The same automatic branch exists in docs/evidence/e-kiro-1/run.sh; distinguish its historical measurements from protocol-compliant acceptance evidence.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7ca5bb4.

  • executor-run.sh now stops at unknown when there is no safe machine evidence. It saves the listings and a state file, and records owner-attested only after an explicit --finish <part> --attested-by <who>. Smoke-tested with one kiro-cli run through both phases.
  • The historical e-kiro-1/run.sh stays as it ran. Each evidence README gains a "How stops were attested" section: those records were attested automatically from 0.3 s process scans, which is not the confirmation dispatch steps 10–11 require, and their accepted verdicts are measurement artifacts, not protocol-compliant acceptance.

vunm-dev and others added 2 commits September 25, 2026 14:15
Review round 1 on #21, plus one finding made while fixing it:

- Claim-free reads need an enforced read-only mode. The Claude Code and
  kiro-cli read-only behavior was measured only against the operator's own
  inherited settings: Claude inherits the user's permissions, hooks and MCP
  servers, and kiro-cli's auto-approved MCP servers were not measured. Both
  cards now mark read_only_mode unverified, so a read attempt takes the
  writer claim. For Codex, two new runs (R, R2) measured `--sandbox
  read-only --ignore-user-config` refusing a shell write and an apply_patch
  with no plugin or MCP activity. Its card keeps verified and names that
  invocation.
- The measurement runner no longer attests by itself. executor-run.sh
  stops at unknown and records owner-attested only after `--finish <part>
  --attested-by <who>`. The three evidence READMEs say that their records
  were attested automatically, and that their accepted verdicts are
  measurement artifacts.
- The Claude Code card is now claude-code. A card file named claude.md is
  CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as
  project instructions while this change was being made. `validate --as
  card` now refuses a card named like an agent instruction file (claude,
  agents, gemini), and tests/attempt.sh covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The S6 notes gain the claude-code naming and review round 1 on #21:
claim-free reads only with an enforced read-only mode, and the runner's
explicit attestation. CHANGELOG follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@vunm-dev vunm-dev left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review: the Claude/Kiro read-only capability downgrade and the explicit --finish attestation step address the previous findings. One remaining claim-free path needs the same isolation treatment. Card validation, all 150 skill checks, and bundled helper/schema synchronization passed; GitHub CI is green. No live executor was launched during this review.

descendants_may_outlive: verified
cancel_signal_honored: verified
read_only_mode: verified
sandbox_confined_writes: verified

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Close the inherited-permissions gap for snapshot+sandbox reads too

The read-only invocation now requires --ignore-user-config, but sandbox_confined_writes remains verified and the documented workspace-write invocation does not require that isolation. Dispatch also permits claim-free reads via --mutation-guard snapshot+sandbox when this capability is verified. I confirmed that the helper still creates claim: null for this guard, without launching an executor. As the card now explicitly notes, inherited MCP servers/plugins operate outside the sandbox; running the worker in a snapshot does not prevent those tools from writing to the original repository alongside its claimed writer. Mark sandbox_confined_writes unverified until an isolated profile is measured, or measure and explicitly require the isolated invocation for snapshot+sandbox as well, with coverage for this second claim-free path.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c146016: the Codex card now marks sandbox_confined_writes: unverified, so a snapshot+sandbox read on Codex takes the writer claim.

  • W refused a shell write outside the workspace, but it ran with the user's MCP servers and plugins loaded, which act outside the sandbox. apply_patch outside the workspace and writes to /tmp were not measured either.
  • Rather than measure an isolated workspace-write profile now, v0.5 ships one claim-free read path only: Codex's measured read-only invocation, --sandbox read-only --ignore-user-config (runs R and R2).
  • The card, the E-CODEX-1 README, the S6 design notes and CHANGELOG say so. The PR description's "writes confined" row is superseded by this.

Review round 2 on #21: the Codex card still marked sandbox_confined_writes
verified, which lets `--mutation-guard snapshot+sandbox` create a
claim-free read. The measuring run (W) refused a shell write outside the
workspace, but it ran with the user's MCP servers and plugins loaded, and
those act outside the sandbox; apply_patch outside the workspace and
writes to /tmp were not measured. The capability is now unverified. The
only claim-free read path shipped in v0.5 is Codex's measured read-only
invocation (`--sandbox read-only --ignore-user-config`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vunm-dev
vunm-dev merged commit e92db66 into release/v0.5.0 Sep 25, 2026
7 checks passed
vunm-dev added a commit that referenced this pull request Sep 25, 2026
Review round 1 on #21, plus one finding made while fixing it:

- Claim-free reads need an enforced read-only mode. The Claude Code and
  kiro-cli read-only behavior was measured only against the operator's own
  inherited settings: Claude inherits the user's permissions, hooks and MCP
  servers, and kiro-cli's auto-approved MCP servers were not measured. Both
  cards now mark read_only_mode unverified, so a read attempt takes the
  writer claim. For Codex, two new runs (R, R2) measured `--sandbox
  read-only --ignore-user-config` refusing a shell write and an apply_patch
  with no plugin or MCP activity. Its card keeps verified and names that
  invocation.
- The measurement runner no longer attests by itself. executor-run.sh
  stops at unknown and records owner-attested only after `--finish <part>
  --attested-by <who>`. The three evidence READMEs say that their records
  were attested automatically, and that their accepted verdicts are
  measurement artifacts.
- The Claude Code card is now claude-code. A card file named claude.md is
  CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as
  project instructions while this change was being made. `validate --as
  card` now refuses a card named like an agent instruction file (claude,
  agents, gemini), and tests/attempt.sh covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vunm-dev added a commit that referenced this pull request Sep 25, 2026
The S6 notes gain the claude-code naming and review round 1 on #21:
claim-free reads only with an enforced read-only mode, and the runner's
explicit attestation. CHANGELOG follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vunm-dev
vunm-dev deleted the feat/pdn-0004-s6-cards branch September 25, 2026 07:46
vunm-dev added a commit that referenced this pull request Sep 30, 2026
Review round 1 on #21, plus one finding made while fixing it:

- Claim-free reads need an enforced read-only mode. The Claude Code and
  kiro-cli read-only behavior was measured only against the operator's own
  inherited settings: Claude inherits the user's permissions, hooks and MCP
  servers, and kiro-cli's auto-approved MCP servers were not measured. Both
  cards now mark read_only_mode unverified, so a read attempt takes the
  writer claim. For Codex, two new runs (R, R2) measured `--sandbox
  read-only --ignore-user-config` refusing a shell write and an apply_patch
  with no plugin or MCP activity. Its card keeps verified and names that
  invocation.
- The measurement runner no longer attests by itself. executor-run.sh
  stops at unknown and records owner-attested only after `--finish <part>
  --attested-by <who>`. The three evidence READMEs say that their records
  were attested automatically, and that their accepted verdicts are
  measurement artifacts.
- The Claude Code card is now claude-code. A card file named claude.md is
  CLAUDE.md on case-insensitive file systems, and Claude Code loaded it as
  project instructions while this change was being made. `validate --as
  card` now refuses a card named like an agent instruction file (claude,
  agents, gemini), and tests/attempt.sh covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vunm-dev added a commit that referenced this pull request Sep 30, 2026
The S6 notes gain the claude-code naming and review round 1 on #21:
claim-free reads only with an enforced read-only mode, and the runner's
explicit attestation. CHANGELOG follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vunm-dev added a commit that referenced this pull request Sep 30, 2026
Review round 2 on #21: the Codex card still marked sandbox_confined_writes
verified, which lets `--mutation-guard snapshot+sandbox` create a
claim-free read. The measuring run (W) refused a shell write outside the
workspace, but it ran with the user's MCP servers and plugins loaded, and
those act outside the sandbox; apply_patch outside the workspace and
writes to /tmp were not measured. The capability is now unverified. The
only claim-free read path shipped in v0.5 is Codex's measured read-only
invocation (`--sandbox read-only --ignore-user-config`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant