feat: measured executor cards for kiro-cli, claude and codex

CHANGELOG.md

@@ -106,6 +106,32 @@ Beta build for the `release/v0.5.0` testing window — not a GitHub release.
     guessed around. The example workspace gains a handoff with an open
     attempt, and `docs/SMOKE_TEST.md` covers both skills against a live
     attempt.
+- Executor cards (PDN-0004, slice S6):
+  - `passdown-attempt validate --file <card> --as card` checks a card's
+    format, the evidence behind every measured value, and what a launch
+    needs. `tests/skills.sh` runs it on every shipped card.
+  - Measured cards for `kiro-cli` 2.24.0, Claude Code 2.1.193 (executor
+    and card name `claude-code`: a `claude.md` file is `CLAUDE.md` on
+    case-insensitive file systems) and codex-cli 0.157.0. Each is based on runs through the real attempt
+    lifecycle, recorded in `docs/evidence/e-kiro-1/`, `e-claude-1/` and
+    `e-codex-1/` with a shared fixture and runner. All three support
+    headless structured output, session resume and a read-only mode, and
+    end their tree on `SIGINT` to the process group. None can enforce
+    passdown's result schema natively. Claude Code and Codex run tool
+    shells in separate process groups that survive `SIGKILL` and keep
+    writing, so an empty process group is never a safe stop for them;
+    every attempt on the three executors ends with owner attestation
+    unless it ran in a containment scope. Only Codex has a measured
+    read-only mode that does not depend on inherited configuration
+    (`--sandbox read-only --ignore-user-config`), so only Codex reads can
+    skip the writer claim, and only through that read-only mode: no card
+    marks `sandbox_confined_writes` verified.
+  - The measurements found that a worker follows a hand-written
+    description of the result schema exactly, mistakes included. The
+    dispatch skill now puts the schema itself into the prompt.
+  - `docs/EXECUTOR_SETUP.md` is now the guide to measuring and writing a
+    card. The executor README drops the invocation hints now covered by
+    cards; only `agy` remains, as an unmeasured note.
 - The reference host (`tests/harness/ref-host`) prints the skill's step names,
   and a new harness scenario (`tests/interrupt.sh STEPS`) checks that the
   host performs the skill's numbered steps by name and in order. The reference

README.md

@@ -233,7 +233,8 @@ scripts/validate-plugin.ps1  # same, for Windows PowerShell (Git Bash/WSL path i
 examples/basic-workspace/  # a worked example: inbox note, OpenSpec change, session log
 docs/SMOKE_TEST.md         # manual verification checklist for install + skills
 docs/INTEGRATIONS.md       # standalone, OpenSpec, and Superpowers workflows
-docs/EXECUTOR_SETUP.md     # pre-flight checklist before adding a dispatch executor
+docs/EXECUTOR_SETUP.md     # measuring and writing an executor card
+docs/evidence/             # recorded measurements behind the shipped executor cards
 docs/GITHUB_SETTINGS.md    # required merge methods and main ruleset
 ```
 

docs/EXECUTOR_SETUP.md

@@ -1,57 +1,113 @@
-# Executor setup
+# Executor setup: measuring and writing a card
 
-Pre-flight checklist for adding an executor to the `executors` list in a
-workspace's `## passdown` section. The dispatch skill routes tasks to these
-executors at runtime, but it **cannot reconfigure them mid-dispatch** —
-permission gates and sandbox policy are user setup work, done here, once,
-before the executor earns its place in the list.
+An executor becomes eligible for delegated dispatch when it has an **executor
+card**: a small Markdown file with YAML frontmatter that says how to launch it
+headless, where its output lands, how to extract the `passdown.result/v1`
+payload, how to cancel it, and which of its capabilities were measured. The
+dispatch skill keeps the policy; the card keeps these volatile mechanics. The
+format is section 18 of the v0.5 design
+(`docs/design/PDN-0004-v05-acceptance-recovery.md`), summarized in
+`plugins/passdown/skills/passdown-dispatch/references/executors/README.md`.
 
-An executor that fails these checks doesn't have to be removed — but its
-limits must be recorded as `executor notes` in the workspace AGENTS.md so
-routing can veto incompatible tasks.
+Cards live in two places. passdown ships measured cards in
+`plugins/passdown/skills/passdown-dispatch/references/executors/`. A
+workspace can add its own, or replace a shipped one, in the directory its
+`## passdown` section names as `executor_refs`. A workspace card replaces
+the shipped card of the same name as a whole file, so it must be complete.
 
 First identify the host. An executor name is a target, not the current host:
-when passdown already runs in Codex, `codex` is a self-target and must be
-skipped. Use `main` or an explicitly authorized native `subagent` instead.
+when passdown already runs in Codex, `codex` is a self-target and is skipped.
 
-## 1. Non-interactive run completes
+## 1. Pre-flight: the executor can run unattended
 
-Non-interactive modes (`--print`, `exec`, app-server) have nobody to answer
-permission prompts. A prompt the CLI cannot answer becomes a silent
-hang-to-timeout, not an error.
+Non-interactive modes (`--print`, `exec`, `--no-interactive`, stream output)
+have nobody to answer permission prompts. Depending on the CLI, an
+unanswerable prompt becomes a silent hang or an immediate denial.
 
-- [ ] Run a trivial prompt end to end, e.g. "run `echo ok` and report the
-      output" — it returns within seconds, no hang
-- [ ] If it hangs: configure the executor's own auto-approval mechanism
-      (allowlist, hook, or policy file — never a blanket
-      skip-all-permissions flag), then re-test
+- [ ] Run a trivial prompt end to end ("reply with exactly the word: pong")
+      and note how long it takes and what the exit code is
+- [ ] Find the executor's own permission mechanism (a trust list, an
+      allowlist, a policy file). Never use a blanket skip-all-permissions
+      flag as the default
+- [ ] Run one representative toolchain command through the executor
+      (`<tool> --version` is not enough: pick one that builds or tests).
+      Toolchain wrappers that write to their own install dir, like Flutter's
+      SDK cache, fail inside a write-restricted sandbox
+- [ ] If the work needs package fetches, decide on network access
+      explicitly, with its prompt-injection and exfiltration trade-off
+- [ ] For cross-repo work, check that every target repository is inside the
+      executor's writable roots
 
-## 2. Sandbox scope matches the work
+## 2. Measure
 
-Sandboxed executors typically restrict writes to the workspace and block
-the network. Toolchain wrappers are the classic trap: they write to their
-own install dir on every invocation (Flutter's SDK cache is a known
-example), so *any* command through them fails — or worse, sends the agent
-off improvising workarounds.
+Measure through the real attempt lifecycle, in a disposable fixture
+repository, never in a repository you care about. `docs/evidence/fixture.sh`
+builds one; `docs/evidence/executor-run.sh` drives one run through `new`,
+`arm`, the launch, `observe`, `probe`, `result`, `inspect` and a verdict, and
+records the prompt, transport, receipt and the process tree seen while it
+ran. It knows `kiro-cli`, `claude` (card `claude-code`) and `codex` (`EK_EXECUTOR`); for another
+executor, add its launch line, session field and result extraction. The
+shipped cards' records (`docs/evidence/e-kiro-1/`, `e-claude-1/`,
+`e-codex-1/`) are worked examples, including the runs that went wrong.
 
-- [ ] Run one representative toolchain command through the executor
-      (`<tool> --version` is not enough — pick one that builds or tests)
-- [ ] If it fails on writes outside the repo: extend the sandbox's writable
-      roots (e.g. Codex `~/.codex/config.toml` →
-      `[sandbox_workspace_write] writable_roots`)
-- [ ] If the work needs package fetches or dependency resolution: enable
-      sandbox network access, and note the prompt-injection/exfiltration
-      trade-off that comes with it
-- [ ] For cross-repo intake/dispatch, verify every inbox and target repo is
-      inside writable workspace roots. If not, open the parent workspace or
-      grant the exact additional root before dispatching
-
-## 3. Record the outcome
-
-- [ ] Add what passed *and* what failed (with root cause) to
-      `executor notes` in the workspace AGENTS.md
-- [ ] Tasks that the executor cannot run (network, out-of-repo writes,
-      GUI/toolchain builds) are routed elsewhere — the notes are the veto
-      list the dispatch skill reads before routing
-- [ ] Never record a workaround that redirects HOME, creates substitute
-      caches, or edits project configuration to evade the sandbox
+Use a plan with four tasks: a read-only task, a bounded write to one named
+file, a task that is deliberately missing a decision, and a long-running
+task that writes slowly (a loop with `sleep 1`). Then:
+
+| Run | What to do | Capabilities it decides |
+|---|---|---|
+| A | Read-only task, output in the executor's machine-readable format | `headless`, `structured_output`, `exit_code_meaningful`, the `result_extraction` rule; with no trust flags, half of `read_only_mode` |
+| B | Bounded write, with the narrowest permission that allows it; also once with none | the `permissions` mechanism; scope behavior; the other half of `read_only_mode` (writes denied without trust) |
+| C | The ambiguous task; the prompt says not to guess | whether the executor returns `needs_input` without writing |
+| D | Answer C's question by resuming its session, with a prompt that leaves the task text out | `session_id_observable`, `resume_session`, `invocation.resume` |
+| E | The long task, interrupted with the cancel signal to the process group, then separately with `KILL`, both a few seconds after the task starts writing (`--interrupt SIG@file:<path>+<seconds>`); record the process tree during the run and after the exit, and whether the file keeps growing | `cancel_signal_honored`, `descendants_may_outlive`, `stop.containment`, `settle_seconds`, `discovery_hint` |
+
+Check `--help` of the installed version for an option that enforces an
+output schema (`native_schema_enforcement`), and record the CLI version with
+every run.
+
+Put the result schema itself in every prompt (or pass it through the
+executor's schema flag). E-KIRO-1 found that a worker follows a hand-written
+description of the schema exactly, including its mistakes.
+
+## 3. Write the card
+
+- `verified` only when a run of the recorded version produced the evidence;
+  `unsupported` only for a measured negative; everything else `unverified`.
+- `descendants_may_outlive: unsupported` makes a plain process-group-empty
+  exit a safe stop. Mark it only when no process started for an attempt can
+  leave the process group. An executor with a shell tool can start a
+  detached process on request, so it normally stays `unverified`, and its
+  attempts end with owner attestation or a containment scope. Measure
+  `KILL` as well as the cancel signal: Claude Code and Codex run their tool
+  shells in separate process groups that end on `SIGINT` but survive
+  `SIGKILL` and keep writing, which makes the capability `verified`.
+- With `descendants_may_outlive` not `unsupported`, `stop.settle_seconds`
+  must be above zero.
+- A card is eligible to launch only with `invocation.headless`,
+  `invocation.output_capture` and `invocation.result_extraction`.
+- Point `measured.evidence` at the recorded runs.
+
+Then check it:
+
+```bash
+scripts/passdown-attempt validate --file <card>.md --as card
+```
+
+It refuses unknown keys and capability values, a `verified` or `unsupported`
+value without evidence, and a card that lacks what a launch needs. For a card
+shipped with passdown, `tests/skills.sh` runs the same check and requires the
+evidence to exist in the repository.
+
+When the installed CLI version moves past the card's `cli_version`, the card
+is stale: hosts may still use it and flag the mismatch, but no capability is
+upgraded by assumption. Re-measure what the release notes touch.
+
+## 4. Record environment limits as executor notes
+
+Some limits are about the machine, not the executor: sandbox write scope,
+network, a toolchain that writes outside the repository. Keep those as
+`executor notes` in the workspace `AGENTS.md`; dispatch reads them as routing
+vetoes. Record what failed and why. Never record a workaround that redirects
+`HOME`, creates substitute caches, or edits project configuration to get past
+a sandbox.

docs/design/PDN-0004-v05-acceptance-recovery.md

@@ -2075,6 +2075,77 @@ ships it.
   new delegated attempts (S4 eligibility rule); the helper still reads a
   missing card as all-`unverified`, so existing receipts stay readable.
 
+**Implementation notes (S6).**
+
+- **Card lint lives in the helper.** `validate --file <card> --as card`
+  applies the §18 rules and the S4 eligibility rule with the helper's own
+  card parser. Hosts, `tests/skills.sh` and the attempt snapshot therefore
+  read a card the same way.
+- **Versions moved.** kiro-cli had gone from 2.22.1 (this design) to
+  2.24.0 by the time E-KIRO-1 ran. codex-cli 0.142.5 could no longer run its
+  default model, and the owner updated it to 0.157.0 before E-CODEX-1. The
+  `claude` CLI needed a login of its own; the desktop session's
+  authentication does not carry over. Each card records the version that
+  was measured.
+- **Claude and Codex were measured too** (E-CLAUDE-1, E-CODEX-1) with the
+  E-KIRO-1 method, using a shared fixture (`docs/evidence/fixture.sh`) and
+  runner (`docs/evidence/executor-run.sh`). §26.2 asked only that their
+  cards "list only what was measured"; measuring all three with the same
+  runs made the cards comparable.
+- **The §8.3 safe-stop rule was exercised on real CLIs.** Claude Code and
+  Codex run tool shells in their own process groups. After `SIGKILL` of the
+  worker those shells survived and kept writing while the launch process
+  group was already empty: a live case of F18. The runner treated it by the
+  protocol: it did not attest, the attempt stayed `unknown`, and the claim
+  stayed held until the operator stopped the survivor. Their cards mark
+  `descendants_may_outlive: verified`. kiro-cli kept every process in the
+  launch group, but a trusted shell could detach one, so it stays
+  `unverified`. On macOS, every attempt on the three executors ends with
+  owner attestation unless it ran in a containment scope.
+- **No executor enforces passdown's result schema.** kiro-cli has no schema
+  option. Claude's `--json-schema` produced no structured output for it.
+  Codex's `--output-schema` was rejected by the API, which accepts only a
+  stricter JSON Schema subset. The first E-KIRO-1 write run also showed a
+  worker copying a hand-written schema description, mistakes included. The
+  dispatch skill (step 4) therefore puts the schema itself into the prompt.
+- **Exit codes differ.** Claude exits 0 after `SIGINT`; kiro-cli and Codex
+  exit 1. The Claude card marks `exit_code_meaningful: unsupported`, and all
+  three cards say to judge a run by its final event and the host checks.
+- **The Claude card is `claude-code`, not `claude`.** A card file named
+  `claude.md` is the same file as `CLAUDE.md` on case-insensitive file
+  systems (macOS and Windows defaults), so Claude Code loads it as project
+  instructions for anyone working in that directory. §18 names the card
+  `claude`; the shipped name is `claude-code`, which is also the product's
+  name. Configure the executor as `claude-code`.
+- **Review round 1 on [#21](https://github.com/vunm-io/passdown/pull/21).**
+  - **Claim-free reads need an enforced read-only mode.** The Claude and
+    kiro-cli read-only behavior was measured only against the operator's
+    own inherited settings. The Claude invocation inherits the user's
+    permissions, hooks and MCP servers, and kiro-cli's auto-approved MCP
+    servers were not measured. Both cards therefore mark `read_only_mode`
+    unverified, so a read attempt takes the writer claim. Codex has an
+    invocation that does not depend on inherited configuration: `--sandbox
+    read-only --ignore-user-config`. Two runs (R, R2) measured it refusing a
+    shell write and an `apply_patch`, with no plugin or MCP activity, so its
+    card keeps `verified` and names that invocation.
+  - **The same goes for `snapshot+sandbox` reads** (review round 2). Codex's
+    workspace-write run W refused a shell write outside the workspace, but
+    it ran with the user's MCP servers and plugins loaded, and those act
+    outside the sandbox. So `sandbox_confined_writes` is `unverified` for
+    Codex as well, and the only claim-free read path shipped in v0.5 is
+    Codex's measured read-only invocation.
+  - **The measurement runner no longer attests by itself.**
+    `docs/evidence/executor-run.sh` stops at `unknown` and records
+    `owner-attested` only after `--finish <part> --attested-by <who>`. The
+    records produced before this change say so, and their `accepted`
+    verdicts are measurement artifacts.
+- **Operator lessons** now in `docs/EXECUTOR_SETUP.md`:
+  - interrupt a few seconds after the task starts writing, not at a fixed
+    time (Codex took about 25 s to start);
+  - close stdin for `codex exec`;
+  - pass Claude's variadic `--allowedTools` with `=`;
+  - measure `KILL` as well as the cancel signal.
+
 **S7 — Real-host evidence, docs, migration, release.**
 - Files: `docs/evidence/v0.5.0/`, README, `docs/INTEGRATIONS.md`,
   `docs/SMOKE_TEST.md`, CHANGELOG `[0.5.0]`, VERSION + manifests.