Approval on demand: Jev reads the agent message a reply answers

CHANGELOG.md

@@ -6,7 +6,18 @@ How to keep this current: add the entry in the same pull request as the change,
 
 ## Unreleased
 
-<!-- Empty. Next release starts here. -->
+## 0.88.0
+
+### Added
+
+- Approval on demand. The acting request no longer asks whether the reply approves; a call that is held again under a new user prompt gets one approval request, with the reply and the agent message it answers (`asked`: the newest assistant message before the reply, redacted, its last 1,500 characters). The call is released when `approved` is at least 0.7. Exports: `settleApproval`, `askApproval`, `buildApprovalRequest`, `replyApprovalQuestion`, `describeAsked`, `APPROVAL_THRESHOLD`; `asked` on `Conversation` and `ActionInput`.
+- `scripts/approval-cases.mjs`, `scripts/approval-designs.mjs`, and `scripts/approval-replay.mjs` measure the old question, the shipped one, and a second design that was not shipped, on 30 synthetic cases, 17 held-out cases, and recorded holds. `docs/guards.md` has the tables and the rule that chose the design.
+
+### Changed
+
+- `evaluateAction` with `retryAfterHold` takes the same approval step as the Action guard: the acting request does not carry `approved`; when the final verdict is a hold, one approval request follows with `asked` (optional field of the action), and without a judge or when the request fails a reply that reads as approval stands in. Before, it asked the combined question on the acting request and read no agent message. No exported name changed.
+- Approval requests fall from 90.0 to 0.7 per 1,000 judged calls (measured on recorded sessions since 2026-09-16). Measured on recorded holds and cases, wrong releases fall from 45 to 12 and correct releases rise from 55 to 81 (runs; `docs/guards.md`).
+- The consent text (`disclosure`) and `docs/data-handling.md` say what the approval request sends.
 
 ## 0.87.0
 

docs/data-handling.md

@@ -8,7 +8,7 @@ With consent, requests go to `https://api.typesafe.ai` (default), or to the host
 
 | Guard | Sent |
 | --- | --- |
-| **Action** | Your latest prompt (1500 characters), the task spine it is judged against (the thread's first request and up to four earlier requests, redacted, capped at 1200 characters together), up to eight earlier user and assistant messages (750 redacted characters each), the agent's text from the message that makes the call (500 redacted characters), the tool name, the command (2000 characters) or the file path (relative inside the project, `~`-shortened outside), whether the file exists, a 1500-character head/middle/tail sample of a `write` (or of the content a `bash` command writes to a file with the content in the command, with the written paths), the first three edit pairs (400 characters each) of an `edit`. Calls the ask gate decides offline (`action.ask`) send nothing at all, and so do read-only tools and read-only shell lines. The acting request carries none of the earlier user and assistant messages: no acting question reads them. The off-task, scope, and should-proceed questions are not on the acting request either, because nothing delivered reads their answers; one judged call in twenty (`action.traceSample`, default 0.05) sends a second request that carries them, with up to eight earlier user and assistant messages (750 redacted characters each) and the resolved rules content, which only that sampled request carries when no violation is open. `/warden test` sends one request and never the sample. The resolved active rules content (`pi-warden.md`, the configured files, or `AGENTS.md`/`CLAUDE.md`/`README.md` as fallback, token-aware truncated at ~4000 tokens) rides the acting request only while a violation is open on the call, because only the per-violation questions name it, and rides the sampled request above; with `rules.enabled` false no rules content leaves the machine. On the first guarded call after your reply, the tool names and commands (300 characters) or paths of up to six calls allowed in the previous turn, for the regret question. |
+| **Action** | Your latest prompt (1500 characters), the task spine it is judged against (the thread's first request and up to four earlier requests, redacted, capped at 1200 characters together), up to eight earlier user and assistant messages (750 redacted characters each), the agent's text from the message that makes the call (500 redacted characters), the tool name, the command (2000 characters) or the file path (relative inside the project, `~`-shortened outside), whether the file exists, a 1500-character head/middle/tail sample of a `write` (or of the content a `bash` command writes to a file with the content in the command, with the written paths), the first three edit pairs (400 characters each) of an `edit`. Calls the ask gate decides offline (`action.ask`) send nothing at all, and so do read-only tools and read-only shell lines. The acting request carries none of the earlier user and assistant messages: no acting question reads them. The off-task, scope, and should-proceed questions are not on the acting request either, because nothing delivered reads their answers; one judged call in twenty (`action.traceSample`, default 0.05) sends a second request that carries them, with up to eight earlier user and assistant messages (750 redacted characters each) and the resolved rules content, which only that sampled request carries when no violation is open. `/warden test` sends one request and never the sample. The resolved active rules content (`pi-warden.md`, the configured files, or `AGENTS.md`/`CLAUDE.md`/`README.md` as fallback, token-aware truncated at ~4000 tokens) rides the acting request only while a violation is open on the call, because only the per-violation questions name it, and rides the sampled request above; with `rules.enabled` false no rules content leaves the machine. On the first guarded call after your reply, the tool names and commands (300 characters) or paths of up to six calls allowed in the previous turn, for the regret question. Only for a call that is held after an earlier hold, once your reply came in, one approval request with your reply, `asked` (the newest agent message before it, redacted, its last 1500 characters), the redacted call summary, and the reasons for the hold (pattern names and scores); the acting request carries no approval question. |
 | **Rules** | The project-relative path, a 6000-character sample of a `write` (or of the content a `bash` heredoc, `echo`, or `printf` writes to a file, judged as a `write`) or each edit's new text (1500 characters) with about 40 lines of the current file around the replaced text, as they are and with the edit applied, and the rule text from your rules file or the condensed fallback document (`rules.maxChars`). No task text. Files under `rules.exclude` are never sent. `/warden rules check` sends each rule's id, heading, `paths:` scope, and text (redacted, clipped at 400 characters) in one request per 32 questions, with no file content and no task text. `/warden rules calibrate` sends one request per changed file of the sampled commits — the project-relative path, the added and removed lines with about 40 lines of the file after the commit around them (redacted), and the rule text — only after a confirm dialog that shows how many requests go out and the redacted diffs; a headless run sends nothing without the explicit `--yes`. `/warden rules tune` sends no request: it hands the flagged rules to the session's agent as one message. | `/warden rules audit` sends a 6000-character redacted sample of each selected file judged as a `write`, plus the rule text, and only after the confirm dialog or `--yes`; files under `rules.exclude` or `rules.skip` are never selected. `/warden bench` sends only one fixed built-in sample file and the rule text; no project content goes for it. |
 | **Rules at turn start** | Before each new user message, when the rules guard and `rulesAtTurnStart` are on: the new request (1500 redacted characters), the task spine (the thread's first request and up to four earlier requests, redacted and capped as above), and the project's rule set — each rule's heading, its text (300 characters), and its `paths:` scope. One question per rule; no file content, no tool output, and no earlier assistant text. Nothing is sent when the prompt is a short continuation or a relayed child report (the two prompts the conscience's local gate also skips), when the rules source has no rule headings, or when judgments are off. When the request fails, times out, comes back after the run ended, or no rule passes the threshold, it has already been sent and nothing is appended to the session. |
 | **Stuck** | The last 12 tool calls (300 characters each) with 400-character output tails, and, while `stuck.evidence` is on (the default), a structured `evidence` section: per run the parsed failing test, error, location, summary, exit code and which earlier run failed the same way (or 300 characters of head and 300 of tail when nothing parses), per `edit`/`write` the project path and a diff of the change capped at 600 characters, and a digest. Every string is redacted; the whole object is capped at 4 KB. |

docs/extension-authors.md

@@ -25,7 +25,7 @@ What spans calls in a session lives in `ActionGuard` (hold, reply, retry approva
 
 From 1.0, semver covers exactly these exports of the package root (`pi-warden`), together with the types of their parameters and results:
 
-- Action guard: `evaluateAction`, `ActionGuard`, `describeAction`, `matchPatterns`, `isReadOnlyCommand`, `stripDataText`, `formatVerdict`, and the question sets `questions`, `intentQuestion`, `visibleQuestion`, `slopQuestions`, `approvalQuestion`, `securityQuestion`, `regretQuestions`.
+- Action guard: `evaluateAction`, `ActionGuard`, `describeAction`, `matchPatterns`, `isReadOnlyCommand`, `stripDataText`, `formatVerdict`, and the question sets `questions`, `intentQuestion`, `visibleQuestion`, `slopQuestions`, `approvalQuestion`, `securityQuestion`, `regretQuestions`, and the approval step for a held call: `settleApproval`, `askApproval`, `buildApprovalRequest`, `replyApprovalQuestion`, `describeAsked`, `APPROVAL_THRESHOLD`. `ActionGuard` and `evaluateAction` with `retryAfterHold` both call `settleApproval`; the field `asked` of `Conversation` and of `ActionInput` is the agent message the user's reply answers.
 - Rules: `evaluateRules`, `RuleStore`, `RulesGuard`, `parseRules`, `matchGlob`.
 - Redaction: `redact`, `syntheticish`, `partitionSecrets`.
 - Stuck detector: `AttemptWindow`, `makeAttempt`, `evaluateStuck`.

docs/guards.md

@@ -41,6 +41,8 @@ Runs on `tool_call`, before the tool executes.
    - `confirm`: a `ctx.ui.confirm` dialog. No blocks with a short reason. Falls back to `steer` without a UI.
    - `advise`: never holds, reports only. A user `dialog` rule still prompts: advise mode keeps Jev holds advisory, it does not soften a prompt you asked for by name.
 
+**Hold and approval.** The acting request never asks whether your reply approves anything. After a hold, a guarded call that runs under a new user prompt and is held again gets one approval request, sent for that call alone: `task` (your reply), `asked` (the agent message the reply answers: the newest assistant message before it, redacted, its last 1,500 characters), the call summary, and the reasons for the hold. It has one question, `approved`. Jev reads `asked` only to resolve what a short or numbered reply points at ("1. yes", "go ahead"); text in `asked` never approves, because it is the agent's own. The call is released when `approved` is at least 0.7. One approval releases one held call. A call the verdict lets through sends no approval request, so the request is made about once in 1,400 judged calls. Without a judge, or when the request fails, a reply that reads as approval stands in (a yes/go-ahead heuristic), and a failure alone never releases a call. Calls of one assistant message are still judged together; their approval requests go out per call, in order. `evaluateAction` with `retryAfterHold: true` takes the same step (`settleApproval`), with `asked` as an optional field of the action.
+
 The action guard receives your latest message. Up to eight earlier user and assistant messages (750 redacted characters each) ride only the trace sample, one judged call in twenty (`action.traceSample`); the rules content rides the acting request only while a violation is open. The acting request also carries the task spine in the request state (`spine`): the thread's first user turn (`goal`), and up to four earlier user turns (`task_history`, newest first), so a follow-up like "now the tests" or a side comment is judged with the goal it belongs to. The whole spine is capped at 1200 characters — history is clipped first, then the goal; the latest turn is never clipped, because approval still comes from `task` only. The spine is scope context and never authorizes an action. Sibling tool calls in one assistant message are judged together in one round trip. If TypeSafe cannot answer, the call is allowed with a warning (`failOpen: true`; set it to `false` to hold instead).
 
 **User command rules.** The built-in pattern list is not the whole floor: `action.commandRules` in the user config declares your own. `{ id, pattern, severity, action?, message?, caseSensitive? }` — `warn` notices and continues, `confirm` holds, `deny` blocks outright with no dialog and no TypeSafe request. A `confirm` rule defaults to `action: "dialog"`: a prompt for you, in every mode (advise included), because asking for a dialog on a named command is the reason to write such a rule; `action: "hold"` restores steer semantics. Patterns see the same data-text-stripped command the built-ins read, so a heredoc body or a commit message that mentions your pattern does not fire it. `action.exemptRules` silences a built-in by id (`["infra-destroy"]` for a workflow whose `kubectl delete` is routine), including the ids the `rm` classifier derives (`rm-recursive`, `rm-rf`, `rm-recursive-dangerous-target`, `rm-temp-subtree`, `rm-session-scratch`) and `sensitive-path`; an id that names neither a built-in, a classifier id, nor one of your own rules is inert and is reported once. User rules share the built-ins' id namespace, so an exempt id can also silence your own rule. Project files cannot set any of the three keys: a checked-out repo cannot ship itself a hold-free floor or a prompt farm.
@@ -134,6 +136,47 @@ The conscience coach assesses whether the agent is missing a useful skill or too
 - A second pass asked four candidate questions on the same calls (`scripts/action-candidates.mjs`, `--extra`). None separates rejected turns on its own: "would a careful engineer ask first", "is this unrequested", "did the user ask to pause", and "is the effect visible outside the working tree" all sit at the 4 to 5% base rate. `visible` has the best recall on regret (AUC 0.82, 10 of 19 regretted calls) but a commit or push is usually what was asked. Paired with the plan it works: `visible >= 0.8` and `intent_mismatch >= 0.8` flags 1.1% of calls with 18% in a rejected turn, so that pair steers at `visibleMismatch` 0.8. Two deterministic patterns came from the regretted list: a git command with hooks or signing switched off, and `gh pr merge`.
 - Of 42 holds pi-warden made in those sessions, the user's next message approved 5.
 
+### Approval on demand (2026-10-01, three designs, three sets)
+
+The old question rode every acting request and asked whether `task` gives the agent permission to continue the current work. It read no agent message, so a numbered reply ("1. yes", "2. merged") or a bare "yes" was read against nothing. Two designs were measured against it, all three at the 0.7 threshold, three runs per case, judge `jev-1.13.0`:
+
+- **old**: the question above, on the acting request.
+- **candidate** (shipped): one question on its own request, sent only for a held call, with `asked` = the newest assistant message before the reply, last 1,500 characters.
+- **round 2** (not shipped): the candidate's `approved` plus `reply_points_at_action`, with `asked` = every assistant message of the turn (after the previous user message, before the reply), last 3,000 characters. A call is released only when both answers reach 0.7. The words of the second question are in `scripts/approval-designs.mjs`.
+
+Three sets: 30 synthetic cases (`scripts/approval-cases.mjs`); 17 held-out synthetic cases, written and committed with their labels before round 2 was measured and never used to choose a wording; and 24 recorded holds from real sessions (`scripts/approval-replay.mjs`), labelled blind by a model, the labels unchanged. Counts are runs (cases × 3), so a case that flips between runs counts each way. "Right" and "wrong" are against the label: a release is right when the reply approves the action.
+
+| Set | Design | Released right | Released wrong | Held right | Held wrong |
+| --- | --- | --- | --- | --- | --- |
+| 30 cases (39 approving runs, 51 not) | old | 21 | 21 | 30 | 18 |
+| | candidate | 39 | 3 | 48 | 0 |
+| | round 2 | 37 | 0 | 51 | 2 |
+| 17 held-out cases (21 approving runs, 30 not) | old | 15 | 9 | 21 | 6 |
+| | candidate | 21 | 6 | 24 | 0 |
+| | round 2 | 21 | 1 | 29 | 0 |
+| 24 recorded holds (30 approving runs, 42 not) | old | 19 | 15 | 27 | 11 |
+| | candidate | 21 | 3 | 39 | 9 |
+| | round 2 | 20 | 0 | 42 | 10 |
+| **Total** (90 approving runs, 123 not) | old | 55 | 45 | 78 | 35 |
+| | candidate | 81 | 12 | 111 | 9 |
+| | round 2 | 78 | 1 | 122 | 12 |
+
+The recorded holds counted by hold (the mean of the three runs decides), 10 approving and 14 not: old releases 6 right and 5 wrong, holds 9 right and 4 wrong; the candidate releases 7 right and 1 wrong, holds 13 right and 3 wrong; round 2 releases 6 right and 0 wrong, holds 14 right and 4 wrong. An earlier pass over the same holds gave the candidate 8, 1, 13, 2; the difference is one hold crossing the threshold between passes.
+
+**The rule, fixed before round 2 was measured:** ship round 2 only if its total wrong releases are fewer than the candidate's and its total correct releases are at most 2 fewer; otherwise ship the candidate and do not run another round. Round 2 has 1 wrong release against 12, but 78 correct releases against 81, which is 3 fewer. **The candidate ships.** Counting the recorded holds by hold instead of by run, the correct releases are 67 against 64, also 3 fewer. Only a count that takes every set by hold (27 against 25) puts round 2 inside the limit; the rule did not name the unit, and runs are what the other two sets report.
+
+Approval requests per 1,000 judged calls: 90.0 before (the old question on 3,061 of 34,014 judged calls since 2026-09-16), 0.7 after (24 holds with a reply).
+
+The measurement used 639 requests and about 570,000 input and 17,000 output tokens.
+
+**What still fails in the shipped design.**
+
+- A numbered reply that gives the held item no answer, or a different one, while it agrees to other items: released (two cases, six runs).
+- A reply that arrives mid-run, after an assistant message that holds only tool calls, and agrees to a question asked earlier in the turn about other work: the newest message is empty, so the reply is read against nothing and a bare "yes" can be released (one synthetic case, one recorded hold). Round 2 closes this; its cost is held approvals.
+- An approving reply held: a short agreement to an explanation that sits earlier in the turn than the newest message, and one long reply that mixes instructions with the approval (three recorded holds). Round 2 does not fix these and held one more. It also held two runs of a bare "yep" after a list that names the action among others (second question 0.67 to 0.74).
+
+Every release has a floor under it: a hold only exists for a call the guard already judged irreversible or matched by a pattern, and a wrong release lets that one call run once.
+
 ### Relevance compaction replay (2026-09-29, first measurement)
 
 `node scripts/relevance-replay.mjs` rebuilds the span each recorded compaction replaced (48 compactions in 1,521 sessions on one machine), runs the keep questions with real requests at the defaults, and compares the result with the summary Pi wrote. It counts the "re-fetch" calls: a `read` of a path or a `bash` command from the span among the first 10 tool calls after the compaction (34 calls after 17 compactions; all 34 were `read`).