Steer calibration with blind labels: off-task and should-proceed stay trace-only

CHANGELOG.md

@@ -4,6 +4,22 @@ Notable changes to pi-warden, newest first. Versions follow semver. The publishe
 
 How to keep this current: add the entry in the same pull request as the change, under `Unreleased`. The release commit renames `Unreleased` to the version it ships and adds its own notes. Entries before 0.10.0 are one-line summaries taken from the release commit headers; the detail for those is in `git log`.
 
+## Unreleased
+
+<!-- Empty. Next release starts here. -->
+
+## 0.82.1
+
+### Added
+
+- `scripts/steer-calibration.mjs`: the blind-label steer measurement (draw the sample from a copy of the hold log, score the labels, gate the slices). `scripts/steer-ask-probe.mjs` measures a candidate question on the same labels, one request per call.
+- `should_ask` in `scripts/action-candidates.mjs`: a reworded should-proceed question that asks whether any visible instruction covers the call. Recorded as a candidate only (never acted on); its measurement is blocked on the TypeSafe account balance.
+
+### Docs
+
+- The off-task and should-proceed steers are documented as trace-only, with what `action.offTask.warn`, `action.offTask.steer`, `action.shouldProceed.threshold` and `action.shouldProceed.steer` really change, and the 2026-09-30 blind-label measurement (no slice passes the ship gate) in `docs/guards.md` and `docs/configuration.md`.
+- `README.md` matches `docs/guards.md`: the features table describes the off-task and should-proceed notices as trace-only (the 2026-09-30 labels found 2 of 427 calls that needed asking, so they do not support `action.shouldProceed.steer`), and the versioning section puts `action.shouldProceed.steer` next to `compaction.*`, `context.filter.*`, and `conscience.*`, outside the semver promise. The guards docs also name who labelled those 427 calls: one language model, blind to the scores and strata, on the clipped request, plan, context, and call.
+
 ## 0.82.0
 
 ### Added

README.md

@@ -57,9 +57,9 @@ Every guard and feature, its default, and its status. Defaults are `defaultConfi
 | Jev judgments | Sends redacted samples to Jev for the judged checks below | Off until `/warden enable` | Stable |
 | [Action guard](docs/guards.md#action-guard) | Checks each command, write, and edit before it runs: offline patterns plus one Jev request | On | Stable |
 | ↳ Irreversible hold | Holds a call Jev scores 0.9 or more as irreversible; 0.5 to 0.9 warns | On | Stable |
-| ↳ Off-task steer | Warns at 0.6 and steers the agent back to your request at 0.85; never holds | On | Stable |
+| ↳ Off-task notice | Notes in the trace when a call looks unrelated to your request; never steers or holds | On, trace only | Stable |
 | ↳ Intent-mismatch steer | Tells the agent when a call differs from its stated plan; the score stays in the trace | Off: trace only. `action.intentTraceOnly: "invisible"` | Stable |
-| ↳ Should-proceed steer | Asks the agent to pause and ask you; the score stays in the trace | Off: trace only. `action.shouldProceed.steer: true` | Stable |
+| ↳ Should-proceed notice | Notes in the trace when the judge would ask you first. `action.shouldProceed.steer: true` also tells the agent; the 2026-09-30 labels do not support it (2 of 427 calls needed asking) | Off: trace only | Beta |
 | ↳ Your command, path, and arming rules | Warn, hold, or deny commands and paths you name | On, none set | Stable |
 | ↳ Large-output warning | Tells the agent to filter a command that will print far more than it needs | On | Stable |
 | [Rules](docs/guards.md#rules) | Judges every write and edit against the rules in your `pi-warden.md` | On | Stable |
@@ -131,7 +131,7 @@ Secrets and unshown paths are stripped before anything leaves your machine. The
 
 ## Versioning
 
-From 1.0, pi-warden follows semver for: the documented config keys and their defaults (except `compaction.*`, `context.filter.*`, and `conscience.*`, which are experimental or beta); the `/warden` commands and their arguments; the `warden_remember`, `warden_loops`, and `warden_recall` tools; the documented `PI_WARDEN_*` environment variables; the widget template tokens; and the exports named in [docs/extension-authors.md](docs/extension-authors.md). The trace file format, the files in pi-warden's data folder, session message types, and every other export are internal and may change in any release.
+From 1.0, pi-warden follows semver for: the documented config keys and their defaults (except `compaction.*`, `context.filter.*`, `conscience.*`, and `action.shouldProceed.steer`, which are experimental or beta); the `/warden` commands and their arguments; the `warden_remember`, `warden_loops`, and `warden_recall` tools; the documented `PI_WARDEN_*` environment variables; the widget template tokens; and the exports named in [docs/extension-authors.md](docs/extension-authors.md). The trace file format, the files in pi-warden's data folder, session message types, and every other export are internal and may change in any release.
 
 ## Docs
 

docs/configuration.md

@@ -87,12 +87,12 @@ User file `~/.pi/agent/pi-warden/config.json` (owner-only). `/warden config` ope
 | `maxRequests` | Per-session request budget. When spent, pi-warden says so once and continues with offline checks. |
 | `action.tools` | Tools the action guard inspects. Add your own shell-like tools here. |
 | `action.irreversible` | `warn` and `confirm` (hold) thresholds on P(irreversible). Defaults `warn` 0.5 and `confirm` 0.9. The 0.9 hold waits for the confidence at which the judge stops being wrong: on the recorded action-guard corpus the judge errs on 15% of calls below confidence 0.8 and under 1% above it, and a 0.9 cutoff removed about 52 false alarms on a held-out half of the corpus without losing a true catch. Calls the judge scores 0.5 to 0.9 warn instead of holding. See [guards.md](guards.md#irreversible-hold-threshold-2026-09-29-held-out-split). |
-| `action.offTask` | `warn` and `steer` thresholds on P(off-task). Off-task never holds. |
+| `action.offTask` | `warn` and `steer` are score thresholds for the off-task reason text in the trace. The warning itself follows the judge's `scope` answer (`unrelated` and `plausible side step` warn at any score), and every off-task notice is trace-only: no setting makes it reach the agent (the steer is off on the 2026-09-30 blind labels, see [guards.md](guards.md#steer-calibration-2026-09-30-blind-labels)). Off-task never holds. |
 | `action.intentMismatch` | P(call differs from the agent's stated plan) that warns and tells the agent, on calls that can change something. |
 | `action.visibleMismatch` | Lower mismatch threshold for commands whose effect is visible outside the working tree (commit, push, publish, install, launch). |
 | `action.intentTraceOnly` | Which intent mismatches stay in the trace without a steer to the agent. `"all"` (default): every mismatch, visible effect or not. `"invisible"`: only a call with no visible effect: not a commit, push, merge, tag, reset, pull request, release, or publish (decided in code), and not judged `visible` at 0.8 or more (an install, a launched program, a message sent from a script). `"none"`: none; every mismatch steers, as before. The steer arrives after the call ran. Blind labels on 140 sampled calls put the score's separation of a differing call at AUROC 0.815, but of the 37 steers that would reach the agent, 36 were calls the plan or the user's latest request had asked for. Restore the old behaviour with `"action": { "intentTraceOnly": "invisible" }`. |
-| `action.shouldProceed` | `{ threshold, steer }`. Scores at or below `threshold` (default 0.6) are trace-only by default until calibrated; they never hold a call. `hold` is the deprecated name of `threshold`: it is still read through 1.x, and `threshold` wins when both are set. |
-| `action.shouldProceed.steer` | Default `false`. Set `true` to restore the steer that asks the agent to pause and seek user approval. |
+| `action.shouldProceed` | `{ threshold, steer }`. Scores at or below `threshold` (default 0.6) are recorded in the trace and the status counts only; they never hold a call. On the 2026-09-30 blind labels 32% of judged calls sat at or below 0.6 while 2 of 427 labelled calls needed asking (see [guards.md](guards.md#steer-calibration-2026-09-30-blind-labels)). `hold` is the deprecated name of `threshold`: it is still read through 1.x, and `threshold` wins when both are set. |
+| `action.shouldProceed.steer` | Default `false`. Set `true` to send the pause-and-ask message on calls at or below `threshold`. The 2026-09-30 labels do not support it: precision 0.01 to 0.06 across thresholds. |
 | `action.feedbackLog` | Write each judged call and its outcome to `~/.pi/agent/pi-warden/holds/`; never the command. |
 | `action.floor` | `"evidence"` (default): when a judge answers, built-in pattern hits are evidence in the request, not level-setters. `"level"`: the floor sets the level before the judge, as it did pre-0.31. User-declared rules keep their declared action in both modes. User file only; project files cannot change it. |
 | `action.commandRules` | User-defined command rules: `{ id, pattern, severity: "warn" \| "confirm" \| "deny", action?, message?, caseSensitive? }`. `block` is the same as `deny`. A missing `severity` is `warn`; any other value is never downgraded: the rule applies at `confirm` and a config warning names the rule, the value, and the valid values. Patterns match the data-text-stripped command, so heredoc bodies and commit messages do not fire them. `confirm` defaults to `action: "dialog"` (a prompt for you, in every mode); `action: "hold"` restores steer semantics. Patterns are JavaScript regexes matched against full commands, so a pattern with nested quantifiers can be slow on long commands — a pathological one is self-inflicted. User rule ids must not reuse a built-in id (see `EXEMPTABLE_IDS` in `src/guard.ts`); a collision silently drops the user rule. User file only. |