From a7a56d954284652df1bc3718b62aedca6ba2cdb1 Mon Sep 17 00:00:00 2001 From: Ryan Gapac Date: Tue, 29 Sep 2026 09:56:34 +0800 Subject: [PATCH 1/2] feat: keep intent-mismatch verdicts in the trace by default The intent-mismatch steer did not earn its delivery. Hand labels on 140 sampled calls, blind to the score, put the score's separation of a differing call at AUROC 0.815, but of the 37 steers that would reach the agent, 16 were calls that did exactly what the plan said, 20 went beyond the plan on something the user's latest request had asked for, and 1 caught something the user had not asked for. action.intentTraceOnly now defaults to "all": every mismatch keeps its score, trace entry, and status count, and no steer reaches the agent. "invisible" restores the old delivery, where a call with a visible effect (a commit, push, merge, publish, install, launched program, or a script's message) still tells the agent, and "none" steers on every mismatch. The score, thresholds, trace, counters, and the user and project overrides are unchanged. Tests that need a delivered steer set "invisible" or "none" explicitly. Docs and CHANGELOG carry the numbers and the key that restores the old behaviour. --- CHANGELOG.md | 4 +++- docs/configuration.md | 4 ++-- docs/guards.md | 6 +++++- src/config.ts | 4 ++-- src/guard.ts | 8 +++++--- src/shape.ts | 4 ++-- tests/extension.test.ts | 30 ++++++++++++++++-------------- tests/guard.test.ts | 17 +++++++++-------- 8 files changed, 44 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 49776f6..f3113e2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,9 @@ How to keep this current: add the entry in the same pull request as the change, ## Unreleased - +### Changed + +- The intent-mismatch verdict is trace-only by default: `action.intentTraceOnly` is `"all"` instead of `"invisible"`, so a mismatch on a call with a visible effect (a commit, push, merge, tag, reset, pull request, release, publish, install, launched program, or a message sent from a script) no longer reaches the agent. The score, the trace entry, the `/warden status` counters, and the `visibleMismatch` and `intentMismatch` thresholds are unchanged; hand labels on 140 sampled calls, blind to the score, put the score's separation of a differing call at AUROC 0.815, but of the 37 steers that would reach the agent, 36 were calls the plan or the user's latest request had asked for. Restore the old delivery with `"action": { "intentTraceOnly": "invisible" }`. ## 0.75.0 diff --git a/docs/configuration.md b/docs/configuration.md index af1d426..c52ce5d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -25,7 +25,7 @@ User file `~/.pi/agent/pi-warden/config.json` (owner-only). `/warden config` ope "offTask": { "warn": 0.6, "steer": 0.85 }, "intentMismatch": 0.9, "visibleMismatch": 0.8, - "intentTraceOnly": "invisible", + "intentTraceOnly": "all", "shouldProceed": { "hold": 0.6, "steer": false }, "feedbackLog": true, "floor": "evidence", @@ -89,7 +89,7 @@ User file `~/.pi/agent/pi-warden/config.json` (owner-only). `/warden config` ope | `action.offTask` | `warn` and `steer` thresholds on P(off-task). Off-task never holds. | | `action.intentMismatch` | P(call differs from the agent's stated plan) that warns and tells the agent, on calls that can change something. | | `action.visibleMismatch` | Lower mismatch threshold for commands whose effect is visible outside the working tree (commit, push, publish, install, launch). | -| `action.intentTraceOnly` | Which intent mismatches stay in the trace without a steer to the agent. `"invisible"` (default): a call with no visible effect: not a commit, push, merge, tag, reset, pull request, release, or publish (decided in code), and not judged `visible` at 0.8 or more (an install, a launched program, a message sent from a script). `"all"`: every mismatch. `"none"`: none; every mismatch steers, as before. The steer arrives after the call ran. | +| `action.intentTraceOnly` | Which intent mismatches stay in the trace without a steer to the agent. `"all"` (default): every mismatch, visible effect or not. `"invisible"`: only a call with no visible effect: not a commit, push, merge, tag, reset, pull request, release, or publish (decided in code), and not judged `visible` at 0.8 or more (an install, a launched program, a message sent from a script). `"none"`: none; every mismatch steers, as before. The steer arrives after the call ran. Blind labels on 140 sampled calls put the score's separation of a differing call at AUROC 0.815, but of the 37 steers that would reach the agent, 36 were calls the plan or the user's latest request had asked for. Restore the old behaviour with `"action": { "intentTraceOnly": "invisible" }`. | | `action.shouldProceed` | `{ hold, steer }`. Scores at or below `hold` (default 0.6) are trace-only by default until calibrated; they never hold a call. | | `action.shouldProceed.steer` | Default `false`. Set `true` to restore the steer that asks the agent to pause and seek user approval. | | `action.feedbackLog` | Write each judged call and its outcome to `~/.pi/agent/pi-warden/holds/`; never the command. | diff --git a/docs/guards.md b/docs/guards.md index 55e8295..4adcf78 100644 --- a/docs/guards.md +++ b/docs/guards.md @@ -25,7 +25,7 @@ Runs on `tool_call`, before the tool executes. In **evidence mode** (`action.floor: "evidence"`, the default), built-in pattern hits listed above are fed to the judge as `floor_hits` in the request state and traced as `(evidence)` in reasons, but they do not set the hold level. The judge's `irreversible` score against the configured thresholds decides warn and confirm. This prevents the floor from overriding a present, confident judge. Without a judge (TypeSafe unavailable, consent off, request failed), or in **level mode** (`action.floor: "level"`), the floor applies as before: destructive hits hold, risky/sensitive hits warn, outside-project existing-file writes hold. 3. **Jev**, with consent: one request with `{ task, spine, context, plan, action, floor_hits }` and five questions. `irreversible` (yes/no), `off_task` (yes/no), `mutates` (does it change anything), `scope` (expected step, plausible side step, unrelated, unclear), `should_proceed` (yes/no, inverted: low = steer). Defaults: irreversible at 0.5 warns and at 0.9 holds; the 0.5 to 0.9 band warns instead of holding (see [Calibration](#calibration)). Off-task never holds: at 0.6 it warns, and at 0.85 with `unrelated` on a call that can change something the agent is also steered back to your request (an unrelated `grep` is warned about only). `should_proceed` steers but never holds: when P(yes) drops below 0.6 the agent is told to pause and ask the user. In evidence mode, built-in pattern hits are listed in `floor_hits` so the judge weighs them; in level mode, patterns set the floor and Jev can only raise it. - `plan` is the agent's own words in the message that makes the call, or in the text-only message right before it with no tool call in between (500 redacted characters). Text from before an earlier tool call described that call, so it is not sent and the intent question is not asked, except for a shell command with a visible effect (a `git` commit, push, merge, tag, or reset, `gh pr`, `gh release`, `npm publish`): that call is still judged against the latest text since your prompt. It tells Jev which step this is, so a verification fixture the agent just announced is not judged unrelated; it never authorizes anything. When there is a plan, a fifth question `intent_mismatch` asks whether the call does something materially different from it: a delete where the plan said list, a force push where it said push. At `action.intentMismatch` (0.9) on a call that can change something, the call is warned about and the agent is told to keep its words and its calls in step. A command whose effect is visible outside the working tree (`visible`: a commit, push, merge, publish, message, install, launched program) needs only `action.visibleMismatch` (0.8): on recorded sessions that pair is what users objected to. Never held on that alone. The warning reaches the agent after the call ran, so by default (`action.intentTraceOnly: "invisible"`) only a call with a visible effect steers the agent: a commit, push, merge, tag, reset, pull request, release, or publish (decided in code), or a call Jev judges `visible` at 0.8 or more (an install, a launched program, a message sent from a script); any other mismatch stays in the trace and the status count. On 275 recorded steers, all 275 arrived after the call, and a strict course change followed 8%. `"none"` steers on every mismatch; `"all"` on none. The trace shows the plan under each verdict. + `plan` is the agent's own words in the message that makes the call, or in the text-only message right before it with no tool call in between (500 redacted characters). Text from before an earlier tool call described that call, so it is not sent and the intent question is not asked, except for a shell command with a visible effect (a `git` commit, push, merge, tag, or reset, `gh pr`, `gh release`, `npm publish`): that call is still judged against the latest text since your prompt. It tells Jev which step this is, so a verification fixture the agent just announced is not judged unrelated; it never authorizes anything. When there is a plan, a fifth question `intent_mismatch` asks whether the call does something materially different from it: a delete where the plan said list, a force push where it said push. At `action.intentMismatch` (0.9) on a call that can change something, the call is warned about and the agent is told to keep its words and its calls in step. A command whose effect is visible outside the working tree (`visible`: a commit, push, merge, publish, message, install, launched program) needs only `action.visibleMismatch` (0.8): on recorded sessions that pair is what users objected to. Never held on that alone. The warning reaches the agent after the call ran, so by default (`action.intentTraceOnly: "all"`) every mismatch stays in the trace and the status count and the agent is not told. On 275 recorded steers, all 275 arrived after the call, and a strict course change followed 8%; on blind labels of 140 sampled calls the score is informative (AUROC 0.815), but of the 37 steers that would reach the agent, 36 were calls the plan or the user's latest request had asked for (see [Calibration](#intent-mismatch-2026-09-29-blind-labels)). `"invisible"` steers only on a call with a visible effect: a commit, push, merge, tag, reset, pull request, release, or publish (decided in code), or a call Jev judges `visible` at 0.8 or more (an install, a launched program, a message sent from a script). `"none"` steers on every mismatch. The trace shows the plan under each verdict. 4. **Act**, by mode: - `steer` (default): a hold blocks the call and returns the judgment to the agent as its tool result, with the two acceptable next moves: find a recoverable alternative, or explain the action to you and wait. If your reply approves it, the retry goes through (Jev reads your reply; offline, a yes/go-ahead heuristic does). - `confirm`: a `ctx.ui.confirm` dialog. No blocks with a short reason. Falls back to `steer` without a UI. @@ -79,6 +79,10 @@ The conscience coach assesses whether the agent is missing a useful skill or too - A second pass asked four candidate questions on the same calls (`scripts/action-candidates.mjs`, `--extra`). None separates rejected turns on its own: "would a careful engineer ask first", "is this unrequested", "did the user ask to pause", and "is the effect visible outside the working tree" all sit at the 4 to 5% base rate. `visible` has the best recall on regret (AUC 0.82, 10 of 19 regretted calls) but a commit or push is usually what was asked. Paired with the plan it works: `visible >= 0.8` and `intent_mismatch >= 0.8` flags 1.1% of calls with 18% in a rejected turn, so that pair steers at `visibleMismatch` 0.8. Two deterministic patterns came from the regretted list: a git command with hooks or signing switched off, and `gh pr merge`. - Of 42 holds pi-warden made in those sessions, the user's next message approved 5. +### Intent mismatch (2026-09-29, blind labels) + +140 sampled calls with a plan were labelled by hand, without the score in view, for whether the call did something other than the agent's stated plan. The `intent_mismatch` score separates the two (AUROC 0.815), but the steer it would deliver does not: of the 37 calls that would reach the agent, 16 did exactly what the plan said, 20 went beyond the plan on something the user's latest request had asked for, and 1 caught something the user had not asked for. The score, the trace entry, the `/warden status` counters, and the thresholds are unchanged; `action.intentTraceOnly` defaults to `"all"` for this reason, and `"invisible"` restores the old delivery. + ### should_proceed calibration (2026-09-20) 100 targeted sessions (50 pi-warden, 50 millia) selected by correction density: 980 labelled turns, 10,612 not-read-only calls, 2 regretted. The `should_proceed` question asks: "How confident are you that this action is safe to run without consulting the user?" Inverted: low P(yes) = a trace-only finding by default. diff --git a/src/config.ts b/src/config.ts index f32e254..fb02272 100644 --- a/src/config.ts +++ b/src/config.ts @@ -53,7 +53,7 @@ export interface ActionGuardConfig { intentMismatch: number; /** The same, for a command whose effect is visible outside the working tree (commit, push, merge, publish, launch): less mismatch is enough. */ visibleMismatch: number; - /** Which intent mismatches stay in the trace without a steer: "invisible" (default) a call with no visible effect (neither a commit, push, merge, tag, reset, pull request, release, or publish by `isVisibleCommand`, nor judged `visible` at 0.8 or more), "all" every one, "none" none. The steer arrives after the call ran: 275 of 275 recorded steers did. */ + /** Which intent mismatches stay in the trace without a steer: "all" (default) every one, "invisible" only a call with no visible effect (neither a commit, push, merge, tag, reset, pull request, release, or publish by `isVisibleCommand`, nor judged `visible` at 0.8 or more), "none" none. The steer arrives after the call ran: 275 of 275 recorded steers did. On blind labels of 140 sampled calls the score separates a differing call well (AUROC 0.815), but of 37 steers that would reach the agent, 36 were calls the plan or the user's latest request had asked for. */ intentTraceOnly: "invisible" | "all" | "none"; /** Low P(should_proceed) is trace-only unless steer is enabled; hold is the inclusive threshold, not a blocking decision. Calibration: AUC 0.26 against regret, 44% flagged at 0.6 (100 targeted sessions, 2026-09-20). */ shouldProceed: { hold: number; steer: boolean }; @@ -492,7 +492,7 @@ export function defaultConfig(): WardenConfig { offTask: { warn: 0.6, steer: 0.85 }, intentMismatch: 0.9, visibleMismatch: 0.8, - intentTraceOnly: "invisible", + intentTraceOnly: "all", shouldProceed: { hold: 0.6, steer: false }, feedbackLog: true, commandRules: [], diff --git a/src/guard.ts b/src/guard.ts index 1b5980d..ed05e52 100644 --- a/src/guard.ts +++ b/src/guard.ts @@ -2211,9 +2211,11 @@ export async function evaluateAction(action: ActionInput, options: EvaluateOptio const visibleDrift = judgment.intentMismatch !== undefined && (judgment.visible ?? 0) >= VISIBLE_THRESHOLD && judgment.intentMismatch >= config.visibleMismatch; const mismatch = judgment.intentMismatch !== undefined && canChange && (judgment.intentMismatch >= config.intentMismatch || visibleDrift); // The steer reaches the agent after the call ran (275 of 275 recorded steers), and a strict course change followed 8% of - // them. A call with no visible effect keeps the finding in the trace only; a visible one still tells the agent. Visible - // is either rule: the code's (commit, push, merge, tag, reset, pull request, release, publish) or the judge's `visible` - // score at 0.8, which also covers an install, a launched program, or a message sent from a script. + // them; blind labels of 140 sampled calls found that of the 37 steers that would reach the agent, 36 were calls the plan + // or the user's latest request had asked for, so the default keeps every mismatch in the trace only ("all"). Under + // "invisible" only a call with a visible effect tells the agent: the code's (commit, push, merge, tag, reset, pull + // request, release, publish) or the judge's `visible` score at 0.8, which also covers an install, a launched program, or + // a message sent from a script. const visibleEffect = (view?.shell === true && isVisibleCommand(view.command)) || (judgment.visible ?? 0) >= VISIBLE_THRESHOLD; const intentTraceOnly = mismatch && (config.intentTraceOnly === "all" || (config.intentTraceOnly === "invisible" && !visibleEffect)); let intentTraceOnlyReasonIndex: number | undefined; diff --git a/src/shape.ts b/src/shape.ts index 4158f21..03c48b5 100644 --- a/src/shape.ts +++ b/src/shape.ts @@ -48,7 +48,7 @@ export function completeConfig(loaded: Partial | undefined): Shape steerVisible: source.steerVisible ?? false, notices: source.notices ?? false, steerBudget: typeof source.steerBudget === "number" && source.steerBudget >= 0 ? source.steerBudget : 3, - action: section("action", { ...off, tools: [], failOpen: true, timeoutMs: 5000, irreversible: { warn: 1, confirm: 1 }, offTask: { warn: 1, steer: 1 }, intentMismatch: 1, visibleMismatch: 1, intentTraceOnly: "invisible", shouldProceed: { hold: 0.6, steer: false }, feedbackLog: false, commandRules: [], commandDenyRules: [], exemptRules: [], pathRules: [], armingRules: [], escalationThreshold: 0.85, floor: "evidence" }), + action: section("action", { ...off, tools: [], failOpen: true, timeoutMs: 5000, irreversible: { warn: 1, confirm: 1 }, offTask: { warn: 1, steer: 1 }, intentMismatch: 1, visibleMismatch: 1, intentTraceOnly: "all", shouldProceed: { hold: 0.6, steer: false }, feedbackLog: false, commandRules: [], commandDenyRules: [], exemptRules: [], pathRules: [], armingRules: [], escalationThreshold: 0.85, floor: "evidence" }), stuck: section("stuck", { ...off, window: 12, minFailures: 3, cooldown: 3, sameStrategy: 1, churnThreshold: 5, nudge: false, repeatSteer: false, evidence: false, diffLimit: 3000, tailLimit: 1000 }), done: section("done", { ...off, claimsDone: 1, nudge: false, uiProof: false, uiFiles: [], visualTools: { commands: [], commandWords: [], tools: [], images: [] } }), slop: section("slop", { ...off, threshold: 1, prose: proseOff() }), @@ -105,7 +105,7 @@ export function completeConfig(loaded: Partial | undefined): Shape if (typeof config.action.feedbackLog !== "boolean") config.action = { ...config.action, feedbackLog: true }; if (typeof config.action.intentMismatch !== "number") config.action = { ...config.action, intentMismatch: 0.9 }; if (typeof config.action.visibleMismatch !== "number") config.action = { ...config.action, visibleMismatch: 0.8 }; - if (config.action.intentTraceOnly !== "invisible" && config.action.intentTraceOnly !== "all" && config.action.intentTraceOnly !== "none") config.action = { ...config.action, intentTraceOnly: "invisible" }; + if (config.action.intentTraceOnly !== "invisible" && config.action.intentTraceOnly !== "all" && config.action.intentTraceOnly !== "none") config.action = { ...config.action, intentTraceOnly: "all" }; if (typeof config.action.shouldProceed !== "object" || config.action.shouldProceed === null || typeof config.action.shouldProceed.hold !== "number") config.action = { ...config.action, shouldProceed: { hold: 0.6, steer: false } }; if (typeof config.action.shouldProceed.steer !== "boolean") config.action = { ...config.action, shouldProceed: { ...config.action.shouldProceed, steer: false } }; if (typeof config.action.escalationThreshold !== "number") config.action = { ...config.action, escalationThreshold: 0.85 }; diff --git a/tests/extension.test.ts b/tests/extension.test.ts index 0669454..d664b0f 100644 --- a/tests/extension.test.ts +++ b/tests/extension.test.ts @@ -1285,8 +1285,8 @@ test("the agent's plan comes from the message that makes the call or the text-on sentMessages.length = 0; assert.equal(await toolCall("bash", { command: "npm run clean" }, earlier), undefined, "a mismatch warns; it never holds"); assert.equal(requests.at(-1)!.state.plan, "Let me first list what is in build/ before removing anything."); - assert.ok(!sentMessages.some(sent => sent.message.customType === "pi-warden-steer"), "npm run clean has no visible effect: trace-only by default"); - assert.match(notices.at(-1)!.text, /^warden · bash: intent mismatch 0\.91 \(the call differs from the agent's stated plan; trace-only, no visible effect\)$/); + assert.ok(!sentMessages.some(sent => sent.message.customType === "pi-warden-steer"), "by default every mismatch stays in the trace"); + assert.match(notices.at(-1)!.text, /^warden · bash: intent mismatch 0\.91 \(the call differs from the agent's stated plan; trace-only\)$/); assert.match(widgets.at(-1)![0]!, /^WARN\s+action\s+bash · .*off plan$/, "the mismatch leads the line as a warn chip"); // A tool-calls-only message after an earlier tool call: the text before that call described it, so no plan, no question. @@ -1310,13 +1310,13 @@ test("the agent's plan comes from the message that makes the call or the text-on await runCommand("status"); const status = notices.at(-1)!.text; assert.match(status, /1 off plan \(1 trace-only\)/); - assert.match(status, /intent mismatch 0\.9 \(0\.8 on a visible action, trace-only: invisible\);/); + assert.match(status, /intent mismatch 0\.9 \(0\.8 on a visible action, trace-only: all\);/); const logPath = status.match(/Log: (.+?\.jsonl)\./)![1]!; const lines = await readLog(logPath, 4, false); assert.deepEqual(lines.map(record => [record.planChars, (record.scores as Record | undefined)?.intentMismatch]), [["Now a live verification step: I will write a small fixture under /tmp. TOKEN=[redacted]".length, 0.1], ["Let me first list what is in build/ before removing anything.".length, 0.91], [0, undefined], [0, undefined]], "planChars says how often the agent called without a word"); }); -test("intentTraceOnly: an invisible mismatch is traced without a steer, a visible one steers; \"none\" and \"all\" set every call", async () => { +test("intentTraceOnly: every mismatch is trace-only by default; \"invisible\" steers only a visible effect, \"none\" every mismatch", async () => { prompt = "Verify the RPC endpoint end to end"; const plan = "Let me first list what is in build/ before removing anything."; const branch = (command: string) => context({ hasUI: false, sessionManager: { getBranch: () => [ @@ -1333,20 +1333,22 @@ test("intentTraceOnly: an invisible mismatch is traced without a steer, a visibl return intentSteers().length; }; - // Default "invisible": no steer, no headless warn notice, and one trace line that names the mismatch. + // Default "all": no steer, no headless warn notice, and one trace line that names the mismatch. assert.equal(await run(undefined, "npm run clean"), 0); assert.ok(!sentMessages.some(sent => /ran with a warning/.test(sent.message.content)), "the headless warn steer drops the trace-only reason too"); await runCommand("trace", context({ hasUI: false })); const trace = sentMessages.at(-1)!.message.content; - assert.equal(trace.match(/intent mismatch 0\.91 \(the call differs from the agent's stated plan; trace-only, no visible effect\)/g)?.length, 1); - - // A push is visible by code, an install the judge scores visible: the steer still reaches the agent for both. - assert.equal(await run(undefined, "git push origin main"), 1); - assert.equal(await run(undefined, "npm install left-pad", 0.85), 1); - assert.equal(await run(undefined, "npm install left-pad", 0.5), 0); - // "none" restores the steer on every mismatch; "all" sends none, visible calls included. + assert.equal(trace.match(/intent mismatch 0\.91 \(the call differs from the agent's stated plan; trace-only\)/g)?.length, 1); + + // A push is visible by code and an install the judge scores visible: the default silences both. + assert.equal(await run(undefined, "git push origin main"), 0); + assert.equal(await run(undefined, "npm install left-pad", 0.85), 0); + // "invisible" tells the agent only about a call with a visible effect. + assert.equal(await run("invisible", "git push origin main"), 1); + assert.equal(await run("invisible", "npm install left-pad", 0.85), 1); + assert.equal(await run("invisible", "npm install left-pad", 0.5), 0); + // "none" restores the steer on every mismatch. assert.equal(await run("none", "npm run clean"), 1); - assert.equal(await run("all", "git push origin main"), 0); assert.equal(await run("all", "npm run clean"), 0); await runCommand("status"); assert.match(notices.at(-1)!.text, /1 off plan \(1 trace-only\)/); @@ -1361,7 +1363,7 @@ test("adaptive steers: an intent-mismatch steer the model does not follow become { type: "message", message: { role: "user", content: prompt } }, assistantEntry({ type: "text", text: plan }, { type: "toolCall", id: "call-1", name: "bash", arguments: { command } }), ] } }); - await writeFile(configPath(), JSON.stringify({ typesafe: true, notices: false, rules: { enabled: false }, slop: { enabled: false }, security: { enabled: false }, action: { feedbackLog: false }, steers: { minSteers: 2, recheckEvery: 4, probeEvery: 2 }, ...STACK_BAR })); + await writeFile(configPath(), JSON.stringify({ typesafe: true, notices: false, rules: { enabled: false }, slop: { enabled: false }, security: { enabled: false }, action: { feedbackLog: false, intentTraceOnly: "invisible" }, steers: { minSteers: 2, recheckEvery: 4, probeEvery: 2 }, ...STACK_BAR })); const intentSteers = () => sentMessages.filter(sent => sent.message.customType === "pi-warden-steer" && /what you said you were about to do/.test(sent.message.content)).length; /** One mismatching push; the agent's next two messages carry on without a course change. Returns whether the steer was sent. */ const run = async (id: string) => { diff --git a/tests/guard.test.ts b/tests/guard.test.ts index 7d332cc..98401b7 100644 --- a/tests/guard.test.ts +++ b/tests/guard.test.ts @@ -1012,8 +1012,8 @@ test("the agent's plan travels with the request and is judged for intent mismatc assert.equal(drift.level, "warn"); assert.equal(drift.intentMismatch, true); assert.equal(drift.judgment?.intentMismatch, 0.9); - assert.match(drift.reasons.join("; "), /intent mismatch 0\.90 \(the call differs from the agent's stated plan; trace-only, no visible effect\)/); - assert.equal(drift.intentTraceOnly, true, "rm has no effect outside the working tree: trace-only by default"); + assert.match(drift.reasons.join("; "), /intent mismatch 0\.90 \(the call differs from the agent's stated plan; trace-only\)/); + assert.equal(drift.intentTraceOnly, true, "the default keeps every mismatch in the trace only"); assert.equal(drift.plan, "Let me first list what is in build/ before removing anything."); assert.match(formatVerdict(drift), /off plan · warn$/); assert.match(intentSteer(drift), /^pi-warden: this bash call does something different from what you said you were about to do \(intent mismatch 0\.90\)\. It ran\./); @@ -1043,7 +1043,8 @@ test("the agent's plan travels with the request and is judged for intent mismatc }); test("a visible action (commit, push, merge, launch) needs less plan mismatch to be steered than a file edit", async () => { - const config = defaultConfig().action; + const defaults = defaultConfig().action; + const config = { ...defaults, intentTraceOnly: "invisible" as const }; const withVisible = (mismatch: number, visible: number): Judge => ({ async evaluate(request) { const base = answers(0.1, 0.1, "expected_step", 0.9, 0.9) as { answers: Record }; @@ -1060,17 +1061,17 @@ test("a visible action (commit, push, merge, launch) needs less plan mismatch to assert.match(drift.reasons.join("; "), /intent mismatch 0\.83 on a visible action \(0\.96; a commit, push, merge, publish, or launch the plan did not describe\)/); assert.match(intentSteer(drift), /and its effect is visible outside the working tree/); assert.match(formatVerdict(drift), /off plan · warn$/); - assert.equal(drift.intentTraceOnly, undefined, "a push and a pull request keep the steer by default"); + assert.equal(drift.intentTraceOnly, undefined, "under \"invisible\" a push and a pull request keep the steer"); const install = { tool: "bash", input: { command: "npm install left-pad" }, cwd, task: "get the PR ready", plan: "I will run the tests once more before touching the PR." }; const judgedVisible = await evaluateAction(install, { config, judge: withVisible(0.91, 0.85) }); assert.equal(judgedVisible.intentMismatch, true); assert.equal(judgedVisible.intentTraceOnly, undefined, "not visible by code, but the judge scores it visible: the steer stays"); const judgedLocal = await evaluateAction(install, { config, judge: withVisible(0.91, 0.5) }); assert.equal(judgedLocal.intentTraceOnly, true, "neither code nor judge finds a visible effect: trace-only"); - const silenced = await evaluateAction(call, { config: { ...config, intentTraceOnly: "all" }, judge: withVisible(0.83, 0.96) }); - assert.equal(silenced.intentTraceOnly, true, "\"all\" keeps even a visible mismatch in the trace only"); - assert.equal(silenced.intentTraceOnlyReasonIndex, silenced.reasons.findIndex(reason => reason.startsWith("intent mismatch"))); - assert.match(silenced.reasons.join("; "), /the plan did not describe; trace-only\)/); + const traced = await evaluateAction(call, { config: defaults, judge: withVisible(0.83, 0.96) }); + assert.equal(traced.intentTraceOnly, true, "the default keeps even a visible mismatch in the trace only"); + assert.equal(traced.intentTraceOnlyReasonIndex, traced.reasons.findIndex(reason => reason.startsWith("intent mismatch"))); + assert.match(traced.reasons.join("; "), /the plan did not describe; trace-only\)/); const quiet = await evaluateAction(call, { config, judge: withVisible(0.83, 0.2) }); assert.equal(quiet.intentMismatch, undefined, "the same mismatch on an action nobody else sees is below the bar"); assert.equal(quiet.level, "allow"); From 068a446a85190a4e95b5daaf05ef83d46ee33838 Mon Sep 17 00:00:00 2001 From: Ryan Gapac Date: Tue, 29 Sep 2026 10:08:48 +0800 Subject: [PATCH 2/2] chore: version bump to 0.76.0, finalize CHANGELOG --- CHANGELOG.md | 4 ++++ package.json | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f3113e2..74c50f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ How to keep this current: add the entry in the same pull request as the change, ## Unreleased + + +## 0.76.0 + ### Changed - The intent-mismatch verdict is trace-only by default: `action.intentTraceOnly` is `"all"` instead of `"invisible"`, so a mismatch on a call with a visible effect (a commit, push, merge, tag, reset, pull request, release, publish, install, launched program, or a message sent from a script) no longer reaches the agent. The score, the trace entry, the `/warden status` counters, and the `visibleMismatch` and `intentMismatch` thresholds are unchanged; hand labels on 140 sampled calls, blind to the score, put the score's separation of a differing call at AUROC 0.815, but of the 37 steers that would reach the agent, 36 were calls the plan or the user's latest request had asked for. Restore the old delivery with `"action": { "intentTraceOnly": "invisible" }`. diff --git a/package.json b/package.json index 97db3d5..61df8b3 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "pi-warden", - "version": "0.75.0", + "version": "0.76.0", "description": "Makes the Pi agent follow your project's rules. Jev judges every write against your pi-warden.md and quotes the broken rule back to the agent, names slop, breaks stuck loops, calls out unverified done claims, compresses large tool output, and holds the rare destructive command. Built on pi-typesafe.", "type": "module", "license": "MIT",