feat: opt-in context filter keeps the passages of a large output that matter for the task

CHANGELOG.md

@@ -8,6 +8,12 @@ How to keep this current: add the entry in the same pull request as the change,
 
 <!-- Empty. Next release starts here. -->
 
+## 0.77.0
+
+### Added
+
+- Context filter (beta, off by default). With `"context": { "filter": { "enabled": true } }`, an output that would get the generic head/diagnostic/tail excerpt is split at line boundaries and Jev scores each chunk for the agent's current task; chunks scoring at least `minScore` (1.5) are kept word for word in original order, up to `maxKeptChars` (6000) with the last 1000 characters always kept, and each gap is marked. Parser excerpts, `all`, duplicates, and repeated runs are unchanged. Any error, a timeout (`timeoutMs`, 4000), judgments off, an exhausted request budget, or no passing chunk keeps today's excerpt. `/warden status`, the trace, and the new offline `scripts/filter-report.mjs` count filtered and excerpt outputs apart (count, recalls, kept size, requests, time, fallbacks), so a trial can be judged.
+
 ## 0.76.0
 
 ### Changed

docs/configuration.md

@@ -118,6 +118,7 @@ User file `~/.pi/agent/pi-warden/config.json` (owner-only). `/warden config` ope
 | `context.dedupeMessages` | Default `false`. With `context.dedupeRuns` also on, cut repeated runs in new user and custom messages the same way. Off by default because a repeat the user sends can itself carry meaning ("here it is again, still failing"), and on recent sessions messages gave about 0.8% of their bytes back. A custom message that Pi appends without an agent turn (`triggerTurn: false`, or unset while the agent is idle) does not pass Pi's `message_end` hook and stays whole. |
 | `context.largeOutput.enabled` | Add one question to each judged `bash` request: will the command print far more than the agent needs? Off keeps the question out of the request. Read-only commands (`cat`, `find`, `git log`) skip the judge, so the question does not ride them. |
 | `context.largeOutput.threshold` | P(large output) at or above which the agent is told, once per command family (`npm test`, `git log`, `find`) per session, to redirect or filter the command before it runs one like it again. The call is never held or warned. Default `0.85`. |
+| `context.filter` | Beta, default `{ "enabled": false, "chunkChars": 2000, "minScore": 1.5, "maxKeptChars": 6000, "timeoutMs": 4000 }`. When on, an output that would get the generic head/diagnostic/tail excerpt is split at line boundaries into chunks of about `chunkChars`; Jev scores each chunk 0 to 3 for the agent's current task, and chunks at or above `minScore` are kept word for word in original order, up to `maxKeptChars` including the last 1000 characters. Parser excerpts, `all`, duplicates, and repeated runs are unchanged. On an error, a timeout after `timeoutMs`, no consent, an exhausted request budget, or no chunk at `minScore`, the excerpt is used. Costs one or more requests per filtered output. `enabled` is read from the user file only: a project file may tune `chunkChars`, `minScore`, `maxKeptChars`, and `timeoutMs`, but cannot turn the filter on, because that sends whole redacted outputs to the judge and spends requests. See [guards.md](guards.md#context-filter-beta-off-by-default). |
 | `runaway.*` | Repeat counts that abort a reply, minimum size, whether the agent gets one recovery turn. |
 | `notify.*` | Desktop notifications, cooldown, optional relay command (user file only). |
 | `judge.failuresBeforeCooldown` | Consecutive timeout, network, or other judge failures before judgments pause for the session (3). One auth or configuration failure pauses at once. |

docs/data-handling.md

@@ -11,7 +11,7 @@ With consent, requests go to `https://api.typesafe.ai` (default), or to the host
 | **Action** | Your latest prompt (1500 characters), the task spine it is judged against (the thread's first request and up to four earlier requests, redacted, capped at 1200 characters together), up to eight earlier user and assistant messages (750 redacted characters each), the agent's text from the message that makes the call (500 redacted characters), the tool name, the command (2000 characters) or the file path (relative inside the project, `~`-shortened outside), whether the file exists, a 1500-character head/middle/tail sample of a `write` (or of the content a `bash` command writes to a file with the content in the command, with the written paths), the first three edit pairs (400 characters each) of an `edit`. The resolved active rules content (`pi-warden.md`, the configured files, or `AGENTS.md`/`CLAUDE.md`/`README.md` as fallback, token-aware truncated at ~4000 tokens) is sent with every action request, unless `rules.enabled` is false: with the rules guard off, no rules content leaves the machine. On the first guarded call after your reply, the tool names and commands (300 characters) or paths of up to six calls allowed in the previous turn, for the regret question. |
 | **Rules** | The project-relative path, a 6000-character sample of a `write` (or of the content a `bash` heredoc, `echo`, or `printf` writes to a file, judged as a `write`) or each edit's new text (1500 characters) with about 40 lines of the current file around the replaced text, as they are and with the edit applied, and the rule text from your rules file or the condensed fallback document (`rules.maxChars`). No task text. Files under `rules.exclude` are never sent. `/warden rules check` sends each rule's id, heading, `paths:` scope, and text (redacted, clipped at 400 characters) in one request per 32 questions, with no file content and no task text. `/warden rules calibrate` sends one request per changed file of the sampled commits — the project-relative path, the added and removed lines with about 40 lines of the file after the commit around them (redacted), and the rule text — only after a confirm dialog that shows how many requests go out and the redacted diffs; a headless run sends nothing without the explicit `--yes`. `/warden rules tune` sends no request: it hands the flagged rules to the session's agent as one message. | `/warden rules audit` sends a 6000-character redacted sample of each selected file judged as a `write`, plus the rule text, and only after the confirm dialog or `--yes`; files under `rules.exclude` or `rules.skip` are never selected. `/warden bench` sends only one fixed built-in sample file and the rule text; no project content goes for it. || **Stuck** | The last 12 tool calls (300 characters each) with 400-character output tails, and, while `stuck.evidence` is on (the default), a structured `evidence` section: per run the parsed failing test, error, location, summary, exit code and which earlier run failed the same way (or 300 characters of head and 300 of tail when nothing parses), per `edit`/`write` the project path and a diff of the change capped at 600 characters, and a digest. Every string is redacted; the whole object is capped at 4 KB. |
 | **Done-check and prose** | The agent's final message (2000 and 2500 characters), the run's check commands, the audience description. |
-| **Output checks** | A redacted head/tail sample up to 6000 characters plus size, line counts, and tool name. |
+| **Output checks** | A redacted head/tail sample up to 6000 characters plus size, line counts, and tool name. With the context filter on (`context.filter.enabled`, off by default), an output that would get the generic excerpt is also sent whole, redacted, in line-bounded chunks, with your latest prompt (1500 characters), the task spine, the agent's text for the call (1500 characters), the tool name, and the command or tool input (500 characters). |
 | **Conscience** (recommend mode) | Your current request (2000 redacted characters), the same task spine (the thread's first request and up to four earlier requests, redacted, capped at 1200 characters together), up to four recent user/assistant text messages (500 redacted characters each with roles), and sanitized candidate metadata (skill/tool name, role, lead, useWhen, examples when an index entry matches; bare description otherwise). Full skill instructions never go to Jev. The index is built locally by the session model; only sanitized entries reach Jev; advertised locations never do. Sent only when TypeSafe consent is given and the conscience module is enabled. |
 | **Conscience** (load mode) | Same judge payload as recommend mode, plus: the selected skill file is read from disk (bounded by `maxSkillBytes` and `maxLoadedBytes`), frontmatter is stripped, credentials are checked, and the complete body is supplied to the main model via a custom message. Skill bodies never go to Jev. |
 | **Subagent triage** | Only for a child report that names a failure, a stop, a timeout, or a question (an incremental progress line or a clean completion is answered in code and sends nothing): a redacted 1500-character head plus 500-character tail of the report, the notification type, whether it is an incremental notify, its length, and your latest prompt (1000 characters). |

docs/guards.md

@@ -400,6 +400,20 @@ Only the newest tool result or message is ever changed, before it enters the ses
 
 Set `context.enabled: false` to turn it off. Full-output files can contain secrets and stay in the OS temporary directory until removed.
 
+### Context filter (beta, off by default)
+
+`context.filter.enabled: true` changes one case only: a single text block for which the saver would build the generic head/diagnostic/tail excerpt (retention `errors_and_summary` or `summary_only`, and no format parser fits). Parser excerpts, `all`, duplicates, repeated runs, multi-block results, and outputs below `tailMinChars` are unchanged.
+
+1. **Chunks.** The output is split at line boundaries into chunks of about `chunkChars` (2000) characters. A line is split only when it alone is longer than `chunkChars`.
+2. **One score question per chunk.** The request state carries the task (your latest request and the task spine), the agent's own words for the call when it gave any, the tool and command, and the chunks as named fields (`c1`, `c2`, …), all redacted. Each chunk gets one `score` question for the agent's current task with four levels: 0 "Unrelated to the question", 1 "Same topic, but does not help answer the question", 2 "Partially answers the question or gives useful supporting facts", 3 "Directly answers the question with specific facts". Chunks share requests up to pi-typesafe's limits (64 KiB of JSON, 32 questions); the requests run in parallel.
+3. **Threshold, then budget.** Chunks scoring at least `minScore` (1.5: they at least partly answer) are kept word for word, in original order, up to `maxKeptChars` (6000). When more qualify, the highest scores are kept and the original order is restored. The last 1000 characters (the final status) are always kept and count toward `maxKeptChars`. Each gap is marked `[… N lines omitted …]`.
+4. **Header and footer.** `[pi-warden: filtered; N original characters, M lines. Passages selected for the current task; omitted text is in the full-output file.]`, then the kept text, then the same full-output footer as the excerpt.
+5. **Fallback.** On a Jev error, a timeout (`timeoutMs`, 4000), judgments off (no consent, no key, or an exhausted request budget), a judge cooldown, or no chunk at the threshold, the excerpt is used unchanged. A filtered output that would be longer than the excerpt by more than `maxKeptChars` also falls back.
+
+**Measuring it.** While the filter is on, `/warden status` adds a line that counts filtered outputs and excerpt outputs apart: count, recalls (whole-file and scoped), characters kept, requests, milliseconds, and fallbacks by reason. Each filtered output leaves one trace entry with the chunks kept, the characters kept, the requests, and the milliseconds; a fallback adds its reason to the excerpt's trace entry. `node scripts/filter-report.mjs --since <ISO date>` reads Pi session files offline and prints the same comparison (outputs, original and kept size, recall rates) from the header texts; it sends no request and prints counts only.
+
+**Method source and limits.** The method is GPT Researcher's Jev context filter, measured on 28 research tasks: one score question per chunk, a fixed threshold, original order. There the threshold, not the ranking, made the gain: 73% of kept passages were relevant with it, 50% without. It has not yet been measured on tool output; this beta is for that trial. Batching several chunk questions into one request is a known compromise: other questions in the same request shift probabilities by about 0.05 (arXiv 2609.26550), and a replay on this codebase found 85% agreement on keep decisions between many questions per request and one per request.
+
 ## Call waste
 
 Every tool call re-reads the whole conversation, so the number of calls drives what a run costs. Four patterns spend calls without gaining anything a single call would not. Each earns one advisory sentence, attached to the tool result that triggers it: the result already goes to the model, so the note costs no extra call and never makes a request of its own.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-warden",
-  "version": "0.76.0",
+  "version": "0.77.0",
   "description": "Makes the Pi agent follow your project's rules. Jev judges every write against your pi-warden.md and quotes the broken rule back to the agent, names slop, breaks stuck loops, calls out unverified done claims, compresses large tool output, and holds the rare destructive command. Built on pi-typesafe.",
   "type": "module",
   "license": "MIT",