diff --git a/CHANGELOG.md b/CHANGELOG.md index 00e0a471..8f64a765 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,30 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in ## Unreleased +- `collect-remote` refuses empty repository, topic, client-ref, prior and prefer + values by name before creating an output directory or reading a token (#287). +- Invalid `--snapshot-max-age-ms` values are usage errors with exit 2. Library callers + receive `INPUT-SNAPSHOT-MAX-AGE` / `INCOMPLETE` rather than a false stale-snapshot + verdict; finite non-negative allowances, including zero, retain their meaning (#343). +- `decompose --limit` consistently requires a whole number of at least 1 before + transport selection. `--max-scrapes 0` remains supported (#320). +- Auto-collect Git failure diagnostics quote the first `fatal:` or `error:` cause, + retaining the final-line fallback for output without either marker (#326). +- A leading ledger byte-order mark is tolerated while reading, without changing the + evidence file or its entry hashes. Tail repair leaves an intact BOM ledger alone + and preserves the prefix when dropping a torn append. Handoff gives unreadable lines the tail-repair + or restore-from-git remedy; secondary chain failures no longer imply failed travel + or lost entries while parsing is unresolved (#296). +- Finding extraction remembers the winning paragraph start and builds its excerpt + forward only until truncation is established. Long unbroken captures avoid the + quadratic array insertion while retaining the same finding and excerpt (#334). +- Recipe loading refuses existing files without dimensions and blank dimension labels, + naming the source and line. Decompose reports invalid recipe content with exit 2 + before writing the map (#356). +- Oversized panel requests receive the intended 413 JSON refusal before the + connection closes. Overflow discards retained body chunks and later data, while + a complete JSON body at the existing byte limit remains accepted (#324). + - ZIP archives declare UTF-8 names in both headers, so ordinary extractors preserve non-ASCII paths declared by the package manifest (#345). - Machine config saves preserve unpatched raw role and posture values, including typos diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a35be350..23221272 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -64,6 +64,12 @@ about defects that were already fixed. Nothing was deleted; it was moved and lin ## Search and collection invocation +`bin/collect-remote.mjs` applies the existing value checker to `--repository`, `--topic`, +`--client-ref`, `--prior` and `--prefer`, as well as its workflow and budget flags. +An empty equals/separate value or bare optional value flag is refused by name with +exit 3 before output-directory creation, token lookup or dispatch (#287). Absent +required flags keep their existing refusal, including the repository-only pickup path. + The CLI chooses providers before invoking its coordinator. `transport.mjs` returns the fetch adapter, search adapter or merged adapters, selection reasons, and selection readiness metadata. Automatic fetch selection may perform a memoized Firecrawl CLI `--status` probe when the executable is present. Selection does not search or fetch a page. The coordinators decide which queries and pages to attempt; the shared search session and collection writer carry out those decisions. ```mermaid @@ -185,7 +191,7 @@ is an open decision (2026-09-30): | `disclosure.mjs` | **What a public run actually exposes, measured rather than asserted.** Written because the opposite claim was documented first: the collector's header said dispatch inputs are visible to anyone with read access, and checking it found the topic in none of the readable responses. `probeRun` sends **unauthenticated** requests - there is no token parameter, and a probe that quietly authenticated would answer a different question while looking like this one. `summarise` separates `exposedNames` (true on any public repo, and the reason the topic is kept out of run and artifact names) from `exposedContent` and `exposedSubject`. `needleChecked` exists so a report can never let "false" read as "no": without a topic to search for, the rendering says **not checked**. An unreachable endpoint is a refusal to conclude, never a clean result - **enforced since 2026-09-27**: until then that sentence was true of the findings and false of the verdict, and with no network every probe was unreachable and the command exited 0, "only the shape is public". `summarise` now reports `unmeasured` (probes with status 0) and `conclusive`; `exitCode(report)` keeps a measured exposure (2, 1) whatever else failed, and otherwise needs every probe answered for 0, else 3; the rendering says UNREACHABLE and "not tried" instead of "refused", shows `not measured` for a false it could not establish, and closes NOT MEASURED instead of the shape paragraph. An unreachable probe names the cause of the failed fetch through `fetchFailure` (2026-09-30), not the bare `fetch failed`. `probeRun` takes the artifact and job ids from the listings its probes already read (2026-09-30); it had fetched both again inside one try/catch, so a transient failure on the second request left the artifact download and the log untried. | ADR-0035 | | `dispatch.mjs` | **Asking the collector to run, from outside GitHub** - the seam an agent or a desktop app uses. `dispatchCollection` (pins `X-GitHub-Api-Version: 2026-03-10` on the REQUEST, not merely in a comment), `waitForRun`, `listArtifacts`, `downloadArtifact`, `unwrapArtifact`, plus `tokenFromEnv` and `redact`. **No token may be passed as an argument** - `tokenFromEnv` is the only way in, because a credential on a command line reaches the shell history, `ps`, and any log that echoes its own command; a test asserts the source contains no token flag. `redact` is PATTERN-based rather than "remove the token we hold", so it also catches a credential the process was never given - one echoed back by a server, a proxy, a redirect. A **204** is a named `NO_RUN_ID` failure, never an empty success: "the run started and I cannot tell which one" is the failure a caller is least able to diagnose from a status code. **Both routes to a run id are requested**, not just the pinned version: the `X-GitHub-Api-Version: 2026-03-10` header AND `return_run_details: true`, the optional parameter GitHub added 2026-02-19 that returns the run id on `2022-11-28` as well. Measured before it was trusted - four calls on 2026-09-21 established that the old version honours the parameter and the new version accepts rather than rejects it, so sending both is safe and a server that strips or downgrades the header still answers with a run id. That retired ADR-0031's original fallback, which was to poll `/actions/runs` and correlate by `created_at` - a race the design never actually needed. `NO_RUN_ID` now means BOTH routes were ignored, which is a far narrower diagnosis. `waiting` is reported as a status, not a stall - it means a protection rule is holding the job. `unwrapArtifact` opens GitHub's outer ZIP through the same distrustful reader, and passes a hostile or ambiguous container through untouched for the validator to refuse rather than guessing at packaging depth. Every request goes through an injectable `fetch`, so the whole module is tested offline. **Pages by URL** (2026-09-27): `queriesInput` builds the one `queries` input from search queries and pages to fetch, one per line, and refuses a URL that is not http(s) before anything is dispatched. `collect.yml` fetches a line that is an http(s) URL instead of searching it, and with only URLs it searches nothing. `collect-remote --url` and the MCP `collect` tool's `urls` share this function, so all three agree. Until then the collector could only search, and a page the caller already knew had to be collected locally. **`usableOutDir` checks the folder before anything is spent** (2026-09-27): it creates the folder and tests write access, and an unusable one is a named `OUT_DIR` failure with a remedy. `fetchCorpus` calls it before it lists artifacts, the MCP `fetch_corpus` tool before it reads the run, and `collect-remote` before it dispatches. The folder had been checked only after the download, as a raw ENOTDIR the MCP server could report only as `UNKNOWN`, and for `collect-remote` that came after a paid run and up to half an hour of waiting. `fetchCorpus` writes the package it downloaded through `writeBytes` (2026-09-28, ADR-0079): `fs.writeFileSync` emptied a package already at that name when the write failed. `bin/collect-remote.mjs` checks `--max-pages` (1..25), `--depth`, `--runner`, `--search-transport` and `--timeout` against the ranges and choices `collect.yml` enforces, before it asks for a token (2026-09-29, Arena break test): `--max-pages 8o` was dispatched as typed, and the paid run failed on the runner. A dispatch that never left names the cause of the failed fetch through `fetchFailure` (2026-09-30), still passed through `redact`; so does `.github/scripts/check-action-pins.mjs` for a pin it could not check. Both had printed the bare `fetch failed`. `getRun` and `listArtifacts` read their body through `readJsonBody` (2026-09-30): a 200 whose body is not JSON (a maintenance page, a proxy) is `DispatchError` `BAD_BODY` with the status and a remedy, where it had thrown a raw `SyntaxError` that `collect-remote` could only call `UNKNOWN`. `getRun`, `listArtifacts` and `downloadArtifact` reach GitHub through `callApi` (2026-09-30), so a request that never left is `DispatchError` `NETWORK` with its cause (`fetchFailure`, redacted) and a remedy, as a dispatch already was; they had thrown the raw `fetch failed`, which `collect-remote` could only call `UNKNOWN`. fetchCorpus treats names from the API as data: a wrapper entry is flattened to its file name, an artifact name holding a separator, `:`, `..` or a control character is refused (`ARTIFACT_NAME`) and nothing is written (2026-10-01). unwrapArtifact hands on a wrapper whose inner package cannot be decompressed unchanged, for the validator to judge, instead of throwing ZIP-READ (2026-10-01). `waitForRun` retries a failed poll that may pass on its own - no connection, a 429 or 5xx, a body that is not JSON - up to `MAX_POLL_RETRIES` (5) in a row, reset by a good poll and never past the deadline; 401, 403 and 404 stay fatal (2026-10-01). Every failed poll had ended a watch of up to 30 minutes with RUN_FAILED while the run went on. `getRun` gives a 401, 403 or 404 its own remedy (a missing run, a refused token), and the timeout remedy names `--run-id` (2026-10-01). `splitRepository` admits only two nonempty `[A-Za-z0-9._-]+` components other than `.` and `..`, so dispatch, run lookup, artifact listing and download refuse route-changing delimiters and control characters before their first request (#379, 2026-10-12). `queriesInput` refuses CR/LF in either list before trimming and serialization (#376, 2026-10-12), retaining existing empty-input and one-line normalization. A declared item cannot become additional workflow lines. `dispatchCollection` also uses `readJsonBody`: a successful non-JSON response is `BAD_BODY` with the proxy/maintenance-page remedy; valid JSON missing the run ID retains `NO_RUN_ID` (#377, 2026-10-12). Output folders resolve to absolute paths before access checks; the MCP caller passes that same path to fetchCorpus so the write and resource link identify the same file (#372, 2026-10-13). Default corpus downloads use a fresh private mkdtemp folder per call; existing or dangling symbolic links at the package name are refused as OUT_DIR before the write, and MCP describes the private default (#373). Dispatch and run reads classify a rate-limited 403 from headers or body as RATE_LIMITED, retaining Retry-After seconds and the UTC reset time; permission-only 403 remedies remain unchanged (#374). | ADR-0033, ADR-0032 | | `brief.mjs` | The phase-1 → phase-2 handoff: renders `research/BRIEF.md` from the corpus, and **owns the brief's shape** — `BRIEF_SECTIONS` (six sections, the two that are judged marked), the markers that distinguish *still the scaffold* from *the drafter wrote this* (`briefState` → `template \| legacy \| draft \| authored`), and the section reader (`briefSection` / `judgedSection`) that `lib/audit.mjs` consumes instead of regexes of its own. The renderer writes headings **from** that definition, so the writer and its readers cannot drift; the scaffold ships `BRIEF_FILE_MARKER`, which the substitution pass leaves alone — the alternative, matching the template's prose, made a sentence of English load-bearing. The verified table has **one row per claim and source, naming the unknowns it closes** (2026-09-26): two unknowns leading with the same E-row used to print its finding twice, word for word - three committed briefs carried such duplicates. **Every E-row an unknown cites** gets a row, not only the first (2026-09-29): a MoonAliza closure resting on E-03 and E-06 showed E-03 alone, and the row it most depended on never reached the builder. With no KNOWN-UNKNOWN declared, the unknowns section reports that status and names any CLOSED unknown citing no `P` row: the earlier primary-source assurance was removed in 2026-09-27 and the unconditional closure assurance in ADR-0143. Its commands are spelled by `documentCommand`, with the project's recorded `kitPath` first (ADR-0084). `--force` over a draft, authored or legacy brief keeps it as `research/BRIEF.md.bak-` (numbered on a second force that day) and says so (2026-09-27): it restored the TODOs and kept nothing of the answers it replaced. **Every draft is stamped** (2026-09-27, ADR-0055) with a hash of its own text, a hash of what it was drafted from (`briefInputsHash`: topic, intent, unknowns, evidence rows, and since ADR-0069 the map rows and the ledger and capture counts), and the verdict it was drafted under (`gate=`), which `bin/preflight.mjs` compares with its own verdict and notes when they differ. `draftStamp` reads the stamp back, wherever it stands (the last one, if several), and text after it is an edit like text before it (2026-10-03, ADR-0138): anchored to the end of the file, a line appended after the stamp - a reviewer's note, the `Reviewed by: agent` line - had made the brief read as unstamped, so a stale brief re-approved the build and `brief-stale` fell silent. An unedited draft is redrafted without `--force` and without a backup, and any edit keeps the refusal. A draft written while the gate failed had kept saying "Gate: FAIL" after PASS. A known unknown's day-one step drops a leading "Day one:" before it is labelled (2026-09-27), so a cell written that way is not rendered "Day-one verification: Day one: ...". **`reviewedBy(text)`** (ADR-0074) reads the brief's `Reviewed by: agent|human` line; the drafted brief carries a placeholder that reads as `undeclared`. A line naming both roles, or two lines that disagree, also read as `undeclared`; a line inside a code block is not a declaration, and an indented one is. A known unknown's Evidence cell that states what is known and then labels its day-one step mid-sentence renders as "Known so far" plus one "Day-one verification" line, never the label twice (2026-09-29). A cell the author wrote as the brief prints it, starting "Known so far:", keeps that label once too. `reviewedBy` reads only `agent` (2026-09-30, ADR-0107): a `Reviewed by: human` line reads as `undeclared`, and the drafted placeholder names only the agent. **PASS reports configured checks, not complete closure or factual correctness** (ADR-0143). The preamble preserves the evaluated warning reasons through `renderGateWarnings`; the known-unknown section reports declared status without claiming all unknowns are closed, including on FAIL. The contradictions placeholder asks for corroboration rather than asserting agreement. On FAIL or without an evaluation, the verified section explicitly labels its CLOSED rows as declarations whose verification has not succeeded or has not been evaluated. Six sections are retained. | ADR-0051, ADR-0084, ADR-0143 | -| `corpus.mjs` | Capture and Raw-path checks use projectFile; capture reads stay inside the capture folder by real path, without reading an internal private-file link (#351, ADR-0109 trigger). `readCaptures(root, { known, sketches })` reads only the captures a caller does not already hold and sketches them only when asked (2026-10-02, break-test pass 3 F6): the collector's per-fetch refresh had re-read, parsed and MinHash-sketched every capture on disk inside the lock - 41% of a 2000-page run - and `rememberCapture` had pushed every entry again, n(n+1)/2 copies after n fetches; it now remembers a file once. Sketches serve one gate check (`corroboration`) and are the gate's to compute. `readCaptures` indexes every capture by exact URL (`byUrl`, with the asked-for URL of a redirected fetch as an alias) and by `urlKey` (`byKey`, 2026-10-02): `cacheDecision` asks the exact spelling first and the key second, so a plan that spells a page differently from its capture - `www.`, a trailing slash, a fragment, http for https - is a cache hit and spends nothing; `rememberCapture` keeps both indexes for a page collected mid-run. Also builds two per-capture indexes alongside the entries, both because `readCaptures` is already holding every body: `sketches` (MinHash, for `corroboration`'s mirror grouping) and `renderFailures` (`countRenderFailures`, `RENDER_FAILURE_MARKERS` — how many times a page printed a notice saying its own content failed to load). Neither rides inside `captureEntry`, which is serialised into artifact manifests. The research corpus: **one reader** (`readCorpus`, `readCaptures`, `parseTable`, `parseCapture`, `readOverrides`) and **one writer family** (`appendRow`, `upsertRow`, `appendJsonLine`, `writeRaw` lives in collect), plus the **cache decision** over the capture index (`cacheDecision`) — the one freshness predicate the collector, the run, and phase 0 consume. Table machinery: `splitRow`/`escapeCell`/`tableRow`/`repairRowArity`. Malformation is reported, never absorbed. The capture index has one constructor (`captureEntry`, used by `readCaptures` **and** by the collector when it has just written a page) and one writer (`rememberCapture`), so the index a run keeps in memory holds the same shape as the index read from disk — callers store the entry they are handed and shape nothing. The format's **joins** are here too: `traceOf` (evidence row → the fetch that produced it and the capture behind it), `captureOf` (row → capture, via the by-file index the reader builds), and `claimOf` (unknown → the claim it rests on, first cited row, falling back to the unknown's own text). Five consumers re-derived those joins, including the subtle "an empty Raw cell matches on the URL alone" clause; now a format change lands in one place. A `research/kit.json` that does not parse is a `kit-unparsed` problem, blocking in `corpus-shape` like `plan-unparsed` (2026-09-27): the gate read it as absent and fell back to its default code paths, silently unguarding the ones the operator declared. A row with more cells than its header folds the extras into the table's prose column (`Finding`, `Unknown`, `Why it matters`, `Title`), not the last one, so the structured columns to its right stay in place, and the `table-arity` warning says a `|` inside a cell splits it and to write `\|` (2026-09-27): folded into `Raw`, a Finding quoting `a || b` corrupted the capture path and preflight reported the page missing. A `Raw` cell's backslashes are always read as separators (2026-09-27): splitting on `path.sep` converted them only on Windows, so a row written there passed on Windows and failed on a Linux builder. `readCaptures` does not read a capture that links outside the project; it records `capture-outside`, which corpus-shape blocks on (ADR-0076). `CAPTURE_MAX_BYTES` (10 MB) is checked with `stat` before a capture is read: larger is `capture-too-large`, unreadable is `capture-unreadable`, both blocking - a 100 MB file crashed every reader and a 600 MB one was skipped in silence (ADR-0077). An EVIDENCE Raw cell that lands outside the project is a blocking `raw-outside` problem and is never read; it counted as present before (2026-09-28). A duplicate key in `plan.json` or `kit.json` is reported as `plan-unparsed` / `kit-unparsed` (blocking), found by the strict reader in `release/json.mjs`: `JSON.parse` kept the last value silently (2026-09-28). `duplicateKey(text)` is exported (2026-09-30) so a command that spends reads a hand-edited JSON file by the same rule the gate does. `byUrl` also answers to the URL a redirected fetch was **asked for** (`requestedUrl`, read out of the capture's command line; 2026-10-02): a capture records where the bytes came from, the plan keeps naming where it asked, and the index that knew only the first missed the cache on every later run - the page was fetched, paid for and appended to the ledger and the table again as a "first capture". A capture of the asked-for URL itself always outranks an alias; `rememberCapture` keeps the same rule for a page collected mid-run. **The newest capture of a URL on one date** (2026-10-03, outside audit): `newer(entry, held, rank)` decides every `byUrl`/`byKey`/alias slot - by retrieval date, then the ledger's order (`readCorpus` reads the ledger first and hands `readCaptures` a capture-file -> fetch-position map), then a higher revision of the same name, then the later name; filename order alone reopened twelve same-day revisions with `.r9.md` current and a re-titled page under its older title, while the collector's own run (`rememberCapture`) had it right. `ledgerRank(entries)` builds that map for both readers, and `rememberCapture` takes the rank and decides through `newer` too (2026-10-03, second audit): it had replaced the direct-URL slot unconditionally, so the collector's refresh under the lock undid the reopen's answer and a stale snapshot took `.r9.md` as its cache hit. **One slot rule on every path** (2026-10-03, fourth audit round): `rememberCapture` decides the current slot for a known file too - a refetch whose bytes match an earlier capture reuses that file and records it last (A -> B -> A) - with a direct capture over any redirect alias, the fetch that just happened (`latest`) over whatever was there, then `newer`; `newer` lets a fetch on record beat a file no fetch produced. Rank alone had let a later alias keep the slot from a direct capture where the reopen gives it to the direct one. **A data row whose ID is not in its table's form is recorded, never dropped** (2026-10-03, output-reliability audit G2): `readCorpus` pushes a `malformed-id` problem - artifact, the ID, the parser's line, a remedy naming the form - for every non-empty unknowns, evidence or subtopic ID the filters would discard; `U_99` had been neither an unknown nor a problem, and a blocking question had vanished on a typo. A row empty in every cell stays silent, because the templates ship empty tables; an empty ID cell on a row that holds content is recorded like a typo, since the row says its piece and no check can find it (the first version kept every empty ID silent on the premise that the templates ship empty rows; review of G2, 2026-10-03). **A row-shaped line after the table ended is recorded, never dropped** (2026-10-03, gap audit rank 1): `parseTable` scans from the table's end to the next heading and pushes a `table-split` problem for every row-shaped line there, naming its line and first cell; a blank line ends a table as GFM renders it, so GitHub showed such a row as prose while the parser dropped it in silence, and a blocking unknown written below a blank line had left the gate, which printed PASS. A header followed by its separator starts another table and is not flagged (the root `research/SOURCES.md` holds two). The row stays out of the table, as rendered; `hygiene` fails it by name. | ADR-0003 | `readCaptures` skips `*.source.html` (ADR-0140, 2026-10-03): a capture's source sibling is named and verified by the ledger and is never a capture - not indexed, cited, graded or exported. The scan runs to the end of the table's SECTION - a heading of the same or a higher level than the one above the table; a deeper `###` note does not end it and `#tag` prose is not a heading - skips fenced code and HTML comments, where GitHub shows a row as code or hides it, treats a lone separator as not a row, and names a second table whose separator is malformed once, as that (breaker review, 2026-10-04: a `###` note had hidden a stray row below it, and a row in a code fence had failed the gate). A `*.source.html` no ledger entry names is a `source-unnamed` problem: a sibling is never indexed, so nothing else would see a planted one. +| `corpus.mjs` | Capture and Raw-path checks use projectFile; capture reads stay inside the capture folder by real path, without reading an internal private-file link (#351, ADR-0109 trigger). `readCaptures(root, { known, sketches })` reads only the captures a caller does not already hold and sketches them only when asked (2026-10-02, break-test pass 3 F6): the collector's per-fetch refresh had re-read, parsed and MinHash-sketched every capture on disk inside the lock - 41% of a 2000-page run - and `rememberCapture` had pushed every entry again, n(n+1)/2 copies after n fetches; it now remembers a file once. Sketches serve one gate check (`corroboration`) and are the gate's to compute. `readCaptures` indexes every capture by exact URL (`byUrl`, with the asked-for URL of a redirected fetch as an alias) and by `urlKey` (`byKey`, 2026-10-02): `cacheDecision` asks the exact spelling first and the key second, so a plan that spells a page differently from its capture - `www.`, a trailing slash, a fragment, http for https - is a cache hit and spends nothing; `rememberCapture` keeps both indexes for a page collected mid-run. Also builds two per-capture indexes alongside the entries, both because `readCaptures` is already holding every body: `sketches` (MinHash, for `corroboration`'s mirror grouping) and `renderFailures` (`countRenderFailures`, `RENDER_FAILURE_MARKERS` — how many times a page printed a notice saying its own content failed to load). Neither rides inside `captureEntry`, which is serialised into artifact manifests. The research corpus: **one reader** (`readCorpus`, `readCaptures`, `parseTable`, `parseCapture`, `readOverrides`) and **one writer family** (`appendRow`, `upsertRow`, `appendJsonLine`, `writeRaw` lives in collect), plus the **cache decision** over the capture index (`cacheDecision`) — the one freshness predicate the collector, the run, and phase 0 consume. Table machinery: `splitRow`/`escapeCell`/`tableRow`/`repairRowArity`. Malformation is reported, never absorbed. The capture index has one constructor (`captureEntry`, used by `readCaptures` **and** by the collector when it has just written a page) and one writer (`rememberCapture`), so the index a run keeps in memory holds the same shape as the index read from disk — callers store the entry they are handed and shape nothing. The format's **joins** are here too: `traceOf` (evidence row → the fetch that produced it and the capture behind it), `captureOf` (row → capture, via the by-file index the reader builds), and `claimOf` (unknown → the claim it rests on, first cited row, falling back to the unknown's own text). Five consumers re-derived those joins, including the subtle "an empty Raw cell matches on the URL alone" clause; now a format change lands in one place. A `research/kit.json` that does not parse is a `kit-unparsed` problem, blocking in `corpus-shape` like `plan-unparsed` (2026-09-27): the gate read it as absent and fell back to its default code paths, silently unguarding the ones the operator declared. A row with more cells than its header folds the extras into the table's prose column (`Finding`, `Unknown`, `Why it matters`, `Title`), not the last one, so the structured columns to its right stay in place, and the `table-arity` warning says a `|` inside a cell splits it and to write `\|` (2026-09-27): folded into `Raw`, a Finding quoting `a || b` corrupted the capture path and preflight reported the page missing. A `Raw` cell's backslashes are always read as separators (2026-09-27): splitting on `path.sep` converted them only on Windows, so a row written there passed on Windows and failed on a Linux builder. `readCaptures` does not read a capture that links outside the project; it records `capture-outside`, which corpus-shape blocks on (ADR-0076). `CAPTURE_MAX_BYTES` (10 MB) is checked with `stat` before a capture is read: larger is `capture-too-large`, unreadable is `capture-unreadable`, both blocking - a 100 MB file crashed every reader and a 600 MB one was skipped in silence (ADR-0077). An EVIDENCE Raw cell that lands outside the project is a blocking `raw-outside` problem and is never read; it counted as present before (2026-09-28). A duplicate key in `plan.json` or `kit.json` is reported as `plan-unparsed` / `kit-unparsed` (blocking), found by the strict reader in `release/json.mjs`: `JSON.parse` kept the last value silently (2026-09-28). `duplicateKey(text)` is exported (2026-09-30) so a command that spends reads a hand-edited JSON file by the same rule the gate does. `byUrl` also answers to the URL a redirected fetch was **asked for** (`requestedUrl`, read out of the capture's command line; 2026-10-02): a capture records where the bytes came from, the plan keeps naming where it asked, and the index that knew only the first missed the cache on every later run - the page was fetched, paid for and appended to the ledger and the table again as a "first capture". A capture of the asked-for URL itself always outranks an alias; `rememberCapture` keeps the same rule for a page collected mid-run. **The newest capture of a URL on one date** (2026-10-03, outside audit): `newer(entry, held, rank)` decides every `byUrl`/`byKey`/alias slot - by retrieval date, then the ledger's order (`readCorpus` reads the ledger first and hands `readCaptures` a capture-file -> fetch-position map), then a higher revision of the same name, then the later name; filename order alone reopened twelve same-day revisions with `.r9.md` current and a re-titled page under its older title, while the collector's own run (`rememberCapture`) had it right. `ledgerRank(entries)` builds that map for both readers, and `rememberCapture` takes the rank and decides through `newer` too (2026-10-03, second audit): it had replaced the direct-URL slot unconditionally, so the collector's refresh under the lock undid the reopen's answer and a stale snapshot took `.r9.md` as its cache hit. **One slot rule on every path** (2026-10-03, fourth audit round): `rememberCapture` decides the current slot for a known file too - a refetch whose bytes match an earlier capture reuses that file and records it last (A -> B -> A) - with a direct capture over any redirect alias, the fetch that just happened (`latest`) over whatever was there, then `newer`; `newer` lets a fetch on record beat a file no fetch produced. Rank alone had let a later alias keep the slot from a direct capture where the reopen gives it to the direct one. **A data row whose ID is not in its table's form is recorded, never dropped** (2026-10-03, output-reliability audit G2): `readCorpus` pushes a `malformed-id` problem - artifact, the ID, the parser's line, a remedy naming the form - for every non-empty unknowns, evidence or subtopic ID the filters would discard; `U_99` had been neither an unknown nor a problem, and a blocking question had vanished on a typo. A row empty in every cell stays silent, because the templates ship empty tables; an empty ID cell on a row that holds content is recorded like a typo, since the row says its piece and no check can find it (the first version kept every empty ID silent on the premise that the templates ship empty rows; review of G2, 2026-10-03). **A row-shaped line after the table ended is recorded, never dropped** (2026-10-03, gap audit rank 1): `parseTable` scans from the table's end to the next heading and pushes a `table-split` problem for every row-shaped line there, naming its line and first cell; a blank line ends a table as GFM renders it, so GitHub showed such a row as prose while the parser dropped it in silence, and a blocking unknown written below a blank line had left the gate, which printed PASS. A header followed by its separator starts another table and is not flagged (the root `research/SOURCES.md` holds two). The row stays out of the table, as rendered; `hygiene` fails it by name. | ADR-0003 `readLedger` tolerates only a leading editor BOM outside the hashed entries, without rewriting evidence or relaxing entry/chain verification (#296). | `readCaptures` skips `*.source.html` (ADR-0140, 2026-10-03): a capture's source sibling is named and verified by the ledger and is never a capture - not indexed, cited, graded or exported. The scan runs to the end of the table's SECTION - a heading of the same or a higher level than the one above the table; a deeper `###` note does not end it and `#tag` prose is not a heading - skips fenced code and HTML comments, where GitHub shows a row as code or hides it, treats a lone separator as not a row, and names a second table whose separator is malformed once, as that (breaker review, 2026-10-04: a `###` note had hidden a stray row below it, and a row in a code fence had failed the gate). A `*.source.html` no ledger entry names is a `source-unnamed` problem: a sibling is never indexed, so nothing else would see a planted one. | `transport.mjs` | **Adapter selection**, and the one reader of the operator's transport choice — now for **two sides** (ADR-0027). The *fetch* ladder is unchanged: `--transport`, `RESEARCH_KIT_TRANSPORT`, the config's `transport`, then auto-detection (`probeFirecrawl` — authenticated or anonymous CLI wins, otherwise keyless). The *search* ladder mirrors it: `--search-transport`, `RESEARCH_KIT_SEARCH_TRANSPORT`, the config's `searchTransport`, then one auto-detect rule — a SerpAPI key wins, otherwise **the search side IS the fetch provider**, so a machine with no key behaves exactly as before. `selectTransport` returns `{ name, adapter, why }` still meaning the FETCH side, with `search` added alongside; that is what let six call sites stay unedited. Exposes `TRANSPORTS`/`TRANSPORT_NAMES` (whole-shape adapters), `SEARCH_PROVIDERS`/`SEARCH_PROVIDER_NAMES` (a **separate** map — a search-only module in the first would make `ADAPTER_SHAPE` a claim it no longer keeps), `FETCH_SHAPE`, `SEARCH_SHAPE`, `satisfies`, `isSearchOnly`, `selectSearch`. `isSearchOnly` reads a module's declared `CONTRACTS` rather than duck-typing, because a search-only module exports `scrape` on purpose — to refuse by name. **`unusedKeyNote(chosen, env)`** (2026-09-27) is the line `research.mjs` and `decompose.mjs` print under the transport line when `FIRECRAWL_API_KEY` is set but auto-detection fell back to keyless because the CLI is not on PATH: the key is unused, and the line gives the install command. Until then only doctor said so. An explicit `--transport http-keyless` gets no note. **`browser` is the third transport** (2026-09-28, ADR-0088), chosen only by name; `SEARCH_PROVIDERS` holds only the transports that search, and `selectSearch` gives a fetch-only transport the keyless search, saying so. With a SerpAPI key the merged search pairs SerpAPI with the fetch side only if it searches; a fetch-only transport (the browser) hands its share to `http-keyless` (2026-09-29, found by the first live browser run). `SEARCH_PROVIDERS` includes `searxng` (2026-09-30, ADR-0104), selected only explicitly; `selectSearch` reports it `notReady` without an instance URL, as it does SerpAPI without a key. `selectSearch` hands the searches of a Firecrawl CLI with no key to the keyless route, as it does a fetch-only transport's: Firecrawl still scrapes for that CLI but refuses its searches ("Alexandria requires a Firecrawl API key", CLI 1.24.6, 2026-10-01). **`canSearch`** sits beside `isSearchOnly` because both are a transport's own capability, asked of any provider: whether it can search at all — the browser fetches only (ADR-0088), and a Firecrawl CLI with no key has a search function and no search. **`SEARCH_KEY`** (2026-10-07, ADR-0147) is the metered search provider's credential for a surface that must not name the vendor - its env and config names, `read`, `state` (set, and from where; never the key) and `redact` - consumed by the desktop panel. Registry lookups accept only own members, so inherited names such as constructor, toString and __proto__ receive the existing unknown-name refusal at every precedence level (#349, 2026-10-13). | ADR-0005, ADR-0027, ADR-0088, ADR-0147 | | `serpapi.mjs` | A **search-only provider**: the first module here that satisfies one contract instead of the whole adapter shape, declared as `CONTRACTS = ['search']`. Produces no capture, so nothing it returns reaches the fetch ledger and `transport-provenance` is untouched. Sends exactly `engine`, `q`, `api_key` — **no result-count parameter**, because none is documented and a response costs the same whatever it carries, so `limit` is applied client-side; `no_cache` and `async` are never sent, which keeps the vendor's free 1-hour cache. The key is read from `SERPAPI_API_KEY` or the config's `serpapiKey`, **never from a repository**, and travels to the child on **stdin** rather than argv so it never enters the process table. The request URL — the one string that contains the key — is built in the child and never returned; `redact()` scrubs it from vendor error text on the way back, and `command()` renders it pre-redacted. **A query that contains the key is refused before transmission**, because `cmd` is written into the hash-chained ledger (a committed file) and because sending it would store the credential as a search term on the vendor's systems for 31 days; redacting the rendering alone would have fixed the smaller half. Async `fetch` behind a synchronous shape via the same self-exec rendezvous `http-transport.mjs` uses. `status()` answers from configuration alone and **declines to report a remaining allowance**, because no endpoint this project has captured reports one. `requestUrl(query, key, endpoint)` is exported so the exact parameter set is testable rather than buried in the child — the guarantee worth pinning is negative, and negative guarantees rot silently. The **endpoint is overridable and guarded**: `allowedEndpoint` permits the vendor's own host over https, or loopback, and nothing else. That exists so the whole path — parent, `spawnSync`, child, real `fetch`, real socket — can be driven against a local stand-in instead of being exercised once by hand and called verified; an overridable endpoint on a request carrying an API key is otherwise an exfiltration route, so **both halves check it**, the parent before spawning and the child before fetching, because the child reads its job off a pipe and does not get to trust it. The one thing no stand-in can prove — that the two meters really are separate — is covered by `test/live-collect.test.mjs`, an **opt-in** test (`RESEARCH_KIT_LIVE=1` plus a key) that drives `bin/research.mjs` through a real collecting run in a temp project and asserts the Firecrawl spend against the pages fetched. Opt-in because running it on every suite would cost every developer credits forever; existing because "it costs money" is a reason to gate a test, not to skip it. **It also says what the meter's numbers mean** (`METER_NOTES`, 2026-09-26): the 50/hour and 250/month `searchUsage` reports are the *documented* Free Plan (E-25, E-27), while SerpAPI's own Account API reported 250/hour for this account (U-9); and SerpAPI's month is a billing cycle, not the calendar month the kit counts (U-10). `research.mjs --status` prints the selected search adapter's notes, so the vendor stays named only here (NFR-3) and a machine without a key hears nothing about it (FR-5) **The key it searches with is the key that selected it**: `search()` reads the machine config when none is passed, as `selectSearch` does. It defaulted to `null` until 2026-09-26, so a key held only in `~/.agents/research-kit.config.json` selected SerpAPI and then failed every search "no SerpAPI key" - a merged run degraded silently to the other provider on every query **It reads the vendor's own meter** (`account()`, ADR-0040): `serpapi.com/account.json`, free and uncounted (E-26), through the same child-process path, endpoint lock and redaction as `search()`, copying only whitelisted fields (the payload carries the email, the account id and the key). `research.mjs --status` asks the selected search adapter for it and prints a `vendor meter` line; no key, no call. `status()` stays offline. **Its child uses a configured proxy** (2026-09-27, ADR-0047): `runJob` starts it with `fetchEnv(env)` - it had been started without, after the keyless child was fixed, so behind a proxy every search went around it and failed "fetch failed"; `search()` and `account()` hand their `env` to the child. A child past its timeout is reported as "SerpAPI did not answer within Ns", not `spawnSync ... ETIMEDOUT` (2026-09-27, seen on a real run) The child runs with `maxBuffer: CHILD_OUTPUT_LIMIT` and names an overflow (2026-09-28, ADR-0082): under the 1 MiB default a large answer was lost to ENOBUFS. `normalizeSearch` drops a result row that is not an object and reads a field of the wrong type as empty (2026-09-28): one null row threw and discarded every valid result beside it. The child reads the answer through `boundedText` (2026-09-28, ADR-0082): with `response.text()` a 17 MiB answer was accepted, since the output limit bounds only what reaches the parent. | ADR-0027, ADR-0040, ADR-0047 | | `searxng.mjs` | The second **search-only provider** (2026-09-30), with the contract `serpapi.mjs` defines: `GET /search?q=...&format=json` against a SearXNG instance the operator runs, named by `SEARXNG_URL` or `searxngUrl`. No key and no meter. JSON is off in a default instance, so an HTTP 403 is named as "enable json in `search.formats`", never "no results"; a row with no string `url` is dropped (the docs type it `str \| None`). Chosen only explicitly. An instance behind basic auth (`http://user:pass@host`, 2026-09-30) has its credentials taken out of the URL - fetch refuses one that carries them - and sent as an `Authorization` header (`basicAuth`); every message that names the instance shows it through `shownInstance`, password as `***`. Every search had failed, and the error printed the password. `shownInstance` redacts a password from any URL that parses, whatever its scheme, and by pattern from one that does not (2026-09-30): a mistyped `htp://user:pass@host` - exactly when the instance is shown in an error - had printed the password. `basicAuth` sends a literal `%` as typed instead of throwing `URIError`. A password holding an unencoded / ? # or @ - which no URL parser accepts - is redacted by pattern up to the last @ (2026-09-30). requestUrl builds the endpoint on the pathname, so a query or fragment on the instance URL no longer drops its sub-path (2026-09-30). | ADR-0104 | @@ -194,7 +200,7 @@ is an open decision (2026-09-30): | `browser-transport.mjs` | **The `browser` transport** (2026-09-28, ADR-0088): renders a page with a local Chromium, `--headless --disable-gpu --virtual-time-budget --dump-dom ` as an argv array, and grades the DOM with the keyless extractor (`mainContent`, `htmlToMarkdown`, `gradeCompleteness`). `findBrowser` (config `browserPath`, `RESEARCH_KIT_BROWSER`, usual paths), `browserArgs` (`--proxy-server` from the proxy environment, `--no-sandbox` only as root, never a TLS-weakening flag), `chromeErrorOf` (Chromium's own error page, which `--dump-dom` prints with exit 0). Fetch only: `CONTRACTS = ['fetch']`, `map` is the keyless map, and `transport.mjs` searches keyless for it. A signal other than the timeout's is reported as `killed by ` with Chromium's own cause - the last FATAL stderr line, else the last ERROR line, else the last line - never as the timeout (2026-09-29). On Linux it prefers Chrome stable (`/opt/google/chrome/chrome`), the binary Ubuntu 23.10+'s AppArmor lets sandbox itself, and a "No usable sandbox" crash names the remedies that keep the sandbox - never `--no-sandbox` for non-root (ADR-0092). Its recorded command is `browser ` (2026-09-30): it had used the keyless formatter whole, so every browser capture's ledger entry read `http-keyless browser `. `scrape` renders through `renderGuarded`, which runs the guard child (`browser-guard.mjs`) under spawnSync and reads back the DOM, Chromium's exit and the guard's refusals; a page that landed on a refusal is reported by name and not captured (ADR-0118, superseding the ADR-0115 pre-check). The refusal named is the one the dump holds - the guard's refusal page carries its host and reason - and only a dump that is Chromium's own error page falls back to the last recorded (2026-10-02): a dead third-party host refused after the navigation, or on a no-network host the browser's own www.google.com, had been named as what the page reached for. `browserArgs` never names a proxy: the child adds the guard's, and switches Chromium's background traffic off - which Chromium 141 ignores for its own services, so the guard drops those by name (ADR-0124). A timeout is judged by what Chromium printed, not by its exit (2026-10-01, CI, three times in one afternoon on Windows and Ubuntu): a whole dump after the timeout is the render, because Chromium prints the DOM once when its own budget says the page is done and a hang at exit does not unrender it; a dump holding the guard's refusal is the refusal; no dump at all is the timeout, and the message now carries the last thing Chromium said on stderr, which the timeout branch used to drop. Chromium is given its own deadline, `--timeout`, ten seconds under the transport's kill timeout and never below five (2026-10-01): with the virtual-time budget alone, one resource that never answers froze the budget - virtual time advances only when the page is idle - so headless never dumped, printed nothing, and was killed with nothing to show, which is what CI's four hangs that afternoon looked like and what a page with a never-answering image reproduces here. At the deadline Chromium dumps the DOM it has and exits, so the page is captured as it stands. A capture is conservatively graded `partial` when the guard still has pending loads at process completion or process elapsed minus startup reaches the configured deadline threshold. `cutAtDeadline` reports the configured loading deadline and the guard's request state **at process completion** (2026-10-08), rather than claiming an observed DOM-dump time or assigning the historical wait to the browser: neither a late process end nor the final pending snapshot proves those facts. The same note rides a refusal's message. A timeout's message says what the launch cost - "took N s to make its first request" or "never made a request in N s" (ADR-0119). **The origin's HTTP status comes from Chromium's own net log** (2026-10-03, ADR-0137, output-reliability audit G5): `--dump-dom` reports no status, so every browser capture carried `statusCode: ''` and the collector's >= 400 rule, which reads a number, never fired - an origin's 403 page was graded `full`. `documentStatusFromNetLog(log, requested)` reads the log the guard child had Chromium write (`--log-net-log`): event and source names are looked up through the log's own `constants`, never by number; the document is the `URL_REQUEST` source whose first `URL_REQUEST_START_JOB` matches the requested URL (compared in canonical form without the fragment, a `main frame` request winning); each later start on that source is a redirect, and the status is the `HTTP/x NNN` line of the last `HTTP_TRANSACTION_READ_RESPONSE_HEADERS` in the final job. `scrape` then fails a non-2xx answer the way the keyless transport does (`HTTP N - the server said: ""`), refuses a final URL that is internal by spelling and not the requested host:port (the guard's own exemption), carries `statusCode` as a number and the post-redirect URL, and grades a render whose log gave no status `partial` with `omitted` saying the origin's status was not observed (rule 2 of the ADR) - never guessed from the page's text. A JavaScript navigation is a new source, not a redirect, and is not followed. The guard child's `EINTERNAL` - a name asked for refused before any launch under an explicit refusal - is reported as that refusal, not as a browser that could not start (ADR-0155, 2026-10-11). `scrape` and `status` given no `config` read the machine's with `loadConfig(env)` (2026-10-11): the collector calls `runScrape(url)` with no options, and the empty default meant `browserPath` was never consulted. | ADR-0088, ADR-0092 | The rendered DOM is returned as `source` beside the Markdown (ADR-0140, 2026-10-03), so the collector keeps the text the capture was converted from. **`guardChildTimeout(timeout)`** (2026-10-04, break-test pass 6 F1): the parent waits `2 × timeout + 10 s` for the guard child - the launch allowance, the render budget (each `timeout`, ADR-0119) and 10 s for the child to kill the browser and report. At `timeout + 10 s`, as before, the parent was itself the timeout for any render whose launch took more than the grace, and the child's record of where the time went died with it; the verdict read "never made a request in 0 s". A child the parent gave up on is now named as such, with the time it was given; the result carries `gaveUp: true`, and the transport reads that flag, never the message's wording (GPT's review, 2026-10-04). | `browser-guard.mjs` | **The guard proxy Chromium renders through, and the child that runs both** (ADR-0118). `startGuard` listens on a loopback port and judges every plain request and CONNECT with `judge`: the keyless rule (`isInternal`), a name by the addresses it resolves to, the connection pinned to the address judged; only the `host:port` asked for is exempt, `allowInternal` is the operator's own decision, and a configured proxy is the upstream (absolute URLs, CONNECT passed through, `Proxy-Authorization` from the URL). A refusal is a 403 carrying `REFUSAL_MARKER` and is recorded. `renderThroughGuard` runs the browser with `--proxy-server` for the guard and `--proxy-bypass-list=<-loopback>`, its own process group on POSIX (killed whole on a timeout), the DOM capped at `MAX_PAGE_BYTES`, the exit reported after a short grace even when a forked helper holds stdout open. Run as a process it takes one job on stdin and prints one JSON line. The guard keeps a record of every request it carried - `seen`, each with its outcome (a status, `refused`, `tunnel`, an error, or `unanswered` once the guard closes) - and `pending()` lists the ones still unanswered, with how long they have waited; the child reports `requests`, `pending` and `elapsedMs` beside the dump, and `seen` as a timeline from the launch - each request's offset, duration and outcome - because a Windows run then stalled 28 s with nothing pending at the guard, so where the time went became the question (2026-10-01). The child keeps the first 30 lines of the browser's stderr as well as the last 60, with the omission counted between them, and stamps the launch (`startedAt`): a hung Windows launch's tail showed profile creation 43 s after the launch, so the wait was in the head the tail could not show (2026-10-01). The render timeout starts at the browser's first request through the guard, and the launch is allowed as long again (ADR-0119): the child arms the kill at launch and re-arms it on the first request, and reports `startupMs`: three Windows CI runs in a row ran the first launch to its deadline and nothing said on what, and the guard is the one place every load passes through. **The browser's own service traffic is dropped** (2026-10-02, ADR-0124): `BROWSER_SERVICE_HOSTS` names Chromium's service hosts - `clients*.google.com`, `*.clients.google.com`, `mtalk.google.com`, `accounts.google.com`, `www.google.com`, the update, safe-browsing, variations and autofill endpoints on `googleapis.com`, `*.gvt1.com`, `dl.google.com` - and `isBrowserService` answers for one; a request there is answered 403 with `DROP_MARKER`, logged `dropped`, and is neither judged, carried nor counted as a refusal, the page asked for exempt by host and port. Rendering a plain loopback page, Chromium 141 had made nine such connections per render, carried as external, and on a no-network host the last of them - `www.google.com`, unresolvable - became the verdict of the LIVE test about an internal address. **A request before the page's own is the browser's** (ADR-0125): with `pageFirst`, which `renderThroughGuard` passes, every request is dropped until the host and port asked for has been requested - CI's Chrome stable preconnected to `www.gstatic.com` 256 ms into a render, before the page, and that host is one real pages use too; the order settles what a list cannot. The CONNECT socket gets its error listener before any answer is written (2026-10-02): a browser resets a CONNECT it has given up on with the 403's body unread, and an ECONNRESET on the handed-over socket with no listener was an uncaught exception that killed the guard child mid-render - an empty report, read by the transport as "the browser exited null", on three CI legs at random points of the render. **The child has Chromium write its net log** (2026-10-03, ADR-0137): `renderThroughGuard` makes a private folder (`rk-netlog-`), adds `--log-net-log=<dir>/net-log.json` before `--dump-dom`, reads the file after the render (a log over `MAX_PAGE_BYTES` is dropped as empty) and deletes the folder on every way out; a folder that cannot be made renders without the flag, which the transport grades as status not observed. The child also reports `allowInternal`, the internal-address decision it made by resolving the requested URL, so the transport judges the final URL by the same rule. The log travels in full through the child's JSON stdout and is never written to the corpus. **A name asked for is judged by what it resolves to, exempt or not** (ADR-0155, 2026-10-11): `judge` refuses a name resolving inward, or `localhost`, unless the caller allowed this machine's network, whether or not it is the host and port asked for; the exemption covers only the literal-address judgment. Under `allowInternalRedirects: false` the child judges a name asked for before launching and answers `EINTERNAL` with no render. | ADR-0118, ADR-0124, ADR-0125 | | `collect.mjs` | One URL's journey: `collectOne` (cache, force, fail entries, evidence/source rows) and `writeRaw` (the capture writer, which hands back the corpus's own index entry). Consumes the `runScrape` seam; stamps `transport` and `completeness` from what the adapter declared — invents neither. The Finding cell a row is born with is `lib/finding.mjs`'s: the collector **imports** `firstFinding` and does not re-export it, so the extractor has one address. A fetch reports **why it happened in words** (`whyFetched`: first capture, refreshed with the age, or --force), not `cacheDecision`'s code - verbatim, a fresh fetch logged as "collected <url> - not-collected" (found in a live-collection log, 2026-09-26). **`writeRaw` never overwrites a capture with different bytes** (2026-09-27, ADR-0058): a same-day re-collection of an unchanged page reuses the file, and a changed page is written as `<name>.r2.md` (then `.r3`...), which sorts after the original so `readCaptures` still takes it as the day's latest. The earlier reading had been replaced, and its ledger hash broken. **An error page is a failed fetch** (2026-09-28): a result whose `statusCode` is 400 or more is recorded as a `fail` ledger entry and writes no capture and no EVIDENCE row - Firecrawl answered ok with Google's "Error 503" page and the kit had filed it as evidence; the keyless transport already refused non-2xx answers. `recentlyGone` (ADR-0100) answers `gone` without fetching when the ledger's last entry for the URL is a 404 or 410 inside the refresh window. writeRaw refuses (`OUTSIDE_PROJECT`, a named write refusal in core) a link at the capture's generated name - dangling or not - and a capture folder whose real path leaves the project, before it reads or writes; a project living under a link is judged by real path and still writes (2026-10-01). A front-matter value with a line break is written on one line, so an adapter title or error cannot add a field; the body is written verbatim (2026-10-01). A page whose text exceeds `CAPTURE_MAX_BYTES` is a **failed fetch**, named with its size and the limit, before anything is written: the reader refuses such a file as `capture-too-large` and the gate blocks on it, so a 40 MB answer had become a capture, a ledger entry and a row the corpus could not hold (2026-10-02, break-test pass 3). `refreshCaptures`, the re-read inside the lock before every fetch, asks `readCaptures` only for files the snapshot does not hold and for no sketches (2026-10-02): per-capture cost at 400 rows went from 74 ms to 13 ms. A forced re-fetch that comes back byte-identical reuses the capture (`writeRaw` never overwrites) and lets the row that cites it stand - the fetch is in the ledger, the reason says `the page is unchanged, so E-01 stands`; it had appended a second row that superseded the first, so every unknown citing unchanged content had to move (break-test pass 3, 2026-10-02). A changed page still goes to `.r2` with a row of its own. `writeRaw` writes the body with LF line endings only, whatever the transport received (2026-10-02): the scaffold's `.gitattributes` makes git store a capture as LF, and a keyless capture written with the CR bytes copyright.gov served hashed one way on the collector and checked out another way everywhere else, failing handoff on a capture nobody had touched. The one such capture already committed (a licence decision project, E-18) is pinned byte for byte with a `-text` attribute in that project. `refreshCaptures` reads the new captures with the ledger's rank (2026-10-03): one ordering rule with the reopen, so a capture another collector wrote since the snapshot is current only if its fetch came later. The collector remembers its own fetch as `latest`, and `refreshCaptures` decides every known capture the ledger names again with the rank, since a reused file's position moves while its bytes do not (2026-10-03). | ADR-0016 | **The text a capture was converted from is kept beside it** (2026-10-03, ADR-0140, gap audit rank 3): `writeSource(root, captureFile, source, markdown)` writes `<capture>.source.html` (`sourcePathFor`), LF-folded like the body, and returns its path and hash for `collectOne` to hand to `appendFetch`; nothing is kept for a verbatim answer (the source is the capture), a source over `CAPTURE_MAX_BYTES`, or a sibling already on disk with other content, which is never overwritten. The converted Markdown was all the corpus kept, so G3's conversion defect was permanent and the ledger certified it. When a source was given and is not kept, `writeSource` returns `{ omitted }` with the reason - over `CAPTURE_MAX_BYTES`, a sibling already on disk with other content, a sibling name that cannot be written - and the ledger entry carries it as `sourceOmitted` (breaker review, 2026-10-04): a link planted at the sibling's name had made `assertCapturePath` throw after the capture was written, leaving an orphan capture with no ledger entry that the next run took for a cache hit; the write is now inside the try, and the fetch is ledgered without a source. -| `finding.mjs` | The auto-extracted **Finding** cell: `firstFinding(markdown, fallback)` — page markdown in, one line of prose out — over the heuristics behind it (boilerplate/tracking/nav/campaign regex families, the sentence splitter, the table reader, the tier-hero block skip, the graded score, the 220-character cap). Two arguments over ~460 lines: a deep module that was buried in the middle of another one until ADR-0016. Pure by construction — no filesystem, no corpus, no adapter — the one module a caller can exercise with a string. **`findingWithContext` is the real entry point** and returns `{ finding, excerpt, heading, signals, score }`: the same selection, plus where on the page it came from, the surrounding paragraph bounded to `EXCERPT_CHARS`, and which signals fired. `firstFinding` is a thin wrapper over it, so the sentence the collector writes and the excerpt a reviewer reads cannot be chosen by different code. **Ranking is one declared table, `RULES`** — 21 named rules, each with a weight and the reason for it, and `explainScore` returns the rules that fired alongside the total. Weights used to be two separate runs of `value += n`, one in `score()` and one in the selector, so the priority order could not be stated without executing the code and the number a reviewer saw could differ from the number that chose the sentence; a test now asserts no weight is applied outside `RULES`, and that the reported signals sum to the score. Heading context is an **ancestry stack**, not the last heading seen, so a claim under `#### Capacity` still knows it sits inside `## Pricing`. The policy is evidence-first: a measured quantity or a metering rule (**how** usage is counted, which often carries no digit) is strongest; where a sentence sits counts as much as what it says, so a terms-bearing heading and an FAQ answer both score while blog and careers copy is pushed down; and things that only look concrete — one row of a pricing comparison matrix, a number inside a sign-up prompt — are penalised enough to lose to plain prose. The `comparison-row` rule's tier test is three linear scans (`tierThenPeriodThenDigit`, per line, since `.` never crossed a newline) instead of `\b(plan|tier)\b.*\b(month|year|user)\b.*\d`, which was quadratic in the row - every tier word on a non-matching row scanned to its end before failing - and cost 15 s on a 309 KB row, the size of a line this corpus already holds, twice per row (break-test 2026-10-01). | ADR-0016 (superseding ADR-0009) An underscore inside a word is the word (2026-10-02): `stripMarkdown` dropped every `_` as emphasis, so a drafted finding named `return_run_details` as "returnrundetails" and `check_suite_id` as "checksuiteid" - parameters no page carries, and a quote copied from the draft could never be found; emphasis marks at a word's edge still go. | +| `finding.mjs` | The auto-extracted **Finding** cell: `firstFinding(markdown, fallback)` — page markdown in, one line of prose out — over the heuristics behind it (boilerplate/tracking/nav/campaign regex families, the sentence splitter, the table reader, the tier-hero block skip, the graded score, the 220-character cap). Two arguments over ~460 lines: a deep module that was buried in the middle of another one until ADR-0016. Pure by construction — no filesystem, no corpus, no adapter — the one module a caller can exercise with a string. **`findingWithContext` is the real entry point** and returns `{ finding, excerpt, heading, signals, score }`: the same selection, plus where on the page it came from, the surrounding paragraph bounded to `EXCERPT_CHARS`, and which signals fired. `firstFinding` is a thin wrapper over it, so the sentence the collector writes and the excerpt a reviewer reads cannot be chosen by different code. **Ranking is one declared table, `RULES`** — 21 named rules, each with a weight and the reason for it, and `explainScore` returns the rules that fired alongside the total. Weights used to be two separate runs of `value += n`, one in `score()` and one in the selector, so the priority order could not be stated without executing the code and the number a reviewer saw could differ from the number that chose the sentence; a test now asserts no weight is applied outside `RULES`, and that the reported signals sum to the score. Heading context is an **ancestry stack**, not the last heading seen, so a claim under `#### Capacity` still knows it sits inside `## Pricing`. The policy is evidence-first: a measured quantity or a metering rule (**how** usage is counted, which often carries no digit) is strongest; where a sentence sits counts as much as what it says, so a terms-bearing heading and an FAQ answer both score while blog and careers copy is pushed down; and things that only look concrete — one row of a pricing comparison matrix, a number inside a sign-up prompt — are penalised enough to lose to plain prose. The `comparison-row` rule's tier test is three linear scans (`tierThenPeriodThenDigit`, per line, since `.` never crossed a newline) instead of `\b(plan|tier)\b.*\b(month|year|user)\b.*\d`, which was quadratic in the row - every tier word on a non-matching row scanned to its end before failing - and cost 15 s on a 309 KB row, the size of a line this corpus already holds, twice per row (break-test 2026-10-01). Prose candidates retain the physical paragraph start; excerpt recovery runs forward and stops once the existing 420-character cap is exceeded. Blank/heading boundaries, including ones inside skipped fences, remain the same; no backward array insertion makes a long unbroken capture quadratic (#334). | ADR-0016 (superseding ADR-0009) An underscore inside a word is the word (2026-10-02): `stripMarkdown` dropped every `_` as emphasis, so a drafted finding named `return_run_details` as "returnrundetails" and `check_suite_id` as "checksuiteid" - parameters no page carries, and a quote copied from the draft could never be found; emphasis marks at a word's edge still go. | | `evidence-context.mjs` | Raw cells pass projectFile for both project and capture-folder boundaries before any content is read; private project files, traversal and links leaving research/raw are refused, and directories are named as non-captures (#351, ADR-0109 trigger). One unknown, everything it rests on, in one read-only answer: the unknown's wording and why it blocks, every evidence row it cites, each row's source title / URL / type / retrieval date, the Finding, a bounded excerpt of the capture chosen for overlap with that Finding (and, in a block longer than the bound, windowed where the most distinct tokens of the Finding fall together - 2026-09-27: a JSON capture is one block, and the excerpt was its first 420 characters whatever the Finding quoted), rows that supersede it, and other rows for the same URLs. **It decides nothing**: it writes no status, offers no verdict, and reports a gap rather than filling it — a test asserts the rendering contains no conclusion and says plainly that the judgement is the reader's, and structural tests assert it imports no verdict machinery and contains no write call at all - the architectural property, which a phrase-based check cannot give. The recorded status is printed LAST and labelled as a recording, because a verdict shown before the evidence anchors the reader. Where the extractor would now pick a different sentence than the row records, that is shown beside it, labelled a suggestion. Reads no environment variable and spawns nothing, so it cannot use a credential. A Raw cell outside the project is reported ("is outside the project, so it is not read") and never read: it printed the outside file into an agent's context (2026-09-28, ADR-0076). `bin/evidence-context.mjs --all` over a contract with no unknowns yet answers "no unknowns" (or `[]` with `--json`) and exits 0 (2026-09-29, break-test): it exited 2 asking for the `--all` it was given. | — | | `credits.mjs` | **What a run does when the paying provider runs dry** — ADR-0086's one switch to the fallback, ADR-0129's stop-and-report. The one owner of a run's exhaustion state: `fellBack` / `stopped`, the `uncollected` and `skippedSearches` accounting, and the single `credits-exhausted` row the failure log keeps per run. The coordinator sequences and asks it which provider serves each call (`exhausted`, `live`, `onExhaustion`, `askLive`) and reads the state back for the summary; before the extraction the state lived as closures inside `runResearch` and `stopped.uncollected` was incremented from three loop sites. | ADR-0086, ADR-0129 | | `search-session.mjs` | **How a query is put to the search providers** (ADR-0135), for both coordinators - `runResearch` and `decompose` each carried a copy, and read side by side on 2026-10-03 the copies had drifted in six places (a failed first ask's usage counted in one, a plain failure in `searchFailures` in the other, `searchTransport` spelled `serpapi` by one and `serpapi+firecrawl-cli` by the other under a merge, the `notReady` refusal in one only, three log lines in one only, `covered` in the other only). `searchSession({ adapter, searchAdapter, searchAdapters, limit, maxRateLimitRetries, sleep, log, record, credits, askOne })` returns `name` (the paying provider, what the usage row's `searchTransport` says), `meter` (the dry run's label, every provider under a merge), `providers`, `assertReady()` (a `searchAdapter` reporting `notReady` throws `SEARCH_PROVIDER_NOT_READY` before any ask), `ask(text)` and `meters()`. `ask` is the loop: every provider under a merge, interleaved by `mergeByRank` and attributed, else the one provider with one bounded degrade to the fetch provider when it `canSearch` and is not the provider that failed; it answers `{ ok, query, results, provider, providers, merged, searchId, degraded }` or `{ ok: false, ..., error }`. **One accounting rule**: what an adapter reports is counted on its meter for every ask, succeeded or failed, and every failed ask is one failure on its provider - a query that failed on both providers is two. **The failure row is the session's, the sink the coordinator's**: `record` gets `{ query, error, provider }`, with `degraded: true, fellBackTo` on a degrade and `covered: true` when another provider of a merge answered; it is handed over at once (so a failure log keeps its chronology beside a `credits-exhausted` row) and gains `covered` afterwards. Provider-level lines (`waiting`, `search failed on`, `degrading to`, `searched`, `search found N`, `search failed`, `search failed on every provider`) are the session's; what a coordinator makes of an answer stays its own. `searchPatiently` and `mergeByRank` live here, re-exported by `research-run.mjs`. With a `credits` policy the policy's `askLive`/`live` decide which provider serves a call (ADR-0133); `askOne` is the one patient call a coordinator shares with that policy. A session built with no provider at all answers the failed search `searchPatiently` already names, charged to `this transport`, rather than throwing on the missing name (2026-10-03, probing after the module landed; only a library caller can reach it). `resultRows` keeps the shape every reader of a row assumes - an array, each row an object - on both paths (2026-10-03, probing): a provider's `null` row had ended both coordinators on `row.url`, and results that were not an array had ended research's `noteOutcome` on `.some`; whether a row's URL is usable stays the coordinator's question. `assertReady` reads `notReady` off every provider the session may ask, the merge's partners included, not the search side alone (2026-10-03, probing). **A provider that throws is a failed search that says so** (`searchPatiently`, 2026-10-03, probing): every adapter returns its failures by contract, so a throw is an adapter's bug the kit cannot prevent; re-thrown it ended both coordinators, and the searches paid for before it never reached the usage row, written at the end of a run - the meter under-recorded real spend. Named `<provider> threw: <message>`, not retried, degraded and recorded like any failure. An adapter that returns a Promise - an `async search()` - is a failed search too, saying adapters are synchronous (review of that guard, the same day): it had reached the `ok` test as a failure with no error text, and its rejection, awaited by nobody, ended the process after all; the Promise is given a catch so it cannot. A thrown value whose own rendering throws (a Symbol, a poisoned `message` getter) is named `a value that cannot be rendered as text` (`thrownText`). | ADR-0027, ADR-0133, ADR-0135 | @@ -214,9 +220,9 @@ is an open decision (2026-09-30): | `release/paths.mjs` | Containment (`inside`, `safePath`). Security-sensitive: a registry names its own record paths and can arrive from another machine with a corpus, so without this a manifest could point anywhere. Checked before the read and again against `realpath`. | — | | `release/order.mjs` | `compareText(a, b)`: the one order for identifiers - slugs, file names, ids, ISO dates - by UTF-16 code unit. Total, the same on every machine, what `canonicalJson` hashes keys by and what the Python conformance runners sort by (`key=utf-16-be`). In the ported layer so the validators may use it (ADR-0029); `core.mjs` re-exports it. `localeCompare` with no locale read the machine's: under da_DK `Firecrawl` before `firecrawl`, under sv_SE `zebra` before `ålder`, under th_TH `a-b` equal to `ab` (break-test PR #185, 2026-10-01). | — | | `path-authority-validator.mjs` | Read-only Git-origin/path-authority snapshot validator: strict envelope schema conformance, self-excluding canonical envelope hashes, portable root/realpath containment, isolated-environment coverage, required Git-origin proof, artifact-diff classification, and deterministic status reduction. It never writes snapshots, transcripts, profiles, Git config, or release state. Files sort by `compareText` (2026-10-01). | Git-origin/path-authority snapshot validator contract | -| `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. **A failed search reaches the map and the terminal** (2026-09-26): MAP.md lists each failure (query, provider, one scrubbed line of the vendor's error, and any fallback) and tells three empty maps apart - a dry run, a quiet topic, an outage - and `searchSummary` gives the CLI its count. Until then the failures were kept in the result (F20) and shown nowhere: after every search failed, the map told the operator to stop dry-running. The error is **scrubbed** because MAP.md is committed and vendor error text has echoed request URLs with keys in them. **Its searches are metered** (2026-09-27): phase 0 runs four searches and now writes a usage row like research-run's (`command: 'decompose'`, `searchesUsed` summed from what each provider reports), so `research.mjs --status` counts them. Until then two runs made 8 SerpAPI searches that the vendor's meter showed and the local one - labelled "this box", though it is one project on this machine - did not. A dry run still seeds the map (its documented purpose) and now logs each search a real run would make and the provider whose meter it would spend, as research's dry run does (2026-09-27): it had previewed none of the four searches. The CLI's closing steps name `research/plan.json`. **`resolveTopic` decides what is mapped** (2026-09-27, ADR-0056): `plan.json`'s topic when `--topic` is omitted, and a refusal naming both when `--topic` differs (spacing and case aside). A project still called `UNTITLED_TOPIC` accepts any `--topic`. The CLI had rewritten the map's topic to whatever it was given, and the brief took its title from the map. `loadRecipe` names the recipes that exist when one is not found (2026-09-27): `--recipe api` had answered with only the folder it looked in. Its usage row carries `searchCreditsEstimate` too (ADR-0065), counted once per search including a fallback search (a duplicated line had counted fallbacks twice until 2026-09-27). With `--max-scrapes`, a candidate is scraped only if it carries the query that found it (`matchesQuery`, ADR-0068); every candidate still appears in the map. Under a **merged** search selection it asks every provider each query and interleaves by rank (`mergeByRank`, as research does; 2026-09-28): the CLI passed only the first while printing that both were searched, so phase 0 scraped what SerpAPI alone had ranked. The usage row carries `searchesOn`; a provider that missed a query another answered is listed in the map as `answered by the other provider` and is not counted lost. **A compound topic is searched part by part** (2026-09-28, ADR-0085): `topicQueries` reads `subject: part, part; part` as one search per part (a one- or two-word part carries the subject, and so does a part that refers back with a pronoun, ADR-0098; at most `MAX_TOPIC_PARTS`, 6) instead of the whole topic four times, and `interleaveByQuery` shares the candidate list one per part. Searched whole, MoonAliza's three-question topic mapped forum threads and a coffee-scale blog, and in search order its third question got none of the map's 20 candidates. A plain topic keeps its four searches. **Outlines** (2026-09-29, ADR-0091): after gathering, `outlineOf(body)` reads the `##`/`###` headings of every material page with a capture on disk (at most `MAX_OUTLINES` pages, 12 headings each; code blocks, links, zero-width marks, repeats and `OUTLINE_FURNITURE` dropped) into a `## Outlines seen in the material` section - STORM's table-of-contents step without its model, unjudged, no fetch. Furniture also covers GitHub's file-view chrome and Discourse per-post bylines (`OUTLINE_FURNITURE_PATTERNS`, 2026-09-29). The scrape budget follows `scrapeOrder`: likely-owner hosts first, by `docsHosts` score, then the rest (ADR-0094). Owners are counted by `ownerOf` (ADR-0096): on a shared host - github.com, gitlab.com, huggingface.co, medium.com and the like - the owner is the account (the segment after `orgs/` or `users/`; the host's own sections such as `topics` stay the host's), so five strangers' repositories no longer pool into one "likely owner" that outranks the vendor's docs. A page on the owners tying for the top score (at least `MIN_NAMED_OWNER_SCORE`, 3) that the relevance floor skips is listed in the map under "Skipped on the likely owner", with `plan.json` `urls` as the way in (ADR-0097): the owner's terse "OpenAI compatibility" page had been skipped with only a log line to show for it. The floor itself still holds for it - trusting the owner past it re-admitted RR-9's postgresql.org home page. A compound-topic part that opens with "the" carries the subject, as a pronoun part does (2026-09-30, ADR-0103): "the bot limiter", searched alone, filled a SearXNG map with audio mastering limiters. `resolveTopic` refuses a plan the corpus reader flagged for a duplicate object key (2026-09-30), in the reader's words: `JSON.parse` keeps the last `topic`, so every search was built from the second while the operator read the first. Its searches go through `searchPatiently` (one call, no retries), and a failed search degrades only to a fetch provider that can search - the browser cannot (2026-10-01). Only an http(s) search result with a host (`isWebUrl`) is taken into the material, so only such pages are listed or scraped (2026-10-01). **A damaged ledger is refused before any provider is asked** (2026-10-03, outside audit): `assertAppendable(root)` at the top of `decompose`, as `runResearch` has had since ADR-0122 - phase 0 spends too, and it made four searches and one fetch before `collectOne` threw. **Its searches go through the search session** (2026-10-03, ADR-0135), the same one `runResearch` asks: `decompose` builds it with no rate-limit patience and a sink that keeps each failure row for the map, refuses a `notReady` provider through `session.assertReady()` before the ledger check (until then only the CLI refused, so a library caller spent), asks `session.ask(query)` per query and keeps every http(s) result as material, and reads `session.meters()` into its usage row. With the move it gained the three lines `runResearch` already logged (`searched`, `search found N`, `search failed on every provider`), its usage row's `searchTransport` names the paying provider under a merge as research's does (it had joined every provider with `+`), its `searchFailures` counts every failed ask (a covered miss and a degraded first ask included) and its `degraded` counts only the degrades that answered - research's counts, under the ADR's one rule. **A bare `--limit` or `--max-scrapes` is refused** (2026-10-11, #269): both go through `checkFlagValues` (`int`, `min: 0`) at the top of the CLI with the other value flags, so a flag with no value fails as `--<name> needs a value` instead of being read as `true` - which the hand-written `Number(raw)` check it replaces turned into 1, so `--dry-run --max-scrapes` reported "scrape up to 1" and a real run paid for one page. | ADR-0085, ADR-0091, ADR-0094, ADR-0096, ADR-0097, ADR-0098, ADR-0135 | +| `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. **A failed search reaches the map and the terminal** (2026-09-26): MAP.md lists each failure (query, provider, one scrubbed line of the vendor's error, and any fallback) and tells three empty maps apart - a dry run, a quiet topic, an outage - and `searchSummary` gives the CLI its count. Until then the failures were kept in the result (F20) and shown nowhere: after every search failed, the map told the operator to stop dry-running. The error is **scrubbed** because MAP.md is committed and vendor error text has echoed request URLs with keys in them. **Its searches are metered** (2026-09-27): phase 0 runs four searches and now writes a usage row like research-run's (`command: 'decompose'`, `searchesUsed` summed from what each provider reports), so `research.mjs --status` counts them. Until then two runs made 8 SerpAPI searches that the vendor's meter showed and the local one - labelled "this box", though it is one project on this machine - did not. A dry run still seeds the map (its documented purpose) and now logs each search a real run would make and the provider whose meter it would spend, as research's dry run does (2026-09-27): it had previewed none of the four searches. The CLI's closing steps name `research/plan.json`. **`resolveTopic` decides what is mapped** (2026-09-27, ADR-0056): `plan.json`'s topic when `--topic` is omitted, and a refusal naming both when `--topic` differs (spacing and case aside). A project still called `UNTITLED_TOPIC` accepts any `--topic`. The CLI had rewritten the map's topic to whatever it was given, and the brief took its title from the map. `loadRecipe` names the recipes that exist when one is not found (2026-09-27): `--recipe api` had answered with only the folder it looked in. Its usage row carries `searchCreditsEstimate` too (ADR-0065), counted once per search including a fallback search (a duplicated line had counted fallbacks twice until 2026-09-27). With `--max-scrapes`, a candidate is scraped only if it carries the query that found it (`matchesQuery`, ADR-0068); every candidate still appears in the map. Under a **merged** search selection it asks every provider each query and interleaves by rank (`mergeByRank`, as research does; 2026-09-28): the CLI passed only the first while printing that both were searched, so phase 0 scraped what SerpAPI alone had ranked. The usage row carries `searchesOn`; a provider that missed a query another answered is listed in the map as `answered by the other provider` and is not counted lost. **A compound topic is searched part by part** (2026-09-28, ADR-0085): `topicQueries` reads `subject: part, part; part` as one search per part (a one- or two-word part carries the subject, and so does a part that refers back with a pronoun, ADR-0098; at most `MAX_TOPIC_PARTS`, 6) instead of the whole topic four times, and `interleaveByQuery` shares the candidate list one per part. Searched whole, MoonAliza's three-question topic mapped forum threads and a coffee-scale blog, and in search order its third question got none of the map's 20 candidates. A plain topic keeps its four searches. **Outlines** (2026-09-29, ADR-0091): after gathering, `outlineOf(body)` reads the `##`/`###` headings of every material page with a capture on disk (at most `MAX_OUTLINES` pages, 12 headings each; code blocks, links, zero-width marks, repeats and `OUTLINE_FURNITURE` dropped) into a `## Outlines seen in the material` section - STORM's table-of-contents step without its model, unjudged, no fetch. Furniture also covers GitHub's file-view chrome and Discourse per-post bylines (`OUTLINE_FURNITURE_PATTERNS`, 2026-09-29). The scrape budget follows `scrapeOrder`: likely-owner hosts first, by `docsHosts` score, then the rest (ADR-0094). Owners are counted by `ownerOf` (ADR-0096): on a shared host - github.com, gitlab.com, huggingface.co, medium.com and the like - the owner is the account (the segment after `orgs/` or `users/`; the host's own sections such as `topics` stay the host's), so five strangers' repositories no longer pool into one "likely owner" that outranks the vendor's docs. A page on the owners tying for the top score (at least `MIN_NAMED_OWNER_SCORE`, 3) that the relevance floor skips is listed in the map under "Skipped on the likely owner", with `plan.json` `urls` as the way in (ADR-0097): the owner's terse "OpenAI compatibility" page had been skipped with only a log line to show for it. The floor itself still holds for it - trusting the owner past it re-admitted RR-9's postgresql.org home page. A compound-topic part that opens with "the" carries the subject, as a pronoun part does (2026-09-30, ADR-0103): "the bot limiter", searched alone, filled a SearXNG map with audio mastering limiters. `resolveTopic` refuses a plan the corpus reader flagged for a duplicate object key (2026-09-30), in the reader's words: `JSON.parse` keeps the last `topic`, so every search was built from the second while the operator read the first. Its searches go through `searchPatiently` (one call, no retries), and a failed search degrades only to a fetch provider that can search - the browser cannot (2026-10-01). Only an http(s) search result with a host (`isWebUrl`) is taken into the material, so only such pages are listed or scraped (2026-10-01). **A damaged ledger is refused before any provider is asked** (2026-10-03, outside audit): `assertAppendable(root)` at the top of `decompose`, as `runResearch` has had since ADR-0122 - phase 0 spends too, and it made four searches and one fetch before `collectOne` threw. **Its searches go through the search session** (2026-10-03, ADR-0135), the same one `runResearch` asks: `decompose` builds it with no rate-limit patience and a sink that keeps each failure row for the map, refuses a `notReady` provider through `session.assertReady()` before the ledger check (until then only the CLI refused, so a library caller spent), asks `session.ask(query)` per query and keeps every http(s) result as material, and reads `session.meters()` into its usage row. With the move it gained the three lines `runResearch` already logged (`searched`, `search found N`, `search failed on every provider`), its usage row's `searchTransport` names the paying provider under a merge as research's does (it had joined every provider with `+`), its `searchFailures` counts every failed ask (a covered miss and a degraded first ask included) and its `degraded` counts only the degrades that answered - research's counts, under the ADR's one rule. **A bare `--limit` or `--max-scrapes` is refused** (2026-10-11, #269): both go through `checkFlagValues` (`int`, `min: 0`) at the top of the CLI with the other value flags, so a flag with no value fails as `--<name> needs a value` instead of being read as `true` - which the hand-written `Number(raw)` check it replaces turned into 1, so `--dry-run --max-scrapes` reported "scrape up to 1" and a real run paid for one page. Recipe loading refuses front matter with no dimensions and blank dimension labels, naming the file and physical line. The CLI presents INVALID_RECIPE as usage failure (exit 2) before map writes (#356). | ADR-0085, ADR-0091, ADR-0094, ADR-0096, ADR-0097, ADR-0098, ADR-0135 | | `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — | -| `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' `verifyLedger` reports a ledger-named capture it cannot read as `raw-unreadable` instead of throwing out of preflight (ADR-0077). It refuses a capture that is not a REGULAR FILE before it opens it, for the reason `readText` does (2026-09-28, Arena break test 12): a fifo named by the ledger hung every command that verifies a corpus, and a ledger symlinked to `dev/zero` ended them with `std::bad_alloc` rather than a verdict. A lock write that fails (a full disk) closes its descriptor and removes the empty lock it just created before the error reaches the caller; it leaked both (2026-09-28). | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). A source entry without `sourceSha256` is a `body-unmodified/unhashed` problem, never skipped (2026-10-04, break-test pass 4 F4): the no-hash skip was written for the first ledgers' captures, and a source never had that history, so a rechained entry naming an unverifiable sibling is refused. | ADR-0003 (corpus-adjacent), ADR-0020 `verifyLedger` does not read a capture that lies outside the project (2026-09-30, ADR-0076); it reports `raw-outside`, which preflight fails. A chain naming `../elsewhere` or a link leading out of `research/raw` had verified a file the corpus does not hold. `verifyLedger` reads the path through `projectFile` (2026-09-30, ADR-0109). `acquire` throws `LOCK_STUCK` when a lock it judged stale cannot be removed for any reason but ENOENT (already gone is the race the loop retries): the failed removal fell through to `continue`, skipping the deadline and the sleep, so a read-only research/raw holding a dead pid's lock, or a lock path that is a directory, spun a whole core forever and printed nothing - unreachable by `--report-on-signal`, since a synchronous loop never returns to the event loop (break-test 2026-10-01). The wait-out refusal is coded `LOCK_HELD`; `withLock(root, fn, { waitMs })` is the test seam for its 120 s wait, in the spirit of `materializeIndex`'s injectable git. Evicting on age stays rejected (ADR-0025). **`assertAppendable(root)`** (ADR-0122, 2026-10-02) is the one function that establishes a ledger can be appended to: an unparsed line (`LEDGER_DAMAGED`), a torn tail (`LEDGER_TORN_TAIL`), and a chain whose `seq`/`prev`/`entrySha256` no longer link (`LEDGER_CHAIN_BROKEN`, named by line and rule, remedy `git checkout -- research/raw/.fetches.jsonl` - the kit never rewrites a hash). `chainProblems(entries)` is the link check alone, shared with `verifyLedger`; `appendFetch` asks `assertAppendable` under the lock, and `runResearch` asks it before the first fetch. Until then a parsed-but-broken chain was appended to, and the paid-for fetch was refused by handoff with the rest. `isLineEndingRewrite` keeps its own CRLF-only fold, deliberately not core's `foldLineEndings` (2026-10-03): it asks whether autocrlf explains a mismatch, and autocrlf never makes a lone CR. | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not. The two sanctioned rewrites, `repairLedgerTail` and `rebuildLedger`, write through `writeText` (2026-09-28, ADR-0079): they emptied the ledger before writing it, so one cut short left a truncated ledger. Both look for the ledger **before** taking the lock (2026-09-28): the lock lives in `research/raw/`, a gate marker, so `doctor --fix-arity` in a folder with no ledger made it a gated project missing its scaffold. A fetch entry names its source sibling (`source`, `sourceSha256`, ADR-0140, 2026-10-03), and `verifyLedger` checks every file an entry names the same way - the capture and the source - so an edited source is `body-unmodified` and a missing one `raw-missing`, and the commit gate refuses them as it refuses an edited capture. `sourceOmitted` rides on an entry whose source was given and not kept (2026-10-04). +| `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' `verifyLedger` reports a ledger-named capture it cannot read as `raw-unreadable` instead of throwing out of preflight (ADR-0077). It refuses a capture that is not a REGULAR FILE before it opens it, for the reason `readText` does (2026-09-28, Arena break test 12): a fifo named by the ledger hung every command that verifies a corpus, and a ledger symlinked to `dev/zero` ended them with `std::bad_alloc` rather than a verdict. A lock write that fails (a full disk) closes its descriptor and removes the empty lock it just created before the error reaches the caller; it leaked both (2026-09-28). | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). A source entry without `sourceSha256` is a `body-unmodified/unhashed` problem, never skipped (2026-10-04, break-test pass 4 F4): the no-hash skip was written for the first ledgers' captures, and a source never had that history, so a rechained entry naming an unverifiable sibling is refused. | ADR-0003 (corpus-adjacent), ADR-0020 `verifyLedger` does not read a capture that lies outside the project (2026-09-30, ADR-0076); it reports `raw-outside`, which preflight fails. A chain naming `../elsewhere` or a link leading out of `research/raw` had verified a file the corpus does not hold. `verifyLedger` reads the path through `projectFile` (2026-09-30, ADR-0109). `acquire` throws `LOCK_STUCK` when a lock it judged stale cannot be removed for any reason but ENOENT (already gone is the race the loop retries): the failed removal fell through to `continue`, skipping the deadline and the sleep, so a read-only research/raw holding a dead pid's lock, or a lock path that is a directory, spun a whole core forever and printed nothing - unreachable by `--report-on-signal`, since a synchronous loop never returns to the event loop (break-test 2026-10-01). The wait-out refusal is coded `LOCK_HELD`; `withLock(root, fn, { waitMs })` is the test seam for its 120 s wait, in the spirit of `materializeIndex`'s injectable git. Evicting on age stays rejected (ADR-0025). **`assertAppendable(root)`** (ADR-0122, 2026-10-02) is the one function that establishes a ledger can be appended to: an unparsed line (`LEDGER_DAMAGED`), a torn tail (`LEDGER_TORN_TAIL`), and a chain whose `seq`/`prev`/`entrySha256` no longer link (`LEDGER_CHAIN_BROKEN`, named by line and rule, remedy `git checkout -- research/raw/.fetches.jsonl` - the kit never rewrites a hash). `chainProblems(entries)` is the link check alone, shared with `verifyLedger`; `appendFetch` asks `assertAppendable` under the lock, and `runResearch` asks it before the first fetch. Until then a parsed-but-broken chain was appended to, and the paid-for fetch was refused by handoff with the rest. `isLineEndingRewrite` keeps its own CRLF-only fold, deliberately not core's `foldLineEndings` (2026-10-03): it asks whether autocrlf explains a mismatch, and autocrlf never makes a lone CR. Tail repair recognizes the leading BOM on a one-entry ledger as the reader does, so a valid first entry is never dropped as a torn tail; repairing a later append preserves the prefix (#296). | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not. The two sanctioned rewrites, `repairLedgerTail` and `rebuildLedger`, write through `writeText` (2026-09-28, ADR-0079): they emptied the ledger before writing it, so one cut short left a truncated ledger. Both look for the ledger **before** taking the lock (2026-09-28): the lock lives in `research/raw/`, a gate marker, so `doctor --fix-arity` in a folder with no ledger made it a gated project missing its scaffold. A fetch entry names its source sibling (`source`, `sourceSha256`, ADR-0140, 2026-10-03), and `verifyLedger` checks every file an entry names the same way - the capture and the source - so an edited source is `body-unmodified` and a missing one `raw-missing`, and the commit gate refuses them as it refuses an edited capture. `sourceOmitted` rides on an entry whose source was given and not kept (2026-10-04). | `quotes.mjs` | **Quote anchors** (2026-09-28, ADR-0087): `quoteAnchors(text)` reads the `[quote: ...]` markers of a Finding cell, split into fragments on an ellipsis - a marker may hold balanced brackets one level deep, a code subscript or a Markdown link (2026-09-29: it ended at the first `]`, and the stub it kept was what got checked); `normalizeForMatch` is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); `anchorFound(fragments, body)` requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, with no fuzzy fallback; `MIN_QUOTE_WORDS` is 3, and `MIN_QUOTE_CHARS` is 40 (2026-10-02, ADR-0126): a quote is short only under both, measured by `quoteWords` and `quoteChars` across the fragments - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; `checks.mjs` reads the capture and reports. `quoteAnchors` takes one pair of quotation marks around the whole passage off before matching (`unwrap`): `[quote: "the sentence"]` was quote-not-found for a sentence the capture held word for word (2026-10-01). `normalizeForMatch` also drops Unicode's format characters (`\p{Cf}`: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: `## [\u200B](...#bundled-skills) Bundled skills` in this repository's own corpus made `[quote: ## Bundled skills]` unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. | ADR-0087 | | `measure.mjs` | **How a corpus's citations hold up, without a judge** (2026-09-28, ADR-0089): `measureCorpus(root)` - link works (capture on disk, in the ledger, unmodified, not an error page), quote-anchored rows and quotes found (`quotes.mjs`), full captures, closed unknowns on a `P` row and on two or more hosts, known unknowns, ledger verifies; `factCheck` is named and never computed. `renderMeasure` for people; `bin/measure.mjs` prints either, `--json` for machines, refuses outside a project, and never gates. | ADR-0089 | | `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the ledger's hash and seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`. `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | @@ -226,11 +232,11 @@ is an open decision (2026-09-30): | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 | | `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, **what phase 1 may commit** (ADR-0048: `research/`, the root `SCAFFOLDING` - `.gitattributes`, `.gitignore`, `AGENTS.md`, `START_HERE.md` - and `docs/ARCHITECTURE.md` while `isEmptyArchitectureMap` says its staged bytes, read by `readStagedText`, are still the template's; until 2026-09-27 only `research/`, so a new project's first `git add -A` was refused and its `.gitattributes` could not travel), and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **Staged paths remain data, with checked acquisition** (ADR-0154): the default commit CLI lazily calls `readStagedPaths` after the ungated and `GATE_OFF` returns. One actual `git diff --cached --name-only --no-renames -z` supplies paths only on successful complete output; the existing 64 MiB capture rejects nonzero exit and buffer failure, and the reader rejects malformed NUL data. Successful zero bytes means `[]`; unreadable scope means null and raises the existing architecture-map breach even after green preflight, ordinary exit 1 under either posture. Explicit arrays/null and the existing stdin/repeated `--staged` options retain their meaning; edits never acquire a list. **One deliberate exception to "no CLI parsing"** remains for explicitly supplied stdin data (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"**, preserving unknown scope rather than assuming an empty set. No list read claims an atomic snapshot with later index materialization. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). The architecture-map block's fix says to **update** the map, then `git add` it (2026-09-27): a bare `git add docs/ARCHITECTURE.md` stages nothing while the map is unchanged, so the printed fix left the commit blocked. `bin/gate.mjs` no longer closes that block with "Phase 1 is not done until preflight prints PASS" - a map block happens only when the gate passes. `isPhaseOneEdit(root, file)` answers the same question for the edit hook, about one file. The empty-index verdict — a gated project whose `research/` is untracked, every fresh project's first commit — removes the scratch dir `materializeIndex` made before it learned the index was empty (2026-09-28): it returned before `scratch` was assigned, so the `finally` that removes it never ran, and every such verdict leaked one temp directory. **Integrity before scope** (ADR-0093): a failing verdict whose findings include an `INTEGRITY_RULES` provenance rule blocks the commit even when it is confined to `research/` - the ADR-0048 allowance covers completeness findings only. `stdinIsReadable(stdin = process.stdin)` takes the stream to ask about, so the hook's terminal case is testable (2026-10-01). **The suite rule** (ADR-0120, 2026-10-01): `suiteBreach(root, stagedPaths, { run })` runs after the map rule, in the commit gate only, and only where `isKitCheckout` finds the kit's four markers and a staged path is under `research-kit/` (or the list is unknown); `runSuiteHere` spawns the checkout's `bin/selftest.mjs` with `RESEARCH_KIT_RESULT_FILE` pointed at a scratch file and `RESEARCH_KIT_ALLOW_UNSUP=1` (unsupported is not red, ADR-0108), git's hook environment stripped (`GIT_INDEX_FILE`, `GIT_DIR`, `GIT_PREFIX` and kin - the same `REPOSITORY_LOCATION` list as the edit hook, else every scratch repository's git in the suite would act on the index being committed), and reads the file, never stdout: failures block naming the counts, no result blocks as a crashed runner, a run past its budget is stopped and blocks as unfinished, the fix naming `RESEARCH_KIT_GATE_TIMEOUT` before the override, and a green run whose runner exited non-zero (the README count check) blocks too. `suiteOwed(root, stagedPaths)` is the predicate alone; `suiteBreach`'s `announce` (`evaluate`'s `announceSuite`) is called right before the run, so `bin/gate.mjs` says the suite is running only when it is - not when the map rule or preflight refused the commit first. `evaluate` takes `runSuite` so the verdict is testable without a suite. **The suite's budget follows the gate's watchdog** (ADR-0142, 2026-10-05): `suiteBudgetMs(env)` is `SUITE_TIMEOUT_MS` (65 min, twice the slowest honest run measured, 1825 s on the operator's Windows PC) unless `RESEARCH_KIT_GATE_TIMEOUT` is set to a positive duration `gateTimeoutSeconds` reads (seconds, an optional fraction, an optional `s`/`m`/`h`/`d`, at most nine digits before the point - the grammar the hook's `gate_timeout_readable` checks too; `0` and anything outside it leave the default), and then that watchdog less `GATE_REST_S` (120 s, the hook's default watchdog for a gate that runs no suite), never more than half a short one, at least 1 ms and at most 2^31 - 1 - so the gate stops the suite and blocks before the hook's watchdog fires, whenever its own work fits what is left. `runSuiteHere` stops the runner with SIGKILL: the harness's SIGTERM listener kept a synchronously blocked runner alive past its budget on POSIX; a stopped run's scratch is left in the temp folder. The budget had been 20 minutes, which that PC's green suite met on every kit commit, and the variable had reached only the watchdog: set below the budget, the watchdog killed the gate first and a fail-open machine allowed a commit whose suite never reported. **The fix lines no longer say `--no-verify` is recorded** (2026-10-03, outside audit): git skips this hook, so nothing can record it; `checks.mjs` had said so since ADR-0035 while the gate's own fix text, the README, AGENTS.md and CONTEXT.md said the opposite. **Red suite diagnostics** (2026-10-08): `suiteBreach` forwards up to five `failed` identifiers, each limited to 160 characters with controls normalized, in its existing detail string; omitted or clipped data is named, and legacy missing labels are explicit. Counts and verdict keys are unchanged. `runSuiteHere` still removes its scratch result; no result retention system is added. | ADR-0002, ADR-0007, ADR-0008, ADR-0020, ADR-0093, ADR-0120, ADR-0142, ADR-0154 | | `doctor.mjs` | Read-only diagnostics: machine + project + gate + chain health in one report. Consumes the verdict context (`verdictContext` / `readGateState`) for the chain-verified corpus and the gate-state triple; adds one machine read of its own (the git-resolved `hooksPathEffective`) for report. **Role-aware** (ADR-0010): the Firecrawl findings are informational on a `builder`, and the handoff findings (ADR-0011) are blockers there — the collection entrypoints' refusal is what makes that split honest. The handoff findings come from the owner, not from a second composition: doctor passes the corpus it already read to `verifyHandoff(root, { corpus })`, so the arrival question has one implementation and one set of pins. One deliberate write: the repository-local hooksPath override log (the third override is silent by nature — the bypassed hook cannot report it). `gateHealth` and `runDoctor` accept explicit Git/config/settings paths and a child environment so diagnostics can run against disposable machine fixtures without consulting host state. **`commitGateState`** asks WHOSE pre-commit is at `core.hooksPath` - `current` / `foreign` / `unusable` - because executable is not the same as ours. It used to ask only whether some executable hook was there, the same existence-for-correspondence mistake the `deploy` check made; `settingsState` has asked the question of the EDIT gate since a registration here named our hook inside a different checkout, and `core.hooksPath` is a single machine-wide setting that husky, lefthook and pre-commit all rewrite. Pure and separate from `gateHealth`, which reads real git config - a rule testable only by fishing a finding out of an array on a host-dependent path is a rule nobody tests (ADR-0004), which is why the older test for it asserts the severity is one of three. **Ownership is decided before executability**, because the first version asked about the mode first and so reported a non-executable FOREIGN hook as `unusable` with `chmod +x` beside it - advice to repair somebody else's gate while leaving this one just as absent. That ordering bug was invisible on Windows, where `hookExecutability` returns ok for any file that exists, and surfaced on Linux in CI. On a **collector**, handoff findings are `info` in the collector's words ("does not exist yet - nothing has been collected here; a builder handed this corpus would be refused"), since 2026-09-26: they were graded `pass` with the builder's wording, so a fresh project printed "pass ... the corpus arrived without its ledger". On a builder they remain blockers, worded as before. It reports the **Node line** and, when a proxy variable is set, whether this Node sends the kit's requests through it A collector without the Firecrawl CLI is told the exact install command, `npm install -g firecrawl-cli@<tested>` from `cliInstallSpec()` (2026-09-27; it said "install the Firecrawl CLI" and nothing anywhere said how). (`proxy`: keyless pages, search requests and remote collection; a warning naming the version to move to - or, for a value Node cannot parse, the spelling it needs - since the Firecrawl CLI has its own proxy handling) - measured on 22.20.0, where the fetch went around the proxy (ADR-0046, ADR-0047). A collector that explicitly chose a keyless or browser transport passes the Firecrawl checks; nothing chosen still fails (ADR-0095). The Firecrawl fixes (`firecrawl-cli`, `firecrawl-auth`) name the no-account route beside the install or login - `"transport": "http-keyless"` in the machine config, the setting that makes them pass (ADR-0095) - where the CLI fix had offered only a per-run flag and the login fix nothing (2026-09-29). `editGateState` reads an entry whose `hooks` is not a list, or a hook that is `null`, as naming nothing (2026-09-30); `[null]` had thrown a `TypeError`. `scanForSecrets` catches a folder it cannot read where it reads it, and skips a folder it has already walked by real path (2026-09-30): one try/catch around the whole walk had ended the scan at the first unreadable folder, leaving every file after it unscanned and reporting nothing found. `scanForSecrets` records a cap stop separately from the scanned count (2026-10-08, bounded-scan audit repair): it keeps the 2000-text-file limit and reports incomplete coverage only when that cap actually prevents further traversal; ending at the final file's boundary is not itself incomplete. `runDoctor` reports a capped scan as a `secret-scan` warning, beside any unchanged critical `secret` hits, instead of reporting a capped no-hit scan as passing. It names no unobserved remaining-file count and makes no whole-repository clearance claim. `bin/doctor.mjs` reports `KIT_VERSION`, as `kitVersion` in `--json` and on its last line (ADR-0116). Reports each ill-typed posture key as a `config-ill-typed` warning naming the key, the values the kit reads, and the state it resolved to (ADR-0121). The `firecrawl-cli` finding judges the installed version with the run's own rule (`cliCompatibility`): an unsupported major is a **fail** with the remedy on a collector that needs the CLI and information elsewhere, a newer minor passes and says what it was tested against - doctor had printed `pass 2.0.0` for a CLI the run then refused (2026-10-02, break-test pass 3). `scanForSecrets` skips a binary file and counts it (`skippedBinary`, named in the coverage line) instead of matching credential patterns against an executable's string table (ADR-0128, 2026-10-02): run on a Windows home folder it reported fourteen committed keys, every one the PEM marker OpenSSH and libssh2 parse, or random bytes spelling `sk-`. **`handOnState`** (2026-10-03, ADR-0134; since 2026-10-04 defined in `installer.mjs` and re-exported here, ADR-0141, so the install judges what it records by the same rule): what the hooks hand on to - none, missing, kit, other - and `gateHealth` warns on kit and missing; the maintainer's machine handed every commit on to the previous implementation's gate, which crashed and failed open while doctor said READY. `handOnState` resolves a relative folder from the repository's top level (`base`), where git runs its hooks from; doctor had resolved `.custom-hooks` against its own working directory and from a subdirectory called a working hand-on missing (fourth audit round, 2026-10-03). `scanForSecrets` skips `research/raw` at ANY depth (2026-10-04): the skip matched the relative path `research/raw`, so only the root's captures were outside the scan and a nested decision project's were read as the project's own files - CI tripped `google-api-key` on a Google page's source sibling (ADR-0140), Google's own public keys; a capture is page content, and the folder exists wherever a nested project does (ADR-0030). The shared unusable-proxy check also rejects proxy URLs carrying a non-root path, query or fragment, so those configurations warn instead of promising proxy use (#380, 2026-10-12). `machineHealth` uses the shared `envProxyChoice` with injectable `execArgv` (#381, 2026-10-12). An explicit disabled choice warns that requests go direct and names how to enable the proxy, rather than promising proxy use from Node version support alone; the raw variable value is never echoed. | ADR-0010, ADR-0011, ADR-0095, ADR-0134 | -| `panel.mjs` | **The desktop panel** (ADR-0147, 2026-10-07): `createPanel({ project, env, kitRoot, kitHome })` serves `research-kit/panel/` (one page, a script, a stylesheet) and a small JSON API on **127.0.0.1 only**, a thin shell over commands that already exist. It owns no verdict: doctor, handoff, preflight and install run as children with the fixed argument lists in `PANEL_COMMANDS`, in the chosen project folder, and their output is shown with the search key redacted. **No command from the page collects** - `research.mjs` and `decompose.mjs` are absent from `PANEL_COMMANDS`; the one way pages are fetched through the panel is auto-collect (`lib/auto-collect.mjs`, ADR-0148), for a builder's validated request, on a collector, within the operator's caps. Its routes are `GET /api/requests` (the queue and the meter), `POST /api/autocollect` (the settings), `/api/requests/check` and `/api/requests/resume`. It writes two things: the search provider's key into the machine config through `transport.mjs`'s `SEARCH_KEY` (so the vendor's name stays in the adapter and the registry, NFR-3), refused when the config resolves inside a git repository (Rule 6), or with HTTP 409 and a role-repair remedy when an unreadable config has no usable snapshot (#364); and which project it shows, a full path to an existing folder, never searched for. The topic and `BRIEF.md` are read, never written (ADR-0056). `builderInstructions` is the hand-off text: the collector's push, ledger included, then the builder's own steps in `$HOME` spelling (ADR-0050) - declare the role, handoff, preflight, inspect an authored complete brief and its current inputs, never collect. Its readiness checklist requires classified map rows and rewritten Findings; a file-presence observation grants no build authorization. Commit-report placeholders require actual observed results. A collected Git-request result waits for the collector to record review/closure before dependent building. Every request passes four checks: the bound address, a Host header naming it (DNS rebinding), an Origin, when sent, that is its own, and for `/api/` the per-launch token, initialized through the launch URL fragment and required in x-panel-token. The launcher prints that capability URL; terminal-log and browser-history privacy are not guaranteed by this module. POSTs must be JSON, so a plain cross-site form is refused. Update is refused when the panel runs from the installed copy, which has nothing newer to install. Command runs, the project switch and auto-collect cycles share one lock (`running`, handed to `createAutoCollect` as `lock`): none runs beside another, so no preflight or update reads the corpus or the kit tree while a cycle writes it. | ADR-0147, ADR-0148 | +| `panel.mjs` | **The desktop panel** (ADR-0147, 2026-10-07): `createPanel({ project, env, kitRoot, kitHome })` serves `research-kit/panel/` (one page, a script, a stylesheet) and a small JSON API on **127.0.0.1 only**, a thin shell over commands that already exist. It owns no verdict: doctor, handoff, preflight and install run as children with the fixed argument lists in `PANEL_COMMANDS`, in the chosen project folder, and their output is shown with the search key redacted. **No command from the page collects** - `research.mjs` and `decompose.mjs` are absent from `PANEL_COMMANDS`; the one way pages are fetched through the panel is auto-collect (`lib/auto-collect.mjs`, ADR-0148), for a builder's validated request, on a collector, within the operator's caps. Its routes are `GET /api/requests` (the queue and the meter), `POST /api/autocollect` (the settings), `/api/requests/check` and `/api/requests/resume`. It writes two things: the search provider's key into the machine config through `transport.mjs`'s `SEARCH_KEY` (so the vendor's name stays in the adapter and the registry, NFR-3), refused when the config resolves inside a git repository (Rule 6), or with HTTP 409 and a role-repair remedy when an unreadable config has no usable snapshot (#364); and which project it shows, a full path to an existing folder, never searched for. The topic and `BRIEF.md` are read, never written (ADR-0056). `builderInstructions` is the hand-off text: the collector's push, ledger included, then the builder's own steps in `$HOME` spelling (ADR-0050) - declare the role, handoff, preflight, inspect an authored complete brief and its current inputs, never collect. Its readiness checklist requires classified map rows and rewritten Findings; a file-presence observation grants no build authorization. Commit-report placeholders require actual observed results. A collected Git-request result waits for the collector to record review/closure before dependent building. Every request passes four checks: the bound address, a Host header naming it (DNS rebinding), an Origin, when sent, that is its own, and for `/api/` the per-launch token, initialized through the launch URL fragment and required in x-panel-token. The launcher prints that capability URL; terminal-log and browser-history privacy are not guaranteed by this module. POSTs must be JSON, so a plain cross-site form is refused. Update is refused when the panel runs from the installed copy, which has nothing newer to install. Command runs, the project switch and auto-collect cycles share one lock (`running`, handed to `createAutoCollect` as `lock`): none runs beside another, so no preflight or update reads the corpus or the kit tree while a cycle writes it. Request-body overflow discards retained chunks and stops parsing further input. The handler sends the existing 413 JSON refusal with Connection: close, allowing the response to flush before the socket closes; the exact byte limit remains admitted (#324). | ADR-0147, ADR-0148 | | `requests.mjs` | **A builder's fact request as a file** (ADR-0148, 2026-10-07): `research/requests/<id>.json`, the `fact-request` skill made machine-readable. `requestProblems(request, { perRequestPages })` validates it as untrusted input: the fixed fields (`fact`, `blocks`, `urls`, `queries`, `prefer`, `unknown`, `maxPages`, `topic`), bounded sizes, and `urlProblem` refuses a non-http(s) URL, one with a user name or password, and one naming this machine or an internal address (`isInternal` from `http-transport.mjs`, ADR-0110). A request over the per-request cap is refused, not trimmed. `listRequests` reads the folder, skipping the collector's own `<id>.result.json` and `<id>.plan.json` and ignoring a name that is not an id, and a request whose file, `.plan.json` or `.result.json` is a symbolic link (a builder's link would make the collector read or overwrite a file of its own machine; `writeText` follows links). `applyRequest` and `writeResult` refuse to write through a link as well. `applyRequest` writes the contract row (a new OPEN `U-n`, or an existing non-CLOSED row; it refuses requests naming a closed unknown) and the `plan.json` entries, and the plan for that run alone, with `maxScrapes` set to the request's pages. `contractUnknownStatus` reads the status of a contract row for continuation checks. `writeResult` records the outcome beside the request. Terminal and partial results are trusted only when `recordTrustedResult` has recorded hashes of both request and result in the machine-local `research-kit.autocollect.json`; `isTrustedResult` checks those hashes so a builder cannot submit a result as if the collector wrote it. The auto-collect settings (`off`/`auto`, pages a request, pages a day, interval, the folder for new topics) and the daily meter live in `research-kit.autocollect.json` beside the machine config (`readAutoCollect`, `saveAutoCollect`, `recordSpend`, `setPaused`); no key is in them. | ADR-0148 | -| `auto-collect.mjs` | **The collector listening** (ADR-0148, 2026-10-07): `createAutoCollect({ project, env, kitRoot, exec, lock })` returns `cycle`, `status`, `schedule`, `resume`, `stop`. A cycle resolves the project folder to its final path (`fs.realpathSync.native`) before comparing it with git's top level, so a Windows 8.3 short name or a symlinked folder cannot turn a path git stages into one outside the repository. `status()` uses that same canonical project path so its trust lookup matches collector records across alternate Windows path spellings. A cycle refuses unless `collectionPolicy` says the machine may collect, and skips while another run holds the shared `lock` (the panel's; its own when none is given). Otherwise it runs `git pull --ff-only`; a failed pull is surfaced and blocks the cycle before requests are read or processed. On success, it goes through each request. Results received over git are ignored unless their request and result hashes match collector-local trust records, including in `status()`; trusted terminal results retry pending commit/push delivery without collecting again, and trusted `partial` results carry continuation state and resume only while their contract unknown is not CLOSED. An invalid one gets a refused result, committed and pushed. A valid one, within the per-request and daily caps and with `research/` clean, is applied (`requests.mjs`), then its target-specific `research/requests/<id>.plan.json` runs, followed by `preflight.mjs`, whose verdict is recorded, not acted on. It then stages `research/` and the ledger by name (`git add -f`), commits, and pushes when there is an upstream. Accounting travels on the fixed Node research child's private IPC descriptor (ADR-0149), never its human output: exactly one bounded, typed report with measured pages within the admitted allowance and the exhaustion provider. Missing, invalid, duplicate or over-allowance reporting reserves the remaining admitted allowance against the daily meter, labels actual spend unknown, records a terminal failed result and pauses. Earlier measured continuation attempts stay counted once. A valid received report survives failed exit or later interruption; interruption also pauses. Diagnostic output keeps a bounded redacted tail with a truncation notice, and failure detail is bounded separately. The legacy `spentPages`/`creditsStopped` exports remain diagnostic compatibility helpers. It never passes `--fallback`: received exhaustion accounting makes the loop commit the partial corpus and continuation state, pause until the operator resumes, then reuse the saved target, unknown and remaining page allowance (ADR-0129). A `topic` request gets `new-project.mjs` and `decompose.mjs --max-scrapes 0` in a dated folder under the folder the operator approved, never over an existing project (ADR-0056). It is refused when no folder is approved. Diagnostic details, refusal reasons and preflight verdicts are scrubbed with the existing redactor before result files or commit reports are written (2026-10-08). Status also scrubs inherited trusted result details, delivery diagnostics and pause text without rewriting those files or changing their trust hashes; the configured redactor defines which secret is recognized. Every child gets a fixed argument list; `exec` is injectable for tests. The panel labels its unchanged numeric meter as pages used or reserved. **Allowance regression coverage** (2026-10-09): the four-cycle credit exhaustion and resume fixture runs real synthetic Node scripts against persisted request, plan, result and meter files, with a finite injected Git executor. Native round-trip, partial-delivery and topic-resume fixtures retain real Git and corpus-transfer assertions. The test watchdog remains 60 seconds. **Complete Git answers** (ADR-0151): the bounded child byte buffer reports a private `truncated` flag. Repository and Git metadata paths, upstream, ahead count, commit subject and changed paths, dirty state, and pending delivery status/diff all refuse truncated output before it can authorize work; a malformed ahead count or missing quiet-diff exit also blocks delivery. A newly written partial result is trusted before fingerprinting, so refused inspection stages nothing and cannot make a restart collect it again. Exit-only operations retain bounded diagnostic logs. **Admission survives collector termination** (ADR-0152): `reserveCollection` saves the newly admitted allowance and an owned in-flight pause atomically before the fixed research child starts; a failed meter write prevents spawn. `settleCollection` refunds unused allowance once on a valid receipt, preserves earlier spend and distinct pauses, and retains the full charge for unknown/spawn-failed attempts. Whole collector termination leaves its charge/pause across restart or project switch. Continuation reserves only its remaining newly admitted pages. Day-aware settlement never refunds yesterday against today; cross-day completion charging is conservative, and an over-cap completion remains paused. The existing file writer is atomic, while the mutex remains per collector instance; racing independent instances and power-loss durability are not certified by this correction. Successful result.next says captures were delivered: the builder reads/judges, the collector records Git Findings, closures, MAP and the current authored BRIEF; another fetch is only for a missing external fact. ADR-0052 local received-package review remains available, and derived approval stays role-agnostic (2026-10-08, ADR-0150). | ADR-0148, ADR-0149, ADR-0150, ADR-0151, ADR-0152 | +| `auto-collect.mjs` | **The collector listening** (ADR-0148, 2026-10-07): `createAutoCollect({ project, env, kitRoot, exec, lock })` returns `cycle`, `status`, `schedule`, `resume`, `stop`. A cycle resolves the project folder to its final path (`fs.realpathSync.native`) before comparing it with git's top level, so a Windows 8.3 short name or a symlinked folder cannot turn a path git stages into one outside the repository. `status()` uses that same canonical project path so its trust lookup matches collector records across alternate Windows path spellings. A cycle refuses unless `collectionPolicy` says the machine may collect, and skips while another run holds the shared `lock` (the panel's; its own when none is given). Otherwise it runs `git pull --ff-only`; a failed pull is surfaced and blocks the cycle before requests are read or processed. On success, it goes through each request. Results received over git are ignored unless their request and result hashes match collector-local trust records, including in `status()`; trusted terminal results retry pending commit/push delivery without collecting again, and trusted `partial` results carry continuation state and resume only while their contract unknown is not CLOSED. An invalid one gets a refused result, committed and pushed. A valid one, within the per-request and daily caps and with `research/` clean, is applied (`requests.mjs`), then its target-specific `research/requests/<id>.plan.json` runs, followed by `preflight.mjs`, whose verdict is recorded, not acted on. It then stages `research/` and the ledger by name (`git add -f`), commits, and pushes when there is an upstream. Accounting travels on the fixed Node research child's private IPC descriptor (ADR-0149), never its human output: exactly one bounded, typed report with measured pages within the admitted allowance and the exhaustion provider. Missing, invalid, duplicate or over-allowance reporting reserves the remaining admitted allowance against the daily meter, labels actual spend unknown, records a terminal failed result and pauses. Earlier measured continuation attempts stay counted once. A valid received report survives failed exit or later interruption; interruption also pauses. Diagnostic output keeps a bounded redacted tail with a truncation notice, and failure detail is bounded separately. The legacy `spentPages`/`creditsStopped` exports remain diagnostic compatibility helpers. It never passes `--fallback`: received exhaustion accounting makes the loop commit the partial corpus and continuation state, pause until the operator resumes, then reuse the saved target, unknown and remaining page allowance (ADR-0129). A `topic` request gets `new-project.mjs` and `decompose.mjs --max-scrapes 0` in a dated folder under the folder the operator approved, never over an existing project (ADR-0056). It is refused when no folder is approved. Diagnostic details, refusal reasons and preflight verdicts are scrubbed with the existing redactor before result files or commit reports are written (2026-10-08). Status also scrubs inherited trusted result details, delivery diagnostics and pause text without rewriting those files or changing their trust hashes; the configured redactor defines which secret is recognized. Every child gets a fixed argument list; `exec` is injectable for tests. The panel labels its unchanged numeric meter as pages used or reserved. **Allowance regression coverage** (2026-10-09): the four-cycle credit exhaustion and resume fixture runs real synthetic Node scripts against persisted request, plan, result and meter files, with a finite injected Git executor. Native round-trip, partial-delivery and topic-resume fixtures retain real Git and corpus-transfer assertions. The test watchdog remains 60 seconds. **Complete Git answers** (ADR-0151): the bounded child byte buffer reports a private `truncated` flag. Repository and Git metadata paths, upstream, ahead count, commit subject and changed paths, dirty state, and pending delivery status/diff all refuse truncated output before it can authorize work; a malformed ahead count or missing quiet-diff exit also blocks delivery. A newly written partial result is trusted before fingerprinting, so refused inspection stages nothing and cannot make a restart collect it again. Exit-only operations retain bounded diagnostic logs. **Admission survives collector termination** (ADR-0152): `reserveCollection` saves the newly admitted allowance and an owned in-flight pause atomically before the fixed research child starts; a failed meter write prevents spawn. `settleCollection` refunds unused allowance once on a valid receipt, preserves earlier spend and distinct pauses, and retains the full charge for unknown/spawn-failed attempts. Whole collector termination leaves its charge/pause across restart or project switch. Continuation reserves only its remaining newly admitted pages. Day-aware settlement never refunds yesterday against today; cross-day completion charging is conservative, and an over-cap completion remains paused. The existing file writer is atomic, while the mutex remains per collector instance; racing independent instances and power-loss durability are not certified by this correction. Successful result.next says captures were delivered: the builder reads/judges, the collector records Git Findings, closures, MAP and the current authored BRIEF; another fetch is only for a missing external fact. ADR-0052 local received-package review remains available, and derived approval stays role-agnostic (2026-10-08, ADR-0150). Git command failures select the first `fatal:` or `error:` cause before applying the existing diagnostic bound; output without a cause marker and kit run summaries retain the final-line fallback (#326). | ADR-0148, ADR-0149, ADR-0150, ADR-0151, ADR-0152 | | `machine.mjs` | **Config saves preserve declarations** (#364): `saveConfig` patches a raw JSON object or its usable last-good snapshot, retaining unpatched role and posture typos for diagnostics rather than serializing resolved defaults. A broken config without a usable snapshot refuses an unrelated save with `CONFIG-UNREADABLE`; an explicit valid role declaration can repair it. The existing `claudeGate` migration remains, and retired keys are dropped. The config carries `searxngUrl` (2026-09-30, ADR-0104): a URL, not a secret. `shape()` keeps only the non-empty strings of `skillRoots`: every other key it reads is type-checked, and this one was checked as an array and passed its contents to `path.join`, so `{"skillRoots":[42]}` killed `doctor` with a raw ERR_INVALID_ARG_TYPE stack (break-test 2026-10-01). A dropped entry is silent; naming it would be a new finding, which the freeze does not allow. `editGate.settingsPath` is taken only as a non-empty string, else the anchor (2026-10-02, arena #198): it was the one key `shape()` passed through raw, and a numeric value reached `path.dirname` in `installEditGate` and killed `install-hooks` with a raw stack - the sibling of the `skillRoots` check, found a day later because the first fix was never swept to it. **An ill-typed posture key resolves to its restrictive state** (ADR-0121, 2026-10-02): `shape()` resolves a `failOpen`, `evidencePolicy` or `editGate.mode` that is present and not of its type to `false`, `strict` and `hard-block` through one `postureKey` helper, lists it in `illTyped`, and `readMachineConfig`/`posture` carry `illTypedKeys` beside `retiredKeys` on every path, the snapshot one included. Absent keys keep their defaults. Until then a typo fell to the lax default, the only place a typo relaxed a gate in silence. `browserPath` is read and kept (2026-10-11): it was missing from `shape()`, so the key the browser transport's errors name was never read, and every `saveConfig` - `install-hooks --role` among them - erased it. Only non-array JSON objects may create or restore a last-good snapshot (#368, 2026-10-13). A parsed scalar/array leaves an existing good snapshot intact; an unusable backup falls back to fail-closed with role unknown. | **Everything outside the project**: machine config (fail-open posture, `editGate` mode and settings path, `evidencePolicy`, `role`), read in **three states** — `absent` (nothing configured; the fail-open default is the answer), `readable`, and **`unreadable`** (the file exists and does not parse), which holds the last config that parsed from a snapshot beside it and, with nothing to hold, **fails closed**. `readMachineConfig()` returns the settings *and* `state`/`source`/`error`, because the settings alone cannot say whether fail-closed was configured or is the fallback this machine adopted (ADR-0020). Install state, git `core.hooksPath` (scope-aware readers that expand Git include and includeIf directives, including scoped reads, so installation preserves and restores the included hooks folder; #369), and the **runtime anchors** — `RUNTIME_ANCHORS`, the one place a runtime's paths are named (settings file, personal skill roots, project-relative skill dir), each a default that `runtimePaths()` / `skillLocations()` let the machine config override (ADR-0012); Git readers/writers accept an explicit child environment, preserving production defaults while allowing R1's disposable machine/Git fixture to isolate global/system origins; reads a retired config key's value into the current shape rather than Its six home-based paths (settings, skills, `agentsHome`, `kitHome`, the config, the install state) use `homeDir()`, so `HOME=""` no longer makes them relative (2026-09-28). dropping a setting the operator had tightened, and reports a retired ENVIRONMENT variable (`CLAUDE_SETTINGS_PATH`) without ever honouring it. Owns the machine role and the collection policy that follows from it (`machineRole`, `collectionPolicy`, `collectionRefusal`) — the single answer to "may this machine collect?". The only module that reaches past the project root. **`localHooksPathOverride` asks git from `cwd`** (2026-10-03, outside audit): git finds the repository by walking up, so a subdirectory - a nested decision project, the folder install-hooks is run from - is governed by the same local `core.hooksPath` as the root; the probe looked for `.git` beside `cwd` and said `null` below it, and `doctor`'s `gateHealth` goes through the same probe now. `previousHooksPath(opts)` reads the folder the install replaced (2026-10-03), the reader beside `setPreviousHooksPath`, with `--type=path` as `hand-on.sh` reads it - the raw value made doctor call a working `~/custom-hooks` missing (second audit, 2026-10-03). `repoTopLevel(opts)` is `git rev-parse --show-toplevel` from a cwd, or null outside a repository (2026-10-03). `skillRoots(env)` is the personal skill roots - the config's `skillRoots`, else the anchor's - which `skillLocations()` and the skill set's deploy both read (2026-10-06, ADR-0146). Install-state reads require an object and shape existing path, path-array and skill-hash fields before consumers use them; doctor names an existing unreadable or pathless record in its deploy warning (#365). Snapshot resolution keeps source naming the broken config and carries snapshotSource separately; doctor and collection refusal distinguish held settings/role from fallback-closed instead of claiming the snapshot failed to parse (#366). | ADR-0002, ADR-0010, ADR-0012 | -| `handoff.mjs` | The **arrival question**: `verifyHandoff(root)` reports a ledger that is absent or empty, an `EVIDENCE.md` that is absent or holds no evidence table (`handoff-evidence-missing`, `handoff-evidence-unparsed`; the report carries `rows`, and the CLI and doctor print the count - 2026-10-02: with no rows there were no citations, so "every cited capture on disk" held vacuously and handoff said OK over a corpus whose table a sync tool had dropped), every capture an evidence row names that is not on disk, and a chain that does not verify — each named. Read-only by design: the machine that asks cannot collect the missing bytes. The report also carries `lineEndings`, the captures whose body hash fails *only* because git rewrote CRLF into them on checkout, and **`handoffRemedy(report)` picks the remedy from the cause** rather than printing one text for every way a corpus can look broken (ADR-0020): `HANDOFF_REMEDY` when something did not travel (the collector must push `research/raw/`, dotfiles included), `lineEndingRemedy` when everything travelled and this machine rewrote it (pin with `.gitattributes`, re-check out, touch no collector and spend no credits), both when a report holds both. Consumed by `bin/handoff.mjs` and doctor (blocker on `role=builder`, silent on a collector; doctor passes the snapshot it already read, so the question is composed once for both). The report carries `nothingCollected` (no ledger entry, no evidence row, no capture) since 2026-09-27, so a caller can tell a project never collected from a corpus that failed to travel. The finding names are unchanged, so the doctor and the artifact validator read them as before. The report also carries `brief: { state, todo }` (`briefReview`, 2026-09-27, ADR-0061): the CLI's "handoff OK" is followed by a note when the brief is a draft (naming the judged sections still TODO) or has not been drafted. It is never a finding, and the exit stays 0. **The line-ending remedy is judged by running it** (2026-09-27, ADR-0062): per affected capture it prints `git rm --cached` (the index entry only) then `git checkout HEAD --`, which rewrites the file as LF through the pin, and it prints the pin (`PIN_LINES`, to add in an editor) only when `.gitattributes` lacks it. Every printed command is a plain git command with no comment, so cmd and PowerShell run it as written (ADR-0070). `--renormalize` had fixed only the index, so the capture handoff reads stayed CRLF. A **third cause** (2026-09-28): a ledger present and empty while captures are on disk lost its entries where the captures are - a one-line ledger whose torn tail `doctor --fix-arity` dropped - so `ledgerLost` gets `ledgerLostRemedy()`: restore it from git, or re-collect with `--force`. It was given the push remedy, which on the collector pushes the same empty ledger. A ledger that is merely **short** is the same cause: a cited capture on disk that no entry records is `handoff-capture-unledgered` (2026-09-28) and gets the same remedy - handoff had said OK and exited 0 while preflight failed it as `fetch-entry-exists`. **`CORPUS_ADD`** (2026-09-28) is how the kit prints putting a corpus into git - `git add research/`, then `git add -f research/raw/.fetches.jsonl` - shared by the push remedy and the commit gate's fix: both said `git add -f` on the whole folder, which also committed the machine-local `.usage.jsonl`, `.failures.jsonl`, `.diagnostics.jsonl` and `.fetches.lock`. A **fourth cause** (2026-10-04, break-test pass 4, F5b): a capture whose body hash fails and NOT by line endings (`body-unmodified/modified`) was changed after its fetch, and the report carries it as `altered`; it no longer counts as "did not travel", and **`alteredRemedy(files, { isRepo })`** prints, per file, `git status --porcelain --` to tell a local change from a committed one, then `git checkout HEAD --` followed by handoff again (no folder-wide checkout, which would discard uncommitted work beside it), and for the committed case `git log --oneline --` to find the commit that held the fetched bytes and restore from it, here or on the collector; without git metadata it prints no command. Judged by running it (2026-10-04, external review): a dirty status says only that this checkout changed the file after the commit - a capture committed altered and edited again is dirty too - so the remedy no longer claims the committed copy is the fetched one; and `--force` is no longer offered, because a new fetch writes a new capture beside this one, or refuses to overwrite a differing source sibling, while `verifyLedger` still checks the entry that names this file. The push remedy it used to get sends the same bytes again when the change was committed, and sends an operator to another machine when the committed copy here was one checkout away. **`safePathspec(file)`** (2026-10-04, two external reviews) decides whether a capture path is printed inside a git command at all: letters, digits, `.`, `_`, `-` and `/`, the alphabet the collector's own names come from. Anything else is named but never printed as a command, because no one spelling is read literally by cmd, PowerShell and sh alike and by git: a space unquoted is two pathspecs, `$copy` inside double quotes is expanded to nothing by PowerShell, and `[12]` is a git glob whatever the shell did, so `git checkout HEAD --` restored a neighbour and discarded its uncommitted work. For those the remedy prints commands that type no name at all: the operator lists them, one per line, in `handoff-names.txt` (`NAMES_FILE`) and runs `git --literal-pathspecs checkout HEAD --pathspec-from-file=handoff-names.txt` (the line-ending remedy adds the `git rm --cached` step the same way) - "quote it for your shell and prefix `:(literal)`" was not enough, because cmd expands `%NAME%` inside double quotes and PowerShell `$name` (third external review, 2026-10-04, verified by running in cmd); `--pathspec-from-file` hands git the bytes of the file with no shell in between, and `--literal-pathspecs` switches pattern syntax off (both git 2.25+). Such a name reaches the ledger rarely but not only by hand: `captureName` embeds the URL's host as it is, so an IPv6 literal or an adapter-supplied host outside DNS's alphabet produces one (third review). The classification runs over EVERY affected file (`classify`), not the five the text names: an unsafe sixth file got no note and "handle the remaining files the same way, each by name" - an instruction to type the glob that restores a neighbour (third external review, 2026-10-04). **`insideRepository(root)`** (2026-10-04, two external reviews) decides whether the runnable remedies are printed, and it asks git: `git -C <root> rev-parse --is-inside-work-tree`. It was `root/.git` alone, so a nested decision project (ADR-0030), which has none of its own, was told it had "no git metadata"; then a walk up for a `.git` entry, which walked past a bare repository the project sat inside and printed a checkout git refused with "this operation must be run in a work tree". The commands are git's, so git decides whether they run; without git on PATH the walk is the fallback: it walks the PHYSICAL path (realpath first - a project reached through a junction into a repository had read as outside it, and one linked out of a repository as inside; third review) and reads `GIT_CEILING_DIRECTORIES` as git does: absolute entries only, each through realpath until an empty entry, after which the rest are taken as written. Git's pathspecs are relative to the cwd, so the printed commands run unchanged from the project folder. The tests that want "no git metadata" from a scratch fixture set `GIT_CEILING_DIRECTORIES` above it and go through `handoffRemedy`, so a detection that always says "repository" fails them. **`repositoryState(root)`** is the full answer, `{ inside, refused }`: when git answered with something other than yes, "not a git repository" or "must be run in a work tree" - `fatal: detected dubious ownership` - `refused` carries its first line and both remedies say git refused to read the repository and to fix that first, where they said "no git metadata, re-copy the corpus" (third review, 2026-10-04); no command is printed either way. | ADR-0011, ADR-0020 | +| `handoff.mjs` | The **arrival question**: `verifyHandoff(root)` reports a ledger that is absent or empty, an `EVIDENCE.md` that is absent or holds no evidence table (`handoff-evidence-missing`, `handoff-evidence-unparsed`; the report carries `rows`, and the CLI and doctor print the count - 2026-10-02: with no rows there were no citations, so "every cited capture on disk" held vacuously and handoff said OK over a corpus whose table a sync tool had dropped), every capture an evidence row names that is not on disk, and a chain that does not verify — each named. Read-only by design: the machine that asks cannot collect the missing bytes. The report also carries `lineEndings`, the captures whose body hash fails *only* because git rewrote CRLF into them on checkout, and **`handoffRemedy(report)` picks the remedy from the cause** rather than printing one text for every way a corpus can look broken (ADR-0020): `HANDOFF_REMEDY` when something did not travel (the collector must push `research/raw/`, dotfiles included), `lineEndingRemedy` when everything travelled and this machine rewrote it (pin with `.gitattributes`, re-check out, touch no collector and spend no credits), both when a report holds both. Consumed by `bin/handoff.mjs` and doctor (blocker on `role=builder`, silent on a collector; doctor passes the snapshot it already read, so the question is composed once for both). The report carries `nothingCollected` (no ledger entry, no evidence row, no capture) since 2026-09-27, so a caller can tell a project never collected from a corpus that failed to travel. The finding names are unchanged, so the doctor and the artifact validator read them as before. The report also carries `brief: { state, todo }` (`briefReview`, 2026-09-27, ADR-0061): the CLI's "handoff OK" is followed by a note when the brief is a draft (naming the judged sections still TODO) or has not been drafted. It is never a finding, and the exit stays 0. **The line-ending remedy is judged by running it** (2026-09-27, ADR-0062): per affected capture it prints `git rm --cached` (the index entry only) then `git checkout HEAD --`, which rewrites the file as LF through the pin, and it prints the pin (`PIN_LINES`, to add in an editor) only when `.gitattributes` lacks it. Every printed command is a plain git command with no comment, so cmd and PowerShell run it as written (ADR-0070). `--renormalize` had fixed only the index, so the capture handoff reads stayed CRLF. A **third cause** (2026-09-28): a ledger present and empty while captures are on disk lost its entries where the captures are - a one-line ledger whose torn tail `doctor --fix-arity` dropped - so `ledgerLost` gets `ledgerLostRemedy()`: restore it from git, or re-collect with `--force`. It was given the push remedy, which on the collector pushes the same empty ledger. A ledger that is merely **short** is the same cause: a cited capture on disk that no entry records is `handoff-capture-unledgered` (2026-09-28) and gets the same remedy - handoff had said OK and exited 0 while preflight failed it as `fetch-entry-exists`. **`CORPUS_ADD`** (2026-09-28) is how the kit prints putting a corpus into git - `git add research/`, then `git add -f research/raw/.fetches.jsonl` - shared by the push remedy and the commit gate's fix: both said `git add -f` on the whole folder, which also committed the machine-local `.usage.jsonl`, `.failures.jsonl`, `.diagnostics.jsonl` and `.fetches.lock`. A **fourth cause** (2026-10-04, break-test pass 4, F5b): a capture whose body hash fails and NOT by line endings (`body-unmodified/modified`) was changed after its fetch, and the report carries it as `altered`; it no longer counts as "did not travel", and **`alteredRemedy(files, { isRepo })`** prints, per file, `git status --porcelain --` to tell a local change from a committed one, then `git checkout HEAD --` followed by handoff again (no folder-wide checkout, which would discard uncommitted work beside it), and for the committed case `git log --oneline --` to find the commit that held the fetched bytes and restore from it, here or on the collector; without git metadata it prints no command. Judged by running it (2026-10-04, external review): a dirty status says only that this checkout changed the file after the commit - a capture committed altered and edited again is dirty too - so the remedy no longer claims the committed copy is the fetched one; and `--force` is no longer offered, because a new fetch writes a new capture beside this one, or refuses to overwrite a differing source sibling, while `verifyLedger` still checks the entry that names this file. The push remedy it used to get sends the same bytes again when the change was committed, and sends an operator to another machine when the committed copy here was one checkout away. **`safePathspec(file)`** (2026-10-04, two external reviews) decides whether a capture path is printed inside a git command at all: letters, digits, `.`, `_`, `-` and `/`, the alphabet the collector's own names come from. Anything else is named but never printed as a command, because no one spelling is read literally by cmd, PowerShell and sh alike and by git: a space unquoted is two pathspecs, `$copy` inside double quotes is expanded to nothing by PowerShell, and `[12]` is a git glob whatever the shell did, so `git checkout HEAD --` restored a neighbour and discarded its uncommitted work. For those the remedy prints commands that type no name at all: the operator lists them, one per line, in `handoff-names.txt` (`NAMES_FILE`) and runs `git --literal-pathspecs checkout HEAD --pathspec-from-file=handoff-names.txt` (the line-ending remedy adds the `git rm --cached` step the same way) - "quote it for your shell and prefix `:(literal)`" was not enough, because cmd expands `%NAME%` inside double quotes and PowerShell `$name` (third external review, 2026-10-04, verified by running in cmd); `--pathspec-from-file` hands git the bytes of the file with no shell in between, and `--literal-pathspecs` switches pattern syntax off (both git 2.25+). Such a name reaches the ledger rarely but not only by hand: `captureName` embeds the URL's host as it is, so an IPv6 literal or an adapter-supplied host outside DNS's alphabet produces one (third review). The classification runs over EVERY affected file (`classify`), not the five the text names: an unsafe sixth file got no note and "handle the remaining files the same way, each by name" - an instruction to type the glob that restores a neighbour (third external review, 2026-10-04). **`insideRepository(root)`** (2026-10-04, two external reviews) decides whether the runnable remedies are printed, and it asks git: `git -C <root> rev-parse --is-inside-work-tree`. It was `root/.git` alone, so a nested decision project (ADR-0030), which has none of its own, was told it had "no git metadata"; then a walk up for a `.git` entry, which walked past a bare repository the project sat inside and printed a checkout git refused with "this operation must be run in a work tree". The commands are git's, so git decides whether they run; without git on PATH the walk is the fallback: it walks the PHYSICAL path (realpath first - a project reached through a junction into a repository had read as outside it, and one linked out of a repository as inside; third review) and reads `GIT_CEILING_DIRECTORIES` as git does: absolute entries only, each through realpath until an empty entry, after which the rest are taken as written. Git's pathspecs are relative to the cwd, so the printed commands run unchanged from the project folder. The tests that want "no git metadata" from a scratch fixture set `GIT_CEILING_DIRECTORIES` above it and go through `handoffRemedy`, so a detection that always says "repository" fails them. **`repositoryState(root)`** is the full answer, `{ inside, refused }`: when git answered with something other than yes, "not a git repository" or "must be run in a work tree" - `fatal: detected dubious ownership` - `refused` carries its first line and both remedies say git refused to read the repository and to fix that first, where they said "no git metadata, re-copy the corpus" (third review, 2026-10-04); no command is printed either way. Unparsed ledger lines carry their physical line and tail position in `ledgerUnparsed`; handoff chooses the existing tail-repair or git-restore remedy. Until parsing is restored, secondary sequence/hash/link failures and unledgered captures do not establish travel failure or entry loss; independently missing captures still get the travel remedy (#296). | ADR-0011, ADR-0020 | | `bundle.mjs` | The **archive this project arrived as**, and how far the tree has moved from it. `BUNDLE_INDEX.md` lists 80 SHA-256 digests taken at the 2026-09-17 handover; until ADR-0028 **nothing had ever read them**, which is how ADR-0022 could treat them as a constraint for three days without noticing the constraint was already violated. `verifyBundle(root)` sorts every entry into unchanged / drifted / unexpected / missing, where "drifted" means a file in `EXPECTED_TO_DRIFT` — the corpus, the map, the ADR index, the manifest itself, (from 2026-09-20) `CONTEXT.md` and the root `README.md`, which are documentation ABOUT the code and go stale the moment it moves, and (from 2026-09-21) `AGENTS.md`, which is the file the agent protocol is *written in* — caught drifting by this check when ADR-0030 gave decision research a committed home and the rule that sends an agent there had to be where agents read their rules — each of which moves because one of this project's own rules *requires* it to. The list is deliberately narrow: exempting a planning document would leave the report unable to say anything, and a test asserts none is exempted. Hashes **bytes**, not decoded text. Reports; never fails a build — a changed file is a fact about history, and the only thing that makes it a problem is nobody knowing. `bin/bundle.mjs --json` prints `verifyBundle`'s result as JSON (2026-09-28): the flag was accepted and never read, so a caller parsing it got the markdown report. `START_HERE.md` joined `EXPECTED_TO_DRIFT` on 2026-09-29: the archived copy held the author's machine paths, and the portable one is the template rendered with `$HOME`. | ADR-0028 | | `git-config.mjs`, `config.mjs` | Compatibility shims → `machine.mjs`. | ADR-0002 | | `runtime.mjs` | The host prerequisites a command actually needs: `checkNode` (22+), `checkGit`, `checkPython` (3.11+ since 2026-09-28, ADR-0078 - one number with the READMEs and CI's pin, reading either stream because Python 2 printed its version to stderr, trying `python3` **first** and continuing past an old interpreter rather than stopping at it — `python` is an alias for 2.7 on the most common Linux and older-macOS layout, and stopping there reported "no usable Python" with a good 3.12 one name away; an interpreter whose `--version` prints no version is passed over like an old one, and a refusal quotes what it printed - it was accepted as "proceeding" until 2026-09-28, and failed later in the conformance tests; it names the interpreter it chose as `exe` (2026-09-28), and the test harness's `findPython` runs that one - the harness had its own probe, `python` first with no version check, so an old alias failed every cross-language test), **`MAX_PAGE_BYTES`, `boundedText`, `CHILD_OUTPUT_LIMIT` and `outputOverflow`** (2026-09-28, ADR-0082) bound every transport child - `boundedText` what a child downloads, the other two what it hands back: they ran under spawnSync's 1 MiB default, so a page just over it failed with ENOBUFS - and a Firecrawl scrape that large was paid for and lost - and an overflow is now one sentence, its partial output discarded; and `requireRuntime({ node, git, python })` which exits **3** — distinct from a product failure — naming the missing piece and how to install it. Deliberately per-command rather than one gate that demands everything: Python is used by the cross-language conformance runners and nothing else, and a command that only reads the corpus has no business refusing to run because a language it never calls is absent. **The Node line, judged** (2026-09-27, ADR-0046): `nodeLine` passes a supported line, warns on an odd line below 27 (never LTS, end-of-life after six months) and fails below the floor; `nodeHonoursEnvProxy` says whether fetch can use a configured proxy - 22.21+ on the 22 line, 24.0+ after, never 23 (docs/decisions/2026-09-27-node-support). **A configured proxy, for every request the kit makes** (ADR-0047): `PROXY_VARIABLES` / `proxyVariable`; `fetchEnv` for a child that fetches; and for a command that fetches in its own process, `envProxyPlan` / `honourEnvProxy` - `set` through `http.setGlobalProxyFromEnv()` on 24.14+/25.4+, `reexec` with the flag on 22.21+/24.0-24.13 (the same standard streams, stop signals forwarded, the child's exit code, and `--disable-warning=UNDICI-EHPA` - Node 22's "EnvHttpProxyAgent is experimental" on every run, and only that warning), `unsupported` with one stderr line naming the version to move to, `none` when no proxy is set or the operator chose at startup (the variable even at `0`, `--use-env-proxy` in argv or `NODE_OPTIONS`), `invalid` when a value Node reads (lowercase wins) is not an http(s) URL - `unusableProxy`; with the flag on such a value crashes every fetch inside Node, so it is left off, and the message shows `proxySpelling` without ever echoing the value. **`fallbackCost(name)`** (2026-09-30) words what falling back onto the fetch provider's search costs: free on `http-keyless` and `browser`, which have no meter, and "this spends fetch credits" for any other, because a provider the kit does not know may be metered. The line had said "spends fetch credits" on the keyless transport. **`fetchFailure(err)`** (2026-09-30) is the keyless and SerpAPI children's error text: Node's `fetch failed` plus the reason on `err.cause` (`getaddrinfo ENOTFOUND host`, `ECONNREFUSED`, `redirect count exceeded`), with any URL in the cause cut to its host so a key in a query string never rides along. Every network failure had read `fetch failed`. `fetchFailure` also reads the first inner error of an `AggregateError` cause that carries no message or code of its own (2026-09-30), as a dual-stack connection failure can. `fetchFailure` cuts every URL to its host in the failure's own message as well as in its cause (`hostOnly`, both return paths): Node throws a credential-bearing or unparseable URL with no cause and the whole URL in the message, and one signed URL in a plan was written nine times into the ledger as that entry's error (break-test 2026-10-01). **A body is read in the charset the server declared** (2026-10-03, output-reliability audit G7): `charsetOf(contentType)` reads the label, `decoderFor(contentType, bytes, who)` picks the decoder - a byte-order mark first, then the declared label through `TextDecoder`, a label it does not know falling back to UTF-8 with a sentence in `fallback` - and `boundedBody` returns `{ text, charset, fallback }`; `boundedText` is its text alone, for the JSON answers of SerpAPI, SearXNG and the Wayback Machine, which RFC 8259 makes UTF-8. Every body had been decoded as UTF-8 whatever the header said, so a `windows-1252` page's euro sign captured as U+FFFD and the hash proved a page the server never sent. A legacy label over bytes that are valid multi-byte UTF-8 - the common server misconfiguration - is read as UTF-8, because such bytes are almost never intended as Latin-1 and the always-UTF-8 reading had got those pages right by accident. The review of that fix (2026-10-03) added three readings and took one back: bytes that are not valid UTF-8 under a UTF-8 label, or none - the page that names its charset only in a `<meta>` tag - are a named fallback instead of silent U+FFFD; a UTF-16 label over a body with no NUL byte (`hasNul`), which UTF-16 text of a page cannot avoid, is not believed and the body is read as UTF-8, where it had decoded to CJK garbage end to end; an unknown label over pure ASCII is an exact reading, not a fallback; and `boundedText` reads as UTF-8 whatever the label says, because the JSON callers had always had that reading and a parse decides what it means - `boundedBytes` is the shared bounded read beneath both. `charsetOf` drops a stray quote (`charset="utf-8`). `nodeDecodesWindows1252` is the version rule behind `nodeLine`'s second `fail` (2026-10-11, ADR-0159): Node 22.13.0 through 22.22.0 read `windows-1252` as Latin-1 (nodejs/node#60888), so the declared-charset reading above silently lost 0x80-0x9F on those builds; `doctor` names the defect and the first good build, 22.22.1. `usableProxyUrl` mirrors Undici origin parsing: pathname must be `/` and search/hash must be empty (#380, 2026-10-12). A non-origin proxy URL follows the existing invalid plan, preserving direct requests and the named diagnostic rather than crashing during proxy initialization. `envProxyChoice` distinguishes an unspecified startup choice from enabled/disabled, with the final explicit flag taking precedence over the variable and command-line flags over `NODE_OPTIONS` (#381, 2026-10-12). `proxyChosen` uses that reader to retain the existing no-override rule; only an exact variable value of `1` enables the proxy. | ADR-0046, ADR-0047, ADR-0159 | @@ -268,7 +274,7 @@ build approval. The shared verdict library and all gated formatting stay unchang (→ evaluate; `--staged-stdin` reads the piped path list; `--posture` answers 0 allow / 1 fail-closed / 2 unreadable-and-closed, so no shell has to parse JSON), `doctor.mjs` (`--fix-arity`), `install.mjs` (deploy + `--into` per-project binding, role-aware next steps; collector keyless instructions name the selected machine config rather than a per-run research flag, builder instructions require complete/current review, and an unresolved role prints diagnostics before role-specific advice (2026-10-08, ADR-0150); `tolerateClosedStdout` before its first write, so a reader that leaves early does not turn a finished deploy into exit 1, 2026-09-30), `install-hooks.mjs` (`--fail-closed`, `--role`; a `CONFIG-UNREADABLE` save refusal is named with exit 2 before the requested config patch is written, #364), -`new-project.mjs` (its first next step omits `--topic` once the project has one, ADR-0056), `decompose.mjs` (phase 0 CLI, refused on a builder by role), +`new-project.mjs` (its first next step omits `--topic` once the project has one, ADR-0056), `decompose.mjs` (phase 0 CLI, refused on a builder by role; `--limit` requires at least 1 at the CLI boundary, matching the library's positive search-result limit, while `--max-scrapes` still allows 0, #320), `timeline.mjs`, `audit.mjs` (lists and resolves versions through the manifest's reader), `brief.mjs` (drafts the handoff with all evaluated warning reasons, remedies and row/unknown targets, labeled as whole-corpus observations; it reports the brief's state; it, @@ -287,7 +293,11 @@ environment variable at all**, so it cannot depend on a credential or a config. maps to exit code — `PASS` 0, `FAIL`/`REOPEN` 1, `INCOMPLETE` 2, `BLOCKED` 3, unknown 4 — and a **usage error exits 2 from every subcommand**, printing the usage block to stderr; its argument parser names a value flag without a value before resolving any path, -including when the next token is another flag; +including when the next token is another flag. `validate` refuses blank, non-finite or +negative `--snapshot-max-age-ms` values as usage errors before judging evidence (#343). +The `validateRelease` and `validateSnapshotEvidence` library boundaries report an invalid +age allowance as `INPUT-SNAPSHOT-MAX-AGE` / `INCOMPLETE`; `SNAPSHOT-STALE` is reserved for +an actual snapshot older than a valid allowance; `test/support-policy.test.mjs` pins that by running the CLI. It writes exactly one thing, only when asked: `fi-validate --report <file>`), `selftest.mjs` (every requested group must match a test file before any test starts; an unmatched group is named and refused with exit 2, including beside a valid group; #332. CLI regressions isolate child report paths from the parent suite; the whole suite, queued by the harness — which **awaits** every test, diff --git a/research-kit/README.md b/research-kit/README.md index c06356d9..483b1a98 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1845 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1861 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/bin/collect-remote.mjs b/research-kit/bin/collect-remote.mjs index d0166a30..b1884b6c 100644 --- a/research-kit/bin/collect-remote.mjs +++ b/research-kit/bin/collect-remote.mjs @@ -150,12 +150,9 @@ checkFlagValues(flags, { 'search-transport': { choices: ['auto', 'serpapi', 'firecrawl-cli'] }, timeout: { int: true, min: 1 }, out: {}, workflow: {}, ref: {}, + repository: {}, topic: {}, 'client-ref': {}, prior: {}, prefer: {}, }, { exit: EXIT.CANNOT_START }); -if (flags['client-ref'] === true) { - process.stderr.write('--client-ref needs a value\n'); - process.exit(EXIT.CANNOT_START); -} const clientReference = checkClientRef(flags['client-ref']); if (!clientReference.ok) { process.stderr.write(`${redact(clientReference.detail)}\n`); diff --git a/research-kit/bin/decompose.mjs b/research-kit/bin/decompose.mjs index 397e96f3..235ff8a7 100644 --- a/research-kit/bin/decompose.mjs +++ b/research-kit/bin/decompose.mjs @@ -18,7 +18,7 @@ refuseUnknownFlags(flags, ['help', 'topic', 'dry-run', 'force', 'limit', 'max-sc // made 1, so `--max-scrapes` with no value quietly paid for one page. checkFlagValues(flags, { topic: 'value', recipe: 'value', transport: 'value', 'search-transport': 'value', - limit: { int: true, min: 0 }, 'max-scrapes': { int: true, min: 0 }, + limit: { int: true, min: 1 }, 'max-scrapes': { int: true, min: 0 }, }); const root = process.cwd(); @@ -131,7 +131,7 @@ try { process.stderr.write(`${why}\n`); process.exit(2); } - if (err.code === 'UNKNOWN_RECIPE' || err.code === 'INVALID_BUDGET' || err.code === 'SEARCH_PROVIDER_NOT_READY') { + if (err.code === 'UNKNOWN_RECIPE' || err.code === 'INVALID_RECIPE' || err.code === 'INVALID_BUDGET' || err.code === 'SEARCH_PROVIDER_NOT_READY') { process.stderr.write(`${err.message}\n`); process.exit(2); } diff --git a/research-kit/bin/researcher-release.mjs b/research-kit/bin/researcher-release.mjs index 17d34d11..8ac95b91 100644 --- a/research-kit/bin/researcher-release.mjs +++ b/research-kit/bin/researcher-release.mjs @@ -54,6 +54,12 @@ function main(argv = process.argv.slice(2)) { if (parsed.error) { console.error(`${parsed.error}\n${usage()}`); return 2; } let result; if (parsed.command === 'validate') { + const snapshotMaxAgeMs = parsed.snapshotmaxagems === undefined ? undefined : Number(parsed.snapshotmaxagems); + if (parsed.snapshotmaxagems !== undefined + && (!parsed.snapshotmaxagems.trim() || !Number.isFinite(snapshotMaxAgeMs) || snapshotMaxAgeMs < 0)) { + console.error(`--snapshot-max-age-ms must be a finite non-negative number\n${usage()}`); + return 2; + } if (!parsed.root || !parsed.package || !parsed.registry || !parsed.roles || !parsed.pointers) { console.error(usage()); return 2; @@ -78,7 +84,7 @@ function main(argv = process.argv.slice(2)) { headPath: parsed.head ? path.resolve(parsed.head) : null, genesisHash: parsed.genesishash ?? '0'.repeat(64), publicKeys, - snapshotMaxAgeMs: parsed.snapshotmaxagems === undefined ? undefined : Number(parsed.snapshotmaxagems), + snapshotMaxAgeMs, }); } else if (parsed.command === 'conform') { if (!parsed.schema) { console.error(usage()); return 2; } diff --git a/research-kit/lib/auto-collect.mjs b/research-kit/lib/auto-collect.mjs index 5621d606..578ec5ca 100644 --- a/research-kit/lib/auto-collect.mjs +++ b/research-kit/lib/auto-collect.mjs @@ -114,6 +114,11 @@ const tail = (s, n = RUN_OUTPUT_BYTES) => { return bytes.length > n ? `[... earlier output truncated ...]\n${bytes.subarray(-n).toString('utf8')}` : text; }; const lastLine = (s) => tail(String(s ?? '').trim().split(/\r?\n/).filter(Boolean).pop() ?? '', 1000); +// Git puts the cause before multi-line advice; kit run summaries still use lastLine. +const gitReason = (output) => { + const cause = String(output ?? '').split(/\r?\n/).find((line) => /^(fatal|error):/.test(line)); + return cause === undefined ? lastLine(output) : tail(cause, 1000); +}; const posix = (p) => p.split(path.sep).join('/'); /** Legacy diagnostic parser. Collection accounting never trusts human-readable logs. */ @@ -182,7 +187,7 @@ export function createAutoCollect({ const ahead = await git(['rev-list', '--count', '@{u}..HEAD'], top); if (ahead.truncated) return 'could not inspect local commits: Git output was truncated; deliver manually'; const count = Number(ahead.output.trim()); - if (ahead.code !== 0) return `could not inspect local commits: ${lastLine(ahead.output)}`; + if (ahead.code !== 0) return `could not inspect local commits: ${gitReason(ahead.output)}`; if (!/^\d+$/.test(ahead.output.trim()) || !Number.isSafeInteger(count)) { return 'could not inspect local commits: invalid Git ahead count; deliver manually'; } @@ -195,13 +200,13 @@ export function createAutoCollect({ } const changed = await git(['diff-tree', '--no-commit-id', '--name-only', '-r', '-z', 'HEAD'], top); if (changed.truncated) return 'could not inspect pending commit paths: Git output was truncated; deliver manually'; - if (changed.code !== 0) return `could not inspect pending delivery commit: ${lastLine(changed.output)}`; + if (changed.code !== 0) return `could not inspect pending delivery commit: ${gitReason(changed.output)}`; const allowed = changed.output.split('\0').filter(Boolean).every((file) => paths.some((base) => ( file === base || file.startsWith(`${base.replace(/\/$/, '')}/`) ))); if (!allowed) return 'the pending auto-collect commit contains paths outside this request; push it manually'; const pushed = await git(['push'], top); - return pushed.code === 0 ? 'pushed' : `push failed: ${lastLine(pushed.output)}`; + return pushed.code === 0 ? 'pushed' : `push failed: ${gitReason(pushed.output)}`; } } const scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'research-kit-autocollect-index-')); @@ -209,33 +214,33 @@ export function createAutoCollect({ const isolatedGit = (args) => exec('git', args, { cwd: top, env: indexEnv }); try { const initialized = await isolatedGit(['read-tree', 'HEAD']); - if (initialized.code !== 0) return `could not initialize isolated git index: ${lastLine(initialized.output)}`; + if (initialized.code !== 0) return `could not initialize isolated git index: ${gitReason(initialized.output)}`; for (const p of paths) { const r = await isolatedGit(['add', '--', p]); - if (r.code !== 0) return `git add ${p} failed: ${lastLine(r.output)}`; + if (r.code !== 0) return `git add ${p} failed: ${gitReason(r.output)}`; } for (const p of paths) { if (!(p === 'research' || p.endsWith('/research'))) continue; const ledger = path.posix.join(p.slice(0, -'research'.length) || '.', LEDGER); if (fs.existsSync(path.join(top, ledger))) { const r = await isolatedGit(['add', '-f', '--', ledger]); - if (r.code !== 0) return `git add -f ${ledger} failed: ${lastLine(r.output)}`; + if (r.code !== 0) return `git add -f ${ledger} failed: ${gitReason(r.output)}`; } } const staged = await isolatedGit(['diff', '--cached', '--quiet']); - if (staged.code !== 0 && staged.code !== 1) return `could not inspect isolated git index: ${lastLine(staged.output)}`; + if (staged.code !== 0 && staged.code !== 1) return `could not inspect isolated git index: ${gitReason(staged.output)}`; const hasChanges = staged.code !== 0; if (hasChanges) { const c = await isolatedGit(['commit', '-m', report.subject, '-m', report.body]); - if (c.code !== 0) return `commit refused: ${lastLine(c.output)}`; + if (c.code !== 0) return `commit refused: ${gitReason(c.output)}`; const reset = await git(['reset', '--quiet', 'HEAD', '--', ...paths], top); - if (reset.code !== 0) return `committed; could not refresh staged paths: ${lastLine(reset.output)}`; + if (reset.code !== 0) return `committed; could not refresh staged paths: ${gitReason(reset.output)}`; } const upstream = await git(['rev-parse', '--abbrev-ref', '--symbolic-full-name', '@{u}'], top); if (upstream.truncated) return `${hasChanges ? 'committed; ' : ''}could not inspect upstream: Git output was truncated; deliver manually`; if (upstream.code !== 0) return hasChanges ? 'committed; no upstream to push to' : 'no upstream to push to'; const pushed = await git(['push'], top); - if (pushed.code !== 0) return hasChanges ? `committed; push failed: ${lastLine(pushed.output)}` : `push failed: ${lastLine(pushed.output)}`; + if (pushed.code !== 0) return hasChanges ? `committed; push failed: ${gitReason(pushed.output)}` : `push failed: ${gitReason(pushed.output)}`; return hasChanges ? 'committed and pushed' : 'pushed'; } finally { fs.rmSync(scratch, { recursive: true, force: true }); @@ -290,7 +295,7 @@ export function createAutoCollect({ if (!inside(canonical(top), physical)) return { problem: 'the approved topics folder resolves outside the repository' }; const gitDir = await git(['rev-parse', '--absolute-git-dir'], top); if (gitDir.truncated) return { problem: 'could not inspect Git metadata: Git output was truncated; inspect the target manually' }; - if (gitDir.code !== 0) return { problem: `could not inspect Git metadata: ${lastLine(gitDir.output)}` }; + if (gitDir.code !== 0) return { problem: `could not inspect Git metadata: ${gitReason(gitDir.output)}` }; if (gitDir.code === 0 && inside(canonical(gitDir.output.trim()), physical)) { return { problem: 'new topics cannot be created in Git metadata' }; } @@ -327,12 +332,12 @@ export function createAutoCollect({ const pathspecs = [...paths, `:(exclude)${excludedPath}`]; const status = await git(['status', '--porcelain=v1', '-z', '--untracked-files=all', '--', ...pathspecs], top); if (status.truncated) throw new Error('could not inspect pending delivery: Git output was truncated; deliver manually'); - if (status.code !== 0) throw new Error(`could not inspect pending delivery: ${lastLine(status.output)}`); + if (status.code !== 0) throw new Error(`could not inspect pending delivery: ${gitReason(status.output)}`); const hash = crypto.createHash('sha256'); hash.update(status.output); const diff = await git(['diff', '--binary', 'HEAD', '--', ...pathspecs], top); if (diff.truncated) throw new Error('could not fingerprint pending delivery: Git output was truncated; deliver manually'); - if (diff.code !== 0) throw new Error(`could not fingerprint pending delivery: ${lastLine(diff.output)}`); + if (diff.code !== 0) throw new Error(`could not fingerprint pending delivery: ${gitReason(diff.output)}`); hash.update(diff.output); for (const record of status.output.split('\0')) { if (!record.startsWith('?? ')) continue; @@ -563,7 +568,7 @@ export function createAutoCollect({ const top = await repoTop(dir); if (!top) { note(`${dir} is not in a git repository; requests travel by git`); return status(); } const pulled = await git(['pull', '--ff-only'], top); - if (pulled.code !== 0) note(`git pull did not run cleanly (${lastLine(pulled.output)}); collection blocked until synchronization succeeds`); + if (pulled.code !== 0) note(`git pull did not run cleanly (${gitReason(pulled.output)}); collection blocked until synchronization succeeds`); const requests = pulled.code === 0 ? listRequests(dir) : []; for (const item of requests) { diff --git a/research-kit/lib/corpus.mjs b/research-kit/lib/corpus.mjs index 08d88e78..48bff133 100644 --- a/research-kit/lib/corpus.mjs +++ b/research-kit/lib/corpus.mjs @@ -495,7 +495,9 @@ export function readLedger(root) { const file = resolve(root, PATHS.ledger); const text = readText(file); if (text === null) return { present: false, entries: [], problems: [], lines: [] }; - const lines = text.split(/\r?\n/); + // An editor's leading BOM is outside the entries the chain hashes. Ignore it while + // reading, as the JSON and Markdown readers do; never rewrite the evidence file. + const lines = text.replace(/^\uFEFF/, '').split(/\r?\n/); const entries = []; const problems = []; lines.forEach((line, idx) => { diff --git a/research-kit/lib/decompose.mjs b/research-kit/lib/decompose.mjs index 8c642592..684cc15a 100644 --- a/research-kit/lib/decompose.mjs +++ b/research-kit/lib/decompose.mjs @@ -28,7 +28,7 @@ export function parseRecipe(text) { let name = ''; if (match) { let inList = false; - for (const line of match[1].split(/\r?\n/)) { + for (const [offset, line] of match[1].split(/\r?\n/).entries()) { const named = line.match(/^name:\s*(.*)$/); if (named) { name = named[1].trim(); continue; } if (/^dimensions:\s*$/.test(line)) { inList = true; continue; } @@ -36,6 +36,11 @@ export function parseRecipe(text) { const item = line.match(/^\s*-\s+(.*)$/); if (!item) { if (line.trim()) inList = false; continue; } const [label, why] = item[1].split(/\s*\|\s*/); + if (!label.trim()) { + const err = new Error(`line ${offset + 2}: the dimension label is blank`); + err.code = 'INVALID_RECIPE'; + throw err; + } dimensions.push({ name: label.trim(), why: (why ?? '').trim() }); } } @@ -54,7 +59,17 @@ export function loadRecipe(nameOrPath) { err.code = 'UNKNOWN_RECIPE'; throw err; } - return { ...parseRecipe(text), file: direct }; + let parsed; + try { parsed = parseRecipe(text); } catch (err) { + if (err.code === 'INVALID_RECIPE') err.message = `recipe "${direct}" ${err.message}`; + throw err; + } + if (!parsed.dimensions.length) { + const err = new Error(`recipe "${direct}" holds no dimensions - use a front-matter dimensions list of "- label | why" lines; see ${RECIPE_DIR}`); + err.code = 'INVALID_RECIPE'; + throw err; + } + return { ...parsed, file: direct }; } /** Where the facts probably live: the official hosts a search keeps pointing at. */ diff --git a/research-kit/lib/finding.mjs b/research-kit/lib/finding.mjs index ae983462..c91f17ea 100644 --- a/research-kit/lib/finding.mjs +++ b/research-kit/lib/finding.mjs @@ -331,6 +331,9 @@ export function findingWithContext(markdown, fallback = '') { for (let i = 0; i < lines.length; i += 1) { const raw = lines[i]; + // Excerpts use physical paragraph boundaries, including ones inside skipped fences. + // Remember the start while scanning so recovery never inserts each line at the front. + if (!raw.trim() || /^\s{0,3}#{1,6}\s/.test(raw)) blockStart = i + 1; if (/^\s*```/.test(raw)) { inFence = !inFence; continue; } if (inFence) continue; @@ -363,7 +366,7 @@ export function findingWithContext(markdown, fallback = '') { // An FAQ question is usually plain text, not a heading, so the heading rules cannot // see it. Remember whether the previous non-empty line asked something. const answersQuestion = lastLine.endsWith('?'); - if (line) { if (!block.length) blockStart = i; block.push(line); lastLine = line; } + if (line) { block.push(line); lastLine = line; } if (isNoise(line)) continue; for (const sentence of sentences(line)) { const text = sentence.trim(); @@ -371,7 +374,7 @@ export function findingWithContext(markdown, fallback = '') { if (isNoise(text)) continue; // The excerpt is resolved after the paragraph closes, so a sentence carries the // whole paragraph it came from rather than just itself. - candidates.push({ text, fromTable: false, heading, headings: stack.filter(Boolean).join(' / '), answersQuestion, lineIndex: i }); + candidates.push({ text, fromTable: false, heading, headings: stack.filter(Boolean).join(' / '), answersQuestion, lineIndex: i, blockStart }); } } flushBlock(); @@ -396,10 +399,13 @@ export function findingWithContext(markdown, fallback = '') { // Recover the paragraph the winner sat in, bounded. A table row already has one. let excerpt = best.excerpt ?? ''; if (!excerpt && best.lineIndex !== undefined) { - const para = []; - for (let i = best.lineIndex; i >= 0 && lines[i]?.trim() && !/^\s{0,3}#{1,6}\s/.test(lines[i]); i -= 1) para.unshift(stripMarkdown(lines[i])); - for (let i = best.lineIndex + 1; i < lines.length && lines[i]?.trim() && !/^\s{0,3}#{1,6}\s/.test(lines[i]); i += 1) para.push(stripMarkdown(lines[i])); - excerpt = para.filter(Boolean).join(' ').trim(); + for (let i = best.blockStart; i < lines.length && lines[i]?.trim() && !/^\s{0,3}#{1,6}\s/.test(lines[i]); i += 1) { + const line = stripMarkdown(lines[i]); + if (!line) continue; + excerpt += `${excerpt ? ' ' : ''}${line}`; + // One character beyond the cap proves truncation, including an exact-cap prefix. + if (excerpt.length > EXCERPT_CHARS) break; + } } if (excerpt.length > EXCERPT_CHARS) excerpt = `${excerpt.slice(0, EXCERPT_CHARS).trimEnd()}…`; diff --git a/research-kit/lib/handoff.mjs b/research-kit/lib/handoff.mjs index 879d3bcd..26ac6e43 100644 --- a/research-kit/lib/handoff.mjs +++ b/research-kit/lib/handoff.mjs @@ -319,6 +319,8 @@ export function verifyHandoff(root, { corpus = null } = {}) { const missingCaptures = []; const unledgered = []; const lineEndings = chain.lineEndings ?? []; + const ledgerUnparsed = chain.problems.filter((p) => p.rule === 'ledger-unparsed'); + const parseConsequences = new Set(); if (!snapshot.ledger.present) { findings.push({ @@ -326,7 +328,7 @@ export function verifyHandoff(root, { corpus = null } = {}) { severity: 'fail', detail: `${PATHS.ledger} is not in this checkout - the corpus arrived without its ledger`, }); - } else if (!snapshot.ledger.entries.length) { + } else if (!snapshot.ledger.entries.length && !ledgerUnparsed.length) { findings.push({ name: 'handoff-ledger-empty', severity: 'fail', @@ -378,19 +380,24 @@ export function verifyHandoff(root, { corpus = null } = {}) { const structural = chain.problems.filter((p) => p.rule !== 'ledger-missing'); for (const problem of structural) { - findings.push({ + const finding = { name: 'handoff-chain-broken', severity: 'fail', // One line per capture, for the same reason the remedy is stated once: a // paragraph per capture buries the capture names. detail: `${problem.rule}${problem.kind ? `/${problem.kind}` : ''}: ${problem.detail}`, kind: problem.kind, - }); + }; + findings.push(finding); + // Missing sequence links cannot establish travel or loss while entries are unreadable. + // Restore parsing first; missing captures and altered bodies still have their own cause. + if (ledgerUnparsed.length && ['ledger-unparsed', 'seq', 'prev', 'entry-hash'].includes(problem.rule)) parseConsequences.add(finding); } // Present, empty, and captures on disk: lost here, not left behind. // Or entries missing for captures that are here - a torn last line of a longer ledger. const ledgerLost = snapshot.ledger.present + && !ledgerUnparsed.length && ((!snapshot.ledger.entries.length && snapshot.captures.entries.length > 0) || unledgered.length > 0); // A capture changed after its fetch is here, and so is its ledger entry: nothing failed to // travel, the bytes changed (alteredRemedy). A broken chain beside it still did. @@ -399,6 +406,7 @@ export function verifyHandoff(root, { corpus = null } = {}) { .map((p) => ({ file: p.file, line: p.line })); const travelled = findings.filter((f) => (f.name !== 'handoff-chain-broken' || (f.kind !== 'line-endings' && f.kind !== 'modified')) && f.name !== 'handoff-capture-unledgered' + && !parseConsequences.has(f) && !(ledgerLost && f.name === 'handoff-ledger-empty')); const report = { root, @@ -412,6 +420,11 @@ export function verifyHandoff(root, { corpus = null } = {}) { // changed after its fetch, or a ledger that lost entries here. didNotTravel: travelled.length > 0, ledgerLost, + ledgerUnparsed: ledgerUnparsed.map((p) => ({ + line: p.line, + // Physical line numbers include blank lines, just as readLedger and tail repair do. + tail: p.line === snapshot.ledger.lines.findLastIndex((line) => line.trim()) + 1, + })), entries: snapshot.ledger.entries.length, rows: snapshot.evidence.length, // Not a finding: the corpus can arrive whole while the brief is unreviewed. The CLI @@ -419,7 +432,7 @@ export function verifyHandoff(root, { corpus = null } = {}) { brief: briefReview(snapshot), // No ledger entry, no evidence row, no capture: nothing was ever collected, so there is // nothing that could have failed to travel. The CLI says so on a collector. - nothingCollected: !snapshot.ledger.entries.length && !snapshot.evidence.length && !snapshot.captures.entries.length, + nothingCollected: !ledgerUnparsed.length && !snapshot.ledger.entries.length && !snapshot.evidence.length && !snapshot.captures.entries.length, }; const remedy = handoffRemedy(report); if (remedy) report.findings.push({ name: 'handoff-remedy', severity: 'info', detail: remedy }); @@ -490,6 +503,11 @@ export function handoffRemedy(report) { const parts = []; if (report.didNotTravel) parts.push(HANDOFF_REMEDY); if (report.ledgerLost) parts.push(ledgerLostRemedy()); + for (const problem of report.ledgerUnparsed ?? []) { + parts.push(problem.tail + ? `${PATHS.ledger} line ${problem.line}: the last line is torn (an interrupted append): ${kitCommand('doctor.mjs', '--fix-arity')} drops it. Run handoff again after repair.` + : `${PATHS.ledger} line ${problem.line}: a broken line inside the chain cannot be repaired here: restore research/raw/.fetches.jsonl from git, then run handoff again.`); + } // Whether this is inside a repository at all decides which remedies are even runnable. const { inside: isRepo, refused } = repositoryState(report.root ?? '.'); if (report.altered?.length) parts.push(alteredRemedy(report.altered.map((e) => e.file), { isRepo, refused })); diff --git a/research-kit/lib/panel.mjs b/research-kit/lib/panel.mjs index 3bf71156..498ecff0 100644 --- a/research-kit/lib/panel.mjs +++ b/research-kit/lib/panel.mjs @@ -270,12 +270,22 @@ export function createPanel({ return new Promise((resolve, reject) => { let size = 0; const chunks = []; + let oversized = false; req.on('data', (chunk) => { + if (oversized) return; size += chunk.length; - if (size > MAX_BODY_BYTES) { reject(Object.assign(new Error('the request is too large'), { status: 413 })); req.destroy(); return; } + if (size > MAX_BODY_BYTES) { + oversized = true; + chunks.length = 0; + // The handler must send its refusal before the connection closes. Destroying + // the request here also destroys the socket carrying that response. + reject(Object.assign(new Error('the request is too large'), { status: 413 })); + return; + } chunks.push(chunk); }); req.on('end', () => { + if (oversized) return; const text = Buffer.concat(chunks).toString('utf8'); try { resolve(text ? JSON.parse(text) : {}); } catch { reject(Object.assign(new Error('the request is not JSON'), { status: 400 })); } }); @@ -401,6 +411,7 @@ export function createPanel({ try { return await api(req, res, route); } catch (err) { + if (err.status === 413) res.setHeader('Connection', 'close'); return refuse(res, err.status ?? 500, err.status ? err.message : 'the panel hit an error; the terminal that started it has the detail'); } } diff --git a/research-kit/lib/provenance.mjs b/research-kit/lib/provenance.mjs index de24e8c5..af3f7e07 100644 --- a/research-kit/lib/provenance.mjs +++ b/research-kit/lib/provenance.mjs @@ -544,7 +544,9 @@ export function repairLedgerTail(root) { const tail = lines[lines.length - 1]; let torn = false; - try { JSON.parse(tail); } catch { torn = true; } + // A one-entry ledger can carry the same leading editor BOM readLedger tolerates. + // It is not a torn append; later-line BOMs remain invalid and the file is untouched. + try { JSON.parse(lines.length === 1 ? tail.replace(/^\uFEFF/, '') : tail); } catch { torn = true; } if (!torn) return { repaired: false, reason: 'tail-parses' }; const head = lines.slice(0, -1).join('\n'); diff --git a/research-kit/lib/release-validator.mjs b/research-kit/lib/release-validator.mjs index 00c5fb1d..cc3c0f72 100644 --- a/research-kit/lib/release-validator.mjs +++ b/research-kit/lib/release-validator.mjs @@ -694,6 +694,10 @@ function snapshotRefValue(ref) { */ export function validateSnapshotEvidence(record, { root = null, now = Date.now(), maxAgeMs = SNAPSHOT_MAX_AGE_MS } = {}) { const errors = []; + if (!Number.isFinite(maxAgeMs) || maxAgeMs < 0) { + add(errors, 'INPUT-SNAPSHOT-MAX-AGE', 'snapshot age allowance must be a finite non-negative number', '--snapshot-max-age-ms', 'INCOMPLETE'); + return { status: resultStatus(errors), errors }; + } const meta = metaOf(record); const refs = snapshotRefs(record); if (!refs || typeof refs !== 'object' || Array.isArray(refs)) { @@ -719,7 +723,7 @@ export function validateSnapshotEvidence(record, { root = null, now = Date.now() const capturedMs = typeof capturedAt === 'string' && UTC.test(capturedAt) ? Date.parse(capturedAt) : NaN; if (!Number.isFinite(capturedMs)) add(errors, 'SNAPSHOT-TIME', `${phase} capturedAt must be UTC ISO-8601 with Z`, `${at}.capturedAt`, 'INCOMPLETE'); else if (capturedMs > now) add(errors, 'SNAPSHOT-FUTURE', `${phase} snapshot is dated in the future`, `${at}.capturedAt`); - else if (!Number.isFinite(maxAgeMs) || maxAgeMs < 0 || now - capturedMs > maxAgeMs) add(errors, 'SNAPSHOT-STALE', `${phase} path-authority snapshot is older than the allowed ${maxAgeMs}ms`, `${at}.capturedAt`); + else if (now - capturedMs > maxAgeMs) add(errors, 'SNAPSHOT-STALE', `${phase} path-authority snapshot is older than the allowed ${maxAgeMs}ms`, `${at}.capturedAt`); const fileName = ref.path ?? ref.file ?? ref.evidencePath; if (fileName === undefined) { @@ -987,6 +991,10 @@ export function validateRelease({ root, package: packageName, registryPath, role recoveryReceiptIds: [], errors, }; + if (!Number.isFinite(snapshotMaxAgeMs) || snapshotMaxAgeMs < 0) { + add(errors, 'INPUT-SNAPSHOT-MAX-AGE', 'snapshot age allowance must be a finite non-negative number', '--snapshot-max-age-ms', 'INCOMPLETE'); + return output; + } if (!root || !fs.existsSync(root) || !inside(process.cwd(), root) && !path.isAbsolute(root)) add(errors, 'INPUT-ROOT', 'root must name an existing directory', '--root', 'INCOMPLETE'); if (!PACKAGES.has(packageName)) add(errors, 'INPUT-PACKAGE', 'package must be R28, R29, R30, R31, or R32', '--package', 'INCOMPLETE'); const registryLoaded = readJson(registryPath); diff --git a/research-kit/test/auto-reservation.test.mjs b/research-kit/test/auto-reservation.test.mjs index 71aed4ef..311fc5cd 100644 --- a/research-kit/test/auto-reservation.test.mjs +++ b/research-kit/test/auto-reservation.test.mjs @@ -63,6 +63,32 @@ function launch(file, dir, marker, held, env) { return { child, closed }; } +test('Git failure notes quote the first fatal or error cause instead of the advice tail', async () => { + const { env, projects } = fixture(); + const dir = projects[0]; + const cases = [ + ["fatal: '/nonexistent/remote.git' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n", + "fatal: '/nonexistent/remote.git' does not appear to be a git repository"], + ['progress\r\nerror: cannot lock ref\r\nPlease try again.\r\n', 'error: cannot lock ref'], + ['notice\nfinal diagnostic\n\n', 'final diagnostic'], + ['', ''], + ]; + for (const [output, reason] of cases) { + const calls = []; + const exec = async (command, args) => { + assertEqual(command, 'git', 'a failed pull must not start collection'); + calls.push(args); + if (args[0] === 'rev-parse') return { code: 0, output: dir }; + assertEqual(args[0], 'pull'); + return { code: 1, output }; + }; + const collector = createAutoCollect({ project: () => dir, env, kitRoot: KIT_ROOT, exec }); + const status = await collector.cycle(); + assert(status.notes.some((note) => note.includes(`git pull did not run cleanly (${reason}); collection blocked`)), JSON.stringify(status.notes)); + assertEqual(calls.length, 2, 'synchronization failure must block all further commands'); + } +}); + async function waitFile(file) { const started = Date.now(); while (!fs.existsSync(file)) { diff --git a/research-kit/test/cli.test.mjs b/research-kit/test/cli.test.mjs index 3f6c7a1d..8a3ef86b 100644 --- a/research-kit/test/cli.test.mjs +++ b/research-kit/test/cli.test.mjs @@ -33,7 +33,7 @@ test('collect-remote refuses unsafe client references before token lookup or cre const bare = run('collect-remote.mjs', ['--repository', 'o/r', '--topic', 't', '--client-ref'], { root }); assert.equal(bare.status, 3); assert.match(bare.err, /--client-ref needs a value/); - for (const clientRef of ['', 'job-1.a_b', 'a'.repeat(64)]) { + for (const clientRef of ['job-1.a_b', 'a'.repeat(64)]) { const r = run('collect-remote.mjs', ['--repository', 'o/r', '--topic', 't', '--client-ref', clientRef, '--no-wait'], { root }); assert.equal(r.status, 3); assert.match(r.err, /no GitHub token/, 'valid input must reach the existing missing-token refusal'); @@ -841,6 +841,24 @@ test('collect-remote refuses a bare --out, --workflow or --ref before it creates } }); +test('collect-remote refuses empty required and optional values before token lookup or creating out', () => { + for (const flag of ['repository', 'topic', 'client-ref', 'prior', 'prefer']) { + const required = flag === 'repository' ? ['--topic', 't'] + : flag === 'topic' ? ['--repository', 'o/r'] : ['--repository', 'o/r', '--topic', 't']; + const spellings = [[`--${flag}=`], [`--${flag}`, '']]; + if (!['repository', 'topic'].includes(flag)) spellings.push([`--${flag}`]); + for (const spelling of spellings) { + const root = project(); + const out = path.join(root, 'not-created'); + const r = run('collect-remote.mjs', [...required, '--out', out, ...spelling], { root }); + assert.equal(r.status, 3, r.all); + assert.equal(r.err.split('\n')[0], `--${flag} needs a value`, r.all); + assert.doesNotMatch(r.all, /no GitHub token/); + assert.equal(fs.existsSync(out), false, 'invalid flags must not create the output directory'); + } + } +}); + // Found 2026-10-01: lib/dispatch.mjs told the caller to "retry with --run-id <n>", exit 4 said // "the run id is on stdout", and collect-remote had no --run-id - it refused it as an unknown // option. A watch that ended (a timeout, a dropped connection, a closed laptop) could not be @@ -1032,7 +1050,9 @@ test('a flag given a bad value, or none, is refused before anything runs', () => // Number() makes 1 - so a real run paid for one page nobody asked for. ['decompose.mjs', ['--dry-run', '--max-scrapes'], /--max-scrapes needs a value/], ['decompose.mjs', ['--dry-run', '--limit'], /--limit needs a value/], - ['decompose.mjs', ['--dry-run', '--limit', 'abc'], /--limit must be a whole number of at least 0, not "abc"/], + ['decompose.mjs', ['--dry-run', '--limit', 'abc'], /--limit must be a whole number of at least 1, not "abc"/], + ['decompose.mjs', ['--dry-run', '--limit', '0'], /--limit must be a whole number of at least 1, not "0"/], + ['decompose.mjs', ['--dry-run', '--limit', '-1'], /--limit must be a whole number of at least 1, not "-1"/], ['preflight.mjs', ['--check'], /--check needs a value/], ['gate.mjs', ['--gate', 'push'], /--gate must be one of/], ['prior.mjs', ['--file'], /--file needs a value/], @@ -1047,6 +1067,13 @@ test('a flag given a bad value, or none, is refused before anything runs', () => assert.equal(only.status, 2, `preflight --only is accepted and never read:\n${only.all.slice(0, 300)}`); }); +test('decompose accepts a search limit of one while max-scrapes still accepts zero', () => { + const r = run('decompose.mjs', ['--dry-run', '--limit', '1', '--max-scrapes', '0', '--transport', 'http-keyless'], { + root: planned('budget-floor'), + }); + assert.equal(r.status, 0, r.all); +}); + // Found 2026-09-27: new-project over an existing project with a different --topic reported // "wrote 0, kept 13" and printed a fresh project's next steps. The topic asked for was not // applied, and nothing said so. diff --git a/research-kit/test/corpus.test.mjs b/research-kit/test/corpus.test.mjs index 4c15921f..a3bd45f8 100644 --- a/research-kit/test/corpus.test.mjs +++ b/research-kit/test/corpus.test.mjs @@ -4,7 +4,7 @@ import { spawnSync } from 'node:child_process'; import { test, describe, assert, makePassingProject, makeProject, corrupt, fs, requireSymlink } from './harness.mjs'; import { PATHS, HEADERS, resolve, writeText, readText, sha256File, isRegularFile } from '../lib/core.mjs'; import { - readCorpus, readCaptures, parseTable, parseCapture, splitRow, escapeCell, tableRow, + readCorpus, readCaptures, readLedger, parseTable, parseCapture, splitRow, escapeCell, tableRow, repairRowArity, captureEntry, rememberCapture, cacheDecision, traceOf, captureOf, claimOf, sectionOf, appendRow, upsertRow, nextId, citedIds, stripReviewNotes, requestedUrl, } from '../lib/corpus.mjs'; @@ -12,6 +12,18 @@ import { runPreflight } from '../lib/preflight.mjs'; describe('corpus'); +test('a leading ledger byte-order mark preserves entries without rewriting the file', () => { + const dir = makePassingProject(); + const file = resolve(dir, PATHS.ledger); + const before = readLedger(dir); + const saved = `\uFEFF${fs.readFileSync(file, 'utf8')}`; + fs.writeFileSync(file, saved); + const after = readLedger(dir); + assert.deepEqual(after.entries, before.entries); + assert.deepEqual(after.problems, []); + assert.equal(fs.readFileSync(file, 'utf8'), saved, 'reading must not edit the ledger'); +}); + test('splitRow honours escaped pipes; escapeCell round-trips through them', () => { assert.deepEqual(splitRow('| a | b\\|c | d |'), ['a', 'b|c', 'd']); assert.equal(escapeCell('b|c'), 'b\\|c'); diff --git a/research-kit/test/decompose.test.mjs b/research-kit/test/decompose.test.mjs index 3295ab35..38735b8e 100644 --- a/research-kit/test/decompose.test.mjs +++ b/research-kit/test/decompose.test.mjs @@ -1,7 +1,8 @@ // Phase 0 gathers material and seeds the checklist. It contains NO judgment: it hands // over statuses BLANK, and `subtopic-coverage` is what judges them afterwards. -import { test, describe, assert, makeProject, makePassingProject, fs } from './harness.mjs'; +import { spawnSync } from 'node:child_process'; +import { test, describe, assert, makeProject, makePassingProject, fs, path, KIT_ROOT } from './harness.mjs'; import { PATHS, resolve, readText, listFiles } from '../lib/core.mjs'; import { decompose, parseRecipe, loadRecipe, docsHosts, scrapeOrder, ownerOf, topicQueries, RECIPE_DIR, searchSummary, outlineOf, resolveTopic } from '../lib/decompose.mjs'; import { UNIVERSAL_DIMENSIONS, seedRows, coverageOfUniversals } from '../lib/dimensions.mjs'; @@ -161,6 +162,47 @@ test('an unknown recipe is an error naming where recipes live', () => { assert.throws(() => loadRecipe('no-such-recipe'), /no recipe/); }); +const invalidRecipes = [ + ['plain.md', '# Just a document\n\nNo dimensions.\n', /holds no dimensions/], + ['empty.md', '---\nname: empty\ndimensions:\n---\n', /holds no dimensions/], + ['blank.md', '---\r\nname: blank\r\ndescription: test\r\ndimensions:\r\n - | a reason only\r\n---\r\n', /line 5.*blank/], +]; + +test('an existing recipe with no dimensions or a blank label is refused by file and line', () => { + const dir = makeProject(); + for (const [name, text, reason] of invalidRecipes) { + const file = resolve(dir, name); + fs.writeFileSync(file, text); + assert.throws(() => loadRecipe(file), (err) => { + assert.equal(err.code, 'INVALID_RECIPE'); + assert.ok(err.message.includes(file), err.message); + assert.match(err.message, reason); + return true; + }); + } +}); + +test('invalid recipe content is CLI usage failure and leaves the map and ledger untouched', () => { + for (const [name, text, reason] of invalidRecipes) { + const dir = makePassingProject(); + // Exercise recipe validation rather than the earlier refusal to replace judged rows. + fs.writeFileSync(resolve(dir, PATHS.map), '# An unclassified draft\n'); + const file = resolve(dir, name); + fs.writeFileSync(file, text); + const beforeMap = readText(resolve(dir, PATHS.map)); + const beforeLedger = readText(resolve(dir, PATHS.ledger)); + const result = spawnSync(process.execPath, [path.join(KIT_ROOT, 'bin/decompose.mjs'), + '--recipe', file, '--dry-run', '--transport', 'http-keyless', '--search-transport', 'http-keyless'], + { cwd: dir, encoding: 'utf8', timeout: 20000, windowsHide: true }); + assert.equal(result.status, 2, result.stdout + result.stderr); + assert.match(result.stderr, reason); + assert.ok(result.stderr.includes(file), result.stderr); + assert.doesNotMatch(result.stderr, /\n\s+at /, 'usage failures must not print a stack'); + assert.equal(readText(resolve(dir, PATHS.map)), beforeMap); + assert.equal(readText(resolve(dir, PATHS.ledger)), beforeLedger); + } +}); + test('parseRecipe reads the machine-readable frontmatter', () => { const { name, dimensions } = parseRecipe('---\nname: sample\ndimensions:\n - First thing | why it matters\n - Second thing | another reason\n---\n\nprose\n'); assert.equal(name, 'sample'); diff --git a/research-kit/test/finding.test.mjs b/research-kit/test/finding.test.mjs index f5e068ca..4a441009 100644 --- a/research-kit/test/finding.test.mjs +++ b/research-kit/test/finding.test.mjs @@ -183,6 +183,39 @@ test('the excerpt carries the surrounding paragraph, bounded, so a reviewer can assert.ok(result.excerpt.length <= EXCERPT_CHARS + 1, `excerpt is ${result.excerpt.length} chars, which is not bounded`); }); +test('long unbroken paragraphs retain the same bounded excerpt without quadratic recovery', () => { + const claim = 'The free plan allows 1,000 requests per minute and retains logs for 30 days.'; + const line = '- item is listed here\n'; + const count = Math.ceil(8 * 1024 * 1024 / line.length); + const page = line.repeat(count) + claim; + const started = Date.now(); + const result = findingWithContext(page, 'fallback'); + const elapsed = Date.now() - started; + assert.equal(result.finding, claim); + const prefix = 'item is listed here '.repeat(count).trim(); + assert.equal(result.excerpt, `${prefix.slice(0, EXCERPT_CHARS).trimEnd()}…`); + // The reported recovery takes tens of seconds at the accepted capture size. Leave a + // generous bound for loaded machines while keeping that failure clearly outside it. + assert.ok(elapsed < 8000, `an 8 MB unbroken paragraph took ${elapsed}ms`); +}); + +test('paragraph recovery preserves empty Markdown lines, heading boundaries and the exact cap', () => { + const claim = 'The free plan allows 1,000 requests per minute and retains logs for 30 days.'; + const cases = [ + [`# Old heading\nIgnored before the boundary.\n## Limits\n**\n${claim}\nRequests beyond that return HTTP 429.\n\nNot this paragraph.`, `${claim} Requests beyond that return HTTP 429.`], + [` \n${claim}\n## Other\nNot this heading.`, claim], + [`${'x'.repeat(EXCERPT_CHARS - 1)}\n${claim}`, `${'x'.repeat(EXCERPT_CHARS - 1)}…`], + [`${'x'.repeat(EXCERPT_CHARS)}\n**\n${claim}`, `${'x'.repeat(EXCERPT_CHARS)}…`], + [`${claim}\n${'x'.repeat(EXCERPT_CHARS)}`, `${(`${claim} ${'x'.repeat(EXCERPT_CHARS)}`).slice(0, EXCERPT_CHARS).trimEnd()}…`], + [`${claim}\n\`\`\`\nunchosen code\n\`\`\`\n `, `${claim} unchosen code`], + ]; + for (const [page, excerpt] of cases) { + const result = findingWithContext(page, 'fallback'); + assert.equal(result.finding, claim); + assert.equal(result.excerpt, excerpt, JSON.stringify(page)); + } +}); + test('firstFinding is exactly findingWithContext.finding, so the two cannot drift', () => { // firstFinding is what the collector writes into the evidence table; findingWithContext // is what a reviewer reads. If they ever selected differently, the excerpt would be diff --git a/research-kit/test/handoff.test.mjs b/research-kit/test/handoff.test.mjs index dfd15d50..05ce41eb 100644 --- a/research-kit/test/handoff.test.mjs +++ b/research-kit/test/handoff.test.mjs @@ -8,6 +8,8 @@ import { PATHS, resolve, writeText } from '../lib/core.mjs'; import { verifyHandoff, handoffRemedy, HANDOFF_REMEDY, lineEndingRemedy, alteredRemedy, safePathspec, insideRepository, repositoryState, NAMES_FILE, PIN_LINES } from '../lib/handoff.mjs'; import { readCorpus } from '../lib/corpus.mjs'; import { runDoctor } from '../lib/doctor.mjs'; +import { appendFetch } from '../lib/provenance.mjs'; +import { runPreflight } from '../lib/preflight.mjs'; describe('handoff'); @@ -32,6 +34,57 @@ test('a corpus that arrived whole passes', () => { assert.equal(report.remedy, ''); }); +test('a leading ledger byte-order mark leaves the chain and handoff intact', () => { + const dir = makePassingProject(); + corrupt(dir, PATHS.ledger, (text) => `\uFEFF${text}`); + const report = verifyHandoff(dir); + assert.equal(report.ok, true, JSON.stringify(report.findings)); + assert.equal(report.didNotTravel, false); + assert.equal(report.ledgerLost, false); + assert.equal(runPreflight(dir).pass, true, 'the provenance gate must accept the same intact chain'); +}); + +test('unparsed ledger lines get the local repair remedy instead of lost-entry or travel advice', () => { + const tail = makePassingProject(); + fs.appendFileSync(resolve(tail, PATHS.ledger), '\n\n{"seq":2,"url":"unfinished'); + let report = verifyHandoff(tail); + assert.equal(report.ok, false); + assert.equal(report.didNotTravel, false); + assert.equal(report.ledgerLost, false); + assert.match(report.remedy, /doctor\.mjs"? --fix-arity/); + assert.doesNotMatch(report.remedy, /Something did not travel|ledger lost entries/); + + const middle = makePassingProject(); + corrupt(middle, PATHS.ledger, (text) => `not JSON\n${text}`); + report = verifyHandoff(middle); + assert.equal(report.ok, false); + assert.equal(report.didNotTravel, false); + assert.equal(report.ledgerLost, false); + assert.match(report.remedy, /broken line inside the chain.*restore research\/raw\/\.fetches\.jsonl from git/); + assert.doesNotMatch(report.remedy, /Something did not travel|ledger lost entries|--fix-arity/); + + const brokenEntry = makePassingProject(); + appendFetch(brokenEntry, { op: 'fail', url: 'https://example.invalid/second', transport: 'http-keyless' }); + corrupt(brokenEntry, PATHS.ledger, (text) => text.replace(/^[^\n]+/, 'unreadable first entry')); + report = verifyHandoff(brokenEntry); + assert.equal(report.ok, false); + assert.equal(report.didNotTravel, false, 'unparsed entries cause secondary seq/prev failures'); + assert.equal(report.ledgerLost, false, 'a capture may be recorded in the unreadable entry'); + assert.match(report.remedy, /restore research\/raw\/\.fetches\.jsonl from git/); + assert.doesNotMatch(report.remedy, /Something did not travel|ledger lost entries/); +}); + +test('an unreadable ledger does not hide a capture that really did not travel', () => { + const dir = makePassingProject(); + const capture = readCorpus(dir).captures.entries[0]; + fs.rmSync(resolve(dir, capture.file)); + fs.appendFileSync(resolve(dir, PATHS.ledger), '{"seq":2'); + const report = verifyHandoff(dir); + assert.equal(report.didNotTravel, true); + assert.match(report.remedy, /Something did not travel/); + assert.match(report.remedy, /--fix-arity/); +}); + test('a missing ledger is named handoff-ledger-missing', () => { const dir = makePassingProject(); fs.rmSync(resolve(dir, PATHS.ledger)); diff --git a/research-kit/test/panel.test.mjs b/research-kit/test/panel.test.mjs index b2fa0439..17a308be 100644 --- a/research-kit/test/panel.test.mjs +++ b/research-kit/test/panel.test.mjs @@ -6,7 +6,7 @@ import http from 'node:http'; import { spawnSync } from 'node:child_process'; import { test, describe, assert, assertEqual, tempDir, fs, path, makeProject, requireGit, requireSymlink, fixtureInitArgs, KIT_ROOT } from './harness.mjs'; import { writeText, readText } from '../lib/core.mjs'; -import { createPanel, PANEL_COMMANDS, PANEL_HOST, keyProblem, projectTopic, builderInstructions } from '../lib/panel.mjs'; +import { createPanel, PANEL_COMMANDS, PANEL_HOST, MAX_BODY_BYTES, keyProblem, projectTopic, builderInstructions } from '../lib/panel.mjs'; import { writeResult, recordTrustedResult, setPaused } from '../lib/requests.mjs'; describe('panel'); @@ -66,6 +66,31 @@ const api = (ctx, route, body) => request(ctx.port, { body, }); +test('an oversized POST receives the 413 JSON refusal and the panel keeps answering', async () => { + const ctx = await started({ project: makeProject(undefined, { content: true }) }); + try { + for (const size of [MAX_BODY_BYTES + 1, 3 * 1024 * 1024]) { + const body = `"${'x'.repeat(size - 2)}"`; + const res = await api(ctx, '/api/run', body); + assertEqual(res.status, 413); + assertEqual(res.json?.error, 'the request is too large'); + assertEqual(res.headers.connection, 'close'); + assertEqual((await api(ctx, '/api/state')).status, 200); + } + } finally { await ctx.panel.close(); } +}); + +test('a complete JSON body at the panel limit reaches route handling', async () => { + const ctx = await started({ project: makeProject(undefined, { content: true }) }); + try { + const body = `"${'x'.repeat(MAX_BODY_BYTES - 2)}"`; + assertEqual(Buffer.byteLength(body), MAX_BODY_BYTES); + const res = await api(ctx, '/api/not-a-route', body); + assertEqual(res.status, 404); + assertEqual(res.json?.error, 'not a panel route'); + } finally { await ctx.panel.close(); } +}); + test('the panel listens on 127.0.0.1 only and puts its token in the fragment, never the path', async () => { const ctx = await started({ project: makeProject(undefined, { topic: 'Panel topic', content: true }) }); try { diff --git a/research-kit/test/provenance.test.mjs b/research-kit/test/provenance.test.mjs index 7fad45c2..286a3183 100644 --- a/research-kit/test/provenance.test.mjs +++ b/research-kit/test/provenance.test.mjs @@ -18,6 +18,20 @@ test('happy path: the chain verifies and preflight passes', () => { assert.equal(runPreflight(dir).pass, true); }); +test('tail repair leaves an intact BOM ledger alone and preserves its prefix when dropping a torn append', () => { + const dir = makePassingProject(); + const file = resolve(dir, PATHS.ledger); + const saved = `\uFEFF${fs.readFileSync(file, 'utf8')}`; + fs.writeFileSync(file, saved); + assert.deepEqual(repairLedgerTail(dir), { repaired: false, reason: 'tail-parses' }); + assert.equal(fs.readFileSync(file, 'utf8'), saved); + assert.equal(verifyLedger(dir).ok, true); + fs.appendFileSync(file, '{"seq":2,"url":"unfinished'); + assert.equal(repairLedgerTail(dir).repaired, true); + assert.equal(fs.readFileSync(file, 'utf8'), saved, 'repair must remove only the torn append'); + assert.equal(verifyLedger(dir).ok, true); +}); + test('a tampered capture fails body-unmodified', () => { const dir = makePassingProject(); const capture = readCorpus(dir).captures.entries[0]; diff --git a/research-kit/test/release-validator.test.mjs b/research-kit/test/release-validator.test.mjs index fc01a5c8..4c34a2b0 100644 --- a/research-kit/test/release-validator.test.mjs +++ b/research-kit/test/release-validator.test.mjs @@ -205,6 +205,72 @@ test('researcher-release names a missing value before resolving paths', () => { } }); +test('researcher-release refuses invalid snapshot ages as usage before judging a valid package', async () => { + const root = tempRoot(); + try { + const registryPath = path.join(root, 'registry.json'); + const rolesPath = path.join(root, 'roles.json'); + const pointers = path.join(root, 'pointers'); + const record = envelope(); + record.payloadSha256 = await sha256(canonical(record.payload)); + writeJson(path.join(root, 'records', 'R28-01.json'), record); + writeJson(registryPath, registryFor()); + writeJson(rolesPath, roster()); + fs.mkdirSync(pointers); + const args = ['research-kit/bin/researcher-release.mjs', 'validate', '--root', root, + '--package', 'R28', '--registry', registryPath, '--roles', rolesPath, '--pointers', pointers, '--json']; + for (const value of ['abc', '', ' ', 'NaN', 'Infinity', '1e309', '-1']) { + const run = spawnSync(process.execPath, [...args, '--snapshot-max-age-ms', value], { encoding: 'utf8' }); + assertEqual(run.status, 2, run.stdout || run.stderr); + assert(run.stderr.includes('--snapshot-max-age-ms'), run.stderr); + assert(run.stderr.includes('Usage:'), run.stderr); + assertEqual(run.stdout, '', 'usage must not produce an evidence verdict'); + assert(!/SNAPSHOT-STALE|allowed .*ms/.test(run.stderr), run.stderr); + } + for (const value of ['0', '1', '1.5', '1e3']) { + const run = spawnSync(process.execPath, [...args, '--snapshot-max-age-ms', value], { encoding: 'utf8' }); + assertEqual(run.status, 0, run.stdout || run.stderr); + assertEqual(JSON.parse(run.stdout).status, 'PASS'); + } + } finally { cleanup(root); } +}); + +test('release and snapshot facades distinguish an invalid age allowance from stale evidence', async () => { + const { validateRelease, validateSnapshotEvidence } = await import('../lib/release-validator.mjs'); + const root = tempRoot(); + try { + const registryPath = path.join(root, 'registry.json'); + const rolesPath = path.join(root, 'roles.json'); + writeJson(registryPath, registryFor()); + writeJson(rolesPath, roster()); + const now = Date.parse('2026-01-01T00:00:00Z'); + const baseRecord = envelope(); + baseRecord.payloadSha256 = await sha256(canonical(baseRecord.payload)); + writeJson(path.join(root, 'records', 'R28-01.json'), baseRecord); + const baseline = validateRelease({ root, package: 'R28', registryPath, rolesPath }); + assertEqual(baseline.status, 'PASS', JSON.stringify(baseline)); + const refs = Object.fromEntries(['before', 'after', 'recovery'].map((phase) => [phase, { + snapshotId: phase, capturedAt: '2026-01-01T00:00:00Z', status: 'PASS', envelopeCanonicalSha256: 'c'.repeat(64), + }])); + const record = envelope({ payload: { pathAuthoritySnapshots: refs } }); + for (const value of [NaN, Infinity, -Infinity, -1, null, '0']) { + const results = [ + validateRelease({ root, package: 'R28', registryPath, rolesPath, snapshotMaxAgeMs: value }), + validateSnapshotEvidence(record, { root, now, maxAgeMs: value }), + ]; + for (const result of results) { + assertEqual(result.status, 'INCOMPLETE', JSON.stringify(result)); + assert(result.errors.some((error) => error.code === 'INPUT-SNAPSHOT-MAX-AGE'), JSON.stringify(result)); + assert(!result.errors.some((error) => error.code === 'SNAPSHOT-STALE'), JSON.stringify(result)); + } + } + const fresh = validateSnapshotEvidence(record, { root, now, maxAgeMs: 0 }); + assert(!fresh.errors.some((error) => error.code === 'INPUT-SNAPSHOT-MAX-AGE' || error.code === 'SNAPSHOT-STALE'), JSON.stringify(fresh)); + const stale = validateSnapshotEvidence(record, { root, now: now + 1, maxAgeMs: 0 }); + assertEqual(stale.errors.filter((error) => error.code === 'SNAPSHOT-STALE').length, 3, JSON.stringify(stale)); + } finally { cleanup(root); } +}); + test('researcher-release conform validates explicit files and maps schema failure to exit 1', async () => { const root = tempRoot(); try {