feat(evidence): Phase 2b readiness gates + rebuild auto-regen flag - #54
Conversation
Operationalise "once the would-reject ratio settles" from docs/evidence-envelope.md: instead of a vibe call, the evidence report now publishes a Phase 2b readiness verdict against three numeric gates (4-week total writes >= 50, every weekly wouldRejectRatio < 5%, search abstain ratio <= 30% when >= 10 searches). Operators read the gates in wiki/evidence-report.md and flip AGENT_WIKI_EVIDENCE_REJECT_UNSUPPORTED when the report shows "READY". Connects existing pieces - no new MCP tool, no new agent-visible field. wiki_admin rebuild gains an opt-in `evidence_report: true` flag so the dashboard can stay fresh without a separate evidence-report call. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00c341cea0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| lines.push( | ||
| `> All gates pass. To flip Phase 2b on, set ` + | ||
| `\`AGENT_WIKI_EVIDENCE_REJECT_UNSUPPORTED=true\` (env var) or ` + | ||
| `\`evidence.rejectUnsupportedWrites: true\` in \`.agent-wiki.yaml\`. ` + |
There was a problem hiding this comment.
When an operator follows this ready-state tip and adds evidence.rejectUnsupportedWrites: true to .agent-wiki.yaml, the config loader will ignore it: the YAML parser reads evidenceData.reject_unsupported_writes into wiki.config.evidence.rejectUnsupportedWrites (checked in src/wiki.ts), so only evidence.reject_unsupported_writes: true enables Phase 2b via YAML. This makes the report's primary non-env flip instruction ineffective exactly in the scenario where the gates say it is safe to enable.
Useful? React with 👍 / 👎.
High: - Render tip and docs/evidence-envelope.md now name the correct YAML key evidence.reject_unsupported_writes (snake_case). The loader at wiki.ts:581 only reads the snake form; the camel form silently no-ops. - Batch dedup now carries `evidence_report:true` through to the end-of-batch rebuild. Previously the flag was dropped along with the deduped wiki_admin op, so batch callers got no report despite ok:true. Medium: - Inline rebuild path surfaces evidence-report regen failures in the ok-message instead of swallowing them. The operator explicitly asked for the report; silent failure leaves them no signal. Low / polish: - "Currently enabled" line drops the env-var attribution (state can come from YAML too). - Drop dead `applicable: true` literal from the two always-applicable gate shapes; keep the discriminant on searchAbstain. - Un-export the four PHASE2B_* threshold constants; no external consumer. - Rebuild handler's evidence_report flag uses `?? false` to match the surrounding boolean-coercion idiom. - Anchor the per-week table regex to full-line shape so a regression on indent or trailing pipe would fail rather than silently match. Tests: - Regression test for the snake_case key in the rendered tip. - Two batch tests covering evidence_report:true on/off through dedup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Medium: - Extract `regenerateEvidenceReport` helper used by both the inline wiki_admin rebuild handler and the end-of-batch path. Batch now surfaces failure as a follow-up `wiki_admin warning` entry in `results` instead of swallowing it, matching the inline path's operator-requested contract. The misleading "mirrors the inline rebuild path's tolerance" comment is gone. - Switch the Phase 2b flip-criteria link from `../docs/...` to an absolute GitHub URL. The relative path dead-links when wiki/evidence-report.md is written under a user-supplied wiki/ dir (docs/ does not ship in the npm package). Low / polish: - Align boolean coercion: inline rebuild path and wiki_evidence_report both now use `args.foo === true`, matching the batch sniff. Same flag, same semantics on every entry point. - Decouple the Phase 2b gate window from `WEEKS_OF_TREND`. Introduce `PHASE2B_GATE_WEEKS = 4` and slice `trend.slice(-PHASE2B_GATE_WEEKS)` in `assessPhase2bReadiness` so widening the display sparkline later doesn't silently halve the gate threshold. - Narrow failingWeeks via a typed predicate; drop the unreachable `(w.ratio ?? 0) * 100` defensiveness. Tests: - Render assertion that the flip tip uses the absolute GitHub URL and does not contain a relative `../docs/` path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Medium:
- Batch end-of-batch rebuild now runs `migrateExistingPagesForEvidence`
alongside `rebuildIndex` + `rebuildTimeline`, mirroring the inline
wiki_admin rebuild handler. Without it, a fresh batch-driven rebuild
on a never-migrated wiki produced an evidence report that bucketed
legacy pages as `other` instead of `legacyUnsupported`. Migration is
marker-file-gated and one-shot, so the call is a no-op after the
first run on any given workspace.
- Batch evidence-report failure now mutates the deferred result entry
in place rather than pushing a sibling `{ tool, warning }` entry.
The old shape (a) inflated `count: results.length` so callers iterating
ops paired with results broke on failure, and (b) introduced a
`warning` key absent elsewhere in `src/`. Operators get the regen
status on the same op slot as the rebuild itself, in a uniform shape.
Low / polish:
- User-visible "4 weeks" strings (the reason text and the per-window
gate label) interpolate `PHASE2B_GATE_WEEKS` instead of a hardcoded
literal — matches the precedent set by `Trend (last ${WEEKS_OF_TREND}
weeks)` and survives any future re-calibration.
- Move the orphan JSDoc block from above `PHASE2B_GATE_WEEKS` to attach
to `export interface Phase2bReadiness`, where it documents the actual
public surface. The constant keeps its own focused docblock.
Tests:
- Batch test asserts the 1-op→1-result invariant explicitly (count, length,
and the deferred entry shape).
- New batch test asserts `.agent-wiki/evidence-migrated` marker exists
after batch rebuild, pinning the migration step against future drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b2fc63b085
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| }; | ||
|
|
||
| const perWeek = gateBuckets.map((b) => { | ||
| const ratio = b.totalWrites === 0 ? null : b.unsupportedOrRejected / b.totalWrites; |
There was a problem hiding this comment.
When a corpus still has migrated legacyUnsupported pages being edited, warn-mode writes are logged as kind: "legacy" (src/wiki.ts:1554), and the Phase 2b path rejects the same updates (src/wiki.ts:1514). This readiness gate divides only unsupportedOrRejected by total writes, so 50+ legacy edits with no fresh unsupported writes can report READY even though enabling Phase 2b would block those edits. Please include legacy writes in the numerator or add a separate gate for unresolved legacy edit volume.
Useful? React with 👍 / 👎.
Medium:
- Extract `runPostRebuildEvidence(wiki, { regenerateReport })` as the
single source of truth for migration + weekly log digest + optional
evidence-report regen. Both `wiki_admin action:rebuild` (inline) and
the end-of-batch rebuild now call it, eliminating the copy-paste
pattern that produced previous parity bugs (missing weekly log
digest in batch). Adding a new step now lands in one place.
- Inline rebuild emits a structured `evidenceReport: { ok: false,
error }` field alongside `ok: true` when regen fails — programmatic
callers can detect partial-success without sniffing `message`.
Matches the batch path's deferred-entry mutation shape.
Low / polish:
- Revert booleans in `wiki_evidence_report.write`, inline rebuild's
`evidence_report`, and the batch sniff to the `(args.X as boolean)
?? false` convention used by sibling booleans (`apply_fixes`, etc).
The strict `=== true` check was an over-correction that introduced
an undocumented behavior change on `wiki_evidence_report.write`.
- `runEvidenceReport` gains overloads so `write: true` narrows
`writtenTo` to `string`. `regenerateEvidenceReport` drops the
unreachable `?? "wiki/evidence-report.md"` fallback.
- Add `pickGateBuckets` helper that throws when `PHASE2B_GATE_WEEKS
> trend.length` — surfaces a calibration mismatch (e.g. someone
raising the gate window without widening `WEEKS_OF_TREND`) instead
of silently making the min-writes gate easier to pass.
- Replace the bespoke `FailingWeek` type guard with a plain filter +
`w.ratio!` at the use site. The invariant (`!passing → ratio !==
null`) is documented in a comment; a discriminated-union refactor
isn't worth it for two lines.
- Lift the flip-criteria URL to a module-level `FLIP_CRITERIA_URL`
constant so a repo rename or fork updates one literal. The previous
comment claimed `docs/` doesn't ship in the npm package, which is
factually wrong (`package.json#files` lists it) — replaced with the
real reason (wiki dir's position relative to docs/ is not stable).
Tests:
- Pin: inline rebuild with evidence_report:true emits no
`evidenceReport` field on success — the field appears only on
failure (parity with the batch path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6204aa98ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const bookkeeping = runPostRebuildEvidence(wiki, { | ||
| regenerateReport: needsEvidenceReport, | ||
| }); |
There was a problem hiding this comment.
When a workspace has no .agent-wiki/evidence-migrated marker, any batch containing wiki_write now reaches this helper because wiki_write sets needsRebuild; the helper runs migrateExistingPagesForEvidence, whose rule 4 clears unsupported: true and replaces it with legacyUnsupported: true for pages without sources. That means the first unsupported page created via a batch is immediately grandfathered as legacy instead of remaining a fresh unsupported write, so source coverage/read downgrades no longer reflect what the user just wrote. Limit the migration helper to explicit rebuild ops or run migration before processing batch writes.
Useful? React with 👍 / 👎.
Low:
- Inline `trend.slice(-PHASE2B_GATE_WEEKS)` at its single call site
and drop the `pickGateBuckets` helper. Both constants live in the
same file, both equal 4, and `aggregateTrend` deterministically
returns exactly `WEEKS_OF_TREND` buckets — the throw was unreachable.
Keep a 1-line guard adjacent to the slice so a future calibration
bump still fails loud instead of silently making the gate easier.
- Drop the `runEvidenceReport` overload pair. Only beneficiary was
`regenerateEvidenceReport`; the cost was `wiki_evidence_report`
paying a runtime-meaningless ternary purely to satisfy overload
resolution, and the narrowed `writtenTo` was JSON-stringified
either way. `regenerateEvidenceReport` now null-checks at the use
site; the wiki_evidence_report call collapses to a single
`runEvidenceReport(wiki, { write })`.
- Add `await new Promise((r) => setImmediate(r))` between
`rebuildIndex` and `rebuildTimeline` in the batch end-of-batch
rebuild. Mirrors the inline path (commit 764c2fe rationale: slow
filesystems like OneDrive-synced workspaces) so the MCP transport
can answer client pings during long batched rebuilds — direct
parity that the round-3 commit message claimed but missed.
- `evidence-report.test.ts` saves/clears `AGENT_WIKI_EVIDENCE_REJECT
_UNSUPPORTED` in `beforeEach`/`afterEach` so a developer running
tests with the env var set (a scenario this PR explicitly
documents) doesn't see the `currentlyEnabled: false` assertion
fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 063d1867c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const totalWritesValue = gateBuckets.reduce((acc, b) => acc + b.totalWrites, 0); | ||
| const totalWritesGate = { | ||
| value: totalWritesValue, | ||
| threshold: PHASE2B_MIN_TOTAL_WRITES, | ||
| passing: totalWritesValue >= PHASE2B_MIN_TOTAL_WRITES, |
There was a problem hiding this comment.
When code-generated pages are produced by code_parse or wiki_admin rebuild, they are written through wiki.write as well (for example src/server.ts:2462, src/server.ts:2473, and src/server.ts:2581), and the counter has no source filter, so these generated grounded/synthesis writes are included in gateBuckets. In COBOL-heavy workspaces, repeated parses/rebuilds can therefore satisfy the ≥50-write gate and keep the ratio low with little or no actual wiki_write activity, causing the Phase 2b report to show READY even though the sample did not measure user/agent unsupported-write behavior.
Useful? React with 👍 / 👎.
Round-6 review caught a JSDoc on PHASE2B_GATE_WEEKS still pointing at the pickGateBuckets helper that was inlined in round 5. Rewrite to reference the actual enforcement site (the inline trend.length guard in assessPhase2bReadiness). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 participant
Summary
phase2bReadinessverdict (ready/not-ready/insufficient-data) against three concrete numeric gates instead of leaving "once the would-reject ratio settles" as a vibe call.wiki_admin evidence-reportoutput (operator-facing), andwiki_admin rebuildgains an opt-inevidence_report: trueflag so the dashboard can stay fresh as part of a regular rebuild.docs/evidence-envelope.mdso the numbers are explicit rather than implicit.Gates
wouldRejectRatioBelow-50-total flags as
insufficient-data(statistically thin); a single bad week or a high search abstain rate flags asnot-ready. Conservative on purpose — a false-positivereadywould block legitimate writes.When
status: readyandcurrentlyEnabled: no, the rendered report includes the exactAGENT_WIKI_EVIDENCE_REJECT_UNSUPPORTED=trueknob to flip.Test plan
assessPhase2bReadinesscover: insufficient-data, ready, not-ready (weekly ratio), zero-write weeks passing by default, search-gate skipping under threshold, search-gate triggering above threshold,currentlyEnabledreflecting wiki configwiki_admin --action rebuild evidence_report:truewriteswiki/evidence-report.mdand includes the Phase 2b section; rebuild without the flag leaves the file untouched🤖 Generated with Claude Code