Skip to content

docs(guide/11): correct the release rule, and stop understating the licence risk - #33

Merged
ancsingabor merged 2 commits into
mainfrom
docs/guide-11-review
Sep 30, 2026
Merged

ancsingabor merged 2 commits into
mainfrom
docs/guide-11-review

Conversation

@ancsingabor

Copy link
Copy Markdown
Owner

The questions themselves are good — I'd keep every one. Two of the answers were wrong, and this is the one page where a wrong answer costs something, because it is the page a reviewer reads to decide whether to trust the rest.

The release rule (Q3) was wrong in both directions

It said: "a retrieval or prompt change must attach an eval diff, and a regression blocks it."

the page said the rule actually says
a recall@10 regression blocks the merge named and justified, not vetoed — a pp threshold at n=40–60 is false precision, which is why the rule stopped being ">2pp recall@10" in PR #21
what blocks (not stated) any drop in citation validity or quote fidelity
who enforces implied CI review; CI does not run the eval
what triggers it retrieval, prompt retrieval, chunking, embedding, prompt

The missing qualifier matters most: telling a reviewer "a regression blocks it" invites them to assume a CI gate, and there isn't one. And omitting chunking from the trigger list is exactly how someone re-tunes the chunker without a diff — the thing .claude/project.md rule 6 and chunk.ts both forbid.

The licence weakness still had its pre-#24 framing

The row read: "May corpus text be sent to a hosted embedding API? … The bake-off starts with local models, which don't raise the question."

ADR-003 § Extended widened that to generation. Composing a prompt puts the retrieved units themselves into a request to Anthropic — in production, per question, for as long as the system runs. Local candidates mitigate the bake-off half; nothing mitigates the other one, and there is no local fallback anywhere in the design. So the mitigation as written was false for the half that matters.

It now gets a short row plus a paragraph naming it as the only open item that can invalidate a design rather than narrow a choice. Understating it on the page whose own TL;DR says "a design explained without its costs is not credible" was the worst possible place for it to sit.

A question a reviewer certainly asks, that wasn't here

"Why Claude?" Four files assume Anthropic, nothing ever compared it, no ADR argues it — and the harness cannot settle it, because it scores no generation model at all. Added as question 6 with the honest answer, plus a weakness row.

That made "The first five" into "the six that come up first" — and that counted heading is itself the tripwire named in the new § What I would do differently bullet:

Treat a counted claim as a tripwire. "Seven silent greens", "five war stories", "three pytest modules" — each drifted while the prose around it stayed true.

The summa:I.q2.a3 bullet also now carries why the fix took six passes: the last two copies were prose, so no grep for the locator form could find them.

1,415 words, in band. docs:lint green.

🤖 Generated with Claude Code

ancsingabor and others added 2 commits September 30, 2026 14:22
…icence risk

Two answers on this page were wrong in ways a reviewer would catch, which is
the one page where that costs something.

**The release rule (Q3) was wrong in both directions.** It said "a retrieval or
prompt change must attach an eval diff, and a regression blocks it". The actual
rule (docs/evaluation.md, moved there in PR #21):

- a `recall@10` regression is **named and justified, not blocked** — a pp
  threshold at n=40–60 is false precision, which is why the rule stopped being
  ">2pp recall@10" in the first place;
- what *does* block is a drop in citation validity or quote fidelity;
- it is **enforced in review; CI does not run the eval** — the qualifier that
  matters most here, since "blocks it" invites a reviewer to assume a CI gate;
- it covers retrieval, chunking, embedding **or prompt**. Chunking was missing,
  and omitting it is exactly how someone re-tunes the chunker without a diff —
  the thing `.claude/project.md` rule 6 and `chunk.ts` both forbid.

**The licence weakness still had its pre-#24 framing.** The row read "May
corpus text be sent to a hosted *embedding* API? … the bake-off starts with
local models, which don't raise the question". ADR-003 § Extended widened that
to generation: composing a prompt puts the retrieved units into a request to
Anthropic, in production, per question. Local candidates mitigate the bake-off
half and nothing mitigates the other one, so the mitigation as written was
false for the half that matters. Now a short row plus a paragraph saying it is
the only open item that invalidates a design rather than narrowing a choice.
Understating this on the page whose stated purpose is that "a design explained
without its costs is not credible" was the worst place for it to sit.

Also:

- **A missing question a reviewer certainly asks: why Claude?** Four files
  assume Anthropic, nothing compared it, no ADR argues it — and the harness
  cannot settle it, since it scores no generation model. Added as question 6
  with the honest answer, and as a weakness row. "The first five" became "the
  six that come up first"; that counted heading is itself the tripwire the new
  § What I would do differently bullet names.
- The `summa:I.q2.a3` bullet now carries why the fix took six passes: the last
  two copies were prose, so no grep for the locator form could find them.

1,415 words.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Branch protection requires an up-to-date head branch; #31 and #32 landed first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ancsingabor
ancsingabor merged commit 717d831 into main Sep 30, 2026
2 checks passed
@ancsingabor
ancsingabor deleted the docs/guide-11-review branch September 30, 2026 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant