Repository navigation
docs(guide/11): correct the release rule, and stop understating the licence risk - #33
Merged
Merged
Conversation
…icence risk Two answers on this page were wrong in ways a reviewer would catch, which is the one page where that costs something. **The release rule (Q3) was wrong in both directions.** It said "a retrieval or prompt change must attach an eval diff, and a regression blocks it". The actual rule (docs/evaluation.md, moved there in PR #21): - a `recall@10` regression is **named and justified, not blocked** — a pp threshold at n=40–60 is false precision, which is why the rule stopped being ">2pp recall@10" in the first place; - what *does* block is a drop in citation validity or quote fidelity; - it is **enforced in review; CI does not run the eval** — the qualifier that matters most here, since "blocks it" invites a reviewer to assume a CI gate; - it covers retrieval, chunking, embedding **or prompt**. Chunking was missing, and omitting it is exactly how someone re-tunes the chunker without a diff — the thing `.claude/project.md` rule 6 and `chunk.ts` both forbid. **The licence weakness still had its pre-#24 framing.** The row read "May corpus text be sent to a hosted *embedding* API? … the bake-off starts with local models, which don't raise the question". ADR-003 § Extended widened that to generation: composing a prompt puts the retrieved units into a request to Anthropic, in production, per question. Local candidates mitigate the bake-off half and nothing mitigates the other one, so the mitigation as written was false for the half that matters. Now a short row plus a paragraph saying it is the only open item that invalidates a design rather than narrowing a choice. Understating this on the page whose stated purpose is that "a design explained without its costs is not credible" was the worst place for it to sit. Also: - **A missing question a reviewer certainly asks: why Claude?** Four files assume Anthropic, nothing compared it, no ADR argues it — and the harness cannot settle it, since it scores no generation model. Added as question 6 with the honest answer, and as a weakness row. "The first five" became "the six that come up first"; that counted heading is itself the tripwire the new § What I would do differently bullet names. - The `summa:I.q2.a3` bullet now carries why the fix took six passes: the last two copies were prose, so no grep for the locator form could find them. 1,415 words. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The questions themselves are good — I'd keep every one. Two of the answers were wrong, and this is the one page where a wrong answer costs something, because it is the page a reviewer reads to decide whether to trust the rest.
The release rule (Q3) was wrong in both directions
It said: "a retrieval or prompt change must attach an eval diff, and a regression blocks it."
recall@10regressionThe missing qualifier matters most: telling a reviewer "a regression blocks it" invites them to assume a CI gate, and there isn't one. And omitting chunking from the trigger list is exactly how someone re-tunes the chunker without a diff — the thing
.claude/project.mdrule 6 andchunk.tsboth forbid.The licence weakness still had its pre-#24 framing
The row read: "May corpus text be sent to a hosted embedding API? … The bake-off starts with local models, which don't raise the question."
ADR-003 § Extended widened that to generation. Composing a prompt puts the retrieved units themselves into a request to Anthropic — in production, per question, for as long as the system runs. Local candidates mitigate the bake-off half; nothing mitigates the other one, and there is no local fallback anywhere in the design. So the mitigation as written was false for the half that matters.
It now gets a short row plus a paragraph naming it as the only open item that can invalidate a design rather than narrow a choice. Understating it on the page whose own TL;DR says "a design explained without its costs is not credible" was the worst possible place for it to sit.
A question a reviewer certainly asks, that wasn't here
"Why Claude?" Four files assume Anthropic, nothing ever compared it, no ADR argues it — and the harness cannot settle it, because it scores no generation model at all. Added as question 6 with the honest answer, plus a weakness row.
That made "The first five" into "the six that come up first" — and that counted heading is itself the tripwire named in the new
§ What I would do differentlybullet:The
summa:I.q2.a3bullet also now carries why the fix took six passes: the last two copies were prose, so no grep for the locator form could find them.1,415 words, in band.
docs:lintgreen.🤖 Generated with Claude Code