Repository navigation
docs(guide): say what a unit is and what a chunk is, in the pitch that uses both - #32
Merged
Merged
Conversation
…t uses both
The guide README introduces "chunk" twice: once as the thing other systems get
wrong ("their citations are therefore chunk IDs, which mean nothing outside the
system") and once as something we do ("chunking follows each source's own
structure"), with nothing in between saying they are two different objects.
A reader who arrives here and then reads chapter 02 meets the distinction for
the first time three chapters in. It now states it where both words are first
used: **a unit is what we cite, a chunk is what we embed**, plus chunk.ts's
actual payoff — re-tuning the chunker cannot invalidate a stored citation,
which is what makes chunking a variable the harness can move.
── The example was the counter-example ─────────────────────────────────────
Both READMEs used `Summa I q.2 a.3` as the specimen of a verifiable citation.
An article is a chunk, not a unit (ADR-002 § Amendment), and q.2 a.3 is the
worst possible choice: it asks whether God exists, so its objections argue
that he does not. `chunk.ts` spells out the failure — retrieved without its
role, "videtur quod Deus non sit" resolves to a real address, quotes
byte-exactly, passes the gate, and cites Aquinas for the non-existence of God.
The headline example of the project's central guarantee was an address that
cannot deliver it.
It survived four earlier passes because those grepped `summa:I.q2.a3`, the
locator form. These were written as prose — `*Summa* I q.2 a.3` — so no search
for the locator ever touched them. The 30-second pitch now uses only
`Catechism §1730`, and the 2-minute one uses q.2 a.3 as the worked example of
why units are leaf-level. Top-level README corrected to `I q.2 a.3 co.`
── An internal contradiction, two bullets apart ───────────────────────────
"Ingestion … writes the result to Postgres with pgvector. That half is built"
sat directly above "the query path waits on one open decision: which embedding
model to use". Both cannot be true: nothing is embedded, and `embed` is
deliberately unimplemented until ADR-008 is decided. Now stated once, with the
reason the two halves share the same blocker — a query and the corpus compare
only if the same model embedded both.
Also: the query path "retrieves units". It retrieves chunks and cites the units
inside them (guide 06's own sequence diagram: `chunks → units (context)`), and
that flattening is the exact vagueness this PR is about. And the Conventions
list defined a locator as `source:address` without saying it names a *unit* —
the omission that let the Summa example through.
1,187 words.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Branch protection requires an up-to-date head branch; #31 landed first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Your instinct was right, and it found the sharpest thing in this whole review.
The example was the counter-example
Both READMEs used
Summa I q.2 a.3as the specimen of a verifiable citation. An article is a chunk, not a unit (ADR-002 § Amendment) — and q.2 a.3 is the worst possible choice, because it asks whether God exists, so its objections argue that he does not.lib/corpus/chunk.tsspells the failure out:So the headline example of the project's central guarantee was an address that cannot deliver that guarantee.
It survived four earlier passes because each grepped
summa:I.q2.a3— the locator form. These two were written as prose,*Summa* I q.2 a.3, so no search for the locator ever touched them.Catechism §1730.README.md→I q.2 a.3 *co.*"chunk" was introduced as the enemy, then used as ours
The page says other systems' "citations are therefore chunk IDs, which mean nothing outside the system" — and then, nine lines later, "chunking follows each source's own structure", with nothing in between saying these are two different objects. A reader meets the distinction for the first time in chapter 02. It is now stated where both words are first used: a unit is what we cite, a chunk is what we embed, plus
chunk.ts's real payoff — re-tuning the chunker cannot invalidate a stored citation, which is what makes chunking a variable the harness can move rather than a decision that must be right first time.An internal contradiction, two bullets apart
Both cannot be true. Nothing is embedded —
embedis deliberately unimplemented until ADR-008 is decided. Now stated once, naming why the two halves share one blocker: a query and the corpus compare only if the same model embedded both.Two smaller ones, same family:
chunks → units (context)). That flattening is the exact vagueness this PR is about.source:addresswithout saying it names a unit — the omission that let the Summa example through. Fixed.1,187 words (was 917), under the budget.
docs:lintgreen.🤖 Generated with Claude Code