Skip to content

docs(guide): say what a unit is and what a chunk is, in the pitch that uses both - #32

Merged
ancsingabor merged 2 commits into
mainfrom
docs/guide-readme-review
Sep 30, 2026
Merged

ancsingabor merged 2 commits into
mainfrom
docs/guide-readme-review

Conversation

@ancsingabor

Copy link
Copy Markdown
Owner

Your instinct was right, and it found the sharpest thing in this whole review.

The example was the counter-example

Both READMEs used Summa I q.2 a.3 as the specimen of a verifiable citation. An article is a chunk, not a unit (ADR-002 § Amendment) — and q.2 a.3 is the worst possible choice, because it asks whether God exists, so its objections argue that he does not. lib/corpus/chunk.ts spells the failure out:

An objection states a position Aquinas is about to reject, often in crisp, quotable, highly retrievable prose. Retrieved without its role, "videtur quod Deus non sit" is a chunk that resolves to a real locator, quotes byte-exactly, passes the citation gate, and cites Thomas Aquinas for the non-existence of God.

So the headline example of the project's central guarantee was an address that cannot deliver that guarantee.

It survived four earlier passes because each grepped summa:I.q2.a3 — the locator form. These two were written as prose, *Summa* I q.2 a.3, so no search for the locator ever touched them.

  • 30-second pitch → uses only Catechism §1730.
  • 2-minute pitch → uses q.2 a.3 deliberately, as the worked example of why units are leaf-level.
  • Top-level README.md → I q.2 a.3 *co.*

"chunk" was introduced as the enemy, then used as ours

The page says other systems' "citations are therefore chunk IDs, which mean nothing outside the system" — and then, nine lines later, "chunking follows each source's own structure", with nothing in between saying these are two different objects. A reader meets the distinction for the first time in chapter 02. It is now stated where both words are first used: a unit is what we cite, a chunk is what we embed, plus chunk.ts's real payoff — re-tuning the chunker cannot invalidate a stored citation, which is what makes chunking a variable the harness can move rather than a decision that must be right first time.

An internal contradiction, two bullets apart

said
ingestion bullet "writes the result to Postgres with pgvector. That half is built"
query-path bullet "waits on one open decision: which embedding model to use"

Both cannot be true. Nothing is embedded — embed is deliberately unimplemented until ADR-008 is decided. Now stated once, naming why the two halves share one blocker: a query and the corpus compare only if the same model embedded both.

Two smaller ones, same family:

  • "Query path — it retrieves units." It retrieves chunks and cites the units inside them (guide 06's own diagram: chunks → units (context)). That flattening is the exact vagueness this PR is about.
  • Conventions defined a locator as source:address without saying it names a unit — the omission that let the Summa example through. Fixed.

1,187 words (was 917), under the budget. docs:lint green.

🤖 Generated with Claude Code

ancsingabor and others added 2 commits September 30, 2026 14:11
…t uses both

The guide README introduces "chunk" twice: once as the thing other systems get
wrong ("their citations are therefore chunk IDs, which mean nothing outside the
system") and once as something we do ("chunking follows each source's own
structure"), with nothing in between saying they are two different objects.
A reader who arrives here and then reads chapter 02 meets the distinction for
the first time three chapters in. It now states it where both words are first
used: **a unit is what we cite, a chunk is what we embed**, plus chunk.ts's
actual payoff — re-tuning the chunker cannot invalidate a stored citation,
which is what makes chunking a variable the harness can move.

── The example was the counter-example ─────────────────────────────────────

Both READMEs used `Summa I q.2 a.3` as the specimen of a verifiable citation.
An article is a chunk, not a unit (ADR-002 § Amendment), and q.2 a.3 is the
worst possible choice: it asks whether God exists, so its objections argue
that he does not. `chunk.ts` spells out the failure — retrieved without its
role, "videtur quod Deus non sit" resolves to a real address, quotes
byte-exactly, passes the gate, and cites Aquinas for the non-existence of God.
The headline example of the project's central guarantee was an address that
cannot deliver it.

It survived four earlier passes because those grepped `summa:I.q2.a3`, the
locator form. These were written as prose — `*Summa* I q.2 a.3` — so no search
for the locator ever touched them. The 30-second pitch now uses only
`Catechism §1730`, and the 2-minute one uses q.2 a.3 as the worked example of
why units are leaf-level. Top-level README corrected to `I q.2 a.3 co.`

── An internal contradiction, two bullets apart ───────────────────────────

"Ingestion … writes the result to Postgres with pgvector. That half is built"
sat directly above "the query path waits on one open decision: which embedding
model to use". Both cannot be true: nothing is embedded, and `embed` is
deliberately unimplemented until ADR-008 is decided. Now stated once, with the
reason the two halves share the same blocker — a query and the corpus compare
only if the same model embedded both.

Also: the query path "retrieves units". It retrieves chunks and cites the units
inside them (guide 06's own sequence diagram: `chunks → units (context)`), and
that flattening is the exact vagueness this PR is about. And the Conventions
list defined a locator as `source:address` without saying it names a *unit* —
the omission that let the Summa example through.

1,187 words.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Branch protection requires an up-to-date head branch; #31 landed first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ancsingabor
ancsingabor merged commit eaf3e3b into main Sep 30, 2026
2 checks passed
@ancsingabor
ancsingabor deleted the docs/guide-readme-review branch September 30, 2026 12:33
ancsingabor added a commit that referenced this pull request Sep 30, 2026
Branch protection requires an up-to-date head branch; #31 and #32 landed first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant