Summary
Add a SemanticChunker strategy to the chunkers module introduced in #12.
Depends on #12 (the chunkers/ module + Chunker ABC + registry must exist first). This issue tracks the specific semantic strategy; it was the original motivation for splitting chunking out of the model layer.
What
Instead of slicing text into fixed N-token windows (FixedTokenChunker), a semantic chunker splits on meaning boundaries — keeping semantically coherent spans together rather than cutting mid-thought at an arbitrary token count.
Typical approaches to evaluate:
- Sentence/paragraph grouping — split into sentences, then greedily pack sentences into a chunk until the token budget is hit (cheap, no model).
- Embedding-similarity boundaries — embed sentences, start a new chunk where adjacent-sentence cosine similarity drops below a threshold (the "semantic chunking" people usually mean). Costs an extra embedding pass.
Design fit
SemanticChunker is just another strategy behind the same Chunker ABC (chunk(text) -> list[str]) and registry, parameterized by the same max_tokens budget (= min across configured models) so its output still fits every model. Selected via the chunking: config block:
chunking:
strategy: semantic
max_tokens: 256 # optional override of the derived budget
# similarity_threshold: 0.6 # for the embedding-similarity variant
Open questions
- Sentence-grouping (no model) vs embedding-similarity (extra pass) — start with the cheaper one?
- For the embedding-similarity variant: which model embeds the sentences (a configured one vs a small dedicated one), and does that reintroduce model coupling we just removed?
- Sentence splitter dependency (spaCy / nltk / a lightweight regex splitter).
Summary
Add a
SemanticChunkerstrategy to thechunkersmodule introduced in #12.Depends on #12 (the
chunkers/module +ChunkerABC + registry must exist first). This issue tracks the specific semantic strategy; it was the original motivation for splitting chunking out of the model layer.What
Instead of slicing text into fixed N-token windows (
FixedTokenChunker), a semantic chunker splits on meaning boundaries — keeping semantically coherent spans together rather than cutting mid-thought at an arbitrary token count.Typical approaches to evaluate:
Design fit
SemanticChunkeris just another strategy behind the sameChunkerABC (chunk(text) -> list[str]) and registry, parameterized by the samemax_tokensbudget (= min across configured models) so its output still fits every model. Selected via thechunking:config block:Open questions