Skip to content

SemanticChunker strategy (depends on #12) #13

Description

Summary

Add a SemanticChunker strategy to the chunkers module introduced in #12.

Depends on #12 (the chunkers/ module + Chunker ABC + registry must exist first). This issue tracks the specific semantic strategy; it was the original motivation for splitting chunking out of the model layer.

What

Instead of slicing text into fixed N-token windows (FixedTokenChunker), a semantic chunker splits on meaning boundaries — keeping semantically coherent spans together rather than cutting mid-thought at an arbitrary token count.

Typical approaches to evaluate:

  • Sentence/paragraph grouping — split into sentences, then greedily pack sentences into a chunk until the token budget is hit (cheap, no model).
  • Embedding-similarity boundaries — embed sentences, start a new chunk where adjacent-sentence cosine similarity drops below a threshold (the "semantic chunking" people usually mean). Costs an extra embedding pass.

Design fit

SemanticChunker is just another strategy behind the same Chunker ABC (chunk(text) -> list[str]) and registry, parameterized by the same max_tokens budget (= min across configured models) so its output still fits every model. Selected via the chunking: config block:

chunking:
  strategy: semantic
  max_tokens: 256        # optional override of the derived budget
  # similarity_threshold: 0.6   # for the embedding-similarity variant

Open questions

  • Sentence-grouping (no model) vs embedding-similarity (extra pass) — start with the cheaper one?
  • For the embedding-similarity variant: which model embeds the sentences (a configured one vs a small dedicated one), and does that reintroduce model coupling we just removed?
  • Sentence splitter dependency (spaCy / nltk / a lightweight regex splitter).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions