Repository navigation
test: run chunker tests on a stand-in tokenizer - #544
Merged
Merged
Conversation
Every test that ran the chunker loaded the BGE-M3 tokenizer, so on a machine without it in the Hugging Face cache (a fresh clone, a cloud sandbox) 106 tests failed rather than skipped. Those tests check chunk structure, not BGE-M3's subword splits. An autouse fixture now gives the chunker a deterministic stand-in: tokens are runs of up to four non-space characters, with real character offsets, so windowing and truncation slice text exactly as they would with the real tokenizer. requires_model tests are left alone. The real tokenizer stays pinned: a requires_model module re-runs the classes that assert on token counts, windows, truncation and the golden fixtures, plus a check that it really is the real tokenizer. Nothing in ingest/ changes. Measured on Linux with no model available: the full suite goes from 214 failures to 108, all 106 fixed being the chunker-bound tests, and none new. The remaining failures are server startup loading the embedding model, plus 16 proxy errors in test_formal_release_pin. Closes #543.
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #543, using option A from that issue (the stand-in always applies in tests).
Every test that ran the chunker loaded the BGE-M3 tokenizer. On a machine without it in the Hugging Face cache, such as a fresh clone or a cloud sandbox, 106 tests failed rather than skipped. Those tests check chunk structure, not BGE-M3's subword splits.
tests/_stand_in_tokenizer.py: a deterministic stand-in for the three calls the chunker makes:encode,decode, and__call__(…, return_offsets_mapping=True). A token is a run of up to four non-space characters, with real character offsets, so windowing and truncation slice text exactly as the real tokenizer's offsets would.tests/conftest.py: an autouse fixture installs the stand-in asingest.chunker._tokenizerfor every test exceptrequires_modelones, andmonkeypatchrestores the previous value afterwards.tests/test_chunker_real_tokenizer.py(requires_model): it re-runs thetest_chunker.pyclasses that assert on token counts, proof windows, statement truncation and the 10 golden fixtures, this time on the real tokenizer. A guard test fails if the stand-in leaks in.Nothing in
ingest/changes.Verification
Run on Ubuntu (Linux) at
ad16670, on top ofmainatd9ddaa8, with no BGE-M3 files available (huggingface.coblocked, empty HF cache).test_chunker.py,test_textbook_chunker.py,test_notebook_chunks_backfill.py,test_chunker_ids.pyandtest_preamble.pyrun 281 tests: 0 failures, 1 skip. Before, 106 of them failed.pytest: 108 failures and errors, down from 214, with none new. The 106 fixed are exactly the chunker-bound tests. Of the rest:httpxproxy errors intest_formal_release_pin.py.-m requires_model: 44 fail here withCan't load the configuration of 'BAAI/bge-m3', which confirms they really use the real tokenizer rather than the stand-in. The 3 that pass do not tokenize.ruff check .is clean.Needed before merge, on the workstation:
pytest -m requires_model tests/test_chunker_real_tokenizer.py. This is the run that proves the real tokenizer still satisfies the windowing, truncation and golden-fixture assertions. It cannot run here.Notes
make teststops exercising real BGE-M3 tokenization;-m requires_modelcovers it. This trade was chosen on Chunker tests need the BGE-M3 tokenizer download; use a stand-in #543 so that a green run means the same thing on every machine.2307.00007: this multi-window fixture also passes on the stand-in, possibly because both tokenizers produce the same window count for that proof. If a fixture is ever regenerated so the two disagree, the stand-in suite needs that fixture's real-tokenizer assertions moved to therequires_modelmodule.Generated by Claude Code