Skip to content

test: run chunker tests on a stand-in tokenizer - #544

Merged
chris-dare-dev merged 1 commit into
mainfrom
claude/chunker-test-tokenizer
Nov 15, 2026
Merged

chris-dare-dev merged 1 commit into
mainfrom
claude/chunker-test-tokenizer

Conversation

@chris-dare-dev

Copy link
Copy Markdown
Owner

What

Closes #543, using option A from that issue (the stand-in always applies in tests).

Every test that ran the chunker loaded the BGE-M3 tokenizer. On a machine without it in the Hugging Face cache, such as a fresh clone or a cloud sandbox, 106 tests failed rather than skipped. Those tests check chunk structure, not BGE-M3's subword splits.

  • tests/_stand_in_tokenizer.py: a deterministic stand-in for the three calls the chunker makes: encode, decode, and __call__(…, return_offsets_mapping=True). A token is a run of up to four non-space characters, with real character offsets, so windowing and truncation slice text exactly as the real tokenizer's offsets would.
  • tests/conftest.py: an autouse fixture installs the stand-in as ingest.chunker._tokenizer for every test except requires_model ones, and monkeypatch restores the previous value afterwards.
  • tests/test_chunker_real_tokenizer.py (requires_model): it re-runs the test_chunker.py classes that assert on token counts, proof windows, statement truncation and the 10 golden fixtures, this time on the real tokenizer. A guard test fails if the stand-in leaks in.

Nothing in ingest/ changes.

Verification

Run on Ubuntu (Linux) at ad16670, on top of main at d9ddaa8, with no BGE-M3 files available (huggingface.co blocked, empty HF cache).

  • Chunker files: test_chunker.py, test_textbook_chunker.py, test_notebook_chunks_backfill.py, test_chunker_ids.py and test_preamble.py run 281 tests: 0 failures, 1 skip. Before, 106 of them failed.
  • Full pytest: 108 failures and errors, down from 214, with none new. The 106 fixed are exactly the chunker-bound tests. Of the rest:
    • ~90 boot the server, which loads the BGE-M3 embedding model; that is a separate change.
    • 16 are httpx proxy errors in test_formal_release_pin.py.
  • Real-tokenizer module, default run: all 47 tests skip (opt-in).
  • Real-tokenizer module, -m requires_model: 44 fail here with Can't load the configuration of 'BAAI/bge-m3', which confirms they really use the real tokenizer rather than the stand-in. The 3 that pass do not tokenize.
  • ruff check . is clean.

Needed before merge, on the workstation: pytest -m requires_model tests/test_chunker_real_tokenizer.py. This is the run that proves the real tokenizer still satisfies the windowing, truncation and golden-fixture assertions. It cannot run here.

Notes


Generated by Claude Code

Every test that ran the chunker loaded the BGE-M3 tokenizer, so on a
machine without it in the Hugging Face cache (a fresh clone, a cloud
sandbox) 106 tests failed rather than skipped.

Those tests check chunk structure, not BGE-M3's subword splits. An
autouse fixture now gives the chunker a deterministic stand-in: tokens
are runs of up to four non-space characters, with real character
offsets, so windowing and truncation slice text exactly as they would
with the real tokenizer. requires_model tests are left alone.

The real tokenizer stays pinned: a requires_model module re-runs the
classes that assert on token counts, windows, truncation and the
golden fixtures, plus a check that it really is the real tokenizer.
Nothing in ingest/ changes.

Measured on Linux with no model available: the full suite goes from
214 failures to 108, all 106 fixed being the chunker-bound tests, and
none new. The remaining failures are server startup loading the
embedding model, plus 16 proxy errors in test_formal_release_pin.

Closes #543.
@chris-dare-dev
chris-dare-dev merged commit d1b9533 into main Nov 15, 2026
1 check passed
@chris-dare-dev
chris-dare-dev deleted the claude/chunker-test-tokenizer branch November 15, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chunker tests need the BGE-M3 tokenizer download; use a stand-in

2 participants