Add test cases for altered_text_cause - #7
Merged
IvoLeist merged 1 commit intoSep 2, 2026
Merged
Conversation
The function picks between two causes for the same symptom, but nothing tested that it picks the right one. - multibyte_persian.txt with the token splitter: gpt2 splits Persian into very small tokens, so a cut lands inside a multi-byte character and the chunk comes back with the replacement character. - repeated_separator.txt with a discarded comma: the run of separators is rebuilt as a single one, so the chunk text no longer occurs in the input. Each test asserts the cause it expects and that the other one is absent, so swapping the two would fail.
IvoLeist
merged commit Sep 2, 2026
55b76de
into
IvoLeist:add-langchain_text_splitters
8 of 10 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on your review point:
altered_text_cause()had no test cases.You were right. The function is called once, in
split_text.py, and nothingexercised it. It picks between two causes for the same symptom — a chunk whose
text no longer occurs in the input — and the two causes have opposite remedies,
so picking the wrong one sends the user in a circle. That choice was untested.
What is added
Two test cases, one per branch, plus a small input file for each:
multibyte_persian.txtwith the token splitter. gpt2 cuts Persian intovery small tokens, so a cut lands inside a multi-byte character and the chunk
comes back with the replacement character. Expected cause: the token cut.
repeated_separator.txtwith a discarded comma. The run of separators isrebuilt as a single one, so the chunk text no longer occurs in the input.
Expected cause: the separator.
Each test asserts the cause it expects and that the other cause is absent,
so swapping the two branches makes both tests fail rather than one.
Notes
change the tool's behaviour.
four splitters, both
keep_separatorvalues, four chunk sizes and four inputshapes, and it never fired. It looks like a safety net rather than a case that
can be reached, so I left it uncovered instead of contriving an input for it.
planemo lintis clean and both new tests pass locally (31 tests total, was 29).