Skip to content

Add test cases for altered_text_cause - #7

Merged
IvoLeist merged 1 commit into
IvoLeist:add-langchain_text_splittersfrom
arash77:langchain-text-splitters-tests
Sep 2, 2026
Merged

Add test cases for altered_text_cause#7
IvoLeist merged 1 commit into
IvoLeist:add-langchain_text_splittersfrom
arash77:langchain-text-splitters-tests

Conversation

@arash77

@arash77 arash77 commented Sep 2, 2026

Copy link
Copy Markdown

Follows up on your review point: altered_text_cause() had no test cases.

You were right. The function is called once, in split_text.py, and nothing
exercised it. It picks between two causes for the same symptom — a chunk whose
text no longer occurs in the input — and the two causes have opposite remedies,
so picking the wrong one sends the user in a circle. That choice was untested.

What is added

Two test cases, one per branch, plus a small input file for each:

  • multibyte_persian.txt with the token splitter. gpt2 cuts Persian into
    very small tokens, so a cut lands inside a multi-byte character and the chunk
    comes back with the replacement character. Expected cause: the token cut.
  • repeated_separator.txt with a discarded comma. The run of separators is
    rebuilt as a single one, so the chunk text no longer occurs in the input.
    Expected cause: the separator.

Each test asserts the cause it expects and that the other cause is absent,
so swapping the two branches makes both tests fail rather than one.

Notes

  • Only tests and test data. No change to the tool logic, so nothing here can
    change the tool's behaviour.
  • The function has a third, fallback cause. I tried 96 combinations across the
    four splitters, both keep_separator values, four chunk sizes and four input
    shapes, and it never fired. It looks like a safety net rather than a case that
    can be reached, so I left it uncovered instead of contriving an input for it.

planemo lint is clean and both new tests pass locally (31 tests total, was 29).

The function picks between two causes for the same symptom, but nothing
tested that it picks the right one.

- multibyte_persian.txt with the token splitter: gpt2 splits Persian into
  very small tokens, so a cut lands inside a multi-byte character and the
  chunk comes back with the replacement character.
- repeated_separator.txt with a discarded comma: the run of separators is
  rebuilt as a single one, so the chunk text no longer occurs in the input.

Each test asserts the cause it expects and that the other one is absent,
so swapping the two would fail.
@IvoLeist
IvoLeist merged commit 55b76de into IvoLeist:add-langchain_text_splitters Sep 2, 2026
8 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants