Skip to content

Fix output correctness issues found in review - #2

Merged
IvoLeist merged 1 commit into
IvoLeist:add-langchain_text_splittersfrom
arash77:langchain-text-splitters-review-fixes
Aug 5, 2026
Merged

Fix output correctness issues found in review#2
IvoLeist merged 1 commit into
IvoLeist:add-langchain_text_splittersfrom
arash77:langchain-text-splitters-review-fixes

Conversation

@arash77

@arash77 arash77 commented Aug 5, 2026

Copy link
Copy Markdown

The fixes for the points from my review on bgruening#1947, as discussed there.

Output correctness

  • Read the input as UTF-8 bytes instead of Path.read_text(), which silently rewrote CRLF/CR line endings and shifted the start indices.
  • TSV: escape \, \t, \r and \n and write the row without the csv module, so each chunk stays on one row and the escaping can be reversed. csv.writer additionally quoted every chunk containing a ".
  • Write all outputs as UTF-8 instead of the system locale encoding.

Errors

  • Clear message for invalid UTF-8 input, for a disallowed special token on the character splitter, and for unsupported or out-of-range custom separator escapes.
  • Reject input that is empty or whitespace only instead of writing empty outputs.
  • Pass a custom separator as --separator=VALUE, so a separator starting with a dash (e.g. ---) is not read as an option.

Output format

  • start_index is null when the splitter returns an invalid position, instead of replacing the number with a message string. The warning on stdout is unchanged and still carries the value.
  • One length key per chunk together with the existing length_unit, instead of character_count/token_count.
  • strip_whitespace is reported as false in token mode, where the splitter never applies it.

Tests and docs

  • New tests: whitespace-only input, disallowed special token on the character splitter, dash-prefixed custom separator, predefined separator inside the repeat.
  • The two overlapping token tests now expect start_index: null and the warning, instead of the not_has_text check that only passed because the old code wrote a string there. They stay green now and will fail once the upstream bug (TokenTextSplitter start indices are sometimes -1 langchain-ai/langchain#29884) is fixed, so nothing has to be re-enabled by hand.
  • Help documents the tiktoken encoding cache, the TSV escaping and the JSON fields.
  • The two tiktoken command line blocks are merged into a macro.

planemo lint clean. planemo test 16/16 before, 20/20 after.

Input handling:
- Decode the input as UTF-8 instead of Path.read_text(), which silently
  rewrote CRLF/CR line endings and shifted the reported start indices.
- Report invalid UTF-8 input with a clear message instead of a traceback.
- Reject input that is empty or whitespace only instead of writing empty
  outputs.

TSV output:
- Escape backslash, tab, carriage return and newline so that the chunk text
  stays on one row and the escaping can be reversed. The previous escaping
  could not be undone, and csv.writer additionally quoted any chunk
  containing a double quote.

Errors and metadata:
- Report a disallowed special token with a clear message on the character
  splitter as well, not only on the token splitter.
- Report an invalid start index as null instead of replacing the number with
  a string, so the JSON field keeps a single type. The warning on stdout is
  unchanged.
- Use a single "length" key per chunk together with the existing
  "length_unit" instead of character_count/token_count.
- Report strip_whitespace as false in token mode, where the splitter never
  applies it.
- Write all outputs as UTF-8 instead of the system locale encoding.

Command line and separators:
- Pass a custom separator as --separator=VALUE so that a separator starting
  with a dash is not read as an option.
- Reject unsupported and out-of-range custom separator escapes with a clear
  message instead of crashing or passing them through silently.
- Make --encoding-name and --model-name mutually exclusive.

Tests and docs:
- Add tests for whitespace-only input, a disallowed special token on the
  character splitter, a dash-prefixed custom separator, and a predefined
  separator inside the repeat.
- Expect null start indices in the two overlapping token tests, which hit
  the same upstream bug as the dedicated repro test.
- Document the tiktoken encoding cache, the TSV escaping and the JSON
  fields in the tool help.
- Deduplicate the two tiktoken command line blocks into a macro.
@IvoLeist
IvoLeist merged commit b46322d into IvoLeist:add-langchain_text_splitters Aug 5, 2026
6 checks passed
@arash77
arash77 deleted the langchain-text-splitters-review-fixes branch August 6, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants