Skip to content

Stop before Galaxy's dataset limit and fix two misleading warnings #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
IvoLeist merged 1 commit into from
Sep 2, 2026
Merged

Stop before Galaxy's dataset limit and fix two misleading warnings #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 24 additions & 12 deletions tools/langchain_text_splitters/langchain_text_splitters.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
--chunks-dir chunks
--chunk-size '$chunk_size'
--chunk-overlap '$chunk_overlap'
## The instance's own ceiling on datasets discovered per job, so the tool
## stops early instead of being failed by Galaxy after all the work.
--max-chunk-files '$__app__.config.max_discovered_files'
#if $splitter.method == "character"
#if $splitter.character_splitter.type == "recursive"
--splitter-type recursive_character
Expand All @@ -44,7 +47,7 @@
#if $splitter.character_splitter.type == "recursive"
#if $splitter.character_splitter.separator_settings.mode == "custom"
--separator-specs
#for $separator in $splitter.character_splitter.separator_settings.separators
#for $separator in $splitter.character_splitter.separator_settings.separators
#if $separator.separator_definition.sep_opt == "custom"
'custom:$separator.separator_definition.value'
#else
Expand Down Expand Up @@ -88,7 +91,7 @@
<param name="input" type="data" format="txt" label="Input text dataset"/>
<conditional name="splitter">
<param name="method" type="select" label="Text splitting boundaries" help="Choose where the input should be split. Recommended is on a character level, by paragraphs, lines, spaces etc. In between token can be useful for a LLM workflow which expects a specific token count per chunk.">
<option value="character" selected="true">
<option value="character" selected="true">
Character
</option>
<option value="sentence">
Expand Down Expand Up @@ -629,7 +632,8 @@
</assert_contents>
</output>
<assert_stdout>
<has_text text="WARNING: Potential upstream langchain-text-splitters bug: invalid start index returned for chunk(s): 2 (start_index: -1)"/>
<!-- Depends on https://github.com/langchain-ai/langchain/issues/29884 ; revisit on a version bump. -->
<has_text text="WARNING: The reported position of the following chunk(s)"/>
</assert_stdout>
<expand macro="assert_chunks_tsv"/>
<output_collection name="chunks_collection" type="list" count="3">
Expand Down Expand Up @@ -1079,7 +1083,8 @@
</assert_contents>
</output>
<assert_stdout>
<has_text text="WARNING: Potential upstream langchain-text-splitters bug: invalid start index returned for chunk(s): 2 (start_index: -1)"/>
<!-- Depends on https://github.com/langchain-ai/langchain/issues/29884 ; revisit on a version bump. -->
<has_text text="WARNING: The reported position of the following chunk(s)"/>
</assert_stdout>
<expand macro="assert_chunks_tsv"/>
<output_collection name="chunks_collection" type="list" count="3">
Expand Down Expand Up @@ -1288,12 +1293,19 @@
<output name="chunks_json">
<assert_contents>
<has_json_property_with_value property="number_of_chunks" value="3"/>
<!-- The position of the last chunk. Unlike the chunk texts and the
first chunk's offset, this changes as soon as the splitting does,
so it is what catches a regression here. -->
<has_json_property_with_value property="start_index" value="663"/>
<!-- An invalid start index is reported as null so that the field keeps a single JSON type. -->
<has_text text="&quot;start_index&quot;: null"/>
</assert_contents>
</output>
<expand macro="assert_chunks_tsv" min_lines="3"/>
<output_collection name="chunks_collection" type="list" count="3"/>
<assert_stdout>
<has_text text="WARNING: Potential upstream langchain-text-splitters bug: invalid start index returned for chunk(s): 2 (start_index: -1)"/>
<!-- Depends on https://github.com/langchain-ai/langchain/issues/29884 ; revisit on a version bump. -->
<has_text text="WARNING: The reported position of the following chunk(s)"/>
</assert_stdout>
</test>
<!-- A custom separator that starts with dashes must not be read as a command line option. -->
Expand Down Expand Up @@ -1472,7 +1484,7 @@
**What it does**

This tool splits a text dataset into chunks with `langchain-text-splitters`.
It is useful before retrieval-augmented generation, embedding, summarization,
It is useful before retrieval-augmented generation, embedding, summarization,
and other LLM workflows where long text needs to be bounded by a chunk size.

**Leveraged Methods of `langchain-text-splitters`**
Expand All @@ -1488,11 +1500,11 @@ and other LLM workflows where long text needs to be bounded by a chunk size.
Both sentence splitters preserve whitespace between sentences by default; in
that case a chunk starts with the line break that follows the previous sentence.
Leading and trailing whitespace can optionally be removed from each returned
chunk. However, for this we do not use the strip_whitespace option of the underlying splitters,
since in spaCy it would also delete the whitespace between sentences and in NLTK it could
cause the start index of a chunk to be reported incorrectly. Instead our python wrapper applies
the whitespace stripping only after LangChain has assembled a finished chunk.
Thus, preventing the two issues stated above.
chunk. Whitespace between two sentences *inside* the same chunk is preserved;
whitespace that falls on a chunk boundary is removed together with the chunk's
own leading and trailing whitespace. Where the position of a chunk in the input
cannot be established reliably, the start index is reported as null and a
warning is written to the job log.

The chunk overlap is applied in whole sentences: an overlapping sentence is only carried over if it
fits into the next chunk together with the following sentence, otherwise the
Expand Down Expand Up @@ -1533,7 +1545,7 @@ chunks.
- Galaxy collection of text files, with one raw chunk per file

**License**
Langchain-Text-Splitters is licensed under the MIT License.
Langchain-Text-Splitters is licensed under the MIT License.
</help>
<citations>
<citation type="bibtex">
Expand Down
91 changes: 70 additions & 21 deletions tools/langchain_text_splitters/split_text.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ def parse_args():
parser.add_argument("--separator")
parser.add_argument("--separator-specs", nargs="+", default=None)
parser.add_argument("--strip-whitespace", action="store_true")
parser.add_argument("--max-chunk-files", required=True)
return parser.parse_args()


Expand Down Expand Up @@ -289,6 +290,10 @@ def build_splitter(args, input_text, length_function, tiktoken_options):
"chunk_size": args.chunk_size,
"chunk_overlap": args.chunk_overlap,
"add_start_index": True,
# Never let langchain strip; main() does it once on the finished chunks.
# langchain would strip each sentence before joining them, welding
# "mat." to "The", and would misreport the start index. Defaults to True.
"strip_whitespace": False,
}

if args.splitter_type == "token":
Expand All @@ -303,30 +308,22 @@ def build_splitter(args, input_text, length_function, tiktoken_options):
}

if args.splitter_type == "nltk":
# strip_whitespace is handled on the finished chunks in main() instead of
# here. separator="" together with strip_whitespace=False keeps the chunk text
# identical to the matching slice of the input, so the reported start
# indices stay usable. Note that the span based tokenizer drops whatever
# follows the last sentence, see the warning in main().
# separator="" keeps the chunk text identical to the matching slice of
# the input, so the reported start indices stay usable. The span based
# tokenizer drops whatever follows the last sentence, see main().
return NLTKTextSplitter(
**length_options,
language=args.sentence_language,
separator="",
use_span_tokenize=True,
strip_whitespace=False,
)

if args.splitter_type == "spacy":
# strip_whitespace is handled on the finished chunks in main() instead of
# here. langchain strips every sentence before joining them, and
# separator="" joins with nothing, so letting it strip would also delete
# the space *between* two sentences and produce "mat.The".
splitter = SpacyTextSplitter(
**length_options,
pipeline=args.spacy_pipeline,
max_length=args.spacy_max_length,
separator="",
strip_whitespace=False,
)
# langchain only forwards max_length to the pipelines it loads with
# spacy.load(). The sentencizer is built from English() instead and
Expand All @@ -341,7 +338,6 @@ def build_splitter(args, input_text, length_function, tiktoken_options):

character_options = {
**length_options,
"strip_whitespace": args.strip_whitespace,
"keep_separator": KEEP_SEPARATOR_VALUES[args.keep_separator],
}

Expand Down Expand Up @@ -373,6 +369,35 @@ def build_splitter(args, input_text, length_function, tiktoken_options):
return RecursiveCharacterTextSplitter(**character_options)


def altered_text_cause(args):
"""Name the cause that fits this run instead of guessing a single one.

The two causes produce the same symptom but have opposite remedies, and
naming the wrong one sends the user in a circle.
"""
if args.splitter_type == "token" or args.length_mode == "token":
return (
"A cut between two tokens can fall inside a character that is "
"encoded in several bytes, which then becomes the Unicode "
"replacement character. Use one of the character based splitters "
"for text that is not plain ASCII."
)

if args.keep_separator == "false":
return (
"Discarding the separator drops the empty pieces between two "
"adjacent separators, so a run of separators is rebuilt as a "
"single one and the characters in between are lost. Choose a "
"setting that keeps the separator, or split on a separator that "
"does not occur several times in a row."
)

return (
"The splitter did not return the input unchanged; the chunks can "
"therefore no longer be traced back to a position in it."
)


def diagnose_chunk(
start_index,
chunk_text,
Expand All @@ -387,7 +412,9 @@ def diagnose_chunk(
"""
if (
start_index >= 0
and (previous_start_index is None or start_index > previous_start_index)
# Two chunks may legitimately begin at the same offset when the overlap
# repeats a whole split, so only a *backwards* jump is a real defect.
and (previous_start_index is None or start_index >= previous_start_index)
and input_text.startswith(chunk_text, start_index)
):
return None
Expand Down Expand Up @@ -434,9 +461,13 @@ def write_text_output(

def write_chunk_files(chunks_dir, chunks):
chunks_dir.mkdir(parents=True, exist_ok=True)
# Galaxy sorts the discovered elements lexically by file name, so the pad
# has to be wide enough for the largest index or chunk_10000 would sort
# between chunk_1000 and chunk_1001.
width = max(4, len(str(len(chunks))))

for chunk in chunks:
chunk_path = chunks_dir / f"chunk_{chunk['index']:04d}.txt"
chunk_path = chunks_dir / f"chunk_{chunk['index']:0{width}d}.txt"
chunk_path.write_text(
chunk["text"],
encoding="utf-8",
Expand Down Expand Up @@ -640,11 +671,8 @@ def main():
"WARNING: The text of the following chunk(s) does not occur in the "
f"input: {', '.join(str(number) for number in altered_text_warnings)}. "
"The splitter returned text it had modified, so no position in the "
"input describes it and the start index is reported as null. The "
"known cause is splitting between tokens: a cut can fall inside a "
"character that is encoded in several bytes, which then becomes the "
"Unicode replacement character. Use one of the character based "
"splitters for text that is not plain ASCII.",
"input describes it and the start index is reported as null. "
+ altered_text_cause(args),
flush=True,
)

Expand All @@ -654,12 +682,29 @@ def main():
for chunk_number, start_index in start_index_warnings
)
print(
"WARNING: Potential upstream langchain-text-splitters bug: "
f"invalid start index returned for chunk(s): {affected_chunks}. "
"WARNING: The reported position of the following chunk(s) does not "
f"describe where their text sits in the input: {affected_chunks}. "
"The start index of these chunks is reported as null.",
flush=True,
)

# Stop here, not after writing: Galaxy fails the job on the file count anyway.
# "None" means the admin set no limit, so isdigit() instead of int().
max_chunk_files = (
int(args.max_chunk_files) if args.max_chunk_files.isdigit() else 0
)

if max_chunk_files and len(chunks) > max_chunk_files:
sys.exit(
f"The selected settings produce {len(chunks)} chunks. The tool "
"writes one dataset per chunk, and this Galaxy instance refuses a "
f"job that produces more than {max_chunk_files} datasets, so the "
"job would be failed after the split had already run. Raise the "
f"target chunk size: the input holds {len(input_text)} characters, "
f"so a chunk size above {len(input_text) // max_chunk_files + 1} "
"keeps the count under the limit."
)

if empty_chunks:
print(
f"WARNING: {empty_chunks} chunk(s) held nothing but whitespace and "
Expand All @@ -673,6 +718,10 @@ def main():
# whatever follows is dropped and the chunks no longer add up to the input.
# The first span always starts at offset 0, so nothing is lost in front.
# The spaCy splitter keeps everything.
# Only reported when the chunks were not stripped. Punkt leaves nothing but
# whitespace after the last span, so with stripping on this could only ever
# report whitespace the user asked to discard, inflated by whatever strip()
# itself took off the end of the last chunk.
if args.splitter_type == "nltk" and chunks and not args.strip_whitespace:
last_chunk = chunks[-1]
last_text = last_chunk["text"]
Expand Down
Loading