Stop before Galaxy's dataset limit and fix two misleading warnings - #6
Merged
IvoLeist merged 1 commit intoSep 2, 2026
Conversation
…two misleading warnings - fail early when the settings would produce more chunks than the instance's max_discovered_files, instead of splitting everything and then letting Galaxy fail the job on the file count - name the cause that fits the run in the altered-text warning instead of always blaming multi-byte tokens - stop reporting a null start index for overlapping chunks that legitimately begin at the same offset - set strip_whitespace once for every splitter instead of per branch - correct the help on whitespace at a chunk boundary, and let three tests assert the warning instead of pinning the upstream -1
arash77
force-pushed
the
langchain-text-splitters-round-3
branch
from
September 1, 2026 08:36
878b07d to
4331ee4
Compare
IvoLeist
merged commit Sep 2, 2026
09ebe0d
into
IvoLeist:add-langchain_text_splitters
8 of 10 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bgruening#1947
Three things that go wrong for a user and are hard to understand from the job log.
The job dies after all the work is done
The tool writes one dataset per chunk. Galaxy refuses a job that produces more
datasets than
max_discovered_files, but only after the split has run, so theuser waits, then gets a failure that does not say what to change.
A 4.9 MB text with chunk size 400 makes 14142 files. The tool now checks the
instance's own setting first and stops with a message that names the number of
chunks and a chunk size that would fit.
The warning about changed text named the wrong cause
When a chunk can no longer be found in the input, the warning always blamed a
cut through a multi-byte character. That is only one of two causes. Dropping the
separator also loses text, and the two need opposite fixes, so the wrong name
sends the user in a circle. The warning now names the cause that fits the run.
Overlapping chunks were called broken
With overlap, two chunks can honestly start at the same character. That was
treated as an error, so correct chunks got a null start index and a warning
about an upstream bug. Only a jump backwards is a real defect now.
Also
strip_whitespaceis set once for all splitters instead of in three placesa chunk, but not at a chunk boundary
-1; they now assert the warning, so a fixupstream will not turn them red
29/29 tests pass,
planemo lintis clean. Also run against a local Galaxy withmax_discovered_filesset to 25 to confirm the tool reads the instance value.