Fix whitespace handling in the sentence splitters

[arash77]

bgruening#1947
Fixes two bugs in the sentence splitters that silently changed the user's text, and the failing test.

spaCy + "strip whitespace" removed the space between sentences

langchain strips each sentence before joining, and we join with an empty separator, so the space is gone:

'The cat sat on the mat. The dog barked loudly.'  ->  'mat.The'

The chunk then does not occur in the input any more, so start_index became null and the warning blamed a UTF-8 token problem that was not there.

Test 13 passed because sentence_spacy.txt separates sentences with \n, which spaCy attaches to the next sentence. I added a fixture with space separated sentences, which is the normal case for prose.

Fix: build the splitter with strip_whitespace=False and strip the finished chunk instead. That is what the option's label already promises ("Whitespace within chunks is preserved"), and the chunk stays a slice of the input, so the start index stays valid.

Whitespace-only chunks were dropped for every splitter

With the token splitter on a chapter break, 64 characters disappeared and the remaining chunks were renumbered. They are now only dropped when stripping was asked for and nothing is left.

The spaCy input limit was one number for two pipelines

10 million characters allows about 44 GB with the English model, which needs about 4 kB per character. The limit is now per pipeline: 2 million for the model, 20 million for the sentencizer, which needs about 0.2 kB per character.

About the trailing newline TODO

Test 9 is correct as it is. The dropped character is the \n Galaxy appends, and punkt reports sentence spans with trailing whitespace trimmed, so it falls outside the last span. I flipped the assertion to expect the warning, so the behaviour is pinned instead of red.

removesuffix("\n") would fix it in the wrong place. An uploaded abc and an uploaded abc\n both arrive as abc\n, so we cannot tell them apart, and we would cut a real character off every input that already ends in a newline. It would also not settle NLTK, because punkt drops all trailing whitespace, so a file ending in a blank line still loses more.

Adding a newline to the test files does not silence the warning either, I tested it. It is still worth doing so the file on disk matches what Galaxy uploads, but it changes no result.

Also

• the punkt_tab notes in the code and the help now match the nltk_data requirement; both still said the data had to be installed by hand
• strip_whitespace is reported once from the argument instead of being hardcoded per splitter, so it cannot report false while stripping

28/28 tests pass. I also ran seven parameter combinations against a live Galaxy, including three the test suite does not cover.