Skip to content

improve max length handling #733

@jordanxlau

Description

@jordanxlau

Bug description

Currently, the algorithm in textsplit.py will cut a sentence off at the max_length, even if this splits up a word. Instead, this algorithm should always preserve words and perhaps even preserve whole sentences or phrases.

How to reproduce the bug

# Sample code to reproduce the problem

Error messages and logs

# Error messages and logs here please

Environment

Current environment
# Please paste the output of `everyvoice --diagnostic` here
# EveryVoice Diagnostic information

More info

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions