v1.1.0

[chonknick]

What's Changed

• Update docs with Cloud SDKs by @shreyash-chonkie in Update docs with Cloud SDKs #189
• Update README.md by @chonknick in Update README.md #192
• Add static typing to the cloud chunkers + Workflow for mypy testing on PRs by @chonknick in Add static typing to the cloud chunkers + Workflow for mypy testing on PRs #194
• Fix: Remove the mypy_errors.txt file by @chonknick in Fix: Remove the mypy_errors.txt file #201
• Update all the chunkers to use "character" instead of "gpt2" by @chonknick in Update all the chunkers to use "character" instead of "gpt2" #202
• Feat: Shift the tokenizer to be character type to make the tokenizers optional by @chonknick in Feat: Shift the tokenizer to be character type to make the tokenizers optional #205
• Fix: return_type is removed for returning only chunks to simplify mypy static typing by @chonknick in Fix: return_type is removed for returning only chunks to simplify mypy static typing #206
• Fix: Remove return_type='texts' to increase type consistency and resolve complex mypy issues by @chonknick in Fix: Remove return_type='texts' to increase type consistency and resolve complex mypy issues #207
• Update version to 1.1.0 in __init__.py by @chonknick in Update version to 1.1.0 in __init__.py #209

Full Changelog: v1.0.10...v1.1.0

---

This discussion was created from the release v1.1.0.

[dantetemplar]

Why we want to simplify life for mypy but not for users?

Texts option was handy, moreover it's breaking change that will crash almost every app that uses chonkie.

Also, moving to character tokenizer may cause unnoticed problems.

[shreyash-chonkie]

Hey @dantetemplar,

I hear you on the breaking changes. That's never ideal, unfortunately sometimes unavoidable.

Regarding the texts option: We removed this to simplify the package and improve type safety. The same result is easily achievable with [chunk.text for chunk in chunks] and our usage research showed most users just did the list comprehension, rarely relying on the texts option.

Regarding the character tokenizer: This change makes the tokenizers package optional, significantly reducing installation size. Existing code will work unchanged if you have tokenizers installed and specify tokenizer="gpt2" in your chunker initialization.

The benefits are a lighter, cleaner package with better type safety. We think this package will be easier to work with for users in the long run

[dantetemplar]

I am okay with that but I see same opinion about breaking change in NLP chat in telegram.
https://t.me/natural_language_processing

[chonknick]

Hey @dantetemplar! 👋

Chonkie's every v1.x version would be a breaking change, with non-breaking changes (like new features, bug fixes etc.) in patch releases (like v1.0.10).

Users can ensure that their app doesn't break by having locked versions in the form of chonkie >= 1.0.0, <1.1. We recommend this approach.

Since Chonkie as a library is also growing, we hope to improve how we build it — and in the process making breaking changes is a necessity. To ensure that we're moving quickly, while not giving a hard time to the end users, we make breaking changes rarely (usually once or twice a quarter).

Thanks 😊