fix(fts): slice fuzzy query prefix on a char boundary

[glitch-ux]

What

Fixes a panic in fuzzy full-text search: a MatchQuery with fuzziness > 0 and prefix_length > 0 crashes when the query term starts with a multibyte UTF-8 character (e.g. über, café).

Closes #7891.

Why

InvertedPartition::collect_fuzzy_candidates took the leading prefix of the query token with a raw byte slice, but prefix_length is a character count. When the byte offset falls inside a multibyte character the slice panics, aborting the query while expanding the term.

How

Advance prefix_length characters with char_indices() and slice on the resulting boundary, clamping to the whole token when it is shorter.

Testing

• Added a regression test for a multibyte query token.
• cargo test -p lance-index --lib fuzzy, clippy -D warnings, and rustfmt all clean.

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

Fuzzy prefix UTF-8 handling

Layer / File(s)	Summary
Character-safe prefix matching rust/lance-index/src/scalar/inverted/index.rs	Fuzzy candidate collection treats prefix_length as a character count and slices query tokens at valid UTF-8 boundaries; a regression test verifies that "über" matches with prefix_length=1.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: hamersaw

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main fix: slicing fuzzy query prefixes on UTF-8 character boundaries.
Description check	✅ Passed	The description is directly about the multibyte fuzzy-search panic and the character-boundary fix.
Linked Issues check	✅ Passed	The change matches #7891 by avoiding byte-based slicing, using char boundaries, and adding a multibyte regression test.
Out of Scope Changes check	✅ Passed	The patch stays focused on the reported fuzzy-prefix panic and its regression test, with no evident unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

rust/lance-index/src/scalar/inverted/index.rs-10807-10858 (1)

10807-10858: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Extend the regression test to validate prefix semantics, not only panic avoidance.

Because the test indexes and queries only "über", it would pass even if prefix filtering were ignored. Add a near-edit-distance token such as "uber" and assert it is excluded with prefix_length = 1; also cover lengths exceeding the token’s character count and the stated "café"/CJK cases. As per coding guidelines, use rstest for the resulting parameterized Rust cases.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-index/src/scalar/inverted/index.rs` around lines 10807 - 10858,
The regression coverage in
test_fuzzy_prefix_length_handles_multibyte_query_token only verifies that “über”
matches; parameterize the test with rstest to validate prefix filtering as well.
Add a near-edit-distance “uber” candidate and assert it is excluded for
prefix_length = 1, and include cases where the prefix length exceeds token
character counts plus the documented “café” and CJK multibyte scenarios,
preserving the expected match/exclusion results for each case.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Other comments:
In `@rust/lance-index/src/scalar/inverted/index.rs`:
- Around line 10807-10858: The regression coverage in
test_fuzzy_prefix_length_handles_multibyte_query_token only verifies that “über”
matches; parameterize the test with rstest to validate prefix filtering as well.
Add a near-edit-distance “uber” candidate and assert it is excluded for
prefix_length = 1, and include cases where the prefix length exceeds token
character counts plus the documented “café” and CJK multibyte scenarios,
preserving the expected match/exclusion results for each case.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 07f90237-f730-494c-8d23-fd7b38bd2606

📥 Commits

Reviewing files that changed from the base of the PR and between f91da0a and 49b8971.

📒 Files selected for processing (1)

• rust/lance-index/src/scalar/inverted/index.rs

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

rust/lance-index/src/scalar/inverted/index.rs-10807-10858 (1)

10807-10858: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Extend the regression test to cover all promised UTF-8 cases.

This verifies the original über panic, but not oversized prefix_length clamping or other multibyte layouts. Add cases such as café with an oversized prefix and CJK text, asserting each expected match. As per coding guidelines, use rstest for Rust parameterized tests.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-index/src/scalar/inverted/index.rs` around lines 10807 - 10858,
Extend test_fuzzy_prefix_length_handles_multibyte_query_token into an
rstest-parameterized regression test covering the existing über case, café with
a prefix_length larger than the token, and a CJK token with a multibyte prefix
boundary. Parameterize the indexed token, query token, prefix length, and
expected row ID, then assert each bm25_search call returns the corresponding
match while preserving the existing setup and regression coverage.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Other comments:
In `@rust/lance-index/src/scalar/inverted/index.rs`:
- Around line 10807-10858: Extend
test_fuzzy_prefix_length_handles_multibyte_query_token into an
rstest-parameterized regression test covering the existing über case, café with
a prefix_length larger than the token, and a CJK token with a multibyte prefix
boundary. Parameterize the indexed token, query token, prefix length, and
expected row ID, then assert each bm25_search call returns the corresponding
match while preserving the existing setup and regression coverage.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 75f8dafb-649a-483a-9f26-9c7ec909d96c

📥 Commits

Reviewing files that changed from the base of the PR and between 49b8971 and bad1b1f.

📒 Files selected for processing (1)

• rust/lance-index/src/scalar/inverted/index.rs