Skip to content

Improve search quality with BM25 path_terms and word tokenizer #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
soomtong merged 19 commits into from
May 27, 2026
Merged

Improve search quality with BM25 path_terms and word tokenizer #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
b0586ad
Unsafe 제거, 비동기 검색, diff 정렬, Recall@5 수정
soomtong May 26, 2026
6c3de97
Bump version to 0.9.2
soomtong May 26, 2026
91e990c
Fix format
soomtong May 26, 2026
16b2712
Spec: search quality improvements design (BM25 tokenizer, path_terms,…
soomtong May 26, 2026
6cac9b0
Plan: search quality improvements implementation (10 tasks, TDD)
soomtong May 26, 2026
dafe990
Add report of glc semantic search
soomtong May 26, 2026
16c4714
Drop unreachable diff.rs from incremental indexing fixture answer
soomtong May 27, 2026
14222b7
Add text_prep helpers for camelCase split and path tokenization
soomtong May 27, 2026
0a590a5
Add path_terms field and word tokenizer to BM25 schema
soomtong May 27, 2026
d768f45
Write path_terms and camelCase-split title in BM25 add_doc
soomtong May 27, 2026
1eba81f
Include path_terms in query parser with boost for path matching
soomtong May 27, 2026
9995b57
Raise WholeFile threshold to 16KB to keep mid-size files searchable b…
soomtong May 27, 2026
0ce950e
Extract top-level trait_item and type_item as Rust symbols
soomtong May 27, 2026
877a57b
Bump INDEX_VERSION to 6 for path_terms + word tokenizer schema
soomtong May 27, 2026
8d8fe31
Format chunk/file.rs
soomtong May 27, 2026
25ce5e8
Update search quality report after BM25 path_terms and word tokenizer…
soomtong May 27, 2026
60aed77
Add report of glc semantic search
soomtong May 27, 2026
1f32d93
Accept commit.rs as valid hit for revwalk fixture and refresh report
soomtong May 27, 2026
7753ef7
Bump version to 0.9.3
soomtong May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "gluck"
version = "0.9.1"
version = "0.9.3"
edition = "2021"

[[bin]]
Expand All @@ -16,8 +16,8 @@ clap = { version = "4", features = ["derive"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
anyhow = "1"
tree-sitter = "0.22"
tree-sitter-highlight = "0.22"
tree-sitter = "0.23"
tree-sitter-highlight = "0.23"
tree-sitter-rust = "0.23"
tree-sitter-python = "0.23"
tree-sitter-javascript = "0.23"
Expand Down
Loading