feat(core): make the FTS5 tokenizer configurable (CJK recall)

[Cyb3rN8]

Closes #9 — but as an opt-in rather than a default change, since switching every user to trigram would be a regression for Latin-script transcripts.

Problem

messages_fts is created without a tokenize option (packages/core/src/schema.sql:37), so FTS5 falls back to unicode61, which indexes an unbroken run of CJK as one token. The repro from #9:

CREATE VIRTUAL TABLE t USING fts5(c);  -- default unicode61, same as messages_fts
INSERT INTO t VALUES ('步2 重跑gen-itgc后50条悬空引用清零，修复已验证');
SELECT count(*) FROM t WHERE t MATCH '悬空';   -- 0
SELECT count(*) FROM t WHERE t MATCH 'itgc';   -- 0  (glued to CJK)

One correction to the issue, from measuring a real index rather than the minimal repro: CJK phrases are not uniformly zero-hit. A phrase matches when it happens to be delimited by punctuation or ASCII, so the real failure mode is silent partial recall, which is worse than an obvious zero. Against 379,877 messages (2.5 GB of transcripts), with LIKE as ground truth:

query	LIKE	unicode61	trigram
额度调整链	12	7	12
所有权制度	21	4	21
itgc (glued to CJK)	368	361	368

Approach

OBELISK_FTS_TOKENIZER selects any built-in FTS5 tokenizer. Unset — the default — nothing changes at all.

schema.sql is deliberately untouched. Because it is CREATE VIRTUAL TABLE IF NOT EXISTS, an existing FTS table keeps its original tokenizer forever, so switching means dropping and repopulating. The migration rewrites the tokenize clause of the table's own sqlite_master definition instead of restating the columns, so schema.sql stays the single source of truth and this keeps working if the FTS columns ever change. Triggers live on the content tables and survive the drop.

It runs alongside migrateCoreSchemaColumns at all three call sites (core openDb, app migrateDb, app installSchema), so the CLI and the desktop app converge on the same tokenizer.

Two details worth review:

• The build debounce yields to a pending switch. Only the build path opens a write connection, so a newly configured tokenizer would otherwise stay silently inactive for up to 30s — or indefinitely while the app holds write ownership. daemon_active still wins, since the app runs the same migration itself.
• No transaction around drop/create/rebuild, on purpose: if it is interrupted, the next schema.sql pass recreates the table and the migration runs again. Wrapping it would add a long write transaction for no additional safety.

The env value is interpolated into DDL, so it is whitelisted against ^(unicode61|ascii|porter|trigram)( [A-Za-z0-9_=]+)*$ and throws on anything else.

Trade-offs (measured, not estimated)

On the same 379,877-message index:

	unicode61	trigram
full build	95 s	113 s
whole database	817 MB	945 MB (+15.7%)
messages_fts inverted index	28 MB	151 MB (5.4x)
one-time migration	—	~35 s

The inverted index grows 5.4x, but only +15.7% of the database, since messages.text is 30 MB of an 817 MB file dominated by tool_results.

The real cost is the 3-character floor: trigram cannot match a shorter query and returns zero without an error, which is a sharp edge for 2-character CJK words and for ok / id-style ASCII. That is precisely why this is opt-in. It is documented in the README next to the setting, pointing at sql() + LIKE for short queries.

Verification

• New tests/fts-tokenizer.test.mjs (10 tests): whitelist rejection, no-op when unset, both tables switching, rows written before the switch becoming searchable after the rebuild, insert/update/delete triggers still feeding the rebuilt table, idempotence, reversibility, the debounce interaction, and daemon precedence.
• Full suite, typecheck, lint, build:core, build:cli: no regression. Pre-existing failures on my machine (better-sqlite3 cannot build on Node 26) are identical with and without this change — I could not exercise the app runtime path locally, only typecheck it, so that call site deserves a second look.
• End-to-end against the real 2.5 GB transcript corpus in an isolated HOME: default build stays unicode61; setting the variable migrates both tables and every query above goes from partial to exact agreement with LIKE; switching back to unicode61 and forward to trigram again both work, the second one inside the debounce window.

Not included

The skill still tells agents to translate the request into English topic terms first, which is the right default for unicode61 but leaves recall on the table once trigram is on. Happy to follow up there, and on surfacing this in the app's settings UI, if the approach looks right.

🤖 Generated with Claude Code