Conversation
…ction tool Collapse the three flat tool registrations into a single sema(action=search|stats|reindex) tool. The execute and render logic of the three former tools is kept verbatim; only the tool surface changes, so three per-round schema payloads become one. The open output schema root records the producing sub-tool via __act so render dispatches; an explicit additionalProperties value is required by the dsh schema compiler. 【中文】将 sema_search/sema_reindex/sema_stats 三件平铺工具聚合为 sema(action=search|stats|reindex) 单件。三件原工具的 execute 与 render 逻辑逐字保留，仅收敛工具面——每轮注入 schema 从三件降为 一件。输出 schema 根保持开放（additionalProperties 显式声明—— dsh schema 编译器拒省略态），内部 __act 字段记录产出子工具供 render 分发。
The 16000-char default assumes roughly 0.25 tokens per character, which holds for English but not for CJK text (close to one token per character). A default-sized CJK chunk exceeds embedding endpoints' per-item token limit, every batch is rejected with HTTP 400, and the index silently degrades to the local lexical provider. Lower the default to 3000 chars, which keeps CJK chunks within typical per-item limits at a small cost for ASCII-heavy corpora. 【中文】默认 maxCharsPerText=16000 按英文 token 比标定；中文接近 1 字≈1 token，默认尺寸中文 chunk 超出 embedding 端点单条 token 上限——整批 400、allowFallback 下静默永久降级为本地 lexical。 默认降为 3000 字符：中文安全，英文语料损失小。
|
The default 3000-character limit is acceptable for the DSH-first baseline because it reduces embedding endpoint failures, but it must not be a hard-coded universal limit. Please add a configurable provider option such as |
Per review: the 3000-char cap stays the CJK-safe default but is no longer a hard-coded universal limit. It is now a first-class provider option — declared in the Config schema, resolved with the same clamp-and-reject contract as provider.timeoutMs (100..16000), and threaded through ResolvedProviderConfig into createProvider. ASCII-heavy corpora can raise it without touching code. Test covers default, override, and out-of-range rejection. 按评审意见：3000 字符限保留为 CJK 安全缺省，但不再是硬编码通用限 ——升为 provider 一等可配项（Config schema 声明、与 timeoutMs 同 clamp 拒绝契约 100..16000、经 ResolvedProviderConfig 透传至 createProvider）。ASCII 重语料可直接调高。测试覆盖缺省/覆盖/超界 拒绝三态。
|
Addressed in 1e90766: |
2 participants
Lower default maxCharsPerText for CJK corpora
Base: on top of pr-1 · stacked series: pr-1 → this PR → pr-4 → pr-5 → pr-3
Motivation
createProviderhardcodesmaxCharsPerText: 16000, calibrated for English (~0.25 tokens per character ≈ 4000 tokens). CJK text runs close to one token per character, so a 16000-char CJK chunk is ~16000 tokens — far beyond any common per-item token limit.Measured against one OpenAI-compatible endpoint (
embedding-3-class): a 6900-char CJK chunk already fails withHTTP 400 / error code 1210("invalid parameter"); 16000 ASCII chars (~4000 tokens) pass. WithallowFallback(the default), the consequence is silent: every embed call fails, the index degrades to the local lexical provider, andsema(action=stats)keeps reportingdegraded: truewith no visible error trail.What changes
One line plus rationale: the factory default becomes
maxCharsPerText: 3000. CJK chunks stay within typical per-item token limits; ASCII-heavy corpora lose at most ~12000 chars of tail context per chunk (~4000 tokens), which the chunker's line-based splitting already mitigates.Notes for reviewers
If you would rather keep 16000 for ASCII-first deployments, the natural extension is a
provider.maxCharsPerTextconfig knob with a CJK-safe default; happy to rework in that direction.Testing
tsc --noEmitclean; full suite 80/80 pass (existing provider tests construct explicitmaxCharsPerText, so they pin behavior independently of the factory default).