Repository navigation
F1: mixed Chinese/English typing without switching (rule-based) - #2
Merged
Merged
Conversation
Adds Source/Engine/MixedScript/ (LatinLexicon, BopomofoShapeTracker, MixedScriptTracker, LatinPassthroughLM) implementing the three-rule decision from zhuyin-ime-personal.md's P1 design: rule A converts a structurally-impossible Bopomofo run straight to literal English; rule B registers a dictionary-word run as a low-scored alternate candidate at its Chinese reading; rule C promotes that alternate to the active value when followed by a space/end-of-input. mixedScriptEnabled defaults on and is a no-op path when off, leaving existing Chinese typing byte-for-byte unchanged (verified via ctest and eval200 --mixed off). Wires the engine into KeyHandler.mm (minimal hooks around the existing BPMF key path, backspace, Esc, Tab/candidate selection), McBopomofoLM (merges LatinPassthroughLM into unigram dispatch), LanguageModelManager (loads the bundled + user word lists), and Preferences (mixedScriptEnabled, mixedScriptLatinOnSpace). Adds a curated tech-term seed list and a build_lexicon.py generator over macOS's system word list (see ACKNOWLEDGEMENTS.md for its license/provenance). Extends the eval harness (mixime-eval --mixed on|off, run_eval.py --mixed) to drive the same decision engine for parity between the CLI and the app. On the private eval200 corpus: F1 token retention 0.0% -> 95.6% (target >=70%), F2 char accuracy unchanged at 95.3% (target >=94.8%), latency ~2.6ms/sentence avg (target <5ms). 32 new gtest cases; 156/156 total pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Round 2 on f1-mixed-script, addressing every blocking finding in
docs/REVIEW-P1-2026-09-10.md. The branch previously cost 5.1 points of
pure-Chinese accuracy, silently swallowed whole sentences, and was on by
default with four upstream tests red.
Rules
- Drop the automatic "dictionary word + space => English" default. On the
standard layout space *is* the tone-1 key, so that rule and ordinary
Chinese input are the same gesture: "up " stopped producing 因, "fu/ "
stopped producing 清, and the non-letter keys inside those syllables
were dropped outright. A trailing space now promotes a run to English
only when the word is in the user's own lexicon -- a word they picked by
hand before. Everything else stays Chinese with the English form as a
candidate. kMinAmbiguousWordLength 2 -> 3; the preference is renamed
MixedScriptLatinOnSpaceForUserWords to match what it now controls.
- Score the rule-B Latin alternate just under the reading's top unigram
instead of a fixed -99, so it is the candidate window's second row and
the first Tab reaches it (it used to be last of 39-72). Also stop
registering an alternate for runs that are not dictionary words at all.
- Only register a rule-B alternate as "learnable": an explicit pick now
has to match an alternate this composition registered, not merely be a
candidate that happens to be spelled with ASCII letters.
KeyHandler
- A space that ends a Latin run inserts a real space instead of falling
into the candidate-window branch, where every following letter was
swallowed ("acer api gmail" committed "acer"). A letter typed into a
candidate window now cancels it and keeps typing rather than beeping
the key away, and Enter still commits after flushing a run.
- Esc clears the reading buffer as well as the tracker; backspacing a
locked run to empty releases the lock; force-commit no longer drops a
pending run; buildInputtingState is preference-gated like every other
mixedScript path; a tone key straight after English no longer parks an
orphan tone mark; charCode is range-checked before being truncated to
char, so a non-ASCII character whose low byte lands in a-z is not
treated as that letter and the arrow keys reach the run-boundary branch
on purpose rather than by coincidence.
Loading and persistence
- The 204k-word list loads on a background queue and is published to the
key thread when ready (it was adding ~150 ms to every activateServer:).
A null lexicon simply means the dictionary rules do not apply yet.
- isPrefix()'s index is kept ordered as the lists load instead of being
sorted lazily on first use, which cost 294 ms on the key thread.
- rememberWord() reports a failed write, creates the user data folder,
and re-resolves the path so it follows useCustomUserPhraseLocation.
Default and measurement
- mixedScriptEnabled defaults to false; see README for the opt-in, and
for the note that builds before today wrote `true` into the plist.
- New McBopomofoTests/MixedScriptKeyHandlerTests.swift: 33 tests, one per
reproduction sequence in the review plus Esc/backspace/arrow-key/
force-commit/two-words/long-run/Shift coverage, and
testEval200ThroughKeyHandler, which types the whole corpus into a real
KeyHandler. That is now the acceptance measurement; tools/eval is
relabelled engine-only. App path: F1 349/385 = 90.6%, and pure-Chinese
accuracy 4369/4586 = 95.3% with the feature on -- identical to the
baseline and to the feature off, i.e. no regression at all.
- Engine tests: 176 pass (was 156), including the two mutations that
survived the review's mutation run.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e is on Follow-up to the round-2 fixes. The word lists are loaded off the key thread now, but they were still loaded for everyone: KeyHandler's initializer asked LanguageModelManager for the lexicon, and +loadDataModels warmed it unconditionally. With the preference defaulting to off, that is ~20 MB and a background read nobody asked for. The tracker now starts with no lexicon and is handed one on the first key actually handled with mixed typing enabled, which is also what starts the load. handleForceCommit's new pending-run check is gated on the preference and input mode for the same reason as every other path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…is not a choice Re-verify (docs/REVERIFY-P1-2026-09-10.md) found the one remaining blocker: BopomofoShapeTracker assumes one key -> one Bopomofo component, which is only true for the standard layout; on 許氏/倚天26 the feature turned whole Chinese sentences into raw letters (0.3% char accuracy). Every mixedScript code path now goes through -[KeyHandler _mixedScriptAvailable], which also requires Preferences.keyboardLayout == standard. Test drives four non-standard layouts ON vs OFF and requires identical output; verified red with the gate removed. R5: a "_latin_" node has a single candidate, so Tab-cycling over it no longer counts as picking English and no longer appends the word to latin-user.txt. 160 XCTests, 176 engine tests. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lmanchu
pushed a commit
that referenced
this pull request
Sep 11, 2026
…eval, isolation)
1. Rebuild Source/Data/latin-words.txt entirely from SCOWL/ESDB (size<=70,
American spelling, variant level <=1) instead of macOS's 1934 web2 word
list with SCOWL only as a frequency-tier annotation. web2 had no
inflected forms at all; the new list carries them directly and every
word now has a rank tier (was ~39%). tools/lexicon/build_lexicon.py
rewritten to build both the word set and its tiers from a local SCOWL
repo checkout in one pass; tools/lexicon/scowl-size-tiers.tsv (the old
annotation-only file) is removed. 141,697 words (was 204,217): 80,067
added, 142,587 removed, 61,630 kept -- ACKNOWLEDGEMENTS.md and
README.markdown updated to match. P1 regression: F1 app-path retention
holds at 349/385 = 90.6%, pure-Chinese ON=OFF still identical character
for character; a sweep of every tone-1 single-syllable reading in
data.txt found exactly 2 (of 138 with a key sequence >= 3 letters)
changed Rule-B ambiguity status ("ful"/ㄑㄧㄠ newly ambiguous,
"tji"/ㄔㄨㄛ no longer is) -- neither is used by any existing test, and
Rule B never overrides the default Chinese candidate on its own.
2. Learn from Latin runs the user actually types and commits (Enter/
space/punctuation), not only from an explicit Tab/candidate-window
completion accept -- new Preferences.latinLearnTypedWords (default
on). KeyHandler's _commitMixedScriptLatinRun: gains a mayLearn flag so
Tab landing there (no completion was on offer) never counts as
evidence the user meant to finish the word, and
MixedScriptTracker::latinRunAlreadyRemembered() prevents double-
counting a run a completion accept already wrote. Esc/Backspace never
reach the commit path, so a canceled run is never learned.
3. Tab (and the completion tooltip/Shift+Tab candidate window) must not
grow an already-finished word into a longer dictionary entry sharing
its prefix -- new KeyHandler _isAlreadyCompleteWord:lexicon: gate
(word is itself a known word whose rank is at least as good as the
best longer completion), applied at the pending-run, committed-node,
and tooltip call sites. "acer" (tech-seed) stays "acer", not
"acerbic"; "the" stays "the". Rule-B alternates are exempt so e.g.
"coo" can still complete into "cook" on a second Tab.
testR5_TabOnARuleARunDoesNotWriteTheUserLexicon restored to its
original semantics; testTabDoesNotExtendACompleteWord added.
4. LatinCompletionKeyHandlerTests' eval now classifies "never completable"
tokens into proper-noun/abbreviation, inflected-form, and other, and
adds a "with history" pass that replays eval200 in corpus order,
teaching each row's eligible words (mirroring fix #2) before scoring
the next row. Inflected-form count is 0 in both passes, confirming
fix #1's dictionary switch. BASELINE.md's P3 section replaced with
both tables.
5. KeyHandlerBopomofoTests (upstream, mixedScript-unaware) now forces
mixedScriptEnabled/latinCompletionEnabled/latinLearnTypedWords off in
setUp and restores the real values in tearDown, so it no longer
inherits whatever this machine's live plist happens to have (it was
failing testInvalidBpmf whenever MixedScriptEnabled=1 on disk). Full
suite verified green with the real pref both true and false, and the
real plist ends each run unchanged.
6. New testing-only LanguageModelManager.resetLatinLexiconForTesting()
(and LatinLexicon::reset()) reloads the process-wide Latin lexicon
fresh before every KeyHandler-level test, fixing the cross-test
pollution from REVERIFY-P1-2026-09-10.md's R12 (an accepted word in
one test used to silently change another test's top completion).
Waits for any load already in flight before resetting to avoid a
data race with the test host app's own startup load. Word-choice
workaround comments from the pollution era removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lmanchu
pushed a commit
that referenced
this pull request
Sep 12, 2026
…eval, isolation)
1. Rebuild Source/Data/latin-words.txt entirely from SCOWL/ESDB (size<=70,
American spelling, variant level <=1) instead of macOS's 1934 web2 word
list with SCOWL only as a frequency-tier annotation. web2 had no
inflected forms at all; the new list carries them directly and every
word now has a rank tier (was ~39%). tools/lexicon/build_lexicon.py
rewritten to build both the word set and its tiers from a local SCOWL
repo checkout in one pass; tools/lexicon/scowl-size-tiers.tsv (the old
annotation-only file) is removed. 141,697 words (was 204,217): 80,067
added, 142,587 removed, 61,630 kept -- ACKNOWLEDGEMENTS.md and
README.markdown updated to match. P1 regression: F1 app-path retention
holds at 349/385 = 90.6%, pure-Chinese ON=OFF still identical character
for character; a sweep of every tone-1 single-syllable reading in
data.txt found exactly 2 (of 138 with a key sequence >= 3 letters)
changed Rule-B ambiguity status ("ful"/ㄑㄧㄠ newly ambiguous,
"tji"/ㄔㄨㄛ no longer is) -- neither is used by any existing test, and
Rule B never overrides the default Chinese candidate on its own.
2. Learn from Latin runs the user actually types and commits (Enter/
space/punctuation), not only from an explicit Tab/candidate-window
completion accept -- new Preferences.latinLearnTypedWords (default
on). KeyHandler's _commitMixedScriptLatinRun: gains a mayLearn flag so
Tab landing there (no completion was on offer) never counts as
evidence the user meant to finish the word, and
MixedScriptTracker::latinRunAlreadyRemembered() prevents double-
counting a run a completion accept already wrote. Esc/Backspace never
reach the commit path, so a canceled run is never learned.
3. Tab (and the completion tooltip/Shift+Tab candidate window) must not
grow an already-finished word into a longer dictionary entry sharing
its prefix -- new KeyHandler _isAlreadyCompleteWord:lexicon: gate
(word is itself a known word whose rank is at least as good as the
best longer completion), applied at the pending-run, committed-node,
and tooltip call sites. "acer" (tech-seed) stays "acer", not
"acerbic"; "the" stays "the". Rule-B alternates are exempt so e.g.
"coo" can still complete into "cook" on a second Tab.
testR5_TabOnARuleARunDoesNotWriteTheUserLexicon restored to its
original semantics; testTabDoesNotExtendACompleteWord added.
4. LatinCompletionKeyHandlerTests' eval now classifies "never completable"
tokens into proper-noun/abbreviation, inflected-form, and other, and
adds a "with history" pass that replays eval200 in corpus order,
teaching each row's eligible words (mirroring fix #2) before scoring
the next row. Inflected-form count is 0 in both passes, confirming
fix #1's dictionary switch. BASELINE.md's P3 section replaced with
both tables.
5. KeyHandlerBopomofoTests (upstream, mixedScript-unaware) now forces
mixedScriptEnabled/latinCompletionEnabled/latinLearnTypedWords off in
setUp and restores the real values in tearDown, so it no longer
inherits whatever this machine's live plist happens to have (it was
failing testInvalidBpmf whenever MixedScriptEnabled=1 on disk). Full
suite verified green with the real pref both true and false, and the
real plist ends each run unchanged.
6. New testing-only LanguageModelManager.resetLatinLexiconForTesting()
(and LatinLexicon::reset()) reloads the process-wide Latin lexicon
fresh before every KeyHandler-level test, fixing the cross-test
pollution from REVERIFY-P1-2026-09-10.md's R12 (an accepted word in
one test used to silently change another test's top completion).
Waits for any load already in flight before resetting to avoid a
data race with the test host app's own startup load. Word-choice
workaround comments from the pollution era removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P1 of the plan, after three rounds (implementation → fresh-context review → fixes → re-verify → gate fix).
A new engine module
Source/Engine/MixedScript/(Latin lexicon, Bopomofo-shape tracker, decision rules,LatinPassthroughLM) plus hooks inKeyHandler.mm. Letters whose Bopomofo shape is structurally dead stay English (rule A — covers 88.8% of English tokens in the corpus, 0 false positives across every single-syllable reading indata.txt); a dictionary word that is also a valid tone-1 syllable stays Chinese (space is the tone-1 key) with the English form at candidate row 2 and Tab flipping; only words the user has explicitly picked before auto-commit as English on space. Everything is behindMixedScriptEnabled(default off — dogfood withdefaults write org.openvanilla.inputmethod.McBopomofo MixedScriptEnabled -bool true) and only active on the standard (大千) layout.Numbers on the private 200-sentence corpus, through the real
KeyHandler(tools/eval/BASELINE.md, "app path"):Reviews:
docs/REVIEW-P1-2026-09-10.md(7 blocking — the first round's CLI numbers did not describe the app; rule C conflicted with tone-1 typing; 4 upstream tests red) → all closed in round 2;docs/REVERIFY-P1-2026-09-10.md(B1–B7 confirmed closed, mutations 0/6 survived; one new blocking: no keyboard-layout gate, 26-key layouts turned Chinese into raw letters) → closed in round 3 with a test that drives four non-standard layouts ON vs OFF and is red without the gate.Tests: 160 XCTest (upstream 125 + 35 KeyHandler-level mixed-script tests, incl.
testEval200ThroughKeyHandler), 176 engine tests. Not installed on this machine (by decision).Known non-blocking items carried to P4 (from the re-verify): a lowercase letter dismisses any candidate window (upstream beeps); two spaces after an English word open a candidate window; picking English via Tab is also learned by the UserOverrideModel so
MixedScriptLatinOnSpaceForUserWords=falsecannot fully switch auto-English off;xcodebuild testrewritesBASELINE.mdlatency numbers; rule B in practice governs only ~15 dictionary words (docs should say so).🤖 Generated with Claude Code