Repository navigation
Eval harness: headless engine CLI, 200-sentence zh/en corpus tooling, baseline numbers - #1
Merged
Merged
Conversation
Source/Engine/CMakeLists.txt referenced GTest::gtest_main but never called
find_package/FetchContent when configured standalone (cmake -S Source/Engine),
only Source/CMakeLists.txt did. Move GTest resolution (prefer brew-installed
GTest via find_package(GTest CONFIG), fall back to FetchContent) above the
add_subdirectory(gramambular2)/add_subdirectory(Mandarin) calls so their
GTest::gtest_main links resolve too. No test code changed; all 124 engine
tests still pass.
Add tools/eval/mixime-eval, a headless CLI that drives the real engine
(BopomofoReadingBuffer + ReadingGrid + McBopomofoLM) for eval scripts,
mirroring KeyHandler.mm's isValidKey->combineKey->composeReading->
insertReading->walk order rather than reimplementing engine logic:
- keys: raw ASCII standard-layout key sequence -> composed text
- readings: space-separated Bopomofo readings -> composed text (for F2)
- keyseq: CJK sentence -> readings + standard-layout keys, via
McBopomofoLM::getReading() (longest match, no keyboard map reimplemented)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build_corpus.py filters ~/Dev/mixime-private/corpus_candidates.txt (private,
mostly AI-generated summaries/Slack text) down to 200 short, zh/en-mixed
chat-style sentences, normalizes each to CJK+ASCII-letters+spaces only,
segments into zh/en spans, and gets each zh segment's Bopomofo readings and
standard-layout keys from `mixime-eval --mode keyseq` (no keyboard map or
dictionary reimplemented). The private corpus never enters this repo
(.gitignore's *.private.* rule); only a synthetic-only 10-row sample lives
at tools/eval/fixtures/sample10.tsv.
run_eval.py measures two baseline metrics against the stock engine:
F1 (English segment retention via mode `keys`): 0/385 tokens = 0.0%,
confirming the engine has no English-awareness yet (P1's gap to close).
F2 (homophone/candidate accuracy via mode `readings`): 4369/4586 = 95.3%,
the unmodified Viterbi walk's accuracy given correct readings.
Both average ~2.2-2.5ms/sentence. Full numbers, commands, and data
provenance (sha256) in tools/eval/BASELINE.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Same numbers within normal timing jitter (F1 0.0%, F2 95.3%); re-ran as part of verifying the P0.5 acceptance commands end to end. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Claude Code Review Failed The automated Claude review encountered an error and could not complete. You can:
This does not affect the PR approval process. |
lmanchu
pushed a commit
that referenced
this pull request
Sep 11, 2026
…eval, isolation)
1. Rebuild Source/Data/latin-words.txt entirely from SCOWL/ESDB (size<=70,
American spelling, variant level <=1) instead of macOS's 1934 web2 word
list with SCOWL only as a frequency-tier annotation. web2 had no
inflected forms at all; the new list carries them directly and every
word now has a rank tier (was ~39%). tools/lexicon/build_lexicon.py
rewritten to build both the word set and its tiers from a local SCOWL
repo checkout in one pass; tools/lexicon/scowl-size-tiers.tsv (the old
annotation-only file) is removed. 141,697 words (was 204,217): 80,067
added, 142,587 removed, 61,630 kept -- ACKNOWLEDGEMENTS.md and
README.markdown updated to match. P1 regression: F1 app-path retention
holds at 349/385 = 90.6%, pure-Chinese ON=OFF still identical character
for character; a sweep of every tone-1 single-syllable reading in
data.txt found exactly 2 (of 138 with a key sequence >= 3 letters)
changed Rule-B ambiguity status ("ful"/ㄑㄧㄠ newly ambiguous,
"tji"/ㄔㄨㄛ no longer is) -- neither is used by any existing test, and
Rule B never overrides the default Chinese candidate on its own.
2. Learn from Latin runs the user actually types and commits (Enter/
space/punctuation), not only from an explicit Tab/candidate-window
completion accept -- new Preferences.latinLearnTypedWords (default
on). KeyHandler's _commitMixedScriptLatinRun: gains a mayLearn flag so
Tab landing there (no completion was on offer) never counts as
evidence the user meant to finish the word, and
MixedScriptTracker::latinRunAlreadyRemembered() prevents double-
counting a run a completion accept already wrote. Esc/Backspace never
reach the commit path, so a canceled run is never learned.
3. Tab (and the completion tooltip/Shift+Tab candidate window) must not
grow an already-finished word into a longer dictionary entry sharing
its prefix -- new KeyHandler _isAlreadyCompleteWord:lexicon: gate
(word is itself a known word whose rank is at least as good as the
best longer completion), applied at the pending-run, committed-node,
and tooltip call sites. "acer" (tech-seed) stays "acer", not
"acerbic"; "the" stays "the". Rule-B alternates are exempt so e.g.
"coo" can still complete into "cook" on a second Tab.
testR5_TabOnARuleARunDoesNotWriteTheUserLexicon restored to its
original semantics; testTabDoesNotExtendACompleteWord added.
4. LatinCompletionKeyHandlerTests' eval now classifies "never completable"
tokens into proper-noun/abbreviation, inflected-form, and other, and
adds a "with history" pass that replays eval200 in corpus order,
teaching each row's eligible words (mirroring fix #2) before scoring
the next row. Inflected-form count is 0 in both passes, confirming
fix #1's dictionary switch. BASELINE.md's P3 section replaced with
both tables.
5. KeyHandlerBopomofoTests (upstream, mixedScript-unaware) now forces
mixedScriptEnabled/latinCompletionEnabled/latinLearnTypedWords off in
setUp and restores the real values in tearDown, so it no longer
inherits whatever this machine's live plist happens to have (it was
failing testInvalidBpmf whenever MixedScriptEnabled=1 on disk). Full
suite verified green with the real pref both true and false, and the
real plist ends each run unchanged.
6. New testing-only LanguageModelManager.resetLatinLexiconForTesting()
(and LatinLexicon::reset()) reloads the process-wide Latin lexicon
fresh before every KeyHandler-level test, fixing the cross-test
pollution from REVERIFY-P1-2026-09-10.md's R12 (an accepted word in
one test used to silently change another test's top completion).
Waits for any load already in flight before resetting to avoid a
data race with the test host app's own startup load. Word-choice
workaround comments from the pollution era removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lmanchu
pushed a commit
that referenced
this pull request
Sep 12, 2026
…eval, isolation)
1. Rebuild Source/Data/latin-words.txt entirely from SCOWL/ESDB (size<=70,
American spelling, variant level <=1) instead of macOS's 1934 web2 word
list with SCOWL only as a frequency-tier annotation. web2 had no
inflected forms at all; the new list carries them directly and every
word now has a rank tier (was ~39%). tools/lexicon/build_lexicon.py
rewritten to build both the word set and its tiers from a local SCOWL
repo checkout in one pass; tools/lexicon/scowl-size-tiers.tsv (the old
annotation-only file) is removed. 141,697 words (was 204,217): 80,067
added, 142,587 removed, 61,630 kept -- ACKNOWLEDGEMENTS.md and
README.markdown updated to match. P1 regression: F1 app-path retention
holds at 349/385 = 90.6%, pure-Chinese ON=OFF still identical character
for character; a sweep of every tone-1 single-syllable reading in
data.txt found exactly 2 (of 138 with a key sequence >= 3 letters)
changed Rule-B ambiguity status ("ful"/ㄑㄧㄠ newly ambiguous,
"tji"/ㄔㄨㄛ no longer is) -- neither is used by any existing test, and
Rule B never overrides the default Chinese candidate on its own.
2. Learn from Latin runs the user actually types and commits (Enter/
space/punctuation), not only from an explicit Tab/candidate-window
completion accept -- new Preferences.latinLearnTypedWords (default
on). KeyHandler's _commitMixedScriptLatinRun: gains a mayLearn flag so
Tab landing there (no completion was on offer) never counts as
evidence the user meant to finish the word, and
MixedScriptTracker::latinRunAlreadyRemembered() prevents double-
counting a run a completion accept already wrote. Esc/Backspace never
reach the commit path, so a canceled run is never learned.
3. Tab (and the completion tooltip/Shift+Tab candidate window) must not
grow an already-finished word into a longer dictionary entry sharing
its prefix -- new KeyHandler _isAlreadyCompleteWord:lexicon: gate
(word is itself a known word whose rank is at least as good as the
best longer completion), applied at the pending-run, committed-node,
and tooltip call sites. "acer" (tech-seed) stays "acer", not
"acerbic"; "the" stays "the". Rule-B alternates are exempt so e.g.
"coo" can still complete into "cook" on a second Tab.
testR5_TabOnARuleARunDoesNotWriteTheUserLexicon restored to its
original semantics; testTabDoesNotExtendACompleteWord added.
4. LatinCompletionKeyHandlerTests' eval now classifies "never completable"
tokens into proper-noun/abbreviation, inflected-form, and other, and
adds a "with history" pass that replays eval200 in corpus order,
teaching each row's eligible words (mirroring fix #2) before scoring
the next row. Inflected-form count is 0 in both passes, confirming
fix #1's dictionary switch. BASELINE.md's P3 section replaced with
both tables.
5. KeyHandlerBopomofoTests (upstream, mixedScript-unaware) now forces
mixedScriptEnabled/latinCompletionEnabled/latinLearnTypedWords off in
setUp and restores the real values in tearDown, so it no longer
inherits whatever this machine's live plist happens to have (it was
failing testInvalidBpmf whenever MixedScriptEnabled=1 on disk). Full
suite verified green with the real pref both true and false, and the
real plist ends each run unchanged.
6. New testing-only LanguageModelManager.resetLatinLexiconForTesting()
(and LatinLexicon::reset()) reloads the process-wide Latin lexicon
fresh before every KeyHandler-level test, fixing the cross-test
pollution from REVERIFY-P1-2026-09-10.md's R12 (an accepted word in
one test used to silently change another test's top completion).
Waits for any load already in flight before resetting to avoid a
data race with the test host app's own startup load. Word-choice
workaround comments from the pollution era removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P0.5 of the plan: measure before building.
Source/Engine/CMakeLists.txt: standalonecmake -S Source/Enginenow resolves brew GoogleTest (find_package(GTest CONFIG), FetchContent fallback); 124 engine tests pass headlessly.tools/eval/mixime-eval: C++ CLI over the real engine —keys(standard-layout key sequence → composed sentence, mirrors KeyHandler's compose order),readings(Bopomofo readings → sentence),keyseq(Chinese sentence → readings + key sequence via the layout's ownkeySequenceFromSyllable).tools/eval/build_corpus.py+run_eval.py: build a private 200-sentence mixed zh/en corpus (kept outside the repo; only a 10-row synthetic fixture is committed) and report F1 (English retention), F2 (homophone accuracy), latency.Baseline (upstream f5ba010, standard layout): F1 0.0% (by design — every letter is a valid Bopomofo key), F2 95.3% char-level, ~2.4 ms/sentence. See
tools/eval/BASELINE.md.Finding worth knowing before P1: tone 1 has no key, so a tone-1 syllable only composes on space/Enter; the harness emits a space after every syllable.
🤖 Generated with Claude Code