F3: English prediction + Tab completion, SCOWL lexicon, learning from typed words

ACKNOWLEDGEMENTS.md

@@ -9,11 +9,48 @@
 專案的[國字讀音整理檔](https://github.com/ButTaiwan/bpmfvs/tree/master/phonetic)，
 該專案作者為 But Ko，採 Apache License v2.0 發布。
 
-`Source/Data/latin-words.txt`（P1 中英混打詞典，見
-`tools/lexicon/build_lexicon.py`）由 macOS 內建 `/usr/share/dict/words`
-（symlink 至 `web2`）過濾產生。該檔案源自 Webster's Second International
-Dictionary（1934 年版權已過期／公有領域，依 FreeBSD 隨附之
-`/usr/share/dict/README` 所載），並由 FreeBSD 專案整理維護、隨 macOS base
-system 附帶散布。`Source/Data/latin-tech-seed.txt` 為本專案原創整理的科技
-詞彙補充清單，非衍生自任何第三方詞庫。
+`Source/Data/latin-words.txt`（中英混打詞典本體＋詞頻分級，見
+`tools/lexicon/build_lexicon.py`）2026-09-11 起完全衍生自 **SCOWL /
+ESDB**（English Speller Database，原名 Spell Checker Originated Word Lists；
+https://github.com/en-wl/wordlist ，作者 Kevin Atkinson）size ≤70 的
+American-English、variant level ≤1 詞表累積輸出（`./scowl --db scowl.db
+word-list <35|40|50|60|70> A 1 --categories=`），只取「一個詞第一次出現在
+哪個 size 桶」當粗粒度常用度分級，不含 ESDB 資料庫本身或其原始碼。
+
+此前（P1／P3 第一輪，2026-09-09～2026-09-11）詞表本體來自 macOS 內建
+`/usr/share/dict/words`（symlink 至 `web2`，源自 Webster's Second
+International Dictionary，1934 年版權已過期／公有領域，依 FreeBSD 隨附之
+`/usr/share/dict/README` 所載，由 FreeBSD 專案整理維護、隨 macOS base
+system 附帶散布），只拿 SCOWL 的 size 分桶做詞頻分級（`scowl-size-tiers.tsv`
+標註檔）。web2 本身完全沒有字詞變化形（games／typing／comments 等一律缺席，
+只有其字典形存在），2026-09-11 已停用，不再是本專案任何檔案的來源；
+`scowl-size-tiers.tsv` 這個獨立標註檔也隨之移除（詞頻分級現在是產生詞表
+本體時直接算好、寫進 `latin-words.txt` 的欄位，不再需要另一個檔案）。
+
+`Source/Data/latin-tech-seed.txt` 為本專案原創整理的科技詞彙補充清單，非衍
+生自任何第三方詞庫。
+
+ESDB 專案 `Copyright` 檔的授權原文（適用於本檔案性質的「由資料庫產生的詞
+表」）：
+
+> Permission to use, copy, modify, distribute, and sell any part of the
+> English Speller Database (ESDB, previously known as SCOWLv2), or word
+> lists created from it, is hereby granted without fee, provided that the
+> above copyright notice appears in all copies and that both the above
+> copyright notice and this notice appear in supporting documentation.
+> Kevin Atkinson makes no representations about the suitability of this
+> database for any purpose. It is provided "as is" without express or
+> implied warranty.
+>
+> Copyright 2000-2026 by Kevin Atkinson
+
+（ESDB 本身彙整自多個以 Public Domain 為主的來源，含 12dicts／ENABLE2K；本
+專案只使用到 size ≤70、American 拼法、variant-level 1 的一般詞表，未使用
+Australian 或 UKACD 相關內容，故上述通用授權即已足夠，不觸發 `Copyright`
+檔案中 `=== AU` / `=== UKACD` 段落的額外條件。）
+
+`first20hours/google-10000-english`（Google Trillion Word Corpus 衍生的
+頻率詞表）**已查證但未採用**：其 `LICENSE.md` 將底層語料綁定 LDC
+（Linguistic Data Consortium）授權與研究／合理使用範圍，並明文「不建議
+商業用途」，不符合本專案「授權明確可再散布」的門檻，故未使用。
 

AGENTS.md

@@ -58,6 +58,62 @@ xcodebuild -project McBopomofo.xcodeproj -target Data -configuration Debug build
 - Framework: XCTest with Swift `Testing` module
 - Run in Xcode with ⌘+U or test navigator
 
+**Run the suite serially. `-parallel-testing-enabled YES` does not work
+and will report failures that have nothing to do with your change.**
+
+The reason is not fixable from inside a test: `Preferences` reads and
+writes `UserDefaults.standard`, i.e. the real
+`org.openvanilla.inputmethod.McBopomofo` domain, which is one per-user
+file shared by *every* parallel test-runner process. `PreferencesTests`
+removes every key in that domain in its initializer and asserts on
+default values, while the KeyHandler suites are simultaneously setting
+`MixedScriptEnabled`, `LatinCompletionEnabled` and
+`LatinLearnTypedWords` from other processes -- so each suite sees the
+others' writes. (The process-wide `LatinLexicon` and
+`resetLatinLexiconForTesting()`'s wait were the two suspects in
+`docs/REVIEW-P3-2026-09-11.md`'s N15; both are per-process and neither
+is the cause.) Making this work means giving `Preferences`' property
+wrappers a test-specific `UserDefaults` suite -- each wrapper already
+has a `container` property, so it is a change of defaults plus a place
+to set them from, not a rewrite, but it has not been done.
+
+What *has* been fixed is the part that damaged real data rather than
+just reporting noise. A parallel run used to leave 130 eval-corpus words
+in `~/Library/Application Support/McBopomofo/latin-user.txt` -- the
+installed input method's own word list -- because the KeyHandler suites
+redirected their user-data folder by writing
+`CustomUserPhraseLocation`, and one worker clearing that key while
+another resolved it landed the second worker's writes in the default
+folder (`docs/REVERIFY-P3-2026-09-12.md`'s P-2). Tests now redirect it
+with `LanguageModelManager.dataFolderOverrideForTesting`, which is
+process-local and cannot be raced.
+
+**Preferences are the developer's own live input-method settings.** Any
+test that touches `Preferences` must call
+`PreferenceSandbox.install(on: self)` as the first statement of
+`setUpWithError()` -- writing saved values back in `tearDownWithError` is
+not enough, because it both creates keys that were never in the file and
+is skipped entirely when a test fails. Anything that needs the
+*user-data folder* must set
+`LanguageModelManager.dataFolderOverrideForTesting` in `setUpWithError()`
+and clear it in a teardown block; never write
+`UseCustomUserPhraseLocation` / `CustomUserPhraseLocation` from a test.
+Note also that the test host is the McBopomofo app itself, so
+`main.swift` runs before any test does; its `Preferences.populateDefaults()`
+is skipped under XCTest (`Preferences.isRunningUnderXCTest`) because
+those writes land earlier than `PreferenceSandbox` can snapshot them.
+Verify all of it with:
+
+```bash
+tools/eval/check_plist_unchanged.sh \
+  xcodebuild -project McBopomofo.xcodeproj -scheme McBopomofo \
+    -configuration Debug -derivedDataPath build \
+    CODE_SIGN_IDENTITY="-" CODE_SIGNING_REQUIRED=NO DEVELOPMENT_TEAM="" test
+```
+
+which fails if the run changed a single preference key, or created or
+modified `~/Library/Application Support/McBopomofo/`.
+
 #### C++ Engine Tests
 ```bash
 cd Source/Engine

McBopomofo.xcodeproj/project.pbxproj

@@ -62,6 +62,8 @@
 		D4314F0D2ED3690F0071DD71 /* NumberInputHelper.swift in Sources */ = {isa = PBXBuildFile; fileRef = D4314F0C2ED369090071DD71 /* NumberInputHelper.swift */; };
 		D436E23F2CD20BA700C6155D /* AssociatedPhrasesTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = D436E23E2CD20B9E00C6155D /* AssociatedPhrasesTests.swift */; };
 		D4E5C211AAA0020000000000 /* MixedScriptKeyHandlerTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = D4E5C211AAA0010000000000 /* MixedScriptKeyHandlerTests.swift */; };
+		D4E5C212AAA0020000000000 /* LatinCompletionKeyHandlerTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = D4E5C212AAA0010000000000 /* LatinCompletionKeyHandlerTests.swift */; };
+		D4E5C212AAB0020000000000 /* PreferenceSandbox.swift in Sources */ = {isa = PBXBuildFile; fileRef = D4E5C212AAB0010000000000 /* PreferenceSandbox.swift */; };
 		D43737C92DF9C35800D9707C /* InputMethodController+KeyHandlerDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = D43737C82DF9C35100D9707C /* InputMethodController+KeyHandlerDelegate.swift */; };
 		D43737CB2DF9C48300D9707C /* InputMethodController+CandidateControllerDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = D43737CA2DF9C48200D9707C /* InputMethodController+CandidateControllerDelegate.swift */; };
 		D43FC40B2B23788400ED5A1C /* InputMacro.swift in Sources */ = {isa = PBXBuildFile; fileRef = D43FC40A2B23788400ED5A1C /* InputMacro.swift */; };
@@ -218,6 +220,8 @@
 		D4314F0C2ED369090071DD71 /* NumberInputHelper.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = NumberInputHelper.swift; sourceTree = "<group>"; };
 		D436E23E2CD20B9E00C6155D /* AssociatedPhrasesTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AssociatedPhrasesTests.swift; sourceTree = "<group>"; };
 		D4E5C211AAA0010000000000 /* MixedScriptKeyHandlerTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = MixedScriptKeyHandlerTests.swift; sourceTree = "<group>"; };
+		D4E5C212AAA0010000000000 /* LatinCompletionKeyHandlerTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = LatinCompletionKeyHandlerTests.swift; sourceTree = "<group>"; };
+		D4E5C212AAB0010000000000 /* PreferenceSandbox.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PreferenceSandbox.swift; sourceTree = "<group>"; };
 		D43737C82DF9C35100D9707C /* InputMethodController+KeyHandlerDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = "InputMethodController+KeyHandlerDelegate.swift"; sourceTree = "<group>"; };
 		D43737CA2DF9C48200D9707C /* InputMethodController+CandidateControllerDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = "InputMethodController+CandidateControllerDelegate.swift"; sourceTree = "<group>"; };
 		D43FC40A2B23788400ED5A1C /* InputMacro.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = InputMacro.swift; sourceTree = "<group>"; };
@@ -553,6 +557,8 @@
 			children = (
 				D436E23E2CD20B9E00C6155D /* AssociatedPhrasesTests.swift */,
 				D4E5C211AAA0010000000000 /* MixedScriptKeyHandlerTests.swift */,
+				D4E5C212AAA0010000000000 /* LatinCompletionKeyHandlerTests.swift */,
+				D4E5C212AAB0010000000000 /* PreferenceSandbox.swift */,
 				D47D73A327A5D43900255A50 /* KeyHandlerBopomofoTests.swift */,
 				D4A8E43527A9E982002F7A07 /* KeyHandlerPlainBopomofoTests.swift */,
 				D485D3B82796A8A000657FF3 /* PreferencesTests.swift */,
@@ -827,6 +833,8 @@
 				D4A8E43627A9E982002F7A07 /* KeyHandlerPlainBopomofoTests.swift in Sources */,
 				D436E23F2CD20BA700C6155D /* AssociatedPhrasesTests.swift in Sources */,
 				D4E5C211AAA0020000000000 /* MixedScriptKeyHandlerTests.swift in Sources */,
+				D4E5C212AAA0020000000000 /* LatinCompletionKeyHandlerTests.swift in Sources */,
+				D4E5C212AAB0020000000000 /* PreferenceSandbox.swift in Sources */,
 				D449AD5F2B393C00000C5812 /* InputMacroTests.swift in Sources */,
 				D4C2A9872B4309D700113711 /* UTF8HelperTest.mm in Sources */,
 				D47D73A427A5D43900255A50 /* KeyHandlerBopomofoTests.swift in Sources */,

McBopomofoTests/KeyHandlerBopomofoTests.swift

@@ -34,22 +34,38 @@ func charCode(_ string: String) -> UInt16 {
 class KeyHandlerBopomofoTests: XCTestCase {
 
     var handler = KeyHandler()
-    var savedKeyboardLayout: KeyboardLayout = .standard
-    var chineseConversionEnabled: Bool = false
-
+    // P3 fix #5 (XCTest/machine-preference isolation, see
+    // ~/.claude/plans/zhuyin-ime-personal.md's P3 fix #5): these upstream
+    // tests know nothing about mixedScript and never save/restore its
+    // preferences themselves, so on a machine that already has
+    // MixedScriptEnabled=1 in its real plist (Lman dogfoods a master
+    // build) they would otherwise inherit that value and see mixedScript
+    // rewrite readings it never used to (e.g. testInvalidBpmf's "ni4"
+    // becomes a Rule-A literal run instead of the upstream
+    // EmptyIgnoringPreviousState). Forced off here regardless of the
+    // machine's real value, which PreferenceSandbox puts back --
+    // this never touches the plist's own final on-disk state.
     override func setUpWithError() throws {
-        savedKeyboardLayout = Preferences.keyboardLayout
-        chineseConversionEnabled = Preferences.chineseConversionEnabled
+        // Must come before the first Preferences write, and is the only
+        // restore mechanism in this class -- see
+        // LatinCompletionKeyHandlerTests.setUpWithError and
+        // docs/REVIEW-P3-2026-09-11.md's N4 for why writing saved values
+        // back in tearDownWithError was not enough.
+        PreferenceSandbox.install(on: self)
+
         Preferences.chineseConversionEnabled = false
         Preferences.keyboardLayout = .standard
+        Preferences.mixedScriptEnabled = false
+        Preferences.latinCompletionEnabled = false
+        Preferences.latinLearnTypedWords = false
         LanguageModelManager.loadDataModels()
         handler = KeyHandler()
         handler.inputMode = .bopomofo
     }
 
     override func tearDownWithError() throws {
-        Preferences.chineseConversionEnabled = chineseConversionEnabled
-        Preferences.keyboardLayout = savedKeyboardLayout
+        // Preferences are restored by PreferenceSandbox, not here -- see
+        // setUpWithError.
     }
 
     func testSyncWithPreferences() {

McBopomofoTests/KeyHandlerPlainBopomofoTests.swift

@@ -27,24 +27,24 @@ import XCTest
 
 class KeyHandlerPlainBopomofoTests: XCTestCase {
 
-    var savedKeyboardLayout: KeyboardLayout = .standard
     var handler = KeyHandler()
 
     override func setUpWithError() throws {
+        // Must come before the first Preferences write, and is the only
+        // restore mechanism in this class -- see
+        // LatinCompletionKeyHandlerTests.setUpWithError and
+        // docs/REVIEW-P3-2026-09-11.md's N4 for why writing saved values
+        // back in tearDownWithError was not enough.
+        PreferenceSandbox.install(on: self)
+
         LanguageModelManager.loadDataModels()
         handler = KeyHandler()
         handler.inputMode = .plainBopomofo
 
-        savedKeyboardLayout = Preferences.keyboardLayout
-
         // Punctuation-related tests only work when the layout is Standard.
         Preferences.keyboardLayout = KeyboardLayout.standard
     }
 
-    override func tearDownWithError() throws {
-        Preferences.keyboardLayout = savedKeyboardLayout
-    }
-
     // Regression test for #292.
     func testUppercaseLetterWhenEmpty() {
         let tmp = Preferences.letterBehavior