Skip to content

Add classical-context phrases to STPhrases for one-to-many s2t characters - #1479

Merged
frankslin merged 1 commit into
BYVoid:masterfrom
frankslin:upstream-master
Aug 19, 2026
Merged

frankslin merged 1 commit into
BYVoid:masterfrom
frankslin:upstream-master

Conversation

@frankslin

Copy link
Copy Markdown
Collaborator

Book titles from pre-modern Chinese corpora daizhige.org were mis-converted because characters such as 征/咸/斗/里/范/岳/冲/郁/谷 fall back to STCharacters, whose first candidate is chosen for modern usage. Adding phrase entries fixes 46 of 47 wrong titles found in the 殆知閣 collection without changing any modern-sense conversion.

Detailed Changes:

  • STPhrases.txt:
    • Added conquest senses of 征 (征南/征东/征西/征北/征缅/征台/征乌梁海/从征实录/从征缅).
    • Added reign titles and adverbial senses of 咸 (咸淳/咸平/咸通/咸康/咸熙/咸雍/咸同/咸宾/少长咸集/咸有一德).
    • Added constellation senses of 斗 (斗姆/斗母/四斗/贯斗/说东斗/说西斗/东斗星君/西斗星君).
    • Added place names and transliterations for 里 (里乘/步里客谈/碧里杂存/甫里/锦里/花里活/摩里支/矩里迦/村屯里数).
    • Added surname senses of 范 and 岳 (范文正/范太史/范忠宣/范德机/范叔/范氏/范公/岳武穆/岳王/岳王庙/景岳).
    • Added Daoist senses of 冲 (道冲/圣诞冲举/冲碧丹经) and Sanskrit transliterations of 郁 (郁迦罗越/郁多罗).
    • Changed 谷神 to keep 谷 (《老子》「谷神不死」); 谷神星 still maps to 穀神星 via the longer key.
    • Added 干禄, 姜斋, 泛胜之, 授时历, 大统历, 历指, 历引, 沈馆录, 石屋余沈, 石屋续沈, 警寤钟, 薛仁贵.
  • testcases.json:
    • Added eight s2t cases covering each character group.
    • Added a guard case asserting modern senses stay unchanged (股東鬥毆, 服從徵召, 散步裏面, 學歷指標, 示範公司, 鹹魚鹹菜).

…ters

Book titles from pre-modern Chinese corpora were mis-converted because
characters such as 征/咸/斗/里/范/岳/冲/郁/谷 fall back to STCharacters,
whose first candidate is chosen for modern usage. Adding phrase entries
fixes 46 of 47 wrong titles found in the 殆知閣 collection without
changing any modern-sense conversion.

Detailed Changes:
- **STPhrases.txt**:
  - Added conquest senses of 征 (征南/征东/征西/征北/征缅/征台/征乌梁海/从征实录/从征缅).
  - Added reign titles and adverbial senses of 咸 (咸淳/咸平/咸通/咸康/咸熙/咸雍/咸同/咸宾/少长咸集/咸有一德).
  - Added constellation senses of 斗 (斗姆/斗母/四斗/贯斗/说东斗/说西斗/东斗星君/西斗星君).
  - Added place names and transliterations for 里 (里乘/步里客谈/碧里杂存/甫里/锦里/花里活/摩里支/矩里迦/村屯里数).
  - Added surname senses of 范 and 岳 (范文正/范太史/范忠宣/范德机/范叔/范氏/范公/岳武穆/岳王/岳王庙/景岳).
  - Added Daoist senses of 冲 (道冲/圣诞冲举/冲碧丹经) and Sanskrit transliterations of 郁 (郁迦罗越/郁多罗).
  - Changed 谷神 to keep 谷 (《老子》「谷神不死」); 谷神星 still maps to 穀神星 via the longer key.
  - Added 干禄, 姜斋, 泛胜之, 授时历, 大统历, 历指, 历引, 沈馆录, 石屋余沈, 石屋续沈, 警寤钟, 薛仁贵.
- **testcases.json**:
  - Added eight s2t cases covering each character group.
  - Added a guard case asserting modern senses stay unchanged (股東鬥毆, 服從徵召, 散步裏面, 學歷指標, 示範公司, 鹹魚鹹菜).
@frankslin

Copy link
Copy Markdown
Collaborator Author

這批詞條的來歷

本 PR 的詞條不是憑印象挑的,而是來自一份外部語料庫產生的「已標註錯誤集」。

背景。 daizhige.org(殆知閣)是一個古籍語料庫,每筆記錄的
metadata 同時帶 zh-hans 與 zh-hant 兩個篇名欄位,其中 zh-hant 最初是由簡體篇名
逐字自動轉換而來。全庫 15,317 筆記錄中,95% 的 zh-hant 與現行 OpenCC s2t 輸出
完全一致
——換句話說,這份語料等同於一組針對古籍文本的 s2t 盲測集;而編者後來
手工訂正過 zh-hant 的地方,正好就是自動轉換出錯的地方。

錯誤集。 殆知閣近期一個 commit 訂正了 56 條這類篇名(去重後 52 條)。把這些簡體
篇名重新餵給 opencc -c s2t,52 條中有 47 條重現了錯誤。而且全部敗在同一個機制上:
STPhrases 沒有詞條命中,於是逐字回退到 STCharacters,而它的首選項是按現代漢語
語感排的:

冲	衝 沖      范	範 范
咸	鹹 咸      表	表 錶
岳	嶽 岳      谷	谷 穀
征	徵 征      郁	鬱 郁
斗	鬥 斗      里	裏 里 哩

這些首選項對現代文本是對的(鹹魚、鬥毆、徵收、範例),不應更動。修正該落在
STPhrases,也就是本 PR 所做的事。

擴大掃描。 沒有止步於這 56 條,而是把全庫 18,056 條不重複的簡體篇名全部轉換後與
zh-hant 比對。為了把真正的一簡多繁選字錯誤,從雜訊(zh-hant 根本沒轉完的篇名,
或本庫自身偏好的異體如 滙/誌/唸/註)中分離出來,只有當語料庫用的那個字本身就
列在該簡體字於 STCharacters.txt 的候選項之中
時,才計為一次錯誤。這樣才浮出本 PR
中額外的幾條:干祿字書、薑齋詩話、氾勝之書、瀋館錄、石屋餘瀋、授時曆、大統曆,
以及《崇禎曆書》的曆指/曆引各卷篇名——加入前均已比對維基百科/維基文庫/ctext/
中央研究院數位典藏核實。

詞條在回歸測試後做過收窄

初稿為每個義項取了最短的自然鍵。用現代語句探測後發現有幾個會跨詞邊界誤命中,遂改為
較長且無歧義的鍵:

初稿 誤轉 實際採用
东斗 / 西斗 股東斗毆、房東斗地主、向東斗爭 说东斗 / 说西斗 / 东斗星君 / 西斗星君
从征 服從征召、遵從征稅 从征实录 / 从征缅
步里 / 碧里 散步里面、碧里透紅 步里客谈 / 碧里杂存
冲举 / 冲碧 向前沖舉起手、汽車沖碧海 圣诞冲举 / 冲碧丹经
沈故 / 余沈 瀋故意的、餘瀋默不語 石屋余沈 / 石屋续沈
咸集 咸集團 少长咸集

薛仁贵 薛仁貴 是因為另一個原因加的:既有詞條 贵征 貴徵 會搶先匹配,導致
薛仁贵征东 一直轉成 薛仁貴徵東,直到補上這個較長的姓名鍵才解決。

有兩條提案完全撤下:虫部 與 谷部。《本草綱目》需要蟲部/穀部,但《說文解字》
《康熙字典》的部首名需要虫部/谷部——沒有任何鍵能區分兩者。《瀋故》基於同樣理由撤下
(二字書名,與姓氏「沈」直接衝突)。

驗證

  1. 目標集——殆知閣那 52 條篇名:47 條錯 → 1 條錯。僅存的一條是《唐宋元名表》;
    名表 名錶 對現代漢語是正確的,也找不到安全的更長鍵。
  2. 全庫回歸——18,056 條篇名分別用新舊詞典轉換:85 條輸出改變,全部是修正,
    零回歸。
    除了已回報的那批,還順帶修好了 大宋中興通俗演義岳王傳、大統曆志、
    石屋續瀋、金華沖碧丹經祕旨傳、《干祿字書》全部七個分卷,以及《崇禎曆書》
    所有曆指/曆引篇名。
  3. 現代語感護欄——54 條刻意構造、讓每個新詞條落在現代義項上的句子(股東鬥毆、
    服從徵召、散步裏面、學歷指標、廣泛勝出、示範公司、模範氏族、主幹祿口機場、
    生薑齋飯……)轉換全部正確。精簡版已固化為 classical_entries_modern_sense_guard_s2t
    測試案例。
  4. 測試套件——bazel test //data/... //test/...:82 tests pass,含
    dictionary_STPhrases_test、command_line_converter_test(讀取新增的 9 個案例)
    與 golden_convert_test。

回饋給語料庫

這次掃描也找出兩條是殆知閣自己錯、而非 OpenCC 錯的篇名——大宋中興通俗演義嶽王傳
(岳王即岳飛,姓氏)與 大統歷志(曆法;四庫本正文作「大統厯志」)。兩條均已在
來源語料庫訂正。

@frankslin
frankslin merged commit 3acca98 into BYVoid:master Aug 19, 2026
40 checks passed
@frankslin
frankslin deleted the upstream-master branch August 19, 2026 02:22
@frankslin frankslin mentioned this pull request Aug 22, 2026
frankslin added a commit that referenced this pull request Aug 22, 2026
Bump every version string to 1.4.2, finalize the NEWS entry, and adjust
the Debian packaging so the release build stamps the new version. The
C++ ABI is unchanged from 1.4.1, so OPENCC_ABI_VERSION stays at 1.4.

Detailed Changes:
- **Version strings**:
  - `cmake/GitVersion.cmake`: `_OPENCC_FALLBACK_REVISION` 1 -> 2.
  - `scripts/compute-version.sh`: `FALLBACK_VERSION` 1.2.1 -> 1.4.2; it
    had drifted out of sync with `cmake/GitVersion.cmake` since 1.2.1.
  - `setup.py`, `MODULE.bazel`, `node/BUILD.bazel`,
    `scripts/build-cli-windows-zig.sh`,
    `scripts/build-node-windows-zig.sh`: bumped to 1.4.2.
  - `package.json` and `plugins/jieba/node/package.json`: version and the
    five `@opencc/*` optional dependencies bumped to 1.4.2.
  - `package-lock.json`: version bumped and the `node_modules/@opencc/*`
    entries dropped; their integrity hashes are regenerated after the
    scoped binary packages are published, as in 1.4.1.
- **NEWS.md**:
  - `Unreleased` becomes `Version 1.4.2` with the release date.
  - Added the STPhrases classical-context entries for one-to-many s2t
    characters found while cleaning up the Daizhige corpus (#1479).
  - Noted the fallback-version bump and the resulting Debian patch drop.
- **README.md**: prebuilt binary links and the release-focus paragraph
  updated to 1.4.2.
- **Debian packaging**:
  - Added a `1.4.2-1` changelog entry; `release-deb.yml` names its
    artifacts from `dpkg-parsechangelog`, which would otherwise still
    report `1.4.1+ds1-7`.
  - Dropped `0010-fix-git-version-fallback.patch`: it moves the fallback
    revision from 0 to 1, which neither applies nor reverse-applies once
    upstream ships 2, and would abort `scripts/build-deb-packages.sh`.
  - `packaging/debian/README.md`: refreshed the baseline, patch list and
    refresh instructions to describe the two deltas above.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant