Repository navigation
README: 「NFKC で足りるのでは」に、API リファレンスより前に実測で答える - #71
Merged
Merged
Conversation
…e measurement
"Why this exists" compared this package against npm `itaiji` and stopped
there. But `itaiji` is not what a reader tries first — `text.normalize("NFKC")`
is. It costs nothing, it is already in the language, and until the README
answers it, a reader has no reason to believe a 2.5 MB table is worth taking.
The answer is measured against the same denominator as the `itaiji`
comparison, so both percentages in the README are over one set: of the 27,661
character-to-character reductions the shipped data carries, NFKC folds 77
(0.3%). NFC folds the same 77 — the compatibility half of NFKC buys nothing
here, because 崎 and 﨑 are two characters with two identities rather than a
character and a compatibility form of one.
The percentage alone would still be dismissible as a rare-character tail, so
the seven variants that actually turn up in a name column are checked one at a
time (﨑/崎, 髙/高, 邉/辺, 邊/辺, 德/徳, 濵/浜, 栁/柳). Unicode normalization
folds none of the seven. Variation selectors get the same treatment: 葛 +
U+E0100 is still two code points after NFKC and still unequal to 葛, while
toMatchingKey folds both to one key.
Also named: `kanji-processor`, the other package npm surfaces for "kanji
variant". It is a different job (異体字 → 親字 for Yomitan) built from
三省堂 全訳漢辞海 第四版, its MIT notice covers the code with nothing said
about the dictionary data, and 1.0.2 as published does not load on Node.js 22.
Naming it and saying what it is beats leaving a reader to assume the field is
empty — and the reason the field is nearly empty is worth one sentence too,
since it is the same reason this package exists.
The new assertions live in test/itaiji-comparison.test.ts rather than a new
file: mjNonTrivialPairs() and the 27,661 baseline are already there, and
counting the same pairs twice in two files is how two numbers in one document
drift apart. Verified the README grep fails when the figure is edited.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KhUyurPQ3BUUQq7tdK1aYT
Self-review. The new section quotes 27,661 pairs; the paragraph above it quotes 26,668. They are the same set minus the 993 that npm `itaiji` also carries, and a reader with both numbers in view has no way to know that — two close-but-different totals in one section read as one of them being wrong. Both READMEs now say which is which. No figure changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KhUyurPQ3BUUQq7tdK1aYT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
「なぜ作ったか」は npm
itaijiとの比較で終わっていた。しかし読者が最初に試すのはitaijiではなくtext.normalize("NFKC")のほう。無料で、言語に最初から入っていて、そこに答えが無いかぎり「2.5MB の表を持ち込む理由」が読者側に立たない。
追加した内容と、その裏づけ
1. Unicode 正規化との比較(実測)
分母は
itaiji比較と同じ MJ の非自明な対応 27,661 組(同じ文書の中で分母の違う割合が並ばないように、意図的に揃えてある)。
NFKC の「互換」の部分がここでは何も買っていないことになる——崎 と 﨑 は、ある文字と
その互換形ではなく、別々の同一性を持つ2つの文字なので。
割合だけでは「畳めないのは稀な字だろう」と読めてしまうので、名前の列で実際に
出会う7組を1つずつ検証している: 﨑/崎、髙/高、邉/辺、邊/辺、德/徳、濵/浜、栁/柳。
Unicode 正規化は7組とも畳まない。
異体字セレクタも同様。葛 + U+E0100 は NFKC のあとも2コードポイントで、素の 葛 とは
一致しない(
toMatchingKeyはどちらも同じキーに落とす)。2.
kanji-processorを名指ししたnpm で "kanji variant" を引くともう1つ出てくる。目的が違い(Yomitan 向けの
異体字→親字)、三省堂 全訳漢辞海 第四版から作られていて、同梱
LICENSEはコードに対する MIT 表示でデータについては何も書かれていない。公開されている 1.0.2 は
Node.js 22 で読み込めない(ESM が import attribute 無しで JSON を import している)。
「この領域には何も無い」と読者に推測させるより、あるものを名指しして位置づけるほうが
正しい。 そのうえで「なぜこれだけしか無いのか」(根拠のあるデータの取得と加工が要る)
を1文だけ足した。
テストの置き場所
新規ファイルにせず
test/itaiji-comparison.test.tsに describe を1つ足した。mjNonTrivialPairs()と 27,661 の基準値がすでにそこにあり、同じ組を2ファイルで別々に数えるのは、1つの文書の中の2つの数値がずれていく経路そのものなので。
検査
npm test139 passed /npm run typecheck緑🤖 Generated with Claude Code
Generated by Claude Code