From b83a91669f72889d8bd716233598f902797624b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 05:29:38 +0000 Subject: [PATCH 1/2] README: answer "isn't NFKC enough?" before the API reference, with the measurement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Why this exists" compared this package against npm `itaiji` and stopped there. But `itaiji` is not what a reader tries first — `text.normalize("NFKC")` is. It costs nothing, it is already in the language, and until the README answers it, a reader has no reason to believe a 2.5 MB table is worth taking. The answer is measured against the same denominator as the `itaiji` comparison, so both percentages in the README are over one set: of the 27,661 character-to-character reductions the shipped data carries, NFKC folds 77 (0.3%). NFC folds the same 77 — the compatibility half of NFKC buys nothing here, because 崎 and 﨑 are two characters with two identities rather than a character and a compatibility form of one. The percentage alone would still be dismissible as a rare-character tail, so the seven variants that actually turn up in a name column are checked one at a time (﨑/崎, 髙/高, 邉/辺, 邊/辺, 德/徳, 濵/浜, 栁/柳). Unicode normalization folds none of the seven. Variation selectors get the same treatment: 葛 + U+E0100 is still two code points after NFKC and still unequal to 葛, while toMatchingKey folds both to one key. Also named: `kanji-processor`, the other package npm surfaces for "kanji variant". It is a different job (異体字 → 親字 for Yomitan) built from 三省堂 全訳漢辞海 第四版, its MIT notice covers the code with nothing said about the dictionary data, and 1.0.2 as published does not load on Node.js 22. Naming it and saying what it is beats leaving a reader to assume the field is empty — and the reason the field is nearly empty is worth one sentence too, since it is the same reason this package exists. The new assertions live in test/itaiji-comparison.test.ts rather than a new file: mjNonTrivialPairs() and the 27,661 baseline are already there, and counting the same pairs twice in two files is how two numbers in one document drift apart. Verified the README grep fails when the figure is edited. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KhUyurPQ3BUUQq7tdK1aYT --- CHANGELOG.md | 18 ++++++++++ README.ja.md | 36 +++++++++++++++++++ README.md | 36 +++++++++++++++++++ test/itaiji-comparison.test.ts | 64 +++++++++++++++++++++++++++++++++- 4 files changed, 153 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4862651..2d45c91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,24 @@ Notable changes to this package. This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## Unreleased + +### Documented + +- Both READMEs now answer the question a reader asks before installing + anything: **is `String.normalize("NFKC")` enough?** Measured, not asserted — + of the 27,661 character-to-character reductions in the shipped data, NFKC + folds 77 (0.3%), NFC folds the same 77, and none of the seven surname + variants anyone actually hits (﨑/崎, 髙/高, 邉/辺, 邊/辺, 德/徳, 濵/浜, + 栁/柳) is among them. Variation selectors survive normalization as well. + The computation lives in `test/itaiji-comparison.test.ts` next to the + `itaiji` comparison, sharing its denominator so the two percentages in the + README are over the same set. +- `kanji-processor`, the other package npm surfaces for "kanji variant", is + named and placed: a different job (異体字 → 親字 for Yomitan), from a + commercial dictionary, with the package's MIT notice covering the code and + nothing said about the data. + ## 0.2.0 — 2026-08-17 ### Changed (behaviour) diff --git a/README.ja.md b/README.ja.md index 2db7608..3a3331e 100644 --- a/README.ja.md +++ b/README.ja.md @@ -47,6 +47,42 @@ JIS表現可能なペア(啞→唖、鷗→鴎 等)は、どちらも縮退の 「縮退」の対象にならない。旧字体→新字体の一方向置換が目的なら、この違いが 自分のユースケースに影響するか確認してほしい。 +### `String.normalize()` だけで、どこまで届くか + +npm を引くより先に試されるのは `text.normalize("NFKC")` のほう。無料で、 +標準で入っていて、しかも**本パッケージでも第一段階として実際に使っている** +(`toMatchingKey` は既定で NFKC を通す)。ただし、それ**だけ**では異体字に +ほとんど効かない。 + +本パッケージのデータが持つ文字対文字の縮退 **27,661 組**のうち、NFKC が +畳むのは **77 組——0.3%**。NFC でも同じ 77 組なので、NFKC の「互換」の部分は +ここでは何も買っていない。崎 と 﨑 は、ある文字とその互換形ではなく、 +**別々の同一性を持つ2つの文字**だから。 + +この 0.3% は「畳めないのは稀な字だけ」でもない。名前の列で実際に出会う +異体字——﨑/崎、髙/高、邉/辺、邊/辺、德/徳、濵/浜、栁/柳——について、 +Unicode 正規化は**7組とも畳まない**。本パッケージは7組とも等価として扱う。 + +異体字セレクタも正規化を素通りする。葛 に U+E0100 が続く文字列は NFKC の +あとも2コードポイントのままで、素の 葛 とは一致しない。`toMatchingKey` は +どちらも同じキーに落とす。以上はすべて +[`test/itaiji-comparison.test.ts`](./test/itaiji-comparison.test.ts) で実測している。 + +### npm にあるもう1つのパッケージ + +npm で "kanji variant" を引くと +[`kanji-processor`](https://www.npmjs.com/package/kanji-processor) も出てくる。 +[Yomitan](https://github.com/yomidevs/yomitan) 向けに、三省堂 全訳漢辞海 第四版を +使って異体字→親字を対応づけるもの。**目的も出所も違う**——同梱の `LICENSE` は +コードに対する MIT 表示で、データの元になった辞書については何も書かれていない。 +公開されている 1.0.2 は Node.js 22 で読み込めない(ESM のエントリポイントが +import attribute 無しで JSON を import していて、Node が拒否する)。 + +**この領域にこれだけしか無いのは偶然ではない。** 根拠のある異体字表は記憶から +書き起こせるものではなく、公開データセットを取得して加工する工程が要る。 +存在する2つは、いずれも安いほうの経路——手作りの一覧と、手元にあった辞書——を +通っている。 + ## インストール ```sh diff --git a/README.md b/README.md index e0197fb..50e3d4c 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,42 @@ sides are already independently JIS-representable (e.g. 啞→唖, 鷗→鴎) ar is specifically old-style→new-style kanji substitution, check whether that distinction matters for you. +### What `String.normalize()` gets you on its own + +Most people reach for `text.normalize("NFKC")` before they reach for npm. It is +free, built in, and it is genuinely step one here — `toMatchingKey` runs it by +default. On its own it does almost nothing about kanji variants. + +Of the **27,661** character-to-character reductions in this package's data, +NFKC collapses **77 — 0.3%**. NFC collapses the same 77, so the compatibility +half of NFKC is buying nothing: 崎 and 﨑 are two characters with two +identities, not a character and a compatibility form of one. + +Nor is that 0.3% a tail of rare characters. Of the variants you actually meet +in a name column — 﨑/崎, 髙/高, 邉/辺, 邊/辺, 德/徳, 濵/浜, 栁/柳 — Unicode +normalization folds **none of the seven**, and this package treats all seven as +equivalent. + +Variation selectors survive normalization too: 葛 followed by U+E0100 is still +two code points after NFKC, and still unequal to plain 葛. `toMatchingKey` +folds both to one key. All of this is measured in +[`test/itaiji-comparison.test.ts`](./test/itaiji-comparison.test.ts). + +### The other package on npm + +Searching npm for "kanji variant" also surfaces +[`kanji-processor`](https://www.npmjs.com/package/kanji-processor), which maps +異体字 to 親字 for [Yomitan](https://github.com/yomidevs/yomitan) using +三省堂 全訳漢辞海 第四版. Different job, and different provenance: its `LICENSE` +is an MIT notice for the code, with nothing said about the dictionary the data +comes from. Version 1.0.2 as published also does not load on Node.js 22 — its +ESM entry point imports JSON without an import attribute, which Node rejects. + +That there is so little here is not an accident. A grounded variant table is +not something you can write down from memory: it has to be obtained from a +published dataset and reshaped, and the two packages that exist took the two +cheaper routes — a hand-built list, and a dictionary someone already owned. + ## Install ```sh diff --git a/test/itaiji-comparison.test.ts b/test/itaiji-comparison.test.ts index bf4f4c3..43f35ff 100644 --- a/test/itaiji-comparison.test.ts +++ b/test/itaiji-comparison.test.ts @@ -1,9 +1,12 @@ import { describe, expect, it } from "vitest"; import { readFileSync } from "node:fs"; import { createRequire } from "node:module"; -import { isVariant, reduce } from "../src/index.js"; +import { isVariant, reduce, toMatchingKey } from "../src/index.js"; import { REDUCE_BY_UCS } from "../src/generated/tables.js"; +// このファイルは、両 README が「代わりに何を検討するか」に答えている箇所の +// 数値の出どころ。比較対象は2つある——npm の `itaiji` と、Unicode 正規化。 +// // 両 README は npm の `itaiji` との比較を断定形で載せている——「約19倍の対応数」 // 「1,404組のうち約86%」。出どころは `docs/phase0-report.md`(2026-08-04 の // 実装前調査)で、**再現手段がリポジトリに無かった**。 @@ -39,6 +42,65 @@ function mjNonTrivialPairs(): ReadonlySet { return pairs; } +/** + * 読者が npm を引くより前に試すのは `String.prototype.normalize` のほう。 + * 「まず NFKC を通せば済むのでは」に答えるのがこの比較で、両 README の + * 「Unicode 正規化では届かない範囲」の節がここから数値を取っている。 + * + * 分母は `itaiji` 比較と同じ MJ の非自明な対応 27,661 組。別々に数えると、 + * 同じ文書の中で分母の違う割合が並ぶことになる。 + */ +describe("Unicode 正規化との比較(README の数値の出どころ)", () => { + const MJ_PAIRS = [...mjNonTrivialPairs()].map((pair) => [...pair] as [string, string]); + + it("MJ の対応 27,661 組のうち、NFKC が畳むのは 77 組(0.3%)", () => { + expect(MJ_PAIRS.length).toBe(27_661); + const folded = MJ_PAIRS.filter(([a, b]) => a.normalize("NFKC") === b.normalize("NFKC")); + expect(folded.length).toBe(77); + expect(Math.round((folded.length / MJ_PAIRS.length) * 1000) / 10).toBe(0.3); + expect(README_EN).toContain("27,661"); + expect(README_JA).toContain("27,661"); + }); + + it("NFC でも同じ 77 組。互換分解を足しても漢字の異体字には効かない", () => { + // NFKC は「互換文字を畳む」正規化なので、NFC より広く畳むはずだが、 + // 異体字については差が出ない——効いているのは正規合成のほうだけ、 + // という事実。README がその言い方をしている根拠。 + const nfc = MJ_PAIRS.filter(([a, b]) => a.normalize("NFC") === b.normalize("NFC")); + expect(nfc.length).toBe(77); + }); + + it("名寄せで実際に効いてほしい姓の異体字は、1組も畳まれない", () => { + // 割合だけでは「畳めない 0.3% は稀な字なのだろう」と読めてしまう。 + // README が名指ししている組を1つずつ確かめる。 + const SURNAMES = [ + ["﨑", "崎"], + ["髙", "高"], + ["邉", "辺"], + ["邊", "辺"], + ["德", "徳"], + ["濵", "浜"], + ["栁", "柳"], + ] as const; + for (const [variant, base] of SURNAMES) { + expect(variant.normalize("NFKC"), `NFKC(${variant}) は ${base} にならない`).not.toBe(base); + expect(isVariant(variant, base), `${variant} と ${base} は異体字`).toBe(true); + expect(README_EN).toContain(variant); + expect(README_JA).toContain(variant); + } + }); + + it("NFKC は異体字セレクタを落とさない", () => { + // IVS 付きの字は、NFKC を通しても base とは別の文字列のまま。 + const base = "\u845B"; // 葛 + const withSelector = `${base}\u{E0100}`; + expect(withSelector.normalize("NFKC")).not.toBe(base); + expect([...withSelector.normalize("NFKC")]).toHaveLength(2); + // この package 側は同じキーに落とす。 + expect(toMatchingKey(withSelector).key).toBe(toMatchingKey(base).key); + }); +}); + describe(`npm itaiji ${ITAIJI_VERSION} との比較(README の数値の出どころ)`, () => { it("itaiji の辞書は 1,404 組の 1:1 対応", () => { expect(PAIRS.length).toBe(1_404); From 765492b1f0aea66e14acadb0689cbf304adf7c44 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 05:36:02 +0000 Subject: [PATCH 2/2] Say how 27,661 relates to the 26,668 two paragraphs above it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-review. The new section quotes 27,661 pairs; the paragraph above it quotes 26,668. They are the same set minus the 993 that npm `itaiji` also carries, and a reader with both numbers in view has no way to know that — two close-but-different totals in one section read as one of them being wrong. Both READMEs now say which is which. No figure changed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KhUyurPQ3BUUQq7tdK1aYT --- README.ja.md | 4 ++-- README.md | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/README.ja.md b/README.ja.md index 3a3331e..dbd5f73 100644 --- a/README.ja.md +++ b/README.ja.md @@ -54,8 +54,8 @@ npm を引くより先に試されるのは `text.normalize("NFKC")` のほう (`toMatchingKey` は既定で NFKC を通す)。ただし、それ**だけ**では異体字に ほとんど効かない。 -本パッケージのデータが持つ文字対文字の縮退 **27,661 組**のうち、NFKC が -畳むのは **77 組——0.3%**。NFC でも同じ 77 組なので、NFKC の「互換」の部分は +本パッケージのデータが持つ文字対文字の縮退 **27,661 組**——上の 26,668 と同じ集合で、 +`itaiji` にもある 993 組を引く前の数——のうち、NFKC が畳むのは **77 組——0.3%**。NFC でも同じ 77 組なので、NFKC の「互換」の部分は ここでは何も買っていない。崎 と 﨑 は、ある文字とその互換形ではなく、 **別々の同一性を持つ2つの文字**だから。 diff --git a/README.md b/README.md index 50e3d4c..7ac956a 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,8 @@ Most people reach for `text.normalize("NFKC")` before they reach for npm. It is free, built in, and it is genuinely step one here — `toMatchingKey` runs it by default. On its own it does almost nothing about kanji variants. -Of the **27,661** character-to-character reductions in this package's data, +Of the **27,661** character-to-character reductions in this package's data — +the same set as above, before subtracting the 993 that `itaiji` also has — NFKC collapses **77 — 0.3%**. NFC collapses the same 77, so the compatibility half of NFKC is buying nothing: 崎 and 﨑 are two characters with two identities, not a character and a compatibility form of one.