[frankslin]

Conversion::AppendConverted and MaxMatchSegmentation advance one character
at a time and pay a trie prefix lookup per position, even for characters
that cannot possibly begin a dictionary key. PrefixMatch now builds a
key-start table at construction time and exposes SkipUnmatchable(), letting
both hot loops consume whole runs of unmatchable characters — ASCII,
punctuation, emoji, and CJK characters that start no key — with a single
scan and one append.

Filtering is by exact code point rather than by UTF-8 lead byte: the common
CJK lead bytes 0xE4..0xE9 all begin some key, so lead-byte filtering skips
only ~10% of characters in typical Traditional-Chinese input, while 69-76%
of positions match no key at all. Character-granularity filtering skips
60-67% of prefix lookups.

Conversion output is byte-for-byte unchanged, enforced by a differential
fuzz test against a per-character reference loop and verified across 9
configs on a corpus with 3 MB of random bytes prepended to 22 MB of Chinese
text. Whole-run CLI times (22 MB Traditional corpus, dictionary load
included):

  config   before   after
  t2s       3.40s   0.52s
  t2tw      3.43s   0.30s
  s2twp     8.40s   4.86s
  t2s (4 MB mixed Chinese/ASCII)
            0.83s   0.11s

Startup is unaffected: single-character input takes the same time as before
(s2twp 5.01s vs 4.87s over 20 runs), and max RSS and page fault counts are
unchanged — .ocd2 dictionaries are fread into a heap buffer at load, so the
skip-table trie walk touches only already-resident pages.

Detailed Changes:

• Utf8SkipScan.hpp (new private header, not installed):
  • Utf8SkipTable: per-byte candidate table of key start bytes, plus an
    optional per-code-point bitmap over the BMP (8 KiB). 2- and 3-byte
    characters are filtered by exact code point; 4-byte characters keep
    lead-byte filtering. Character-level mode is on exactly when the bitmap
    is non-empty — no separate flag to keep in sync.
  • DecodeCodePoint23(): the single decoder shared by the table builder and
    the scanner, so a key's first character always maps to the bit the
    scanner tests. It never validates continuation bytes, and the scan
    consumes the same byte count as the per-character path, keeping
    behavior equivalent on malformed input.
  • Finalize(): marks the ideographic description operators and their lead
    bytes from UTF8Util::IdeographicDescriptionOperatorArity() over the
    operator range, so the conversion loop's IDS grouping is preserved;
    non-operator characters sharing the 0xE2 lead byte (em dashes, curly
    quotes) stay skippable. A static_assert pins the operator range to the
    3-byte UTF-8 block.
  • AsciiRunLength(): word-at-a-time (SWAR) scan of the leading ASCII run,
    in plain C++ with no intrinsics and no architecture or endianness
    special cases. SSE2/NEON/WASM-SIMD kernels were written and measured
    against it and won by at most 3% end to end, inside run-to-run noise on
    half the cases — the candidate table has already removed the
    per-character trie lookups that dominated, so widening the scan from 8
    to 16 bytes per step no longer moves the total. simdutf was considered
    and rejected: it exposes no API for scanning against a caller-supplied
    candidate table, which is where the speedup actually comes from.
• MarisaDict::EnumerateKeys():
  • New non-virtual method that invokes a callback per key by walking the
    trie directly, reusing the lexicon when already materialized so lazy
    loading is never forced. Deliberately not a Dict virtual: Dict and
    MarisaDict are both OPENCC_EXPORT and installed, so adding a vtable
    slot would require an ABI bump; the skip-table builder dynamic_casts
    instead, matching how SupportsFastPrefixMatch() already special-cases
    MarisaDict/DartsDict. OPENCC_ABI_VERSION is unchanged.
  • Keys may not be NUL-terminated; the skip-table builder reads only
    within the reported length.
• PrefixMatch:
  • SkipUnmatchable() returns how many leading bytes may be consumed
    without any prefix lookup. The skip table lives in the opaque Tables
    (no layout or header change for PrefixMatch itself) and is built for
    both the single-dictionary fast path and the table path; both go
    through the existing Tables cache, with fast-path entries marked so a
    matcher-less Tables can never be handed to the table path.
  • MarkKeyFirstChar() marks each key's exact first code point. A key with
    an invalid lead byte only marks its lead (the scanner already stops
    there); a key whose first character is truncated disables
    character-level filtering for the whole table, since its raw byte
    prefix cannot be represented as a code point.
  • CollectSkipTable() recurses into groups via the existing
    GetDictGroupItems() virtual and falls back to GetLexicon() for
    non-Marisa leaves.
• Conversion / MaxMatchSegmentation:
  • After a failed match, extend the unmatched advance over the following
    unmatchable run in one step instead of one lookup per character.
• Tests:
  • Differential fuzz (300 random inputs per path mixing dictionary keys,
    CJK, IDS operators, and raw invalid/truncated bytes) comparing
    Conversion::Convert with a per-character reference loop on both the
    single-dictionary fast path and the table path.
  • An invariant test that the scan never crosses any dictionary key's
    first character; a pin that every recognized IDS operator stops the
    scan; character-level cases for characters sharing a lead byte with
    keys, 0xE2-lead punctuation vs IDS operators, and 2-byte characters.
  • AsciiRunLength boundary/offset sweep; ASCII-candidate, invalid,
    truncated, and 4-byte cases; mixed ASCII/Chinese conversion and
    segmentation cases.

[frankslin]

Benchmark before: (https://github.com/BYVoid/OpenCC/actions/runs/29971656400/job/89094753296)

Benchmark after: (https://github.com/BYVoid/OpenCC/actions/runs/30123499046/job/89581405003?pr=1464)