Read the Script property for every character, and match HarfBuzz's default cluster level

[albedosehen]

Closes #288

Four commits, one per audit item. Each one passes the core and C API tests on its own.

1. N3: default cluster level (src/buffer/flags.rs). ClusterLevel::default(), and so a new Buffer, is now MonotoneGraphemes. HarfBuzz defaults to this level, and so does the C API (hb_buffer_create already used HB_BUFFER_CLUSTER_LEVEL_MONOTONE_GRAPHEMES, so the C API is unchanged). Tests that read clusters at the default level now expect HarfBuzz's default-level output. docs/STABILITY.md says so.
2. A13: Script property (src/unicode/mod.rs, script_table.rs, tests/unicode_table_gen.rs). script_of returns the bucket of each character's Unicode Script property, from the same Scripts.txt 18.0.0 snapshot that already fed the table.
   • The old block arms are gone, and so is TABLE_SCRIPTS. The generator maps every ISO code to its bucket, with Hira and Kana folding into Han. Common, Inherited, Unknown and the scripts without a bucket map to Other.
   • Script runs and the guessed script skip Unknown characters (unassigned and private use) as well as Common and Inherited ones, as hb_buffer_guess_segment_properties does.
   • The hand-listed ASCII and Latin-1 ranges are dropped, so U+00AA and U+00BA are Latin letters. The hand-listed combining-mark blocks stay.
   • A kana segment picks kana by the Script property instead of by block.
   • The implicit vertical layout for Mongolian (a sigilbuzz default when no direction is set; HarfBuzz has none) now applies to text that starts with a Mongolian character of the Mongolian block. The old trigger read the block buckets. Driven by the Script property instead, it would have turned some texts vertical that used to be horizontal (a leading space, or a Mongolian Supplement ornament). The new rule only ever turns vertical texts horizontal.
3. N5: leading marks before a tatweel (src/shape/native_direction.rs).
   • Cause: text with no script-bearing character took its native direction from its first strong character. The tatweel is AL, so an LTR buffer of Arabic marks plus a tatweel counted as non-native, and its graphemes were reversed.
   • Fix: such text is now natively LTR, as HarfBuzz's HB_SCRIPT_INVALID is.
   • The mark sorting itself was already right.
4. A12: symbols in the stch word (src/unicode/general_category*.rs, src/shape/stch.rs).
   • The General_Category table gets a Symbol class (Sc, Sk, Sm, So). The snapshot is refreshed from the same Unicode 17.0.0 DerivedGeneralCategory.txt, keeping the four symbol categories.
   • The stch stretch counts symbols into its word.
   • GeneralCategoryClass (under the hidden unicode:: path) gains the variant and becomes #[non_exhaustive].

New parity file tests/script_property_parity.rs: 21 cases, all HarfBuzz 14.5.0 output, on the repo's own fonts (Rubik, Amiri, Noto Sans Mongolian / Devanagari / Sinhala / Syriac). It covers Latin Extended Additional, Arabic Extended-A and the math letters, the Mongolian Supplement, Devanagari Extended / Extended-A, Sinhala Archaic Numbers, leading marks before a tatweel, and symbols in an stch word. 0.23.1 fails 19 of the 21 cases. Unit tests cover script_of, is_common_or_inherited, the Symbol class, stch word membership, native direction with no script, and the Mongolian vertical trigger. The generator test now checks every code point: script_of must equal the bucket of script_code.

Before and after (HarfBuzz 14.5.0 through uharfbuzz 0.56.2)

The issue's repros, each a buffer with guessed segment properties:

font	text	0.23.1	this PR	HarfBuzz 14.5.0
Roboto	U+1EF7 U+0020 U+0066 U+0069 ("ỷ fi")	[1139,4,74,77], adv 2684	[1139,4,444], adv 2610	[1139,4,444], adv 2610
SansSerifCollection	U+1EF0 U+0061 ("Ựa")	[838,68], adv 1341	[838,68], adv 1311	[838,68], adv 1311
arial	U+08A0 U+0628 U+0644	[4006,911,989] LTR	[989,913,4008]	[989,913,4008]
segoeui	same	[4866,2318,2341]	[2393,3623,4867]	[2393,3623,4867]
tahoma	same	[3573,911,989]	[990,914,3575]	[990,914,3575]
times	same	[4003,911,989]	[990,914,4005]	[990,914,4005]
calibri	same	[4748,4729,5211]	[5232,5656,4769]	[5232,5656,4769]
MSUIGHUB	same	[1147,71,94]	[321,267,1149]	[321,267,1149]
micross	same	[3207,858,881]	[881,791,3208]	[881,791,3208]
arial	U+064F U+064B U+0640	[757,753,752]	[753,757,752]	[753,757,752]

Rows that differ from HarfBuzz:

suite	rows	0.23.1	this PR
audit scripts.tsv (first letter in the old block)	72	0	0
audit scripts2.tsv (Vietnamese, 6 Latin fonts)	252	14	0
audit scripts3.tsv (Arabic Extended-A, Mongolian Supplement, 8 fonts)	48	16	0
audit scripts4.tsv	48	0	0
audit marks.tsv	6	1	0
first character outside the old blocks: every older bucket, 30 Windows and repo fonts, h and ttb	11,043	1,258	8
the same over the repo's test fonts	1,059	255	0
consumer differential (554 cases), defaults, clusters compared	554	73	0
consumer differential, glyphs and positions only	554	0	0

The 8 remaining rows are SansSerifCollection texts with a Coptic letter followed by a space. Coptic has no bucket (see Not done).

Broad sweep in the style of review283/shape_cmp.py. It covers 26 fonts (13 from the repo, 13 Windows and downloaded fonts; 9 are variable, at 2 random settings each besides the default). Per font it shapes 1,000 random cmap characters and 500 random triples, and where the font maps combining marks, 200 base+mark+mark texts and 100 mark+mark+base texts, in h and v. sigilbuzz shapes each text twice: once without guess_segment_properties, as the original harness did, and once with it. HarfBuzz always guesses. Rows are font × setting × text:

mode	rows	0.23.1 != HB	PR != HB	fixed	went from match to mismatch
h, guessed, glyphs and positions	55,782	448	54	394	0
h, guessed, clusters too	55,782	9,672	54	9,618	0
v, guessed, glyphs and positions	55,782	702	504	198	0
v, guessed, clusters too	55,782	9,566	504	9,062	0
h, not guessed, glyphs and positions	55,782	5,293	4,772	535	14
h, not guessed, clusters too	55,782	12,796	4,777	8,032	13
v, not guessed, glyphs and positions	55,782	1,069	855	215	1
v, not guessed, clusters too	55,782	9,692	855	8,837	0

• Guessed modes: no text goes from matching to not matching.
• Not-guessed modes: 28 rows do, all from 6 mixed-script random triples:
  • Rubik U+045D U+0116 U+FB01, at 3 settings;
  • Roboto Flex U+043B U+0139 U+1EB0, U+039F U+00DC U+1EB0, U+03B8 U+1EBD U+1EE9 and U+0437 U+00D8 U+1EB4;
  • Tahoma U+055C U+FE23 U+08F0.
• Why those 6 change: HarfBuzz shapes each of them as one buffer in the script of its first letter (Cyrillic, Greek or Armenian). sigilbuzz splits a buffer whose script is not set into script runs:
  • 0.23.1 cut the Latin run in two, because U+FB01, U+1EB0 and the like were Other. The kerning pair across the cut was lost, and by chance that matched HarfBuzz's Cyrillic or Greek shaping.
  • This PR keeps the Latin letters in one Latin run.
• Check: shaping the same script runs separately with HarfBuzz gives exactly this PR's output for all 28 rows.
• Other buckets: the vertical and pre-existing mismatches (Noto Sans KR VF vertical, the Coptic and Armenian Other texts) are unchanged.

Table size

build	0.23.1	this PR	change
libsigilbuzz.rlib, release, --no-default-features	9,527,106 B	9,529,216 B	+2,110 B
wasm32-unknown-unknown cdylib that guesses and shapes one buffer (opt-level = "z", LTO, panic = "abort", stripped)	532,022 B	535,047 B	+3,025 B (+0.6%)

The Script table keeps its size. Only its bucket column changes, and the block arms of script_of are gone. The General_Category table grows by 242 ranges, which is the symbol class.

Verification

• cargo fmt --all -- --check: clean.
• cargo +1.99.0 clippy with --workspace --all-targets --all-features, with --workspace --all-targets --no-default-features, and with -p sigilbuzz --no-default-features, all with -D warnings: clean.
• cargo test --workspace --all-features: 195 test binaries, 3,682 tests pass. That includes the Indic, USE, Khmer, Myanmar, Hangul, bidi, cluster-level, Mongolian, Tibetan, line-breaking and word-breaking parity suites and the C API tests.
• cargo build --no-default-features: OK. MSRV, with CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback cargo +stable generate-lockfile, then cargo +1.81 check --workspace --lib --no-default-features and --lib --bins --all-features: OK.
• shape fuzz target, 10 minutes in Docker (rustlang/rust:nightly, -O -a, seeds are the repo fonts under 65 KB): 102,391 runs, 1,706 new units, peak RSS 467 MB. No crash, timeout or leak.
• unicode_table_gen regenerates every table byte for byte from the snapshots.

Not done

• Scripts without a bucket: Armenian, Georgian, Coptic, Ethiopic, Thaana and the rest still shape as Other under DFLT, where HarfBuzz tries their own tag first (armn, copt, ...). The 8 Coptic rows above and most of the remaining SansSerifCollection rows in the guessed sweep come from this. Fixing it needs Other segments to carry their ISO code.
• Kana tag for a buffer-wide script: a buffer whose script is set or guessed as Han because its first letter is kana takes hani; HarfBuzz uses kana. Unguessed segments already get kana.
• Letter case in stch: stch still tells Lm and Lo from cased letters through Rust's case properties, so a titlecase letter, or a modifier letter with Other_Lowercase, is misjudged.
• Unicode versions: the General_Category table stays at Unicode 17.0.0, its existing version, and the Script table at 18.0.0, its existing version. HarfBuzz 14.5.0 reads 18.0.0 for both.
• Files left alone: CHANGELOG.md and docs/ROADMAP.md are not edited. Two bullets in the ROADMAP "Gaps" list are done by this PR and can go: the stch symbols bullet and the "older UnicodeScript buckets" bullet.

CHANGELOG lines (for the 0.24.0 section)

Changed:

• The default cluster level is now MonotoneGraphemes, HarfBuzz's default (Read the Script property for every character, and match HarfBuzz's default cluster level #288). ClusterLevel::default(), and so every new Buffer, was MonotoneCharacters. Under the defaults, a combining mark, a variation selector, the second regional indicator of a flag, the rest of a ZWJ emoji sequence and the other characters a grapheme merges now share the cluster of the character they continue, as in HarfBuzz. Glyphs and positions do not change. Code that expects a cluster per character calls buffer.set_cluster_level(ClusterLevel::MonotoneCharacters). The C API already defaulted to HB_BUFFER_CLUSTER_LEVEL_MONOTONE_GRAPHEMES and is unchanged. The sigilbuzz shape command builds its buffer with the default, so its cluster= values change the same way and now match hb-shape. Under defaults, all 554 consumer differential cases now match HarfBuzz 14.5.0's clusters.
• script_of returns the bucket of a character's Unicode Script property for every bucket (Read the Script property for every character, and match HarfBuzz's default cluster level #288). The older buckets, Latin through Modi, covered the Unicode blocks of their scripts. So letters outside those blocks were Other: Latin Extended Additional, Arabic Extended-A, -B and -C, the Arabic mathematical letters, the CJK extensions past B and the compatibility ideographs, the Mongolian Supplement, Devanagari Extended and others. They now take their script's bucket. The Common and Inherited characters inside the blocks took the block's bucket and are now Other. These include digits, punctuation, the space, the tatweel, the dandas and combining marks, so script_of('0') was Latin. The Coptic letters of the Greek and Coptic block are Other, not Greek.
• Script runs (Buffer::script_runs and the segmentation of shape) and Buffer::guess_segment_properties skip unassigned and private-use characters, as well as Common and Inherited ones, as hb_buffer_guess_segment_properties does (Read the Script property for every character, and match HarfBuzz's default cluster level #288). A private-use icon inside Latin text stays in the Latin run. The ordinal indicators U+00AA and U+00BA count as Latin letters, so inside text of another script they start a Latin run when the buffer has no script set.
• The vertical default for Mongolian, used when no direction is set, applies to text that starts with a Mongolian character of the Mongolian block (Read the Script property for every character, and match HarfBuzz's default cluster level #288). Text that starts with a quotation mark, the Mongolian comma, or a letter of another script stays horizontal, as HarfBuzz lays out all text with no direction. Set Direction::Ttb for vertical Mongolian.
• unicode::general_category::GeneralCategoryClass has a Symbol class (Sc, Sk, Sm, So) and is #[non_exhaustive] (Read the Script property for every character, and match HarfBuzz's default cluster level #288). general_category_class returns Some(Symbol) for the symbols, which were None. The table grows by 242 ranges.

Fixed:

• Text whose first letter sits outside the blocks of the older script buckets shapes with its script, as in HarfBuzz (Read the Script property for every character, and match HarfBuzz's default cluster level #288). It shaped as Other under DFLT:

  • Roboto "ỷ fi" lost the fi ligature: [1139, 4, 74, 77], where HarfBuzz gives [1139, 4, 444].
  • The Windows 11 sans "Ựa" lost its kerning.
  • Arabic Extended-A shaped left to right with no joining in all 7 fonts tested (Tahoma U+08A0 U+0628 U+0644).
  • A Mongolian Supplement birga lost the joining forms of the letters after it.

  Over 11,043 such texts in 30 fonts, mismatches against HarfBuzz 14.5.0 fall from 1,258 to 8 (Coptic, which has no bucket).

• Text with no script-bearing character shapes left to right, as HarfBuzz's invalid script does (Read the Script property for every character, and match HarfBuzz's default cluster level #288). Leading Arabic marks before a tatweel were reversed: Arial U+064F U+064B U+0640 gave [757, 753, 752], where HarfBuzz gives [753, 757, 752].

• The stch stretch (U+070F SYRIAC ABBREVIATION MARK) counts symbols (General_Category Sc, Sk, Sm and So) into the word it fills, as HarfBuzz does (Read the Script property for every character, and match HarfBuzz's default cluster level #288). A symbol ended the word, so the stretch came out short.