fix(expr)!: match Python semantics in character classification functions

[mwiebe]

Fixes: #309

What was the problem/requirement? (What/Why)

Background: OpenJD job template expressions include string classification
functions — isdigit, isalpha, isalnum, and friends — that are named
after, and meant to behave like, Python's str methods. "Is this character a
digit?" sounds like it has one answer, but it doesn't: every language draws
its own lines through the Unicode character set. Rust's standard library
draws different lines than Python does, and this implementation was using
Rust's.

Issue #309 reported the visible symptom: for '٣' (the Arabic-Indic digit
three), isdigit said false while isalnum said true — the predicates
contradicted each other, because isdigit was checking only ASCII 0-9
while isalnum accepted anything Unicode calls alphabetic or numeric.

Comparing every one of Unicode's ~1.1 million code points against CPython
showed the problem was wider than the report:

• isdigit was wrong for 878 characters (all non-ASCII digit systems, plus
  superscripts like ²)
• isalpha/isalnum disagreed with Python on thousands of characters
  (Roman numerals like Ⅻ, combining marks, circled letters)
• isspace missed four control characters Python counts as whitespace
• isupper/islower ignored the wrong characters: Python skips uncased
  characters (so 'a五' is lowercase), this implementation skipped
  non-alphabetic ones

What was the solution? (How)

Since the expression language dialect is defined by Python's behavior, the
predicates now use lookup tables generated directly from CPython by a new
script, scripts/generate_unicode_tables.py. The script asks CPython itself
("is this character a digit to you?") for every code point, writes the
answers into crates/openjd-expr/src/functions/unicode_tables.rs as compact
range tables (checked in, currently CPython 3.14.3 / Unicode 16.0.0), and
verifies the generated tables round-trip exactly before writing. The six
predicates are now binary searches over those tables, with isupper/islower
implementing Python's cased-character rule.

To adopt a newer Unicode version later, rerun the script with a newer CPython.

What is the impact of this change?

The classification functions now return exactly what Python returns for any
input. Results change only for non-ASCII input (and the four isspace
control characters).

How was this change tested?

• 22 new integration tests covering every divergence class found in the
  analysis, with CPython-verified expected values.
• Generated tables carry embedded unit tests (well-formedness invariants and
  an exhaustive binary-search-vs-linear-scan check over all code points).
• The generator itself asserts table/CPython agreement for all 1,114,112
  code points at generation time.
• Full openjd-expr suite: 3,313 tests pass; cargo test --workspace green.
• Full OpenJD conformance suite: 1,117 passed, 0 failed — including a new
  Unicode classification conformance test (submitted separately to
  openjd-specifications) that fails against the previous implementation and
  passes against this one.
• Clippy -D warnings, rustfmt, and copyright header checks all clean.

Was this change documented?

Yes — specs/expr/function-library.md documents the Python-parity semantics,
the cased-character rule, and the table regeneration procedure. A companion
openjd-specifications PR clarifies RFC 0006 and the Expression Language spec,
which previously didn't say which character-class convention applies.

Is this a breaking change?

Yes — the commit is marked fix(expr)! with a BREAKING CHANGE footer.
Classification results change for non-ASCII input: e.g. isdigit('٣') is now
true, isalpha('Ⅻ') is now false, and islower('a五') is now true.
No code changes are required of users; templates that relied on the old
answers for non-ASCII input will see the Python-correct results.

Does this change impact security?

No.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[leongdl]

Reviewed at tip 0258acf with the full suite green locally (3615 passed) and a mutation harness over the new behaviours — every claimed fix has a named falsifying test (7/7 mutants caught, including the table binary search being pinned by its own exhaustive binary-vs-linear test, which is a genuinely independent oracle). Expected values in the new integration tests reproduce 0-mismatch against a local CPython. Nice work — the generator-verifies-against-CPython-before-writing design is exactly right.

One real (low-severity, non-blocking) finding below on guard ordering, verified three ways rather than by inspection.

[leongdl]

Reviewer orientation — TLDR and call stack (posted to help others review; verified locally at 45d3a9d, full suite 3,626 passed / 0 failed)

TLDR

The expression functions isdigit/isalpha/isalnum/isspace/isupper/islower are named after, and specified to behave like, Python's str methods — but they used Rust std classification, which draws different lines through Unicode (#309's symptom: isdigit('٣') false while isalnum('٣') true, self-contradictory). The three commits:

1. Classification predicates → binary searches over range tables generated from CPython itself (scripts/generate_unicode_tables.py asks a running CPython about all 1,114,112 code points and verifies round-trip before writing). isupper/islower also gain Python's cased-character rule: ignore uncased chars, require ≥1 cased — so islower('a五') is now true.
2. title()/capitalize() → CPython do_title/do_capitalize semantics: word boundaries at uncased chars (title('1st') = '1St'), ToTitleFull mapping (capitalize('džx') → 'Džx', not 'DŽx'), and the Final_Sigma context rule (Σ lowers to ς at word end, σ otherwise).
3. int()/float() → closes a gap commit 1 created: once isdigit('٣') became true, the common guard int(Param.X) if isdigit(Param.X) else 0 passed the guard and then failed in int(), which was still ASCII-only. Nd digits are now normalized to ASCII before parsing, exactly like CPython's _PyUnicode_TransformDecimalAndSpaceToASCII. int('²') correctly stays an error (Numeric_Type=Digit, not Decimal), matching Python.

Call stack

job template "{{ isdigit(Param.X) }}"          (untrusted input)
└─ openjd-expr evaluator → FunctionLibrary dispatch (default_library.rs)
   └─ string.rs::isdigit_fn … islower_fn                      [CHANGED]
      ├─ ctx.count_string_ops(s.len())      pre-existing op-budget guard
      └─ unicode_tables::in_table(TABLE, c)                   [NEW]
         binary search over static CPython-derived (start, end) ranges

   └─ string.rs::title_fn / capitalize_fn                     [CHANGED]
      ├─ collect Vec<char> + ctx.check_memory (ordering under discussion above)
      ├─ push_titled(c)      → unicode_tables::title_mapping  (ToTitleFull)
      └─ push_lowered(..)    → is_final_sigma(..)             (Final_Sigma context)

"{{ int(Param.X) }}"
└─ conversion.rs::int_from_string / float_from_string         [CHANGED, commit 3]
   └─ normalize_decimal_digits(s.trim())                      [NEW]
      ├─ ASCII fast path → Cow::Borrowed (zero cost for the common case)
      └─ unicode_tables::decimal_digit_value(c)               [NEW]
         value = (cp − range_start) % 10   ← relies on the zero-aligned-run invariant
   └─ .parse::<i64/f64>()   → existing "Cannot convert" error path

All error paths return ExpressionError (Result), no panics added.

What makes the tables trustworthy (the crux of the review)

• The generator verifies every emitted range table round-trips against CPython for all code points before writing, with an independent sweep (not the same code path that built the ranges).
• The % 10 digit-value trick requires every DECIMAL range to be a zero-aligned run of ten; the generator checks each code point's value against unicodedata.decimal(). I independently re-verified the checked-in table against a local CPython: 760 code points, zero value/alignment errors (the only unverifiable entries were Unicode-16-only digit blocks my older host unicodedata doesn't know — a version-skew note, not a defect).
• Mutation testing: 8 mutants across the three commits (revert isdigit to ASCII, weaken the cased rule, disable Final_Sigma, revert title word boundaries, revert ToTitleFull, off-by-one the table binary search, revert isspace, disable Nd normalization) — all 8 caught by named tests, including the guard-pattern test int_isdigit_guard_pattern_nd_digit for commit 3, and the table's own exhaustive binary-vs-linear test for the off-by-one (a genuinely independent oracle).

Known-open items are in the inline threads above: the check_memory-after-allocation ordering (acknowledged, fix pending) and the capture-name XID check cross-referenced from #337.