Add UTF8.decode-at, an incremental single-code-point decoder; re-express valid? in terms of it

[carpentry-agent]

Follow-up to the discussion on parsec#24, where the same RFC 3629 / Table 3-7 validation was about to be duplicated between this library's valid? and parsec's decode. Veit asked to "do the more principled fix right away" — centralize the logic here rather than duplicate it. This PR is the enabling half: a public incremental decoder that becomes the single source of truth for UTF-8 code-point validation.

What changed

New public primitive UTF8.decode-at:

(sig decode-at (Fn [(Ref (Array Byte)) Int] (Maybe (Pair Int Int))))

It decodes exactly one code point starting at byte offset pos and returns (Just (Pair codepoint width)) — the scalar value and its byte width (1–4) — or (Nothing) on end-of-input, truncation, or malformed input. It applies the full Table 3-7 checks: overlong encodings, UTF-16 surrogates (U+D800..U+DFFF), code points beyond U+10FFFF, and out-of-range continuation bytes.

Two deliberate signature choices, both driven by the parsec use case:

• (Ref (Array Byte)), not a String — a String round-trip truncates at the first NUL, silently accepting garbage after it. On raw bytes, an embedded NUL (the valid one-byte code point U+0000) decodes like any other byte.
• Returns the code point as an Int — sidesteps the Char-is-a-byte (utf8.carp) vs Char-is-a-code-point (parsec) ambiguity across the two libraries.

valid? re-expressed in terms of decode-at — it now loops: decode-at at the cursor, advance by the returned width, fail on Nothing. The Table 3-7 logic that was inline now lives in one place.

valid? is byte-for-byte unchanged

Its observable behavior is identical. Beyond the existing suite still passing, I ran an exhaustive differential of the old vs. new implementation:

Input space	Cases	Divergences
every 1-byte sequence	256	0
every 2-byte sequence	65 536	0
every 3-byte sequence	16 777 216	0
4-byte, leads EE–F7 × full first continuation × boundary tail bytes	~262 k	0
multi-code-point, length-4 over a 16-byte mixed alphabet	65 536	0

The harness was checked non-vacuous: swapping in a deliberately-broken validator makes it report 128 divergences on the 1-byte sweep, so the zeros above are meaningful.

The value-range checks are the algebraic equivalent of the old lead-byte-range dispatch (e.g. an E0 lead requiring first continuation ≥ 0xA0 ⇔ code point ≥ U+0800; ED first continuation ≤ 0x9F ⇔ code point < U+D800; likewise F0/F4). This is the same equivalence the reviewer verified on parsec#24, now with the code point computed rather than only its byte ranges bounded.

Tests

test/utf8.carp gains a decode-at matrix (75/0 total, 46 baseline + 29 new): per-width min/max boundaries with the exact expected code point and width (U+0080, U+07FF, U+0800, U+FFFF, U+10000, U+10FFFF); the surrogate hole (U+D7FF/U+E000 accepted, U+D800/U+DFFF rejected); overlong forms (C0 80, E0 80 80, F0 80 80 80); truncated and lone-continuation sequences; F5/F8/FF; EOF/negative offsets; an embedded NUL; and decoding at a non-zero offset inside a mixed buffer.

docs/UTF8.html regenerated (decode-at is now public; decode-cont is hidden/private). No CHANGELOG exists, so none was added.

Sequencing (parsec follow-up)

This PR stands on its own. The parsec rewire — route Parser.UTF8.decode through decode-at and delete its duplicated inline validation — can't land atomically, since parsec can only load a tagged utf8.carp. It has to follow once this is merged and released. I've left a note on parsec#24 asking whether you'd prefer to hold #24 as the interim self-contained fix until then, or close it in favor of the rewire.

---

Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.

[carpentry-reviewer]

Build & Tests

Checked out claude/incremental-decoder, built and ran the suite locally (armhf, compiled via carp -x): 75/0 pass. CI is green on ubuntu + macos.

Findings

I went past the suite and tried to break the two central claims independently.

1. valid? is byte-for-byte unchanged — confirmed exhaustively. I embedded the pre-PR valid? (copied from the removed lines) alongside the new one and diffed their verdicts directly:

• all 1+2+3-byte sequences — 16,843,008 cases, 0 divergences.
• 4-byte boundary sweep (leads 0xEE–0xFF × full first continuation × below/at/inside/above-range tail bytes) — 903,168 cases, 0 divergences.
• Non-vacuity control: the same harness against an accept-all validator reports 128 divergences on the 1-byte sweep, so the zeros are meaningful.

This makes the F0/F4/E0/ED equivalence concrete: the new value-range checks (>= min-cp, surrogate hole, <= 0x10FFFF) reproduce the old lead-byte first-continuation tightening on every input, including the always-overlong C0/C1 leads (old code rejected them at the lead; new code lets them into decode-cont and rejects them via min-cp — same verdict).

2. decode-at's returned code point is correct — the valid? differential doesn't cover this (it only exercises the width, never Pair.a). So I checked the scalar value separately against an independent textbook recompute over every accepted 1/2/3-byte sequence: 8,976,384 accepted, 0 wrong code points, 0 wrong widths.

Bounds safety: decode-cont's (> (+ pos ncont) (- len 1)) guard runs before any unsafe-nth, and decode-at guards pos in [0, len) before dispatching, so every continuation-byte access is in-bounds — no OOB read path. Width is always >= 1, so the valid? loop can't stall.

Nothing to fix. docs/UTF8.html regenerated with decode-at public and decode-cont correctly hidden/private; the diff carries no unrelated doc drift. No CHANGELOG (none exists) — correct.

Verdict: merge

The code is correct and merge-ready — one of the cleaner changes I've reviewed: the primitive is well-chosen ((Ref (Array Byte)) + Int code point are the right calls for the parsec rewire), and the "no behavior change" claim holds under exhaustive differential testing. The draft status is intentional — it's yours to un-draft once you've settled the hold-vs-close question on parsec#24, so I'm not touching that. Verdict reflects code quality, not a nudge to merge now.