Repository navigation
Prove truncated UTF-8 recovery and record mutation evidence - #8
Conversation
727f860 to
ea92619
Compare
zero-aska
left a comment
There was a problem hiding this comment.
Requesting changes: the new SPEC.md section says the fourteenth UTF-16 planted bug cannot be found and that no runner exists. A runner in this repository finds it, and all fourteen bugs are caught on this head.
This review is for head ea92619. It is the same patch as 727f860, the head I reviewed before, rebased onto main 2cdbc29. git range-diff shows the commit unchanged. The two heads differ only by #7's LICENSE and lakefile license line, which come from main. So the earlier finding still stands, and the earlier note about the old base is resolved.
This PR is the bottom of a stack. #9 is built on this head and must merge after it.
What I checked
- I read the whole diff against main: Spec.lean, Proofs/Main.lean, README.md and SPEC.md, +20/-4.
lake buildon this head, Lean 4.35.0: it passes with 37 jobs. The only warnings are the 35 expectedsorryones in Spec.lean.bar/check.py --offline --run-testson this head, with the same rows as on main:- Item 1 is OPEN, since no approval is recorded. The spec hash is now a0563cd0.... No approval existed, so nothing is voided.
- Items 2 to 6 PASS. Item 3:
lake comparator --paranoidaccepts all 35 theorems under every bundled kernel. - Item 5: 2,818,540 inputs, seed 20261025, 0 failures. The regenerated
tests/TEST-RESULTS.mdmatches the committed one, apart from the Python version line. - Item 7 SKIP (offline).
- Statement check: I dumped every type and definition in the spec modules. Against main, the only change is one new conjunct in
Spec.substitution_vectors:Utf8.decodeLossy [0xE2, 0x82, 0x3C]is U+FFFD followed by<. Without that conjunct, the type is exactly main's. No definition changes. - The new vector is right. E2 82 is one maximal subpart (Unicode 3.9.6), so it gives one U+FFFD and keeps the
<. Python'sreplacehandler agrees on all seven conjuncts. A decoder that took the three bytes the lead announces would fail it. - The README and SPEC rows now say "three cut-short sequences followed by a character". That is right.
- CI: two
bar / barruns on ea92619, both success.git merge-treeonto main is clean.
Blocking
1. The "Mutation evidence limits" section in SPEC.md is wrong, and it is in the wrong file.
It says the #4 list "identifies only thirteen" and that "No fourteenth mutation or reproducible mutation runner is recorded in this repository". So it calls the fourteen-bug claim unverified. That does not hold.
- The first bullet in #4 is "surrogate ranges off by one". "Ranges" is plural: one bug for the high range, one for the low range.
- Branch
claude/review-followupsin this repository hastests/planted_bugs.pyandtests/PLANTED-BUGS.md(commit 2e018fe, from 2026-10-26). It comes from the same Claude session that wrote #3 and #4, so it is the original author's record that #5 asks for. It lists 14 UTF-16 bugs, with "High surrogate range misses 0xDBFF" and "Low surrogate range misses 0xDC00" as separate rows. - I planted each of that runner's 14 UTF-16 bugs in a copy of this head. For each one, I ran
lake build utf8, thenpython3 tests/run_tests.py --count 20000(2,415,209 inputs), thenlake build Proofs.- With no bug planted, the tests report 0 failures.
- All 14 are caught. The failure counts are 3943, 8679, 969, 30786, 11129, 3569, 7677, 10627, 764, 998, 73811, 2642, 8192 and 1441. Every count equals the record.
- With each bug, the proofs also fail to build.
- An earlier run also caught all 8 lossy UTF-8 bugs, with the record's counts.
Reproduction:
git fetch origin claude/review-followups
git show FETCH_HEAD:tests/PLANTED-BUGS.md | sed -n 1,45p
# in a copy of this PR's tree:
git show FETCH_HEAD:tests/planted_bugs.py > tests/planted_bugs.py
python3 tests/planted_bugs.py --count 20000
Two more problems:
- SPEC.md is in
bar.toml's spec files, so it is part of the hash a spec reviewer approves. A history of past PR bodies is not specification. Correcting it later would void an approval. - The PR body's answer to #5 about the fourteen bugs makes the same claim, so it needs the same correction.
Suggested fix: remove the section from SPEC.md. Then either bring tests/planted_bugs.py and tests/PLANTED-BUGS.md over from claude/review-followups, here or in a separate PR, or add a short note under tests/ that links #3, #4 and that record. Answer #5's fourteenth-bug question with a link to the record.
Non-blocking
- The Spec.lean docstring of
Spec.substitution_vectorsstill says "two cases where replacing too many bytes would swallow the next character". There are now three. The comment above them, in Spec.lean and Proofs/Main.lean, still says "then<andA". This is the challenge file, so please fix both in the same push. - If you bring
tests/planted_bugs.pyover, note that it runslake build Proofs. #9 renames that library toUtfProofs, and on #9's headlake build Proofsgives "unknown targetProofs". So #9 would need to update the script. claude/review-followupsalso adds four more cut-short vectors to the same theorem (commit 9ffcdbf). It edits the same rows to say "six". Please coordinate so the work is not done twice.- The body says the U+FEFF policy for #2 is already on main. That is true (SPEC.md line 35, and README's "Not proved" list). But the #2 point in #5 may be about the invisible U+FEFF and U+E000 literals in the Lean files. Main now writes them as escapes, but no file records that rule. Commit 5731b9b on
claude/review-followupshas a paragraph for it.
Merge order
This merges first, and #9 merges after it. Suppose you fix this with a new commit on this branch (dropping the section and fixing the docstring), and it is merged with a merge commit. Then #9's current head still merges cleanly. The section is gone from the result, and the docstring fix lands in Utf/Spec.lean. I simulated that with git merge-tree. If you amend or rebase this branch instead, #9 needs a rebase and a fresh review.
Generated by Claude Code
Remove the incorrect "Mutation evidence limits" section that claimed the fourteenth UTF-16 planted bug was unverifiable. Add tests/planted_bugs.py and tests/PLANTED-BUGS.md from the review-followups record; the runner catches all 14 UTF-16 mutations (28 of 30 total, two marked equivalent). Update substitution_vectors docstrings for three cut-short cases. Co-authored-by: ahraz <ahrazzle@users.noreply.github.com>
The requested fixes are in ca29613: the wrong Mutation evidence limits block is gone from SPEC.md, tests/planted_bugs.py and PLANTED-BUGS.md are present, and the substitution_vectors docstrings and added conjunct are corrected. Verified on the box.
zero-aska
left a comment
There was a problem hiding this comment.
Verified at ca29613 on the box.
The earlier change request is addressed: the wrong "Mutation evidence limits" block is gone from SPEC.md; tests/planted_bugs.py and PLANTED-BUGS.md are in the tree; the substitution_vectors docstrings now say three cut-short sequences, and the added conjunct Utf8.decodeLossy [0xE2, 0x82, 0x3C] = ['�', '<'] is present.
- lake build passed, including the Spec and Proofs libraries. tests/run_tests.py: 8,714,312/8,714,312 requests, 0 failures. tests/cross_check.py: 2,358,234 byte strings OK. tests/planted_bugs.py --count 20000: 28 of 30 planted bugs caught; the two at 0 failures ("Leads 0xF5 to 0xF7 start four-byte sequences" and "A bad last byte given its own branch") are marked equivalent mutations, and lake build Proofs still fails for both. The generated record matches the committed PLANTED-BUGS.md. mission bar/check.py --offline --run-tests at current mission main: in-progress, no failures; item 1 remains OPEN. CI bar/bar is green on this head.
- Versus main, the only Spec theorem-statement change is that added substitution_vectors conjunct. Docstrings were updated to match. No statement is weakened. No new sorry, axiom, native_decide, implemented_by, extern, or unsafe outside the spec statement files.
Approving the fix. This is not a specification approval.
The tree is unchanged. This refreshes the merge-base so a last-push approval can apply.
Add a third proved lossy UTF-8 case: a truncated three-byte sequence followed by
<preserves the character (Utf8.decodeLossy [0xE2, 0x82, 0x3C] = ['�', '<']inSpec.substitution_vectors). Record in SPEC.md ("Mutation evidence limits") how the eight UTF-8 mutations were counted and the limit of the historical UTF-16 evidence: the published list identifies thirteen mutations, leaving the claimed fourteenth unverified.Refs #5. How each item of #5 is answered:
EF BB BF) is read as the character U+FEFF ... does not strip it"; README.md "Not proved" list).Merge order: land this before #9 (module move). #9 is rebased on top of this branch.
Validation, on the head rebased onto current main (2cdbc29):
lake buildpasses, including the extendedSpec.substitution_vectorsproof.python3 tests/run_tests.py: 8714312/8714312 requests agree, 0 failures (record unchanged).🤖 Generated with Claude Code
https://claude.ai/code/session_01QwrvNdsMA6RtgKxXn1Pmhf