Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions Proofs.lean

This file was deleted.

72 changes: 36 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,44 +41,44 @@ From Lean:

## What is proved

The statements are in `Spec.lean`, the definitions they use in `Spec/Defs.lean` and `Spec/Utf16.lean`, and the proofs in `Proofs/`.
The statements are in `Utf/Spec.lean`, the definitions they use in `Utf/Spec/Defs.lean` and `Utf/Spec/Utf16.lean`, and the proofs in `Utf/Proofs/`.

| Claim | Theorem |
|-------|---------|
| The encoder writes each character as the table of RFC 3629, section 3, sets out. | `Spec.encode_eq` |
| `decode` returns a list of characters exactly when the bytes are its encoding. So every byte string it accepts has one reading. It refuses everything else: overlong forms, surrogate halves, values past U+10FFFF, stray continuation bytes, the bytes 0xC0, 0xC1 and 0xF5 to 0xFF, and sequences cut short. | `Spec.decode_iff` |
| Decoding what the encoder wrote gives the characters back. | `Spec.decode_encode` |
| The decoder accepts exactly the byte sequences that Table 3-7 of the Unicode Standard calls well formed. | `Spec.decode_wellFormed` |
| Lean core's `String.toUTF8`, as core defines it, writes the same bytes as the encoder. | `Spec.encode_toUTF8` |
| Lean core's `String.fromUTF8?`, as core defines it, accepts exactly the bytes the decoder accepts and reads the same characters. | `Spec.fromUTF8?_eq` |
| The examples of RFC 3629, section 7, both ways; the boundaries of each length and of the surrogates; and a selection of malformed inputs, most of them from Markus Kuhn's UTF-8 decoder test file. Every case in that file follows from `Spec.decode_iff`, and the tests run them all. | `Spec.rfc3629_examples`, `Spec.boundary_vectors`, `Spec.reject_vectors` |
| `decodeLossy` gives exactly what section 3.9.6 of the Unicode Standard, *U+FFFD Substitution of Maximal Subparts*, sets out: each well-formed sequence becomes its character, and each maximal subpart of bad bytes becomes one U+FFFD. | `Spec.decodeLossy_iff` |
| On well-formed bytes, `decodeLossy` reads what `decode` reads; on any other bytes it gives at least one U+FFFD. | `Spec.decodeLossy_of_decode`, `Spec.decodeLossy_replaces` |
| Well-formed text before or after bad bytes comes through unchanged: no U+FFFD takes in any of it. | `Spec.decodeLossy_encode_append`, `Spec.decodeLossy_append_encode` |
| The examples of Tables 3-8 to 3-11 of the Unicode Standard, and three cut-short sequences followed by a character. | `Spec.substitution_vectors` |
| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Spec.utf16_encode_eq` |
| `Utf16.decode` returns a list of characters exactly when the code units are its encoding. So every unit list it accepts has one reading, and it refuses everything else. | `Spec.utf16_decode_iff` |
| Decoding what the UTF-16 encoder wrote gives the characters back. | `Spec.utf16_decode_encode` |
| `Utf16.decode` accepts exactly the unit lists that D91 and Table 3-5 of the Unicode Standard call well formed. | `Spec.utf16_decode_wellFormed` |
| `Utf16.decode` fails exactly when some unit is an isolated surrogate: a high surrogate with no low surrogate after it, or a low surrogate with no high surrogate before it. | `Spec.utf16_decode_none_iff` |
| The number of UTF-16 units is one for each character, and one more for each character from U+10000 up. | `Spec.utf16_encode_length` |
| `Utf16.decodeLossy` gives exactly the conversion of section 3.9.6, written for UTF-16 code units. | `Spec.utf16_decodeLossy_iff` |
| An isolated surrogate becomes one U+FFFD, and decoding goes on at the very next unit, so a high surrogate never takes the unit after it with it. That is what section 3.9.6 comes to in UTF-16. The proofs go through a lemma, `Utf16.Proofs.maximalSubpart_iff`, that each maximal subpart in UTF-16 is one isolated surrogate. It is a fact about the definitions alone, so the bar does not take it as one of the approved statements. | `Spec.utf16_decodeLossy_unpaired` |
| On well-formed units, `Utf16.decodeLossy` reads what `Utf16.decode` reads; on any other units it gives at least one U+FFFD. | `Spec.utf16_decodeLossy_of_decode`, `Spec.utf16_decodeLossy_replaces` |
| Well-formed text before or after bad units comes through unchanged. | `Spec.utf16_decodeLossy_encode_append`, `Spec.utf16_decodeLossy_append_encode` |
| `toWellFormed` keeps the number of units, puts 0xFFFD in place of each isolated surrogate and keeps every other unit, as ECMAScript's `String.prototype.toWellFormed` does. | `Spec.utf16_toWellFormed` |
| UTF-16BE writes each unit's high byte first, and UTF-16LE its low byte first, with no byte order mark. | `Spec.utf16_encodeBytes` |
| The UTF-16BE and UTF-16LE decoders accept exactly the encodings, so they refuse an odd number of bytes and read a leading byte order mark as U+FEFF. | `Spec.utf16be_decode_iff`, `Spec.utf16le_decode_iff` |
| The decoder for the UTF-16 encoding scheme reads a byte order mark as D98 and RFC 2781, section 4.3, say: FE FF for big-endian, FF FE for little-endian, the mark not part of the text, and big-endian with no mark. | `Spec.utf16_decodeScheme_iff` |
| The lossy byte decoders give exactly what the WHATWG Encoding Standard's UTF-16BE and UTF-16LE decoders give, run as the standard writes them. | `Spec.utf16_whatwg` |
| On whole units, the lossy byte decoders give the conversion of section 3.9.6, so the WHATWG decoders and the Unicode Standard agree. | `Spec.utf16_decodeLossyBytes_units` |
| The examples of D91, D96 and D97 of the Unicode Standard and of RFC 2781, section 5, both ways, and the texts of D98 and RFC 2781 with a byte order mark, read; the ends of the ranges; byte order marks; isolated surrogates refused and replaced; and a high surrogate cut off by an odd byte, which the WHATWG decoders turn into one U+FFFD. | `Spec.utf16_vectors` |
| The encoder writes each character as the table of RFC 3629, section 3, sets out. | `Utf.Spec.encode_eq` |
| `decode` returns a list of characters exactly when the bytes are its encoding. So every byte string it accepts has one reading. It refuses everything else: overlong forms, surrogate halves, values past U+10FFFF, stray continuation bytes, the bytes 0xC0, 0xC1 and 0xF5 to 0xFF, and sequences cut short. | `Utf.Spec.decode_iff` |
| Decoding what the encoder wrote gives the characters back. | `Utf.Spec.decode_encode` |
| The decoder accepts exactly the byte sequences that Table 3-7 of the Unicode Standard calls well formed. | `Utf.Spec.decode_wellFormed` |
| Lean core's `String.toUTF8`, as core defines it, writes the same bytes as the encoder. | `Utf.Spec.encode_toUTF8` |
| Lean core's `String.fromUTF8?`, as core defines it, accepts exactly the bytes the decoder accepts and reads the same characters. | `Utf.Spec.fromUTF8?_eq` |
| The examples of RFC 3629, section 7, both ways; the boundaries of each length and of the surrogates; and a selection of malformed inputs, most of them from Markus Kuhn's UTF-8 decoder test file. Every case in that file follows from `Utf.Spec.decode_iff`, and the tests run them all. | `Utf.Spec.rfc3629_examples`, `Utf.Spec.boundary_vectors`, `Utf.Spec.reject_vectors` |
| `decodeLossy` gives exactly what section 3.9.6 of the Unicode Standard, *U+FFFD Substitution of Maximal Subparts*, sets out: each well-formed sequence becomes its character, and each maximal subpart of bad bytes becomes one U+FFFD. | `Utf.Spec.decodeLossy_iff` |
| On well-formed bytes, `decodeLossy` reads what `decode` reads; on any other bytes it gives at least one U+FFFD. | `Utf.Spec.decodeLossy_of_decode`, `Utf.Spec.decodeLossy_replaces` |
| Well-formed text before or after bad bytes comes through unchanged: no U+FFFD takes in any of it. | `Utf.Spec.decodeLossy_encode_append`, `Utf.Spec.decodeLossy_append_encode` |
| The examples of Tables 3-8 to 3-11 of the Unicode Standard, and three cut-short sequences followed by a character. | `Utf.Spec.substitution_vectors` |
| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Utf.Spec.utf16_encode_eq` |
| `Utf16.decode` returns a list of characters exactly when the code units are its encoding. So every unit list it accepts has one reading, and it refuses everything else. | `Utf.Spec.utf16_decode_iff` |
| Decoding what the UTF-16 encoder wrote gives the characters back. | `Utf.Spec.utf16_decode_encode` |
| `Utf16.decode` accepts exactly the unit lists that D91 and Table 3-5 of the Unicode Standard call well formed. | `Utf.Spec.utf16_decode_wellFormed` |
| `Utf16.decode` fails exactly when some unit is an isolated surrogate: a high surrogate with no low surrogate after it, or a low surrogate with no high surrogate before it. | `Utf.Spec.utf16_decode_none_iff` |
| The number of UTF-16 units is one for each character, and one more for each character from U+10000 up. | `Utf.Spec.utf16_encode_length` |
| `Utf16.decodeLossy` gives exactly the conversion of section 3.9.6, written for UTF-16 code units. | `Utf.Spec.utf16_decodeLossy_iff` |
| An isolated surrogate becomes one U+FFFD, and decoding goes on at the very next unit, so a high surrogate never takes the unit after it with it. That is what section 3.9.6 comes to in UTF-16. The proofs go through a lemma, `Utf16.Proofs.maximalSubpart_iff`, that each maximal subpart in UTF-16 is one isolated surrogate. It is a fact about the definitions alone, so the bar does not take it as one of the approved statements. | `Utf.Spec.utf16_decodeLossy_unpaired` |
| On well-formed units, `Utf16.decodeLossy` reads what `Utf16.decode` reads; on any other units it gives at least one U+FFFD. | `Utf.Spec.utf16_decodeLossy_of_decode`, `Utf.Spec.utf16_decodeLossy_replaces` |
| Well-formed text before or after bad units comes through unchanged. | `Utf.Spec.utf16_decodeLossy_encode_append`, `Utf.Spec.utf16_decodeLossy_append_encode` |
| `toWellFormed` keeps the number of units, puts 0xFFFD in place of each isolated surrogate and keeps every other unit, as ECMAScript's `String.prototype.toWellFormed` does. | `Utf.Spec.utf16_toWellFormed` |
| UTF-16BE writes each unit's high byte first, and UTF-16LE its low byte first, with no byte order mark. | `Utf.Spec.utf16_encodeBytes` |
| The UTF-16BE and UTF-16LE decoders accept exactly the encodings, so they refuse an odd number of bytes and read a leading byte order mark as U+FEFF. | `Utf.Spec.utf16be_decode_iff`, `Utf.Spec.utf16le_decode_iff` |
| The decoder for the UTF-16 encoding scheme reads a byte order mark as D98 and RFC 2781, section 4.3, say: FE FF for big-endian, FF FE for little-endian, the mark not part of the text, and big-endian with no mark. | `Utf.Spec.utf16_decodeScheme_iff` |
| The lossy byte decoders give exactly what the WHATWG Encoding Standard's UTF-16BE and UTF-16LE decoders give, run as the standard writes them. | `Utf.Spec.utf16_whatwg` |
| On whole units, the lossy byte decoders give the conversion of section 3.9.6, so the WHATWG decoders and the Unicode Standard agree. | `Utf.Spec.utf16_decodeLossyBytes_units` |
| The examples of D91, D96 and D97 of the Unicode Standard and of RFC 2781, section 5, both ways, and the texts of D98 and RFC 2781 with a byte order mark, read; the ends of the ranges; byte order marks; isolated surrogates refused and replaced; and a high surrogate cut off by an odd byte, which the WHATWG decoders turn into one U+FFFD. | `Utf.Spec.utf16_vectors` |

The command-line tool in `Main.lean` answers `decode`, `lossy` and `encode` requests, and each UTF-16 request, with these functions directly.

## What is not proved

- Lean core's compiled check. At run time, `String.fromUTF8?` runs C code, `lean_string_validate_utf8`, in place of the definition `Spec.fromUTF8?_eq` is about. The differential tests compare that C code with the proved decoder.
- Lean core's compiled check. At run time, `String.fromUTF8?` runs C code, `lean_string_validate_utf8`, in place of the definition `Utf.Spec.fromUTF8?_eq` is about. The differential tests compare that C code with the proved decoder.
- Reading requests and printing answers in `Main.lean`: the hex parsing and printing, and the request format. The differential tests cover them, with malformed requests and CR LF line endings.
- Other replacement policies. The lossy decoder follows section 3.9.6 only. It has no mode that puts U+FFFD in place of each bad byte on its own, as a Go `range` loop does. A byte order mark is read as a character, not stripped.
- For UTF-16: byte order marks in the lossy byte decoders. They are the WHATWG decoders themselves, which read a leading FE FF (UTF-16BE) or FF FE (UTF-16LE) as U+FEFF. The standard's other steps are not here. Its `decode` algorithm sniffs a mark first, and a leading EF BB BF, FE FF or FF FE then picks the encoding and is skipped. `TextDecoder`, unless `ignoreBOM` is set, drops a U+FEFF that the decoder gives at the start.
Expand Down Expand Up @@ -118,19 +118,19 @@ lake comparator --paranoid
python3 tests/run_tests.py
```

`lake build` checks every proof in `Proofs/`, and fails if one is broken.
`lake build` checks every proof in `Utf/Proofs/`, and fails if one is broken.

`lake comparator` is part of Lake in Lean 4.35.0, the toolchain that `lean-toolchain` pins and elan installs. `lake comparator --paranoid` builds the statements and the proofs in a sandbox. It checks that each theorem in `Proofs/` proves exactly the statement in `Spec.lean`, using only the axioms `propext`, `Classical.choice`, and `Quot.sound`, and replays it through every kernel shipped with Lean.
`lake comparator` is part of Lake in Lean 4.35.0, the toolchain that `lean-toolchain` pins and elan installs. `lake comparator --paranoid` builds the statements and the proofs in a sandbox. It checks that each theorem in `Utf/Proofs/` proves exactly the statement in `Utf/Spec.lean`, using only the axioms `propext`, `Classical.choice`, and `Quot.sound`, and replays it through every kernel shipped with Lean.

The test script compares the tool with Python's UTF-8 and UTF-16 codecs, strict and with the `replace` error handler. For UTF-8 it runs every byte string of one and two bytes, a grid of three- and four-byte strings, and random and mutated inputs from a recorded seed. For UTF-16 it runs every single code unit, every pair and triple of units at the edges of the surrogate ranges, every byte string of one and two bytes through each byte decoder, and random and mutated inputs. It then rewrites `tests/TEST-RESULTS.md`.

## Layout

- `Utf8/Codec.lean`, `Utf8/Lossy.lean`, `Utf16/Codec.lean` and `Utf16/Bytes.lean`: the code under proof.
- `Spec.lean`: the statements, with `sorry` for proofs. A reviewer approves this file.
- `Spec/Defs.lean`: the definitions the statements use, written from RFC 3629, Table 3-7 and section 3.9.6, with checks against the RFC's examples.
- `Spec/Utf16.lean`: the definitions the UTF-16 statements use, written from RFC 2781, the Unicode Standard, the WHATWG Encoding Standard and ECMAScript, with checks against the examples of the first two and against each step of the WHATWG decoder.
- `Proofs.lean` and `Proofs/`: the proofs, under the same names.
- `Utf/Spec.lean`: the statements, with `sorry` for proofs. A reviewer approves this file.
- `Utf/Spec/Defs.lean`: the definitions the statements use, written from RFC 3629, Table 3-7 and section 3.9.6, with checks against the RFC's examples.
- `Utf/Spec/Utf16.lean`: the definitions the UTF-16 statements use, written from RFC 2781, the Unicode Standard, the WHATWG Encoding Standard and ECMAScript, with checks against the examples of the first two and against each step of the WHATWG decoder.
- `Utf/Proofs.lean` and `Utf/Proofs/`: the proofs, under the same names.
- `Main.lean`: the command-line tool the tests drive.
- `lakefile.toml`: the build. It names the package and its version, its libraries and executable, and the default targets, and sets nothing else.
- `comparator.json`: which theorems `lake comparator` checks, and which axioms it allows.
Expand Down
Loading