diff --git a/Proofs.lean b/Proofs.lean deleted file mode 100644 index a0c5328..0000000 --- a/Proofs.lean +++ /dev/null @@ -1,7 +0,0 @@ -/- - The proofs, under the names the challenge file gives them. The - theorems themselves are in `Proofs/Main.lean`, from the lemmas in the - other files. --/ - -import Proofs.Main diff --git a/README.md b/README.md index d26fbf6..48e3a7d 100644 --- a/README.md +++ b/README.md @@ -41,44 +41,44 @@ From Lean: ## What is proved -The statements are in `Spec.lean`, the definitions they use in `Spec/Defs.lean` and `Spec/Utf16.lean`, and the proofs in `Proofs/`. +The statements are in `Utf/Spec.lean`, the definitions they use in `Utf/Spec/Defs.lean` and `Utf/Spec/Utf16.lean`, and the proofs in `Utf/Proofs/`. | Claim | Theorem | |-------|---------| -| The encoder writes each character as the table of RFC 3629, section 3, sets out. | `Spec.encode_eq` | -| `decode` returns a list of characters exactly when the bytes are its encoding. So every byte string it accepts has one reading. It refuses everything else: overlong forms, surrogate halves, values past U+10FFFF, stray continuation bytes, the bytes 0xC0, 0xC1 and 0xF5 to 0xFF, and sequences cut short. | `Spec.decode_iff` | -| Decoding what the encoder wrote gives the characters back. | `Spec.decode_encode` | -| The decoder accepts exactly the byte sequences that Table 3-7 of the Unicode Standard calls well formed. | `Spec.decode_wellFormed` | -| Lean core's `String.toUTF8`, as core defines it, writes the same bytes as the encoder. | `Spec.encode_toUTF8` | -| Lean core's `String.fromUTF8?`, as core defines it, accepts exactly the bytes the decoder accepts and reads the same characters. | `Spec.fromUTF8?_eq` | -| The examples of RFC 3629, section 7, both ways; the boundaries of each length and of the surrogates; and a selection of malformed inputs, most of them from Markus Kuhn's UTF-8 decoder test file. Every case in that file follows from `Spec.decode_iff`, and the tests run them all. | `Spec.rfc3629_examples`, `Spec.boundary_vectors`, `Spec.reject_vectors` | -| `decodeLossy` gives exactly what section 3.9.6 of the Unicode Standard, *U+FFFD Substitution of Maximal Subparts*, sets out: each well-formed sequence becomes its character, and each maximal subpart of bad bytes becomes one U+FFFD. | `Spec.decodeLossy_iff` | -| On well-formed bytes, `decodeLossy` reads what `decode` reads; on any other bytes it gives at least one U+FFFD. | `Spec.decodeLossy_of_decode`, `Spec.decodeLossy_replaces` | -| Well-formed text before or after bad bytes comes through unchanged: no U+FFFD takes in any of it. | `Spec.decodeLossy_encode_append`, `Spec.decodeLossy_append_encode` | -| The examples of Tables 3-8 to 3-11 of the Unicode Standard, and three cut-short sequences followed by a character. | `Spec.substitution_vectors` | -| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Spec.utf16_encode_eq` | -| `Utf16.decode` returns a list of characters exactly when the code units are its encoding. So every unit list it accepts has one reading, and it refuses everything else. | `Spec.utf16_decode_iff` | -| Decoding what the UTF-16 encoder wrote gives the characters back. | `Spec.utf16_decode_encode` | -| `Utf16.decode` accepts exactly the unit lists that D91 and Table 3-5 of the Unicode Standard call well formed. | `Spec.utf16_decode_wellFormed` | -| `Utf16.decode` fails exactly when some unit is an isolated surrogate: a high surrogate with no low surrogate after it, or a low surrogate with no high surrogate before it. | `Spec.utf16_decode_none_iff` | -| The number of UTF-16 units is one for each character, and one more for each character from U+10000 up. | `Spec.utf16_encode_length` | -| `Utf16.decodeLossy` gives exactly the conversion of section 3.9.6, written for UTF-16 code units. | `Spec.utf16_decodeLossy_iff` | -| An isolated surrogate becomes one U+FFFD, and decoding goes on at the very next unit, so a high surrogate never takes the unit after it with it. That is what section 3.9.6 comes to in UTF-16. The proofs go through a lemma, `Utf16.Proofs.maximalSubpart_iff`, that each maximal subpart in UTF-16 is one isolated surrogate. It is a fact about the definitions alone, so the bar does not take it as one of the approved statements. | `Spec.utf16_decodeLossy_unpaired` | -| On well-formed units, `Utf16.decodeLossy` reads what `Utf16.decode` reads; on any other units it gives at least one U+FFFD. | `Spec.utf16_decodeLossy_of_decode`, `Spec.utf16_decodeLossy_replaces` | -| Well-formed text before or after bad units comes through unchanged. | `Spec.utf16_decodeLossy_encode_append`, `Spec.utf16_decodeLossy_append_encode` | -| `toWellFormed` keeps the number of units, puts 0xFFFD in place of each isolated surrogate and keeps every other unit, as ECMAScript's `String.prototype.toWellFormed` does. | `Spec.utf16_toWellFormed` | -| UTF-16BE writes each unit's high byte first, and UTF-16LE its low byte first, with no byte order mark. | `Spec.utf16_encodeBytes` | -| The UTF-16BE and UTF-16LE decoders accept exactly the encodings, so they refuse an odd number of bytes and read a leading byte order mark as U+FEFF. | `Spec.utf16be_decode_iff`, `Spec.utf16le_decode_iff` | -| The decoder for the UTF-16 encoding scheme reads a byte order mark as D98 and RFC 2781, section 4.3, say: FE FF for big-endian, FF FE for little-endian, the mark not part of the text, and big-endian with no mark. | `Spec.utf16_decodeScheme_iff` | -| The lossy byte decoders give exactly what the WHATWG Encoding Standard's UTF-16BE and UTF-16LE decoders give, run as the standard writes them. | `Spec.utf16_whatwg` | -| On whole units, the lossy byte decoders give the conversion of section 3.9.6, so the WHATWG decoders and the Unicode Standard agree. | `Spec.utf16_decodeLossyBytes_units` | -| The examples of D91, D96 and D97 of the Unicode Standard and of RFC 2781, section 5, both ways, and the texts of D98 and RFC 2781 with a byte order mark, read; the ends of the ranges; byte order marks; isolated surrogates refused and replaced; and a high surrogate cut off by an odd byte, which the WHATWG decoders turn into one U+FFFD. | `Spec.utf16_vectors` | +| The encoder writes each character as the table of RFC 3629, section 3, sets out. | `Utf.Spec.encode_eq` | +| `decode` returns a list of characters exactly when the bytes are its encoding. So every byte string it accepts has one reading. It refuses everything else: overlong forms, surrogate halves, values past U+10FFFF, stray continuation bytes, the bytes 0xC0, 0xC1 and 0xF5 to 0xFF, and sequences cut short. | `Utf.Spec.decode_iff` | +| Decoding what the encoder wrote gives the characters back. | `Utf.Spec.decode_encode` | +| The decoder accepts exactly the byte sequences that Table 3-7 of the Unicode Standard calls well formed. | `Utf.Spec.decode_wellFormed` | +| Lean core's `String.toUTF8`, as core defines it, writes the same bytes as the encoder. | `Utf.Spec.encode_toUTF8` | +| Lean core's `String.fromUTF8?`, as core defines it, accepts exactly the bytes the decoder accepts and reads the same characters. | `Utf.Spec.fromUTF8?_eq` | +| The examples of RFC 3629, section 7, both ways; the boundaries of each length and of the surrogates; and a selection of malformed inputs, most of them from Markus Kuhn's UTF-8 decoder test file. Every case in that file follows from `Utf.Spec.decode_iff`, and the tests run them all. | `Utf.Spec.rfc3629_examples`, `Utf.Spec.boundary_vectors`, `Utf.Spec.reject_vectors` | +| `decodeLossy` gives exactly what section 3.9.6 of the Unicode Standard, *U+FFFD Substitution of Maximal Subparts*, sets out: each well-formed sequence becomes its character, and each maximal subpart of bad bytes becomes one U+FFFD. | `Utf.Spec.decodeLossy_iff` | +| On well-formed bytes, `decodeLossy` reads what `decode` reads; on any other bytes it gives at least one U+FFFD. | `Utf.Spec.decodeLossy_of_decode`, `Utf.Spec.decodeLossy_replaces` | +| Well-formed text before or after bad bytes comes through unchanged: no U+FFFD takes in any of it. | `Utf.Spec.decodeLossy_encode_append`, `Utf.Spec.decodeLossy_append_encode` | +| The examples of Tables 3-8 to 3-11 of the Unicode Standard, and three cut-short sequences followed by a character. | `Utf.Spec.substitution_vectors` | +| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Utf.Spec.utf16_encode_eq` | +| `Utf16.decode` returns a list of characters exactly when the code units are its encoding. So every unit list it accepts has one reading, and it refuses everything else. | `Utf.Spec.utf16_decode_iff` | +| Decoding what the UTF-16 encoder wrote gives the characters back. | `Utf.Spec.utf16_decode_encode` | +| `Utf16.decode` accepts exactly the unit lists that D91 and Table 3-5 of the Unicode Standard call well formed. | `Utf.Spec.utf16_decode_wellFormed` | +| `Utf16.decode` fails exactly when some unit is an isolated surrogate: a high surrogate with no low surrogate after it, or a low surrogate with no high surrogate before it. | `Utf.Spec.utf16_decode_none_iff` | +| The number of UTF-16 units is one for each character, and one more for each character from U+10000 up. | `Utf.Spec.utf16_encode_length` | +| `Utf16.decodeLossy` gives exactly the conversion of section 3.9.6, written for UTF-16 code units. | `Utf.Spec.utf16_decodeLossy_iff` | +| An isolated surrogate becomes one U+FFFD, and decoding goes on at the very next unit, so a high surrogate never takes the unit after it with it. That is what section 3.9.6 comes to in UTF-16. The proofs go through a lemma, `Utf16.Proofs.maximalSubpart_iff`, that each maximal subpart in UTF-16 is one isolated surrogate. It is a fact about the definitions alone, so the bar does not take it as one of the approved statements. | `Utf.Spec.utf16_decodeLossy_unpaired` | +| On well-formed units, `Utf16.decodeLossy` reads what `Utf16.decode` reads; on any other units it gives at least one U+FFFD. | `Utf.Spec.utf16_decodeLossy_of_decode`, `Utf.Spec.utf16_decodeLossy_replaces` | +| Well-formed text before or after bad units comes through unchanged. | `Utf.Spec.utf16_decodeLossy_encode_append`, `Utf.Spec.utf16_decodeLossy_append_encode` | +| `toWellFormed` keeps the number of units, puts 0xFFFD in place of each isolated surrogate and keeps every other unit, as ECMAScript's `String.prototype.toWellFormed` does. | `Utf.Spec.utf16_toWellFormed` | +| UTF-16BE writes each unit's high byte first, and UTF-16LE its low byte first, with no byte order mark. | `Utf.Spec.utf16_encodeBytes` | +| The UTF-16BE and UTF-16LE decoders accept exactly the encodings, so they refuse an odd number of bytes and read a leading byte order mark as U+FEFF. | `Utf.Spec.utf16be_decode_iff`, `Utf.Spec.utf16le_decode_iff` | +| The decoder for the UTF-16 encoding scheme reads a byte order mark as D98 and RFC 2781, section 4.3, say: FE FF for big-endian, FF FE for little-endian, the mark not part of the text, and big-endian with no mark. | `Utf.Spec.utf16_decodeScheme_iff` | +| The lossy byte decoders give exactly what the WHATWG Encoding Standard's UTF-16BE and UTF-16LE decoders give, run as the standard writes them. | `Utf.Spec.utf16_whatwg` | +| On whole units, the lossy byte decoders give the conversion of section 3.9.6, so the WHATWG decoders and the Unicode Standard agree. | `Utf.Spec.utf16_decodeLossyBytes_units` | +| The examples of D91, D96 and D97 of the Unicode Standard and of RFC 2781, section 5, both ways, and the texts of D98 and RFC 2781 with a byte order mark, read; the ends of the ranges; byte order marks; isolated surrogates refused and replaced; and a high surrogate cut off by an odd byte, which the WHATWG decoders turn into one U+FFFD. | `Utf.Spec.utf16_vectors` | The command-line tool in `Main.lean` answers `decode`, `lossy` and `encode` requests, and each UTF-16 request, with these functions directly. ## What is not proved -- Lean core's compiled check. At run time, `String.fromUTF8?` runs C code, `lean_string_validate_utf8`, in place of the definition `Spec.fromUTF8?_eq` is about. The differential tests compare that C code with the proved decoder. +- Lean core's compiled check. At run time, `String.fromUTF8?` runs C code, `lean_string_validate_utf8`, in place of the definition `Utf.Spec.fromUTF8?_eq` is about. The differential tests compare that C code with the proved decoder. - Reading requests and printing answers in `Main.lean`: the hex parsing and printing, and the request format. The differential tests cover them, with malformed requests and CR LF line endings. - Other replacement policies. The lossy decoder follows section 3.9.6 only. It has no mode that puts U+FFFD in place of each bad byte on its own, as a Go `range` loop does. A byte order mark is read as a character, not stripped. - For UTF-16: byte order marks in the lossy byte decoders. They are the WHATWG decoders themselves, which read a leading FE FF (UTF-16BE) or FF FE (UTF-16LE) as U+FEFF. The standard's other steps are not here. Its `decode` algorithm sniffs a mark first, and a leading EF BB BF, FE FF or FF FE then picks the encoding and is skipped. `TextDecoder`, unless `ignoreBOM` is set, drops a U+FEFF that the decoder gives at the start. @@ -118,19 +118,19 @@ lake comparator --paranoid python3 tests/run_tests.py ``` -`lake build` checks every proof in `Proofs/`, and fails if one is broken. +`lake build` checks every proof in `Utf/Proofs/`, and fails if one is broken. -`lake comparator` is part of Lake in Lean 4.35.0, the toolchain that `lean-toolchain` pins and elan installs. `lake comparator --paranoid` builds the statements and the proofs in a sandbox. It checks that each theorem in `Proofs/` proves exactly the statement in `Spec.lean`, using only the axioms `propext`, `Classical.choice`, and `Quot.sound`, and replays it through every kernel shipped with Lean. +`lake comparator` is part of Lake in Lean 4.35.0, the toolchain that `lean-toolchain` pins and elan installs. `lake comparator --paranoid` builds the statements and the proofs in a sandbox. It checks that each theorem in `Utf/Proofs/` proves exactly the statement in `Utf/Spec.lean`, using only the axioms `propext`, `Classical.choice`, and `Quot.sound`, and replays it through every kernel shipped with Lean. The test script compares the tool with Python's UTF-8 and UTF-16 codecs, strict and with the `replace` error handler. For UTF-8 it runs every byte string of one and two bytes, a grid of three- and four-byte strings, and random and mutated inputs from a recorded seed. For UTF-16 it runs every single code unit, every pair and triple of units at the edges of the surrogate ranges, every byte string of one and two bytes through each byte decoder, and random and mutated inputs. It then rewrites `tests/TEST-RESULTS.md`. ## Layout - `Utf8/Codec.lean`, `Utf8/Lossy.lean`, `Utf16/Codec.lean` and `Utf16/Bytes.lean`: the code under proof. -- `Spec.lean`: the statements, with `sorry` for proofs. A reviewer approves this file. -- `Spec/Defs.lean`: the definitions the statements use, written from RFC 3629, Table 3-7 and section 3.9.6, with checks against the RFC's examples. -- `Spec/Utf16.lean`: the definitions the UTF-16 statements use, written from RFC 2781, the Unicode Standard, the WHATWG Encoding Standard and ECMAScript, with checks against the examples of the first two and against each step of the WHATWG decoder. -- `Proofs.lean` and `Proofs/`: the proofs, under the same names. +- `Utf/Spec.lean`: the statements, with `sorry` for proofs. A reviewer approves this file. +- `Utf/Spec/Defs.lean`: the definitions the statements use, written from RFC 3629, Table 3-7 and section 3.9.6, with checks against the RFC's examples. +- `Utf/Spec/Utf16.lean`: the definitions the UTF-16 statements use, written from RFC 2781, the Unicode Standard, the WHATWG Encoding Standard and ECMAScript, with checks against the examples of the first two and against each step of the WHATWG decoder. +- `Utf/Proofs.lean` and `Utf/Proofs/`: the proofs, under the same names. - `Main.lean`: the command-line tool the tests drive. - `lakefile.toml`: the build. It names the package and its version, its libraries and executable, and the default targets, and sets nothing else. - `comparator.json`: which theorems `lake comparator` checks, and which axioms it allows. diff --git a/SPEC.md b/SPEC.md index c3c29f5..f78bb45 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,6 +1,6 @@ # Specification: utf -This library is a UTF-8 encoder, a validating decoder, and a lossy decoder, and the same three for UTF-16. It turns Unicode scalar values into bytes, or into UTF-16 code units, and back. A reviewer who did not write this specification approves it. The approval also covers `Spec.lean`, `Spec/Defs.lean`, `Spec/Utf16.lean`, and the other files `bar.toml` lists. +This library is a UTF-8 encoder, a validating decoder, and a lossy decoder, and the same three for UTF-16. It turns Unicode scalar values into bytes, or into UTF-16 code units, and back. A reviewer who did not write this specification approves it. The approval also covers `Utf/Spec.lean`, `Utf/Spec/Defs.lean`, `Utf/Spec/Utf16.lean`, and the other files `bar.toml` lists. Characters are Lean's `Char`, which holds exactly the Unicode scalar values: 0 to 0xD7FF and 0xE000 to 0x10FFFF. Bytes are `List UInt8`, and UTF-16 code units `List UInt16`. @@ -57,7 +57,7 @@ At run time, core checks the bytes with C code (`lean_string_validate_utf8`), wh ## UTF-16 -The sources are RFC 2781, *UTF-16, an encoding of ISO 10646*, and the Unicode Standard, version 16.0: D72 to D75 for surrogates, D91 and Table 3-5 for the UTF-16 encoding form, D96 to D98 for the encoding schemes, and section 3.9.6 again for lossy decoding. The lossy byte decoders follow the shared UTF-16 decoder of the WHATWG Encoding Standard, section 14.2.1, and `toWellFormed` follows ECMAScript's `String.prototype.toWellFormed`. `Spec/Utf16.lean` quotes RFC 2781, the Unicode Standard and the WHATWG decoder, and sets out ECMAScript's rule in words. +The sources are RFC 2781, *UTF-16, an encoding of ISO 10646*, and the Unicode Standard, version 16.0: D72 to D75 for surrogates, D91 and Table 3-5 for the UTF-16 encoding form, D96 to D98 for the encoding schemes, and section 3.9.6 again for lossy decoding. The lossy byte decoders follow the shared UTF-16 decoder of the WHATWG Encoding Standard, section 14.2.1, and `toWellFormed` follows ECMAScript's `String.prototype.toWellFormed`. `Utf/Spec/Utf16.lean` quotes RFC 2781, the Unicode Standard and the WHATWG decoder, and sets out ECMAScript's rule in words. ### Code units @@ -104,50 +104,50 @@ Like the WHATWG decoders themselves, they read a leading mark in their own order ## What must always hold -Each property is a theorem in `Spec.lean`, under the name given. The definitions the statements use are in `Spec/Defs.lean` and, for UTF-16, `Spec/Utf16.lean`. They are written from RFC 3629, Table 3-7 and section 3.9.6 (D93a, D93b). Those from RFC 3629 and Table 3-7 are checked against the RFC's examples and against the first and last sequence of each row of the table and the bytes just outside its edges. Those from section 3.9.6 have two direct checks there, and are checked against the standard's examples, Tables 3-8 to 3-11, through `Spec.substitution_vectors` together with `Spec.decodeLossy_iff`. +Each property is a theorem in `Utf/Spec.lean`, under the name given. The definitions the statements use are in `Utf/Spec/Defs.lean` and, for UTF-16, `Utf/Spec/Utf16.lean`. They are written from RFC 3629, Table 3-7 and section 3.9.6 (D93a, D93b). Those from RFC 3629 and Table 3-7 are checked against the RFC's examples and against the first and last sequence of each row of the table and the bytes just outside its edges. Those from section 3.9.6 have two direct checks there, and are checked against the standard's examples, Tables 3-8 to 3-11, through `Utf.Spec.substitution_vectors` together with `Utf.Spec.decodeLossy_iff`. | Property | Theorem | |----------|---------| -| The encoder writes each character as the table of RFC 3629, section 3, sets out. | `Spec.encode_eq` | -| `decode` returns `cs` exactly when the bytes are the encoding of `cs`, so every byte string it accepts has one reading, and it refuses everything listed above. | `Spec.decode_iff` | -| Decoding what the encoder wrote gives the characters back. | `Spec.decode_encode` | -| The decoder accepts exactly the byte sequences that Table 3-7 calls well formed. | `Spec.decode_wellFormed` | -| Lean core's `String.toUTF8`, as core defines it, writes the same bytes as the encoder. | `Spec.encode_toUTF8` | -| Lean core's `String.fromUTF8?`, as core defines it, agrees with the decoder. | `Spec.fromUTF8?_eq` | -| The examples of RFC 3629, section 7, both ways. | `Spec.rfc3629_examples` | -| The first and last code points of each length, those on either side of the surrogates, and the empty input. | `Spec.boundary_vectors` | -| A selection of malformed inputs, most of them from Markus Kuhn's UTF-8 decoder test file, and each byte from 0xF5 to 0xFF. Every case in that file follows from `Spec.decode_iff`, and the tests run them all. | `Spec.reject_vectors` | -| `decodeLossy` gives exactly the conversion of section 3.9.6: each character read as `decode` reads it, and one U+FFFD for each maximal subpart. | `Spec.decodeLossy_iff` | -| On well-formed bytes, `decodeLossy` reads what `decode` reads. | `Spec.decodeLossy_of_decode` | -| On bytes that are not well formed, `decodeLossy` gives at least one U+FFFD. | `Spec.decodeLossy_replaces` | -| Well-formed text before or after bad bytes comes through unchanged. | `Spec.decodeLossy_encode_append`, `Spec.decodeLossy_append_encode` | -| The examples of section 3.9.6, Tables 3-8 to 3-11, and three cut-short sequences followed by a character. | `Spec.substitution_vectors` | -| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Spec.utf16_encode_eq` | -| `Utf16.decode` returns `cs` exactly when the units are the encoding of `cs`. | `Spec.utf16_decode_iff` | -| Decoding what the UTF-16 encoder wrote gives the characters back. | `Spec.utf16_decode_encode` | -| `Utf16.decode` accepts exactly the unit lists that D91 and Table 3-5 call well formed. | `Spec.utf16_decode_wellFormed` | -| `Utf16.decode` fails exactly when some unit is an isolated surrogate. | `Spec.utf16_decode_none_iff` | -| The number of units is one for each character and one more for each character from U+10000 up. | `Spec.utf16_encode_length` | -| `Utf16.decodeLossy` gives exactly the conversion of section 3.9.6, for UTF-16. | `Spec.utf16_decodeLossy_iff` | -| An isolated surrogate becomes one U+FFFD, and decoding goes on at the very next unit. | `Spec.utf16_decodeLossy_unpaired` | -| On well-formed units, `Utf16.decodeLossy` reads what `Utf16.decode` reads; on any other units it gives at least one U+FFFD. | `Spec.utf16_decodeLossy_of_decode`, `Spec.utf16_decodeLossy_replaces` | -| Well-formed text before or after bad units comes through unchanged. | `Spec.utf16_decodeLossy_encode_append`, `Spec.utf16_decodeLossy_append_encode` | -| `toWellFormed` keeps the number of units, puts 0xFFFD in place of each isolated surrogate, and keeps every other unit. | `Spec.utf16_toWellFormed` | -| `encodeBE` and `encodeLE` write the bytes of RFC 2781, section 3.1, and D96 and D97. | `Spec.utf16_encodeBytes` | -| `decodeBE` and `decodeLE` return `cs` exactly when the bytes are its UTF-16BE or UTF-16LE encoding. | `Spec.utf16be_decode_iff`, `Spec.utf16le_decode_iff` | -| `decodeScheme` reads the UTF-16 encoding scheme as D98 and RFC 2781, section 4.3, say. | `Spec.utf16_decodeScheme_iff` | -| `decodeLossyBE` and `decodeLossyLE` give exactly what the WHATWG UTF-16BE and UTF-16LE decoders give. | `Spec.utf16_whatwg` | -| On whole units, the lossy byte decoders give the conversion of section 3.9.6. | `Spec.utf16_decodeLossyBytes_units` | -| The examples of D91, D96, D97 and RFC 2781, section 5, both ways, and D98's and RFC 2781's texts with a mark, read; the ends of the ranges; marks; isolated surrogates refused and replaced; and a high surrogate cut off by an odd byte. | `Spec.utf16_vectors` | +| The encoder writes each character as the table of RFC 3629, section 3, sets out. | `Utf.Spec.encode_eq` | +| `decode` returns `cs` exactly when the bytes are the encoding of `cs`, so every byte string it accepts has one reading, and it refuses everything listed above. | `Utf.Spec.decode_iff` | +| Decoding what the encoder wrote gives the characters back. | `Utf.Spec.decode_encode` | +| The decoder accepts exactly the byte sequences that Table 3-7 calls well formed. | `Utf.Spec.decode_wellFormed` | +| Lean core's `String.toUTF8`, as core defines it, writes the same bytes as the encoder. | `Utf.Spec.encode_toUTF8` | +| Lean core's `String.fromUTF8?`, as core defines it, agrees with the decoder. | `Utf.Spec.fromUTF8?_eq` | +| The examples of RFC 3629, section 7, both ways. | `Utf.Spec.rfc3629_examples` | +| The first and last code points of each length, those on either side of the surrogates, and the empty input. | `Utf.Spec.boundary_vectors` | +| A selection of malformed inputs, most of them from Markus Kuhn's UTF-8 decoder test file, and each byte from 0xF5 to 0xFF. Every case in that file follows from `Utf.Spec.decode_iff`, and the tests run them all. | `Utf.Spec.reject_vectors` | +| `decodeLossy` gives exactly the conversion of section 3.9.6: each character read as `decode` reads it, and one U+FFFD for each maximal subpart. | `Utf.Spec.decodeLossy_iff` | +| On well-formed bytes, `decodeLossy` reads what `decode` reads. | `Utf.Spec.decodeLossy_of_decode` | +| On bytes that are not well formed, `decodeLossy` gives at least one U+FFFD. | `Utf.Spec.decodeLossy_replaces` | +| Well-formed text before or after bad bytes comes through unchanged. | `Utf.Spec.decodeLossy_encode_append`, `Utf.Spec.decodeLossy_append_encode` | +| The examples of section 3.9.6, Tables 3-8 to 3-11, and three cut-short sequences followed by a character. | `Utf.Spec.substitution_vectors` | +| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Utf.Spec.utf16_encode_eq` | +| `Utf16.decode` returns `cs` exactly when the units are the encoding of `cs`. | `Utf.Spec.utf16_decode_iff` | +| Decoding what the UTF-16 encoder wrote gives the characters back. | `Utf.Spec.utf16_decode_encode` | +| `Utf16.decode` accepts exactly the unit lists that D91 and Table 3-5 call well formed. | `Utf.Spec.utf16_decode_wellFormed` | +| `Utf16.decode` fails exactly when some unit is an isolated surrogate. | `Utf.Spec.utf16_decode_none_iff` | +| The number of units is one for each character and one more for each character from U+10000 up. | `Utf.Spec.utf16_encode_length` | +| `Utf16.decodeLossy` gives exactly the conversion of section 3.9.6, for UTF-16. | `Utf.Spec.utf16_decodeLossy_iff` | +| An isolated surrogate becomes one U+FFFD, and decoding goes on at the very next unit. | `Utf.Spec.utf16_decodeLossy_unpaired` | +| On well-formed units, `Utf16.decodeLossy` reads what `Utf16.decode` reads; on any other units it gives at least one U+FFFD. | `Utf.Spec.utf16_decodeLossy_of_decode`, `Utf.Spec.utf16_decodeLossy_replaces` | +| Well-formed text before or after bad units comes through unchanged. | `Utf.Spec.utf16_decodeLossy_encode_append`, `Utf.Spec.utf16_decodeLossy_append_encode` | +| `toWellFormed` keeps the number of units, puts 0xFFFD in place of each isolated surrogate, and keeps every other unit. | `Utf.Spec.utf16_toWellFormed` | +| `encodeBE` and `encodeLE` write the bytes of RFC 2781, section 3.1, and D96 and D97. | `Utf.Spec.utf16_encodeBytes` | +| `decodeBE` and `decodeLE` return `cs` exactly when the bytes are its UTF-16BE or UTF-16LE encoding. | `Utf.Spec.utf16be_decode_iff`, `Utf.Spec.utf16le_decode_iff` | +| `decodeScheme` reads the UTF-16 encoding scheme as D98 and RFC 2781, section 4.3, say. | `Utf.Spec.utf16_decodeScheme_iff` | +| `decodeLossyBE` and `decodeLossyLE` give exactly what the WHATWG UTF-16BE and UTF-16LE decoders give. | `Utf.Spec.utf16_whatwg` | +| On whole units, the lossy byte decoders give the conversion of section 3.9.6. | `Utf.Spec.utf16_decodeLossyBytes_units` | +| The examples of D91, D96, D97 and RFC 2781, section 5, both ways, and D98's and RFC 2781's texts with a mark, read; the ends of the ranges; marks; isolated surrogates refused and replaced; and a high surrogate cut off by an odd byte. | `Utf.Spec.utf16_vectors` | Together, these fix every output. - `encode` is fixed for every input. - `decode` is fixed for every input. -- `decodeLossy` is fixed for every input, by `Spec.decodeLossy_iff`. -- A decoder that accepted nothing would fail `Spec.decode_encode`. -- A decoder that accepted too much would fail `Spec.decode_iff`. -- For UTF-16, the same holds by the statements of the same names. Each byte function is fixed for every input: the encoders and the strict decoders by their own statements, `decodeScheme` by `Spec.utf16_decodeScheme_iff`, and the lossy byte decoders by `Spec.utf16_whatwg`. `toWellFormed` is fixed unit by unit. +- `decodeLossy` is fixed for every input, by `Utf.Spec.decodeLossy_iff`. +- A decoder that accepted nothing would fail `Utf.Spec.decode_encode`. +- A decoder that accepted too much would fail `Utf.Spec.decode_iff`. +- For UTF-16, the same holds by the statements of the same names. Each byte function is fixed for every input: the encoders and the strict decoders by their own statements, `decodeScheme` by `Utf.Spec.utf16_decodeScheme_iff`, and the lossy byte decoders by `Utf.Spec.utf16_whatwg`. `toWellFormed` is fixed unit by unit. ## Oracle diff --git a/Utf/Proofs.lean b/Utf/Proofs.lean new file mode 100644 index 0000000..8d07af4 --- /dev/null +++ b/Utf/Proofs.lean @@ -0,0 +1,7 @@ +/- + The proofs, under the names the challenge file gives them. The + theorems themselves are in `Utf/Proofs/Main.lean`, from the lemmas in the + other files. +-/ + +import Utf.Proofs.Main diff --git a/Proofs/Core.lean b/Utf/Proofs/Core.lean similarity index 95% rename from Proofs/Core.lean rename to Utf/Proofs/Core.lean index 8400b20..5ca6509 100644 --- a/Proofs/Core.lean +++ b/Utf/Proofs/Core.lean @@ -6,9 +6,9 @@ -/ import Utf8 -import Spec.Defs -import Proofs.Encode -import Proofs.Decode +import Utf.Spec.Defs +import Utf.Proofs.Encode +import Utf.Proofs.Decode namespace Utf8.Proofs.Core diff --git a/Proofs/Decode.lean b/Utf/Proofs/Decode.lean similarity index 99% rename from Proofs/Decode.lean rename to Utf/Proofs/Decode.lean index a98aa25..c1cf16c 100644 --- a/Proofs/Decode.lean +++ b/Utf/Proofs/Decode.lean @@ -7,7 +7,7 @@ -/ import Utf8 -import Spec.Defs +import Utf.Spec.Defs namespace Utf8.Proofs.Decode diff --git a/Proofs/Encode.lean b/Utf/Proofs/Encode.lean similarity index 89% rename from Proofs/Encode.lean rename to Utf/Proofs/Encode.lean index 5f7ba42..529388d 100644 --- a/Proofs/Encode.lean +++ b/Utf/Proofs/Encode.lean @@ -1,13 +1,13 @@ /- The encoder of `Utf8/Codec.lean` against the specification and against Lean core: `Utf8.encodeChar` writes the same bytes as the - bit-level table of RFC 3629 (`Spec.encodeChar`), and the same bytes - as core's `String.utf8EncodeChar`; so `Utf8.encode` is `Spec.encode`, + bit-level table of RFC 3629 (`Utf.Spec.encodeChar`), and the same bytes + as core's `String.utf8EncodeChar`; so `Utf8.encode` is `Utf.Spec.encode`, and on a string it gives the bytes of `String.toUTF8`. -/ import Utf8 -import Spec.Defs +import Utf.Spec.Defs namespace Utf8.Proofs.Encode @@ -24,8 +24,8 @@ theorem and_63 (x : Nat) : x &&& 0b111111 = x % 64 := Nat.and_two_pow_sub_one_eq_mod x 6 /-- A continuation byte as arithmetic: the marker 0x80 plus six bits. -/ -theorem contByte_eq (n k : Nat) : Spec.contByte n k = (0x80 + n / 2 ^ (6 * k) % 0x40).toUInt8 := by - unfold Spec.contByte +theorem contByte_eq (n k : Nat) : Utf.Spec.contByte n k = (0x80 + n / 2 ^ (6 * k) % 0x40).toUInt8 := by + unfold Utf.Spec.contByte rw [Nat.shiftRight_eq_div_pow, and_63, or_eq_add 128 _ 6 rfl (Nat.mod_lt _ (by decide))] theorem lead2 (n : Nat) (h : n < 0x800) : (0b11000000 ||| n >>> 6) = 0xC0 + n / 0x40 := by @@ -45,9 +45,9 @@ theorem toNat_lt (c : Char) : c.toNat < 0x110000 := by /-! ## One character -/ -theorem encodeChar_eq (c : Char) : Utf8.encodeChar c = Spec.encodeChar c := by +theorem encodeChar_eq (c : Char) : Utf8.encodeChar c = Utf.Spec.encodeChar c := by have hc := toNat_lt c - unfold Utf8.encodeChar Spec.encodeChar + unfold Utf8.encodeChar Utf.Spec.encodeChar simp only [contByte_eq] generalize c.toNat = n at * by_cases h1 : n < 0x80 @@ -60,8 +60,8 @@ theorem encodeChar_eq (c : Char) : Utf8.encodeChar c = Spec.encodeChar c := by · simp [h1, h2, h3, show ¬ n ≤ 0x7F by omega, show ¬ n ≤ 0x7FF by omega, show ¬ n ≤ 0xFFFF by omega, lead4 n hc] -theorem encode_eq (cs : List Char) : Utf8.encode cs = Spec.encode cs := by - unfold Utf8.encode Spec.encode +theorem encode_eq (cs : List Char) : Utf8.encode cs = Utf.Spec.encode cs := by + unfold Utf8.encode Utf.Spec.encode induction cs with | nil => rfl | cons c cs ih => simp [List.flatMap_cons, ih, encodeChar_eq] diff --git a/Proofs/Lossy/Defs.lean b/Utf/Proofs/Lossy/Defs.lean similarity index 97% rename from Proofs/Lossy/Defs.lean rename to Utf/Proofs/Lossy/Defs.lean index 5f49164..4d23893 100644 --- a/Proofs/Lossy/Defs.lean +++ b/Utf/Proofs/Lossy/Defs.lean @@ -5,7 +5,7 @@ -/ import Utf8 -import Spec.Defs +import Utf.Spec.Defs namespace Utf8.Proofs.Lossy diff --git a/Proofs/Lossy/Main.lean b/Utf/Proofs/Lossy/Main.lean similarity index 92% rename from Proofs/Lossy/Main.lean rename to Utf/Proofs/Lossy/Main.lean index e439ef5..a6794af 100644 --- a/Proofs/Lossy/Main.lean +++ b/Utf/Proofs/Lossy/Main.lean @@ -4,13 +4,13 @@ -/ import Utf8 -import Spec.Defs -import Proofs.Encode -import Proofs.Decode -import Proofs.Table -import Proofs.Lossy.Defs -import Proofs.Lossy.Table -import Proofs.Lossy.Step +import Utf.Spec.Defs +import Utf.Proofs.Encode +import Utf.Proofs.Decode +import Utf.Proofs.Table +import Utf.Proofs.Lossy.Defs +import Utf.Proofs.Lossy.Table +import Utf.Proofs.Lossy.Step namespace Utf8.Proofs.Lossy @@ -95,11 +95,11 @@ theorem mn_decodeLossy_encodeChar_append (c : Char) (rest : List UInt8) : theorem mn_maximalSubpart (b0 : UInt8) (rest : List UInt8) (h : decodeChar (b0 :: rest) = none) : - Spec.MaximalSubpart ((b0 :: rest).take (badLength b0 rest)) + Utf.Spec.MaximalSubpart ((b0 :: rest).take (badLength b0 rest)) ((b0 :: rest).drop (badLength b0 rest)) := by have hpos := badLength_pos b0 rest have hle := badLength_le b0 rest - have hu : Spec.Unconvertible (b0 :: rest) := (unconvertible_iff b0 rest).mpr h + have hu : Utf.Spec.Unconvertible (b0 :: rest) := (unconvertible_iff b0 rest).mpr h refine ⟨?_, ?_, ?_, ?_⟩ · intro h0 have := congrArg List.length h0 @@ -130,21 +130,21 @@ theorem mn_maximalSubpart (b0 : UInt8) (rest : List UInt8) (h : decodeChar (b0 : exact Bool.noConfusion this · omega -theorem mn_maximalSubpart_unique {p r p' r' : List UInt8} (h : Spec.MaximalSubpart p r) - (h' : Spec.MaximalSubpart p' r') (he : p ++ r = p' ++ r') : p = p' ∧ r = r' := by +theorem mn_maximalSubpart_unique {p r p' r' : List UInt8} (h : Utf.Spec.MaximalSubpart p r) + (h' : Utf.Spec.MaximalSubpart p' r') (he : p ++ r = p' ++ r') : p = p' ∧ r = r' := by have l1 : p'.length ≤ p.length := h.2.2.2 p' (he ▸ List.prefix_append p' r') h'.2.2.1 have l2 : p.length ≤ p'.length := h'.2.2.2 p (he ▸ List.prefix_append p r) h.2.2.1 exact List.append_inj he (by omega) theorem mn_substitutes_decodeLossy (n : Nat) : ∀ bs : List UInt8, bs.length < n → - Spec.Substitutes bs (decodeLossy bs) := by + Utf.Spec.Substitutes bs (decodeLossy bs) := by induction n with | zero => intro bs h; omega | succ n ih => intro bs hn cases bs with - | nil => exact Spec.Substitutes.nil + | nil => exact Utf.Spec.Substitutes.nil | cons b r => cases hd : decodeChar (b :: r) with | some cr => @@ -156,18 +156,18 @@ theorem mn_substitutes_decodeLossy (n : Nat) : ∀ bs : List UInt8, bs.length < have hp := mn_encodeChar_length_pos c have ih' := ih rest (by omega) rw [he, Utf8.Proofs.Encode.encodeChar_eq] - exact Spec.Substitutes.char ih' + exact Utf.Spec.Substitutes.char ih' | none => rw [mn_decodeLossy_none hd] have hpos := badLength_pos b r have ih' := ih ((b :: r).drop (badLength b r)) (by simp only [List.length_drop, List.length_cons] at hn ⊢; omega) have hm := mn_maximalSubpart b r hd - have := Spec.Substitutes.replace hm ih' + have := Utf.Spec.Substitutes.replace hm ih' rw [List.take_append_drop] at this exact this theorem mn_decodeLossy_of_substitutes {bs : List UInt8} {out : List Char} - (h : Spec.Substitutes bs out) : decodeLossy bs = out := by + (h : Utf.Spec.Substitutes bs out) : decodeLossy bs = out := by induction h with | nil => rfl | @char c rest out _ ih => @@ -186,7 +186,7 @@ theorem mn_decodeLossy_of_substitutes {bs : List UInt8} {out : List Char} rfl theorem decodeLossy_iff (bs : List UInt8) (out : List Char) : - Utf8.decodeLossy bs = out ↔ Spec.Substitutes bs out := by + Utf8.decodeLossy bs = out ↔ Utf.Spec.Substitutes bs out := by constructor · rintro rfl exact mn_substitutes_decodeLossy (bs.length + 1) bs (Nat.lt_succ_self _) diff --git a/Proofs/Lossy/Step.lean b/Utf/Proofs/Lossy/Step.lean similarity index 99% rename from Proofs/Lossy/Step.lean rename to Utf/Proofs/Lossy/Step.lean index 7c5650c..6538895 100644 --- a/Proofs/Lossy/Step.lean +++ b/Utf/Proofs/Lossy/Step.lean @@ -5,9 +5,9 @@ -/ import Utf8 -import Spec.Defs -import Proofs.Decode -import Proofs.Lossy.Defs +import Utf.Spec.Defs +import Utf.Proofs.Decode +import Utf.Proofs.Lossy.Defs namespace Utf8.Proofs.Lossy diff --git a/Proofs/Lossy/Table.lean b/Utf/Proofs/Lossy/Table.lean similarity index 72% rename from Proofs/Lossy/Table.lean rename to Utf/Proofs/Lossy/Table.lean index d8b39b3..6b187c7 100644 --- a/Proofs/Lossy/Table.lean +++ b/Utf/Proofs/Lossy/Table.lean @@ -1,13 +1,13 @@ /- - The definitions of section 3.9.6 (`Spec.Unconvertible`, - `Spec.Initial`) in terms of the strict decoder and `prefixB`. + The definitions of section 3.9.6 (`Utf.Spec.Unconvertible`, + `Utf.Spec.Initial`) in terms of the strict decoder and `prefixB`. -/ import Utf8 -import Spec.Defs -import Proofs.Decode -import Proofs.Table -import Proofs.Lossy.Defs +import Utf.Spec.Defs +import Utf.Proofs.Decode +import Utf.Proofs.Table +import Utf.Proofs.Lossy.Defs namespace Utf8.Proofs.Lossy @@ -29,14 +29,14 @@ theorem tbl_isCont_iff (b : UInt8) : isCont b = true ↔ 0x80 ≤ b.toNat ∧ b. simp [isCont] /-- An encoded character, as a well-formed sequence on its own. -/ -theorem tbl_wellFormed_encodeChar (c : Char) : Spec.WellFormed (Utf8.encodeChar c) := by - have h := Spec.WellFormed.cons (Utf8.Proofs.Table.encodeChar_wellFormed c) Spec.WellFormed.nil +theorem tbl_wellFormed_encodeChar (c : Char) : Utf.Spec.WellFormed (Utf8.encodeChar c) := by + have h := Utf.Spec.WellFormed.cons (Utf8.Proofs.Table.encodeChar_wellFormed c) Utf.Spec.WellFormed.nil rwa [List.append_nil] at h /-- A nonempty well-formed sequence starts with the bytes of one character, and the rest is well formed. -/ -theorem wellFormed_cons {s : List UInt8} (h : Spec.WellFormed s) (hs : s ≠ []) : - ∃ c t, s = Utf8.encodeChar c ++ t ∧ Spec.WellFormed t := by +theorem wellFormed_cons {s : List UInt8} (h : Utf.Spec.WellFormed s) (hs : s ≠ []) : + ∃ c t, s = Utf8.encodeChar c ++ t ∧ Utf.Spec.WellFormed t := by obtain ⟨cs, rfl⟩ := (Utf8.Proofs.Table.wellFormed_iff s).mp h match cs with | [] => exact absurd rfl hs @@ -47,7 +47,7 @@ theorem wellFormed_cons {s : List UInt8} (h : Spec.WellFormed s) (hs : s ≠ []) /-- D93a in terms of the strict decoder: no well-formed sequence starts at the front exactly when no character can be read there. -/ theorem unconvertible_iff (b0 : UInt8) (rest : List UInt8) : - Spec.Unconvertible (b0 :: rest) ↔ Utf8.decodeChar (b0 :: rest) = none := by + Utf.Spec.Unconvertible (b0 :: rest) ↔ Utf8.decodeChar (b0 :: rest) = none := by constructor · intro hu match hd : Utf8.decodeChar (b0 :: rest) with @@ -65,24 +65,24 @@ theorem unconvertible_iff (b0 : UInt8) (rest : List UInt8) : simp at this /-- A prefix of one row of Table 3-7 passes `prefixB`. -/ -theorem tbl_prefixB_of_wellFormedChar {s q : List UInt8} (hs : Spec.WellFormedChar s) +theorem tbl_prefixB_of_wellFormedChar {s q : List UInt8} (hs : Utf.Spec.WellFormedChar s) (hq : q <+: s) : prefixB q = true := by match s, hs with | [b0], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h simp only [List.prefix_cons_iff, List.prefix_nil] at hq rcases hq with rfl | ⟨t, rfl, rfl⟩ · rfl · simp [prefixB]; omega | [b0, b1], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h simp only [List.prefix_cons_iff, List.prefix_nil] at hq rcases hq with rfl | ⟨t, rfl, rfl | ⟨t, rfl, rfl⟩⟩ · rfl · simp [prefixB]; omega · simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff]; omega | [b0, b1, b2], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h simp only [List.prefix_cons_iff, List.prefix_nil] at hq rcases hq with rfl | ⟨t, rfl, rfl | ⟨t, rfl, rfl | ⟨t, rfl, rfl⟩⟩⟩ · rfl @@ -90,7 +90,7 @@ theorem tbl_prefixB_of_wellFormedChar {s q : List UInt8} (hs : Spec.WellFormedCh · simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff]; omega · simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff]; omega | [b0, b1, b2, b3], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h simp only [List.prefix_cons_iff, List.prefix_nil] at hq rcases hq with rfl | ⟨t, rfl, rfl | ⟨t, rfl, rfl | ⟨t, rfl, rfl | ⟨t, rfl, rfl⟩⟩⟩⟩ · rfl @@ -100,7 +100,7 @@ theorem tbl_prefixB_of_wellFormedChar {s q : List UInt8} (hs : Spec.WellFormedCh · simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff]; omega /-- A prefix of a row of Table 3-7 is a prefix of an encoded character. -/ -theorem tbl_prefix_encodeChar {q s : List UInt8} (hs : Spec.WellFormedChar s) (hq : q <+: s) : +theorem tbl_prefix_encodeChar {q s : List UInt8} (hs : Utf.Spec.WellFormedChar s) (hq : q <+: s) : ∃ c, q <+: Utf8.encodeChar c := by obtain ⟨c, rfl⟩ := (Utf8.Proofs.Table.wellFormedChar_iff s).mp hs exact ⟨c, hq⟩ @@ -115,42 +115,42 @@ theorem tbl_encodeChar_of_prefixB (q : List UInt8) (h : prefixB q = true) : simp [prefixB] at h by_cases a1 : b0.toNat ≤ 0x7F · exact tbl_prefix_encodeChar (s := [b0]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; omega) ⟨[], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; omega) ⟨[], rfl⟩ by_cases a2 : b0.toNat ≤ 0xDF · exact tbl_prefix_encodeChar (s := [b0, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x80], rfl⟩ by_cases a3 : b0.toNat = 0xE0 · exact tbl_prefix_encodeChar (s := [b0, 0xA0, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0xA0, 0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0xA0, 0x80], rfl⟩ by_cases a4 : b0.toNat ≤ 0xEF · exact tbl_prefix_encodeChar (s := [b0, 0x80, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x80, 0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x80, 0x80], rfl⟩ by_cases a5 : b0.toNat = 0xF0 · exact tbl_prefix_encodeChar (s := [b0, 0x90, 0x80, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x90, 0x80, 0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x90, 0x80, 0x80], rfl⟩ · exact tbl_prefix_encodeChar (s := [b0, 0x80, 0x80, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x80, 0x80, 0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x80, 0x80, 0x80], rfl⟩ | [b0, b1] => simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff] at h by_cases a2 : b0.toNat ≤ 0xDF · exact tbl_prefix_encodeChar (s := [b0, b1]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; omega) ⟨[], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; omega) ⟨[], rfl⟩ by_cases a4 : b0.toNat ≤ 0xEF · exact tbl_prefix_encodeChar (s := [b0, b1, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x80], rfl⟩ · exact tbl_prefix_encodeChar (s := [b0, b1, 0x80, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x80, 0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x80, 0x80], rfl⟩ | [b0, b1, b2] => simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff] at h by_cases a4 : b0.toNat ≤ 0xEF · exact tbl_prefix_encodeChar (s := [b0, b1, b2]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; omega) ⟨[], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; omega) ⟨[], rfl⟩ · exact tbl_prefix_encodeChar (s := [b0, b1, b2, 0x80]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; simp; omega) ⟨[0x80], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; simp; omega) ⟨[0x80], rfl⟩ | [b0, b1, b2, b3] => simp [prefixB, tbl_isCont_iff, tbl_secondOK_iff] at h exact tbl_prefix_encodeChar (s := [b0, b1, b2, b3]) - (by simp only [Spec.WellFormedChar, Spec.InRange]; omega) ⟨[], rfl⟩ + (by simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange]; omega) ⟨[], rfl⟩ | _ :: _ :: _ :: _ :: _ :: _ => simp [prefixB] at h /-- `prefixB` says what it should. -/ @@ -163,8 +163,8 @@ theorem prefixB_iff (q : List UInt8) : /-- At an unconvertible offset, a prefix is the initial subsequence of a well-formed sequence exactly when `prefixB` holds of it. -/ -theorem initial_iff {q bs : List UInt8} (hu : Spec.Unconvertible bs) (hq : q <+: bs) : - Spec.Initial q ↔ prefixB q = true := by +theorem initial_iff {q bs : List UInt8} (hu : Utf.Spec.Unconvertible bs) (hq : q <+: bs) : + Utf.Spec.Initial q ↔ prefixB q = true := by constructor · rintro ⟨s, hw, hqs⟩ match q, hq, hqs with diff --git a/Proofs/Main.lean b/Utf/Proofs/Main.lean similarity index 71% rename from Proofs/Main.lean rename to Utf/Proofs/Main.lean index 2af762a..3128e8a 100644 --- a/Proofs/Main.lean +++ b/Utf/Proofs/Main.lean @@ -1,44 +1,44 @@ /- - The statements of `Spec.lean`, under the same names, from the lemmas - in the other files of `Proofs/`. The test vectors are checked by + The statements of `Utf/Spec.lean`, under the same names, from the lemmas + in the other files of `Utf/Proofs/`. The test vectors are checked by evaluation. -/ import Utf8 -import Spec.Defs -import Proofs.Encode -import Proofs.Decode -import Proofs.Table -import Proofs.Core -import Proofs.Lossy.Main -import Proofs.Utf16.Units -import Proofs.Utf16.Lossy -import Proofs.Utf16.Bytes -import Proofs.Utf16.Whatwg - -theorem Spec.encode_eq (cs : List Char) : Utf8.encode cs = Spec.encode cs := +import Utf.Spec.Defs +import Utf.Proofs.Encode +import Utf.Proofs.Decode +import Utf.Proofs.Table +import Utf.Proofs.Core +import Utf.Proofs.Lossy.Main +import Utf.Proofs.Utf16.Units +import Utf.Proofs.Utf16.Lossy +import Utf.Proofs.Utf16.Bytes +import Utf.Proofs.Utf16.Whatwg + +theorem Utf.Spec.encode_eq (cs : List Char) : Utf8.encode cs = Utf.Spec.encode cs := Utf8.Proofs.Encode.encode_eq cs -theorem Spec.decode_iff (bs : List UInt8) (cs : List Char) : - Utf8.decode bs = some cs ↔ bs = Spec.encode cs := by +theorem Utf.Spec.decode_iff (bs : List UInt8) (cs : List Char) : + Utf8.decode bs = some cs ↔ bs = Utf.Spec.encode cs := by rw [Utf8.Proofs.Decode.decode_iff, Utf8.Proofs.Encode.encode_eq] -theorem Spec.decode_encode (cs : List Char) : Utf8.decode (Utf8.encode cs) = some cs := +theorem Utf.Spec.decode_encode (cs : List Char) : Utf8.decode (Utf8.encode cs) = some cs := Utf8.Proofs.Decode.decode_encode cs -theorem Spec.decode_wellFormed (bs : List UInt8) : - (Utf8.decode bs).isSome ↔ Spec.WellFormed bs := by +theorem Utf.Spec.decode_wellFormed (bs : List UInt8) : + (Utf8.decode bs).isSome ↔ Utf.Spec.WellFormed bs := by rw [Utf8.Proofs.Table.wellFormed_iff, Option.isSome_iff_exists] simp only [Utf8.Proofs.Decode.decode_iff] -theorem Spec.encode_toUTF8 (s : String) : Utf8.encode s.toList = s.toUTF8.toList := +theorem Utf.Spec.encode_toUTF8 (s : String) : Utf8.encode s.toList = s.toUTF8.toList := Utf8.Proofs.Encode.encode_toUTF8 s -theorem Spec.fromUTF8?_eq (bs : List UInt8) : +theorem Utf.Spec.fromUTF8?_eq (bs : List UInt8) : (String.fromUTF8? bs.toByteArray).map String.toList = Utf8.decode bs := Utf8.Proofs.Core.fromUTF8?_eq bs -theorem Spec.rfc3629_examples : +theorem Utf.Spec.rfc3629_examples : Utf8.encode ['A', '≢', '\u0391', '.'] = [0x41, 0xE2, 0x89, 0xA2, 0xCE, 0x91, 0x2E] ∧ Utf8.encode ['한', '국', '어'] = @@ -56,7 +56,7 @@ theorem Spec.rfc3629_examples : some ['\uFEFF', Char.ofNat 0x233B4] := by decide -theorem Spec.boundary_vectors : +theorem Utf.Spec.boundary_vectors : Utf8.encode ['\u0000', '\u007F'] = [0x00, 0x7F] ∧ Utf8.encode ['\u0080', '\u07FF'] = [0xC2, 0x80, 0xDF, 0xBF] ∧ Utf8.encode ['\u0800', '\uFFFF'] = [0xE0, 0xA0, 0x80, 0xEF, 0xBF, 0xBF] ∧ @@ -68,7 +68,7 @@ theorem Spec.boundary_vectors : Utf8.decode [] = some [] := by decide -theorem Spec.reject_vectors : +theorem Utf.Spec.reject_vectors : Utf8.decode [0xC0, 0xAF] = none ∧ Utf8.decode [0xE0, 0x80, 0xAF] = none ∧ Utf8.decode [0xF0, 0x80, 0x80, 0xAF] = none ∧ @@ -104,27 +104,27 @@ theorem Spec.reject_vectors : Utf8.decode [0xC2, 0x41] = none := by decide -theorem Spec.decodeLossy_iff (bs : List UInt8) (out : List Char) : - Utf8.decodeLossy bs = out ↔ Spec.Substitutes bs out := +theorem Utf.Spec.decodeLossy_iff (bs : List UInt8) (out : List Char) : + Utf8.decodeLossy bs = out ↔ Utf.Spec.Substitutes bs out := Utf8.Proofs.Lossy.decodeLossy_iff bs out -theorem Spec.decodeLossy_of_decode (bs : List UInt8) (cs : List Char) : +theorem Utf.Spec.decodeLossy_of_decode (bs : List UInt8) (cs : List Char) : Utf8.decode bs = some cs → Utf8.decodeLossy bs = cs := Utf8.Proofs.Lossy.decodeLossy_of_decode bs cs -theorem Spec.decodeLossy_replaces (bs : List UInt8) : +theorem Utf.Spec.decodeLossy_replaces (bs : List UInt8) : Utf8.decode bs = none → '�' ∈ Utf8.decodeLossy bs := Utf8.Proofs.Lossy.decodeLossy_replaces bs -theorem Spec.decodeLossy_encode_append (cs : List Char) (bs : List UInt8) : +theorem Utf.Spec.decodeLossy_encode_append (cs : List Char) (bs : List UInt8) : Utf8.decodeLossy (Utf8.encode cs ++ bs) = cs ++ Utf8.decodeLossy bs := Utf8.Proofs.Lossy.decodeLossy_encode_append cs bs -theorem Spec.decodeLossy_append_encode (bs : List UInt8) (cs : List Char) : +theorem Utf.Spec.decodeLossy_append_encode (bs : List UInt8) (cs : List Char) : Utf8.decodeLossy (bs ++ Utf8.encode cs) = Utf8.decodeLossy bs ++ cs := Utf8.Proofs.Lossy.decodeLossy_append_encode bs cs -theorem Spec.substitution_vectors : +theorem Utf.Spec.substitution_vectors : -- Table 3-8, non-shortest forms Utf8.decodeLossy [0xC0, 0xAF, 0xE0, 0x80, 0xBF, 0xF0, 0x81, 0x82, 0x41] = ['�', '�', '�', '�', '�', '�', '�', '�', 'A'] ∧ @@ -143,91 +143,91 @@ theorem Spec.substitution_vectors : Utf8.decodeLossy [0xE2, 0x82, 0x3C] = ['�', '<'] := by decide -theorem Spec.utf16_encode_eq (cs : List Char) : Utf16.encode cs = Spec.encode16 cs := +theorem Utf.Spec.utf16_encode_eq (cs : List Char) : Utf16.encode cs = Utf.Spec.encode16 cs := Utf16.Proofs.encode_eq cs -theorem Spec.utf16_decode_iff (us : List UInt16) (cs : List Char) : - Utf16.decode us = some cs ↔ us = Spec.encode16 cs := by +theorem Utf.Spec.utf16_decode_iff (us : List UInt16) (cs : List Char) : + Utf16.decode us = some cs ↔ us = Utf.Spec.encode16 cs := by rw [Utf16.Proofs.decode_iff, Utf16.Proofs.encode_eq] -theorem Spec.utf16_decode_encode (cs : List Char) : +theorem Utf.Spec.utf16_decode_encode (cs : List Char) : Utf16.decode (Utf16.encode cs) = some cs := Utf16.Proofs.decode_encode cs -theorem Spec.utf16_decode_wellFormed (us : List UInt16) : - (Utf16.decode us).isSome ↔ Spec.WellFormed16 us := by +theorem Utf.Spec.utf16_decode_wellFormed (us : List UInt16) : + (Utf16.decode us).isSome ↔ Utf.Spec.WellFormed16 us := by rw [Utf16.Proofs.wellFormed_iff, Option.isSome_iff_exists] simp only [Utf16.Proofs.decode_iff] -theorem Spec.utf16_decode_none_iff (us : List UInt16) : - Utf16.decode us = none ↔ ∃ l w r, us = l ++ w :: r ∧ Spec.Unpaired l w r := +theorem Utf.Spec.utf16_decode_none_iff (us : List UInt16) : + Utf16.decode us = none ↔ ∃ l w r, us = l ++ w :: r ∧ Utf.Spec.Unpaired l w r := Utf16.Proofs.decode_none_iff us -theorem Spec.utf16_encode_length (cs : List Char) : +theorem Utf.Spec.utf16_encode_length (cs : List Char) : (Utf16.encode cs).length = cs.length + (cs.filter fun c => 0x10000 ≤ c.toNat).length := Utf16.Proofs.encode_length cs -theorem Spec.utf16_decodeLossy_iff (us : List UInt16) (out : List Char) : - Utf16.decodeLossy us = out ↔ Spec.Substitutes16 us out := +theorem Utf.Spec.utf16_decodeLossy_iff (us : List UInt16) (out : List Char) : + Utf16.decodeLossy us = out ↔ Utf.Spec.Substitutes16 us out := Utf16.Proofs.decodeLossy_iff us out -theorem Spec.utf16_decodeLossy_unpaired (w : UInt16) (rest : List UInt16) : - Spec.Unpaired [] w rest → Utf16.decodeLossy (w :: rest) = '�' :: Utf16.decodeLossy rest := +theorem Utf.Spec.utf16_decodeLossy_unpaired (w : UInt16) (rest : List UInt16) : + Utf.Spec.Unpaired [] w rest → Utf16.decodeLossy (w :: rest) = '�' :: Utf16.decodeLossy rest := Utf16.Proofs.decodeLossy_unpaired w rest -theorem Spec.utf16_decodeLossy_of_decode (us : List UInt16) (cs : List Char) : +theorem Utf.Spec.utf16_decodeLossy_of_decode (us : List UInt16) (cs : List Char) : Utf16.decode us = some cs → Utf16.decodeLossy us = cs := Utf16.Proofs.decodeLossy_of_decode us cs -theorem Spec.utf16_decodeLossy_replaces (us : List UInt16) : +theorem Utf.Spec.utf16_decodeLossy_replaces (us : List UInt16) : Utf16.decode us = none → '�' ∈ Utf16.decodeLossy us := Utf16.Proofs.decodeLossy_replaces us -theorem Spec.utf16_decodeLossy_encode_append (cs : List Char) (us : List UInt16) : +theorem Utf.Spec.utf16_decodeLossy_encode_append (cs : List Char) (us : List UInt16) : Utf16.decodeLossy (Utf16.encode cs ++ us) = cs ++ Utf16.decodeLossy us := Utf16.Proofs.decodeLossy_encode_append cs us -theorem Spec.utf16_decodeLossy_append_encode (us : List UInt16) (cs : List Char) : +theorem Utf.Spec.utf16_decodeLossy_append_encode (us : List UInt16) (cs : List Char) : Utf16.decodeLossy (us ++ Utf16.encode cs) = Utf16.decodeLossy us ++ cs := Utf16.Proofs.decodeLossy_append_encode us cs -theorem Spec.utf16_toWellFormed (us : List UInt16) : +theorem Utf.Spec.utf16_toWellFormed (us : List UInt16) : (Utf16.toWellFormed us).length = us.length ∧ ∀ (i : Nat) (h : i < us.length), - (Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → + (Utf.Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → (Utf16.toWellFormed us)[i]? = some 0xFFFD) ∧ - (¬ Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → + (¬ Utf.Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → (Utf16.toWellFormed us)[i]? = some us[i]) := Utf16.Proofs.toWellFormed_spec us -theorem Spec.utf16_encodeBytes (cs : List Char) : - Utf16.encodeBE cs = Spec.serializeBE (Spec.encode16 cs) ∧ - Utf16.encodeLE cs = Spec.serializeLE (Spec.encode16 cs) := +theorem Utf.Spec.utf16_encodeBytes (cs : List Char) : + Utf16.encodeBE cs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs) ∧ + Utf16.encodeLE cs = Utf.Spec.serializeLE (Utf.Spec.encode16 cs) := Utf16.Proofs.encodeBytes cs -theorem Spec.utf16be_decode_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeBE bs = some cs ↔ bs = Spec.serializeBE (Spec.encode16 cs) := +theorem Utf.Spec.utf16be_decode_iff (bs : List UInt8) (cs : List Char) : + Utf16.decodeBE bs = some cs ↔ bs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs) := Utf16.Proofs.decodeBE_iff bs cs -theorem Spec.utf16le_decode_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeLE bs = some cs ↔ bs = Spec.serializeLE (Spec.encode16 cs) := +theorem Utf.Spec.utf16le_decode_iff (bs : List UInt8) (cs : List Char) : + Utf16.decodeLE bs = some cs ↔ bs = Utf.Spec.serializeLE (Utf.Spec.encode16 cs) := Utf16.Proofs.decodeLE_iff bs cs -theorem Spec.utf16_decodeScheme_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeScheme bs = some cs ↔ Spec.SchemeReads bs cs := +theorem Utf.Spec.utf16_decodeScheme_iff (bs : List UInt8) (cs : List Char) : + Utf16.decodeScheme bs = some cs ↔ Utf.Spec.SchemeReads bs cs := Utf16.Proofs.decodeScheme_iff bs cs -theorem Spec.utf16_whatwg (bs : List UInt8) : - Utf16.decodeLossyBE bs = Spec.whatwgDecode true bs ∧ - Utf16.decodeLossyLE bs = Spec.whatwgDecode false bs := +theorem Utf.Spec.utf16_whatwg (bs : List UInt8) : + Utf16.decodeLossyBE bs = Utf.Spec.whatwgDecode true bs ∧ + Utf16.decodeLossyLE bs = Utf.Spec.whatwgDecode false bs := Utf16.Proofs.whatwg bs -theorem Spec.utf16_decodeLossyBytes_units (us : List UInt16) : - Utf16.decodeLossyBE (Spec.serializeBE us) = Utf16.decodeLossy us ∧ - Utf16.decodeLossyLE (Spec.serializeLE us) = Utf16.decodeLossy us := +theorem Utf.Spec.utf16_decodeLossyBytes_units (us : List UInt16) : + Utf16.decodeLossyBE (Utf.Spec.serializeBE us) = Utf16.decodeLossy us ∧ + Utf16.decodeLossyLE (Utf.Spec.serializeLE us) = Utf16.decodeLossy us := Utf16.Proofs.decodeLossyBytes_units us -theorem Spec.utf16_vectors : +theorem Utf.Spec.utf16_vectors : -- D91 and D96 to D98: <004D 0430 4E8C 10302> Utf16.encode ['M', '\u0430', '二', Char.ofNat 0x10302] = [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] ∧ diff --git a/Proofs/Table.lean b/Utf/Proofs/Table.lean similarity index 82% rename from Proofs/Table.lean rename to Utf/Proofs/Table.lean index 3df24c2..e10d26a 100644 --- a/Proofs/Table.lean +++ b/Utf/Proofs/Table.lean @@ -7,7 +7,7 @@ -/ import Utf8 -import Spec.Defs +import Utf.Spec.Defs namespace Utf8.Proofs.Table @@ -36,29 +36,29 @@ theorem eq_toUInt8 (b : UInt8) (m : Nat) (h : b.toNat = m % 256) : b = m.toUInt8 /-! ## One character -/ /-- Each encoded character is one row of Table 3-7. -/ -theorem encodeChar_wellFormed (c : Char) : Spec.WellFormedChar (Utf8.encodeChar c) := by +theorem encodeChar_wellFormed (c : Char) : Utf.Spec.WellFormedChar (Utf8.encodeChar c) := by have hc := toNat_range c unfold Utf8.encodeChar generalize c.toNat = n at * by_cases h1 : n < 0x80 - · simp [h1, Spec.WellFormedChar, Spec.InRange] + · simp [h1, Utf.Spec.WellFormedChar, Utf.Spec.InRange] omega by_cases h2 : n < 0x800 - · simp [h1, h2, Spec.WellFormedChar, Spec.InRange] + · simp [h1, h2, Utf.Spec.WellFormedChar, Utf.Spec.InRange] omega by_cases h3 : n < 0x10000 - · simp [h1, h2, h3, Spec.WellFormedChar, Spec.InRange] + · simp [h1, h2, h3, Utf.Spec.WellFormedChar, Utf.Spec.InRange] omega - · simp [h1, h2, h3, Spec.WellFormedChar, Spec.InRange] + · simp [h1, h2, h3, Utf.Spec.WellFormedChar, Utf.Spec.InRange] omega /-- Each row of Table 3-7 is an encoded character. -/ -theorem wellFormedChar_encode (s : List UInt8) (h : Spec.WellFormedChar s) : +theorem wellFormedChar_encode (s : List UInt8) (h : Utf.Spec.WellFormedChar s) : ∃ c, s = Utf8.encodeChar c := by have hb := UInt8.toNat_lt_size match s, h with | [b0], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h have h0 := hb b0 refine ⟨Char.ofNat b0.toNat, ?_⟩ unfold Utf8.encodeChar @@ -66,7 +66,7 @@ theorem wellFormedChar_encode (s : List UInt8) (h : Spec.WellFormedChar s) : simp only [show b0.toNat < 0x80 by omega, ↓reduceIte, List.cons.injEq, and_true] exact eq_toUInt8 _ _ (by omega) | [b0, b1], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h have h0 := hb b0 have h1 := hb b1 refine ⟨Char.ofNat ((b0.toNat - 0xC0) * 0x40 + (b1.toNat - 0x80)), ?_⟩ @@ -77,7 +77,7 @@ theorem wellFormedChar_encode (s : List UInt8) (h : Spec.WellFormedChar s) : ↓reduceIte, List.cons.injEq, and_true] exact ⟨eq_toUInt8 _ _ (by omega), eq_toUInt8 _ _ (by omega)⟩ | [b0, b1, b2], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h have h0 := hb b0 have h1 := hb b1 have h2 := hb b2 @@ -94,7 +94,7 @@ theorem wellFormedChar_encode (s : List UInt8) (h : Spec.WellFormedChar s) : ↓reduceIte, List.cons.injEq, and_true] exact ⟨eq_toUInt8 _ _ (by omega), eq_toUInt8 _ _ (by omega), eq_toUInt8 _ _ (by omega)⟩ | [b0, b1, b2, b3], h => - simp only [Spec.WellFormedChar, Spec.InRange] at h + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange] at h have h0 := hb b0 have h1 := hb b1 have h2 := hb b2 @@ -113,7 +113,7 @@ theorem wellFormedChar_encode (s : List UInt8) (h : Spec.WellFormedChar s) : exact ⟨eq_toUInt8 _ _ (by omega), eq_toUInt8 _ _ (by omega), eq_toUInt8 _ _ (by omega), eq_toUInt8 _ _ (by omega)⟩ -theorem wellFormedChar_iff (s : List UInt8) : Spec.WellFormedChar s ↔ ∃ c, s = Utf8.encodeChar c := by +theorem wellFormedChar_iff (s : List UInt8) : Utf.Spec.WellFormedChar s ↔ ∃ c, s = Utf8.encodeChar c := by constructor · exact wellFormedChar_encode s · rintro ⟨c, rfl⟩ @@ -125,7 +125,7 @@ theorem encode_cons (c : Char) (cs : List Char) : Utf8.encode (c :: cs) = Utf8.encodeChar c ++ Utf8.encode cs := by simp [Utf8.encode, List.flatMap_cons] -theorem wellFormed_iff (bs : List UInt8) : Spec.WellFormed bs ↔ ∃ cs, bs = Utf8.encode cs := by +theorem wellFormed_iff (bs : List UInt8) : Utf.Spec.WellFormed bs ↔ ∃ cs, bs = Utf8.encode cs := by constructor · intro h induction h with @@ -136,9 +136,9 @@ theorem wellFormed_iff (bs : List UInt8) : Spec.WellFormed bs ↔ ∃ cs, bs = U exact ⟨c :: cs, (encode_cons c cs).symm⟩ · rintro ⟨cs, rfl⟩ induction cs with - | nil => exact Spec.WellFormed.nil + | nil => exact Utf.Spec.WellFormed.nil | cons c cs ih => rw [encode_cons] - exact Spec.WellFormed.cons (encodeChar_wellFormed c) ih + exact Utf.Spec.WellFormed.cons (encodeChar_wellFormed c) ih end Utf8.Proofs.Table diff --git a/Proofs/Utf16/Bytes.lean b/Utf/Proofs/Utf16/Bytes.lean similarity index 76% rename from Proofs/Utf16/Bytes.lean rename to Utf/Proofs/Utf16/Bytes.lean index 1f2d428..9100587 100644 --- a/Proofs/Utf16/Bytes.lean +++ b/Utf/Proofs/Utf16/Bytes.lean @@ -4,7 +4,7 @@ units. -/ -import Proofs.Utf16.Units +import Utf.Proofs.Utf16.Units namespace Utf16.Proofs @@ -21,18 +21,18 @@ theorem by_lo (u : UInt16) : (u.toNat % 0x100).toUInt8 = (u &&& 0xFF).toUInt8 := rw [show (255 : Nat) = 2^8 - 1 from rfl, Nat.and_two_pow_sub_one_eq_mod] omega -theorem by_bytesBE (us : List UInt16) : bytesBE us = Spec.serializeBE us := by +theorem by_bytesBE (us : List UInt16) : bytesBE us = Utf.Spec.serializeBE us := by induction us with | nil => rfl | cons u us ih => - simp only [bytesBE, Spec.serializeBE, List.flatMap_cons, List.map_cons, List.flatten_cons] at * + simp only [bytesBE, Utf.Spec.serializeBE, List.flatMap_cons, List.map_cons, List.flatten_cons] at * rw [ih, by_hi, by_lo] -theorem by_bytesLE (us : List UInt16) : bytesLE us = Spec.serializeLE us := by +theorem by_bytesLE (us : List UInt16) : bytesLE us = Utf.Spec.serializeLE us := by induction us with | nil => rfl | cons u us ih => - simp only [bytesLE, Spec.serializeLE, List.flatMap_cons, List.map_cons, List.flatten_cons] at * + simp only [bytesLE, Utf.Spec.serializeLE, List.flatMap_cons, List.map_cons, List.flatten_cons] at * rw [ih, by_hi, by_lo] theorem by_unit_bytes (u : UInt16) : unit (u >>> 8).toUInt8 (u &&& 0xFF).toUInt8 = u := by @@ -58,25 +58,25 @@ theorem by_lo_unit (a b : UInt8) : (unit a b &&& 0xFF).toUInt8 = b := by simp [unit] <;> omega theorem by_units_serializeBE (us : List UInt16) : - units true (Spec.serializeBE us) = (us, none) := by + units true (Utf.Spec.serializeBE us) = (us, none) := by induction us with | nil => rfl | cons u us ih => - simp only [Spec.serializeBE, List.map_cons, List.flatten_cons, List.cons_append, + simp only [Utf.Spec.serializeBE, List.map_cons, List.flatten_cons, List.cons_append, List.nil_append] at * simp only [units, ih, by_unit_bytes, ite_true] theorem by_units_serializeLE (us : List UInt16) : - units false (Spec.serializeLE us) = (us, none) := by + units false (Utf.Spec.serializeLE us) = (us, none) := by induction us with | nil => rfl | cons u us ih => - simp only [Spec.serializeLE, List.map_cons, List.flatten_cons, List.cons_append, + simp only [Utf.Spec.serializeLE, List.map_cons, List.flatten_cons, List.cons_append, List.nil_append] at * simp only [units, ih, by_unit_bytes, Bool.false_eq_true, ite_false] theorem by_serializeBE_units : (bs : List UInt8) → (us : List UInt16) → - units true bs = (us, none) → bs = Spec.serializeBE us + units true bs = (us, none) → bs = Utf.Spec.serializeBE us | [], us, h => by simp only [units, Prod.mk.injEq] at h rw [← h.1]; rfl @@ -91,11 +91,11 @@ theorem by_serializeBE_units : (bs : List UInt8) → (us : List UInt16) → obtain ⟨h1, h2⟩ := h subst h1 h2 rw [ih rfl] - simp only [Spec.serializeBE, List.map_cons, List.flatten_cons, by_hi_unit, by_lo_unit, + simp only [Utf.Spec.serializeBE, List.map_cons, List.flatten_cons, by_hi_unit, by_lo_unit, List.cons_append, List.nil_append] theorem by_serializeLE_units : (bs : List UInt8) → (us : List UInt16) → - units false bs = (us, none) → bs = Spec.serializeLE us + units false bs = (us, none) → bs = Utf.Spec.serializeLE us | [], us, h => by simp only [units, Prod.mk.injEq] at h rw [← h.1]; rfl @@ -110,20 +110,20 @@ theorem by_serializeLE_units : (bs : List UInt8) → (us : List UInt16) → obtain ⟨h1, h2⟩ := h subst h1 h2 rw [ih rfl] - simp only [Spec.serializeLE, List.map_cons, List.flatten_cons, by_hi_unit, by_lo_unit, + simp only [Utf.Spec.serializeLE, List.map_cons, List.flatten_cons, by_hi_unit, by_lo_unit, List.cons_append, List.nil_append] /-! ## The statements -/ theorem encodeBytes (cs : List Char) : - Utf16.encodeBE cs = Spec.serializeBE (Spec.encode16 cs) ∧ - Utf16.encodeLE cs = Spec.serializeLE (Spec.encode16 cs) := by + Utf16.encodeBE cs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs) ∧ + Utf16.encodeLE cs = Utf.Spec.serializeLE (Utf.Spec.encode16 cs) := by refine ⟨?_, ?_⟩ · rw [encodeBE, by_bytesBE, encode_eq] · rw [encodeLE, by_bytesLE, encode_eq] theorem decodeBE_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeBE bs = some cs ↔ bs = Spec.serializeBE (Spec.encode16 cs) := by + Utf16.decodeBE bs = some cs ↔ bs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs) := by constructor · intro h unfold decodeBE decodeBytes at h @@ -143,7 +143,7 @@ theorem decodeBE_iff (bs : List UInt8) (cs : List Char) : exact decode_encode cs theorem decodeLE_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeLE bs = some cs ↔ bs = Spec.serializeLE (Spec.encode16 cs) := by + Utf16.decodeLE bs = some cs ↔ bs = Utf.Spec.serializeLE (Utf.Spec.encode16 cs) := by constructor · intro h unfold decodeLE decodeBytes at h @@ -163,8 +163,8 @@ theorem decodeLE_iff (bs : List UInt8) (cs : List Char) : exact decode_encode cs theorem decodeScheme_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeScheme bs = some cs ↔ Spec.SchemeReads bs cs := by - unfold Spec.SchemeReads + Utf16.decodeScheme bs = some cs ↔ Utf.Spec.SchemeReads bs cs := by + unfold Utf.Spec.SchemeReads match bs with | [] => simp only [decodeScheme, decodeBE_iff] @@ -192,8 +192,8 @@ theorem decodeScheme_iff (bs : List UInt8) (cs : List Char) : · exact h theorem decodeLossyBytes_units (us : List UInt16) : - Utf16.decodeLossyBE (Spec.serializeBE us) = Utf16.decodeLossy us ∧ - Utf16.decodeLossyLE (Spec.serializeLE us) = Utf16.decodeLossy us := by + Utf16.decodeLossyBE (Utf.Spec.serializeBE us) = Utf16.decodeLossy us ∧ + Utf16.decodeLossyLE (Utf.Spec.serializeLE us) = Utf16.decodeLossy us := by refine ⟨?_, ?_⟩ · unfold decodeLossyBE decodeLossyBytes rw [by_units_serializeBE] diff --git a/Proofs/Utf16/Lossy.lean b/Utf/Proofs/Utf16/Lossy.lean similarity index 82% rename from Proofs/Utf16/Lossy.lean rename to Utf/Proofs/Utf16/Lossy.lean index 36e5a46..0b8b1e7 100644 --- a/Proofs/Utf16/Lossy.lean +++ b/Utf/Proofs/Utf16/Lossy.lean @@ -3,21 +3,21 @@ and `toWellFormed` against ECMAScript. -/ -import Proofs.Utf16.Units +import Utf.Proofs.Utf16.Units namespace Utf16.Proofs /-! ## Units and characters -/ -theorem lo_isHigh_iff (u : UInt16) : isHigh u = true ↔ Spec.High u := by - simp [isHigh, Spec.High] +theorem lo_isHigh_iff (u : UInt16) : isHigh u = true ↔ Utf.Spec.High u := by + simp [isHigh, Utf.Spec.High] -theorem lo_isLow_iff (u : UInt16) : isLow u = true ↔ Spec.Low u := by - simp [isLow, Spec.Low] +theorem lo_isLow_iff (u : UInt16) : isLow u = true ↔ Utf.Spec.Low u := by + simp [isLow, Utf.Spec.Low] -theorem lo_high_not_low {u : UInt16} (h : Spec.High u) : ¬ Spec.Low u := by - unfold Spec.High at h; unfold Spec.Low; omega +theorem lo_high_not_low {u : UInt16} (h : Utf.Spec.High u) : ¬ Utf.Spec.Low u := by + unfold Utf.Spec.High at h; unfold Utf.Spec.Low; omega theorem lo_isHigh_not_isLow {u : UInt16} (h : isHigh u = true) : isLow u = false := by have h1 := lo_high_not_low ((lo_isHigh_iff u).1 h) @@ -84,9 +84,9 @@ theorem lo_encodeChar_cases (c : Char) : · intro hd exact Or.inr ⟨_, _, rfl, lo_decode_two hd⟩ -theorem lo_encodeChar16 (c : Char) : Spec.encodeChar16 c = encodeChar c := by +theorem lo_encodeChar16 (c : Char) : Utf.Spec.encodeChar16 c = encodeChar c := by have h := encode_eq [c] - simpa [encode, Spec.encode16] using h.symm + simpa [encode, Utf.Spec.encode16] using h.symm theorem lo_encodeChar_pair {u v : UInt16} (hu : isHigh u = true) (hv : isLow v = true) : encodeChar (pair u v) = [u, v] := by @@ -127,52 +127,52 @@ theorem lo_encode_head {cs : List Char} {e : UInt16} {X : List UInt16} /-! ## Unconvertible offsets and initial subsequences -/ -theorem lo_wf_head {s : List UInt16} (hs : Spec.WellFormed16 s) : +theorem lo_wf_head {s : List UInt16} (hs : Utf.Spec.WellFormed16 s) : ∀ w t, s = w :: t → - ¬ Spec.Low w ∧ (Spec.High w → ∃ v t', t = v :: t' ∧ Spec.Low v) := by + ¬ Utf.Spec.Low w ∧ (Utf.Spec.High w → ∃ v t', t = v :: t' ∧ Utf.Spec.Low v) := by cases hs with | nil => intro w t h; simp at h | cons hc hr => rename_i s1 rest intro w t h rcases s1 with _ | ⟨x, _ | ⟨y, _ | ⟨z, s2⟩⟩⟩ - · simp [Spec.WellFormedChar16] at hc - · simp only [Spec.WellFormedChar16] at hc + · simp [Utf.Spec.WellFormedChar16] at hc + · simp only [Utf.Spec.WellFormedChar16] at hc simp only [List.singleton_append, List.cons.injEq] at h obtain ⟨rfl, -⟩ := h - unfold Spec.Low Spec.High + unfold Utf.Spec.Low Utf.Spec.High omega - · simp only [Spec.WellFormedChar16] at hc + · simp only [Utf.Spec.WellFormedChar16] at hc simp only [List.cons_append, List.cons.injEq] at h obtain ⟨rfl, rfl⟩ := h exact ⟨lo_high_not_low hc.1, fun _ => ⟨y, rest, rfl, hc.2⟩⟩ - · simp [Spec.WellFormedChar16] at hc + · simp [Utf.Spec.WellFormedChar16] at hc -theorem lo_wf_single {w : UInt16} (h1 : ¬ Spec.High w) (h2 : ¬ Spec.Low w) : - Spec.WellFormed16 [w] := by - have hc : Spec.WellFormedChar16 [w] := by - simp only [Spec.WellFormedChar16] - unfold Spec.High at h1; unfold Spec.Low at h2 +theorem lo_wf_single {w : UInt16} (h1 : ¬ Utf.Spec.High w) (h2 : ¬ Utf.Spec.Low w) : + Utf.Spec.WellFormed16 [w] := by + have hc : Utf.Spec.WellFormedChar16 [w] := by + simp only [Utf.Spec.WellFormedChar16] + unfold Utf.Spec.High at h1; unfold Utf.Spec.Low at h2 omega - have := Spec.WellFormed16.cons hc Spec.WellFormed16.nil + have := Utf.Spec.WellFormed16.cons hc Utf.Spec.WellFormed16.nil simpa using this -theorem lo_wf_pair {w v : UInt16} (h1 : Spec.High w) (h2 : Spec.Low v) : - Spec.WellFormed16 [w, v] := by - have hc : Spec.WellFormedChar16 [w, v] := by - simp only [Spec.WellFormedChar16] +theorem lo_wf_pair {w v : UInt16} (h1 : Utf.Spec.High w) (h2 : Utf.Spec.Low v) : + Utf.Spec.WellFormed16 [w, v] := by + have hc : Utf.Spec.WellFormedChar16 [w, v] := by + simp only [Utf.Spec.WellFormedChar16] exact ⟨h1, h2⟩ - have := Spec.WellFormed16.cons hc Spec.WellFormed16.nil + have := Utf.Spec.WellFormed16.cons hc Utf.Spec.WellFormed16.nil simpa using this theorem lo_unconv_cons (w : UInt16) (r : List UInt16) : - Spec.Unconvertible16 (w :: r) ↔ - Spec.Low w ∨ (Spec.High w ∧ ¬ ∃ v r', r = v :: r' ∧ Spec.Low v) := by + Utf.Spec.Unconvertible16 (w :: r) ↔ + Utf.Spec.Low w ∨ (Utf.Spec.High w ∧ ¬ ∃ v r', r = v :: r' ∧ Utf.Spec.Low v) := by constructor · intro hU - by_cases hl : Spec.Low w + by_cases hl : Utf.Spec.Low w · exact Or.inl hl - by_cases hh : Spec.High w + by_cases hh : Utf.Spec.High w · refine Or.inr ⟨hh, ?_⟩ rintro ⟨v, r', rfl, hv⟩ exact hU [w, v] (by simp) ⟨r', rfl⟩ (lo_wf_pair hh hv) @@ -190,8 +190,8 @@ theorem lo_unconv_cons (w : UInt16) (r : List UInt16) : · obtain ⟨v, t', rfl, hv⟩ := hh.2 hh' exact hn ⟨v, t' ++ t, by rw [← ht]; simp, hv⟩ -theorem lo_initial_two {w v : UInt16} {q : List UInt16} (h : Spec.Initial16 (w :: v :: q)) : - ¬ Spec.Low w ∧ (Spec.High w → Spec.Low v) := by +theorem lo_initial_two {w v : UInt16} {q : List UInt16} (h : Utf.Spec.Initial16 (w :: v :: q)) : + ¬ Utf.Spec.Low w ∧ (Utf.Spec.High w → Utf.Spec.Low v) := by obtain ⟨s, hs, t, rfl⟩ := h have := lo_wf_head hs w (v :: q ++ t) (by simp) refine ⟨this.1, fun hh => ?_⟩ @@ -200,7 +200,7 @@ theorem lo_initial_two {w v : UInt16} {q : List UInt16} (h : Spec.Initial16 (w : rw [he.1]; exact hv theorem lo_maximalSubpart_iff (p rest : List UInt16) : - Spec.MaximalSubpart16 p rest ↔ ∃ w, p = [w] ∧ Spec.Unpaired [] w rest := by + Utf.Spec.MaximalSubpart16 p rest ↔ ∃ w, p = [w] ∧ Utf.Spec.Unpaired [] w rest := by constructor · rintro ⟨hne, hU, hI, -⟩ cases p with @@ -217,7 +217,7 @@ theorem lo_maximalSubpart_iff (p rest : List UInt16) : · left; exact hh | cons v p'' => exfalso - have hI' : Spec.Initial16 (w :: v :: p'') := by + have hI' : Utf.Spec.Initial16 (w :: v :: p'') := by rcases hI with hI | hI · exact hI · simp at hI @@ -237,7 +237,7 @@ theorem lo_maximalSubpart_iff (p rest : List UInt16) : | [_], _, _ => simp | x :: y :: q'', hq, hI => exfalso - have hI' : Spec.Initial16 (x :: y :: q'') := by + have hI' : Utf.Spec.Initial16 (x :: y :: q'') := by rcases hI with hI | hI · exact hI · simp at hI @@ -252,7 +252,7 @@ theorem lo_maximalSubpart_iff (p rest : List UInt16) : /-! ## Lossy decoding -/ -theorem lo_dl_unpaired {w : UInt16} {rest : List UInt16} (hu : Spec.Unpaired [] w rest) : +theorem lo_dl_unpaired {w : UInt16} {rest : List UInt16} (hu : Utf.Spec.Unpaired [] w rest) : decodeLossy (w :: rest) = replacement :: decodeLossy rest := by rcases hu with ⟨hh, hn⟩ | ⟨hl, -⟩ · rw [← lo_isHigh_iff] at hh @@ -267,47 +267,47 @@ theorem lo_dl_unpaired {w : UInt16} {rest : List UInt16} (hu : Spec.Unpaired [] rw [← lo_isLow_iff] at hl rw [lo_dl_low _ hh hl] -theorem lo_subst_dl (us : List UInt16) : Spec.Substitutes16 us (decodeLossy us) := by +theorem lo_subst_dl (us : List UInt16) : Utf.Spec.Substitutes16 us (decodeLossy us) := by induction us using decodeLossy.induct with - | case1 => rw [lo_dl_nil]; exact Spec.Substitutes16.nil + | case1 => rw [lo_dl_nil]; exact Utf.Spec.Substitutes16.nil | case2 u hu v rest' hv ih => rw [lo_dl_pair _ hu hv] - have := Spec.Substitutes16.char (c := pair u v) ih + have := Utf.Spec.Substitutes16.char (c := pair u v) ih rw [lo_encodeChar16, lo_encodeChar_pair hu hv] at this exact this | case3 u hu v rest' hv ih => rw [lo_dl_high_bad _ hu hv] - have hm : Spec.MaximalSubpart16 [u] (v :: rest') := by + have hm : Utf.Spec.MaximalSubpart16 [u] (v :: rest') := by rw [lo_maximalSubpart_iff] refine ⟨u, rfl, Or.inl ⟨(lo_isHigh_iff u).1 hu, ?_⟩⟩ rintro ⟨v', r', he, hv'⟩ simp only [List.cons.injEq] at he rw [← he.1, ← lo_isLow_iff] at hv' exact hv hv' - exact Spec.Substitutes16.replace hm ih + exact Utf.Spec.Substitutes16.replace hm ih | case4 u hu => rw [lo_dl_high_end hu] - have hm : Spec.MaximalSubpart16 [u] [] := by + have hm : Utf.Spec.MaximalSubpart16 [u] [] := by rw [lo_maximalSubpart_iff] refine ⟨u, rfl, Or.inl ⟨(lo_isHigh_iff u).1 hu, ?_⟩⟩ rintro ⟨v', r', he, -⟩ simp at he - exact Spec.Substitutes16.replace hm Spec.Substitutes16.nil + exact Utf.Spec.Substitutes16.replace hm Utf.Spec.Substitutes16.nil | case5 u rest hu hl ih => rw [lo_dl_low _ hu hl] - have hm : Spec.MaximalSubpart16 [u] rest := by + have hm : Utf.Spec.MaximalSubpart16 [u] rest := by rw [lo_maximalSubpart_iff] refine ⟨u, rfl, Or.inr ⟨(lo_isLow_iff u).1 hl, ?_⟩⟩ rintro ⟨l', v, he, -⟩ simp at he - exact Spec.Substitutes16.replace hm ih + exact Utf.Spec.Substitutes16.replace hm ih | case6 u rest hu hl ih => rw [lo_dl_plain _ hu hl] - have := Spec.Substitutes16.char (c := Char.ofNat u.toNat) ih + have := Utf.Spec.Substitutes16.char (c := Char.ofNat u.toNat) ih rw [lo_encodeChar16, lo_encodeChar_plain hu hl] at this exact this -theorem lo_dl_of_subst {us : List UInt16} {out : List Char} (h : Spec.Substitutes16 us out) : +theorem lo_dl_of_subst {us : List UInt16} {out : List Char} (h : Utf.Spec.Substitutes16 us out) : decodeLossy us = out := by induction h with | nil => exact lo_dl_nil @@ -331,20 +331,20 @@ theorem lo_dl_encode (cs : List Char) : decodeLossy (encode cs) = cs := by exact this theorem decodeLossy_iff (us : List UInt16) (out : List Char) : - Utf16.decodeLossy us = out ↔ Spec.Substitutes16 us out := by + Utf16.decodeLossy us = out ↔ Utf.Spec.Substitutes16 us out := by constructor · rintro rfl; exact lo_subst_dl us · exact lo_dl_of_subst /-- In UTF-16 a maximal subpart is always one isolated surrogate. A fact about the definitions alone, so not one of the statements of - `Spec.lean`; `decodeLossy_iff` rests on it. -/ + `Utf/Spec.lean`; `decodeLossy_iff` rests on it. -/ theorem maximalSubpart_iff (p rest : List UInt16) : - Spec.MaximalSubpart16 p rest ↔ ∃ w, p = [w] ∧ Spec.Unpaired [] w rest := + Utf.Spec.MaximalSubpart16 p rest ↔ ∃ w, p = [w] ∧ Utf.Spec.Unpaired [] w rest := lo_maximalSubpart_iff p rest theorem decodeLossy_unpaired (w : UInt16) (rest : List UInt16) : - Spec.Unpaired [] w rest → Utf16.decodeLossy (w :: rest) = '�' :: Utf16.decodeLossy rest := + Utf.Spec.Unpaired [] w rest → Utf16.decodeLossy (w :: rest) = '�' :: Utf16.decodeLossy rest := fun hu => lo_dl_unpaired hu theorem decodeLossy_of_decode (us : List UInt16) (cs : List Char) : @@ -448,14 +448,14 @@ theorem lo_tw_plain {u : UInt16} (r : List UInt16) (hu : ¬ isHigh u = true) def lo_TW (l us out : List UInt16) : Prop := out.length = us.length ∧ ∀ (i : Nat) (h : i < us.length), - (Spec.Unpaired (l ++ us.take i) us[i] (us.drop (i + 1)) → out[i]? = some 0xFFFD) ∧ - (¬ Spec.Unpaired (l ++ us.take i) us[i] (us.drop (i + 1)) → out[i]? = some us[i]) + (Utf.Spec.Unpaired (l ++ us.take i) us[i] (us.drop (i + 1)) → out[i]? = some 0xFFFD) ∧ + (¬ Utf.Spec.Unpaired (l ++ us.take i) us[i] (us.drop (i + 1)) → out[i]? = some us[i]) theorem lo_TW_nil (l : List UInt16) : lo_TW l [] [] := ⟨rfl, fun _ h => absurd h (Nat.not_lt_zero _)⟩ theorem lo_TW_cons {l rest X : List UInt16} {u a : UInt16} - (h1 : Spec.Unpaired l u rest → a = 0xFFFD) (h2 : ¬ Spec.Unpaired l u rest → a = u) + (h1 : Utf.Spec.Unpaired l u rest → a = 0xFFFD) (h2 : ¬ Utf.Spec.Unpaired l u rest → a = u) (h : lo_TW (l ++ [u]) rest X) : lo_TW l (u :: rest) (a :: X) := by refine ⟨by simp [h.1], fun i hi => ?_⟩ match i, hi with @@ -471,7 +471,7 @@ theorem lo_TW_cons {l rest X : List UInt16} {u a : UInt16} exact h.2 i (by simpa using hi) theorem lo_endsHigh_append {l : List UInt16} {x : UInt16} : - (∃ l' v, l ++ [x] = l' ++ [v] ∧ Spec.High v) ↔ Spec.High x := by + (∃ l' v, l ++ [x] = l' ++ [v] ∧ Utf.Spec.High v) ↔ Utf.Spec.High x := by constructor · rintro ⟨l', v, he, hv⟩ have := congrArg List.reverse he @@ -481,7 +481,7 @@ theorem lo_endsHigh_append {l : List UInt16} {x : UInt16} : · intro hx; exact ⟨l, x, rfl, hx⟩ theorem lo_tw_main (us : List UInt16) : ∀ (l : List UInt16), - (∀ v r, us = v :: r → Spec.Low v → ¬ ∃ l' x, l = l' ++ [x] ∧ Spec.High x) → + (∀ v r, us = v :: r → Utf.Spec.Low v → ¬ ∃ l' x, l = l' ++ [x] ∧ Utf.Spec.High x) → lo_TW l us (encode (decodeLossy us)) := by induction us using decodeLossy.induct with | case1 => intro l _; rw [lo_dl_nil]; exact lo_TW_nil l @@ -550,9 +550,9 @@ theorem lo_tw_main (us : List UInt16) : ∀ (l : List UInt16), theorem toWellFormed_spec (us : List UInt16) : (Utf16.toWellFormed us).length = us.length ∧ ∀ (i : Nat) (h : i < us.length), - (Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → + (Utf.Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → (Utf16.toWellFormed us)[i]? = some 0xFFFD) ∧ - (¬ Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → + (¬ Utf.Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → (Utf16.toWellFormed us)[i]? = some us[i]) := by have h := lo_tw_main us [] (by rintro v r - - ⟨l', x, he, -⟩ diff --git a/Proofs/Utf16/Units.lean b/Utf/Proofs/Utf16/Units.lean similarity index 85% rename from Proofs/Utf16/Units.lean rename to Utf/Proofs/Utf16/Units.lean index 35b11b7..e83ca15 100644 --- a/Proofs/Utf16/Units.lean +++ b/Utf/Proofs/Utf16/Units.lean @@ -5,7 +5,7 @@ -/ import Utf16 -import Spec.Utf16 +import Utf.Spec.Utf16 namespace Utf16.Proofs @@ -62,20 +62,20 @@ private theorem un_or_lo (m : Nat) : have hb : m % 2 ^ 10 < 2 ^ 10 := Nat.mod_lt _ (by decide) exact (Nat.two_pow_add_eq_or_of_lt hb 55).symm -private theorem un_encodeChar_spec (c : Char) : encodeChar c = Spec.encodeChar16 c := by +private theorem un_encodeChar_spec (c : Char) : encodeChar c = Utf.Spec.encodeChar16 c := by have hv := un_valid c - unfold encodeChar Spec.encodeChar16 + unfold encodeChar Utf.Spec.encodeChar16 simp only split · rfl · rw [un_or_hi _ (by omega), un_or_lo] -theorem encode_eq (cs : List Char) : Utf16.encode cs = Spec.encode16 cs := by +theorem encode_eq (cs : List Char) : Utf16.encode cs = Utf.Spec.encode16 cs := by induction cs with - | nil => simp [encode, Spec.encode16] + | nil => simp [encode, Utf.Spec.encode16] | cons c cs ih => rw [un_encode_cons, ih, un_encodeChar_spec] - simp only [Spec.encode16, List.map_cons, List.flatten_cons] + simp only [Utf.Spec.encode16, List.map_cons, List.flatten_cons] /-! ## The decoder, one character at a time -/ @@ -165,28 +165,28 @@ theorem decode_encode (cs : List Char) : Utf16.decode (Utf16.encode cs) = some c /-! ## Well-formed sequences -/ private theorem un_wellFormedChar_iff (s : List UInt16) : - Spec.WellFormedChar16 s ↔ ∃ c, s = encodeChar c := by + Utf.Spec.WellFormedChar16 s ↔ ∃ c, s = encodeChar c := by constructor · intro h rcases s with _ | ⟨w, _ | ⟨w2, _ | ⟨w3, s⟩⟩⟩ - · simp [Spec.WellFormedChar16] at h - · simp only [Spec.WellFormedChar16] at h + · simp [Utf.Spec.WellFormedChar16] at h + · simp only [Utf.Spec.WellFormedChar16] at h exact ⟨Char.ofNat w.toNat, (un_encodeChar_single w (by omega)).symm⟩ - · simp only [Spec.WellFormedChar16, Spec.High, Spec.Low] at h + · simp only [Utf.Spec.WellFormedChar16, Utf.Spec.High, Utf.Spec.Low] at h exact ⟨pair w w2, (un_encodeChar_pair w w2 h.1 h.2).symm⟩ - · simp [Spec.WellFormedChar16] at h + · simp [Utf.Spec.WellFormedChar16] at h · rintro ⟨c, rfl⟩ have hv := un_valid c by_cases h : c.toNat < 0x10000 · rw [un_encodeChar_lt c h] - simp only [Spec.WellFormedChar16, un_toNat_toUInt16] + simp only [Utf.Spec.WellFormedChar16, un_toNat_toUInt16] omega · rw [un_encodeChar_ge c h] - simp only [Spec.WellFormedChar16, Spec.High, Spec.Low, un_toNat_toUInt16] + simp only [Utf.Spec.WellFormedChar16, Utf.Spec.High, Utf.Spec.Low, un_toNat_toUInt16] omega theorem wellFormed_iff (us : List UInt16) : - Spec.WellFormed16 us ↔ ∃ cs, us = Utf16.encode cs := by + Utf.Spec.WellFormed16 us ↔ ∃ cs, us = Utf16.encode cs := by constructor · intro h induction h with @@ -197,20 +197,20 @@ theorem wellFormed_iff (us : List UInt16) : exact ⟨c :: cs, (un_encode_cons c cs).symm⟩ · rintro ⟨cs, rfl⟩ induction cs with - | nil => simpa [encode] using Spec.WellFormed16.nil + | nil => simpa [encode] using Utf.Spec.WellFormed16.nil | cons c cs ih => rw [un_encode_cons] - exact Spec.WellFormed16.cons ((un_wellFormedChar_iff _).2 ⟨c, rfl⟩) ih + exact Utf.Spec.WellFormed16.cons ((un_wellFormedChar_iff _).2 ⟨c, rfl⟩) ih /-! ## Unpaired surrogates -/ /-- A unit put before the units on the left of `w` changes nothing, so long as it is not a high surrogate right before `w`. -/ private theorem un_unpaired_cons_iff {a : UInt16} {l : List UInt16} {w : UInt16} - {r : List UInt16} (ha : l = [] → ¬ Spec.High a) : - Spec.Unpaired (a :: l) w r ↔ Spec.Unpaired l w r := by - unfold Spec.Unpaired - have key : (∃ l' v, a :: l = l' ++ [v] ∧ Spec.High v) ↔ (∃ l' v, l = l' ++ [v] ∧ Spec.High v) := by + {r : List UInt16} (ha : l = [] → ¬ Utf.Spec.High a) : + Utf.Spec.Unpaired (a :: l) w r ↔ Utf.Spec.Unpaired l w r := by + unfold Utf.Spec.Unpaired + have key : (∃ l' v, a :: l = l' ++ [v] ∧ Utf.Spec.High v) ↔ (∃ l' v, l = l' ++ [v] ∧ Utf.Spec.High v) := by constructor · rintro ⟨l', v, he, hv⟩ rcases l' with _ | ⟨b, l''⟩ @@ -224,7 +224,7 @@ private theorem un_unpaired_cons_iff {a : UInt16} {l : List UInt16} {w : UInt16} /-- No unit of an encoding is an unpaired surrogate. -/ private theorem un_encode_not_unpaired (cs : List Char) : - ∀ l w r, encode cs = l ++ w :: r → ¬ Spec.Unpaired l w r := by + ∀ l w r, encode cs = l ++ w :: r → ¬ Utf.Spec.Unpaired l w r := by induction cs with | nil => intro l w r h; simp [encode] at h | cons c cs ih => @@ -234,13 +234,13 @@ private theorem un_encode_not_unpaired (cs : List Char) : by_cases hc : c.toNat < 0x10000 · rw [un_encodeChar_lt c hc] at h have hu : c.toNat.toUInt16.toNat = c.toNat := by rw [un_toNat_toUInt16]; omega - have hnh : ¬ Spec.High c.toNat.toUInt16 := by unfold Spec.High; rw [hu]; omega + have hnh : ¬ Utf.Spec.High c.toNat.toUInt16 := by unfold Utf.Spec.High; rw [hu]; omega rcases l with _ | ⟨a, l'⟩ · simp only [List.nil_append, List.singleton_append, List.cons.injEq] at h obtain ⟨rfl, rfl⟩ := h rcases hun with ⟨h1, _⟩ | ⟨h1, _⟩ · exact hnh h1 - · unfold Spec.Low at h1; rw [hu] at h1; omega + · unfold Utf.Spec.Low at h1; rw [hu] at h1; omega · simp only [List.cons_append, List.cons.injEq] at h obtain ⟨rfl, h⟩ := h exact ih l' w r h ((un_unpaired_cons_iff (fun _ => hnh)).1 hun) @@ -253,10 +253,10 @@ private theorem un_encode_not_unpaired (cs : List Char) : rw [un_toNat_toUInt16]; omega generalize (0xD800 + m / 0x400).toUInt16 = hi at h h1 generalize (0xDC00 + m % 0x400).toUInt16 = lo at h h2 - have hhi : Spec.High hi := by unfold Spec.High; rw [h1]; omega - have hlo : Spec.Low lo := by unfold Spec.Low; rw [h2]; omega - have hlo' : ¬ Spec.High lo := by unfold Spec.High; rw [h2]; omega - have hhi' : ¬ Spec.Low hi := by unfold Spec.Low; rw [h1]; omega + have hhi : Utf.Spec.High hi := by unfold Utf.Spec.High; rw [h1]; omega + have hlo : Utf.Spec.Low lo := by unfold Utf.Spec.Low; rw [h2]; omega + have hlo' : ¬ Utf.Spec.High lo := by unfold Utf.Spec.High; rw [h2]; omega + have hhi' : ¬ Utf.Spec.Low hi := by unfold Utf.Spec.Low; rw [h1]; omega rcases l with _ | ⟨a, _ | ⟨b, l''⟩⟩ · simp only [List.nil_append, List.cons_append, List.cons.injEq] at h obtain ⟨rfl, rfl⟩ := h @@ -274,7 +274,7 @@ private theorem un_encode_not_unpaired (cs : List Char) : exact ih l'' w r h hun theorem decode_none_iff (us : List UInt16) : - Utf16.decode us = none ↔ ∃ l w r, us = l ++ w :: r ∧ Spec.Unpaired l w r := by + Utf16.decode us = none ↔ ∃ l w r, us = l ++ w :: r ∧ Utf.Spec.Unpaired l w r := by constructor · intro h induction us using decode.induct with @@ -283,8 +283,8 @@ theorem decode_none_iff (us : List UInt16) : rw [un_decode_pair _ _ _ hu hv, Option.map_eq_none_iff] at h obtain ⟨l, w, r, rfl, hun⟩ := ih h refine ⟨u :: v :: l, w, r, rfl, ?_⟩ - have hvh : ¬ Spec.High v := by - rw [un_isLow_iff] at hv; unfold Spec.High; omega + have hvh : ¬ Utf.Spec.High v := by + rw [un_isLow_iff] at hv; unfold Utf.Spec.High; omega rw [un_unpaired_cons_iff (by simp), un_unpaired_cons_iff (fun _ => hvh)] exact hun | case3 u hu v rest' hv => @@ -304,7 +304,7 @@ theorem decode_none_iff (us : List UInt16) : rw [un_decode_single _ _ hu hl, Option.map_eq_none_iff] at h obtain ⟨l, w, r, rfl, hun⟩ := ih h refine ⟨u :: l, w, r, rfl, ?_⟩ - have huh : ¬ Spec.High u := fun h' => hu ((un_isHigh_iff u).2 h') + have huh : ¬ Utf.Spec.High u := fun h' => hu ((un_isHigh_iff u).2 h') rw [un_unpaired_cons_iff (fun _ => huh)] exact hun · rintro ⟨l, w, r, hus, hun⟩ diff --git a/Proofs/Utf16/Whatwg.lean b/Utf/Proofs/Utf16/Whatwg.lean similarity index 88% rename from Proofs/Utf16/Whatwg.lean rename to Utf/Proofs/Utf16/Whatwg.lean index a38d87c..51742a5 100644 --- a/Proofs/Utf16/Whatwg.lean +++ b/Utf/Proofs/Utf16/Whatwg.lean @@ -4,7 +4,7 @@ -/ import Utf16 -import Spec.Utf16 +import Utf.Spec.Utf16 namespace Utf16.Proofs @@ -128,8 +128,8 @@ theorem wh_bytes (be : Bool) (a b : UInt8) : · simp only [wh_u, unit, ite_true, List.cons.injEq, and_true] constructor <;> apply UInt8.toNat_inj.mp <;> simp <;> omega -theorem wh_run_nil (be : Bool) (fuel : Nat) : Spec.whatwgRun be fuel none none [] = [] := by - cases fuel <;> simp [Spec.whatwgRun] +theorem wh_run_nil (be : Bool) (fuel : Nat) : Utf.Spec.whatwgRun be fuel none none [] = [] := by + cases fuel <;> simp [Utf.Spec.whatwgRun] theorem wh_high_iff (u : UInt16) : isHigh u = true ↔ 0xD800 ≤ u.toNat ∧ u.toNat ≤ 0xDBFF := by simp [isHigh] @@ -145,14 +145,14 @@ theorem wh_pair (h l : UInt16) : theorem wh_run (be : Bool) : ∀ fuel : Nat, (∀ q, 2 * q.length + 1 ≤ fuel → - Spec.whatwgRun be fuel none none q = decodeLossyBytes be q) ∧ + Utf.Spec.whatwgRun be fuel none none q = decodeLossyBytes be q) ∧ (∀ a q, 2 * q.length + 2 ≤ fuel → - Spec.whatwgRun be fuel (some a) none q = decodeLossyBytes be (a :: q)) ∧ + Utf.Spec.whatwgRun be fuel (some a) none q = decodeLossyBytes be (a :: q)) ∧ (∀ x y q, isHigh (wh_u be x y) = true → 2 * q.length + 3 ≤ fuel → - Spec.whatwgRun be fuel none (some (wh_u be x y).toNat) q = + Utf.Spec.whatwgRun be fuel none (some (wh_u be x y).toNat) q = decodeLossyBytes be (x :: y :: q)) ∧ (∀ x y a q, isHigh (wh_u be x y) = true → 2 * q.length + 4 ≤ fuel → - Spec.whatwgRun be fuel (some a) (some (wh_u be x y).toNat) q = + Utf.Spec.whatwgRun be fuel (some a) (some (wh_u be x y).toNat) q = decodeLossyBytes be (x :: y :: a :: q)) := by intro fuel induction fuel with @@ -164,14 +164,14 @@ theorem wh_run (be : Bool) : ∀ fuel : Nat, cases q with | nil => exact wh_run_nil be _ | cons b q => - simp only [Spec.whatwgRun] + simp only [Utf.Spec.whatwgRun] exact ih2 b q (by simp at hq; omega) · intro a q hq cases q with - | nil => simp [Spec.whatwgRun, wh_run_nil, wh_dlb_one, wh_repl] + | nil => simp [Utf.Spec.whatwgRun, wh_run_nil, wh_dlb_one, wh_repl] | cons b q => simp only [List.length_cons] at hq - simp only [Spec.whatwgRun, wh_u_toNat] + simp only [Utf.Spec.whatwgRun, wh_u_toNat] cases hh : isHigh (wh_u be a b) · have hh' : ¬ (0xD800 ≤ (wh_u be a b).toNat ∧ (wh_u be a b).toNat ≤ 0xDBFF) := by rw [← wh_high_iff, hh]; simp @@ -187,17 +187,17 @@ theorem wh_run (be : Bool) : ∀ fuel : Nat, exact ih3 a b q hh (by omega) · intro x y q hh hq cases q with - | nil => simp [Spec.whatwgRun, wh_run_nil, wh_dlb_high_two be x y hh, wh_repl] + | nil => simp [Utf.Spec.whatwgRun, wh_run_nil, wh_dlb_high_two be x y hh, wh_repl] | cons b q => simp only [List.length_cons] at hq - simp only [Spec.whatwgRun] + simp only [Utf.Spec.whatwgRun] exact ih4 x y b q hh (by omega) · intro x y a q hh hq cases q with - | nil => simp [Spec.whatwgRun, wh_run_nil, wh_dlb_high_three be x y a hh, wh_repl] + | nil => simp [Utf.Spec.whatwgRun, wh_run_nil, wh_dlb_high_three be x y a hh, wh_repl] | cons b q => simp only [List.length_cons] at hq - simp only [Spec.whatwgRun, wh_u_toNat, wh_bytes] + simp only [Utf.Spec.whatwgRun, wh_u_toNat, wh_bytes] cases hl : isLow (wh_u be a b) · have hl' : ¬ (0xDC00 ≤ (wh_u be a b).toNat ∧ (wh_u be a b).toNat ≤ 0xDFFF) := by rw [← wh_low_iff, hl]; simp @@ -208,8 +208,8 @@ theorem wh_run (be : Bool) : ∀ fuel : Nat, ih1 q (by omega), wh_pair] theorem whatwg (bs : List UInt8) : - Utf16.decodeLossyBE bs = Spec.whatwgDecode true bs ∧ - Utf16.decodeLossyLE bs = Spec.whatwgDecode false bs := by + Utf16.decodeLossyBE bs = Utf.Spec.whatwgDecode true bs ∧ + Utf16.decodeLossyLE bs = Utf.Spec.whatwgDecode false bs := by constructor · exact ((wh_run true _).1 bs (by omega)).symm · exact ((wh_run false _).1 bs (by omega)).symm diff --git a/Spec.lean b/Utf/Spec.lean similarity index 79% rename from Spec.lean rename to Utf/Spec.lean index 2954937..40983f9 100644 --- a/Spec.lean +++ b/Utf/Spec.lean @@ -2,7 +2,7 @@ The specification as Lean statements: the challenge file. A reviewer who did not write this file approves it, and `bar.toml` - records the hash it had then. The proofs live in `Proofs.lean`, under + records the hash it had then. The proofs live in `Utf/Proofs.lean`, under the same names; `lake comparator` checks that each one proves exactly the statement written here. The `sorry`s below are expected in this file only. @@ -16,14 +16,14 @@ import Utf8 import Utf16 -import Spec.Defs -import Spec.Utf16 +import Utf.Spec.Defs +import Utf.Spec.Utf16 /-! ## Encoding -/ /-- The encoder writes each character as RFC 3629, section 3, sets out. -/ -theorem Spec.encode_eq (cs : List Char) : Utf8.encode cs = Spec.encode cs := sorry +theorem Utf.Spec.encode_eq (cs : List Char) : Utf8.encode cs = Utf.Spec.encode cs := sorry /-! ## Decoding -/ @@ -32,36 +32,36 @@ theorem Spec.encode_eq (cs : List Char) : Utf8.encode cs = Spec.encode cs := sor refuses everything else: overlong forms, surrogate halves, values past U+10FFFF, stray continuation bytes, the bytes 0xC0, 0xC1 and 0xF5 to 0xFF, and sequences cut short. -/ -theorem Spec.decode_iff (bs : List UInt8) (cs : List Char) : - Utf8.decode bs = some cs ↔ bs = Spec.encode cs := sorry +theorem Utf.Spec.decode_iff (bs : List UInt8) (cs : List Char) : + Utf8.decode bs = some cs ↔ bs = Utf.Spec.encode cs := sorry /-- Round trip: decoding what the encoder wrote gives the characters back. -/ -theorem Spec.decode_encode (cs : List Char) : Utf8.decode (Utf8.encode cs) = some cs := sorry +theorem Utf.Spec.decode_encode (cs : List Char) : Utf8.decode (Utf8.encode cs) = some cs := sorry /-- The decoder accepts exactly the byte sequences that Table 3-7 of the Unicode Standard calls well formed. -/ -theorem Spec.decode_wellFormed (bs : List UInt8) : - (Utf8.decode bs).isSome ↔ Spec.WellFormed bs := sorry +theorem Utf.Spec.decode_wellFormed (bs : List UInt8) : + (Utf8.decode bs).isSome ↔ Utf.Spec.WellFormed bs := sorry /-! ## Lean core -/ /-- The encoder agrees with Lean core's own UTF-8 encoding of a string. -/ -theorem Spec.encode_toUTF8 (s : String) : Utf8.encode s.toList = s.toUTF8.toList := sorry +theorem Utf.Spec.encode_toUTF8 (s : String) : Utf8.encode s.toList = s.toUTF8.toList := sorry /-- The decoder agrees with Lean core's `String.fromUTF8?`, as core defines it: core accepts exactly the bytes this decoder accepts, and reads the same characters. (At run time core checks the bytes with compiled C code, which this statement does not cover; the tests compare the two.) -/ -theorem Spec.fromUTF8?_eq (bs : List UInt8) : +theorem Utf.Spec.fromUTF8?_eq (bs : List UInt8) : (String.fromUTF8? bs.toByteArray).map String.toList = Utf8.decode bs := sorry /-! ## Test vectors -/ /-- The examples of RFC 3629, section 7, both ways. -/ -theorem Spec.rfc3629_examples : +theorem Utf.Spec.rfc3629_examples : Utf8.encode ['A', '≢', '\u0391', '.'] = [0x41, 0xE2, 0x89, 0xA2, 0xCE, 0x91, 0x2E] ∧ Utf8.encode ['한', '국', '어'] = @@ -80,7 +80,7 @@ theorem Spec.rfc3629_examples : /-- The first and last code points of each length, the code points on either side of the surrogates, and the empty input. -/ -theorem Spec.boundary_vectors : +theorem Utf.Spec.boundary_vectors : Utf8.encode ['\u0000', '\u007F'] = [0x00, 0x7F] ∧ Utf8.encode ['\u0080', '\u07FF'] = [0xC2, 0x80, 0xDF, 0xBF] ∧ Utf8.encode ['\u0800', '\uFFFF'] = [0xE0, 0xA0, 0x80, 0xEF, 0xBF, 0xBF] ∧ @@ -97,8 +97,8 @@ theorem Spec.boundary_vectors : halves and a surrogate pair, a value past U+10FFFF, each byte from 0xF5 to 0xFF, five and six byte forms, lone continuation bytes, sequences cut short, and a lead byte followed by a byte that is not - a continuation. Every other input follows from `Spec.decode_iff`. -/ -theorem Spec.reject_vectors : + a continuation. Every other input follows from `Utf.Spec.decode_iff`. -/ +theorem Utf.Spec.reject_vectors : Utf8.decode [0xC0, 0xAF] = none ∧ Utf8.decode [0xE0, 0x80, 0xAF] = none ∧ Utf8.decode [0xF0, 0x80, 0x80, 0xAF] = none ∧ @@ -139,31 +139,31 @@ theorem Spec.reject_vectors : the Unicode Standard sets out: each well-formed sequence becomes its character, and each maximal subpart of an ill-formed sequence becomes one U+FFFD. -/ -theorem Spec.decodeLossy_iff (bs : List UInt8) (out : List Char) : - Utf8.decodeLossy bs = out ↔ Spec.Substitutes bs out := sorry +theorem Utf.Spec.decodeLossy_iff (bs : List UInt8) (out : List Char) : + Utf8.decodeLossy bs = out ↔ Utf.Spec.Substitutes bs out := sorry /-- On well-formed bytes, the lossy decoder reads what the strict one reads. -/ -theorem Spec.decodeLossy_of_decode (bs : List UInt8) (cs : List Char) : +theorem Utf.Spec.decodeLossy_of_decode (bs : List UInt8) (cs : List Char) : Utf8.decode bs = some cs → Utf8.decodeLossy bs = cs := sorry /-- On bytes that are not well formed, the lossy decoder puts in at least one U+FFFD. -/ -theorem Spec.decodeLossy_replaces (bs : List UInt8) : +theorem Utf.Spec.decodeLossy_replaces (bs : List UInt8) : Utf8.decode bs = none → '�' ∈ Utf8.decodeLossy bs := sorry /-- Well-formed text before bad bytes comes through unchanged. -/ -theorem Spec.decodeLossy_encode_append (cs : List Char) (bs : List UInt8) : +theorem Utf.Spec.decodeLossy_encode_append (cs : List Char) (bs : List UInt8) : Utf8.decodeLossy (Utf8.encode cs ++ bs) = cs ++ Utf8.decodeLossy bs := sorry /-- Well-formed text after bad bytes comes through unchanged: no replacement swallows any of it. -/ -theorem Spec.decodeLossy_append_encode (bs : List UInt8) (cs : List Char) : +theorem Utf.Spec.decodeLossy_append_encode (bs : List UInt8) (cs : List Char) : Utf8.decodeLossy (bs ++ Utf8.encode cs) = Utf8.decodeLossy bs ++ cs := sorry /-- The examples of section 3.9.6 of the Unicode Standard, Tables 3-8 to 3-11, and three cut-short sequences followed by a character. -/ -theorem Spec.substitution_vectors : +theorem Utf.Spec.substitution_vectors : -- Table 3-8, non-shortest forms Utf8.decodeLossy [0xC0, 0xAF, 0xE0, 0x80, 0xBF, 0xF0, 0x81, 0x82, 0x41] = ['�', '�', '�', '�', '�', '�', '�', '�', 'A'] ∧ @@ -185,34 +185,34 @@ theorem Spec.substitution_vectors : /-- The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. -/ -theorem Spec.utf16_encode_eq (cs : List Char) : Utf16.encode cs = Spec.encode16 cs := sorry +theorem Utf.Spec.utf16_encode_eq (cs : List Char) : Utf16.encode cs = Utf.Spec.encode16 cs := sorry /-- `Utf16.decode` returns `cs` exactly when the units are the encoding of `cs`. So every unit sequence it accepts has one reading, and it refuses everything else: a high surrogate with no low surrogate after it, and a low surrogate with no high surrogate before it. -/ -theorem Spec.utf16_decode_iff (us : List UInt16) (cs : List Char) : - Utf16.decode us = some cs ↔ us = Spec.encode16 cs := sorry +theorem Utf.Spec.utf16_decode_iff (us : List UInt16) (cs : List Char) : + Utf16.decode us = some cs ↔ us = Utf.Spec.encode16 cs := sorry /-- Round trip: decoding what the encoder wrote gives the characters back. -/ -theorem Spec.utf16_decode_encode (cs : List Char) : +theorem Utf.Spec.utf16_decode_encode (cs : List Char) : Utf16.decode (Utf16.encode cs) = some cs := sorry /-- The decoder accepts exactly the unit sequences that D91 and Table 3-5 of the Unicode Standard call well formed. -/ -theorem Spec.utf16_decode_wellFormed (us : List UInt16) : - (Utf16.decode us).isSome ↔ Spec.WellFormed16 us := sorry +theorem Utf.Spec.utf16_decode_wellFormed (us : List UInt16) : + (Utf16.decode us).isSome ↔ Utf.Spec.WellFormed16 us := sorry /-- The decoder fails exactly when some unit is an isolated surrogate, in no surrogate pair. -/ -theorem Spec.utf16_decode_none_iff (us : List UInt16) : - Utf16.decode us = none ↔ ∃ l w r, us = l ++ w :: r ∧ Spec.Unpaired l w r := sorry +theorem Utf.Spec.utf16_decode_none_iff (us : List UInt16) : + Utf16.decode us = none ↔ ∃ l w r, us = l ++ w :: r ∧ Utf.Spec.Unpaired l w r := sorry /-- The number of units: one for each character, and one more for each character from U+10000 up. This is what JavaScript's `length` counts for a string. -/ -theorem Spec.utf16_encode_length (cs : List Char) : +theorem Utf.Spec.utf16_encode_length (cs : List Char) : (Utf16.encode cs).length = cs.length + (cs.filter fun c => 0x10000 ≤ c.toNat).length := sorry /-! ## UTF-16 lossy decoding -/ @@ -221,45 +221,45 @@ theorem Spec.utf16_encode_length (cs : List Char) : of the Unicode Standard sets out, for UTF-16: each well-formed sequence becomes its character, and each maximal subpart of an ill-formed sequence becomes one U+FFFD. -/ -theorem Spec.utf16_decodeLossy_iff (us : List UInt16) (out : List Char) : - Utf16.decodeLossy us = out ↔ Spec.Substitutes16 us out := sorry +theorem Utf.Spec.utf16_decodeLossy_iff (us : List UInt16) (out : List Char) : + Utf16.decodeLossy us = out ↔ Utf.Spec.Substitutes16 us out := sorry /-- An isolated surrogate at the front becomes one U+FFFD, and decoding goes on at the very next unit: a high surrogate never takes the unit after it with it. This is the rule of JavaScript's `toWellFormed`, Rust's `String::from_utf16_lossy` and the WHATWG decoders. -/ -theorem Spec.utf16_decodeLossy_unpaired (w : UInt16) (rest : List UInt16) : - Spec.Unpaired [] w rest → Utf16.decodeLossy (w :: rest) = '�' :: Utf16.decodeLossy rest := sorry +theorem Utf.Spec.utf16_decodeLossy_unpaired (w : UInt16) (rest : List UInt16) : + Utf.Spec.Unpaired [] w rest → Utf16.decodeLossy (w :: rest) = '�' :: Utf16.decodeLossy rest := sorry /-- On well-formed units, the lossy decoder reads what the strict one reads. -/ -theorem Spec.utf16_decodeLossy_of_decode (us : List UInt16) (cs : List Char) : +theorem Utf.Spec.utf16_decodeLossy_of_decode (us : List UInt16) (cs : List Char) : Utf16.decode us = some cs → Utf16.decodeLossy us = cs := sorry /-- On units that are not well formed, the lossy decoder puts in at least one U+FFFD. -/ -theorem Spec.utf16_decodeLossy_replaces (us : List UInt16) : +theorem Utf.Spec.utf16_decodeLossy_replaces (us : List UInt16) : Utf16.decode us = none → '�' ∈ Utf16.decodeLossy us := sorry /-- Well-formed text before bad units comes through unchanged. -/ -theorem Spec.utf16_decodeLossy_encode_append (cs : List Char) (us : List UInt16) : +theorem Utf.Spec.utf16_decodeLossy_encode_append (cs : List Char) (us : List UInt16) : Utf16.decodeLossy (Utf16.encode cs ++ us) = cs ++ Utf16.decodeLossy us := sorry /-- Well-formed text after bad units comes through unchanged: a high surrogate at the end of the bad units never takes the first unit of the text as its partner. -/ -theorem Spec.utf16_decodeLossy_append_encode (us : List UInt16) (cs : List Char) : +theorem Utf.Spec.utf16_decodeLossy_append_encode (us : List UInt16) (cs : List Char) : Utf16.decodeLossy (us ++ Utf16.encode cs) = Utf16.decodeLossy us ++ cs := sorry /-- `toWellFormed` does what ECMAScript's `String.prototype.toWellFormed` does: it keeps the number of units, puts 0xFFFD in place of each isolated surrogate, and keeps every other unit. -/ -theorem Spec.utf16_toWellFormed (us : List UInt16) : +theorem Utf.Spec.utf16_toWellFormed (us : List UInt16) : (Utf16.toWellFormed us).length = us.length ∧ ∀ (i : Nat) (h : i < us.length), - (Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → + (Utf.Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → (Utf16.toWellFormed us)[i]? = some 0xFFFD) ∧ - (¬ Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → + (¬ Utf.Spec.Unpaired (us.take i) us[i] (us.drop (i + 1)) → (Utf16.toWellFormed us)[i]? = some us[i]) := sorry /-! ## UTF-16 as bytes -/ @@ -267,42 +267,42 @@ theorem Spec.utf16_toWellFormed (us : List UInt16) : /-- UTF-16BE writes each unit's high byte first, and UTF-16LE its low byte first, as RFC 2781, section 3.1, and D96 and D97 set out; no byte order mark is written. -/ -theorem Spec.utf16_encodeBytes (cs : List Char) : - Utf16.encodeBE cs = Spec.serializeBE (Spec.encode16 cs) ∧ - Utf16.encodeLE cs = Spec.serializeLE (Spec.encode16 cs) := sorry +theorem Utf.Spec.utf16_encodeBytes (cs : List Char) : + Utf16.encodeBE cs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs) ∧ + Utf16.encodeLE cs = Utf.Spec.serializeLE (Utf.Spec.encode16 cs) := sorry /-- The UTF-16BE decoder returns `cs` exactly when the bytes are the UTF-16BE encoding of `cs`. So it refuses an odd number of bytes, and it reads a leading FE FF as U+FEFF (D96). -/ -theorem Spec.utf16be_decode_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeBE bs = some cs ↔ bs = Spec.serializeBE (Spec.encode16 cs) := sorry +theorem Utf.Spec.utf16be_decode_iff (bs : List UInt8) (cs : List Char) : + Utf16.decodeBE bs = some cs ↔ bs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs) := sorry /-- The UTF-16LE decoder returns `cs` exactly when the bytes are the UTF-16LE encoding of `cs`, so it reads a leading FF FE as U+FEFF (D97). -/ -theorem Spec.utf16le_decode_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeLE bs = some cs ↔ bs = Spec.serializeLE (Spec.encode16 cs) := sorry +theorem Utf.Spec.utf16le_decode_iff (bs : List UInt8) (cs : List Char) : + Utf16.decodeLE bs = some cs ↔ bs = Utf.Spec.serializeLE (Utf.Spec.encode16 cs) := sorry /-- The decoder for the UTF-16 encoding scheme reads `cs` exactly as D98 and RFC 2781, section 4.3, say: after a byte order mark, in the order it gives, the mark not part of the text; with no mark, big-endian. -/ -theorem Spec.utf16_decodeScheme_iff (bs : List UInt8) (cs : List Char) : - Utf16.decodeScheme bs = some cs ↔ Spec.SchemeReads bs cs := sorry +theorem Utf.Spec.utf16_decodeScheme_iff (bs : List UInt8) (cs : List Char) : + Utf16.decodeScheme bs = some cs ↔ Utf.Spec.SchemeReads bs cs := sorry /-- The lossy byte decoders give exactly what the WHATWG Encoding Standard's UTF-16BE and UTF-16LE decoders give, with error mode "replacement". -/ -theorem Spec.utf16_whatwg (bs : List UInt8) : - Utf16.decodeLossyBE bs = Spec.whatwgDecode true bs ∧ - Utf16.decodeLossyLE bs = Spec.whatwgDecode false bs := sorry +theorem Utf.Spec.utf16_whatwg (bs : List UInt8) : + Utf16.decodeLossyBE bs = Utf.Spec.whatwgDecode true bs ∧ + Utf16.decodeLossyLE bs = Utf.Spec.whatwgDecode false bs := sorry /-- On whole units, the lossy byte decoders give the conversion of - section 3.9.6 (`Spec.utf16_decodeLossy_iff`): the WHATWG decoders + section 3.9.6 (`Utf.Spec.utf16_decodeLossy_iff`): the WHATWG decoders and the Unicode Standard agree. -/ -theorem Spec.utf16_decodeLossyBytes_units (us : List UInt16) : - Utf16.decodeLossyBE (Spec.serializeBE us) = Utf16.decodeLossy us ∧ - Utf16.decodeLossyLE (Spec.serializeLE us) = Utf16.decodeLossy us := sorry +theorem Utf.Spec.utf16_decodeLossyBytes_units (us : List UInt16) : + Utf16.decodeLossyBE (Utf.Spec.serializeBE us) = Utf16.decodeLossy us ∧ + Utf16.decodeLossyLE (Utf.Spec.serializeLE us) = Utf16.decodeLossy us := sorry /-! ## UTF-16 test vectors -/ @@ -311,7 +311,7 @@ theorem Spec.utf16_decodeLossyBytes_units (us : List UInt16) : byte order mark, read; the ends of the ranges; isolated surrogates refused and replaced; and the WHATWG decoder's single U+FFFD for a high surrogate cut off with an odd byte. -/ -theorem Spec.utf16_vectors : +theorem Utf.Spec.utf16_vectors : -- D91 and D96 to D98: <004D 0430 4E8C 10302> Utf16.encode ['M', '\u0430', '二', Char.ofNat 0x10302] = [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] ∧ diff --git a/Spec/Defs.lean b/Utf/Spec/Defs.lean similarity index 54% rename from Spec/Defs.lean rename to Utf/Spec/Defs.lean index 8a53590..0281b7e 100644 --- a/Spec/Defs.lean +++ b/Utf/Spec/Defs.lean @@ -3,7 +3,7 @@ Table 3-7 of the Unicode Standard (section 3.9), and from section 3.9.6 of the Unicode Standard, version 16.0 (D93a, D93b), independently of the code in `Utf8/`. Part of the approved specification, with - `Spec.lean`. + `Utf/Spec.lean`. Characters are Lean's `Char`, which holds exactly the Unicode scalar values: 0 to 0xD7FF and 0xE000 to 0x10FFFF. @@ -14,7 +14,7 @@ sequence of each row of the table and the bytes just outside its edges. The definitions of section 3.9.6 have two direct checks here; the standard's own examples, Tables 3-8 to 3-11, are checked through - `Spec.substitution_vectors` together with `Spec.decodeLossy_iff`. + `Utf.Spec.substitution_vectors` together with `Utf.Spec.decodeLossy_iff`. -/ import Utf8 @@ -41,35 +41,35 @@ the preceding one, etc. until all x bits are filled in." -/ /-- The six bits of `n` that go in a continuation byte, `k` bytes from the end, after the marker `10`. -/ -def Spec.contByte (n k : Nat) : UInt8 := +def Utf.Spec.contByte (n k : Nat) : UInt8 := (0b10000000 ||| (n >>> (6 * k) &&& 0b111111)).toUInt8 /-- The octet sequence of RFC 3629, section 3, for one character. -/ -def Spec.encodeChar (c : Char) : List UInt8 := +def Utf.Spec.encodeChar (c : Char) : List UInt8 := let n := c.toNat if n ≤ 0x7F then [n.toUInt8] - else if n ≤ 0x7FF then [(0b11000000 ||| n >>> 6).toUInt8, Spec.contByte n 0] + else if n ≤ 0x7FF then [(0b11000000 ||| n >>> 6).toUInt8, Utf.Spec.contByte n 0] else if n ≤ 0xFFFF then - [(0b11100000 ||| n >>> 12).toUInt8, Spec.contByte n 1, Spec.contByte n 0] + [(0b11100000 ||| n >>> 12).toUInt8, Utf.Spec.contByte n 1, Utf.Spec.contByte n 0] else - [(0b11110000 ||| n >>> 18).toUInt8, Spec.contByte n 2, Spec.contByte n 1, - Spec.contByte n 0] + [(0b11110000 ||| n >>> 18).toUInt8, Utf.Spec.contByte n 2, Utf.Spec.contByte n 1, + Utf.Spec.contByte n 0] /-- The UTF-8 bytes of a list of characters, one after another. -/ -def Spec.encode (cs : List Char) : List UInt8 := - (cs.map Spec.encodeChar).flatten +def Utf.Spec.encode (cs : List Char) : List UInt8 := + (cs.map Utf.Spec.encodeChar).flatten -- The examples of RFC 3629, section 7: "A≢Α." (U+0041 U+2262 U+0391 -- U+002E), the Korean word "hangugeo" (U+D55C U+AD6D U+C5B4), the -- Japanese word "nihongo" (U+65E5 U+672C U+8A9E), and a byte order mark -- followed by U+233B4. -example : Spec.encode ['A', '≢', '\u0391', '.'] = +example : Utf.Spec.encode ['A', '≢', '\u0391', '.'] = [0x41, 0xE2, 0x89, 0xA2, 0xCE, 0x91, 0x2E] := by decide -example : Spec.encode ['한', '국', '어'] = +example : Utf.Spec.encode ['한', '국', '어'] = [0xED, 0x95, 0x9C, 0xEA, 0xB5, 0xAD, 0xEC, 0x96, 0xB4] := by decide -example : Spec.encode ['日', '本', '語'] = +example : Utf.Spec.encode ['日', '本', '語'] = [0xE6, 0x97, 0xA5, 0xE6, 0x9C, 0xAC, 0xE8, 0xAA, 0x9E] := by decide -example : Spec.encode ['\uFEFF', Char.ofNat 0x233B4] = +example : Utf.Spec.encode ['\uFEFF', Char.ofNat 0x233B4] = [0xEF, 0xBB, 0xBF, 0xF0, 0xA3, 0x8E, 0xB4] := by decide /-! ## The Unicode Standard, Table 3-7, Well-Formed UTF-8 Byte Sequences @@ -89,58 +89,58 @@ U+100000..U+10FFFF F4 80..8F 80..BF 80..BF -/ /-- `b` is in the range `lo..hi`, both ends included. -/ -def Spec.InRange (lo hi : Nat) (b : UInt8) : Prop := lo ≤ b.toNat ∧ b.toNat ≤ hi +def Utf.Spec.InRange (lo hi : Nat) (b : UInt8) : Prop := lo ≤ b.toNat ∧ b.toNat ≤ hi /-- One row of Table 3-7: the bytes of a single well-formed sequence. -/ -def Spec.WellFormedChar : List UInt8 → Prop - | [b0] => Spec.InRange 0x00 0x7F b0 - | [b0, b1] => Spec.InRange 0xC2 0xDF b0 ∧ Spec.InRange 0x80 0xBF b1 +def Utf.Spec.WellFormedChar : List UInt8 → Prop + | [b0] => Utf.Spec.InRange 0x00 0x7F b0 + | [b0, b1] => Utf.Spec.InRange 0xC2 0xDF b0 ∧ Utf.Spec.InRange 0x80 0xBF b1 | [b0, b1, b2] => - ((Spec.InRange 0xE0 0xE0 b0 ∧ Spec.InRange 0xA0 0xBF b1) ∨ - (Spec.InRange 0xE1 0xEC b0 ∧ Spec.InRange 0x80 0xBF b1) ∨ - (Spec.InRange 0xED 0xED b0 ∧ Spec.InRange 0x80 0x9F b1) ∨ - (Spec.InRange 0xEE 0xEF b0 ∧ Spec.InRange 0x80 0xBF b1)) ∧ - Spec.InRange 0x80 0xBF b2 + ((Utf.Spec.InRange 0xE0 0xE0 b0 ∧ Utf.Spec.InRange 0xA0 0xBF b1) ∨ + (Utf.Spec.InRange 0xE1 0xEC b0 ∧ Utf.Spec.InRange 0x80 0xBF b1) ∨ + (Utf.Spec.InRange 0xED 0xED b0 ∧ Utf.Spec.InRange 0x80 0x9F b1) ∨ + (Utf.Spec.InRange 0xEE 0xEF b0 ∧ Utf.Spec.InRange 0x80 0xBF b1)) ∧ + Utf.Spec.InRange 0x80 0xBF b2 | [b0, b1, b2, b3] => - ((Spec.InRange 0xF0 0xF0 b0 ∧ Spec.InRange 0x90 0xBF b1) ∨ - (Spec.InRange 0xF1 0xF3 b0 ∧ Spec.InRange 0x80 0xBF b1) ∨ - (Spec.InRange 0xF4 0xF4 b0 ∧ Spec.InRange 0x80 0x8F b1)) ∧ - Spec.InRange 0x80 0xBF b2 ∧ Spec.InRange 0x80 0xBF b3 + ((Utf.Spec.InRange 0xF0 0xF0 b0 ∧ Utf.Spec.InRange 0x90 0xBF b1) ∨ + (Utf.Spec.InRange 0xF1 0xF3 b0 ∧ Utf.Spec.InRange 0x80 0xBF b1) ∨ + (Utf.Spec.InRange 0xF4 0xF4 b0 ∧ Utf.Spec.InRange 0x80 0x8F b1)) ∧ + Utf.Spec.InRange 0x80 0xBF b2 ∧ Utf.Spec.InRange 0x80 0xBF b3 | _ => False /-- A well-formed byte sequence: well-formed sequences from Table 3-7, one after another. -/ -inductive Spec.WellFormed : List UInt8 → Prop where - | nil : Spec.WellFormed [] +inductive Utf.Spec.WellFormed : List UInt8 → Prop where + | nil : Utf.Spec.WellFormed [] | cons {s rest : List UInt8} : - Spec.WellFormedChar s → Spec.WellFormed rest → Spec.WellFormed (s ++ rest) + Utf.Spec.WellFormedChar s → Utf.Spec.WellFormed rest → Utf.Spec.WellFormed (s ++ rest) -example : Spec.WellFormedChar [0xF4, 0x8F, 0xBF, 0xBF] := by - unfold Spec.WellFormedChar Spec.InRange; decide -example : ¬ Spec.WellFormedChar [0xED, 0xA0, 0x80] := by - unfold Spec.WellFormedChar Spec.InRange; decide +example : Utf.Spec.WellFormedChar [0xF4, 0x8F, 0xBF, 0xBF] := by + unfold Utf.Spec.WellFormedChar Utf.Spec.InRange; decide +example : ¬ Utf.Spec.WellFormedChar [0xED, 0xA0, 0x80] := by + unfold Utf.Spec.WellFormedChar Utf.Spec.InRange; decide -- The first and last sequence of each row of Table 3-7. example : - Spec.WellFormedChar [0x00] ∧ Spec.WellFormedChar [0x7F] ∧ - Spec.WellFormedChar [0xC2, 0x80] ∧ Spec.WellFormedChar [0xDF, 0xBF] ∧ - Spec.WellFormedChar [0xE0, 0xA0, 0x80] ∧ Spec.WellFormedChar [0xE0, 0xBF, 0xBF] ∧ - Spec.WellFormedChar [0xE1, 0x80, 0x80] ∧ Spec.WellFormedChar [0xEC, 0xBF, 0xBF] ∧ - Spec.WellFormedChar [0xED, 0x80, 0x80] ∧ Spec.WellFormedChar [0xED, 0x9F, 0xBF] ∧ - Spec.WellFormedChar [0xEE, 0x80, 0x80] ∧ Spec.WellFormedChar [0xEF, 0xBF, 0xBF] ∧ - Spec.WellFormedChar [0xF0, 0x90, 0x80, 0x80] ∧ Spec.WellFormedChar [0xF0, 0xBF, 0xBF, 0xBF] ∧ - Spec.WellFormedChar [0xF1, 0x80, 0x80, 0x80] ∧ Spec.WellFormedChar [0xF3, 0xBF, 0xBF, 0xBF] ∧ - Spec.WellFormedChar [0xF4, 0x80, 0x80, 0x80] ∧ Spec.WellFormedChar [0xF4, 0x8F, 0xBF, 0xBF] := by - simp only [Spec.WellFormedChar, Spec.InRange, UInt8.toNat]; and_intros <;> decide + Utf.Spec.WellFormedChar [0x00] ∧ Utf.Spec.WellFormedChar [0x7F] ∧ + Utf.Spec.WellFormedChar [0xC2, 0x80] ∧ Utf.Spec.WellFormedChar [0xDF, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xE0, 0xA0, 0x80] ∧ Utf.Spec.WellFormedChar [0xE0, 0xBF, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xE1, 0x80, 0x80] ∧ Utf.Spec.WellFormedChar [0xEC, 0xBF, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xED, 0x80, 0x80] ∧ Utf.Spec.WellFormedChar [0xED, 0x9F, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xEE, 0x80, 0x80] ∧ Utf.Spec.WellFormedChar [0xEF, 0xBF, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xF0, 0x90, 0x80, 0x80] ∧ Utf.Spec.WellFormedChar [0xF0, 0xBF, 0xBF, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xF1, 0x80, 0x80, 0x80] ∧ Utf.Spec.WellFormedChar [0xF3, 0xBF, 0xBF, 0xBF] ∧ + Utf.Spec.WellFormedChar [0xF4, 0x80, 0x80, 0x80] ∧ Utf.Spec.WellFormedChar [0xF4, 0x8F, 0xBF, 0xBF] := by + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange, UInt8.toNat]; and_intros <;> decide -- Just outside the edges of the rows. example : - ¬ Spec.WellFormedChar [0x80] ∧ ¬ Spec.WellFormedChar [0xC1, 0xBF] ∧ - ¬ Spec.WellFormedChar [0xC2, 0x7F] ∧ ¬ Spec.WellFormedChar [0xC2, 0xC0] ∧ - ¬ Spec.WellFormedChar [0xE0, 0x9F, 0xBF] ∧ ¬ Spec.WellFormedChar [0xED, 0xA0, 0x80] ∧ - ¬ Spec.WellFormedChar [0xF0, 0x8F, 0xBF, 0xBF] ∧ ¬ Spec.WellFormedChar [0xF4, 0x90, 0x80, 0x80] ∧ - ¬ Spec.WellFormedChar [0xF5, 0x80, 0x80, 0x80] := by - simp only [Spec.WellFormedChar, Spec.InRange, UInt8.toNat]; and_intros <;> decide + ¬ Utf.Spec.WellFormedChar [0x80] ∧ ¬ Utf.Spec.WellFormedChar [0xC1, 0xBF] ∧ + ¬ Utf.Spec.WellFormedChar [0xC2, 0x7F] ∧ ¬ Utf.Spec.WellFormedChar [0xC2, 0xC0] ∧ + ¬ Utf.Spec.WellFormedChar [0xE0, 0x9F, 0xBF] ∧ ¬ Utf.Spec.WellFormedChar [0xED, 0xA0, 0x80] ∧ + ¬ Utf.Spec.WellFormedChar [0xF0, 0x8F, 0xBF, 0xBF] ∧ ¬ Utf.Spec.WellFormedChar [0xF4, 0x90, 0x80, 0x80] ∧ + ¬ Utf.Spec.WellFormedChar [0xF5, 0x80, 0x80, 0x80] := by + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange, UInt8.toNat]; and_intros <;> decide /-! ## The Unicode Standard, section 3.9.6, U+FFFD Substitution of Maximal Subparts @@ -170,54 +170,54 @@ subsequence starting there is a prefix of them (`<+:`). That sequence may hold more than one character. The note under D93b speaks of one character: "as many code units as match the beginning of a well-formed character". Where this definition is used, the two - readings agree. `Spec.MaximalSubpart` asks it only of bytes at an + readings agree. `Utf.Spec.MaximalSubpart` asks it only of bytes at an unconvertible offset. By D93a, no well-formed sequence starts there, not even the bytes of one whole character. So `p` stops inside the bytes of the first character of the well-formed sequence. That is, `p` is the start of one character. The proofs show this as - `Utf8.Proofs.Lossy.initial_iff`, in `Proofs/Lossy/Table.lean`. -/ -def Spec.Initial (p : List UInt8) : Prop := - ∃ s, Spec.WellFormed s ∧ p <+: s + `Utf8.Proofs.Lossy.initial_iff`, in `Utf/Proofs/Lossy/Table.lean`. -/ +def Utf.Spec.Initial (p : List UInt8) : Prop := + ∃ s, Utf.Spec.WellFormed s ∧ p <+: s /-- D93a: no subsequence starting at the front of `bs` is well formed. (The empty one is left out: it starts nowhere.) -/ -def Spec.Unconvertible (bs : List UInt8) : Prop := - ∀ s, s ≠ [] → s <+: bs → ¬ Spec.WellFormed s +def Utf.Spec.Unconvertible (bs : List UInt8) : Prop := + ∀ s, s ≠ [] → s <+: bs → ¬ Utf.Spec.WellFormed s /-- D93b: `p` is the maximal subpart at the front of `p ++ rest`. The front is an unconvertible offset, and `p` is the longest prefix there that is either the start of a well-formed sequence or one byte long. -/ -def Spec.MaximalSubpart (p rest : List UInt8) : Prop := - p ≠ [] ∧ Spec.Unconvertible (p ++ rest) ∧ - (Spec.Initial p ∨ p.length = 1) ∧ - ∀ q, q <+: p ++ rest → (Spec.Initial q ∨ q.length = 1) → q.length ≤ p.length +def Utf.Spec.MaximalSubpart (p rest : List UInt8) : Prop := + p ≠ [] ∧ Utf.Spec.Unconvertible (p ++ rest) ∧ + (Utf.Spec.Initial p ∨ p.length = 1) ∧ + ∀ q, q <+: p ++ rest → (Utf.Spec.Initial q ∨ q.length = 1) → q.length ≤ p.length /-- Conversion with U+FFFD substitution of maximal subparts: converting `bs` gives `out`. Where a character's well-formed sequence starts, it gives the character; at an unconvertible offset, the maximal subpart gives one U+FFFD; either way, conversion proceeds after the bytes used. -/ -inductive Spec.Substitutes : List UInt8 → List Char → Prop where - | nil : Spec.Substitutes [] [] +inductive Utf.Spec.Substitutes : List UInt8 → List Char → Prop where + | nil : Utf.Spec.Substitutes [] [] | char {c : Char} {rest : List UInt8} {out : List Char} : - Spec.Substitutes rest out → Spec.Substitutes (Spec.encodeChar c ++ rest) (c :: out) + Utf.Spec.Substitutes rest out → Utf.Spec.Substitutes (Utf.Spec.encodeChar c ++ rest) (c :: out) | replace {p rest : List UInt8} {out : List Char} : - Spec.MaximalSubpart p rest → Spec.Substitutes rest out → - Spec.Substitutes (p ++ rest) ('�' :: out) + Utf.Spec.MaximalSubpart p rest → Utf.Spec.Substitutes rest out → + Utf.Spec.Substitutes (p ++ rest) ('�' :: out) -- A well-formed sequence starts at the front of `E2 82 AC` (U+20AC), so -- that offset is not unconvertible. -example : ¬ Spec.Unconvertible [0xE2, 0x82, 0xAC] := by +example : ¬ Utf.Spec.Unconvertible [0xE2, 0x82, 0xAC] := by intro h apply h [0xE2, 0x82, 0xAC] (by simp) (List.prefix_refl _) - have : Spec.WellFormedChar [0xE2, 0x82, 0xAC] := by - simp only [Spec.WellFormedChar, Spec.InRange, UInt8.toNat]; decide - simpa using Spec.WellFormed.cons this Spec.WellFormed.nil + have : Utf.Spec.WellFormedChar [0xE2, 0x82, 0xAC] := by + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange, UInt8.toNat]; decide + simpa using Utf.Spec.WellFormed.cons this Utf.Spec.WellFormed.nil -- `F0 9F 98` is the start of a well-formed sequence, `F0 9F 98 80`. -example : Spec.Initial [0xF0, 0x9F, 0x98] := by +example : Utf.Spec.Initial [0xF0, 0x9F, 0x98] := by refine ⟨[0xF0, 0x9F, 0x98, 0x80], ?_, ⟨[0x80], rfl⟩⟩ - have : Spec.WellFormedChar [0xF0, 0x9F, 0x98, 0x80] := by - simp only [Spec.WellFormedChar, Spec.InRange, UInt8.toNat]; decide - simpa using Spec.WellFormed.cons this Spec.WellFormed.nil + have : Utf.Spec.WellFormedChar [0xF0, 0x9F, 0x98, 0x80] := by + simp only [Utf.Spec.WellFormedChar, Utf.Spec.InRange, UInt8.toNat]; decide + simpa using Utf.Spec.WellFormed.cons this Utf.Spec.WellFormed.nil diff --git a/Spec/Utf16.lean b/Utf/Spec/Utf16.lean similarity index 70% rename from Spec/Utf16.lean rename to Utf/Spec/Utf16.lean index f252909..f4b4c19 100644 --- a/Spec/Utf16.lean +++ b/Utf/Spec/Utf16.lean @@ -4,7 +4,7 @@ D96 to D98, and section 3.9.6, D93a and D93b), from the WHATWG Encoding Standard (section 14.2.1, the shared UTF-16 decoder), and from ECMAScript's `String.prototype.toWellFormed`, independently of the code - in `Utf16/`. Part of the approved specification, with `Spec.lean`. + in `Utf16/`. Part of the approved specification, with `Utf/Spec.lean`. Code units are `UInt16`. Characters are Lean's `Char`, which holds exactly the Unicode scalar values: 0 to 0xD7FF and 0xE000 to 0x10FFFF. @@ -14,7 +14,7 @@ ranges, and on the cases of the WHATWG decoder's steps. -/ -import Spec.Defs +import Utf.Spec.Defs /-! ## Surrogates: the Unicode Standard, D72, D74, D75 @@ -32,18 +32,18 @@ D75 Surrogate pair: A representation for a single abstract character -/ /-- D72: a high-surrogate code unit. -/ -def Spec.High (w : UInt16) : Prop := 0xD800 ≤ w.toNat ∧ w.toNat ≤ 0xDBFF +def Utf.Spec.High (w : UInt16) : Prop := 0xD800 ≤ w.toNat ∧ w.toNat ≤ 0xDBFF /-- D74: a low-surrogate code unit. -/ -def Spec.Low (w : UInt16) : Prop := 0xDC00 ≤ w.toNat ∧ w.toNat ≤ 0xDFFF +def Utf.Spec.Low (w : UInt16) : Prop := 0xDC00 ≤ w.toNat ∧ w.toNat ≤ 0xDFFF /-- The unit `w`, with the units `l` before it and `r` after it, is an isolated surrogate, in no surrogate pair (D75): a high surrogate that no low surrogate follows, or a low surrogate that no high surrogate comes right before. -/ -def Spec.Unpaired (l : List UInt16) (w : UInt16) (r : List UInt16) : Prop := - (Spec.High w ∧ ¬ ∃ v r', r = v :: r' ∧ Spec.Low v) ∨ - (Spec.Low w ∧ ¬ ∃ l' v, l = l' ++ [v] ∧ Spec.High v) +def Utf.Spec.Unpaired (l : List UInt16) (w : UInt16) (r : List UInt16) : Prop := + (Utf.Spec.High w ∧ ¬ ∃ v r', r = v :: r' ∧ Utf.Spec.Low v) ∨ + (Utf.Spec.Low w ∧ ¬ ∃ l' v, l = l' ++ [v] ∧ Utf.Spec.High v) /-! ## RFC 2781, section 2.1, Encoding UTF-16 @@ -64,7 +64,7 @@ def Spec.Unpaired (l : List UInt16) (w : UInt16) (r : List UInt16) : Prop := /-- The units of one character, by the four steps of RFC 2781, section 2.1. -/ -def Spec.encodeChar16 (c : Char) : List UInt16 := +def Utf.Spec.encodeChar16 (c : Char) : List UInt16 := let u := c.toNat if u < 0x10000 then [u.toUInt16] else @@ -72,20 +72,20 @@ def Spec.encodeChar16 (c : Char) : List UInt16 := [(0xD800 ||| u' >>> 10).toUInt16, (0xDC00 ||| u' &&& 0x3FF).toUInt16] /-- The UTF-16 code units of a list of characters, one after another. -/ -def Spec.encode16 (cs : List Char) : List UInt16 := - (cs.map Spec.encodeChar16).flatten +def Utf.Spec.encode16 (cs : List Char) : List UInt16 := + (cs.map Utf.Spec.encodeChar16).flatten -- D91: "In UTF-16, the code point sequence <004D, 0430, 4E8C, 10302> is -- represented as <004D 0430 4E8C D800 DF02>". -example : Spec.encode16 ['M', '\u0430', '二', Char.ofNat 0x10302] = +example : Utf.Spec.encode16 ['M', '\u0430', '二', Char.ofNat 0x10302] = [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] := by decide +kernel -- RFC 2781, section 5: U+12345, then "=Ra". -example : Spec.encode16 [Char.ofNat 0x12345, '=', 'R', 'a'] = +example : Utf.Spec.encode16 [Char.ofNat 0x12345, '=', 'R', 'a'] = [0xD808, 0xDF45, 0x003D, 0x0052, 0x0061] := by decide +kernel -- The ends of the ranges: U+D7FF, U+E000, U+FFFF, U+10000, U+10FFFF. -example : Spec.encode16 [Char.ofNat 0xD7FF, Char.ofNat 0xE000, Char.ofNat 0xFFFF, +example : Utf.Spec.encode16 [Char.ofNat 0xD7FF, Char.ofNat 0xE000, Char.ofNat 0xFFFF, Char.ofNat 0x10000, Char.ofNat 0x10FFFF] = [0xD7FF, 0xE000, 0xFFFF, 0xD800, 0xDC00, 0xDBFF, 0xDFFF] := by decide +kernel @@ -111,39 +111,39 @@ Note: wwww = uuuuu - 1 /-- The code units of a single well-formed sequence in UTF-16: one unit outside D800..DFFF, or a high surrogate then a low surrogate. -/ -def Spec.WellFormedChar16 : List UInt16 → Prop +def Utf.Spec.WellFormedChar16 : List UInt16 → Prop | [w] => w.toNat ≤ 0xD7FF ∨ 0xE000 ≤ w.toNat - | [w1, w2] => Spec.High w1 ∧ Spec.Low w2 + | [w1, w2] => Utf.Spec.High w1 ∧ Utf.Spec.Low w2 | _ => False /-- A well-formed UTF-16 code unit sequence: well-formed sequences, one after another. -/ -inductive Spec.WellFormed16 : List UInt16 → Prop where - | nil : Spec.WellFormed16 [] +inductive Utf.Spec.WellFormed16 : List UInt16 → Prop where + | nil : Utf.Spec.WellFormed16 [] | cons {s rest : List UInt16} : - Spec.WellFormedChar16 s → Spec.WellFormed16 rest → Spec.WellFormed16 (s ++ rest) + Utf.Spec.WellFormedChar16 s → Utf.Spec.WellFormed16 rest → Utf.Spec.WellFormed16 (s ++ rest) -example : Spec.WellFormedChar16 [0xD800, 0xDF02] := by - unfold Spec.WellFormedChar16 Spec.High Spec.Low; decide -example : ¬ Spec.WellFormedChar16 [0xDF02, 0xD800] := by - unfold Spec.WellFormedChar16 Spec.High Spec.Low; decide -example : Spec.WellFormedChar16 [0xD7FF] ∧ Spec.WellFormedChar16 [0xE000] ∧ - ¬ Spec.WellFormedChar16 [0xD800] ∧ ¬ Spec.WellFormedChar16 [0xDFFF] := by - unfold Spec.WellFormedChar16; decide +example : Utf.Spec.WellFormedChar16 [0xD800, 0xDF02] := by + unfold Utf.Spec.WellFormedChar16 Utf.Spec.High Utf.Spec.Low; decide +example : ¬ Utf.Spec.WellFormedChar16 [0xDF02, 0xD800] := by + unfold Utf.Spec.WellFormedChar16 Utf.Spec.High Utf.Spec.Low; decide +example : Utf.Spec.WellFormedChar16 [0xD7FF] ∧ Utf.Spec.WellFormedChar16 [0xE000] ∧ + ¬ Utf.Spec.WellFormedChar16 [0xD800] ∧ ¬ Utf.Spec.WellFormedChar16 [0xDFFF] := by + unfold Utf.Spec.WellFormedChar16; decide -- In <0041 D800 DC00 DC00>, the second low surrogate is isolated: the -- unit before it is a low surrogate. The high surrogate is not: a low -- surrogate follows it. -example : Spec.Unpaired [0x0041, 0xD800, 0xDC00] 0xDC00 [] := by - refine Or.inr ⟨by unfold Spec.Low; decide, ?_⟩ +example : Utf.Spec.Unpaired [0x0041, 0xD800, 0xDC00] 0xDC00 [] := by + refine Or.inr ⟨by unfold Utf.Spec.Low; decide, ?_⟩ rintro ⟨l', v, h, hv⟩ have : v = 0xDC00 := by have := congrArg List.getLast? h; simpa using this.symm - subst this; unfold Spec.High at hv; revert hv; decide -example : ¬ Spec.Unpaired [0x0041] 0xD800 [0xDC00, 0xDC00] := by + subst this; unfold Utf.Spec.High at hv; revert hv; decide +example : ¬ Utf.Spec.Unpaired [0x0041] 0xD800 [0xDC00, 0xDC00] := by rintro (⟨-, h⟩ | ⟨hl, -⟩) - · exact h ⟨0xDC00, [0xDC00], rfl, by unfold Spec.Low; decide⟩ - · unfold Spec.Low at hl; revert hl; decide + · exact h ⟨0xDC00, [0xDC00], rfl, by unfold Utf.Spec.Low; decide⟩ + · unfold Utf.Spec.Low at hl; revert hl; decide /-! ## The Unicode Standard, section 3.9.6, for UTF-16 @@ -155,44 +155,44 @@ by one U+FFFD, and conversion goes on after it. /-- `p` is the initial subsequence of a well-formed UTF-16 code unit sequence. - As with `Spec.Initial`, that sequence may hold more than one + As with `Utf.Spec.Initial`, that sequence may hold more than one character. Where this definition is used, that changes no answer. - `Spec.MaximalSubpart16` asks it only of units at an unconvertible + `Utf.Spec.MaximalSubpart16` asks it only of units at an unconvertible offset. By D93a, no well-formed sequence starts there, not even the units of one whole character. So `p` stops inside the units of the first character, which are at most two. That is, `p` is empty or one high surrogate. The proofs show that each maximal subpart is one unit, as `Utf16.Proofs.maximalSubpart_iff`, in - `Proofs/Utf16/Lossy.lean`. -/ -def Spec.Initial16 (p : List UInt16) : Prop := - ∃ s, Spec.WellFormed16 s ∧ p <+: s + `Utf/Proofs/Utf16/Lossy.lean`. -/ +def Utf.Spec.Initial16 (p : List UInt16) : Prop := + ∃ s, Utf.Spec.WellFormed16 s ∧ p <+: s /-- D93a: no subsequence starting at the front of `us` is well formed. (The empty one is left out: it starts nowhere.) -/ -def Spec.Unconvertible16 (us : List UInt16) : Prop := - ∀ s, s ≠ [] → s <+: us → ¬ Spec.WellFormed16 s +def Utf.Spec.Unconvertible16 (us : List UInt16) : Prop := + ∀ s, s ≠ [] → s <+: us → ¬ Utf.Spec.WellFormed16 s /-- D93b: `p` is the maximal subpart at the front of `p ++ rest`. The front is an unconvertible offset, and `p` is the longest prefix there that is either the start of a well-formed sequence or one unit long. -/ -def Spec.MaximalSubpart16 (p rest : List UInt16) : Prop := - p ≠ [] ∧ Spec.Unconvertible16 (p ++ rest) ∧ - (Spec.Initial16 p ∨ p.length = 1) ∧ - ∀ q, q <+: p ++ rest → (Spec.Initial16 q ∨ q.length = 1) → q.length ≤ p.length +def Utf.Spec.MaximalSubpart16 (p rest : List UInt16) : Prop := + p ≠ [] ∧ Utf.Spec.Unconvertible16 (p ++ rest) ∧ + (Utf.Spec.Initial16 p ∨ p.length = 1) ∧ + ∀ q, q <+: p ++ rest → (Utf.Spec.Initial16 q ∨ q.length = 1) → q.length ≤ p.length /-- Conversion with U+FFFD substitution of maximal subparts: converting the units `us` gives `out`. Where a character's units start, it gives the character; at an unconvertible offset, the maximal subpart gives one U+FFFD; either way, conversion proceeds after the units used. -/ -inductive Spec.Substitutes16 : List UInt16 → List Char → Prop where - | nil : Spec.Substitutes16 [] [] +inductive Utf.Spec.Substitutes16 : List UInt16 → List Char → Prop where + | nil : Utf.Spec.Substitutes16 [] [] | char {c : Char} {rest : List UInt16} {out : List Char} : - Spec.Substitutes16 rest out → Spec.Substitutes16 (Spec.encodeChar16 c ++ rest) (c :: out) + Utf.Spec.Substitutes16 rest out → Utf.Spec.Substitutes16 (Utf.Spec.encodeChar16 c ++ rest) (c :: out) | replace {p rest : List UInt16} {out : List Char} : - Spec.MaximalSubpart16 p rest → Spec.Substitutes16 rest out → - Spec.Substitutes16 (p ++ rest) ('�' :: out) + Utf.Spec.MaximalSubpart16 p rest → Utf.Spec.Substitutes16 rest out → + Utf.Spec.Substitutes16 (p ++ rest) ('�' :: out) /-! ## The encoding schemes: RFC 2781, section 3.1, and D96 to D98 @@ -226,41 +226,41 @@ is the character ZERO WIDTH NO-BREAK SPACE. -/ /-- The bytes of code units, big-endian. -/ -def Spec.serializeBE (us : List UInt16) : List UInt8 := +def Utf.Spec.serializeBE (us : List UInt16) : List UInt8 := (us.map fun (u : UInt16) => [(u >>> 8).toUInt8, (u &&& 0xFF).toUInt8]).flatten /-- The bytes of code units, little-endian. -/ -def Spec.serializeLE (us : List UInt16) : List UInt8 := +def Utf.Spec.serializeLE (us : List UInt16) : List UInt8 := (us.map fun (u : UInt16) => [(u &&& 0xFF).toUInt8, (u >>> 8).toUInt8]).flatten /-- D98, read: the bytes `bs`, in the UTF-16 encoding scheme, are the text `cs`. A byte order mark FE FF and the text big-endian; a mark FF FE and the text little-endian; or no mark, and the text big-endian. -/ -def Spec.SchemeReads (bs : List UInt8) (cs : List Char) : Prop := - bs = [0xFE, 0xFF] ++ Spec.serializeBE (Spec.encode16 cs) ∨ - bs = [0xFF, 0xFE] ++ Spec.serializeLE (Spec.encode16 cs) ∨ +def Utf.Spec.SchemeReads (bs : List UInt8) (cs : List Char) : Prop := + bs = [0xFE, 0xFF] ++ Utf.Spec.serializeBE (Utf.Spec.encode16 cs) ∨ + bs = [0xFF, 0xFE] ++ Utf.Spec.serializeLE (Utf.Spec.encode16 cs) ∨ (bs.take 2 ≠ [0xFE, 0xFF] ∧ bs.take 2 ≠ [0xFF, 0xFE] ∧ - bs = Spec.serializeBE (Spec.encode16 cs)) + bs = Utf.Spec.serializeBE (Utf.Spec.encode16 cs)) -- D96, D97 and D98's examples, for <004D 0430 4E8C D800 DF02>. -example : Spec.serializeBE [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] = +example : Utf.Spec.serializeBE [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] = [0x00, 0x4D, 0x04, 0x30, 0x4E, 0x8C, 0xD8, 0x00, 0xDF, 0x02] := by decide +kernel -example : Spec.serializeLE [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] = +example : Utf.Spec.serializeLE [0x004D, 0x0430, 0x4E8C, 0xD800, 0xDF02] = [0x4D, 0x00, 0x30, 0x04, 0x8C, 0x4E, 0x00, 0xD8, 0x02, 0xDF] := by decide +kernel -- D98's three serializations of it all read as its characters. -example : Spec.SchemeReads [0xFE, 0xFF, 0x00, 0x4D, 0x04, 0x30, 0x4E, 0x8C, 0xD8, 0x00, 0xDF, 0x02] +example : Utf.Spec.SchemeReads [0xFE, 0xFF, 0x00, 0x4D, 0x04, 0x30, 0x4E, 0x8C, 0xD8, 0x00, 0xDF, 0x02] ['M', '\u0430', '二', Char.ofNat 0x10302] ∧ - Spec.SchemeReads [0xFF, 0xFE, 0x4D, 0x00, 0x30, 0x04, 0x8C, 0x4E, 0x00, 0xD8, 0x02, 0xDF] + Utf.Spec.SchemeReads [0xFF, 0xFE, 0x4D, 0x00, 0x30, 0x04, 0x8C, 0x4E, 0x00, 0xD8, 0x02, 0xDF] ['M', '\u0430', '二', Char.ofNat 0x10302] ∧ - Spec.SchemeReads [0x00, 0x4D, 0x04, 0x30, 0x4E, 0x8C, 0xD8, 0x00, 0xDF, 0x02] + Utf.Spec.SchemeReads [0x00, 0x4D, 0x04, 0x30, 0x4E, 0x8C, 0xD8, 0x00, 0xDF, 0x02] ['M', '\u0430', '二', Char.ofNat 0x10302] := by - unfold Spec.SchemeReads; decide +kernel + unfold Utf.Spec.SchemeReads; decide +kernel -- With a mark, the mark is not text; text that starts with U+FEFF and has -- no mark cannot be told from text after a mark, and reads as the latter. -example : ¬ Spec.SchemeReads [0xFE, 0xFF, 0x00, 0x41] ['\uFEFF', 'A'] := by - unfold Spec.SchemeReads; decide +kernel +example : ¬ Utf.Spec.SchemeReads [0xFE, 0xFF, 0x00, 0x41] ['\uFEFF', 'A'] := by + unfold Utf.Spec.SchemeReads; decide +kernel /-! ## The WHATWG Encoding Standard, section 14.2.1, shared UTF-16 decoder @@ -315,14 +315,14 @@ two for the end, is enough. /-- The shared UTF-16 decoder, run over the queue `q` from the state `lead`, `sur`, with error mode "replacement". -/ -def Spec.whatwgRun (isBE : Bool) : Nat → Option UInt8 → Option Nat → List UInt8 → List Char +def Utf.Spec.whatwgRun (isBE : Bool) : Nat → Option UInt8 → Option Nat → List UInt8 → List Char | 0, _, _, _ => [] | fuel + 1, lead, sur, [] => -- Steps 1 and 2. - if lead.isSome ∨ sur.isSome then '�' :: Spec.whatwgRun isBE fuel none none [] else [] + if lead.isSome ∨ sur.isSome then '�' :: Utf.Spec.whatwgRun isBE fuel none none [] else [] | fuel + 1, none, sur, byte :: q => -- Step 3. - Spec.whatwgRun isBE fuel (some byte) sur q + Utf.Spec.whatwgRun isBE fuel (some byte) sur q | fuel + 1, some leadByte, sur, byte :: q => -- Steps 4 and 5. let codeUnit : Nat := @@ -332,36 +332,36 @@ def Spec.whatwgRun (isBE : Bool) : Nat → Option UInt8 → Option Nat → List -- Step 6. if 0xDC00 ≤ codeUnit ∧ codeUnit ≤ 0xDFFF then Char.ofNat (0x10000 + ((leadingSurrogate - 0xD800) <<< 10) + (codeUnit - 0xDC00)) :: - Spec.whatwgRun isBE fuel none none q + Utf.Spec.whatwgRun isBE fuel none none q else let byte1 := (codeUnit >>> 8).toUInt8 let byte2 := (codeUnit &&& 0x00FF).toUInt8 let bytes := if isBE then [byte1, byte2] else [byte2, byte1] - '�' :: Spec.whatwgRun isBE fuel none none (bytes ++ q) + '�' :: Utf.Spec.whatwgRun isBE fuel none none (bytes ++ q) | none => -- Steps 7 to 9. - if 0xD800 ≤ codeUnit ∧ codeUnit ≤ 0xDBFF then Spec.whatwgRun isBE fuel none (some codeUnit) q - else if 0xDC00 ≤ codeUnit ∧ codeUnit ≤ 0xDFFF then '�' :: Spec.whatwgRun isBE fuel none none q - else Char.ofNat codeUnit :: Spec.whatwgRun isBE fuel none none q + if 0xD800 ≤ codeUnit ∧ codeUnit ≤ 0xDBFF then Utf.Spec.whatwgRun isBE fuel none (some codeUnit) q + else if 0xDC00 ≤ codeUnit ∧ codeUnit ≤ 0xDFFF then '�' :: Utf.Spec.whatwgRun isBE fuel none none q + else Char.ofNat codeUnit :: Utf.Spec.whatwgRun isBE fuel none none q /-- The WHATWG UTF-16BE (`isBE`) or UTF-16LE decoder over all of `bs`, from the initial state. -/ -def Spec.whatwgDecode (isBE : Bool) (bs : List UInt8) : List Char := - Spec.whatwgRun isBE (3 * bs.length + 2) none none bs +def Utf.Spec.whatwgDecode (isBE : Bool) (bs : List UInt8) : List Char := + Utf.Spec.whatwgRun isBE (3 * bs.length + 2) none none bs -- Each step: a pair (6.3), a leading surrogate then a unit that is not -- trailing (6.4 to 6.7, the unit read again), a trailing surrogate alone -- (8), a plain unit (9), a byte left over (1), a leading surrogate left -- over (1), and both left over, which give one error between them. -example : Spec.whatwgDecode true [0xD8, 0x3D, 0xDE, 0x00] = [Char.ofNat 0x1F600] ∧ - Spec.whatwgDecode true [0xD8, 0x00, 0x00, 0x41] = ['�', 'A'] ∧ - Spec.whatwgDecode true [0xD8, 0x00, 0xD8, 0x00, 0xDC, 0x00] = ['�', Char.ofNat 0x10000] ∧ - Spec.whatwgDecode false [0x00, 0xDC, 0x41, 0x00] = ['�', 'A'] ∧ - Spec.whatwgDecode false [0x41, 0x00] = ['A'] ∧ - Spec.whatwgDecode false [0x41, 0x00, 0x42] = ['A', '�'] ∧ - Spec.whatwgDecode true [0xD8, 0x00] = ['�'] ∧ - Spec.whatwgDecode false [0x00, 0xD8, 0x41] = ['�'] ∧ - Spec.whatwgDecode true [0xFE, 0xFF, 0x00, 0x41] = ['\uFEFF', 'A'] := by +example : Utf.Spec.whatwgDecode true [0xD8, 0x3D, 0xDE, 0x00] = [Char.ofNat 0x1F600] ∧ + Utf.Spec.whatwgDecode true [0xD8, 0x00, 0x00, 0x41] = ['�', 'A'] ∧ + Utf.Spec.whatwgDecode true [0xD8, 0x00, 0xD8, 0x00, 0xDC, 0x00] = ['�', Char.ofNat 0x10000] ∧ + Utf.Spec.whatwgDecode false [0x00, 0xDC, 0x41, 0x00] = ['�', 'A'] ∧ + Utf.Spec.whatwgDecode false [0x41, 0x00] = ['A'] ∧ + Utf.Spec.whatwgDecode false [0x41, 0x00, 0x42] = ['A', '�'] ∧ + Utf.Spec.whatwgDecode true [0xD8, 0x00] = ['�'] ∧ + Utf.Spec.whatwgDecode false [0x00, 0xD8, 0x41] = ['�'] ∧ + Utf.Spec.whatwgDecode true [0xFE, 0xFF, 0x00, 0x41] = ['\uFEFF', 'A'] := by decide +kernel /-! ## ECMAScript, `String.prototype.toWellFormed` @@ -375,5 +375,5 @@ surrogate, or a leading surrogate that no trailing surrogate follows. Read from the left, a trailing surrogate right after a leading one is taken with it as a pair, so a trailing surrogate is reported unpaired exactly when no leading surrogate comes right before it: that is -`Spec.Unpaired`. The statement in `Spec.lean` says it unit by unit. +`Utf.Spec.Unpaired`. The statement in `Utf/Spec.lean` says it unit by unit. -/ diff --git a/bar.toml b/bar.toml index f2e3c79..2d51211 100644 --- a/bar.toml +++ b/bar.toml @@ -13,7 +13,7 @@ spec_authors = ["claude"] entry_points = ["Main.lean"] [spec] -files = ["SPEC.md", "Spec.lean", "Spec/Defs.lean", "Spec/Utf16.lean"] +files = ["SPEC.md", "Utf/Spec.lean", "Utf/Spec/Defs.lean", "Utf/Spec/Utf16.lean"] # The code under proof: the functions the statements are about. code = ["Utf8.encode", "Utf8.decode", "Utf8.decodeLossy", "Utf16.encode", "Utf16.decode", "Utf16.decodeLossy", "Utf16.toWellFormed", diff --git a/comparator.json b/comparator.json index 2667ef1..cee5ba8 100644 --- a/comparator.json +++ b/comparator.json @@ -1,42 +1,42 @@ { - "challenge_module": "Spec", - "solution_module": "Proofs", + "challenge_module": "Utf.Spec", + "solution_module": "Utf.Proofs", "theorem_names": [ - "Spec.encode_eq", - "Spec.decode_iff", - "Spec.decode_encode", - "Spec.decode_wellFormed", - "Spec.encode_toUTF8", - "Spec.fromUTF8?_eq", - "Spec.rfc3629_examples", - "Spec.boundary_vectors", - "Spec.reject_vectors", - "Spec.decodeLossy_iff", - "Spec.decodeLossy_of_decode", - "Spec.decodeLossy_replaces", - "Spec.decodeLossy_encode_append", - "Spec.decodeLossy_append_encode", - "Spec.substitution_vectors", - "Spec.utf16_encode_eq", - "Spec.utf16_decode_iff", - "Spec.utf16_decode_encode", - "Spec.utf16_decode_wellFormed", - "Spec.utf16_decode_none_iff", - "Spec.utf16_encode_length", - "Spec.utf16_decodeLossy_iff", - "Spec.utf16_decodeLossy_unpaired", - "Spec.utf16_decodeLossy_of_decode", - "Spec.utf16_decodeLossy_replaces", - "Spec.utf16_decodeLossy_encode_append", - "Spec.utf16_decodeLossy_append_encode", - "Spec.utf16_toWellFormed", - "Spec.utf16_encodeBytes", - "Spec.utf16be_decode_iff", - "Spec.utf16le_decode_iff", - "Spec.utf16_decodeScheme_iff", - "Spec.utf16_whatwg", - "Spec.utf16_decodeLossyBytes_units", - "Spec.utf16_vectors" + "Utf.Spec.encode_eq", + "Utf.Spec.decode_iff", + "Utf.Spec.decode_encode", + "Utf.Spec.decode_wellFormed", + "Utf.Spec.encode_toUTF8", + "Utf.Spec.fromUTF8?_eq", + "Utf.Spec.rfc3629_examples", + "Utf.Spec.boundary_vectors", + "Utf.Spec.reject_vectors", + "Utf.Spec.decodeLossy_iff", + "Utf.Spec.decodeLossy_of_decode", + "Utf.Spec.decodeLossy_replaces", + "Utf.Spec.decodeLossy_encode_append", + "Utf.Spec.decodeLossy_append_encode", + "Utf.Spec.substitution_vectors", + "Utf.Spec.utf16_encode_eq", + "Utf.Spec.utf16_decode_iff", + "Utf.Spec.utf16_decode_encode", + "Utf.Spec.utf16_decode_wellFormed", + "Utf.Spec.utf16_decode_none_iff", + "Utf.Spec.utf16_encode_length", + "Utf.Spec.utf16_decodeLossy_iff", + "Utf.Spec.utf16_decodeLossy_unpaired", + "Utf.Spec.utf16_decodeLossy_of_decode", + "Utf.Spec.utf16_decodeLossy_replaces", + "Utf.Spec.utf16_decodeLossy_encode_append", + "Utf.Spec.utf16_decodeLossy_append_encode", + "Utf.Spec.utf16_toWellFormed", + "Utf.Spec.utf16_encodeBytes", + "Utf.Spec.utf16be_decode_iff", + "Utf.Spec.utf16le_decode_iff", + "Utf.Spec.utf16_decodeScheme_iff", + "Utf.Spec.utf16_whatwg", + "Utf.Spec.utf16_decodeLossyBytes_units", + "Utf.Spec.utf16_vectors" ], "permitted_axioms": [ "propext", diff --git a/lakefile.toml b/lakefile.toml index e4ae5d0..f10eff8 100644 --- a/lakefile.toml +++ b/lakefile.toml @@ -1,7 +1,7 @@ name = "utf8" version = "0.1.0" license = "Apache-2.0" -defaultTargets = ["Utf8", "Utf16", "Spec", "Proofs", "utf8"] +defaultTargets = ["Utf8", "Utf16", "UtfSpec", "UtfProofs", "utf8"] [[lean_lib]] name = "Utf8" @@ -10,10 +10,12 @@ name = "Utf8" name = "Utf16" [[lean_lib]] -name = "Spec" +name = "UtfSpec" +roots = ["Utf.Spec"] [[lean_lib]] -name = "Proofs" +name = "UtfProofs" +roots = ["Utf.Proofs"] [[lean_exe]] name = "utf8" diff --git a/tests/PLANTED-BUGS.md b/tests/PLANTED-BUGS.md index c4d1c0d..e7098bc 100644 --- a/tests/PLANTED-BUGS.md +++ b/tests/PLANTED-BUGS.md @@ -2,11 +2,12 @@ Written by `python3 tests/planted_bugs.py`. Each row is one small change to the code under proof, made in a copy. Before any change, the copy's tool builds, the -tests report 0 failures, and `lake build Proofs` succeeds. With each change, the -tests ran as `python3 tests/run_tests.py --count 20000` runs them, on 2,415,209 -inputs from their recorded seed, counted as `tests/TEST-RESULTS.md` counts them, -against the changed tool. A bug is caught when they report failures. The last -column says whether `lake build Proofs` still succeeds with the change. +tests report 0 failures, and `lake build UtfProofs` succeeds. With each change, +the tests ran as `python3 tests/run_tests.py --count 20000` runs them, on +2,415,209 inputs from their recorded seed, counted as `tests/TEST-RESULTS.md` +counts them, against the changed tool. A bug is caught when they report +failures. The last column says whether `lake build UtfProofs` still succeeds +with the change. | Bug | File | Failures | Proofs | |-----|------|----------|--------| diff --git a/tests/planted_bugs.py b/tests/planted_bugs.py index 2878ce5..eefcf7f 100755 --- a/tests/planted_bugs.py +++ b/tests/planted_bugs.py @@ -181,7 +181,7 @@ def run(count): sys.exit(f"with no change, the tool does not build\n{tool.stdout[-2000:]}") if test(work, "with no change", count) != 0: sys.exit("with no change, the tests report failures") - proofs = build(work, "Proofs") + proofs = build(work, "UtfProofs") if proofs.returncode != 0: sys.exit(f"with no change, the proofs do not build\n{proofs.stdout[-2000:]}") m = re.search(r"^- Inputs: (\d+)$", (work / "tests" / "TEST-RESULTS.md").read_text(), @@ -202,7 +202,7 @@ def run(count): path.write_text(original) sys.exit(f"{name}: the tool does not build with this change\n{tool.stdout[-2000:]}") n = test(work, name, count) - broke = build(work, "Proofs").returncode != 0 + broke = build(work, "UtfProofs").returncode != 0 path.write_text(original) rows.append((name, rel, old, new, equivalent, n, broke)) print(f"{name}: {n} failures; proofs {'fail to build' if broke else 'build'}", @@ -214,11 +214,11 @@ def write(rows, count, inputs): intro = ( "Written by `python3 tests/planted_bugs.py`. Each row is one small change to the " "code under proof, made in a copy. Before any change, the copy's tool builds, the " - "tests report 0 failures, and `lake build Proofs` succeeds. With each change, the " + "tests report 0 failures, and `lake build UtfProofs` succeeds. With each change, the " f"tests ran as `python3 tests/run_tests.py --count {count}` runs them, on {inputs:,} " "inputs from their recorded seed, counted as `tests/TEST-RESULTS.md` counts them, " "against the changed tool. A bug is caught when they report failures. The last " - "column says whether `lake build Proofs` still succeeds with the change." + "column says whether `lake build UtfProofs` still succeeds with the change." ) lines = ["# Planted bugs", "", *textwrap.wrap(intro, 80), "", "| Bug | File | Failures | Proofs |",