Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions Proofs/Main.lean
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,10 @@ theorem Spec.substitution_vectors :
-- Table 3-11, truncated sequences
Utf8.decodeLossy [0xE1, 0x80, 0xE2, 0xF0, 0x91, 0x92, 0xF1, 0xBF, 0x41] =
['�', '�', '�', '�', 'A'] ∧
-- A cut-short sequence, then `<` and `A`
-- Three cut-short sequences, then `<`, `A`, or `<` again
Utf8.decodeLossy [0xF0, 0x9F, 0x98, 0x3C] = ['�', '<'] ∧
Utf8.decodeLossy [0xE2, 0x41, 0x42] = ['�', 'A', 'B'] := by
Utf8.decodeLossy [0xE2, 0x41, 0x42] = ['�', 'A', 'B'] ∧
Utf8.decodeLossy [0xE2, 0x82, 0x3C] = ['�', '<'] := by
decide

theorem Spec.utf16_encode_eq (cs : List Char) : Utf16.encode cs = Spec.encode16 cs :=
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The statements are in `Spec.lean`, the definitions they use in `Spec/Defs.lean`
| `decodeLossy` gives exactly what section 3.9.6 of the Unicode Standard, *U+FFFD Substitution of Maximal Subparts*, sets out: each well-formed sequence becomes its character, and each maximal subpart of bad bytes becomes one U+FFFD. | `Spec.decodeLossy_iff` |
| On well-formed bytes, `decodeLossy` reads what `decode` reads; on any other bytes it gives at least one U+FFFD. | `Spec.decodeLossy_of_decode`, `Spec.decodeLossy_replaces` |
| Well-formed text before or after bad bytes comes through unchanged: no U+FFFD takes in any of it. | `Spec.decodeLossy_encode_append`, `Spec.decodeLossy_append_encode` |
| The examples of Tables 3-8 to 3-11 of the Unicode Standard, and two cut-short sequences followed by a character. | `Spec.substitution_vectors` |
| The examples of Tables 3-8 to 3-11 of the Unicode Standard, and three cut-short sequences followed by a character. | `Spec.substitution_vectors` |
| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Spec.utf16_encode_eq` |
| `Utf16.decode` returns a list of characters exactly when the code units are its encoding. So every unit list it accepts has one reading, and it refuses everything else. | `Spec.utf16_decode_iff` |
| Decoding what the UTF-16 encoder wrote gives the characters back. | `Spec.utf16_decode_encode` |
Expand Down
2 changes: 1 addition & 1 deletion SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ Each property is a theorem in `Spec.lean`, under the name given. The definitions
| On well-formed bytes, `decodeLossy` reads what `decode` reads. | `Spec.decodeLossy_of_decode` |
| On bytes that are not well formed, `decodeLossy` gives at least one U+FFFD. | `Spec.decodeLossy_replaces` |
| Well-formed text before or after bad bytes comes through unchanged. | `Spec.decodeLossy_encode_append`, `Spec.decodeLossy_append_encode` |
| The examples of section 3.9.6, Tables 3-8 to 3-11, and two cut-short sequences followed by a character. | `Spec.substitution_vectors` |
| The examples of section 3.9.6, Tables 3-8 to 3-11, and three cut-short sequences followed by a character. | `Spec.substitution_vectors` |
| The UTF-16 encoder writes each character as RFC 2781, section 2.1, sets out. | `Spec.utf16_encode_eq` |
| `Utf16.decode` returns `cs` exactly when the units are the encoding of `cs`. | `Spec.utf16_decode_iff` |
| Decoding what the UTF-16 encoder wrote gives the characters back. | `Spec.utf16_decode_encode` |
Expand Down
8 changes: 4 additions & 4 deletions Spec.lean
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,7 @@ theorem Spec.decodeLossy_append_encode (bs : List UInt8) (cs : List Char) :
Utf8.decodeLossy (bs ++ Utf8.encode cs) = Utf8.decodeLossy bs ++ cs := sorry

/-- The examples of section 3.9.6 of the Unicode Standard, Tables 3-8
to 3-11, and two cases where replacing too many bytes would swallow
the next character. -/
to 3-11, and three cut-short sequences followed by a character. -/
theorem Spec.substitution_vectors :
-- Table 3-8, non-shortest forms
Utf8.decodeLossy [0xC0, 0xAF, 0xE0, 0x80, 0xBF, 0xF0, 0x81, 0x82, 0x41] =
Expand All @@ -177,9 +176,10 @@ theorem Spec.substitution_vectors :
-- Table 3-11, truncated sequences
Utf8.decodeLossy [0xE1, 0x80, 0xE2, 0xF0, 0x91, 0x92, 0xF1, 0xBF, 0x41] =
['�', '�', '�', '�', 'A'] ∧
-- A cut-short sequence, then `<` and `A`
-- Three cut-short sequences, then `<`, `A`, or `<` again
Utf8.decodeLossy [0xF0, 0x9F, 0x98, 0x3C] = ['�', '<'] ∧
Utf8.decodeLossy [0xE2, 0x41, 0x42] = ['�', 'A', 'B'] := sorry
Utf8.decodeLossy [0xE2, 0x41, 0x42] = ['�', 'A', 'B'] ∧
Utf8.decodeLossy [0xE2, 0x82, 0x3C] = ['�', '<'] := sorry

/-! ## UTF-16 code units -/

Expand Down
260 changes: 260 additions & 0 deletions tests/PLANTED-BUGS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
# Planted bugs

Written by `python3 tests/planted_bugs.py`. Each row is one small change to the
code under proof, made in a copy. Before any change, the copy's tool builds, the
tests report 0 failures, and `lake build Proofs` succeeds. With each change, the
tests ran as `python3 tests/run_tests.py --count 20000` runs them, on 2,415,209
inputs from their recorded seed, counted as `tests/TEST-RESULTS.md` counts them,
against the changed tool. A bug is caught when they report failures. The last
column says whether `lake build Proofs` still succeeds with the change.

| Bug | File | Failures | Proofs |
|-----|------|----------|--------|
| Three-byte overlong forms read | `Utf8/Codec.lean` | 4833 | fail to build |
| Surrogates read | `Utf8/Codec.lean` | 5902 | fail to build |
| Values past U+10FFFF read | `Utf8/Codec.lean` | 920 | fail to build |
| Leads 0xC0 and 0xC1 read | `Utf8/Codec.lean` | 37877 | fail to build |
| 0xC0 read as a continuation byte | `Utf8/Codec.lean` | 19175 | fail to build |
| U+0800 written in two bytes | `Utf8/Codec.lean` | 250 | fail to build |
| Leads 0xF5 to 0xF7 start four-byte sequences | `Utf8/Codec.lean` | 0, equivalent: with a lead of 0xF5 to 0xF7 the value is at least 5 * 0x40000 = 0x140000, past 0x10FFFF, so the four-byte branch refuses it. `decodeChar` gives `none` on the same inputs as before, and the encoder is unchanged. | fail to build |
| One U+FFFD for each bad byte | `Utf8/Lossy.lean` | 261672 | fail to build |
| Second byte after 0xE0 not narrowed | `Utf8/Lossy.lean` | 8854 | fail to build |
| Second byte after 0xED not narrowed | `Utf8/Lossy.lean` | 9526 | fail to build |
| Second byte after 0xF0 not narrowed | `Utf8/Lossy.lean` | 4731 | fail to build |
| Second byte after 0xF4 not narrowed | `Utf8/Lossy.lean` | 13595 | fail to build |
| Third byte of a cut-short four-byte sequence not counted | `Utf8/Lossy.lean` | 18788 | fail to build |
| Leads 0xF5 to 0xF7 take continuation bytes | `Utf8/Lossy.lean` | 54410 | fail to build |
| Leads 0xC2 to 0xDF and 0xF5 to 0xFF swallow the next byte | `Utf8/Lossy.lean` | 696466 | fail to build |
| A bad last byte given its own branch | `Utf8/Lossy.lean` | 0, equivalent: when `rest` is empty, `badLength b0 []` is 1 for every `b0`, so the old code also gives U+FFFD and drops the one byte, leaving nothing. `lossyStep` gives the same pair on every input. The change was meant to drop a lone lead byte at the end, but as written it gives the same answer. | fail to build |
| High surrogate range misses 0xDBFF | `Utf16/Codec.lean` | 3943 | fail to build |
| Low surrogate range misses 0xDC00 | `Utf16/Codec.lean` | 8679 | fail to build |
| U+10000 written as one unit | `Utf16/Codec.lean` | 969 | fail to build |
| Pair arithmetic off by one | `Utf16/Codec.lean` | 30786 | fail to build |
| Lone low surrogate accepted | `Utf16/Codec.lean` | 11129 | fail to build |
| High surrogate paired with any unit | `Utf16/Codec.lean` | 3569 | fail to build |
| Lossy decoding swallows the unit after a high surrogate | `Utf16/Codec.lean` | 7677 | fail to build |
| Lossy decoding drops a final high surrogate | `Utf16/Codec.lean` | 10627 | fail to build |
| Two U+FFFD for a high surrogate and an odd byte | `Utf16/Bytes.lean` | 764 | fail to build |
| Big-endian mark kept as text | `Utf16/Bytes.lean` | 998 | fail to build |
| Text with no mark read little-endian | `Utf16/Bytes.lean` | 73811 | fail to build |
| UTF-16LE written big-endian | `Utf16/Bytes.lean` | 2642 | fail to build |
| Odd byte ignored by the strict byte decoders | `Utf16/Bytes.lean` | 8192 | fail to build |
| Little-endian mark read big-endian | `Utf16/Bytes.lean` | 1441 | fail to build |

## The changes

**Three-byte overlong forms read** in `Utf8/Codec.lean`:

```
- if n < 0x800 ∨ (0xD800 ≤ n ∧ n ≤ 0xDFFF) then none
+ if (0xD800 ≤ n ∧ n ≤ 0xDFFF) then none
```

**Surrogates read** in `Utf8/Codec.lean`:

```
- if n < 0x800 ∨ (0xD800 ≤ n ∧ n ≤ 0xDFFF) then none
+ if n < 0x800 then none
```

**Values past U+10FFFF read** in `Utf8/Codec.lean`:

```
- if n < 0x10000 ∨ 0x10FFFF < n then none
+ if n < 0x10000 then none
```

**Leads 0xC0 and 0xC1 read** in `Utf8/Codec.lean`:

```
- else if x < 0xC2 then none
+ else if x < 0xC0 then none
```

**0xC0 read as a continuation byte** in `Utf8/Codec.lean`:

```
- if 0x80 ≤ b.toNat ∧ b.toNat ≤ 0xBF then
+ if 0x80 ≤ b.toNat ∧ b.toNat ≤ 0xC0 then
```

**U+0800 written in two bytes** in `Utf8/Codec.lean`:

```
- else if n < 0x800 then
- [(0xC0
+ else if n ≤ 0x800 then
+ [(0xC0
```

**Leads 0xF5 to 0xF7 start four-byte sequences** in `Utf8/Codec.lean`:

```
- else if x < 0xF5 then
+ else if x < 0xF8 then
```

**One U+FFFD for each bad byte** in `Utf8/Lossy.lean`:

```
- (b0 :: rest).drop (badLength b0 rest)
+ (b0 :: rest).drop 1
```

**Second byte after 0xE0 not narrowed** in `Utf8/Lossy.lean`:

```
- if x = 0xE0 then 0xA0 ≤ y
+ if x = 0xE0 then 0x80 ≤ y
```

**Second byte after 0xED not narrowed** in `Utf8/Lossy.lean`:

```
- else if x = 0xED then 0x80 ≤ y && y ≤ 0x9F
+ else if x = 0xED then 0x80 ≤ y && y ≤ 0xBF
```

**Second byte after 0xF0 not narrowed** in `Utf8/Lossy.lean`:

```
- else if x = 0xF0 then 0x90 ≤ y
+ else if x = 0xF0 then 0x80 ≤ y
```

**Second byte after 0xF4 not narrowed** in `Utf8/Lossy.lean`:

```
- else if x = 0xF4 then 0x80 ≤ y && y ≤ 0x8F
+ else if x = 0xF4 then 0x80 ≤ y && y ≤ 0xBF
```

**Third byte of a cut-short four-byte sequence not counted** in `Utf8/Lossy.lean`:

```
- then 3 else 2
+ then 2 else 2
```

**Leads 0xF5 to 0xF7 take continuation bytes** in `Utf8/Lossy.lean`:

```
- if 0xE0 ≤ x ∧ x ≤ 0xF4 then
+ if 0xE0 ≤ x ∧ x ≤ 0xF7 then
```

**Leads 0xC2 to 0xDF and 0xF5 to 0xFF swallow the next byte** in `Utf8/Lossy.lean`:

```
- else 1
-
- /-- One step
+ else if 0xC2 ≤ x ∧ rest ≠ [] then 2 else 1
+
+ /-- One step
```

**A bad last byte given its own branch** in `Utf8/Lossy.lean`:

```
- | none => (replacement, (b0 :: rest).drop (badLength b0 rest))
+ | none => (if rest = [] then (replacement, []) else (replacement, (b0 :: rest).drop (badLength b0 rest)))
```

**High surrogate range misses 0xDBFF** in `Utf16/Codec.lean`:

```
- u.toNat ≤ 0xDBFF
+ u.toNat < 0xDBFF
```

**Low surrogate range misses 0xDC00** in `Utf16/Codec.lean`:

```
- 0xDC00 ≤ u.toNat
+ 0xDC01 ≤ u.toNat
```

**U+10000 written as one unit** in `Utf16/Codec.lean`:

```
- if n < 0x10000 then [n.toUInt16]
+ if n ≤ 0x10000 then [n.toUInt16]
```

**Pair arithmetic off by one** in `Utf16/Codec.lean`:

```
- Char.ofNat (0x10000 + (hi.toNat
+ Char.ofNat (0xFFFF + (hi.toNat
```

**Lone low surrogate accepted** in `Utf16/Codec.lean`:

```
- else if isLow u then none
+ else if false then none
```

**High surrogate paired with any unit** in `Utf16/Codec.lean`:

```
- if isLow v then (decode rest').map (pair u v :: ·) else none
+ (decode rest').map (pair u v :: ·)
```

**Lossy decoding swallows the unit after a high surrogate** in `Utf16/Codec.lean`:

```
- else replacement :: decodeLossy (v :: rest')
+ else replacement :: decodeLossy rest'
```

**Lossy decoding drops a final high surrogate** in `Utf16/Codec.lean`:

```
- | [] => [replacement]
+ | [] => []
```

**Two U+FFFD for a high surrogate and an odd byte** in `Utf16/Bytes.lean`:

```
- if us.getLast?.any isHigh then decodeLossy us else decodeLossy us ++ [replacement]
+ decodeLossy us ++ [replacement]
```

**Big-endian mark kept as text** in `Utf16/Bytes.lean`:

```
- if a = 0xFE ∧ b = 0xFF then decodeBE rest
+ if a = 0xFE ∧ b = 0xFF then decodeBE (a :: b :: rest)
```

**Text with no mark read little-endian** in `Utf16/Bytes.lean`:

```
- else decodeBE (a :: b :: rest)
+ else decodeLE (a :: b :: rest)
```

**UTF-16LE written big-endian** in `Utf16/Bytes.lean`:

```
- [(u.toNat % 0x100).toUInt8, (u.toNat / 0x100).toUInt8]
+ [(u.toNat / 0x100).toUInt8, (u.toNat % 0x100).toUInt8]
```

**Odd byte ignored by the strict byte decoders** in `Utf16/Bytes.lean`:

```
- | (_, some _) => none
+ | (us, some _) => decode us
```

**Little-endian mark read big-endian** in `Utf16/Bytes.lean`:

```
- else if a = 0xFF ∧ b = 0xFE then decodeLE rest
+ else if a = 0xFF ∧ b = 0xFE then decodeBE rest
```
Loading