Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Proofs/Contain.lean
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ theorem gate_plain (name : String) (rel : List String) :
is rejected; anything else is skipped. -/
theorem manifest_eq (dest : List String) (entries : List RawEntry) :
manifest dest entries = entries.map fun e =>
if regularType e.typeflag then
if regularType e.typeflag e.name then
match cleanName e.name with
| some rel => .extract (dest ++ rel) e.data
| none => .reject e.name
Expand Down Expand Up @@ -514,7 +514,7 @@ theorem manifest_lines_eq (dest : String) (entries : List RawEntry) :
funext e
simp only [Function.comp_apply]
unfold outcome Spec.line
have hreg : isRegular e = regularType e.typeflag := rfl
have hreg : isRegular e = regularType e.typeflag e.name := rfl
rw [hreg, gate_eq]
split
· split
Expand Down
17 changes: 13 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ lake build

The theorems are in `Proofs/Contain.lean`, about the code in `TarPilot/Path.lean` and `TarPilot/Archive.lean`. Their statements use the definitions in `Spec/Defs.lean`: a special character (a control character, C0, DEL, or C1, or the Unicode line or paragraph separator), a plain name (not empty, `.` or `..`, and free of `/` and special characters), the components of a path string (the parts between `/`), the gate's rules, the regular typeflags, a path under a destination, how a name is printed, the exact line printed for an entry, and lexical resolution of a relative path. They are about the code definitions `Spec/Defs.lean` lists, which a reviewer reads with them.

The gate rules, which `gate_eq` proves the gate follows exactly: a name that starts with `/` is rejected. A name with a `..` component, or with a special character, is rejected. Empty and `.` components are dropped. A name with nothing left is rejected. Only regular files (typeflag `0` or NUL) are extracted, which `manifest_eq` proves. Directory, link, and other entries are skipped, so a link entry cannot place a later entry outside the destination.
The gate rules, which `gate_eq` proves the gate follows exactly: a name that starts with `/` is rejected. A name with a `..` component, or with a special character, is rejected. Empty and `.` components are dropped. A name with nothing left is rejected. Only regular files are extracted, which `manifest_eq` proves: typeflag `0`, NUL from older archives, or `7`, a contiguous file, which POSIX lets a reader treat as regular; and a name that does not end in `/`, which older archives use for a directory and GNU tar reads as one. So an entry with a regular typeflag and a name ending in `/` gets a `SKIP` line when its size is zero. When its size is not zero, the whole archive fails to parse, since readers disagree on whether data follows (see the list under What is not proved). Directory, link, and other entries are skipped, so a link entry cannot place a later entry outside the destination.

- `gate_eq`: the gate is exactly its rules.
- `gate_plain`: a name that passes the gate gives a non-empty list of plain names.
Expand All @@ -43,11 +43,18 @@ Axiom check: each theorem depends only on the standard Lean axioms (`propext`, `

Read this before trusting the pilot with real archives:

- Header parsing, octal fields, checksums, and archive writing are tested, not proved. The tests compare the Lean code against Python's `tarfile` module and against an independent copy of the extraction spec. Results are in `tests/TEST-RESULTS.md`: 37 checks, 14 of them on the axioms, 0 failures. They cover 23 archive entries with normal files, hostile names (control characters and line separators among them), and a long name split over the ustar prefix field; eight destinations, and six refused: five for a special character or U+FFFD, and one given as bytes that are not UTF-8; and broken inputs (truncated, garbage, bad checksum).
- Header parsing, octal fields, checksums, and archive writing are tested, not proved. The tests compare the Lean code against Python's `tarfile` module and against an independent copy of the extraction spec. Results are in `tests/TEST-RESULTS.md`: 147 checks, 14 of them on the axioms, 0 failures. They cover 23 archive entries with normal files, hostile names (control characters and line separators among them), and a long name split over the ustar prefix field; eight destinations, and six refused: five for a special character or U+FFFD, and one given as bytes that are not UTF-8; broken inputs (truncated, garbage, bad checksum); 29 archives in formats the pilot does not read; 10 archives that break the end rule, including concatenated archives; and 16 more archives it does read, among them GNU format, GNU tar's own output in its default, `gnu` and `oldgnu` formats, names ending in `/`, links, devices, directories and FIFOs whose size fields are not zero, an archive of exactly two zero blocks, and extra zero padding. Both commands are compared with `tarfile` on each of them. `tarfile` reads typeflag NUL with a name ending in `/` as a directory, typeflag 5, so the tool's typeflag 0 on such an entry is compared as 5. Every other typeflag is compared as printed.
- The `list` output is tested, not proved. So is `Main.lean`: that it prints `o.line` for each outcome of the manifest, and refuses a destination with a special character or U+FFFD. Read it with the statements; it is a few lines.
- Extraction works on an in-memory model. Nothing here writes files to disk, so the step from a proved in-memory path to a safe write on a real filesystem (races, symlinks already on disk, symlinked destination folders) is outside this project for now. The proof resolves paths lexically, as POSIX does when no component is a symlink.
- The destination itself. The proof says where a path lands relative to the destination, not whether the destination is a safe place to write. The destination is the command-line argument as the Lean runtime decodes it from UTF-8.
- Formats beyond POSIX ustar. GNU tar extensions (long name entries, pax headers), base-256 size fields, and sparse files are not supported. A base-256 size field fails to parse, but an archive with a GNU long name or a pax header is not refused, as the bar audit's probes show (findings F2 and F3). Such a header is skipped as a non-regular entry, so the file after it is read under the name in its own ustar header: a long name is cut to 100 bytes, and a pax path is ignored. A name or prefix field that is not valid UTF-8 reads as empty. So a short name like that is rejected, but a name split over both fields, with one field invalid, is extracted under another name. In every probe the path stays inside the destination, since the gate still runs.
- Formats beyond POSIX ustar. The pilot reads ustar headers, and GNU headers that carry nothing ustar does not. As POSIX says, and as GNU tar and Python's `tarfile` read them, links, devices, directories and FIFOs store no data, whatever their size field says. An archive that uses anything else fails to parse as a whole, instead of being read in a wrong way:
- typeflags outside the POSIX set (NUL and `0` to `7`), among them GNU long names and long links (`L`, `K`), pax headers (`x`, `g`), and GNU sparse, incremental and volume entries. GNU tar writes long names this way by default, and Python's `tarfile` writes pax headers for long or non-ASCII names;
- a GNU header with anything where ustar keeps its name prefix (bytes 345 to 499), such as the access and change times of an incremental dump. GNU tar 1.35 puts nothing there for short names, in its default, `gnu` and `oldgnu` formats, unless it is asked for an incremental dump, a sparse file or a multi-volume archive;
- a name or prefix field that is not valid UTF-8;
- a base-256 size field, or a size field of spaces, which GNU tar refuses;
- a regular typeflag with a name ending in `/`, which reads as a directory, and a size that is not zero, where readers disagree on whether data follows;
- any other magic, such as an old Unix V7 header.
- The end of the archive. Parsing accepts the entries only when the length is a multiple of 512, the archive ends with two zero blocks, and every byte after the first of those blocks is zero. One zero block, a partial block, nonzero padding, and a second archive concatenated after the end are rejected whole, with nothing printed. Extra blocks of zeros are padding and are read. This is tested, not proved.
- The Lean compiler, runtime, and kernel are trusted, as in every Lean project. A bug in them is outside the proof.

## Re-check it
Expand All @@ -60,7 +67,9 @@ python3 tests/run_tests.py

The build fails if a proof breaks, though it accepts a `sorry` with only a warning. The second command prints the axioms each theorem depends on. The test script builds the project first, fails if any theorem depends on an axiom other than the three standard ones, reruns every comparison, and rewrites `tests/TEST-RESULTS.md`.

`python3 tests/compare_old.py` builds the tool as it was before the containment change (commit 7517a86) and compares the two on 3,000 random archives under five destinations, from a recorded seed. Once the new escapes in names are undone, the output must be the same. It writes `tests/COMPARE-OLD.md`.
`python3 tests/compare_old.py --new 1273382` builds the tool as it was before the containment change (commit 7517a86) and the tool of that change (commit 1273382), and compares the two on 3,000 random archives under five destinations, from a recorded seed. Once the new escapes in names are undone, the output must be the same. It writes `tests/COMPARE-OLD.md`. Without `--new`, it compares this checkout's tool, and runs differ on purpose: some random names end in `/` with a regular typeflag, and this tool skips such an entry, or refuses the archive when the entry has data.

`python3 tests/run_tests.py --tool 1273382` runs the same checks against the tool of the containment change (commit 1273382), from before the parser changes. It leaves out the axiom check, which reads this checkout's proofs, not the tool, and writes `tests/TEST-RESULTS-1273382.md`: 86 of the 133 checks fail there. Nine of the ten end-rule archives fail there, because that tool returns at the first zero block. An archive with no zero block already failed there.

`python3 tests/planted_bugs.py` plants, one at a time in a copy, a `sorry`, an axiom, an extra theorem that is not on the list, and a listed theorem left out of `TarPilot/Verify.lean`. It records that `lake build` passes on each and that the test script fails. It writes `tests/PLANTED-BUGS.md`.

Expand Down
11 changes: 7 additions & 4 deletions Spec/Defs.lean
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,12 @@ def cleanName (name : String) : Option (List String) :=
let kept := parts.filter (fun c => !(c == "" || c == "."))
if kept = [] then none else some kept

/-- The typeflags of regular files: `0`, and NUL from older archives. -/
def regularType (t : Char) : Bool :=
t == '0' || t == Char.ofNat 0
/-- Whether an entry with typeflag `t` and name `name` is a regular file:
typeflag `0`, NUL from older archives, or `7`, a contiguous file,
which POSIX lets a reader treat as regular; and a name that does not
end in `/`, which older archives use for a directory. -/
def regularType (t : Char) (name : String) : Bool :=
(t == '0' || t == Char.ofNat 0 || t == '7') && name.toList.getLast? != some '/'

/-- A relative path `rel` under the destination `dest`, as a string: `rel`
alone when `dest` is empty, `dest` then `rel` when `dest` ends in `/`,
Expand Down Expand Up @@ -135,7 +138,7 @@ def byteSum (b : ByteArray) : Nat :=
whose name fails; `SKIP`, the name, and the typeflag as a decimal
code, for anything else. -/
def line (dest : String) (name : String) (t : Char) (data : ByteArray) : String :=
if regularType t then
if regularType t name then
match cleanName name with
| some rel =>
"OK\t" ++ under dest (String.intercalate "/" rel) ++ "\t" ++ toString data.size ++
Expand Down
38 changes: 27 additions & 11 deletions TarPilot/Archive.lean
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,24 @@ structure RawEntry where
/-- Blocks of the archive, read from offset `off`. `fuel` counts down
the blocks that can remain, which keeps the function total on
hostile input: a size field that points past the end, or a block
count that never reaches a full zero block, gives `none`. An
archive that ends without its zero blocks is truncated, and a
truncated archive is rejected whole, not read as a clean prefix. -/
count that never reaches two zero blocks, gives `none`. The
archive must end with two zero blocks, and every byte after the
first of them must be zero. One zero block, nonzero padding, or
another archive concatenated after the end is rejected whole, not
read as a clean prefix. -/
def parseBlocks (data : ByteArray) (off : Nat) : Nat → Option (List RawEntry)
| 0 => none
| fuel + 1 =>
if off + 512 > data.size then none
else
let block := data.extract off (off + 512)
if block.toList.all (· = 0) then some []
if block.toList.all (· = 0) then
-- Two end blocks, then only zero padding. A reader that
-- stopped at the first zero block would miss an entry hidden
-- after it, or one in a second archive concatenated on the end.
if off + 1024 > data.size then none
else if (data.extract off data.size).toList.all (· = 0) then some []
else none
else match parseBlock block with
| none => none
| some h =>
Expand All @@ -47,15 +55,23 @@ def parseBlocks (data : ByteArray) (off : Nat) : Nat → Option (List RawEntry)
some ({ name := h.name, typeflag := h.typeflag,
data := data.extract dataOff (dataOff + h.size) } :: rest)

/-- Parse a whole archive. -/
/-- Parse a whole archive. A length that is not a multiple of 512 is
rejected, as is an archive that does not end in two zero blocks
with only zero bytes after the first of them. -/
def parseArchive (data : ByteArray) : Option (List RawEntry) :=
parseBlocks data 0 (data.size / 512 + 1)

/-- Regular files only: typeflag `0` or NUL. Directories, links, and
everything else are never extracted, so a link entry cannot be used
to place a later entry outside the destination. -/
if data.size % 512 != 0 then none
else parseBlocks data 0 (data.size / 512 + 1)

/-- Regular files only: typeflag `0`, NUL, or `7` (a contiguous file,
which POSIX lets a reader treat as regular, as Python's `tarfile`
does), and a name that does not end in `/`. Older archives mark a
directory with that `/`, and GNU tar reads any entry named so as a
directory. Directories, links, and everything else are never
extracted, so a link entry cannot be used to place a later entry
outside the destination. -/
def isRegular (e : RawEntry) : Bool :=
e.typeflag == '0' || e.typeflag == Char.ofNat 0
(e.typeflag == '0' || e.typeflag == Char.ofNat 0 || e.typeflag == '7') &&
e.name.toList.getLast? != some '/'

/-- Extract entries into a list of `(path, data)` pairs, in memory.
The path of each pair is the destination components followed by the
Expand Down
Loading