diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 537e0a03..4c892fe7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,9 +2,9 @@ ## What a fresh clone can run, and what it needs first -A clone with nothing else installed but the SDK `global.json` asks for builds, tests and publishes. -The conformance corpus is committed — 856 files and their sidecars — so `dotnet test` runs the -whole thing on a bare clone with no toolchain beyond .NET. +A clone with nothing installed but the SDK that `global.json` asks for can build, test and publish. +The conformance corpus is committed (876 files and their sidecars), so `dotnet test` runs everything +on a bare clone with no toolchain beyond .NET. ``` dotnet build Vorticity.slnx -c Release @@ -12,32 +12,32 @@ dotnet test Vorticity.slnx -c Release dotnet publish tools/vxdump -c Release -r # Native AOT, ~5 MB, no trim warning ``` -Three things are not in the repository, all deliberately, and each one refuses with the command -that produces it rather than with a stack trace. +Three things are deliberately not in the repository, and each one, when missing, fails with the +command that produces it rather than with a stack trace. -**The Rust shim, for anything that compares against the reference.** `--ffi-check` and -`--ratio-check` load a cdylib built from `tools/vxbench-rs`, which means **`bench/gate.sh` is red -on a fresh clone** until you build it. So is `--throughput --check` in any of its forms. +The first is the Rust shim, needed for anything that compares against the reference. `--ffi-check` and +`--ratio-check` load a cdylib built from `tools/vxbench-rs`, which means `bench/gate.sh` is red on a +fresh clone until you build it, and so is `--throughput --check` in any of its forms. ``` cd tools/vxbench-rs && cargo build --release ``` -`bench/crosscheck.sh` needs `cargo` too, and builds a different crate: it has the .NET side write -the corpus out and has Vortex Rust read every file back. +`bench/crosscheck.sh` needs `cargo` too, and builds a different crate: it has the .NET side write the +corpus out and has Vortex Rust read every file back. -**The throughput inputs**, 468 MB for 57 encodings, generated rather than committed — a git history -is not the place for half a gigabyte that a script reproduces. +The second is the throughput inputs, about half a gigabyte for 61 files, generated rather than +committed, since a git history is not the place for data a script reproduces. ``` -bench/gen-throughput.sh # once; or point VORTICITY_THROUGHPUT_CORPUS +bench/gen-throughput.sh # once, or point VORTICITY_THROUGHPUT_CORPUS # at a directory that already holds them ``` -**Vorticity.Zstd's test data**: zstd's golden files and a corpus of frames from zstd's decodecorpus, -binary files written from zstd 1.5.7's sources rather than committed. The tests that read them are -skipped without them, and fail in CI, whose workflow writes them first. -The script needs `curl`, `tar` and a C compiler. +The third is Vorticity.Zstd's test data: zstd's golden files and a corpus of frames from zstd's +decodecorpus, binary files written from zstd 1.5.7's sources rather than committed. The tests that +read them are skipped without them, and fail in CI, whose workflow writes them first. The script needs +`curl`, `tar` and a C compiler. ``` tools/native-ref/testdata.sh # once @@ -45,16 +45,16 @@ tools/native-ref/testdata.sh # once ## What to run before a change is done -The chain runs when a change is finished, not at every commit. Between two, the suite is enough. -Pick the row that matches what moved; doing more is always allowed, doing less is not. +The chain runs when a change is finished, not at every commit. Between two, the suite is enough. Pick +the row that matches what moved. Doing more is always allowed, doing less is not. | what moved | run | |---|---| -| a decoder, a canonical form, the scan, the filter | the suite · `tests/scalar-pass.sh` · `bench/gate.sh` · `--throughput --check` | -| the compressor, a plan, the blob, the writer, with no intent to change a byte | the same · `bench/crosscheck.sh` · `--throughput --write --check` | +| a decoder, a canonical form, the scan, the filter | the suite, `tests/scalar-pass.sh`, `bench/gate.sh`, `--throughput --check` | +| the compressor, a plan, the blob, the writer, with no intent to change a byte | the same, plus `bench/crosscheck.sh` and `--throughput --write --check` | | bytes on disk, deliberately | the same, plus the corpus byte count before and after in the commit message | -| indexes, keys, the dataset | the suite · `tests/scalar-pass.sh` · `bench/gate.sh` | -| a public signature, a visibility, an XML summary | the suite · a Release build with no warning · the public surface record updated in the same commit | +| indexes, keys, the dataset | the suite, `tests/scalar-pass.sh`, `bench/gate.sh` | +| a public signature, a visibility, an XML summary | the suite, a Release build with no warning, and the public surface record updated in the same commit | | the bench or the tools | `bench/gate.sh`, and the command you touched, once | | documents and comments only | a Release build, so the `` links still resolve | @@ -71,81 +71,73 @@ dotnet run -c Release --project bench/Vorticity.Benchmarks -- --throughput --che ``` `bench/gate.sh` is the short answer to "is this safe to push": the ten deterministic ratchets, a -row-count agreement with the reference through FFI, and every ratio against it. It is a command -with an exit code, so a hook can run it: +row-count agreement with the reference through FFI, and every ratio against it. It is a command with +an exit code, so a hook can run it: ``` ln -s ../../bench/gate.sh .git/hooks/pre-push ``` -The two intrinsic modes apply to the suite only. Without intrinsics our side loses the SIMD the +The two intrinsic modes only apply to the suite. Without intrinsics our side loses the SIMD the reference keeps, so the ratio gates are red by construction and mean nothing there. -`bench/crosscheck.sh` is the only oracle that catches a fault our reader and our writer share: it -writes the corpus with this library and has Vortex Rust read it back, scalar by scalar. Needs +`bench/crosscheck.sh` is the only oracle that catches a fault our reader and our writer share. It +writes the corpus with this library and has Vortex Rust read it back, scalar by scalar. It needs `cargo`. ## Tests run side by side -Every test runs beside every other, and each row of a theory is a test of its own: -`tests/xunit.runner.json` sets xunit's `parallelMode` to `all` and pre-enumerates theories, for -the three test projects. A test therefore owns what it writes, a temporary file of its own or a -file of `SharedFiles`, which a run writes once and its tests only read, and changes nothing another -test reads. +Every test runs alongside every other one, and each row of a theory is a test of its own: +`tests/xunit.runner.json` sets xunit's `parallelMode` to `all` and pre-enumerates theories, for the +three test projects. A test therefore owns what it writes (a temporary file of its own, or a file of +`SharedFiles`, which a run writes once and its tests only read) and changes nothing another test +reads. -A test that measures the process rather than its own work, such as its allocations, the blocks -finalized anywhere, its live memory or a JIT that has settled, joins one of the collections that -run alone, one test at a time: `AllocationCollection`, `FinalizerCollection`, `AdviceCollection`. +A test that measures the process rather than its own work (its allocations, the blocks finalized +anywhere, its live memory, a JIT that has settled, the memory pressure on the query budget) joins one +of the collections that run alone, one test at a time: `AllocationCollection`, `FinalizerCollection`, +`AdviceCollection` and `CorePressureCollection`. ## The ratchet rule -Ten test classes hold a number rather than a behaviour — `PathAllocationTests`, -`ScanAllocationTests`, `WriteAllocationTests`, `WrittenSizeTests`, `FlatLayoutDecodeCountTests`, -`RoundTripCountTests`, `LiveMemoryTests`, `CorpusCoverageTests`, `DatasetBudgetTests`, -`WorkCounterTests` — and the benchmark harness holds a reference ratio per axis. They exist to fail -on a regression nobody was watching for. The last counts the work of a group by at one lane, the -rows, groups, arrays and bytes its tables took, which a clock on a shared machine cannot resolve: a -change that moves one of its lines writes the line again in the same commit, with the reason. +Ten test classes hold a number rather than a behaviour (`PathAllocationTests`, `ScanAllocationTests`, +`WriteAllocationTests`, `WrittenSizeTests`, `FlatLayoutDecodeCountTests`, `RoundTripCountTests`, +`LiveMemoryTests`, `CorpusCoverageTests`, `DatasetBudgetTests`, `WorkCounterTests`), and the +benchmark harness holds a reference ratio per axis. They exist to fail on a regression nobody was +watching for. `WorkCounterTests` counts the work of a group by at one lane (the rows, groups, arrays +and bytes its tables took), which a clock on a shared machine cannot resolve, so a change that moves +one of its lines writes the line again in the same commit, with the reason. -**A ratchet comes down, and does not go up to make something pass.** That is the whole rule, and it -is the one that decides whether any of these numbers is still worth having: a reference that stops -following the code stops guarding it, and a reference raised to accommodate a change guards -nothing at all. +A ratchet comes down, and never goes up to make something pass. That is the whole rule, and it +decides whether any of these numbers is still worth having: a reference that stops following the +code stops guarding it, and a reference raised to accommodate a change guards nothing at all. -Three things follow. +Three things follow: -* **Lower one whenever the measurement allows it**, in the commit that earned the gain. A ratchet +* Lower a ratchet whenever the measurement allows it, in the commit that earned the gain. A ratchet left far above its axis is slack, not safety. -* **A red ratio is replayed before it is believed.** Several axes are noisier between processes - than the 15 % margin, and the harness names them when it prints one. Re-run in a second process; - if it is still red, it is real. -* **Raising one is a deliberate, dated, justified edit**, with the measurement written at the site - and the reason a reader can check. `--recalibrate` refuses to loosen on its own, by design. +* Replay a red ratio before believing it. Several axes are noisier between processes than the 15 % + margin, and the harness names them when it prints one. Run it again in a second process, and if it + is still red, it is real. +* Raising a ratchet is a deliberate, dated and justified edit, with the measurement written at the + site and a reason a reader can check. `--recalibrate` refuses to loosen on its own, by design. One ceiling is not portable, and it is worth knowing before you go looking for a regression. `PathAllocationTests` measures a whole read path from a corpus file's absolute path, so the figure -carries that path's string — two bytes per character. The ceilings were set where this repository -lives, and a clone thirty characters deeper is red by a couple of hundred bytes on every axis at -once, in lockstep. Several axes moving together by the same amount is the signature: compare the -two clone paths before believing the code moved. +includes that path's string, two bytes per character. The ceilings were set where this repository +lives, so a clone thirty characters deeper is red by a couple of hundred bytes on every axis at once, +in lockstep. Several axes moving together by the same amount is the signature, so compare the two +clone paths before believing the code moved. The same applies to analyzer warnings. There is no `#pragma` in this repository and no rule turned -off in `.editorconfig`: an analyzer worth silencing is usually right about the code. +off in `.editorconfig`, because an analyzer worth silencing is usually right about the code. ## References in comments -Comments cite things like `BENCH-AUDIT.md B19`, `PERF-AUDIT-v2.md Z1b`, `WRITE-AUDIT.md W-36`, -`IMPL-PLAN.md §1.34` or `PERF-GAPS.md W3.1`. There are 215 of them. - -**These are not links, and the files are not in this repository.** They are the maintainers' -engineering journals — measurement sessions, profiles, the argument behind a decision — written in -French and kept out of the published tree. A reference of that shape names where the evidence -behind a decision is recorded. Read the comment around it: it is written to stand on its own, and -the reference is there so that whoever has the journals can find the measurement, not so that you -can follow a link. - -Do not add new ones. A comment that needs a journal to be understood is a comment that has not been -written yet. +A comment stands on its own. It does not cite documents that are not in this repository (private +notes, plans, audits), finding ids or commit hashes, because a reader cannot open them. If a comment +needs such a document to be understood, it has not been written yet. Say what the measurement or the +constraint is, in the comment itself. ## Commit messages @@ -159,29 +151,28 @@ English, Conventional Commits, and the body carries the argument. BREAKING CHANGE: ``` -**Type** is one of `feat`, `fix`, `perf`, `refactor`, `test`, `docs`, `build`, `ci`, `chore`, -`bench`. **Scope** is optional and names the part that moved: `reader`, `writer`, `scan`, `filter`, -`index`, `dataset`, `rowenc`, `bench`, `tools`, `docs`. A change that breaks a caller takes a `!` -after the scope and a `BREAKING CHANGE:` trailer saying what they have to do. +The type is one of `feat`, `fix`, `perf`, `refactor`, `test`, `docs`, `build`, `ci`, `chore` and +`bench`. The scope is optional and names the part that moved: `reader`, `writer`, `scan`, `filter`, +`index`, `dataset`, `rowenc`, `bench`, `tools`, `docs`. A change that breaks a caller takes a `!` after +the scope and a `BREAKING CHANGE:` trailer saying what they have to do. -**The summary is an instruction, not a report** — `fix the budget`, never `fixed the budget`. Aim -for 72 characters and do not pass 80; a summary that needs more is usually two commits. No full -stop: it is a title. +The summary is an instruction, not a report: `fix the budget`, never `fixed the budget`. Aim for 72 +characters and do not pass 80, since a summary that needs more is usually two commits. Leave out the +full stop, since it is a title. -**The body says why, and the numbers belong in it.** Before and after, measured, with the unit: -that is what makes a commit worth reading a year later. What the diff already shows does not need -saying twice. +The body says why, and the numbers belong in it: before and after, measured, with the unit. That is +what makes a commit worth reading a year later. What the diff already shows does not need saying +twice. -**Every sentence stands on its own.** No document name, no section mark, no numbered step, no -finding id, no commit hash. Those point at things a reader of this repository cannot open — the -journals are not published, and a hash does not survive a rewrite. The reasoning in -[References in comments](#references-in-comments) is the same reasoning: if a message needs a -journal to be understood, it has not been written yet. +Every sentence stands on its own, with no document name, no section mark, no numbered step, no +finding id and no commit hash. Those point at things a reader of this repository cannot open, and a +hash does not survive a rewrite. It is the same reasoning as for [references in +comments](#references-in-comments). ## Style -* No file headers, and no `Copyright` lines: `LICENSE` and `NOTICE` carry that. -* No capitals for emphasis, and no history in a comment — say what the code does now and why it is +* No file headers, and no `Copyright` lines, since `LICENSE` and `NOTICE` carry that. +* No capitals for emphasis, and no history in a comment. Say what the code does now and why it is this way, not what it used to be. -* A comment earns its place by saying something the code cannot: a measurement, a constraint from - the format, a road not taken and the reason. +* A comment earns its place by saying something the code cannot: a measurement, a constraint from the + format, a road not taken and the reason. diff --git a/README.md b/README.md index 516b49d2..9eacd19d 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,17 @@ # Vorticity -Vorticity is a **high-performance**, pure .NET implementation of the [Vortex](https://vortex.dev) -columnar file format, with no dependencies: it reads and writes Vortex files faster than Vortex's -own Rust implementation, and its group bys run faster than DuckDB's. +Vorticity is a pure .NET implementation of the [Vortex](https://vortex.dev) columnar file format, +with no third-party dependency. It reads and writes Vortex files faster than Vortex's own Rust +implementation, and its group bys run faster than DuckDB's. ## Performance -Each speedup below is the other side's time divided by Vorticity's, so anything above 1.00x means -Vorticity is faster. Both sides run as separate processes on the same machine, an Apple M4 Pro with -Native AOT, and do the same work. +Each figure below is a speedup: the other side's time divided by Vorticity's, so anything above +1.00x means Vorticity is faster. Both sides run as separate processes on the same machine, an Apple +M4 Pro, are compiled ahead of time, and do the same work. -Against Vortex's own Rust implementation, Vorticity is faster in every scenario, and often twice as -fast. On a table of four columns and a million rows: +Against Vortex's Rust implementation, Vorticity wins every scenario, often by a factor of two. On a +table of four columns and a million rows: | scenario | speedup, one core | speedup, 14 cores | |---|---:|---:| @@ -22,45 +22,45 @@ fast. On a table of four columns and a million rows: | take 1,000 scattered rows | 1.64x | 1.11x | | write the table back out | 2.37x | 1.35x | -Against DuckDB, its group bys win all 13 queries we measured, on the same files and with the same -number of threads: +Against DuckDB, Vorticity's group bys win all 13 queries we measured, on the same files and with the +same number of threads: | DuckDB reading | speedup, one thread | speedup, 14 threads | |---|---:|---:| | its own table, loaded beforehand | 1.43x to 6.15x | 1.18x to 3.50x | | the file, through its Vortex extension | 1.58x to 5.65x | 1.39x to 4.47x | -You can read [how it is measured](#how-performance-is-measured) below, and find every figure on the -pages [against Rust](docs/guide/benchmarks.md), [against DuckDB](docs/guide/benchmarks-duckdb.md) -and [on x64](docs/guide/benchmarks-x64.md). +The way these numbers are produced is described [below](#how-performance-is-measured), and every +figure is on the pages [against Rust](docs/guide/benchmarks.md), +[against DuckDB](docs/guide/benchmarks-duckdb.md) and [on x64](docs/guide/benchmarks-x64.md). ## Features -* **A typed surface.** A `[VortexRecord]` declares the columns; filters and aggregates are C# - lambdas the scan pushes down to the file, and what cannot be pushed down does not compile. -* Ultra-optimized core: near-zero allocations per batch on hot paths, SIMD (`System.Runtime.Intrinsics`), - no copy before decoding: a mapped file is decoded where it lies, and an uncompressed column is a - view of its bytes. -* **Async-only** public API: every read and every write is awaited (`ValueTask`, `await foreach`), - whatever the source; there is no synchronous path to choose. -* Conformant to the published spec (`VTXF` v1, `core2026.08.3` edition). -* Validated by **cross-testing** against the Rust reference implementation. - -No third-party package: the base class library, plus `System.IO.Hashing`, first party, for the -hashes of the write path and the Bloom filters, and `Vorticity.Zstd`, this repository's managed -Zstandard, for `vortex.zstd`. - -**What parity means here.** Files written by this library are read by Vortex Rust, and every value -in them reads back equal — that is the parity claimed and the cross-check is what proves it. It is -not byte parity: the same data is encoded differently on the two sides by design, and a file from -one is not expected to match the other byte for byte. - -**Status.** This project is developed with the help of LLMs. Its API may change in breaking ways +* Columns are declared by a `[VortexRecord]` type, and filters and aggregates are plain C# lambdas + that the scan pushes down into the file. A filter that cannot be pushed down does not compile. +* The core is built for throughput. It allocates next to nothing per batch on the hot paths, uses + SIMD through `System.Runtime.Intrinsics`, and never copies data before decoding it: a mapped file + is decoded where it lies, and an uncompressed column is a view of its bytes. +* The API is async only. Every read and every write is awaited (`ValueTask`, `await foreach`), + whatever the source, so there is no synchronous path to pick by mistake. +* Files follow the published spec, `VTXF` version 1 up to the `core2026.08.3` edition, and every + file Vorticity writes is read back by the Rust reference in CI. + +Beyond the base class library, the core references only `System.IO.Hashing`, a first-party package +used for the write path's hashes and for Bloom filters, and `Vorticity.Zstd`, this repository's +managed Zstandard, which handles `vortex.zstd`. + +Parity with Rust means that Rust reads every file Vorticity writes and every value reads back equal. +It does not mean byte parity. The two writers choose their encodings differently by design, so the +same rows produce different bytes on each side. + +The project is developed with the help of LLMs, and its API may still change in breaking ways before v1.0. ## A first look -A record is a schema, not a row: its members are the file's columns, in order, under their names. +A record describes the columns of a file. It is a schema rather than a row: each member is a column, +in declaration order and under its own name. ```csharp using Vorticity; @@ -69,7 +69,7 @@ using Vorticity; public partial record struct Reading(int Day, double? Celsius, string City); ``` -Writing records out, and completing the file: +Writing records out and completing the file: ```csharp await using (VortexFileWriter writer = VortexSession.Default.CreateWriter("readings.vortex")) @@ -79,8 +79,8 @@ await using (VortexFileWriter writer = VortexSession.Default.CreateWriter().Where(r => r.Day >= 9 } ``` -The lambda given to `Where` runs once, when the scan is built, and what it records is the plan: the -file statistics, the zone maps and the indexes skip what they can before a block is decoded. -`r.Day >= "900"` and `r.Day % 2 == 0` do not compile. +The lambda given to `Where` runs once, when the scan is built, and what it records becomes the plan. +The file statistics, the zone maps and the indexes then skip whatever they can before a block is +decoded. Expressions such as `r.Day >= "900"` or `r.Day % 2 == 0` do not compile. -The same question, and a few more, without a loop: answered from the statistics when they settle -it, from the encoded blocks otherwise, and never by materialising a batch. +The same question, and a few more, can be asked without a loop. The answer comes from the statistics +when they settle it, from the encoded blocks otherwise, and never from a materialised batch. ```csharp long hotDays = await file.Scan() @@ -118,7 +118,7 @@ Spread spread = await file.Scan() public partial record struct Spread(double? Min, double? Max, long Cities); // several answers go into a record ``` -A group by is a query too, and its result a scan of a record, read as batches like a file's: +A group by is a query too, and its result is a scan of a record, read in batches like a file: ```csharp await foreach (var (city, readings, mean) in file.Scan() @@ -133,60 +133,59 @@ await foreach (var (city, readings, mean) in file.Scan() public partial record struct CityMean(string City, long Readings, double? Mean); ``` -Rows, when rows are what you need, are a sink you ask for and pay for: `ToRecordsAsync()` yields -one `Reading` per row, and LINQ starts after it. A file whose schema is not known when the program -is compiled is read by name, `file.Scan("Day", "City").Where($"Day >= {min}")`, each hole of the -filter typed, which is the surface `vxdump` is written on. +When rows are really what you need, ask for them and pay for them: `ToRecordsAsync()` yields one +`Reading` per row, and LINQ takes over from there. A file whose schema is unknown at compile time is +read by column name, `file.Scan("Day", "City").Where($"Day >= {min}")`, with each hole of the filter +passed as a typed value. That is the surface `vxdump` is built on. ## Requirements -The library targets `net11.0`, and `global.json` pins the SDK to a **.NET 11 release candidate** -with `allowPrerelease`. Until .NET 11 reaches general availability you need that SDK installed — -an otherwise current machine on .NET 10 will be refused by `global.json` before anything builds. -That SDK is all `dotnet build` and `dotnet test` need. Everything that compares against the Rust -reference — the cross-check, the ratio axes of the bench, and the corpus generator — wants a Rust -toolchain on top. +The library targets `net11.0`, and `global.json` pins the SDK to a .NET 11 release candidate with +`allowPrerelease`. Until .NET 11 is generally available you need that SDK installed: a machine with +only .NET 10 is refused by `global.json` before anything builds. That SDK is all `dotnet build` and +`dotnet test` need. Comparing against the Rust reference (the cross-check, the ratio gates of the +bench and the corpus generator) also requires a Rust toolchain. -The library ships on nuget.org, for a project that targets `net11.0`; the generator is a -build-time dependency, which `PrivateAssets` keeps out of whatever your project packs: +The library ships on nuget.org for projects that target `net11.0`. The generator is a build-time +dependency, and `PrivateAssets` keeps it out of whatever your project packs: ```xml - - + + ``` -`Vorticity.Dataset` and `Vorticity.RowEncoding`, both experimental, are referenced the same way, -at the same version as the core. +`Vorticity.Dataset` and `Vorticity.RowEncoding` are both experimental. They are referenced the same +way, at the same version as the core. ## Documentation -| Document | Contents | +| document | contents | |---|---| -| [docs/guide/](docs/guide/README.md) | **Using the library**: pages named by what you are trying to do, each one a program the samples project compiles and runs | -| [docs/guide/benchmarks.md](docs/guide/benchmarks.md) | Every figure the bench publishes: what this costs against the Rust implementation, and each kernel against its baseline | -| [docs/guide/benchmarks-x64.md](docs/guide/benchmarks-x64.md) | The same page from an x64 machine, a Zen 4 processor under Windows, against a Rust built there | -| [docs/guide/benchmarks-duckdb.md](docs/guide/benchmarks-duckdb.md) | Our group bys against DuckDB's on the same files, through its Vortex reader and from its own table, query by query, on one thread and on fourteen | -| [docs/design/](docs/design/README.md) | **Why it is shaped this way**: the design documents, from the scope to the byte layout to the public API | +| [docs/guide/](docs/guide/README.md) | how to use the library: pages named after what you want to do, each backed by a program the samples project compiles and runs | +| [docs/guide/benchmarks.md](docs/guide/benchmarks.md) | every figure the bench publishes: Vorticity against the Rust implementation, and each kernel against its baseline | +| [docs/guide/benchmarks-x64.md](docs/guide/benchmarks-x64.md) | the same page measured on an x64 machine, a Zen 4 processor under Windows, against a Rust built there | +| [docs/guide/benchmarks-duckdb.md](docs/guide/benchmarks-duckdb.md) | our group bys against DuckDB's on the same files, through its Vortex reader and from its own table, on one thread and on fourteen | +| [docs/design/](docs/design/README.md) | why the library is shaped this way: scope, byte layout, engine, public API | ## How performance is measured -Vorticity is measured against Vortex's Rust implementation on the same files and the same -machine, built with Native AOT, on one core and on all of them: a table read and written, every -encoding decoded, taken from and written, and each hot loop against its baseline. Both sides -do the same work: each maps the file and reads it where it lies, decodes every value it returns, -and hands what it writes to a sink that keeps nothing; on one core, Rust is timed under the faster -of its two ways of splitting a scan. The speedup is Rust's time over Vorticity's: above 1.00x, -Vorticity is faster. The table above is read from the file Vorticity writes. +Vorticity is measured against Vortex's Rust implementation on the same files and the same machine, +both built with Native AOT, on one core and on all of them. The bench covers a table read and +written, every encoding decoded, taken from and written, and each hot loop against its baseline. +Both sides do the same work: each maps the file and reads it where it lies, decodes every value it +returns, and hands what it writes to a sink that keeps nothing. On one core, Rust is timed with the +faster of its two ways of splitting a scan. The speedup is Rust's time over Vorticity's, so above +1.00x Vorticity is faster. The table at the top of this page reads the file Vorticity writes. Per encoding, Vorticity is well ahead on nested, text and compressed integer columns (`struct` -6.92x, `varbin` 6.08x, `map` 3.28x, `pco` 3.06x), and level or behind on 10 files by a tenth at -most: plain booleans (0.90x), `onpair` (0.95x), `decimal` (0.96x), masked columns (0.96x to -1.00x), patched bit-packed integers (0.98x), `alprd` and `sequence` (1.00x). Its file is no more -than 5 % larger than Rust's, or smaller, on 44 of 56. Both readers warmed up in one process, Rust -is faster on 1 of 19 cases: a prefix filter on FSST strings (0.93x). +6.92x, `varbin` 6.08x, `map` 3.28x, `pco` 3.06x). It is level with Rust or behind by at most a tenth +on 10 files: plain booleans (0.90x), `onpair` (0.95x), `decimal` (0.96x), masked columns (0.96x to +1.00x), patched bit-packed integers (0.98x), `alprd` and `sequence` (1.00x). Its files are at most +5 % larger than Rust's, or smaller, on 44 of 56 encodings. With both readers warmed up in one +process, Rust is faster on 1 case out of 19, a prefix filter on FSST strings (0.93x). Every figure is on [the benchmark page](docs/guide/benchmarks.md), each section with the machine -and the commit it was measured on, and [its x64 twin](docs/guide/benchmarks-x64.md) holds the -same sections measured on x64. [docs/design/05-benchmarks.md](docs/design/05-benchmarks.md) says what is -compared and how, and [bench/README.md](bench/README.md) how to run each instrument: `bench/gate.sh` -runs everything that gates a commit. +and the commit it was measured on, and [its x64 twin](docs/guide/benchmarks-x64.md) holds the same +sections measured on x64. [docs/design/05-benchmarks.md](docs/design/05-benchmarks.md) explains what +is compared and how, and [bench/README.md](bench/README.md) how to run each instrument. +`bench/gate.sh` runs everything that gates a change before it is pushed. diff --git a/SECURITY.md b/SECURITY.md index 9e87af46..efc0d21f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,27 +2,27 @@ ## What is guaranteed -**A malformed file produces an exception, never worse.** A file that violates the format — bad -offsets, truncation, out-of-range structural fields, exceeded caps — raises a clean -`VortexFormatException` or `VortexUnsupportedException`. No out-of-bounds access, no unbounded -allocation, no hang. Parser fuzzing in CI is what enforces this, on every pull request. +A malformed file produces an exception, never anything worse. A file that violates the format (bad +offsets, truncation, out-of-range structural fields, exceeded caps) raises a clean +`VortexFormatException` or `VortexUnsupportedException`, with no out-of-bounds access, no unbounded +allocation and no hang. Parser fuzzing in CI enforces this on every pull request. ## What is not -**A well-formed file that lies is believed.** Statistics embedded in a file — zone bounds, null -counts, sortedness — are used for pruning, and a file whose structure is valid but whose statistics -are false may yield wrong results. This is inherent to every format with embedded statistics. +A well-formed file that lies is believed. Statistics embedded in a file (zone bounds, null counts, +sortedness) are used for pruning, so a file whose structure is valid but whose statistics are false +may yield wrong results. This is inherent to every format with embedded statistics. `VerifyStatistics` trades throughput for validation when the producer is not trusted. -Two further things are explicitly not claimed: protection against a producer targeting resource -exhaustion *below* the configured caps, and any confidentiality property — the format's encryption -slot is an empty reserved table and this library writes it empty. +Two more things are explicitly not claimed: protection against a producer aiming at resource +exhaustion below the configured caps, and any confidentiality property. The format's encryption slot +is an empty reserved table, and this library writes it empty. ## Reporting -Report a vulnerability through the repository's **Security** tab, with *Report a vulnerability*. -That opens a private advisory visible only to you and the maintainers; a public issue is the wrong -place and cannot be taken back. +Report a vulnerability through the repository's Security tab, with "Report a vulnerability". That +opens a private advisory visible only to you and the maintainers. A public issue is the wrong place, +and cannot be taken back. -Include the file or the generator that reproduces it. A crash on a malformed input is a bug in -scope of the guarantee above and is treated as one. +Include the file, or the generator that reproduces it. A crash on a malformed input is a bug within +the scope of the guarantee above, and is treated as one. diff --git a/bench/README.md b/bench/README.md index d484ed85..3e4eebf2 100644 --- a/bench/README.md +++ b/bench/README.md @@ -1,75 +1,107 @@ # The bench, in one page Every command below is run from the repository root. `PROJ` stands for -`--project bench/Vorticity.Benchmarks`; add `-c Release` always — a Debug benchmark measures the -JIT's unoptimized output and is worth nothing. Forget it and every mode refuses with exit 2 rather -than printing a table. +`--project bench/Vorticity.Benchmarks`. Always add `-c Release`, because a Debug benchmark measures +the JIT's unoptimized output and is worth nothing. If you forget it, every mode refuses with exit 2 +rather than printing a table. ``` dotnet run -c Release PROJ -- ``` -**What the bench publishes is a speedup; what it gates on is a ratio.** Every table of the benchmark -page, and `--report` and BenchmarkDotNet on the console, give the other side's time over the measured -side's — the reference's over ours, a kernel's baseline's over its own — so above 1.00x the measured -side took less. The gates and `ab.sh` hold and print the inverse: the measured side's time, or bytes, -over the side it is compared with — ours over the reference's, the later build's over the earlier's — -so under 1.00 the measured side took less, and a ceiling bounds it from above. +The bench publishes speedups and gates on ratios. Every table of the benchmark page, and `--report` +and BenchmarkDotNet on the console, give the other side's time over the measured side's (the +reference's over ours, a kernel's baseline's over its own), so above 1.00x the measured side took +less. The gates and `ab.sh` hold and print the inverse, the measured side's time or bytes over the +side it is compared with (ours over the reference's, the later build's over the earlier one's), so +under 1.00 the measured side took less, and a ceiling bounds it from above. -**Every figure it publishes is on one page**, [docs/guide/benchmarks.md](../docs/guide/benchmarks.md). -`--out docs/guide/benchmarks.md` makes the report, `--ratio-check`, `--throughput --take`, +Every figure it publishes is on one page, [docs/guide/benchmarks.md](../docs/guide/benchmarks.md). +With `--out docs/guide/benchmarks.md`, the report, `--ratio-check`, `--throughput --take`, `--throughput --write`, `--tradeoffs`, `--tradeoffs --advise` and a BenchmarkDotNet run each write -their own sections of it, dated and signed with the machine and the commit, and leave the others; -a run narrowed to some axes, files or columns writes nothing, and a BenchmarkDotNet run writes the -classes it ran. The page's head lists the command behind each section. -On an x64 machine the same runs take `--out docs/guide/benchmarks-x64.md`, which holds the same -sections measured there; publish the Native AOT runner first, since the report times the one it -finds. +their own sections of it, dated and signed with the machine and the commit, and leave the others +alone. A run narrowed to some axes, files or columns writes nothing, and a BenchmarkDotNet run writes +the classes it ran. The head of the page lists the command behind each section. On an x64 machine +the same runs take `--out docs/guide/benchmarks-x64.md`, which holds the same sections measured +there. Publish the Native AOT runner first, since the report times the one it finds. ## The loop: what you changed, what you run | you changed | you run | it costs | what it answers | |---|---|---|---| -| a kernel | `-- fastlanes` (its class) | 1–8 s | did the kernel move, against the ported arm on the same clock | -| anything, want a direction | no argument at all | **1 min** | the four default classes, 17 benchmarks, fast profile | -| a number about to be written down | `-- --full fastlanes` | 1–4 min | the reference profile, on the ONE class concerned | +| a kernel | `-- fastlanes` (its class) | 1–8 s | whether the kernel moved, against the ported arm on the same clock | +| anything, and you want a direction | no argument at all | 1 min | the four default classes, 17 benchmarks, fast profile | +| a number about to be written down | `-- --full fastlanes` | 1–4 min | the reference profile, on the one class concerned | | a read path | `-- --ratio-check [axis…]` | 56 s, or 5 s for one axis | the twenty-six axes against Rust, interleaved, each held to a ceiling | -| a bitmap kernel | `-- BitmapKernel` | 15 s | each of `Classify`, `CountSet`, `CopyRange`, `PackBytes` against the plain loop it stands in for | -| a gather, a tile, a dictionary | `-- RowKernel` | 9 s | `Gather`, `GatherMasked`, `Tile` against the per-row type switch each replaced | -| a string heap cut into views | `-- ViewKernel` | 9 s | `SumLengths`, `BuildFromLengths`, `RequireAscending` against the per-row loops | -| the OnPair token concatenation | `-- OnPairKernel` | 6 s | the bulk of an OnPair scan, against the per-code switch it stands in for | -| whether your change moved anything | `bench/compare.sh --record before `, then `--record after`, then `bench/compare.sh before after` | 2x the class | Mann-Whitney per case: Faster / Same / Slower | -| a decoder | `-- --throughput ` | ~30 s | ns/value per encoding at a million rows, against Rust. **Reports, never gates**: a run of one file is +32% on our side, the JIT not finished with it | -| the writer, or a decoder | `-- --throughput --check` | 54 s | the same, as a gate over all 57 files | -| a selective decode (`DecodeSelected`) | `-- --throughput --take --check` | 90 s | 64 rows spread over each of the 57 files, against Rust | -| a lane, or the degree of parallelism | `-- LanesBench` (`--full` walks 1, 2, 4, 8) | 15 s | ours at n lanes against the reference on a Tokio runtime of n workers, threads pinned both sides | -| the compressor's decision | `-- CompressorBench` | 12 s | `Choose` and one arm per candidate; `--full` adds the utf8 and f64 columns | -| the writer, per encoding | `-- --throughput --write --check` | ~2 min | each file read back out to a discarding sink on both sides, against Rust, both writers given the rows as our reader delivers them; the page prints each writer's bytes beside its time. **A gate** over 56 files; three are noisier than the ×1.15 margin (`onpair`, `sparse`, `constant`): re-run before believing a red. Only `parquet_variant` produces no ratio, and that is **the reference** refusing to write it | -| every file we write, read by Rust | `bench/crosscheck.sh` | 80 s | 854 files compared scalar by scalar, 2 538 751 rows; needs cargo. `gate.sh --crosscheck` folds it in | -| **anything, before you push** | `bench/gate.sh` | 68 s | the ten ratchets, `--ffi-check`, `--ratio-check`; exit 1 if one is red. `--throughput` adds the full axis (92 s) | -| a change too big for a ported arm | `bench/ab.sh [--after ] [scenario…]` | 7 s | two builds of the library in one process, interleaved, ratio per round | -| a dataset's write path, its compaction policy, anything a long-lived dataset pays | `dotnet run -c Release --project bench/Vorticity.Benchmarks.Churn -- --rows 1000000 --ops 2000` | 45 s; ten million rows and ten thousand commits, 3 min | a dataset loaded, then thousands of small commits — appends, updates and deletes of ten rows, `--mix` and `--batch` to change them — with the caller's compaction and vacuum between them. Per window of commits: each kind's latency, requests, bytes written and allocated, and the compaction it made due. Per checkpoint: eleven reads, from a point lookup and seeks both ways to a full scan, with their requests, and the heap. **A cost that grows with the commits shows as a slope**; `--compact none` shows what a caller who never compacts pays, `--keys tail` a time series, `--marks off` a dataset that rewrites every object a delete or an update touches, `--purge off` one whose compaction leaves marks to the deletes, and `--probe-rounds 5` keeps each probe's fastest round on a machine that runs other work | +| whether your change moved anything | `bench/compare.sh --record before `, then `--record after`, then `bench/compare.sh before after` | 2x the class | Mann-Whitney per case: Faster, Same or Slower | +| a decoder | `-- --throughput ` | ~30 s | ns per value per encoding at a million rows, against Rust. It reports and never gates, since a run of one file reads +32 % on our side, the JIT not being finished with it | +| the writer, or a decoder | `-- --throughput --check` | 54 s | the same, as a gate over the whole per-encoding corpus | +| a selective decode (`DecodeSelected`) | `-- --throughput --take --check` | 90 s | 64 rows spread over each file of the corpus, against Rust | +| a lane, or the degree of parallelism | `-- LanesBench` (`--full` walks 1, 2, 4, 8) | 15 s | ours at n lanes against the reference on a Tokio runtime of n workers, with threads pinned on both sides | +| the compressor's decision | `-- CompressorBench` | 12 s | `Choose` and one arm per candidate, and `--full` adds the utf8 and f64 columns | +| the writer, per encoding | `-- --throughput --write --check` | ~2 min | each file read back out to a discarding sink on both sides, against Rust, both writers given the rows as our reader delivers them, and the page prints each writer's bytes next to its time. It is a gate. Three files are noisier than the ×1.15 margin (`onpair`, `sparse`, `constant`), so run again before believing a red. Only `parquet_variant` produces no ratio, because the reference refuses to write it | +| every file we write, read by Rust | `bench/crosscheck.sh` | 80 s | 854 files compared scalar by scalar, 2 538 751 rows. Needs cargo, and `gate.sh --crosscheck` folds it in | +| anything, before you push | `bench/gate.sh` | 68 s | the ten ratchets, `--ffi-check` and `--ratio-check`, exiting 1 if one is red. `--throughput` adds the full axis (92 s) | +| a change too big for a ported arm | `bench/ab.sh [--after ] [scenario…]` | 7 s | two builds of the library in one process, interleaved, with a ratio per round | +| a dataset's write path, its compaction policy, anything a long-lived dataset pays for | `dotnet run -c Release --project bench/Vorticity.Benchmarks.Churn -- --rows 1000000 --ops 2000` | 45 s, or 3 min for ten million rows and ten thousand commits | a dataset loaded, then thousands of small commits (appends, updates and deletes of ten rows, with `--mix` and `--batch` to change them), with the caller's compaction and vacuum in between. Per window of commits it reports each kind's latency, requests, bytes written and allocated, and the compaction it made due. Per checkpoint it runs eleven reads, from a point lookup and seeks both ways to a full scan, with their requests, and reports the heap. A cost that grows with the commits shows as a slope. `--compact none` shows what a caller who never compacts pays, `--keys tail` gives a time series, `--marks off` a dataset that rewrites every object a delete or an update touches, `--purge off` one whose compaction leaves marks to the deletes, and `--probe-rounds 5` keeps each probe's fastest round on a machine that runs other work | | the FFI harness, or before trusting any ratio | `-- --ffi-check` | < 1 s | both readers return the same rows on the same files | -| **what a user would see**, for the published page | `-- --report` | ~2 min | eight high-level scenarios at 2^20 rows and ten times that, **each side in its own process**, **on one core and on all of them**, on the file our writer makes and on the one the reference's writer makes from the same rows: the action's time with its spread, its throughput over the plain size of the rows returned (`PlainSize.cs`), what our side allocates for a call once warm (the median of six calls after two), peak resident memory, rows rendered, and each file's chunks and encodings per column. Then **every file of the per-encoding corpus**, scanned by both sides twelve times in each of three processes a side, the median of the last ten: the decoders once warm, which `--no-kernels` leaves out. Our side runs as the **Native AOT runner** built for the machine's instruction set (`dotnet publish -c Release bench/Vorticity.Benchmarks.Runner`), whose time divides the reference's in the speedup, as on every table of the page, and on one core also as this framework-dependent host on the JIT. The reference is the `vxbench` **binary** (`cargo build --release` in `tools/vxbench-rs`), built as upstream builds its benchmarks: mimalloc, `target-cpu=native`, one codegen unit, no LTO. On all cores, `--threads all` gives our scans a lane per processor and the reference a Tokio worker per processor. Build both first. `--markdown --out docs/guide/benchmarks.md` writes its two sections of the benchmark page. The fixtures are written again on every run | -| a hot path you want to profile | `-- --profile [seconds]` | as asked | a bare loop for `dotnet-trace`, no harness in the profile | -| where a report scenario spends its cycles, line by line | `bench/profile.sh cycles ` (macOS, Xcode) | 15 s | the **Native AOT runner** sampled every 25–30 µs by Instruments' CPU Profiler, weighed in cycles: self time per function and **per source line, inlined code included**, and every sample charged to the innermost line of this repository. Run from the command line, no Instruments window | -| what a report scenario allocates, and where | `bench/profile.sh allocations ` (macOS) | 15 s–5 min | **every** allocation of three rounds of the Native AOT runner, stopped on under lldb: type, size, stack to the line, managed and native (the C allocator and `mmap`). Not sampled — the managed totals per round equal the runner's own `allocated_bytes` | -| every allocation site over many scenarios, and whether it grows with the rows | `python3 bench/profile/inventory.py . ` | seconds | one table of sites (library line and its source, allocating line, type), cold and warm counts per trace; a scenario traced at two sizes (`scan-1048576`, `scan-10485760`) says which sites are paid per batch or chunk rather than per scan | -| every branch a scenario takes | `bench/profile.sh trace ` | 30 s | Processor Trace, for Instruments to open. An M4 or later, and Processor Trace allowed in System Settings → Privacy & Security → Developer Tools | -| who ran, and on what, when a scenario has lanes | `bench/profile.sh cycles --threads all` | 20 s | the cycles above, on as many lanes as the report's all-core rows, plus `threads.md` from `bench/profile/threads.py`: each thread's share, how many threads ran in each millisecond, and the hottest functions of the caller's thread and of the pool's workers. The profiler records fewer of each thread's cycles as more of them run: take shares from it, and the process's processor time from the runner's `cpu_ms` | -| where a group by spends its cycles | `bench/profile.sh cycles group-total-k6+core ~/.cache/vorticity/queries/spread-random-4000000.vortex 0 --threads 14` | 30 s | the cycles above for a group by of the high-cardinality bench (`group--`, the plan's switches after `+`: `GroupScenarios.cs` in the runner), over the spread files `vortex-queries --matrix small` writes; the runner prints the engine's counts, the core's with it. The bench compiles as it goes, and Instruments names no frame of a method compiled at run time | -| a change to the engine, against the last change to the same part | `bench/runners.sh keep ` once, then `bench/runners.sh alt HEAD --threads 1,14 [--warm] group--[@file] …` | 1 min for three scenarios at one lane | the **Native AOT runner** of a commit published once and kept under `~/.cache/vorticity/runners/`, and two kept runners run in turns, a process each, in pairs whose order alternates: three pairs, then more while the 95 % interval of the median ratio is wider than ±3 % (eight at most), the median of the rounds after the first: the ratio of the two binaries, which a loaded machine moves less than it moves a number of milliseconds taken another day. Two canary runs before and after each scenario, against the session's first, catch what other processes did meanwhile; a scenario framed by a canary 5 % off, or during which swap grew, runs again once. The table prints the interval, the pairs, the ratio of the fastest rounds (a gap of 10 % from the median marks the scenario noisy), the ratio of the rounds' cycles, the load and the processes that were busy. `--warm` keeps each side's file mapped (`VORTICITY_RUNNER_WARM=1`). Without a commit, `keep` publishes the working tree's tracked changes. Every run is kept with its time of day in `bench/.ab/runners-/runs.tsv` | -| our group bys against DuckDB's | `bench/duckdb.sh [--set hc\|db\|all] [--only ] [--rows N] [--threads 1,14] [--bases both\|warm\|cold]`, then `bench/duckdb.sh --publish ` | 15 to 25 min a session for both sets at 1 and 14 threads, its files written; 30 s for `--only db-q4,db-q10 --threads 14` | the same files, the bench's own at edition core2025.10, read once before anything is timed, in **alternated pairs**: one DuckDB process a file kept open and driven through a FIFO, its table loaded once, and the Native AOT runner, a block of each in turn, ABBA, three pairs and then more while the 95 % interval of the median speedup is wider than ±5 % (eight at most); 10 rounds a block, 3 dropped, or 5 and 2 past 100 ms. Two bases: our runner with its file kept mapped against DuckDB's table in memory (warm), mapping it anew against DuckDB's vortex reader (cold). DuckDB's time is its JSON profile's latency under 20 ms (to the nanosecond), its timer's past that (the profile cost 2 % on q10); the CPU of both sides beside it. The canaries, controls and replays of `bench/runners.sh`; each answer held to the others. Everything is kept under `bench/.runs/duckdb-/`; `--publish` builds the page's table from two sessions taken at different hours, a row published when they agree within 10 %. `hc` is the high-cardinality table (keys of 10³ to 10⁷ values), `db` db-benchmark's group by, the queries our native aggregates cover; `--only db-q4,total-k7` keeps the rows named by the runner's scenario without `group-`. Needs `duckdb` and its vortex extension; informs, gates nothing | -| the runner, its file kept mapped, and what a round cost | `VORTICITY_RUNNER_WARM=1 Vorticity.Benchmarks.Runner --scenario … --repeat N` | — | by default each round maps its file anew, as the reference maps it at each call; `VORTICITY_RUNNER_WARM=1` keeps the session's default of 64 mappings, the file already in memory, which is what a comparison against a table loaded beforehand needs. Each round line carries `cpu_us`, `instructions` and `cycles`, the process's on every thread, read with `proc_pid_rusage` as a user (macOS; zero elsewhere): a time that moves while the cycles do not was moved by the machine | +| what a user would see, for the published page | `-- --report` | ~2 min | see [the report](#the-report) | +| a hot path you want to profile | `-- --profile [seconds]` | as asked | a bare loop for `dotnet-trace`, with no harness in the profile | +| where a report scenario spends its cycles, line by line | `bench/profile.sh cycles ` (macOS, Xcode) | 15 s | the Native AOT runner sampled every 25–30 µs by Instruments' CPU Profiler, weighed in cycles: self time per function and per source line, inlined code included, every sample charged to the innermost line of this repository. It runs from the command line, with no Instruments window | +| what a report scenario allocates, and where | `bench/profile.sh allocations ` (macOS) | 15 s–5 min | every allocation of three rounds of the Native AOT runner, stopped on under lldb: type, size, stack down to the line, managed and native (the C allocator and `mmap`). It is not sampled, and the managed totals per round equal the runner's own `allocated_bytes` | +| every allocation site over many scenarios, and whether it grows with the rows | `python3 bench/profile/inventory.py . ` | seconds | one table of sites (library line and its source, allocating line, type), with cold and warm counts per trace. A scenario traced at two sizes (`scan-1048576`, `scan-10485760`) shows which sites are paid per batch or chunk rather than per scan | +| every branch a scenario takes | `bench/profile.sh trace ` | 30 s | Processor Trace, for Instruments to open. It needs an M4 or later, with Processor Trace allowed in System Settings → Privacy & Security → Developer Tools | +| who ran, and on what, when a scenario has lanes | `bench/profile.sh cycles --threads all` | 20 s | the cycles above, on as many lanes as the report's all-core rows, plus `threads.md` from `bench/profile/threads.py`: each thread's share, how many threads ran in each millisecond, and the hottest functions of the caller's thread and of the pool's workers. The profiler records fewer of each thread's cycles as more threads run, so take shares from it, and the process's processor time from the runner's `cpu_ms` | +| where a group by spends its cycles | `bench/profile.sh cycles group-total-k6+core ~/.cache/vorticity/queries/spread-random-4000000.vortex 0 --threads 14` | 30 s | the cycles above for a group by of the high-cardinality bench (`group--`, with the plan's switches after `+`, see `GroupScenarios.cs` in the runner), over the spread files `vortex-queries --matrix small` writes. The runner prints the engine's counts, the core's included. The bench compiles as it goes, and Instruments names no frame of a method compiled at run time | +| a change to the engine, against the last change to the same part | `bench/runners.sh keep ` once, then `bench/runners.sh alt HEAD --threads 1,14 [--warm] group--[@file] …` | 1 min for three scenarios at one lane | see [runners](#comparing-two-runners) | +| our group bys against DuckDB's | `bench/duckdb.sh [--set hc\|db\|all] [--only ] [--rows N] [--threads 1,14] [--bases both\|warm\|cold]`, then `bench/duckdb.sh --publish ` | 15 to 25 min per session for both sets at 1 and 14 threads, files included, or 30 s for `--only db-q4,db-q10 --threads 14` | see [the DuckDB page](../docs/guide/benchmarks-duckdb.md), which describes the method. Everything is kept under `bench/.runs/duckdb-/`, and `--publish` builds the page's tables from two sessions taken at different hours, a row being published when they agree within 10 %. `hc` is the high-cardinality table (keys of 10³ to 10⁷ values) and `db` db-benchmark's group by. `--only db-q4,total-k7` keeps the rows named by the runner's scenario without `group-`. It needs `duckdb` and its vortex extension, and informs without gating anything | +| the runner, its file kept mapped, and what a round cost | `VORTICITY_RUNNER_WARM=1 Vorticity.Benchmarks.Runner --scenario … --repeat N` | | by default each round maps its file anew, as the reference maps it at each call. `VORTICITY_RUNNER_WARM=1` keeps the session's default of 64 mappings, with the file already in memory, which is what a comparison against a table loaded beforehand needs. Each round line carries `cpu_us`, `instructions` and `cycles`, the process's on every thread, read with `proc_pid_rusage` as a user (macOS, zero elsewhere), so a time that moves while the cycles do not was moved by the machine | | a ratchet | `dotnet test Vorticity.slnx -c Release` | ~1 min | the suite plus the ten allocation, count, budget and work ratchets | -| a change, and only the tests it can affect | `dotnet run -c Release --project tools/testimpact -- run` (after `-- map` once) | seconds | the test classes that ran a method the working tree changed against `HEAD` (`--base REF` for another), from a map of what each class runs: every class in a process of its own with inlining off, the JIT's list of compiled methods recorded (35 s). A constant, a build file or the source generator runs everything; a new class runs; a comment or a `using` runs nothing | - -**A fast-profile figure is a direction, not a number.** Anything under about 5 % on a kernel, and -every figure that gets recorded, is confirmed with `--full` on the one class concerned. Why, and -what the fast profile costs in fidelity: `BenchmarkConfig.cs`. - -**Never run the whole BDN suite to decide one change.** Run the class, before and after, on the same -build. The gates are the other instrument: they are *commands*, deliberately not `dotnet test` +| a change, and only the tests it can affect | `dotnet run -c Release --project tools/testimpact -- run` (after `-- map` once) | seconds | the test classes that ran a method the working tree changed against `HEAD` (`--base REF` for another), from a map of what each class runs, built by running every class in a process of its own with inlining off and recording the JIT's list of compiled methods (35 s). A constant, a build file or the source generator runs everything, a new class runs, and a comment or a `using` runs nothing | + +### The report + +`-- --report` runs eight high-level scenarios at 2^20 rows and ten times that, each side in its own +process, on one core and on all of them, on the file our writer makes and on the one the reference's +writer makes from the same rows. It reports the action's time with its spread, its throughput over +the plain size of the rows returned (`PlainSize.cs`), what our side allocates per call once warm (the +median of six calls after two), the peak resident memory, the rows rendered, and each file's chunks +and encodings per column. Then it scans every file of the per-encoding corpus with both sides, twelve +times in each of three processes per side, and keeps the median of the last ten: the decoders once +warm, which `--no-kernels` leaves out. + +Our side runs as the Native AOT runner built for the machine's instruction set (`dotnet publish -c +Release bench/Vorticity.Benchmarks.Runner`), whose time is the divisor in the speedup, as on every +table of the page, and on one core also as this framework-dependent host on the JIT. The reference is +the `vxbench` binary (`cargo build --release` in `tools/vxbench-rs`), built the way upstream builds +its benchmarks: mimalloc, `target-cpu=native`, one codegen unit, no LTO. On all cores, `--threads all` +gives our scans a lane per processor and the reference a Tokio worker per processor. Build both +first. `--markdown --out docs/guide/benchmarks.md` writes its two sections of the benchmark page, and +the fixtures are written again on every run. + +### Comparing two runners + +`bench/runners.sh keep ` publishes the Native AOT runner of a commit once and keeps it under +`~/.cache/vorticity/runners/`. Without a commit, `keep` publishes the working tree's tracked +changes. `bench/runners.sh alt` then runs two kept runners in turns, one process each, in pairs whose +order alternates: three pairs, then more while the 95 % interval of the median ratio is wider than +±3 % (eight at most), taking the median of the rounds after the first. That gives the ratio of the two +binaries, which a loaded machine moves less than it moves a number of milliseconds measured on +another day. + +Two canary runs before and after each scenario, compared with the session's first, catch what other +processes did meanwhile, and a scenario framed by a canary 5 % off, or during which swap grew, runs +again once. The table prints the interval, the pairs, the ratio of the fastest rounds (a gap of 10 % +from the median marks the scenario noisy), the ratio of the rounds' cycles, the load and the +processes that were busy. `--warm` keeps each side's file mapped (`VORTICITY_RUNNER_WARM=1`). Every +run is kept with its time of day in `bench/.ab/runners-/runs.tsv`. + +A fast-profile figure is a direction, not a number. Anything under about 5 % on a kernel, and every +figure that gets recorded, is confirmed with `--full` on the one class concerned. `BenchmarkConfig.cs` +explains why, and what the fast profile costs in fidelity. + +Never run the whole BenchmarkDotNet suite to decide one change. Run the class, before and after, on +the same build. The gates are the other instrument: they are commands, deliberately not `dotnet test` cases, so CI calls them and a red one never silently stops the suite from running. ## Selecting @@ -78,19 +110,20 @@ Three mechanisms, finest first: | form | means | |---|---| -| `-- fsst` | a bare word becomes `--filter '*fsst*'` — the shell would eat the stars | +| `-- fsst` | a bare word becomes `--filter '*fsst*'`, since the shell would eat the stars | | `-- --filter '*Fsst*.Library'` | BenchmarkDotNet's own glob, over `Class.Method` | | `-- --anyCategories kernel` | by category: every class is `kernel`, `path` or `explore` | | `-- --list flat` | what exists under the current selection | -**`-- --help` lists every mode**, its flags and what each costs. `--full`, `--explore` and `--inprocess` are ours and are consumed before BenchmarkDotNet sees the -rest, so `-- --full fastlanes` works as written. Everything else is forwarded. +`-- --help` lists every mode, its flags and what each costs. `--full`, `--explore` and `--inprocess` +are ours and are consumed before BenchmarkDotNet sees the rest, so `-- --full fastlanes` works as +written. Everything else is forwarded. -**`--full` runs out of process; the fast profile does not.** BenchmarkDotNet 0.16.0-preview.1, the +`--full` runs out of process, and the fast profile does not. BenchmarkDotNet 0.16.0-preview.1, the first version that can build a `net11.0` host, gives the reference profile process isolation per -case, GC and runtime jobs, and `--disasm`. -It costs the host's build: `--full` on a two-case class is 2 min 10 rather than 7 s, which is why -the fast profile stays in this process. `--full --inprocess` is the escape hatch. +case, GC and runtime jobs, and `--disasm`. It costs the host's build: `--full` on a two-case class +takes 2 min 10 s rather than 7 s, which is why the fast profile stays in this process. +`--full --inprocess` is the escape hatch. ## Asking what our own writer costs to read back @@ -112,10 +145,10 @@ VORTICITY_THROUGHPUT_CORPUS=/tmp/pair \ dotnet run -c Release --project bench/Vorticity.Benchmarks -- --throughput ``` -That pair locates a slow read of our own rewrite: the same file rewritten with and without zstd, -read by both readers, says whether the frames are the cost. `vxdump --encodings` cannot say which -column holds them — the reference interns all 37 encodings of the registry whatever the file uses, -so only OUR dictionary is informative. What does answer it is `vxdump --layout`, whose `encoding=` +That pair locates a slow read of our own rewrite: the same file rewritten with and without zstd, read +by both readers, says whether the frames are the cost. `vxdump --encodings` cannot say which column +holds them, because the reference interns all 37 encodings of the registry whatever the file uses, so +only our own dictionary is informative. What does answer it is `vxdump --layout`, whose `encoding=` column names the array encoding of every terminal node: ```sh @@ -124,16 +157,15 @@ dotnet run -c Release --project tools/vxdump -- /tmp/pair/ours.vortex --layout | ``` The `vortex.zstd` nodes it counts, and the columns they sit on, put the slowdown on a column rather -than leave it to be inferred from a target edition. +than leaving it to be inferred from a target edition. ## Reading the assembly a kernel actually got -Two routes, and they answer slightly different questions. +There are two routes, and they answer slightly different questions. ```sh # The whole class, through BenchmarkDotNet. Needs --full, because the disassembler needs the -# out-of-process toolchain (bench/README's profile table). Writes -asm.md under the -# artifacts directory. +# out-of-process toolchain. Writes -asm.md under the artifacts directory. dotnet run -c Release --project bench/Vorticity.Benchmarks -- \ --full --filter '*FastLanesKernelBenchmarks.Vector64*' --disasm @@ -144,162 +176,170 @@ DOTNET_JitStdOutFile=/tmp/jit.asm \ dotnet run -c Release --project bench/Vorticity.Benchmarks -- FastLanesKernel ``` -`DOTNET_TieredCompilation=0` is not optional if you want the code that runs in steady state: with -tiering on, the listing's header says `Tier0-FullOpts ... optimized using Synthesized PGO`, which is -the first full-opts compilation and not necessarily the last. With it off the header says `FullOpts` -and there is one listing per generic instantiation. +`DOTNET_TieredCompilation=0` is required if you want the code that runs in steady state. With tiering +on, the listing's header says `Tier0-FullOpts ... optimized using Synthesized PGO`, which is the first +full-opts compilation and not necessarily the last. With it off, the header says `FullOpts` and there +is one listing per generic instantiation. -**Hardware counters do not work here.** `[HardwareCounters]` needs -`BenchmarkDotNet.Diagnostics.Windows` and ETW; on this machine the run prints `Unable to resolve -IHardwareCountersDiagnoser diagnoser using dynamic assembly loading` and then completes **without -counters and without failing** — so a table that looks normal is simply missing the columns you -asked for. Branch mispredictions and cache misses need a Windows machine. +Hardware counters do not work here. `[HardwareCounters]` needs `BenchmarkDotNet.Diagnostics.Windows` +and ETW. On macOS the run prints `Unable to resolve IHardwareCountersDiagnoser diagnoser using dynamic +assembly loading` and then completes without counters and without failing, so a table that looks +normal is simply missing the columns you asked for. Branch mispredictions and cache misses need a +Windows machine. -**No argument means all of them**, not a prompt: without one, BenchmarkDotNet asks the console which -class to run, which makes the default run do nothing under a script or in CI. `Program.cs` supplies -`--filter *` when nothing else has said what to run. +No argument means all of them, not a prompt. Without one, BenchmarkDotNet asks the console which +class to run, which would make the default run do nothing under a script or in CI, so `Program.cs` +supplies `--filter *` when nothing else has said what to run. ## The classes -Ten, and `-- --list flat` is what answers this question for real: four in the default run, six more -behind `--explore`. +There are ten, and `-- --list flat` is what answers this question for real: four in the default run, +and six more behind `--explore`. | class | in the default run | |---|---| -| `FastLanesKernelBenchmarks` | yes — 17 bits; `--full` adds 10 and 33 | +| `FastLanesKernelBenchmarks` | yes, at 17 bits, and `--full` adds 10 and 33 | | `RowEncodingBenchmarks` | yes | | `CompressorBenchmarks` | yes | | `LanesBenchmarks` | yes | -| `RandomAccessBenchmarks` | no — `--explore` | -| `FilterSelectivityBenchmarks` | no — `--explore` | -| `ConstantFormBenchmarks` | no — `--explore` | -| `VarBinFormBenchmarks` | no — `--explore` | -| `ComplexityProbes` | no — `--explore` | -| `MergeProbes` | no — `--explore` | - -`explore` holds the **curves** — a selectivity sweep, a take sweep, a complexity probe — which -answered their question once and do not guard against anything. They stay runnable and stay out of -the default run: 17 benchmarks become 39 with `--explore`. - -A class is deleted rather than demoted when another instrument measures the same thing with a -better estimator: our own bytes read by both readers, for one, are the four `rewritten` axes of +| `RandomAccessBenchmarks` | no, `--explore` | +| `FilterSelectivityBenchmarks` | no, `--explore` | +| `ConstantFormBenchmarks` | no, `--explore` | +| `VarBinFormBenchmarks` | no, `--explore` | +| `ComplexityProbes` | no, `--explore` | +| `MergeProbes` | no, `--explore` | + +`explore` holds the curves (a selectivity sweep, a take sweep, a complexity probe), which answered +their question once and do not guard against anything. They stay runnable but out of the default run, +and the 17 benchmarks become 39 with `--explore`. + +A class is deleted rather than demoted when another instrument measures the same thing with a better +estimator. Our own bytes read by both readers, for instance, are the four `rewritten` axes of `--ratio-check`. ## The gates, and their ceilings -* **`--ratio-check`** — twenty-six axes, ours against Rust, **interleaved against one clock** so - that drift is common to both arms. Ten on the dataset (full scan, full scan upstream-lazy, - projected, projected upstream-lazy, first batch, footer only, read-and-write-back, filtered 1 %, - filtered half, scattered - take); four `rewritten` ones that read a file **our writer produced** beside the reference's — the - only place our own encoding choices are measured at all; three on key order, one of them held to a - take of rows whose positions the reference is given, and one on the exact count; six on predicates - pushed into a string or a packed column (equality and prefix over `fsst` and over `dict`, a band - over `runend` and over `bitpacked`); and two on a fifty-column table. Each has a ceiling in - `RatioCheck.cs`; over it, non-zero exit. 56 s. - A bare word narrows it to the axes whose name contains it — `-- --ratio-check write`, `-- - --ratio-check rewritten`, `-- --ratio-check string` — which is the difference between checking one - change and waiting for twenty-six axes. - - **Do not pin tiered compilation for this gate.** `DOTNET_TieredCompilation=0` costs our side - dynamic profile-guided optimization while the reference, being native, loses nothing: the same - command that is green unpinned reports **several axes over their ceiling** with the pin, `full - scan` among them. `bench/ab.sh` pins on purpose and says why — both of its sides are the same - runtime and the pin removes a JIT difference between them. A ratio against native code is the - opposite case. -* **`--throughput [family…] [--check]`** — 57 encodings at a million rows, where the fixed - open-and-walk cost is under a percent instead of most of the measurement. `--check` makes it a - gate. Its inputs are 468 MB and are not committed: run `bench/gen-throughput.sh` once, and it - finds them in `~/.cache/vorticity/throughput-1M` by itself (`VORTICITY_THROUGHPUT_CORPUS` - overrides). `--check` refuses a corpus with no `manifest.json`, one generated at another row - count, or one whose files do not match their recorded sha256 — fifty-seven ratchets against bytes - that live outside the repository need to know *which* bytes. - - **`--take`** asks the same fifty-seven files for 64 rows spread evenly over each (one every 15 625), - against `vxbench_take`, with its own ratchet table — the two axes do not move together, and that - is the point: a decoder without a `DecodeSelected` override decodes the whole chunk a taken row - falls in, once for the take, where one with it decodes only the taken rows: `zstd` inflates only - the frames that hold the taken rows, and the benchmark page's table says what each encoding makes - of it. `--ratio-check`'s single `scattered take` axis says none of this, because it is one file - whose encodings all have the override. - - **`--write`** reads each file back out into a sink that keeps nothing, against `vxbench_write`, - which gives the reference's writer the rows as our reader gives ours, decoded and a constant kept - as one value, and hands its bytes to a sink that keeps nothing too: the read is inside the - measurement on both sides, so subtract the scan axis before reading the quotient as a statement - about writers. With `--out`, the page prints each writer's bytes beside its time, because a writer - can be fast by compressing less. It has its own ratchet table of 56 references, and it reports an - encoding it cannot write rather than dying on it — that is `parquet_variant` alone, and it is the - reference that declines: its variant, decoded, keeps a lazy slice its writer cannot serialize. - - `--quick` (23 s instead of 70) shortens the warm-up and the per-file budget: a direction, not a - gate. `--recalibrate N` prints a replacement reference table, as it does for `--ratio-check`. - - **A bare family name narrows the report, and does not give the gate's ratio**: a file measured - on its own reads higher than in the full run, on the same bytes, the JIT not finished with it. - Use the narrow form to see a direction; confirm with the full run before believing a red. - -**A ceiling only ever comes down, and only behind a real improvement.** Raising one to make a run -pass is the one thing this directory forbids outright. - -**Both gates decide on an interval, not a point.** Each round times both sides microseconds apart, -so the rounds are paired and a per-round ratio is a sample; the median of those ratios is what is -held, with a 95% bootstrap interval around it. A ratio is `OVER` only when the *whole* interval is -above the ceiling, `STALE` only when all of it is under 0.85 × reference, and `noisy` when the median -is over but the interval straddles, which a single run cannot decide. -Rounds keep coming until half the interval is within 5% of the median, and a call shorter than the -timer's noise floor is repeated inside one timed round (the `k` column). The `mde` column is the -smallest change that axis can currently see. - -**An interval is within one run, and between-run variance is 2–3× larger.** Measured over twenty -processes per axis, a 95 % interval contains the grand median 11–12 times out -of 20 rather than 19. So a ratio can read 1.30 [1.28; 1.31] in eight runs and 1.71 [1.65; 1.77] in -the ninth — narrow and wrong. **Narrow does not mean reproducible**, which is why `--recalibrate` -runs each pass in its own process, and why the replay rule below still stands for a single red. - -**When the code outran a ceiling, `--ratio-check` says `STALE`** — more than 15 % under its reference -— because a ratchet that is never lowered defends nothing: a reference left far above its axis leaves +### `--ratio-check` + +Twenty-six axes compare ours against Rust, interleaved against one clock so that drift is common to +both arms. Ten run on the dataset (full scan, full scan upstream-lazy, projected, projected +upstream-lazy, first batch, footer only, read-and-write-back, filtered 1 %, filtered half, scattered +take). Four `rewritten` axes read a file our writer produced next to the reference's, the only place +our own encoding choices are measured at all. Three are on key order, one of them held to a take of +rows whose positions the reference is given, and one is the exact count. Six are predicates pushed +into a string or a packed column (equality and prefix over `fsst` and over `dict`, a band over +`runend` and over `bitpacked`). The last two read generated tables, one of a million rows and six +columns and one of fifty columns. Each has a ceiling in `RatioCheck.cs`, and going over it gives a +non-zero exit. It takes 56 s. + +A bare word narrows it to the axes whose name contains it (`-- --ratio-check write`, +`-- --ratio-check rewritten`, `-- --ratio-check string`), which is the difference between checking +one change and waiting for twenty-six axes. + +Do not pin tiered compilation for this gate. `DOTNET_TieredCompilation=0` costs our side dynamic +profile-guided optimization, while the reference, being native, loses nothing, so the same command +that is green unpinned reports several axes over their ceiling with the pin, `full scan` among them. +`bench/ab.sh` pins on purpose and says why: both of its sides are the same runtime, and the pin +removes a JIT difference between them. A ratio against native code is the opposite case. + +### `--throughput [family…] [--check]` + +This reads the per-encoding corpus at a million rows per file, where the fixed open-and-walk cost is +under a percent instead of most of the measurement. The generator writes 61 files today, and a corpus +generated before the four `pco` shapes were added holds 57. `--check` makes it a gate. Its inputs are +about half a gigabyte and are not committed: run `bench/gen-throughput.sh` once, and the bench finds +them in `~/.cache/vorticity/throughput-1M` by itself (`VORTICITY_THROUGHPUT_CORPUS` overrides it). +`--check` refuses a corpus with no `manifest.json`, one generated at another row count, or one whose +files do not match their recorded sha256, since ratchets against bytes that live outside the +repository need to know which bytes. + +`--take` asks the same files for 64 rows spread evenly over each (one every 15 625), against +`vxbench_take`, with its own ratchet table. The two axes do not move together, and that is the point: +a decoder without a `DecodeSelected` override decodes the whole chunk a taken row falls in, once for +the take, while one with it only decodes the taken rows, and `zstd` only inflates the frames that hold +them. The benchmark page's table says what each encoding makes of it. `--ratio-check`'s single +`scattered take` axis says none of this, because it reads one file whose encodings all have the +override. + +`--write` reads each file back out into a sink that keeps nothing, against `vxbench_write`, which +gives the reference's writer the rows as our reader gives ours (decoded, with a constant kept as one +value) and hands its bytes to a sink that keeps nothing too. The read is inside the measurement on +both sides, so subtract the scan axis before reading the quotient as a statement about writers. With +`--out`, the page prints each writer's bytes next to its time, because a writer can be fast by +compressing less. It has its own ratchet table, and it reports an encoding it cannot write rather +than dying on it. That only happens with `parquet_variant`, and it is the reference that declines, +because its variant, once decoded, keeps a lazy slice its writer cannot serialize. + +`--quick` (23 s instead of 70) shortens the warm-up and the per-file budget, which gives a direction, +not a gate. `--recalibrate N` prints a replacement reference table, as it does for `--ratio-check`. + +A bare family name narrows the report but does not give the gate's ratio, because a file measured on +its own reads higher than in the full run, on the same bytes, the JIT not being finished with it. Use +the narrow form to see a direction, and confirm with the full run before believing a red. + +### Ceilings + +A ceiling only ever comes down, and only behind a real improvement. Raising one to make a run pass is +the one thing this directory forbids outright. + +Both gates decide on an interval, not a point. Each round times both sides microseconds apart, so +the rounds are paired and a per-round ratio is a sample. The median of those ratios is what is held, +with a 95 % bootstrap interval around it. A ratio is `OVER` only when the whole interval is above the +ceiling, `STALE` only when all of it is under 0.85 × reference, and `noisy` when the median is over +but the interval straddles the ceiling, which a single run cannot decide. Rounds keep coming until +half the interval is within 5 % of the median, and a call shorter than the timer's noise floor is +repeated inside one timed round (the `k` column). The `mde` column is the smallest change that axis +can currently detect. + +An interval is computed within one run, and the variance between runs is 2–3× larger. Measured over +twenty processes per axis, a 95 % interval contains the grand median 11–12 times out of 20 rather than +19. So a ratio can read 1.30 [1.28; 1.31] in eight runs and 1.71 [1.65; 1.77] in the ninth, narrow and +wrong. A narrow interval does not mean a reproducible one, which is why `--recalibrate` runs each pass +in its own process, and why the replay rule below still stands for a single red. + +When the code outruns a ceiling, `--ratio-check` says `STALE` (more than 15 % under its reference), +because a ratchet that is never lowered defends nothing: a reference left far above its axis leaves room for a regression of that size to pass. Lower them with: ``` dotnet run -c Release PROJ -- --ratio-check --recalibrate 3 # ~90 s, prints a table to paste ``` -It runs N passes **in N separate processes** — between-run variance is the larger part and passes -inside one process cannot sample it — takes the max of the per-pass medians, and prints the +It runs N passes in N separate processes (variance between runs is the larger part, and passes inside +one process cannot sample it), takes the maximum of the per-pass medians, and prints the `RatioCheck.References` table ready to paste. `--throughput --recalibrate N` does the same for the -per-encoding table. An axis that measures *above* its reference is printed back **unchanged** and -flagged `HELD`: the command lowers ratchets and never raises one. If such an axis is a real -regression, plain `--ratio-check` says `OVER`, and the answer is the code, not the number. - -`--rebase` is the one exception and it is deliberately awkward: it lets a reference rise, and only on -an axis whose `k` moved, because grouping calls into a round changes what is being measured — k calls -in a row are a *warm* path where a single timed call was cache-cold — or under a reference binary -other than the one the table records. Use it when the harness changed, never when the number did, -and say which change in the commit message. - -**A new reference binary is carried, not recalibrated.** Both gates record the fingerprint of the -`vxbench` build their tables were set under (`CalibratedShim`) and refuse to gate under another: a -ratio through a different denominator is a number about the rebuild. Another machine builds another -binary, and its ratios are its own: there `--ratio-check --out` and `--throughput --take --out` (or -`--write`) measure every axis or file for that machine's page and hold none to a reference. After -rebuilding `tools/vxbench-rs` — a new `vortex` pin, a new toolchain, a changed entry point — keep -the previous build and carry the tables over: +per-encoding table. An axis that measures above its reference is printed back unchanged and flagged +`HELD`, since the command lowers ratchets and never raises one. If such an axis is a real regression, +plain `--ratio-check` says `OVER`, and the answer is the code, not the number. + +`--rebase` is the one exception, and it is deliberately awkward. It lets a reference rise, but only +on an axis whose `k` moved, because grouping calls into a round changes what is being measured (k +calls in a row are a warm path where a single timed call was cache-cold), or under a reference binary +other than the one the table records. Use it when the harness changed, never when the number did, and +say which change in the commit message. + +A new reference binary is carried over, not recalibrated. Both gates record the fingerprint of the +`vxbench` build their tables were set under (`CalibratedShim`) and refuse to gate under another one, +since a ratio through a different denominator is a number about the rebuild. Another machine builds +another binary, and its ratios are its own: there, `--ratio-check --out` and `--throughput --take +--out` (or `--write`) measure every axis or file for that machine's page and hold none to a +reference. After rebuilding `tools/vxbench-rs` (a new `vortex` pin, a new toolchain, a changed entry +point), keep the previous build and carry the tables over: ``` dotnet run -c Release PROJ -- --ratio-check --recalibrate 3 --rebase-from dotnet run -c Release PROJ -- --throughput [--take|--write] --recalibrate 3 --rebase-from ``` -Each pass is a pair of processes, one under each binary, and each reference moves by what the new -binary moved its ratio. Our side is the same build under both, so an axis the table held over its -ceiling, or stale under it, is still over or stale after the carry: a plain `--rebase` to the new -measurement would have absorbed it. Paste the tables and the `CalibratedShim` line they print +Each pass is a pair of processes, one under each binary, and each reference moves by as much as the +new binary moved its ratio. Our side is the same build under both, so an axis the table held over its +ceiling, or stale under it, is still over or stale after the carry, where a plain `--rebase` to the +new measurement would have absorbed it. Paste the tables and the `CalibratedShim` line they print together, the three throughput tables before the fingerprint. -**A red gate is not believed on the first run.** Run-to-run spread reaches a fifth on some axes, -and a run can come out red with no byte changed. Replay three times: two reds out of three is a -regression, otherwise it is noise. +A red gate is not believed on the first run. The spread from run to run reaches a fifth on some axes, +and a run can come out red with no byte changed. Replay it three times: two reds out of three is a +regression, and anything else is noise. ## Where the figures live diff --git a/bench/Vorticity.Benchmarks.Queries/DatasetScenarios.cs b/bench/Vorticity.Benchmarks.Queries/DatasetScenarios.cs index 211f99a5..638c0b8f 100644 --- a/bench/Vorticity.Benchmarks.Queries/DatasetScenarios.cs +++ b/bench/Vorticity.Benchmarks.Queries/DatasetScenarios.cs @@ -16,8 +16,8 @@ namespace Vorticity.Benchmarks.Queries; internal sealed record DatasetScenario(string Name, Func> Query, int ProbeEvery = 1); /// -/// The queries of the datasets' stage (PLAN-QUERIES-STREAMING.md, 6e): the readings as one object, -/// as sixteen, and as sixteen with an eighth of every object deleted. The first batch of a scan is +/// The queries a dataset is judged on: the readings as one object, as sixteen, and as sixteen with +/// an eighth of every object deleted. The first batch of a scan is /// to stay flat from one object to sixteen; a group by on sixteen at a degree of many lanes within /// 15 % of the file's. /// diff --git a/bench/Vorticity.Benchmarks.Queries/EngineScenarios.cs b/bench/Vorticity.Benchmarks.Queries/EngineScenarios.cs index 9609dc80..18d5a67d 100644 --- a/bench/Vorticity.Benchmarks.Queries/EngineScenarios.cs +++ b/bench/Vorticity.Benchmarks.Queries/EngineScenarios.cs @@ -7,45 +7,45 @@ namespace Vorticity.Benchmarks.Queries; /// -/// The queries of the engine's stage (PLAN-QUERIES-STREAMING.md, 6j): each line of the stage is judged -/// on one of them, and a query that tracks its aggregation reports its lanes and its merge. +/// The queries the group-by engine is judged on, one per strategy, and a query that tracks its +/// aggregation reports its lanes and its merge. /// internal static class EngineScenarios { internal static IEnumerable<(string File, Scenario Scenario)> All(int large) { - // 6k: the group by that streams, against the same query forced to block. + // The group by that streams, against the same query forced to block. yield return ("readings", new Scenario("group by city day (composite), count avg, blocking", (file, run) => CityDayAsync(file, run, blocking: true), 1_000)); yield return ("readings", new Scenario("group by day (sorted), welford", (file, run) => DayWelfordAsync(file, run, blocking: false), 100)); yield return ("readings", new Scenario("group by day (sorted), welford, blocking", (file, run) => DayWelfordAsync(file, run, blocking: true), 100)); - // 6l: a filter that keeps 7 % of the rows, contiguous, on a sorted key. + // A filter that keeps 7 % of the rows, contiguous, on a sorted key. yield return ($"readings-{large}", new Scenario($"group by city under a 7 % day filter, {large / 1_000_000}M", CitiesOfSomeDaysAsync)); - // 6m, 6n, 6o, 6p: what follows the pass. + // What follows the pass: a top-k, a chosen row, an order on the key, a window without order. yield return ("names", new Scenario("order by count take 100, text key (1M groups)", TopNamesAsync)); yield return ("requests", new Scenario("order by max take 10 with its row, 1M groups", SlowestUsersAsync)); yield return ("requests", new Scenario("ohlc by day (first, max, min, last)", OhlcAsync)); - // 6n: every group's chosen row, read after the pass, against the same query without it. + // Every group's chosen row, read after the pass, against the same query without it. yield return ("requests", new Scenario("max by user with its row, every group (1M groups)", (file, run) => SlowestOfEveryUserAsync(file, run, row: true))); yield return ("requests", new Scenario("max by user, every group (1M groups)", (file, run) => SlowestOfEveryUserAsync(file, run, row: false))); yield return ("draws", new Scenario("order by key take 10, random int (1M groups)", FirstKeysAsync)); yield return ("readings", new Scenario("order by key descending take 7, 1000 days", LastDaysAsync)); yield return ("draws", new Scenario("take 10 without order, random int (1M groups)", AnyKeysAsync)); - // 6c, 6q, 6d1: a table of keys and a merge, by cardinality. + // A table of keys and a merge, by cardinality. yield return ("draws", new Scenario("group by random int (4 groups), count sum", (file, run) => DrawsAsync(file, run, d => d.K4))); yield return ("draws", new Scenario("group by random int (100 groups), count sum", (file, run) => DrawsAsync(file, run, d => d.K100))); yield return ("draws", new Scenario("group by random int (1000 groups), count sum", (file, run) => DrawsAsync(file, run, d => d.K1000))); yield return ("draws", new Scenario("group by random int (100k groups), count sum", (file, run) => DrawsAsync(file, run, d => d.K100k), 10_000)); yield return ("draws", new Scenario("group by random int (1M groups), count sum", (file, run) => DrawsAsync(file, run, d => d.K1M), 100_000)); - // 6b: a key of two fixed-width columns, at a few thousand groups and at nearly a group a row. + // A key of two fixed-width columns, at a few thousand groups and at nearly a group a row. yield return ("draws", new Scenario("group by (random int, random int) (4 000 groups), count sum", (file, run) => PairsAsync(file, run, d => d.K4, d => d.K1000))); yield return ("draws", new Scenario("group by (random int, random int) (1.8M groups), count sum", (file, run) => PairsAsync(file, run, d => d.K100k, d => d.K100), 100_000)); - // 6d2: a hot key and a long tail, and keys each seen twenty times. + // A hot key and a long tail, and keys each seen twenty times. yield return ("skewed", new Scenario("group by skewed key (one key 30 %, 1M rare), count sum", KeyedAsync, 100_000)); // The same key on a file of many chunks a lane: what the last ranges a queue hands out weigh. @@ -273,7 +273,7 @@ private static async Task PairsAsync(VortexFile file, Run run, FuncA key its zones prove final as the read goes (6g), or the same query forced to block. + /// A key its zones prove final as the read goes, or the same query forced to block. private static async Task LateKeysAsync(VortexFile file, Run run, bool blocking, bool zones = false) { long rows = 0; diff --git a/bench/gen-throughput.sh b/bench/gen-throughput.sh index fcf7e2fa..d2561660 100755 --- a/bench/gen-throughput.sh +++ b/bench/gen-throughput.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Generate the 1M-row, single-encoding inputs `--throughput` measures. # -# 468 MB for 57 encodings, so they are not committed. They go in a CACHE directory rather +# About half a gigabyte for 61 files, so they are not committed. They go in a CACHE directory rather # than /tmp: a reboot wiping them is how a corpus gets regenerated by accident, and a # regenerated corpus is exactly what the manifest exists to catch. # diff --git a/docs/design/01-scope.md b/docs/design/01-scope.md index 26568b5d..74275d96 100644 --- a/docs/design/01-scope.md +++ b/docs/design/01-scope.md @@ -4,80 +4,82 @@ What Vorticity covers, what it deliberately leaves out, and the guarantees it ho ## 1. What Vortex is, and what that implies -Vortex is not another Parquet. It is a **columnar compression framework** whose file format is a +Vortex is not another Parquet. It is a columnar compression framework whose file format is a container, and three of its properties shape this implementation: -1. **The file is self-describing.** How rows are partitioned into chunks, zones and columns is not - fixed by the format: the writer decides, by composing a layout tree that the footer serializes. - A reader is therefore an interpreter of that tree, not a parser of a fixed structure. -2. **Data stays compressed until the last moment.** An array is a tree of encodings, such as +1. The file describes itself. How rows are split into chunks, zones and columns is not fixed by the + format. The writer decides, by composing a layout tree that the footer serializes, so a reader is + an interpreter of that tree rather than a parser of a fixed structure. +2. Data stays compressed until the last moment. An array is a tree of encodings, such as `dict(runend(for(bitpacked)))`, and a filter, a take or a projection can run on the compressed form. Decoding is a progressive canonicalization. -3. **Compatibility is governed by editions.** An edition is a frozen set of component ids. Files - written from Vortex 0.36.0 on, edition `core2025.05.0`, are promised to stay readable. A reader's - scope is a set of editions, not a version of the format, which has stayed at `VTXF` version 1. +3. Compatibility is governed by editions. An edition is a frozen set of component ids. Files written + by Vortex 0.36.0 and later, edition `core2025.05.0`, are promised to stay readable. A reader's + scope is therefore a set of editions, not a version of the format, which has stayed at `VTXF` + version 1. -[02-format.md](02-format.md) condenses the format; [90-registry.md](90-registry.md) lists every +[02-format.md](02-format.md) condenses the format, and [90-registry.md](90-registry.md) lists every component and its state. ## 2. What Vorticity is -A reader and a writer of Vortex files for .NET 11, with no third-party dependency: the FlatBuffers -and Protobuf runtimes are written here, and so is the Zstandard codec, `Vorticity.Zstd`; the one -other package the core references is the first-party `System.IO.Hashing` -([03-architecture.md](03-architecture.md) §1). It is compatible with Native AOT and trimming. +Vorticity is a reader and a writer of Vortex files for .NET 11, with no third-party dependency. The +FlatBuffers and Protobuf runtimes are written here, and so is the Zstandard codec, +`Vorticity.Zstd`. The only other package the core references is Microsoft's `System.IO.Hashing` +(see [the founding constraints](03-architecture.md#1-founding-constraints)). It is compatible with +Native AOT and trimming. | package | what it holds | |---|---| -| `Vorticity` | the session, files, the typed scan and its columns, aggregates, the key cursor, the tool path for schemas known only at run time, the writer and its builders, the I/O seam | -| `Vorticity.Generators` | the `[VortexRecord]` source generator and the analyzers VX1001 to VX1008, build-time only | -| `Vorticity.Dataset` | a versioned dataset over an object store ([13-dataset.md](13-dataset.md)); experimental, `VX0001` | -| `Vorticity.RowEncoding` | the byte-sortable row encoding ([06-row-encoding.md](06-row-encoding.md)); experimental, `VX0002`, because upstream reserves the right to change it | +| `Vorticity` | the session, files, the typed scan and its columns, aggregates, the key cursor, the tool path for schemas known only at run time, the writer and its builders, the I/O interface | +| `Vorticity.Generators` | the `[VortexRecord]` source generator and the analyzers VX1001 to VX1011, build time only | +| `Vorticity.Dataset` | a versioned dataset over an object store ([13-dataset.md](13-dataset.md)), experimental under `VX0001` | +| `Vorticity.RowEncoding` | the byte-sortable row encoding ([06-row-encoding.md](06-row-encoding.md)), experimental under `VX0002` because upstream reserves the right to change it | | `Vorticity.Zstd` | Zstandard compression and decompression in managed C#, which the core's `vortex.zstd` and `vortex.zstd_buffers` go through ([its README](../../src/Vorticity.Zstd/README.md)) | -| `vxdump` | the inspection tool, written against the public surface only | +| `vxdump` | the inspection tool, written against the public API only | ## 3. What it reads -- **Every component a Rust writer emits**: 37 array encodings and 7 layouts. That is every - component of the core editions up to `core2026.08.3`, `vortex.zstd_buffers` from the draft - `zstd2026.02.0` edition, and the three Rust writes outside every edition when an upstream flag - asks for them: `fastlanes.delta`, `vortex.patched` and the experimental `vortex.list` layout. -- **Every file from Rust 0.36.0 to the version the conformance corpus is pinned to**, 0.86.1, - value for value ([04-conformance.md](04-conformance.md)). -- **Lazily.** An id this library does not know fails only the read that needs it: an unprojected - column may use one, and a zone-map aggregate nobody knows only disables its pruning - ([08-semantics.md](08-semantics.md) §4). -- **Refused by name**, with the id and its kind: a legacy two-buffer `vortex.fsst`, a shredded - variant child, and a compressed buffer or segment, which the schema declares and no Vortex release - implements ([08-semantics.md](08-semantics.md) §7). +- Every component a Rust writer emits: 37 array encodings and 7 layouts. That covers every component + of the core editions up to `core2026.08.3`, `vortex.zstd_buffers` from the draft `zstd2026.02.0` + edition, and the three components Rust writes outside every edition when an upstream flag asks for + them: `fastlanes.delta`, `vortex.patched` and the experimental `vortex.list` layout. +- Every file from Rust 0.36.0 up to the version the conformance corpus is pinned to, 0.86.1, value + for value ([04-conformance.md](04-conformance.md)). +- Lazily. An id this library does not know only fails the read that needs it. An unprojected column + may use one, and a zone-map aggregate nobody knows only disables its own pruning (see [unknown + components](08-semantics.md#4-unknown-components-resolve-lazily-fail-on-use)). +- A few things are refused by name, with the id and its kind: a legacy two-buffer `vortex.fsst`, a + shredded variant child, and a compressed buffer or segment, which the schema declares but no + Vortex release implements (see [buffer-level LZ4](08-semantics.md#7-buffer-level-lz4)). ## 4. What it writes -- **Files every Rust reader of the target edition opens.** The target is `VortexEditions.Default`, - `core2026.08.3`, read by Vortex Rust from 0.85.0; lower targets are honoured by dropping - what they cannot carry, and refused where a column needs a component the target lacks - ([90-registry.md](90-registry.md)). The reference reads every file this writer produces, value - for value, in CI. -- **Encodings chosen per chunk**, by exact formulas over statistics computed in one pass, with a - trial only for zstd, pco, FSST, OnPair, ALP and ALP-RD ([11-write-strategy.md](11-write-strategy.md)). -- **Pruning structures**: a zone map per block of 8 192 rows, file statistics, bounded string - bounds, and on request the skipping and locating indexes of [10-indexes.md](10-indexes.md), which - a Rust reader ignores. -- **Appends** that continue a file in place, and a repair for a torn one. +- Files every Rust reader of the target edition opens. The default target is + `VortexEditions.Default`, `core2026.08.3`, read by Vortex Rust 0.85.0 and later. Lower targets are + honoured by dropping what they cannot carry, and refused where a column needs a component the + target lacks ([90-registry.md](90-registry.md)). In CI, the reference reads every file this writer + produces, value for value. +- Encodings chosen per chunk, by exact formulas over statistics computed in one pass, with a trial + only for zstd, pco, FSST, OnPair, ALP and ALP-RD ([11-write-strategy.md](11-write-strategy.md)). +- Pruning structures: a zone map per block of 8 192 rows, file statistics, bounded string bounds, + and on request the skipping and locating indexes of [10-indexes.md](10-indexes.md), which a Rust + reader ignores. +- Appends that continue a file in place, and a repair for a torn one. ## 5. What it answers -A scan reads the columns a record names and nothing else, pushes a filter written as a lambda into -the file statistics, the zone maps and the indexes, and decodes only the blocks that survive. On +A scan reads the columns a record names and nothing else, pushes a filter written as a lambda down +into the file statistics, the zone maps and the indexes, and decodes only the blocks that survive. On top of that: | capability | where | |---|---| | rows by index, decoding only the rows asked for where the encoding allows it | [90-registry.md](90-registry.md) | -| queries in the shape of LINQ — projections, group by, aggregates — on the encoded form, a dictionary by code, a run by its length, flowing as batches as soon as they are final, with sums exact whatever the cut | [16-queries.md](16-queries.md) | -| counts, existence, minimum and maximum answered from the structures before any decode | [12-index-reads.md](12-index-reads.md) §4 | +| queries in the shape of LINQ (projections, group by, aggregates) on the encoded form, a dictionary by its codes and a run by its length, flowing as batches as soon as they are final, with sums exact whatever the split | [16-queries.md](16-queries.md) | +| counts, existence, minimum and maximum answered from the structures before any decode | [answers without rows](12-index-reads.md#4-answers-without-rows) | | a key cursor that seeks, steps, ranks and counts, and batches in key order | [12-index-reads.md](12-index-reads.md) | -| parallel decode on a session's threads, batches still in file order | [09-contracts.md](09-contracts.md) §2 | +| parallel decoding on a session's threads, with batches still in file order | [parallelism](09-contracts.md#2-parallelism) | ## 6. What it leaves out @@ -87,21 +89,21 @@ top of that: | the `tensor`, `spatial` and `json` editions | plugins outside `core` | | encryption | the format's slot is an empty reserved table | | engine integrations (DataFusion, DuckDB, Spark, CUDA) | not a format library's job | -| Apache Arrow interoperability | it would break the zero-dependency rule; an owned `RecordBatch` is where a bridge package would attach | +| Apache Arrow interoperability | it would break the zero-dependency rule. An owned `RecordBatch` is where a bridge package would attach | | a DataFrame, SQL, joins | the plan a scan builds is what such a layer would target | -| deleting or updating rows | a file is immutable and an append adds blocks; mutation belongs to a storage engine above the library | -| an S3 client | the dataset's store is a seam another library implements ([13-dataset.md](13-dataset.md) §11) | +| deleting or updating rows inside a file | a file is immutable and an append only adds blocks. Row deletes and updates live one level up, in the dataset ([deleting and updating rows](13-dataset.md#12-deleting-and-updating-rows)) | +| an S3 client | the dataset's store is an interface another library implements ([the store abstraction](13-dataset.md#11-the-store-abstraction)) | | preserving unknown nodes byte for byte through a rewrite | a separate feature, with its own invariants | ## 7. Guarantees, and what holds each | guarantee | held by | |---|---| -| every corpus file Rust wrote is read value for value | the conformance corpus, 876 files over seven editions ([04-conformance.md](04-conformance.md) §3) | -| Rust reads every file this library writes | the cross-check in CI ([04-conformance.md](04-conformance.md) §4) | -| a scan allocates nothing per batch in steady state | allocation ratchets in the test suite ([05-benchmarks.md](05-benchmarks.md) §5) | -| a full scan within 2× of Rust's time, a decoder within 1.5×, files no larger than 105 % of Rust's | the benchmark gates, and `WrittenSizeTests` for the bytes ([05-benchmarks.md](05-benchmarks.md) §1); where each stands is on [the benchmark page](../guide/benchmarks.md) | -| pruning never removes a row a full scan returns | every filter test run with pruning and indexes on and off ([08-semantics.md](08-semantics.md) §1) | -| malformed input fails with a clean exception, never a crash, a hang or an unbounded allocation | parser fuzzing in CI and the resource caps ([08-semantics.md](08-semantics.md) §6) | -| Native AOT and trimming, no reflection | `vxdump` published ahead of time over the corpus in CI | -| the row encoding is byte-identical to Rust's `vortex-row`, and byte order is tuple order | golden vectors and a randomized order property ([04-conformance.md](04-conformance.md) §7) | +| every corpus file Rust wrote is read value for value | the conformance corpus, 876 files over seven editions ([the golden corpus](04-conformance.md#3-the-golden-corpus)) | +| Rust reads every file this library writes | the cross-check in CI ([the cross-check](04-conformance.md#4-the-cross-check-rust-reads-what-we-write)) | +| a scan allocates nothing per batch in steady state | allocation ratchets in the test suite ([allocations](05-benchmarks.md#5-allocations)) | +| a full scan within 2× of Rust's time, a decoder within 1.5×, files no larger than 105 % of Rust's | the benchmark gates, and `WrittenSizeTests` for the bytes ([the comparison](05-benchmarks.md#1-the-comparison)). Where each stands is on [the benchmark page](../guide/benchmarks.md) | +| pruning never removes a row a full scan returns | every filter test runs with pruning and indexes on and off ([statistic precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)) | +| malformed input fails with a clean exception, never a crash, a hang or an unbounded allocation | parser fuzzing in CI and the resource caps ([resource caps](08-semantics.md#6-resource-caps)) | +| Native AOT and trimming, no reflection | `vxdump` published ahead of time and run over the corpus in CI | +| the row encoding is byte-identical to Rust's `vortex-row`, and byte order is tuple order | golden vectors and a randomized order property ([the row encoding](04-conformance.md#7-the-row-encoding)) | diff --git a/docs/design/02-format.md b/docs/design/02-format.md index beda5adb..c5cfedc8 100644 --- a/docs/design/02-format.md +++ b/docs/design/02-format.md @@ -1,8 +1,8 @@ # The format, condensed The Vortex file format as this library reads and writes it, condensed from the primary sources of -[99-sources.md](99-sources.md); the schemas themselves are vendored in [spec/](../../spec/). -**Everything is little-endian.** +[99-sources.md](99-sources.md). The schemas themselves are vendored in [spec/](../../spec/). +Everything is little-endian. ## 1. File structure @@ -17,11 +17,11 @@ The Vortex file format as this library reads and writes it, condensed from the p ├────────────────────────────┤ │ DType flatbuffer │ optional (may be supplied out of band) ├────────────────────────────┤ -│ Layout flatbuffer │ required — root layout tree +│ Layout flatbuffer │ required: the root layout tree ├────────────────────────────┤ -│ Statistics flatbuffer │ optional — per-field, file-level statistics +│ Statistics flatbuffer │ optional: per-field, file-level statistics ├────────────────────────────┤ -│ Footer flatbuffer │ required — dictionaries (segments, encodings, ...) +│ Footer flatbuffer │ required: dictionaries (segments, encodings, ...) ├────────────────────────────┤ │ Postscript │ ≤ 65527 bytes ├────────────────────────────┤ @@ -29,7 +29,7 @@ The Vortex file format as this library reads and writes it, condensed from the p └────────────────────────────┘ ``` -Frozen constants (these never change): +Frozen constants, which never change: | Constant | Value | |---|---| @@ -40,17 +40,17 @@ Frozen constants (these never change): | `V1_FOOTER_FBS_SIZE` | 32 | | Extension | `.vortex` | -**Open sequence (2 round trips worst case, 1 best case):** +Opening a file takes one round trip at best and two at worst: -1. Read the 64 KiB tail (or the whole file if smaller). By construction this always covers the +1. Read the 64 KiB tail, or the whole file if it is smaller. By construction this always covers the postscript (`MAX_POSTSCRIPT_SIZE` + `EOF_SIZE` ≤ 64 KiB). A local file, whose reads are copies - and cost no round trip, reads its last 8 KiB first; one whose postscript reaches past them reads - the 64 KiB next. -2. Validate magic and version from the last 8 bytes, read the postscript length. -3. Parse the postscript → locate dtype / layout / statistics / footer / metadata. + and cost no round trip, reads its last 8 KiB first, and reads the 64 KiB next only if its + postscript reaches past them. +2. Validate the magic and the version in the last 8 bytes, and read the postscript length. +3. Parse the postscript to locate the dtype, layout, statistics, footer and metadata. 4. If any of those segments falls outside the window already read, issue one targeted second read. -This is exactly why the postscript exists: without it, locating a large DType would cost three +That is exactly why the postscript exists: without it, locating a large DType would cost three round trips. ## 2. Postscript (`footer.fbs`) @@ -70,10 +70,10 @@ table PostscriptSegment { } ``` -Compression and encryption are **inline** in the postscript (rather than indices into the footer) -so a reader does not need the footer before it can decrypt. +Compression and encryption are inline in the postscript, rather than indices into the footer, so a +reader does not need the footer before it can decrypt. -User metadata values are **not** loaded by default: each locator is resolved on demand. +User metadata values are not loaded by default. Each locator is resolved on demand. ## 3. Footer (`footer.fbs`) @@ -100,17 +100,17 @@ table FileStatistics { field_stats: [ArrayStats]; } // one entry per root field ``` `segment_specs` is a vector of fixed-size structs, so it is read as a reinterpreted -`ReadOnlySpan` with no traversal and no allocation, which matters on wide files; so is -`Buffer` (8 bytes) in `array.fbs`. +`ReadOnlySpan` with no traversal and no allocation, which matters on wide files. The +same goes for `Buffer` (8 bytes) in `array.fbs`. ## 4. DType (`dtype.fbs` + `dtype.proto`) -The root DType is a FlatBuffer; the Protobuf variant is used in scalars and statistics. Both -define the same union, with identical and **stable** tags: +The root DType is a FlatBuffer, and the Protobuf variant is used in scalars and statistics. Both +define the same union, with identical and stable tags: | Tag | Type | Fields | |---|---|---| -| 1 | `Null` | — | +| 1 | `Null` | none | | 2 | `Bool` | nullable | | 3 | `Primitive` | ptype, nullable | | 4 | `Decimal` | precision (u8), scale (i8), nullable | @@ -119,18 +119,19 @@ define the same union, with identical and **stable** tags: | 7 | `Struct_` | names[], dtypes[], nullable | | 8 | `List` | element_type, nullable | | 9 | `Extension` | id, storage_dtype, metadata[] | -| 10 | `FixedSizeList` | element_type, size (u32), nullable — *placed after Extension for backward compatibility* | +| 10 | `FixedSizeList` | element_type, size (u32), nullable. Placed after Extension for backward compatibility | | 11 | `Variant` | nullable | | 12 | `Union` | names[], dtypes[], type_ids[], nullable | | 13 | `Map` | key_type, value_type, keys_sorted, nullable | `PType`: `U8=0, U16, U32, U64, I8, I16, I32, I64, F16, F32, F64=10`. -Two parsers therefore share one model, and a property test holds them equivalent: a generated -DType serialized both ways parses back to the same value (`DTypeEquivalenceTests`). +Two parsers therefore share one model, and a property test keeps them equivalent: a generated DType +serialized both ways parses back to the same value (`DTypeEquivalenceTests`). -A file's root DType is **not required to be a Struct**: a file may hold a bare `Float64` or `Bool`, -and a reader must not assume a tabular schema ([07-dotnet-mapping.md](07-dotnet-mapping.md) §5). +A file's root DType does not have to be a struct. A file may hold a bare `Float64` or `Bool`, and a +reader must not assume a tabular schema (see [a root that is not a +struct](07-dotnet-mapping.md#5-a-root-that-is-not-a-struct)). Core-edition extension dtypes: `vortex.date`, `vortex.time`, `vortex.timestamp`, `vortex.uuid`. @@ -142,13 +143,13 @@ Core-edition extension dtypes: `vortex.date`, `vortex.time`, `vortex.timestamp`, [padding] [buffer 0] [padding] [buffer 1] ... [padding] [Array flatbuffer] [u32 flatbuffer length] ``` -Reading: read the last 4 bytes → `fb_length`; the FlatBuffer occupies -`[len-4-fb_length, len-4)`; everything before it is the buffer region. Each buffer's offset is +To read one, take the last 4 bytes as `fb_length`. The FlatBuffer occupies +`[len-4-fb_length, len-4)`, and everything before it is the buffer region. Each buffer's offset is reconstructed by accumulating `padding + length` from 0. The FlatBuffer is 8-byte aligned. Each buffer carries its own required alignment (up to 16 for -`varbinview` views), and the written padding guarantees that a memory-mapped segment is directly -usable — **this is what makes zero-copy possible**. +`varbinview` views), and the written padding guarantees that a memory-mapped segment is usable as +is. This is what makes zero-copy reads possible. ### 5.2 Schema (`array.fbs`) @@ -180,14 +181,14 @@ table ArrayStats { enum Precision : uint8 { Inexact = 0, Exact = 1 } ``` -**Key point:** an `ArrayNode` does **not** carry its DType. The type is inherited or derived from -the parent (the root DType comes from the file). Decoding is therefore a type-guided top-down +The key point is that an `ArrayNode` does not carry its DType. The type is inherited or derived from +the parent, and the root DType comes from the file. Decoding is therefore a type-guided top-down traversal, and every decoder must know how to compute its children's DTypes. ### 5.3 Per-encoding metadata -Each encoding serializes its metadata as **Protobuf** (`prost`) with stable tags. There is no -single `.proto` file upstream: messages are declared beside each encoding. A few examples: +Each encoding serializes its metadata as Protobuf (`prost`) with stable tags. There is no single +`.proto` file upstream: messages are declared next to each encoding. A few examples: ```proto // vortex.dict @@ -215,52 +216,52 @@ message PatchesMetadata { uint64 len = 1; uint64 offset = 2; PType indices_ptype ``` `vortex.primitive`, `vortex.varbinview`, `vortex.struct`, `vortex.chunked`, `vortex.null`, -`vortex.masked`, `vortex.fixed_size_list`, `vortex.ext`, `vortex.bytebool`, `vortex.zigzag` and -`fastlanes.for`: empty metadata. +`vortex.masked`, `vortex.fixed_size_list`, `vortex.ext`, `vortex.bytebool` and `vortex.zigzag` have +empty metadata. -Two traps in that neighbourhood, in opposite directions. `vortex.constant` **is** empty — but its -scalar is not absent, it lives in **buffer 0** as a Protobuf `ScalarValue`, so a decoder that reads -the metadata and stops finds nothing. `fastlanes.for` is **not** empty: its metadata is a bare +Two traps sit nearby, in opposite directions. `vortex.constant` has empty metadata, but its scalar +is not absent: it lives in buffer 0 as a Protobuf `ScalarValue`, so a decoder that reads the +metadata and stops finds nothing. `fastlanes.for` does not have empty metadata: it is a bare `ScalarValue` holding the frame-of-reference value, with zero buffers and one child. And `vortex.zoned`'s layout metadata is a version byte followed by a protobuf, not a bare message. -The complete transcription of every metadata message, with the upstream source path of each, is +The complete transcription of every metadata message, with the upstream source path of each, is in [spec/METADATA.md](../../spec/METADATA.md). Its tag numbers are the contract. #### Unknown Protobuf fields: skip, never reject -The Protobuf runtime **skips fields with unknown numbers**, dispatching on the wire type (varint, -fixed32, fixed64, length-delimited; groups, wire types 3 and 4, are rejected, as proto3 never emits -them). +The Protobuf runtime skips fields with unknown numbers, dispatching on the wire type (varint, +fixed32, fixed64, length-delimited). Groups, wire types 3 and 4, are rejected, since proto3 never +emits them. -This protects the read-forever promise. Upstream may add an *optional* field to a metadata message -without it being a reader-visible evolution: old readers ignore it and the semantics are unchanged, -so upstream need not mint a new id. A parser that rejected unknown fields would fail on legal files, +This protects the read-forever promise. Upstream may add an optional field to a metadata message +without it being a reader-visible change: old readers ignore it and the semantics are unchanged, so +upstream does not need a new id. A parser that rejected unknown fields would fail on legal files, and a corpus pinned to one version would not notice until its next regeneration. The distinction to keep sharp: -* **Reject** = a value outside the contract's domain (a forbidden ptype, `lower_part_count != 0`, - an out-of-range exponent). -* **Tolerate** = a field number the wire format does not recognize. +* Reject a value outside the contract's domain (a forbidden ptype, `lower_part_count != 0`, an + out-of-range exponent). +* Tolerate a field number the wire format does not recognize. -**An encoding id is a frozen contract.** A reader-visible evolution gets a **new id** -(`vortex.foo` → `vortex.foo_v2`), never a silent extension. A decoder therefore rejects a payload -outside the contract of the id it read, even when it could decode the successor's: `vortex.pco` -refuses an 8-bit dtype, which belongs to `vortex.pco.v2`. +An encoding id is a frozen contract. A reader-visible change gets a new id (`vortex.foo` becomes +`vortex.foo_v2`), never a silent extension. A decoder therefore rejects a payload outside the +contract of the id it read, even when it could decode the successor's: `vortex.pco` refuses an 8-bit +dtype, which belongs to `vortex.pco.v2`. ### 5.4 Validity (nullability) -Validity is not a mandatory bitmap. Four states: `NonNullable`, `AllValid`, `AllInvalid`, or a -**boolean array** (itself possibly encoded: constant, dict, runend…). `true` = valid. The validity -array's DType is always non-nullable `Bool`. +Validity is not a mandatory bitmap. It has four states: `NonNullable`, `AllValid`, `AllInvalid`, or +a boolean array, which may itself be encoded (constant, dict, runend…). `true` means valid. The +validity array's DType is always a non-nullable `Bool`. ## 6. Layouts (`layout.fbs`) ```fbs table Layout { encoding: uint16; // index into Footer.layout_specs - row_count: uint64; // rows represented — the basis for pruning + row_count: uint64; // rows represented, the basis for pruning metadata: [ubyte]; // opaque, layout-specific children: [Layout]; segments: [uint32]; // indices into Footer.segment_specs @@ -269,55 +270,55 @@ table Layout { | ID | Role | Children | Segments | Metadata | |---|---|---|---|---| -| `vortex.flat` | one serialized array | 0 | exactly 1 | `optional bytes array_encoding_tree` — when present the array FlatBuffer is inlined in the layout and the segment holds only buffers | -| `vortex.chunked` | row-wise partition | ≥ 1 | 0 | empty (chunk offsets derive from children `row_count`s, whose sum must equal the parent's) | -| `vortex.struct` | one child per field (+ validity) | n fields | 0 | empty | -| `vortex.dict` | dictionary shared with a child | codes + values | — | `PType codes_ptype = 1; optional bool is_nullable_codes = 2; optional bool all_values_referenced = 3;` | -| `vortex.zoned` | statistics zone map for pruning | data + stats table | — | `[u8 version = 1] ++ proto { uint32 zone_len = 1; repeated AggregateSpec aggregate_specs = 2; }` | -| `vortex.stats` | legacy ancestor of `zoned` | same | — | must stay readable (read by the same machinery as `zoned`) | +| `vortex.flat` | one serialized array | 0 | exactly 1 | `optional bytes array_encoding_tree`. When present, the array FlatBuffer is inlined in the layout and the segment holds only buffers | +| `vortex.chunked` | row-wise partition | ≥ 1 | 0 | empty. Chunk offsets derive from the children's `row_count`s, whose sum must equal the parent's | +| `vortex.struct` | one child per field (plus validity) | n fields | 0 | empty | +| `vortex.dict` | dictionary shared with a child | codes + values | | `PType codes_ptype = 1; optional bool is_nullable_codes = 2; optional bool all_values_referenced = 3;` | +| `vortex.zoned` | statistics zone map for pruning | data + stats table | | `[u8 version = 1] ++ proto { uint32 zone_len = 1; repeated AggregateSpec aggregate_specs = 2; }` | +| `vortex.stats` | legacy ancestor of `zoned` | same | | must stay readable, and is read by the same machinery as `zoned` | Aggregates allowed in zone maps (core edition): `vortex.min`, `vortex.max`, `vortex.bounded_min`, -`vortex.bounded_max`, `vortex.nan_count`, `vortex.null_count`. -An unknown aggregate must **not** invalidate the file: it disables the affected pruning only. +`vortex.bounded_max`, `vortex.nan_count`, `vortex.null_count`. An unknown aggregate must not +invalidate the file. It only disables the pruning that depends on it. ## 7. What the Rust writer produces -What a default Rust writer emits is what most files contain. Its pipeline is **split structs → -repartition into 8 192-row blocks → zone maps → dictionary where useful → coalesce toward ~1 MiB → -BtrBlocks compression → flat leaves**. +What a default Rust writer emits is what most files contain. Its pipeline splits structs, +repartitions into 8 192-row blocks, adds zone maps, applies a dictionary where useful, coalesces +toward about 1 MiB, compresses with BtrBlocks and writes flat leaves. -Compression schemes enabled by default, in this order (order breaks ties): -FoR, ZigZag, BitPacking, Sparse, IntDict, RunEnd, Sequence, IntRLE, Delta; ALP, ALPrd, FloatDict, -NullDominatedSparse, FloatRLE; StringDict, FSST, OnPair; BinaryDict, VarBin; Decimal; Temporal. +Compression schemes enabled by default, in this order (the order breaks ties): FoR, ZigZag, +BitPacking, Sparse, IntDict, RunEnd, Sequence, IntRLE, Delta, ALP, ALPrd, FloatDict, +NullDominatedSparse, FloatRLE, StringDict, FSST, OnPair, BinaryDict, VarBin, Decimal, Temporal. Two caveats: -* **Zstd and Pco are not enabled by default** (`with_compact()`, the `zstd` and `pco` features), so - ordinary files need neither. This library reads both, every type, mode and delta pco has, and - writes both: zstd costs it no third-party dependency, since its frames go through - `Vorticity.Zstd`, this repository's managed Zstandard, which writes libzstd's frames byte for - byte, and pco is ported, its encoder tried by the size-first profile as `with_compact()` tries - it. -* **`fastlanes.delta` belongs to no edition.** The Delta scheme is in the list, but the edition - allowlist forbids its id, so it appears only in files written with enforcement turned off. +* Zstd and Pco are not enabled by default (they need `with_compact()` and the `zstd` and `pco` + features), so ordinary files need neither. This library reads both, including every type, mode and + delta pco has, and writes both. Zstd costs it no third-party dependency, since its frames go + through `Vorticity.Zstd`, this repository's managed Zstandard, which writes libzstd's frames byte + for byte. Pco is ported, and its encoder is tried by the size-first profile the way + `with_compact()` tries it. +* `fastlanes.delta` belongs to no edition. The Delta scheme is in the list, but the edition allowlist + forbids its id, so it only appears in files written with enforcement turned off. ## 8. Algorithmic details that are easy to get wrong -* **FastLanes**: 1024-element **transposed** blocks. Bit-packing operates in that order, not - logical order — which is what allows unpacking without cross-lane shuffles, and any access by - logical index must go through the inverse transposition. There are **two** permutations, not one, - and they are easy to conflate: the bit-packing index - `FL_ORDER[row / 8] * 16 + (row % 8) * 128 + lane` (lane over `1024 / bitwidth(T)` lanes), and the - element transposition `lane * 64 + FL_ORDER[order] * 8 + row` (lane always `idx % 16`). Only - `FL_ORDER = [0,4,2,6,1,5,3,7]` is its own inverse; the transposition itself is not, so - `untranspose` is the inverse mapping and not a second `transpose`. Both derivations, their - inverses, and the known-value table to test against are in +* FastLanes uses 1024-element transposed blocks. Bit-packing operates in that order, not in logical + order, which is what allows unpacking without cross-lane shuffles, and any access by logical index + must go through the inverse transposition. There are two permutations, not one, and they are easy + to conflate: the bit-packing index `FL_ORDER[row / 8] * 16 + (row % 8) * 128 + lane` (lane over + `1024 / bitwidth(T)` lanes), and the element transposition `lane * 64 + FL_ORDER[order] * 8 + row` + (lane always `idx % 16`). Only `FL_ORDER = [0,4,2,6,1,5,3,7]` is its own inverse. The transposition + is not, so `untranspose` is the inverse mapping and not a second `transpose`. Both derivations, + their inverses and the known-value table to test against are in [spec/REFERENCE.md](../../spec/REFERENCE.md). -* **FSST**: 255-entry symbol table; code 255 is the escape code. Symbol buffers are padded to that - fixed size. -* **ALP**: the encoded integer is `i32` or `i64`; `exp_e`/`exp_f` must be validated in range for - the target float type. Patches in a modern ALP are read as a `Patched` array wrapping a +* FSST uses a 255-entry symbol table, and code 255 is the escape code. Symbol buffers are padded to + that fixed size. +* In ALP the encoded integer is an `i32` or an `i64`, and `exp_e` and `exp_f` must be validated in + range for the target float type. Patches in a modern ALP are read as a `Patched` array wrapping a patch-free ALP. -* **VarBinView**: 16-byte views (Arrow StringView compatible), `views` buffer aligned to 16. -* **Patches**: one shared structure (indices, values and optional chunk offsets) that BitPacked, - ALP and Sparse all use, implemented once here. +* VarBinView uses 16-byte views (compatible with Arrow's StringView), with the `views` buffer aligned + to 16. +* Patches are one shared structure (indices, values and optional chunk offsets) that BitPacked, ALP + and Sparse all use, implemented once here. diff --git a/docs/design/03-architecture.md b/docs/design/03-architecture.md index 583b1b75..1daf9ad7 100644 --- a/docs/design/03-architecture.md +++ b/docs/design/03-architecture.md @@ -1,254 +1,259 @@ # .NET architecture -The engine under the public surface: the constraints it is built to, where things are, the objects a -scan and a write run on, the I/O seam, and the rules that keep it fast and safe. The surface itself -is [14-public-api.md](14-public-api.md); what each column type becomes in .NET is -[07-dotnet-mapping.md](07-dotnet-mapping.md). +The engine under the public API: the constraints it is built to, where things are, the objects a +scan and a write run on, the I/O interface, and the rules that keep it fast and safe. The public API +itself is described in [14-public-api.md](14-public-api.md), and what each column type becomes in +.NET in [07-dotnet-mapping.md](07-dotnet-mapping.md). ## 1. Founding constraints | constraint | consequence | |---|---| -| **No third-party dependency** | FlatBuffers and Protobuf are read and written by runtimes of this repository, hand-written against the five schemas; no `Google.FlatBuffers`, `protobuf-net` or `Apache.Arrow`. Two packages are referenced: `System.IO.Hashing` from `dotnet/runtime`, for XXH3, which the shared framework does not carry, and `Vorticity.Zstd`, this repository's managed Zstandard, rather than the libzstd the runtime ships | -| **No allocation on the hot path** | no LINQ, no `foreach` over interfaces, no capturing closures, no boxing, no `params` array (only `params ReadOnlySpan`); aligned native buffers, pools for transients, `ref struct` views | -| **SIMD with a scalar twin** | `System.Runtime.Intrinsics`: `Vector128` everywhere, `Vector256` and `Vector512` where the hardware has them, and a scalar path that the suite runs under `DOTNET_EnableHWIntrinsic=0` and compares bit for bit | -| **Async only, where there is I/O** | `ValueTask` and `ConfigureAwait(false)` throughout; no blocking public call and no synchronous twin of an asynchronous one. A source whose reads complete at once, a mapped file or bytes in memory, pays for a completed `ValueTask`, not for a second path. Pure CPU work stays synchronous: filling a builder, row encoding, decode kernels, a merge or a sort cut into parts the caller awaits. A builder never reads | -| **Streaming, bounded** | every source, operator and sink exchanges batches, pulled by its consumer: the async boundary is the batch, and every loop inside one is synchronous. Read-ahead is a bounded window, an operator holds its open state and never its input, a result is a stream of batches like a file's, and a consumer that stops stops the reads. What a query costs before its first answer, and holds while it runs, is measured ([16-queries.md](16-queries.md) §2) | - -**One target framework, `net11.0`.** A single target means no `#if`, no API surface that differs by -target, no test matrix, and no kernel that could diverge between legs. The price is reach: +| no third-party dependency | FlatBuffers and Protobuf are read and written by runtimes in this repository, hand-written against the five schemas. There is no `Google.FlatBuffers`, `protobuf-net` or `Apache.Arrow`. Two packages are referenced: `System.IO.Hashing` from `dotnet/runtime`, for XXH3, which the shared framework does not carry, and `Vorticity.Zstd`, this repository's managed Zstandard, rather than the libzstd the runtime ships | +| no allocation on the hot path | no LINQ, no `foreach` over interfaces, no capturing closures, no boxing, no `params` array (only `params ReadOnlySpan`). Aligned native buffers, pools for transient memory, `ref struct` views | +| SIMD with a scalar twin | `System.Runtime.Intrinsics`: `Vector128` everywhere, `Vector256` and `Vector512` where the hardware has them, and a scalar path that the suite runs under `DOTNET_EnableHWIntrinsic=0` and compares bit for bit | +| async only where there is I/O | `ValueTask` and `ConfigureAwait(false)` throughout, with no blocking public call and no synchronous twin of an asynchronous one. A source whose reads complete at once (a mapped file, bytes in memory) pays for a completed `ValueTask`, not for a second code path. Pure CPU work stays synchronous: filling a builder, row encoding, decode kernels, a merge or a sort split into parts the caller awaits. A builder never reads | +| streaming, bounded | every source, operator and sink exchanges batches, pulled by its consumer. The async boundary is the batch, and every loop inside one is synchronous. Read-ahead is a bounded window, an operator holds its open state and never its input, a result is a stream of batches like a file's, and a consumer that stops also stops the reads. What a query costs before its first answer, and what it holds while it runs, is measured ([the flow](16-queries.md#2-the-flow)) | + +The library has one target framework, `net11.0`. A single target means no `#if`, no API that differs +by target, no test matrix, and no kernel that could diverge between targets. The price is reach: .NET 11 is not a long-term-support release, and a consumer pinned to .NET 10 cannot reference the -library. `netstandard2.1` is out of the question: it lacks `Vector512`, static abstract members, -`INumber`, `SearchValues` and aligned native allocation, which is to say the performance model. -`global.json` pins the SDK, so that a build never changes compiler silently under a library whose -code generation is measured. +library. `netstandard2.1` is out of the question, because it lacks `Vector512`, static abstract +members, `INumber`, `SearchValues` and aligned native allocation, which is to say the whole +performance model. `global.json` pins the SDK, so a build never silently changes compiler under a +library whose code generation is measured. The library enables `AllowUnsafeBlocks` and `IsAotCompatible`. It does not set -`InvariantGlobalization`, an application property with no effect on a library; it avoids +`InvariantGlobalization`, an application property with no effect on a library, and avoids culture-sensitive APIs instead. A module initializer asserts that the process is little-endian and that the inline FlatBuffers structs have their wire sizes, since the zero-copy design reinterprets file bytes in place. -**Cancellation** is taken where it is actionable: an open, a segment read, a scan's enumeration, +Cancellation is accepted wherever it is actionable: an open, a segment read, a scan's enumeration, every answer a scan computes, every write. It is not threaded through decode kernels, which are -short and CPU-bound: the granularity is the batch. +short and CPU-bound. The granularity is the batch. ## 2. Where things are | `src/Vorticity/` | | |---|---| -| `Buffers`, `Serialization` | aligned buffers and leases; the FlatBuffers and Protobuf runtimes and the schema views | -| `Types`, `Schema`, `Editions` | the dtype model and its two serializations; the public schema; the edition registry | -| `File`, `IO`, `Layouts` | the open path, the footer and the file's statistics; the segment seam and its sources; the layout readers | -| `Arrays`, `Compute`, `Expressions` | the canonical arena and one decoder per encoding; kernels, masks, pruning; the filter expressions | -| `Scan`, `Columns`, `Symbolic`, `Records` | the scan and its plan; the borrowed and owned batches; the typed filter algebra; record binding | -| `Aggregation`, `Keys`, `Indexes` | aggregates and group by on encoded forms; the key cursor and its sources; index directory, pruners, policies | -| `Writing`, `Advice` | the writer, its builders and chooser, the index builders; the encoding advisor | -| `Session`, `Diagnostics` | the session, its pool and cache; exceptions, limits, the meter and the event source | - -Beside it: `src/Vorticity.Generators`, `.Dataset` and `.RowEncoding`; `tests/` (unit and property -tests, the conformance corpus, the fuzzer, and code that must not compile); `bench/` -([bench/README.md](../../bench/README.md)); `tools/` (`vxdump`, the Rust corpus generator, the -Rust benchmark shim, the row-encoding vectors, the test-impact selector); `samples/`, the programs -the guide's pages run. +| `Buffers`, `Serialization` | aligned buffers and leases, the FlatBuffers and Protobuf runtimes and the schema views | +| `Types`, `Schema`, `Editions` | the dtype model and its two serializations, the public schema, the edition registry | +| `File`, `IO`, `Layouts` | the open path, the footer and the file's statistics, the segment interface and its sources, the layout readers | +| `Arrays`, `Compute`, `Expressions` | the canonical arena and one decoder per encoding, kernels, masks and pruning, the filter expressions | +| `Scan`, `Columns`, `Symbolic`, `Records` | the scan and its plan, the borrowed and owned batches, the typed filter algebra, record binding | +| `Aggregation`, `Keys`, `Indexes` | aggregates and group by on encoded forms, the key cursor and its sources, the index directory, pruners and policies | +| `Writing`, `Advice` | the writer, its builders and chooser, the index builders, the encoding advisor | +| `Session`, `Diagnostics` | the session, its pool and cache, the exceptions, limits, meter, activity source and event source | + +Alongside it sit `src/Vorticity.Generators`, `.Dataset`, `.RowEncoding` and `.Zstd`, then `tests/` +(unit and property tests, the conformance corpus, the fuzzer, and code that must not compile), +`bench/` ([bench/README.md](../../bench/README.md)), `tools/` (`vxdump`, the Rust corpus generator, +the Rust benchmark shim, the row-encoding vectors, the test-impact selector), and `samples/`, the +programs the guide's pages run. ## 3. Object model -None of this is public. The arenas are built once at open and at `CreateWriter` and never appear; -[14-public-api.md](14-public-api.md) builds the surface over them. +None of this is public. The arenas are built once, at open and at `CreateWriter`, and never appear +in the API. [14-public-api.md](14-public-api.md) builds the public API on top of them. ### 3.1 Buffers A segment is a view over aligned bytes, whatever their origin: a mapped file (zero-copy), pooled -native memory (a positional or remote read), or managed memory the caller already holds. A -`SegmentLease` owns the bytes and releases them when the batch that borrowed them is done. +native memory (a positional or remote read), or memory the caller already holds. A `SegmentLease` +owns the bytes and releases them when the batch that borrowed them is done. ### 3.2 Types A dtype is a `readonly struct` with a tag and a payload, its children in a shared array, rather than -a class hierarchy: a wide schema must not become thousands of objects. Equality and hashing are +a class hierarchy, so a wide schema does not become thousands of objects. Equality and hashing are structural and allocate nothing. ### 3.3 Arrays: an arena, not an object graph A batch of fifty columns, each a tree of three to six encodings rebuilt per chunk, would allocate -hundreds of objects per batch as an object graph. So nodes are structs in a pooled array owned by -the batch (`CanonicalArena`), addressed by index; `CanonicalNode` is a `ref struct` view of one. -This mirrors the format, where `ArrayNode.children` and `buffers` are indices already. The blocks a -batch decodes into are kept for the next batch of the scan, which writes into memory still in the -cache without renting it again, and go back to the pool when the scan ends. The parse of a flat -layout's array blob is kept the same way, in the contexts the scans pool, for the next batch that -holds the same segment bytes. - -A file's encoding ids are resolved **once, at open**, against the registry, into an array indexed -by the file's own `u16`: on the hot path, dispatch is an index, not a string lookup. Kernels are +hundreds of objects per batch as an object graph. Nodes are therefore structs in a pooled array owned +by the batch (`CanonicalArena`), addressed by index, and `CanonicalNode` is a `ref struct` view of +one. This mirrors the format, where `ArrayNode.children` and `buffers` are already indices. The +blocks a batch decodes into are kept for the scan's next batch, which writes into memory that is +still in the CPU cache without renting it again, and go back to the pool when the scan ends. The +parse of a flat layout's array blob is kept the same way, in the contexts the scans pool, for the +next batch that holds the same segment bytes. + +A file's encoding ids are resolved once, at open, against the registry, into an array indexed by +the file's own `u16`, so on the hot path dispatch is an index, not a string lookup. Kernels are generic over the physical type (`where T : unmanaged, INumberBase`) and specialized per type by -the compiler: no `switch` on a type inside a loop, and no virtual call per value. +the compiler. There is no `switch` on a type inside a loop and no virtual call per value. -A decoder canonicalizes a node, and may instead decode only a selection of its rows -(`DecodeSelected`), which is how a take avoids decoding a chunk it needs one row of -([90-registry.md](90-registry.md)). +A decoder canonicalizes a node, or decodes only a selection of its rows (`DecodeSelected`), which is +how a take avoids decoding a whole chunk to get one row of it ([90-registry.md](90-registry.md)). -### 3.4 What the surface sees of it +### 3.4 What the public API sees of it -A scan's batch is borrowed: `Columns`, `Column` and `BatchView` are `ref struct`s over -the scan's arenas, valid until the next `MoveNextAsync`, and the compiler, not a `Dispose`, holds -that lifetime. A batch that must outlive its successor is an owned `RecordBatch`, one copy into -pooled buffers, which the caller disposes. +A scan's batch is borrowed. `Columns`, `Column` and `BatchView` are `ref struct`s over +the scan's arenas, valid until the next `MoveNextAsync`, and the compiler, not a `Dispose`, enforces +that lifetime. A batch that must outlive the next one is an owned `RecordBatch`, one copy into pooled +buffers, which the caller disposes. ### 3.5 I/O -The public seam is `Vorticity.IO.ISegmentSource` (`IO/SegmentSeam.cs`): a length, a single read, -and a batch read of many ranges into many leases. The batch read is the real entry point: a scan -registers every segment a split needs before reading, which lets a source **coalesce** nearby -ranges (1 MiB gap by default) and issue them together, and a scan requests each segment at most -once, every block of it decoding from the one lease. That is what makes or breaks performance on -an object store. - -What an implementer of the seam must honour, since object storage lives outside the core: - -- **All or nothing**: a batch read that fails releases every lease it acquired and throws; retry - policy is the source's. -- **Ownership**: a lease holds its bytes, one block or the pieces a response arrived in, until the - reader disposes it; a caching source therefore reference-counts. -- **Thread safety**: a source is shared by the scans of a session. -- **Cancellation** leaves a source's cache consistent: a segment is present whole or absent. - -The session adds one segment cache and one bound on reads in flight (`MaxConcurrentReads`) for every -file it opens, for sources that do I/O; a mapped file and bytes in memory bypass both. Over a source -that does I/O, a file also serves every segment lying in the tail its open read, which is then -neither asked of the source nor cached: the open reads that tail from a 64-byte boundary, so that a -segment inside keeps the alignment it declares, and a file shorter than the tail is read once. A test-only -HTTP source with injectable latency (`tests/Vorticity.Tests/IO/HttpRangeSegmentSource.cs`) is -the proof that the seam holds for a remote store. - -Three sources ship: `MemoryMappedSegmentSource` (zero-copy), `FileSegmentSource` (positional reads -into aligned buffers) and `MemorySegmentSource` (bytes the caller holds, never copied). A file -opened **from a path** reads positionally until a scan's plan says it will read data, and is mapped -from then on: - -- the open reads the last 8 KiB positionally, or the whole file up to 64 KiB, and a larger footer - whole in a second read: mapping a whole file to read its tail costs more than one read. A file - the open read whole is never mapped: its scans are served from that read; -- the file is opened, locked as `FileShare.Read` locks it, mapped and read with the system's calls - made directly where the platform has them (`NativeFile`): on macOS a third fewer calls than the - framework makes for an open, a scan and a close; -- the first scan whose plan reads any data maps the file, once, and the mapping is the file's until - it is disposed and the last lease on it released. A held mapping reads two to three times faster - than a positional read at every size, and a mapped scan allocates nothing per batch, so no - threshold on the volume is worth its complexity; -- two scans that race to map publish one mapping by compare-and-exchange; a read already under way - finishes positionally; -- the session keeps the mapping once the file is closed (`MappedFileCacheCount`, 64 files by - default): the next open of the same file, known by its device and inode, takes it over with every - page already mapped in it, so a file scanned again pays neither the mapping nor a page fault per - page. A file whose length changed is mapped again, one replaced under its name leaves the cache at - once, and the mapping holds neither the handle nor its lock, so a writer is never kept out. A - file deleted once closed keeps its disk space while it is kept, and `ReleaseMappedFiles` lets - every kept mapping go. On Windows, where a mapped file can be neither deleted nor replaced, - nothing is kept; -- a session created with `MapFiles = false` maps nothing: a path is read by a `FileSegmentSource` - under the session's bound on reads and segment cache, as any source that does I/O. On Linux and - macOS a mapped file cut short by another process faults on the pages past its new end, which - kills the process; a positional read of it throws `VortexFormatException` instead. - -**Coalescing against alignment.** A buffer for a coalesced range `[start, end)` places a segment at -file offset `o` at memory offset `o − start`, arbitrary for an arbitrary `start`. So the start is -rounded down to 64 and the buffer allocated 64-aligned: every segment's offset is a multiple of its -own alignment, at most 64 ([08-semantics.md](08-semantics.md) §6), and so is its distance from the -rounded start. +The public interface is `Vorticity.IO.ISegmentSource` (`IO/SegmentSeam.cs`): a length, a single +read, and a batch read of many ranges into many leases. The batch read is the real entry point. A +scan registers every segment a split needs before reading, which lets a source coalesce nearby +ranges (with a 1 MiB gap by default) and issue them together, and a scan requests each segment at +most once, every block in it decoding from the one lease. That is what makes or breaks performance +on an object store. + +Since object storage lives outside the core, an implementer of the interface must honour four rules: + +- All or nothing: a batch read that fails releases every lease it acquired and throws. Retry policy + belongs to the source. +- Ownership: a lease holds its bytes (one block, or the pieces a response arrived in) until the + reader disposes it, so a caching source has to reference-count. +- Thread safety: a source is shared by all the scans of a session. +- Cancellation leaves a source's cache consistent: a segment is either present whole or absent. + +For sources that do I/O, the session adds one segment cache and one bound on reads in flight +(`MaxConcurrentReads`) shared by every file it opens. A mapped file and bytes in memory bypass both. +Over a source that does I/O, a file also serves every segment lying in the tail its open read, and +those segments are neither requested from the source nor cached. The open reads that tail from a +64-byte boundary, so a segment inside keeps the alignment it declares, and a file shorter than the +tail is read once. A test-only HTTP source with injectable latency +(`tests/Vorticity.Tests/IO/HttpRangeSegmentSource.cs`) proves the interface holds for a remote store. + +Three sources ship. `MemoryMappedSegmentSource` is zero-copy. `FileSegmentSource` does positional +reads into aligned buffers. `MemorySegmentSource` serves bytes the caller holds, and uses them in +place when they start on a 64-byte boundary, otherwise copying them once, when the source is made, +into aligned memory it owns. A file opened from a path reads positionally until a scan's plan says it +will read data, and is mapped from then on: + +- The open reads the last 8 KiB positionally, or the whole file up to 64 KiB, and a larger footer + whole in a second read, because mapping a whole file just to read its tail costs more than one + read. A file the open read whole is never mapped, and its scans are served from that read. +- The file is opened, locked the way `FileShare.Read` locks it, mapped and read with the system calls + made directly where the platform has them (`NativeFile`). On macOS that is a third fewer calls than + the framework makes for an open, a scan and a close. +- The first scan whose plan reads any data maps the file, once, and the mapping belongs to the file + until it is disposed and the last lease on it is released. A held mapping reads two to three times + faster than a positional read at every size, and a mapped scan allocates nothing per batch, so no + threshold on volume is worth its complexity. +- Two scans that race to map publish a single mapping by compare-and-exchange, and a read already + under way finishes positionally. +- The session keeps the mapping after the file is closed (`MappedFileCacheCount`, 64 files by + default). The next open of the same file, recognized by its device and inode (on Windows by its + volume and file id), takes it over with every page already mapped, so a file scanned again pays + neither the mapping nor a page fault per page. A file whose length changed is mapped again, one + replaced under its name leaves the cache at once, and the mapping holds neither the handle nor its + lock, so a writer is never kept out. A file deleted after closing keeps its disk space as long as + its mapping is kept, and `ReleaseMappedFiles` lets every kept mapping go. On Windows a kept file + can be deleted and replaced under its name, but another writer cannot truncate or overwrite it + from its start until the session lets it go, and this library's own writers release it themselves. +- A session created with `MapFiles = false` maps nothing. A path is then read by a + `FileSegmentSource` under the session's read bound and segment cache, like any source that does + I/O. On Linux and macOS, a mapped file truncated by another process faults on the pages past its + new end, which kills the process, whereas a positional read of it throws + `VortexFormatException`. + +Coalescing has to respect alignment. A buffer for a coalesced range `[start, end)` places a segment +at file offset `o` at memory offset `o − start`, which is arbitrary for an arbitrary `start`. So the +start is rounded down to 64 and the buffer allocated 64-aligned. Every segment's offset is a +multiple of its own alignment, at most 64 ([resource caps](08-semantics.md#6-resource-caps)), and so +is its distance from the rounded start. ### 3.6 Two phases: materialize, then execute -Reading a split is two calls: an asynchronous one that reads every segment the split needs, -coalesced, and a synchronous one that decodes. A decoder therefore never awaits: threading -`ValueTask` through every decoder would put state machines on the hot path to serve one case, the -lazily loaded values of a dictionary layout, which belongs to the layout level, where the +Reading a split takes two calls: an asynchronous one that reads every segment the split needs, +coalesced, and a synchronous one that decodes. A decoder therefore never awaits. Threading +`ValueTask` through every decoder would put state machines on the hot path to serve a single case, +the lazily loaded values of a dictionary layout, and that case belongs at the layout level, where the materialization decides what to fetch. ### 3.7 Async enumeration -A compiler-generated `async` enumerator allocates its state machine and possibly more per -`MoveNextAsync`. One allocation per scan is acceptable; one per batch is not. The scan's enumerator +A compiler-generated `async` enumerator allocates its state machine, and possibly more, per +`MoveNextAsync`. One allocation per scan is acceptable, one per batch is not. The scan's enumerator is therefore written by hand over `ManualResetValueTaskSourceCore`, the pattern -`System.Threading.Channels` uses, and so is every enumerator of a query's flow: an operator's -stream of batches, a result's, the scan over a result, a dataset's walk of its objects. A view of -rows over those batches — values, records — steps synchronously within a batch and awaits only at -the next one: an `await` per batch, never per row, which is also what lets a batch be borrowed, a -span never crossing an `await`. +`System.Threading.Channels` uses, and so is every enumerator of a query's flow: an operator's stream +of batches, a result's, the scan over a result, a dataset's walk of its objects. A view of rows over +those batches (values, records) steps synchronously within a batch and only awaits at the next one. +That means one `await` per batch, never per row, which is also what allows a batch to be borrowed, +since a span never crosses an `await`. ### 3.8 The write sink -The writer's seam is a `PipeWriter`: `CreateWriter(path)` builds one over a file handle, +The writer writes to a `PipeWriter`. `CreateWriter(path)` builds one over a file handle, `PipeWriter.Create(stream)` covers a stream, and an object store provides its own. The writer writes -**forward only, in one pass**: segments, then the metadata FlatBuffers, the postscript and the end -record, the order the format is designed for. Padding is computed as each segment is written, so -no file is ever materialized whole; bytes leave the pipe at `FlushAsync` and `CompleteAsync`, so a -sink that applies backpressure slows the producer instead of growing a buffer. A writer that gives -a file up completes the caller's pipe with an error, so that whatever it feeds knows the bytes are -not a file. The postscript's 65 527-byte ceiling is checked when user metadata is given, not at the -last write. +forward only, in one pass: segments, then the metadata FlatBuffers, the postscript and the end +record, the order the format is designed for. Padding is computed as each segment is written, so no +file is ever materialized whole. Bytes leave the pipe at `FlushAsync` and `CompleteAsync`, so a sink +that applies backpressure slows the producer instead of growing a buffer. A writer that gives up on a +file completes the caller's pipe with an error, so whatever the pipe feeds knows the bytes are not a +file. The postscript's 65 527-byte ceiling is checked when user metadata is given, not at the last +write. ## 4. Performance invariants -Each is a test, not a guideline: +Each one is a test, not a guideline: -1. **No managed allocation per batch** in steady state on a full scan (`ScanAllocationTests`), and - a fixed ceiling per operation and per written file ([05-benchmarks.md](05-benchmarks.md) §5). -2. **Zero copy**: a plain column's decode is a view of the segment's bytes +1. No managed allocation per batch in steady state on a full scan (`ScanAllocationTests`), and a + fixed ceiling per operation and per written file ([allocations](05-benchmarks.md#5-allocations)). +2. Zero copy: a plain column's decode is a view of the segment's bytes (`LeafDecoderTests.PrimitiveReadsValuesZeroCopy`). -3. **No per-row dispatch** in decode and write loops: every kernel is generic and specialized +3. No per-row dispatch in decode and write loops: every kernel is generic and specialized (`PerRowDispatchTests`). -4. **Scalar fallback**: CI runs again with hardware intrinsics disabled the test classes that reach - every scalar fallback the suite does (`tests/scalar-pass.txt`), and the conformance corpus. -5. **Native AOT**: CI publishes `vxdump` ahead of time and runs it over the corpus. -6. **The first batch waits for the first split**: the time to first batch of every streaming query - — a scan, a projection, a `Distinct`, a group by on a key that streams — does not grow with the - file or the dataset, locally and over a source with latency - ([16-queries.md](16-queries.md) §13). -7. **Memory is the window and the open state**: a streaming group by's peak stays flat as its - groups grow, a top-k's follows `k`, a blocking group by's follows its groups and not the degree - times them (`LiveMemoryTests`). -8. **Answers do not depend on the cut**: every aggregate that does not depend on the order of rows - by definition, float sums included, is the same bits at every degree, in every order of rows, - and under every cut into chunks, files or objects. +4. Scalar fallback: CI runs again, with hardware intrinsics disabled, the test classes that reach + every scalar fallback the suite covers (`tests/scalar-pass.txt`), plus the conformance corpus. +5. Native AOT: CI publishes `vxdump` ahead of time and runs it over the corpus. +6. The first batch only waits for the first split. The time to first batch of every streaming query + (a scan, a projection, a `Distinct`, a group by on a key that streams) does not grow with the file + or the dataset, locally and over a source with latency ([how queries are + tested](16-queries.md#13-how-it-is-tested)). +7. Memory is the window plus the open state. A streaming group by's peak stays flat as its groups + grow, a top-k's follows `k`, and a blocking group by's follows its groups, not the degree times its + groups (`LiveMemoryTests`). +8. Answers do not depend on how the data is split. Every aggregate whose definition does not depend + on row order, float sums included, gives the same bits at every degree, in every row order, and + under every split into chunks, files or objects. ## 5. Error handling -- On the hot path, `Try` forms and return codes; no exception for control flow. -- At the boundary: `VortexFormatException` for a malformed file; `VortexUnsupportedException` for - a component this build does not implement, which always names the **id and its kind** (`Array`, +- On the hot path, `Try` forms and return codes, never exceptions for control flow. +- At the boundary, `VortexFormatException` for a malformed file, and `VortexUnsupportedException` for + a component this build does not implement, which always names the id and its kind (`Array`, `Layout`, `DType`, `Aggregate`, `Compression`, `Encryption`, `Index` or `Feature`), since that is - what upstream's troubleshooting asks for; `VortexSchemaException` when a record or a column does - not fit the file. -- Unknown components fail lazily, only the read that needs them, and there is no option to make - them fail at open ([08-semantics.md](08-semantics.md) §4). `VortexFile.ArrayEncodings` and - `LayoutEncodings` list what a footer declares, each with whether this build supports it. + what upstream's troubleshooting asks for. `VortexSchemaException` when a record or a column does + not fit the file, and `VortexMemoryException` when a query outgrows its memory or scratch budget. +- Unknown components fail lazily, only in the read that needs them, and there is no option to make + them fail at open ([unknown + components](08-semantics.md#4-unknown-components-resolve-lazily-fail-on-use)). + `VortexFile.ArrayEncodings` and `LayoutEncodings` list what a footer declares, each with whether + this build supports it. ## 6. Parser safety A file is untrusted input. - Every offset and length read from the file is checked against the real size before any access. -- FlatBuffers verification follows the format's actual rules, stated precisely because the obvious - phrasing both over- and under-rejects: - - **uoffsets** (table and vector references) are unsigned and point forward: `uoffset > 0` and the - target in bounds. Cycles are therefore impossible by construction, and no visited set is kept. - - Forward-only offsets exclude cycles but **not sharing**: two slots may resolve to one target, +- FlatBuffers verification follows the format's actual rules, stated precisely here because the + obvious phrasing both over- and under-rejects: + - uoffsets (table and vector references) are unsigned and point forward: `uoffset > 0` and the + target in bounds. Cycles are impossible by construction, so no visited set is kept. + - Forward-only offsets exclude cycles but not sharing. Two slots may resolve to the same target, so the decoded graph is a DAG, and a reader that recurses once per edge is exponential in depth. - A few hundred bytes of shared children pass every bounds and depth rule while describing 2⁶³ + A few hundred bytes of shared children can pass every bounds and depth rule while describing 2⁶³ visits. Total work therefore has its own bound, separate from depth: every table visited is - charged against `VortexLimits.MaxFlatBufferTables`, as the reference verifiers' `max_tables`. - - **soffsets** (vtable references) are signed: a vtable may precede or follow its table, and both + charged against `VortexLimits.MaxFlatBufferTables`, like the reference verifiers' `max_tables`. + - soffsets (vtable references) are signed. A vtable may precede or follow its table, and both directions are bounded. - - **vtable sharing** between tables is legal and routine; a verifier that refused to revisit a + - Vtable sharing between tables is legal and routine, so a verifier that refused to revisit a position would reject valid files. - - Recursion depth is capped for layout trees, array trees and dtypes: a 10 000-deep nested struct - blows the stack while the schema is parsed, before any data is read. -- No allocation is sized by a file-supplied value without a cap; the caps are one table, - [08-semantics.md](08-semantics.md) §6, not a choice per call site. `alignment_exponent` is a `u8`, - so an unchecked file could demand `2²⁵⁵`. -- Semantic hints are classified ([08-semantics.md](08-semantics.md) §5): what memory safety depends - on is validated unconditionally, what only correctness depends on is covered by the threat model - or by the opt-in `VortexOpenOptions.VerifyStatistics`. + - Recursion depth is capped for layout trees, array trees and dtypes. A 10 000-deep nested struct + would otherwise blow the stack while the schema is parsed, before any data is read. +- No allocation is sized by a file-supplied value without a cap, and the caps live in one table + ([resource caps](08-semantics.md#6-resource-caps)), not in a choice per call site. + `alignment_exponent` is a `u8`, so an unchecked file could demand `2²⁵⁵`. +- Semantic hints are classified ([untrusted + hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). What memory safety depends on is + validated unconditionally, and what only correctness depends on is covered by the threat model or + by the opt-in `VortexOpenOptions.VerifyStatistics`. - Children's row counts are checked against their parents, never assumed. diff --git a/docs/design/04-conformance.md b/docs/design/04-conformance.md index 8ddc331c..3d5a95fb 100644 --- a/docs/design/04-conformance.md +++ b/docs/design/04-conformance.md @@ -1,6 +1,6 @@ # Testing and conformance -A format implementation tested only against itself proves nothing: a round trip through its own +A format implementation tested only against itself proves nothing. A round trip through its own writer and reader is self-consistent and can be uniformly wrong. So every claim about bytes is anchored in the Rust reference, in both directions. Tests use xUnit v3 on Microsoft.Testing.Platform. @@ -15,16 +15,17 @@ anchored in the Rust reference, in both directions. Tests use xUnit v3 on Micros ## 2. Unit and property tests -- The FlatBuffers and Protobuf readers on hand-built bytes: a missing vtable field reads its - default, an empty vector, an absent optional table, and a truncated or over-long input fails with - `VortexFormatException` and nothing else. -- Every SIMD kernel against a scalar reference over random inputs; in CI, again with hardware - intrinsics disabled, every test class that reaches a scalar fallback and the conformance corpus. -- The FastLanes transposition: `untranspose(transpose(i)) == i` for every `i` of a block, as the - reference asserts. -- Every operator against the full materialization it replaces: a projection, a filter, a take, a - count or an aggregate equals the same operation applied to every decoded row. -- The allocation ratchets of [05-benchmarks.md](05-benchmarks.md) §5. +- The FlatBuffers and Protobuf readers are tested on hand-built bytes: a missing vtable field reads + its default, an empty vector, an absent optional table, and a truncated or over-long input fails + with `VortexFormatException` and nothing else. +- Every SIMD kernel is checked against a scalar reference over random inputs. In CI, every test class + that reaches a scalar fallback, and the conformance corpus, run again with hardware intrinsics + disabled. +- The FastLanes transposition satisfies `untranspose(transpose(i)) == i` for every `i` of a block, as + the reference asserts. +- Every operator is checked against the full materialization it replaces: a projection, a filter, a + take, a count or an aggregate must equal the same operation applied to every decoded row. +- The allocation ratchets described in [allocations](05-benchmarks.md#5-allocations). ## 3. The golden corpus @@ -33,90 +34,95 @@ anchored in the Rust reference, in both directions. Tests use xUnit v3 on Micros core editions from `core2025.05.0` to `core2026.08.3`. The matrix is deliberate: - every dtype × nullability × row counts of 0, 1, 1 023, 1 024, 1 025, 8 191, 8 192 and 8 193, - because the block edges at 1 024 (FastLanes) and 8 192 (the default block) are where bugs live; -- every encoding forced on its own by an explicit compressor configuration, so that each is - exercised in isolation rather than whichever the sampling happens to pick; -- the default configuration on realistic distributions, which is the production path; -- edge cases: all-null and constant columns, a single-value dictionary, strings over a mebibyte, - deep nesting, a root that is not a struct, a file with no dtype segment, user metadata, and the - layouts a default writer never emits (an inlined array tree, a dictionary layout). - -Beside each file, a **sidecar of expected values** produced by the Rust reader (`SIDECAR.md` gives + because the block edges at 1 024 (FastLanes) and 8 192 (the default block) are where bugs live +- every encoding forced on its own by an explicit compressor configuration, so that each one is + exercised in isolation rather than whichever the sampling happens to pick +- the default configuration on realistic distributions, which is the production path +- edge cases: all-null and constant columns, a single-value dictionary, strings over a mebibyte, deep + nesting, a root that is not a struct, a file with no dtype segment, user metadata, and the layouts + a default writer never emits (an inlined array tree, a dictionary layout) + +Next to each file sits a sidecar of expected values produced by the Rust reader (`SIDECAR.md` gives its format): the dtype, the layout tree, the file statistics, the zone maps and every value, with floats compared by their bits, so a wrong NaN payload or a `-0.0` read as `+0.0` fails. The tests -therefore need no Rust toolchain. A `manifest.json` records the Vortex version, the edition and the -encodings of every file, and its SHA-256. +therefore need no Rust toolchain. A `manifest.json` records the Vortex version, the edition, the +encodings and the SHA-256 of every file. -**The coverage gate**: the union of the corpus's array and layout ids must cover every component +A coverage gate checks that the union of the corpus's array and layout ids covers every component this library claims to read, or the build fails. What no Rust writer can produce is forged by -patching bytes, in `forged/`: the legacy `vortex.stats` layout, and the unknown ids of §6. +patching bytes, in `forged/`: the legacy `vortex.stats` layout, and the unknown ids of the +[negative tests](#6-negative-and-forward-compatibility-tests). ## 4. The cross-check: Rust reads what we write Golden files prove that we read what Rust wrote. They do not prove that Rust reads what we write, -and a writer tested only by its own reader reads back its own mistakes. So a CI job has this -library write the whole corpus out again, and the reference read every file and compare it scalar -by scalar with the source (`bench/crosscheck.sh` locally, `verify_written` in the generator). The -writes rotate the index policies, append in pieces, rewrite files as a dataset's compaction does, -and chunk some columns apart, so that every structure this writer can emit meets a strict Rust -reader. +and a writer tested only by its own reader reads back its own mistakes. So a CI job has this library +write the whole corpus out again, and the reference reads every file and compares it scalar by +scalar with the source (`bench/crosscheck.sh` locally, `verify_written` in the generator). The writes +rotate the index policies, append in pieces, rewrite files the way a dataset's compaction does, and +chunk some columns apart, so every structure this writer can emit meets a strict Rust reader. -The benchmark shim adds an in-process check ([05-benchmarks.md](05-benchmarks.md) §6): both readers -return the same rows and the same checksum of every decoded value on the same files. +The benchmark shim adds an in-process check ([how it is measured](05-benchmarks.md#6-how-it-is-measured)): +both readers must return the same rows and the same checksum of every decoded value on the same +files. ## 5. Fuzzing -- **Target**: open and full scan, over mutated corpus files. -- **Invariant**: only `VortexFormatException` or `VortexUnsupportedException` escapes. An access - violation, an index out of range, an out-of-memory, a hang or a silently wrong value is a bug. -- **Structure-aware mutation**: a naive bit flip on an offset-based format dies at the first bounds - check. The mutator (`tests/Vorticity.Fuzz`) rewrites words in the tail where the offsets live, - truncates, writes plausible Protobuf tags and long varints, and puts extreme values in the - lengths and counts that memory safety depends on ([08-semantics.md](08-semantics.md) §5). -- **Caps are invariants**: a file declaring a `2²⁵⁵` alignment or a segment that expands to 100 GiB - must fail cleanly and fast. -- **Where it runs**: a smoke campaign in every test run (`FuzzSmokeTests`), and 100 000 mutations - with a fixed seed in CI, so a finding reproduces from the log. +- The target is an open followed by a full scan, over mutated corpus files. +- The invariant is that only `VortexFormatException` or `VortexUnsupportedException` escapes. An + access violation, an index out of range, an out-of-memory, a hang or a silently wrong value is a + bug. +- Mutation is structure-aware, because a naive bit flip on an offset-based format dies at the first + bounds check. The mutator (`tests/Vorticity.Fuzz`) rewrites words in the tail where the offsets + live, truncates, writes plausible Protobuf tags and long varints, and puts extreme values in the + lengths and counts that memory safety depends on ([untrusted + hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). +- The caps are invariants too. A file declaring a `2²⁵⁵` alignment, or a segment that expands to + 100 GiB, must fail cleanly and fast. +- A smoke campaign runs in every test run (`FuzzSmokeTests`), and CI runs 100 000 mutations with a + fixed seed, so any finding reproduces from the log. ## 6. Negative and forward-compatibility tests Two fixtures carry an encoding or a layout id that exists in no edition (`forged/negative/unknown_*_id.vortex`): -- reading a **projected** column that uses one fails with `VortexUnsupportedException` naming the - id and its kind; -- scanning **without** projecting it succeeds. This is what locks in lazy resolution - ([08-semantics.md](08-semantics.md) §4): without it, an open path "simplified" into an eager - failure would pass. +- reading a projected column that uses one fails with `VortexUnsupportedException`, naming the id + and its kind +- scanning without projecting it succeeds -On the pruning side, filter tests run with pruning and indexes on and off and must return the same -rows, which is what enforces that pruning never removes a matching row. +The second test is what locks in lazy resolution ([unknown +components](08-semantics.md#4-unknown-components-resolve-lazily-fail-on-use)). Without it, an open +path "simplified" into an eager failure would still pass. + +On the pruning side, filter tests run with pruning and indexes both on and off and must return the +same rows, which is what enforces that pruning never removes a matching row. ## 7. The row encoding -The row encoder's contract is one property, byte order equals tuple order, and a format match -([06-row-encoding.md](06-row-encoding.md)): +The row encoder's contract is one property, byte order equals tuple order, plus a byte-for-byte +format match ([06-row-encoding.md](06-row-encoding.md)): -- **The order property, randomized**: random tuples over a random schema and random sort options; +- The order property is randomized: random tuples over a random schema with random sort options, and sorting by `memcmp` of the encoded rows must give exactly the permutation of a tuple sort. -- **Adversarial values**: `-0.0` and `+0.0`, NaN bit patterns, `i64::MIN`, empty against null - against one-byte strings, strings of 31, 32, 33 and 64 bytes around the block size, prefix pairs, - and nulls under nested structs with garbage in their children. -- **Byte for byte against Rust**: `tools/row-vectors` encodes generated tables with the reference - `vortex-row` and records the bytes in `tests/Vorticity.Conformance/row-vectors`; the tests +- Adversarial values are covered: `-0.0` and `+0.0`, NaN bit patterns, `i64::MIN`, empty against + null against one-byte strings, strings of 31, 32, 33 and 64 bytes around the block size, prefix + pairs, and nulls under nested structs with garbage in their children. +- The bytes are checked against Rust. `tools/row-vectors` encodes generated tables with the reference + `vortex-row` and records the bytes in `tests/Vorticity.Conformance/row-vectors`, and the tests compare every one. `vortex-row` is not published on crates.io, so that crate takes the Vortex repository at tag `0.86.1` as a git dependency, the only one in the repository, and its output is - committed. Being order-compatible is not enough: two encoders can each be consistent and still - disagree, which would break any comparison across languages. + committed. Being order-compatible is not enough, since two encoders can each be consistent and + still disagree, which would break any comparison across languages. ## 8. Read forever Files written under the oldest edition must stay readable, and the corpus holds files written under -each edition from `core2025.05.0` on: the files of the older editions are what catches a reader -that hard-codes the encodings of the newest edition. They are written by the pinned 0.86.1 generator restricted to -each edition, which is the same component set as the 0.36.0 floor with a modern writer; files from -a 0.36.0 writer itself would need a second, older generator. +each edition from `core2025.05.0` on. The files of the older editions are what catches a reader that +hard-codes the encodings of the newest edition. They are written by the pinned 0.86.1 generator +restricted to each edition, which gives the same component set as the 0.36.0 floor with a modern +writer. Files from a 0.36.0 writer itself would need a second, older generator. -A CI job re-vendors the upstream schemas and reports any drift; a change there is a format change, +A CI job vendors the upstream schemas again and reports any drift. A change there is a format change, reviewed against [90-registry.md](90-registry.md), and the corpus is regenerated with the generator pinned to the new version. diff --git a/docs/design/05-benchmarks.md b/docs/design/05-benchmarks.md index 047519b8..6810120c 100644 --- a/docs/design/05-benchmarks.md +++ b/docs/design/05-benchmarks.md @@ -1,226 +1,238 @@ # Benchmarks -How Vorticity is measured against Vortex's Rust implementation, and its kernels against their -baselines: what is compared (§1), what a caller sees (§2), the ratios the gates hold in one -process (§3), each decoder (§4), allocations (§5), and the instruments (§6). The figures are on one -page, [the benchmark page](../guide/benchmarks.md), each section written by the instrument that -measures it; [bench/README.md](../../bench/README.md) says how to run each one. +How Vorticity is measured against Vortex's Rust implementation, and how its kernels are measured +against their baselines. This page covers [what is compared](#1-the-comparison), [what a caller +sees](#2-what-a-caller-sees), [the ratios the gates hold in one +process](#3-in-one-process-after-warm-up), [each decoder](#4-decoders-per-encoding), +[allocations](#5-allocations) and [the instruments](#6-how-it-is-measured). The figures are all on +[the benchmark page](../guide/benchmarks.md), each section written by the instrument that measures +it, and [bench/README.md](../../bench/README.md) explains how to run each one. ## 1. The comparison -* **Against Rust.** The reference is Vortex's own implementation, the one the format was designed - around. A .NET Parquet library would measure a difference of format, not of implementation. -* **The same work.** Both sides read the same bytes and must return the same rows, or the run fails. - Every call opens its file and maps it anew, on both sides, and reads it where it lies: Rust through - `open_buffer` over a `memmap2` mapping, ours through a session that keeps no mapping once a file is - closed. Both decode every value they return to its plain form, except a constant column, which both - keep as one value and a length: our reader's form, and upstream's `Columnar`. A write hands its - bytes to a sink that counts them and keeps none, on both sides. A count is a count on both sides, - and a read the reference has no index for is asked of it as the rows it would have to find. - Neither side keeps a segment cache. -* **Rust's figure is its faster setting.** By default Rust cuts a chunk of more than 100 000 rows - into splits of 100 000, and its reader decodes the whole chunk again for every split: on the - per-encoding corpus, files of one chunk of a million rows, that is ten decodes of each, and the - string-heavy files took ten times as long. Asked for one split per chunk it decodes each once, but - builds a chunk made of smaller arrays in one piece, three to six times slower on those. Neither - wins everywhere, so on one core every Rust figure is the faster of the two, measured on the spot, - file by file and axis by axis; on all cores Rust keeps its default, whose splits spread its work. -* **Each side built for speed.** Vorticity as a Native AOT binary for the machine's instruction - set. Vortex 0.86.1 built as upstream builds its own benchmarks: mimalloc, `-C target-cpu=native`, - one codegen unit, no LTO, upstream's `release_debug` profile and the `RUSTFLAGS` of its benchmark - workflows. -* **The same threads.** Every scenario runs on one core — Rust's single-threaded runtime, our scans - at one lane — and on all of them: a Tokio worker per processor for Rust, a lane per processor for - our scans and as many threads for our writer. -* **A speedup on every page**: Rust's time over Vorticity's, so that above 1.00× Vorticity took - less, and in a kernel's table its baseline's time over its own. The gates hold the inverse, - Vorticity's time over Rust's, under a ceiling (§3); their consoles print that ratio. -* **The targets**: within 2× of Rust's time on a full scan and 1.5× on a decoder, per instruction - set, and files no larger than 105 % of Rust's. The benchmark page says where each time stands; - `WrittenSizeTests` holds the bytes the writer gives the whole conformance corpus under a ceiling - below the size target. -* **One machine per figure.** Each section of the benchmark page names the machine it was measured - on: an Apple M4 Pro, arm64 and the `Vector128` paths, for [the benchmark page](../guide/benchmarks.md), - and an AMD Ryzen 9 7950X, x64 with AVX-512, for [its twin](../guide/benchmarks-x64.md). CI builds - and tests on x64 and arm64 but runs no benchmark. +* The reference is Rust. Vortex's own implementation is the one the format was designed around, and + a .NET Parquet library would measure a difference of format, not of implementation. +* Both sides do the same work. They read the same bytes and must return the same rows, or the run + fails. Every call opens its file and maps it anew, on both sides, and reads it where it lies: Rust + through `open_buffer` over a `memmap2` mapping, ours through a session that keeps no mapping once a + file is closed. Both decode every value they return to its plain form, except a constant column, + which both keep as one value and a length (our reader's form, and upstream's `Columnar`). A write + hands its bytes to a sink that counts them and keeps none, on both sides. A count is a count on + both sides, and a read the reference has no index for is asked of it as the rows it would have to + find. Neither side keeps a segment cache. +* Rust's figure is its faster setting. By default Rust cuts a chunk of more than 100 000 rows into + splits of 100 000, and its reader decodes the whole chunk again for every split. On the + per-encoding corpus, files of one chunk of a million rows, that means ten decodes of each, and the + string-heavy files took ten times as long. Asked for one split per chunk, it decodes each chunk + once but builds a chunk made of smaller arrays in one piece, three to six times slower on those + files. Neither setting wins everywhere, so on one core every Rust figure is the faster of the two, + measured on the spot, file by file and axis by axis. On all cores Rust keeps its default, whose + splits spread its work. +* Each side is built for speed. Vorticity runs as a Native AOT binary for the machine's instruction + set. Vortex 0.86.1 is built the way upstream builds its own benchmarks: mimalloc, + `-C target-cpu=native`, one codegen unit, no LTO, upstream's `release_debug` profile and the + `RUSTFLAGS` of its benchmark workflows. +* Both sides use the same threads. Every scenario runs on one core (Rust's single-threaded runtime, + our scans at one lane) and on all of them (a Tokio worker per processor for Rust, a lane per + processor for our scans and as many threads for our writer). +* Every page shows a speedup: Rust's time over Vorticity's, so above 1.00× Vorticity took less, and + in a kernel's table, its baseline's time over its own. The gates hold the inverse, Vorticity's time + over Rust's, under a ceiling ([in one process](#3-in-one-process-after-warm-up)), and their consoles + print that ratio. +* The targets are within 2× of Rust's time on a full scan and within 1.5× on a decoder, per + instruction set, and files no larger than 105 % of Rust's. The benchmark page shows where each time + stands, and `WrittenSizeTests` holds the bytes the writer produces for the whole conformance corpus + under a ceiling below the size target. +* Each figure comes from one machine. Each section of the benchmark page names the machine it was + measured on: an Apple M4 Pro, arm64 with the `Vector128` paths, for [the benchmark + page](../guide/benchmarks.md), and an AMD Ryzen 9 7950X, x64 with AVX-512, for [its + twin](../guide/benchmarks-x64.md). CI builds and tests on x64 and arm64 but runs no benchmark. ## 2. What a caller sees -Seven actions a program performs on a file, each timed in a fresh process from opening the file to -its last row: the median of five runs, the page cache warm. The data is one table of 2^20 rows and -one of ten times that, four columns: `monotone`, an increasing `i64`; `value`, an `f64`; `label`, -one of five short strings; `flag`, a `bool` with nulls. In their plain form, Arrow's layout, that is -25.5 bytes a row. - -**Two files of the same rows.** A Vortex file can store a column in many encodings, each writer -picks its own, and what a read costs depends on the pick. So the rows are written twice, once by -each implementation's writer, and both readers read both files; the benchmark page lists what each -writer chose at each size. Our writer stores `value` as zstd frames where Rust's chooses ALP-RD: a -much smaller file, slower to scan and to take from. Which of the two `Auto` should pick is the -writer's decision ([11-write-strategy.md](11-write-strategy.md) §3.4); -[the encodings, column by column](../guide/benchmarks.md#encodings-column-by-column) measure the -trade. - -The actions: `open` the file and read no rows; `scan` every column of every row; `project` one -column of four; `filter-narrow`, the rows of a band of `monotone` holding about one in a hundred; -`filter-wide`, a band holding about half; `take` a thousand rows spread across the file; `write`, -read the file and write it back out. `append`, a tenth of the rows appended to a copy of the file, -is ours alone: Rust's harness has no entry point for it. - -**GB/s** is the plain size of the rows returned over the time. **Allocated** is what Vorticity +Seven actions a program performs on a file are each timed in a fresh process, from opening the file +to its last row: the median of five runs, with the page cache warm. The data is one table of 2^20 +rows and one of ten times that, with four columns: `monotone`, an increasing `i64`, `value`, an +`f64`, `label`, one of five short strings, and `flag`, a `bool` with nulls. In their plain form +(Arrow's layout) that is 25.5 bytes per row. + +The same rows are written as two files. A Vortex file can store a column in many encodings, each +writer picks its own, and what a read costs depends on the pick. So the rows are written twice, once +by each implementation's writer, and both readers read both files. The benchmark page lists what each +writer chose at each size. Our writer stores `value` as zstd frames where Rust's chooses ALP-RD, +which gives a much smaller file that is slower to scan and to take from. Which of the two `Auto` +should pick is the writer's decision ([choosing +encodings](11-write-strategy.md#34-choose-exact-verdicts-then-bounded-trials)), and [the encodings, +column by column](../guide/benchmarks.md#encodings-column-by-column) measure the trade-off. + +The actions are: `open` the file and read no rows, `scan` every column of every row, `project` one +column of four, `filter-narrow` (the rows of a band of `monotone` holding about one in a hundred), +`filter-wide` (a band holding about half), `take` a thousand rows spread across the file, and +`write`, which reads the file and writes it back out. `append`, a tenth of the rows appended to a +copy of the file, is ours alone, because Rust's harness has no entry point for it. + +GB/s is the plain size of the rows returned divided by the time. Allocated is what Vorticity allocates on its managed heap for one call, once warm: the median of six calls in a process that has -made two before them. **Peak** is each process's peak resident memory. +made two before them. Peak is each process's peak resident memory. [Reading and writing a table](../guide/benchmarks.md#reading-and-writing-a-table-process-against-process) -has the figures, on one core and on all of them, and what a first call costs on the JIT. Reading -them: - -* **A take from our file inflates a zstd frame per row.** Every row taken inflates the frame that - holds its `value`, 64 KiB for one value, and the inflating is nearly all of the take. -* **On all cores** a take keeps three splits a lane decoded ahead of its consumer, so the lanes - that finish first do not wait for the slowest; our writer summarizes its columns and compresses - its zstd frames on every core, and chooses and writes the encodings on one. -* **Allocations on all cores**: buffers rented from the shared array pool on one thread and returned - on another miss its per-thread caches, so a call now and then allocates several times its median. - It is a known defect, not a cost of the work. -* **Native AOT, and the JIT.** The tables are the native build's. On the JIT, a fresh process - compiles the code the first time it runs it, which the page's last table measures; a process that - stays up pays that once, and then reads the speedups of §3 and §4. Starting a process is not in the - figures, and the page gives it apart. +has the figures, on one core and on all of them, and what a first call costs on the JIT. A few +points help reading them: + +* A take from our file inflates a zstd frame per row. Every row taken inflates the frame that holds + its `value`, 64 KiB for one value, and the inflating is nearly all of the take. +* On all cores, a take keeps three splits per lane decoded ahead of its consumer, so the lanes that + finish first do not wait for the slowest. Our writer summarizes its columns and compresses its + zstd frames on every core, and chooses and writes the encodings on one. +* On all cores, buffers rented from the shared array pool on one thread and returned on another miss + the pool's per-thread caches, so a call now and then allocates several times its median. This is a + known defect, not a cost of the work. +* The tables come from the native build. On the JIT, a fresh process compiles code the first time it + runs it, which the page's last table measures. A process that stays up pays that once, and then + sees the speedups of [in one process](#3-in-one-process-after-warm-up) and + [decoders](#4-decoders-per-encoding). Process start-up is not in the figures, and the page gives it + separately. ## 3. In one process, after warm-up `--ratio-check` calls both implementations in one process, Rust's through a C ABI, in turn and -against one clock: at least 21 rounds, until the 95 % interval of the per-round ratios is within -5 % of their median. Our side runs on the JIT, warmed. Each axis is a gate: its ratio, our time over -Rust's, must stay under a ceiling. The page reads the same rounds the other way up, the median of +against one clock: at least 21 rounds, until the 95 % interval of the per-round ratios is within 5 % +of their median. Our side runs on the JIT, warmed up. Each axis is a gate, whose ratio (our time over +Rust's) must stay under a ceiling. The page reads the same rounds the other way up, as the median of the per-round speedups and its interval. The first eight axes read a corpus file Rust's writer made, `containers/zoned_many_zones_nulls`: 65 536 rows, five columns, 64 zones. The key-order, string and -band axes read files of 65 536 rows our writer makes for them; the last two, generated tables of a -million rows and six columns, and of fifty columns. +band axes read files of 65 536 rows that our writer makes for them, and the last two read generated +tables of a million rows with six columns, and of fifty columns. [In one process](../guide/benchmarks.md#in-one-process-after-warm-up) has the figures. -* **`open to first batch` is a latency.** Both sides decode their first batch. Rust's first batch - costs most of what its whole scan does on this file, 1.4 ms of 1.6, where ours costs - a fifth of ours. The speedup is on the time to the first row, and nothing more. -* **Where the reference has no counterpart, it answers the same question its own way.** A - key-ordered read of a band is held against Rust's filtered scan of that band, which returns the - same rows in file order; an exact count from the index, against Rust's count of the band with no - column decoded. The speedup then prices the index, which is what a caller would weigh. -* The gate has seven more axes, which the page leaves out: two time Rust's scan without decoding, - which asks another question; one holds the uncorrelated key-order read to a take of 64 rows whose - positions the reference is given, the floor that read cannot go under; and four read one file as +* `open to first batch` is a latency. Both sides decode their first batch. Rust's first batch costs + most of what its whole scan does on this file, 1.4 ms of 1.6, where ours costs a fifth of our full + scan. The speedup is about the time to the first row, and nothing more. +* Where the reference has no counterpart, it answers the same question its own way. A key-ordered + read of a band is compared with Rust's filtered scan of that band, which returns the same rows in + file order, and an exact count from the index with Rust's count of the band with no column decoded. + The speedup then prices the index, which is what a caller would weigh. +* The gate has seven more axes that the page leaves out. Two time Rust's scan without decoding, which + asks a different question. One holds the uncorrelated key-order read against a take of 64 rows + whose positions the reference is given, the floor that read cannot go under. Four read one file as each writer wrote it. [bench/README.md](../../bench/README.md) describes them. ## 4. Decoders, per encoding -One file per encoding, and per notable shape of one: 57 files of a million rows -(`bench/gen-throughput.sh`), written by Rust's writer. Each side scans each file 12 times in a -process of its own, on one core, every value decoded to its plain form; a figure is the median of the -last 10 scans and of 3 processes, Native AOT against Rust's `vxbench` binary: what a decoder costs -once warm. [Decoding, per encoding](../guide/benchmarks.md#decoding-per-encoding) has all 57. - -* **Past what memory can move is not decoding.** A column stored in its plain form is handed out as - a view of the mapped file on both sides, nothing copied: the figure measures the walk of the - layout, and the speedup compares two walks. -* **The widest gaps are UTF-8 validation.** In `varbin`, `varbinview`, `struct` and the tables, the - stored arrays are already in their plain form, and the time is the check that every string is - UTF-8, which both sides make: a sampled profile of Rust's scan of `varbinview` spends it in - `from_utf8` and `validate_view`, under `VarBinViewData::try_new`. -* **Each scan opens the file again and maps it anew**, on both sides. - -Take and write per encoding are measured in one process, on the JIT after warm-up, each file held to -a ceiling (`--throughput --take`, `--throughput --write`): -[taking rows](../guide/benchmarks.md#taking-rows-per-encoding) and +There is one file per encoding, and per notable shape of one: 61 files of a million rows +(`bench/gen-throughput.sh`), written by Rust's writer. The arm64 page was measured on the earlier set +of 57, and the four `pco` shapes added since (`pco_f32`, `pco_i16`, `pco_u32`, `pco_nullable`) are on +the x64 page only. Each side scans each file 12 times in a process of its own, on one core, every +value decoded to its plain form. A figure is the median of the last 10 scans and of 3 processes, +Native AOT against Rust's `vxbench` binary, which is what a decoder costs once warm. [Decoding, per +encoding](../guide/benchmarks.md#decoding-per-encoding) has every file. + +* A column stored in plain form is not decoding at all. It is handed out as a view of the mapped file + on both sides, with nothing copied, so the figure measures the walk of the layout and the speedup + compares two walks. +* The widest gaps are UTF-8 validation. In `varbin`, `varbinview`, `struct` and the tables, the + stored arrays are already in plain form, and the time goes to checking that every string is UTF-8, + which both sides do. A sampled profile of Rust's scan of `varbinview` spends it in `from_utf8` and + `validate_view`, under `VarBinViewData::try_new`. +* Each scan opens the file again and maps it anew, on both sides. + +Take and write per encoding are measured in one process, on the JIT after warm-up, with each file +held to a ceiling (`--throughput --take`, `--throughput --write`). See [taking +rows](../guide/benchmarks.md#taking-rows-per-encoding) and [writing](../guide/benchmarks.md#writing-per-encoding), per encoding. Rust's writer declines `parquet_variant`. ### How the hot paths are built -* **FastLanes: the lane is the vector lane, and each packed word is loaded once.** A bit-packed - block is 1 024 values transposed so that a row of the block, and each packed word of all its - lanes, is 128 contiguous bytes whatever the element type: whole-vector loads and stores, no - gather, no scatter. The kernel walks a block a packed word at a time, keeping the word in - registers for every row inside it; what a step does depends on the width alone and is laid out - once per width, so no branch looks at the data. The scalar loop stays as fallback and oracle +* In FastLanes, the lane is the vector lane, and each packed word is loaded once. A bit-packed block + is 1 024 values transposed so that a row of the block, and each packed word across all its lanes, + is 128 contiguous bytes whatever the element type: whole-vector loads and stores, no gather, no + scatter. The kernel walks a block one packed word at a time, keeping the word in registers for + every row inside it. What a step does depends only on the bit width and is laid out once per width, + so no branch looks at the data. The scalar loop stays as fallback and oracle (`FastLanesKernelBenchmarks` on the benchmark page). -* **FSST and OnPair: one wide store per symbol.** A symbol of one to eight bytes is written as one - unaligned 8-byte store, and the output advances by its real length; an OnPair token of up to 16 - bytes, as one `Vector128` store. An exact copy only where the slack runs out at the end of a - buffer. Checking that every code names a symbol and stays inside the destination costs next to - nothing against the bare loop. -* **Zstd: a frame per block, and only the frames a take needs.** Our writer compresses a zstd column - a block of 8 192 rows to a frame, and our reader inflates only the frames that hold the rows it is - asked for. -* **A take reads by the run, and decodes ahead.** Consecutive zones holding a take's rows are read - as one batch, up to sixteen rows a batch on several lanes, and a column whose encoding decodes - ranges without selecting decodes the clusters of rows it is asked for rather than the span between - them. On several lanes a take keeps three splits a lane in flight, and its degree bounds how many - decode at once. -* **Zone maps prune before any decode.** The file keeps the minimum, maximum and null count of every - zone of 8 192 rows. A filter skips the zones it proves empty and hands out the zones it proves full - without testing their rows: the narrow filter of §2 decodes about a hundredth of the rows. -* **Views, not copies.** On a mapped file, a column stored in its plain form is a view of the mapped - bytes; a test checks it by comparing pointers. -* **Nothing allocated per batch.** A scan's arenas and pools are its own, its `RecordBatch` is bound +* FSST and OnPair write one wide store per symbol. A symbol of one to eight bytes is written as one + unaligned 8-byte store and the output advances by its real length, and an OnPair token of up to 16 + bytes as one `Vector128` store. An exact copy is only needed where the slack runs out at the end of + a buffer. Checking that every code names a symbol and stays inside the destination costs next to + nothing compared with the bare loop. +* Zstd uses a frame per block, and a take only inflates the frames it needs. Our writer compresses a + zstd column one block of 8 192 rows per frame, and our reader inflates only the frames that hold + the rows it is asked for. +* A take reads by runs and decodes ahead. Consecutive zones holding a take's rows are read as one + batch, up to sixteen rows per batch on several lanes, and a column whose encoding can decode ranges + without a selection decodes the clusters of rows asked for rather than the span between them. On + several lanes, a take keeps three splits per lane in flight, and its degree bounds how many decode + at once. +* Zone maps prune before any decode. The file keeps the minimum, maximum and null count of every zone + of 8 192 rows. A filter skips the zones it proves empty and hands out the zones it proves full + without testing their rows, so the narrow filter of [what a caller sees](#2-what-a-caller-sees) + decodes about a hundredth of the rows. +* On a mapped file, a column stored in plain form is a view of the mapped bytes rather than a copy, + and a test checks it by comparing pointers. +* Nothing is allocated per batch. A scan's arenas and pools are its own, its `RecordBatch` is bound again rather than rebuilt, and its column views are `ref struct`s the compiler keeps inside the - loop body (§5). -* **Parallel when asked, and thread-safe where shared.** Parallelism is given, never taken: a - session's `MaxDegreeOfParallelism`, 1 unless set, or a scan's or a writer's own + loop body ([allocations](#5-allocations)). +* Parallelism is opt-in, and shared objects are thread-safe. Parallelism is given, never taken: a + session's `MaxDegreeOfParallelism` (1 unless set), or a scan's or a writer's own `DegreeOfParallelism`. A scan decodes its splits side by side, each on a context and arenas of its - own, nothing shared, and delivers its batches in row order. A writer summarizes its columns and + own with nothing shared, and delivers its batches in row order. A writer summarizes its columns and compresses a column's zstd frames on its threads, and writes the same bytes whatever the degree. A - `VortexSession` and a `VortexFile` are thread-safe and meant to be shared, several scans of one - file at once included; a scan, a writer and a batch belong to one consumer at a time - ([09-contracts.md](09-contracts.md) §1–2, [threads.md](../guide/threads.md)). + `VortexSession` and a `VortexFile` are thread-safe and meant to be shared, including several scans + of one file at once, while a scan, a writer and a batch belong to one consumer at a time (see + [concurrency](09-contracts.md#1-concurrency-and-thread-safety), + [parallelism](09-contracts.md#2-parallelism) and [threads.md](../guide/threads.md)). ## 5. Allocations -Held by ratchets in the test suite, counted by the runtime in a process doing nothing else, so -exact: a test fails above its ceiling, and a ceiling only comes down. Each test prints what it -measured beside its ceiling. - -* **Per batch**: nothing on the calling thread, in steady state, on every scan - (`ScanAllocationTests`). Counted on every thread of a scan at degree 2, a few hundred bytes a batch - for its tasks, under a ceiling of 1 000. -* **Per operation**, on the 65 536-row corpus file of §3, the open included (`PathAllocationTests`): - the footer alone, the first batch, a full scan, a projection, a take of 64 rows from 64 splits, a - filter with and without its zone maps, and a full scan of one encoding's 4 096-row file, each - under a ceiling in bytes. A full scan costs what its first batch costs. -* **Per write** (`WriteAllocationTests`): each corpus file written back out, a writer on four threads +Allocations are held by ratchets in the test suite, counted by the runtime in a process doing nothing +else, so the counts are exact. A test fails above its ceiling, and a ceiling only ever comes down. +Each test prints what it measured next to its ceiling. + +* Per batch: nothing on the calling thread, in steady state, on every scan (`ScanAllocationTests`). + Counted on every thread of a scan at degree 2, a few hundred bytes per batch for its tasks, under a + ceiling of 1 000. +* Per operation, on the 65 536-row corpus file used [in one + process](#3-in-one-process-after-warm-up), the open included (`PathAllocationTests`): the footer + alone, the first batch, a full scan, a projection, a take of 64 rows from 64 splits, a filter with + and without its zone maps, and a full scan of one encoding's 4 096-row file, each under a ceiling in + bytes. A full scan costs what its first batch costs. +* Per write (`WriteAllocationTests`): each corpus file written back out, a writer on four threads against one, and a column of a wide schema, each under a ceiling. -The benchmark page gives what each of §2's actions allocates once warm, beside its time. +The benchmark page gives what each of the actions in [what a caller sees](#2-what-a-caller-sees) +allocates once warm, next to its time. ## 6. How it is measured -Every instrument but the ratchets runs as `dotnet run -c Release --project -bench/Vorticity.Benchmarks -- `, and, given `--out docs/guide/benchmarks.md`, writes -its sections of the benchmark page and leaves the others: +Every instrument except the ratchets runs as `dotnet run -c Release --project +bench/Vorticity.Benchmarks -- `, and, given `--out docs/guide/benchmarks.md`, rewrites its +own sections of the benchmark page and leaves the others alone: | instrument | what it gives | arguments | |---|---|---| -| the report | §2, and §4's decode, each side in its own process | `--report --markdown` | -| the ratio gate | §3, both sides in one process | `--ratio-check` | -| the per-encoding gates | §4's take and write, in one process | `--throughput`, with `--take` or `--write`; `--check` holds each file to its ceiling | +| the report | [what a caller sees](#2-what-a-caller-sees) and the per-encoding decode, each side in its own process | `--report --markdown` | +| the ratio gate | [in one process](#3-in-one-process-after-warm-up), both sides in one process | `--ratio-check` | +| the per-encoding gates | the per-encoding take and write, in one process | `--throughput`, with `--take` or `--write`, and `--check` to hold each file to its ceiling | | the trade-offs | every profile and hint on twenty column shapes, and what the advice picks on them | `--tradeoffs`, and `--advise` | -| BenchmarkDotNet | one kernel against its baseline, on one clock | a class name: `fastlanes` | -| the ratchets | §5 | `dotnet test Vorticity.slnx -c Release` | +| BenchmarkDotNet | one kernel against its baseline, on one clock | a class name, such as `fastlanes` | +| the ratchets | [allocations](#5-allocations) | `dotnet test Vorticity.slnx -c Release` | Rust is driven through [tools/vxbench-rs](../../tools/vxbench-rs), built against the same -`vortex = "=0.86.1"` pin the corpus was written with: a C ABI for the in-process gates, a binary for -the report. `--ffi-check` checks that both readers return the same rows and the same checksum of +`vortex = "=0.86.1"` pin the corpus was written with: a C ABI for the in-process gates, and a binary +for the report. `--ffi-check` checks that both readers return the same rows and the same checksum of every decoded value before a ratio is trusted, and the gates refuse to run under a Rust build other -than the one their ceilings were set under. [bench/README.md](../../bench/README.md) has every +than the one their ceilings were set with. [bench/README.md](../../bench/README.md) has every command, what it costs, and how a ceiling moves. -Not measured: - -* **A cold page cache.** Every run reads a warm one; upstream's benchmarks flush it. -* **Pinned cores.** macOS pins no process, and an Apple M-series processor has cores of two kinds. -* **A remote store's latency**: every file is local. -* **Other data.** TPC-H and ClickBench are gigabytes that do not belong in a repository; what makes - the comparison honest is both sides reading the same bytes, whatever they are. -* **The row encoding against Rust's**: the shim cannot hand `vortex-row` a batch, so - `RowEncodingBenchmarks` compares our builds only. -* **CI** runs no benchmark: the gates are commands run before a push, `bench/gate.sh` among them. +What is not measured: + +* A cold page cache. Every run reads a warm one, while upstream's benchmarks flush it. +* Pinned cores. macOS pins no process, and an Apple M-series processor has two kinds of cores. +* A remote store's latency, since every file is local. +* Other datasets. TPC-H and ClickBench are gigabytes that do not belong in a repository, and what + makes the comparison honest is both sides reading the same bytes, whatever they are. +* The row encoding against Rust's. The shim cannot hand `vortex-row` a batch, so + `RowEncodingBenchmarks` only compares our own builds. +* Anything in CI. CI runs no benchmark, and the gates are commands run before a push, `bench/gate.sh` + among them. diff --git a/docs/design/06-row-encoding.md b/docs/design/06-row-encoding.md index a83abe40..40fd1564 100644 --- a/docs/design/06-row-encoding.md +++ b/docs/design/06-row-encoding.md @@ -1,40 +1,39 @@ # Byte-sortable row encoding -A separate specification from the file format, implemented upstream by the `vortex-row` crate. It -converts one or more columnar arrays into a `ListView`: one byte string per row, such that -**lexicographic byte comparison of the encoded rows equals logical tuple comparison of the input -values**, under per-column sort direction and null placement. +The row encoding is a separate specification from the file format, implemented upstream by the +`vortex-row` crate. It converts one or more columnar arrays into a `ListView`, one byte string +per row, such that lexicographic byte comparison of the encoded rows equals logical tuple comparison +of the input values, under per-column sort direction and null placement. -Use cases: sort keys, row keys, range partitioning, merge joins, index entries — anywhere a -comparator must become a `memcmp`. +It serves sort keys, row keys, range partitioning, merge joins and index entries: anywhere a +comparator has to become a `memcmp`. -> **Stability.** Upstream marks this format *experimental*: the byte layout, the supported types and -> the edge cases may change between Vortex releases, and its spec says not to persist these bytes as -> an interchange format. So it ships in its own package, `Vorticity.RowEncoding`, marked +> Upstream marks this format experimental. The byte layout, the supported types and the edge cases +> may change between Vortex releases, and its spec says not to persist these bytes as an interchange +> format. So it ships in its own package, `Vorticity.RowEncoding`, marked > `[Experimental("VX0002")]`, and the encoder names the release its bytes follow (`vortex-row -> 0.86.1`). Encoded bytes are safe to compare within one process, or one cluster running one -> version; a durable index that stores them records that name -> ([10-indexes.md](10-indexes.md) §5.4). +> 0.86.1`). Encoded bytes are safe to compare within one process, or within one cluster running one +> version, and a durable index that stores them records that name (see [composite +> keys](10-indexes.md#54-composite-keys-through-the-row-encoding)). ## 1. Contract -For columns `c0…cn` with per-column `RowSortField { descending: bool, nulls_first: bool }`: +For columns `c0…cn` with a per-column `RowSortField { descending: bool, nulls_first: bool }`: ``` encode(row_a) < encode(row_b) ⟺ (a.c0, …, a.cn) < (b.c0, …, b.cn) ``` -The bytes carry **no type tags, no field names, no sort options**. Two encoded rows are comparable -only if produced from the same schema with the same field settings. `descending` and `nulls_first` -are independent: nulls can sort first or last in either direction. +The bytes carry no type tags, no field names and no sort options. Two encoded rows are only +comparable if they were produced from the same schema with the same field settings. `descending` and +`nulls_first` are independent, so nulls can sort first or last in either direction. -Notation: `||` concatenation, `BE(x)` fixed-width big-endian bytes (never length-prefixed, never -trimmed), `!b` = `b XOR 0xFF`, `zero(n)` / `ff(n)` = n bytes of `0x00` / `0xFF`. +Notation: `||` is concatenation, `BE(x)` fixed-width big-endian bytes (never length-prefixed, never +trimmed), `!b` is `b XOR 0xFF`, and `zero(n)` and `ff(n)` are n bytes of `0x00` and `0xFF`. ## 2. Sentinels -Every value starts with a sentinel byte that classifies nullness before any value byte is -compared. +Every value starts with a sentinel byte that classifies nullness before any value byte is compared. | Family | Case | Asc, nulls first | Desc, nulls first | Asc, nulls last | Desc, nulls last | |---|---|---|---|---|---| @@ -47,9 +46,9 @@ compared. Fixed-width sentinels serve Null, Bool, Primitive, Decimal, Struct and FixedSizeList. Variable-width sentinels serve Utf8 and Binary. -**The null sentinel is never inverted by `descending`** — only non-null value bytes (and, for -variable-width, the empty/non-empty sentinels) are. This is what keeps null placement independent -of sort direction, and it is the single easiest thing to get wrong. +`descending` never inverts the null sentinel. Only non-null value bytes are inverted, plus, for +variable-width types, the empty and non-empty sentinels. That is what keeps null placement +independent of sort direction, and it is the single easiest thing to get wrong. A fixed-width null is followed by zero-filled value bytes, so fixed types keep a constant encoded width on every row. @@ -59,28 +58,29 @@ width on every row. | Type | Ascending | Descending | |---|---|---| | `Null` | `sentinel` (no body) | same | -| `Bool` | `0x01 \|\| 0x01` (false) / `0x01 \|\| 0x02` (true) | `0x01 \|\| 0xFE` (false) / `0x01 \|\| 0xFD` (true) | +| `Bool` | `0x01 \|\| 0x01` (false), `0x01 \|\| 0x02` (true) | `0x01 \|\| 0xFE` (false), `0x01 \|\| 0xFD` (true) | | `u8/u16/u32/u64` | `0x01 \|\| BE(v)` | `0x01 \|\| !BE(v)` | | `i8/i16/i32/i64` (and `i128`) | `0x01 \|\| ordered`, where `ordered = BE(v)` with `ordered[0] ^= 0x80` | `0x01 \|\| !ordered` | | `f16/f32/f64` | `0x01 \|\| BE(ordered)` | `0x01 \|\| !BE(ordered)` | -| `Decimal` | signed-integer encoding of the scaled storage value | idem | -| `Utf8` / `Binary` | variable-width sentinel + block body (§4) | inverted (§4) | -| `Struct` | `sentinel \|\| field_0 \|\| … \|\| field_n`, fields recursive in schema order | idem | -| `FixedSizeList` | `sentinel \|\| elem_0 \|\| … \|\| elem_{N-1}` | idem | +| `Decimal` | signed-integer encoding of the scaled storage value | same | +| `Utf8` / `Binary` | variable-width sentinel + block body ([below](#4-variable-length-body)) | inverted ([below](#4-variable-length-body)) | +| `Struct` | `sentinel \|\| field_0 \|\| … \|\| field_n`, fields recursive in schema order | same | +| `FixedSizeList` | `sentinel \|\| elem_0 \|\| … \|\| elem_{N-1}` | same | -Null bodies for fixed-width types: `null_sentinel || zero(width(T))`. +Null bodies for fixed-width types are `null_sentinel || zero(width(T))`. -**Float transform**, from raw IEEE bits: +Floats are transformed from their raw IEEE bits: ``` ordered = sign_bit(bits) == 0 ? bits XOR sign_bit_mask : bits XOR all_ones ``` -This yields a total order: negatives before positives, `-0.0` before `+0.0`. NaNs order by raw bit -pattern and are **not** canonicalized: two NaNs with different payloads are two keys. +This yields a total order, negatives before positives and `-0.0` before `+0.0`. NaNs order by their +raw bit pattern and are not canonicalized, so two NaNs with different payloads are two different +keys. -**Decimal storage width**, selected from precision: +A decimal's storage width is selected from its precision: | Precision | Storage | |---|---| @@ -92,40 +92,40 @@ pattern and are **not** canonicalized: two NaNs with different payloads are two `Decimal256` is not supported. -**Option inheritance is identity, and inversion happens once.** A nested field uses its parent's -`RowSortField` unchanged — verified in `vortex-row/src/codec.rs`, where `encode_struct` passes the +Option inheritance is the identity, and inversion happens exactly once. A nested field uses its +parent's `RowSortField` unchanged, as `vortex-row/src/codec.rs` shows: `encode_struct` passes the same `field` to `field_encode` for every child. Inversion for `descending` is applied by the leaf -encoder to its own value bytes only; the parent never re-inverts the concatenated body. The classic -double-inversion trap is therefore avoided by *not* doing anything at the parent level. - -**The child null sentinel is chosen by the child's dtype, using the inherited field.** -`child_canonical_null_byte(child_dtype, field)` returns the variable-width null sentinel for -`Utf8`/`Binary` children and the fixed-width null sentinel for everything else (including a nested -struct or FSL acting as a variable-width child). The `nulls_first` that selects between `0x00` and -`0x02`/`0xFF` is the inherited one, i.e. the root column's. For a fixed-width child under a null -parent, the body is that sentinel byte followed by **zero** fill to the child's fixed width — zero +encoder to its own value bytes only, and the parent never inverts the concatenated body again. The +classic double-inversion trap is avoided by doing nothing at the parent level. + +The child null sentinel is chosen by the child's dtype, using the inherited field. +`child_canonical_null_byte(child_dtype, field)` returns the variable-width null sentinel for `Utf8` +and `Binary` children and the fixed-width null sentinel for everything else, including a nested +struct or FSL acting as a variable-width child. The `nulls_first` that selects between `0x00` and +`0x02` or `0xFF` is the inherited one, which is the root column's. For a fixed-width child under a +null parent, the body is that sentinel byte followed by zero fill up to the child's fixed width, zero even under `descending`. -**Nested nulls are canonicalized.** A null struct or null fixed-size list must still emit a body, -so that two null parents compare byte-equal regardless of what their physical child arrays -contain: fixed-width children contribute their fixed-width null encoding; variable-width children -contribute exactly one child null sentinel byte. Skipping this is a correctness bug that only -shows up on arrays with garbage under a null — exactly what a differential test against Rust will -catch and a naive round-trip test will not. +Nested nulls are canonicalized. A null struct or null fixed-size list must still emit a body, so that +two null parents compare byte-equal whatever their physical child arrays contain. Fixed-width +children contribute their fixed-width null encoding, and variable-width children contribute exactly +one child null sentinel byte. Skipping this is a correctness bug that only shows up on arrays with +garbage under a null, which is exactly what a differential test against Rust catches and a naive +round-trip test does not. ## 4. Variable-length body -Non-empty Utf8/Binary bodies are written in blocks of **32 data bytes + 1 marker byte**: +Non-empty Utf8 and Binary bodies are written in blocks of 32 data bytes plus 1 marker byte: -* Ascending: every non-final full block uses marker `0xFF`; the final block is zero-padded to 32 - data bytes and its marker is the real data length in `1..=32`. -* Descending: every data byte inverted, continuation marker `0x00`, final block padded with - `0xFF`, final marker `final_len XOR 0xFF`. +* Ascending: every non-final full block uses marker `0xFF`, and the final block is zero-padded to 32 + data bytes with the real data length, in `1..=32`, as its marker. +* Descending: every data byte is inverted, the continuation marker is `0x00`, the final block is + padded with `0xFF`, and the final marker is `final_len XOR 0xFF`. * If the length is an exact multiple of 32, the final block carries marker `32` and earlier blocks use the continuation marker. -This is what preserves prefix ordering: a shorter value reaches its final marker (`≤ 32`) while a -longer one is still emitting continuation markers (`0xFF`), so the prefix sorts first. +This preserves prefix ordering: a shorter value reaches its final marker (`≤ 32`) while a longer one +is still emitting continuation markers (`0xFF`), so the prefix sorts first. ## 5. Output layout @@ -137,29 +137,34 @@ sizes: per-row byte length Rows are not self-delimiting without `sizes`. The encoder computes the sizes first (a fixed-width column contributes a constant, a variable-width one what its values need), then reuses the `sizes` -array as the per-row write cursor: **one sizing pass, one write pass**, into three pooled buffers. +array as the per-row write cursor. That makes one sizing pass and one write pass, into three pooled +buffers. ## 6. Unsupported types -Variable-size `List`, `Variant`, `Union`, `Extension` and `Decimal256`. The absence is deliberate -upstream: adding one means defining both its order and its bytes. This encoder refuses them as the -reference does, and never invents an ordering. +Variable-size `List`, `Variant`, `Union` and `Decimal256` are not supported. The absence is +deliberate upstream, since adding one means defining both its order and its bytes. This encoder +refuses them as the reference does, and never invents an ordering. -`Extension` being unsupported means a timestamp or a date is not row-encoded directly: a caller -encodes its storage column. Unwrapping extensions silently would make the bytes stop matching -Rust's. +Upstream does not support `Extension` either. Here, an extension column at the top level of the key +(a date, a time, a timestamp, a uuid) is encoded as its storage column, which gives exactly the bytes +Rust produces when a caller encodes that storage column itself. An extension nested inside a struct +or a fixed-size list is refused with `VortexUnsupportedException`, as the reference refuses it, +because the encoder would otherwise be silently choosing an ordering the format has not defined. ## 7. Implementation -- **Column at a time.** The encoder writes one field for every row, then the next field: better - for branch prediction and vector width than a row at a time, and what the two-pass sizing assumes. - Big-endian conversion is a byte swap, descending a bitwise NOT over a span. -- **No allocation per row.** The output size is known before a byte is written. The kernels call - `TryWriteBigEndian`, never `WriteBigEndian`: the latter is a default interface method the - primitive types do not override, so a generic call to it boxes its receiver on every row. -- **Synchronous.** It is CPU work over arrays already in memory; the async-only rule is for I/O. - -The surface is `RowEncoder` (a borrowed batch in, a pooled `RowKeys` out; one key's bytes with -`EncodeKey`), `RowSortField`, and `RowKeyEncoder`, which a writer uses to key a composite index -([10-indexes.md](10-indexes.md) §5.4). The code is `src/Vorticity.RowEncoding`, and -[row-keys.md](../guide/row-keys.md) is the guide's page. +- The encoder works a column at a time. It writes one field for every row, then the next field, + which suits branch prediction and vector width better than a row at a time, and is what the + two-pass sizing assumes. Big-endian conversion is a byte swap, and descending is a bitwise NOT over + a span. +- There is no allocation per row, since the output size is known before a byte is written. The + kernels call `TryWriteBigEndian`, never `WriteBigEndian`: the latter is a default interface method + the primitive types do not override, so a generic call to it boxes its receiver on every row. +- It is synchronous. It is CPU work over arrays already in memory, and the async-only rule is for + I/O. + +The public API is `RowEncoder` (a borrowed batch in, a pooled `RowKeys` out, and one key's bytes +with `EncodeKey`), `RowSortField`, and `RowKeyEncoder`, which a writer uses to key a composite index +(see [composite keys](10-indexes.md#54-composite-keys-through-the-row-encoding)). The code is in +`src/Vorticity.RowEncoding`, and [row-keys.md](../guide/row-keys.md) is the guide's page. diff --git a/docs/design/07-dotnet-mapping.md b/docs/design/07-dotnet-mapping.md index fa7085bd..91801859 100644 --- a/docs/design/07-dotnet-mapping.md +++ b/docs/design/07-dotnet-mapping.md @@ -1,7 +1,8 @@ # DType → .NET mapping -The type contract: what a caller receives for each Vortex dtype, and where a naive mapping would -lose data. The dtypes themselves are [02-format.md](02-format.md) §4. +The type contract: what a caller receives for each Vortex dtype, and where a naive mapping would lose +data. The dtypes themselves are described in [the format's DType +section](02-format.md#4-dtype-dtypefbs--dtypeproto). ## 1. The mapping @@ -9,112 +10,113 @@ What a record member and a `Column` may be, and what the column exposes: | dtype | `T` | what `Column` exposes | notes | |---|---|---|---| -| `Null` | none | — | the schema shows it; no `T` reads it | +| `Null` | none | nothing | the schema shows it, but no `T` reads it | | `Bool` | `bool` | `Bits` as `ReadOnlySpan`, an indexer | zero-copy | | `Primitive` i8…i64, u8…u64 | `sbyte`…`long`, `byte`…`ulong` | `Values` as `ReadOnlySpan`, an indexer | zero-copy | | `Primitive` u16 | `char` too | the same | the UTF-16 code unit, lone surrogates included | | `Primitive` i64, u64 | `nint`, `nuint` too, on a 64-bit host | the same | refused on a 32-bit host, whose native integer is narrower | -| `Primitive` i64 | `TimeSpan` too | an indexer, `CopyTo`, `Storage()` | 100 ns ticks: exact over the whole range | -| `Primitive` f16, f32, f64 | `Half`, `float`, `double` | the same | exact: `Half` is IEEE binary16 | -| `Decimal(p, s)` | `decimal` within §2's bounds, `VortexDecimal` always | an indexer, `Storage()`, `Scale` | never lossy: a column too wide for `decimal` is refused when it binds | -| `Decimal(p, 0)` | `Int128`, `UInt128` for `p ≤ 39`; `BigInteger` always | an indexer, `CopyTo`; `Values` zero-copy over 128-bit storage | §2.1 | -| `Utf8` | `string` | a UTF-8 span per value; `GetString(i)` on request | §4 | +| `Primitive` i64 | `TimeSpan` too | an indexer, `CopyTo`, `Storage()` | 100 ns ticks, exact over the whole range | +| `Primitive` f16, f32, f64 | `Half`, `float`, `double` | the same | exact, since `Half` is IEEE binary16 | +| `Decimal(p, s)` | `decimal` within [the bounds below](#2-decimal-why-systemdecimal-is-not-the-answer), `VortexDecimal` always | an indexer, `Storage()`, `Scale` | never lossy: a column too wide for `decimal` is refused when it binds | +| `Decimal(p, 0)` | `Int128`, `UInt128` for `p ≤ 39`, `BigInteger` always | an indexer, `CopyTo`, and `Values` zero-copy over 128-bit storage | see [wider integers](#21-integers-wider-than-64-bits) | +| `Utf8` | `string` | a UTF-8 span per value, `GetString(i)` on request | see [strings](#4-strings-spans-first-string-on-request) | | `Binary` | `ReadOnlyMemory`, `Memory`, `byte[]` | a span per value | | | `Struct` | a nested `[VortexRecord]` type | `Columns` | | | `List`, `FixedSizeList` | `ReadOnlyMemory`, `Memory`, `T[]`, `List`, `ImmutableArray`, an interface an array implements | a `Range` per row, `Elements` as `Column` | reading rows allocates one array, or one `List`, per list | -| `List` of `Struct` | any of the above over a `[VortexRecord]` type | `ListOf()`: a `Range` per row, `Elements` as `Columns` | the elements read in one pass, then cut per row | -| `Map` | `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | `MapOf()`: a `Range` per row, `Keys` and `Values` | scalar keys and values; a key is never null | +| `List` of `Struct` | any of the above over a `[VortexRecord]` type | `ListOf()`: a `Range` per row, `Elements` as `Columns` | the elements are read in one pass, then cut per row | +| `Map` | `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | `MapOf()`: a `Range` per row, `Keys` and `Values` | scalar keys and values, and a key is never null | | `vortex.date` | `DateOnly` | an indexer, `Storage()` or `Storage()` | | | `vortex.time` | `TimeOnly` | an indexer, `Storage()`, `Unit` | | -| `vortex.timestamp` | `DateTime` when naive or UTC, `DateTimeOffset` with a zone | an indexer, `Storage()`, `Unit`, `TimeZone` | §3 | +| `vortex.timestamp` | `DateTime` when naive or UTC, `DateTimeOffset` with a zone | an indexer, `Storage()`, `Unit`, `TimeZone` | see [temporal extensions](#3-temporal-extensions-one-resolution-at-binding) | | `vortex.uuid` | `Guid` | an indexer | the byte order is converted explicitly, never implied | -| an extension registered on the session | the registered type | an indexer, `Storage()` | [14-public-api.md](14-public-api.md) §2 | +| an extension registered on the session | the registered type | an indexer, `Storage()` | see [the session](14-public-api.md#2-the-session) | `Union` and `Variant` columns appear in a schema but no `T` maps to them. An `enum` member maps to its underlying integer. -**Nullability is carried by `T`**: `Column` is a nullable column and `Column` is -not, and a non-nullable value type over a nullable column is refused when it binds +Nullability is carried by `T`. `Column` is a nullable column and `Column` is not, +and a non-nullable value type over a nullable column is refused when it binds (`VortexSchemaException`). The `T?` names the column, not its storage: `Values` is still the -`ReadOnlySpan` of the slots, a null slot holds an unspecified value, and `ValidityWords` -gives the validity as 64-bit words, empty when every row is valid, so a caller skips per-row checks -on the common case. Only the indexer and row materialization produce a `double?` per value. +`ReadOnlySpan` of the slots, where a null slot holds an unspecified value, and +`ValidityWords` gives the validity as 64-bit words, empty when every row is valid, so a caller can +skip per-row checks in the common case. Only the indexer and row materialization produce a `double?` +per value. ## 2. Decimal: why `System.Decimal` is not the answer -Vortex allows a precision up to **76**, backed by `i256`; `System.Decimal` holds 28 to 29 -significant digits. A default mapping to `decimal` would be **silently lossy over a legal range of -the format**, which is the one failure this library must not have. +Vortex allows a precision of up to 76, backed by `i256`, while `System.Decimal` holds 28 to 29 +significant digits. A default mapping to `decimal` would be silently lossy over a legal range of the +format, which is the one failure this library must not have. So a `decimal` is accepted where it cannot lose anything, and refused everywhere else. A `decimal` -record member, `Column`, `Sym` or `ColumnBuilder` binds to a column -whose precision is at most 28 and whose scale lies in `[0, 28]`, which `System.Decimal` holds -exactly. The check is made once, when the record or the column binds, and a wider column is refused -there with `VortexSchemaException` naming `VortexDecimal`, never rounded value by value. -`VortexDecimal` (`Types/Numerics/VortexDecimal.cs`) binds to every decimal column and is the only -mapping of one beyond those bounds; it converts to `decimal` or `Int128` when the value fits and -formats exactly always. Written from a record, a `decimal` member declares `Decimal(28, 10)` unless -`[VortexColumn]` gives its precision and scale. - -The exact bounds, from the reference and transcribed in -[spec/METADATA.md](../../spec/METADATA.md): +record member, `Column`, `Sym` or `ColumnBuilder` binds to a column whose +precision is at most 28 and whose scale lies in `[0, 28]`, which `System.Decimal` holds exactly. The +check is made once, when the record or the column binds, and a wider column is refused there with a +`VortexSchemaException` naming `VortexDecimal`, never rounded value by value. `VortexDecimal` +(`Types/Numerics/VortexDecimal.cs`) binds to every decimal column and is the only mapping for one +beyond those bounds. It converts to `decimal` or `Int128` when the value fits, and always formats +exactly. Written from a record, a `decimal` member declares `Decimal(28, 10)` unless `[VortexColumn]` +gives its precision and scale. + +The exact bounds, from the reference and transcribed in [spec/METADATA.md](../../spec/METADATA.md): * `1 ≤ precision ≤ 76` * `scale ≤ 76` -* `scale ≤ precision` **only when `scale > 0`**: a negative scale is legal, bounded only by `i8` +* `scale ≤ precision` only when `scale > 0`. A negative scale is legal, bounded only by `i8` A reader that refuses a precision above 38, or requires `scale ≥ −precision`, refuses legal files. -**Storage width** is carried in `vortex.decimal`'s metadata as `values_type`: `i8` (0), `i16` (1), -`i32` (2), `i64` (3), `i128` (4), `i256` (5). The precision selects the *smallest* legal width — -1–2 → `i8`, 3–4 → `i16`, 5–9 → `i32`, 10–18 → `i64`, 19–38 → `i128`, 39–76 → `i256` — but a file may -declare any **wider** one, and often does: upstream allows precision-2 values in an `i256` buffer, -and a decimal column from Arrow or Parquet arrives as `i128` or `i256` whatever its precision. The -width is therefore read from `values_type`, and only a width *narrower* than the precision requires -is refused; `Storage()` exposes what the file declared, for callers who do their own -arithmetic without widening. +The storage width is carried in `vortex.decimal`'s metadata as `values_type`: `i8` (0), `i16` (1), +`i32` (2), `i64` (3), `i128` (4), `i256` (5). The precision selects the smallest legal width (1–2 → +`i8`, 3–4 → `i16`, 5–9 → `i32`, 10–18 → `i64`, 19–38 → `i128`, 39–76 → `i256`), but a file may +declare any wider one, and often does. Upstream allows precision-2 values in an `i256` buffer, and a +decimal column from Arrow or Parquet arrives as `i128` or `i256` whatever its precision. The width is +therefore read from `values_type`, and only a width narrower than the precision requires is refused. +`Storage()` exposes what the file declared, for callers who do their own arithmetic without +widening. -.NET has `Int128` and no `Int256`, so this library carries an internal `Int256` with what the -decimal path needs: comparison, negation, two's-complement bytes in both endiannesses and exact -text. It is correctness insurance, not a hot path: precision ≤ 38 is what files hold in practice. +.NET has `Int128` but no `Int256`, so this library carries an internal `Int256` with what the decimal +path needs: comparison, negation, two's-complement bytes in both endiannesses and exact text. It is +correctness insurance, not a hot path, since precision ≤ 38 is what files hold in practice. -The row encoder's narrower table (up to `i128`, no `Decimal256`) is `vortex-row`'s own limit, not -the format's ([06-row-encoding.md](06-row-encoding.md) §6). +The row encoder's narrower table (up to `i128`, no `Decimal256`) is `vortex-row`'s own limit, not the +format's (see [unsupported types](06-row-encoding.md#6-unsupported-types)). ### 2.1 Integers wider than 64 bits -No primitive dtype is wider than 64 bits, so an `Int128`, a `UInt128` or a `BigInteger` is a -decimal of scale 0: the unscaled value is the integer. The default width is the choice that matters: +No primitive dtype is wider than 64 bits, so an `Int128`, a `UInt128` or a `BigInteger` is stored as +a decimal of scale 0, whose unscaled value is the integer. The default width is the choice that +matters: -* **`Int128` and `UInt128` declare `Decimal(38, 0)`**, stored in 128 bits. Every reader of a - 128-bit decimal reads it, Arrow's `Decimal128` among them, and `Values` is the stored buffer with - no copy. Thirty-eight digits are not every value of either type, whose extremes have 39, so a - value past them is refused when it is written -- never wrapped, never rounded -- with the way - out in the message: `[VortexColumn(Precision = 39)]`, stored in 256 bits. -* **`BigInteger` declares `Decimal(76, 0)`**, as far as a decimal goes; a value past it is refused. +* `Int128` and `UInt128` declare `Decimal(38, 0)`, stored in 128 bits. Every reader of a 128-bit + decimal reads it, Arrow's `Decimal128` among them, and `Values` is the stored buffer with no copy. + Thirty-eight digits do not cover every value of either type, whose extremes have 39, so a value + beyond them is refused when it is written, never wrapped or rounded, and the message gives the way + out: `[VortexColumn(Precision = 39)]`, stored in 256 bits. +* `BigInteger` declares `Decimal(76, 0)`, as far as a decimal goes, and a value beyond it is refused. -Reading, `Int128` and `UInt128` bind to a decimal of scale 0 and at most 39 digits, whatever width -stores it. Thirty-eight digits always fit; a 39th can hold a value past the type, which is refused -as it is read, as a timestamp past `DateTime` is (§3), and a negative value is refused by a -`UInt128`. `BigInteger` binds to any decimal of scale 0 and reads every value. +On the read side, `Int128` and `UInt128` bind to a decimal of scale 0 and at most 39 digits, whatever +width stores it. Thirty-eight digits always fit. A 39th digit can hold a value beyond the type, which +is refused as it is read (the same way a timestamp beyond `DateTime` is), and a `UInt128` refuses a +negative value. `BigInteger` binds to any decimal of scale 0 and reads every value. -The aggregates follow: a sum over such a column is exact in 320 bits whatever its row count, and -converts to the type asked for, `OverflowException` when it does not fit; a sum as `BigInteger` -never throws. +The aggregates follow suit. A sum over such a column is exact in 320 bits whatever its row count, and +converts to the type asked for, throwing `OverflowException` when it does not fit. A sum as +`BigInteger` never throws. ## 3. Temporal extensions: one resolution, at binding -`vortex.timestamp` carries a unit and a time zone in its metadata. Resolving an IANA zone takes the -operating system's time-zone database through `TimeZoneInfo`, which makes the result -**machine-dependent**. +`vortex.timestamp` carries a unit and a time zone in its metadata. Resolving an IANA zone goes +through the operating system's time-zone database via `TimeZoneInfo`, which makes the result +machine-dependent. So the zone is resolved once, when a `DateTimeOffset` member or column binds to the file, by `TimeZoneInfo.FindSystemTimeZoneById`, and never per value. A naive or UTC timestamp binds as -`DateTime`, with `Kind` `Unspecified` or `Utc`, and needs no database. A zone the host cannot -resolve refuses the `DateTimeOffset` binding with `VortexSchemaException`; the column still binds -as `long`, its storage, and `Storage()`, `Unit` and `TimeZone` let a caller convert with a zone of -its own choosing. The machine dependence is confined to one call, at one moment, with a refusal -rather than a guess. +`DateTime`, with `Kind` `Unspecified` or `Utc`, and needs no database. A zone the host cannot resolve +makes the `DateTimeOffset` binding fail with `VortexSchemaException`. The column still binds as +`long`, its storage, and `Storage()`, `Unit` and `TimeZone` let a caller convert with a zone of its +own choosing. The machine dependence is confined to one call, at one moment, with a refusal rather +than a guess. ## 4. Strings: spans first, `string` on request @@ -122,17 +124,17 @@ A `vortex.varbinview` value is a 16-byte view into the file's buffers, so the na UTF-8 span, and a `string` is an allocation the caller asks for (`GetString`). [text-columns.md](../guide/text-columns.md) measures the difference. -The lifetime rule is the sharp edge: **a span borrowed from a batch is invalid once the enumeration -moves on.** `Columns` and `Column` are `ref struct`s, so the compiler keeps them inside -the loop body and out of an `await`, and analyzer VX1001 flags a span stored outside it. A caller -who needs the values later copies them, or takes an owned `RecordBatch`. +The lifetime rule is the sharp edge: a span borrowed from a batch becomes invalid once the +enumeration moves on. `Columns` and `Column` are `ref struct`s, so the compiler keeps +them inside the loop body and out of an `await`, and analyzer VX1001 flags a span stored outside it. +A caller who needs the values later copies them, or takes an owned `RecordBatch`. ## 5. A root that is not a struct -A file's root dtype may be a bare `Float64`, a `Bool`, or any type, and the reader does not assume a -table. The schema of such a file is one field with an empty name, whose type is the root. A record -binds to the fields of a struct root only, and `Scan()` on any other root throws +A file's root dtype may be a bare `Float64`, a `Bool` or any other type, and the reader does not +assume a table. The schema of such a file is one field with an empty name, whose type is the root. A +record only binds to the fields of a struct root, and `Scan()` on any other root throws a `VortexSchemaException` pointing at the tool scan, which reads the root as its single column -([untyped-files.md](../guide/untyped-files.md)). Inventing a synthetic column name instead would -make a round trip through this writer produce another schema than its input, which is worse than an +([untyped-files.md](../guide/untyped-files.md)). Inventing a synthetic column name instead would make +a round trip through this writer produce a different schema from its input, which is worse than an awkward API. diff --git a/docs/design/08-semantics.md b/docs/design/08-semantics.md index a846f2ec..b6e6dcb2 100644 --- a/docs/design/08-semantics.md +++ b/docs/design/08-semantics.md @@ -1,197 +1,206 @@ # Query semantics: pruning, predicates, unknown components -The rules that decide whether a scan returns *correct* rows. None of them is visible in the byte -format, and each produces silently wrong results if guessed. Where a rule was settled by reading the -reference implementation rather than its prose, the source is cited. +The rules that decide whether a scan returns correct rows. None of them is visible in the byte +format, and each one produces silently wrong results if guessed. Where a rule was settled by reading +the reference implementation rather than its prose, the source is cited. ## 1. Statistic precision: what `Inexact` licenses -`ArrayStats` carries `min_precision` / `max_precision` with values `Exact` / `Inexact` -([02-format.md](02-format.md) §5.2). The reference defines the semantics unambiguously -(`vortex-array/src/expr/stats/precision.rs`): +`ArrayStats` carries `min_precision` and `max_precision`, with the values `Exact` and `Inexact` (see +[the array schema](02-format.md#52-schema-arrayfbs)). The reference defines their meaning without +ambiguity (`vortex-array/src/expr/stats/precision.rs`): > Inexact statistics form a range of possible values that the statistic could be. This is statistic > specific, for max this will be an upper bound. Meaning that the actual max in an array is > guaranteed to be less than or equal to the inexact value, but equal to the exact value. -So **`Inexact` is a conservative bound, not an unqualified approximation**: `min` is a lower bound -on the true minimum, `max` an upper bound on the true maximum. The consequences are exact: +So `Inexact` is a conservative bound, not a loose approximation: `min` is a lower bound on the true +minimum and `max` an upper bound on the true maximum. The consequences are precise: | Operation | `Exact` | `Inexact` | |---|---|---| -| Range pruning (`x > k` vs zone `max`) | legal | **legal** — the bound points the safe way | -| Equality shortcut (`min == max` ⇒ constant zone) | legal | forbidden | -| Reporting the value to the caller as a true min/max | legal | forbidden | +| Range pruning (`x > k` against the zone's `max`) | legal | legal, since the bound points the safe way | +| Equality shortcut (`min == max` means a constant zone) | legal | forbidden | +| Reporting the value to the caller as a true min or max | legal | forbidden | | Skipping decode because `min == max` | legal | forbidden | -A third state, `Absent`, exists in the reference and maps to `Inexact` with no value on the wire. -A statistic with no value licenses nothing. +A third state, `Absent`, exists in the reference and maps to `Inexact` with no value on the wire. A +statistic with no value licenses nothing. -**The invariant that governs every pruning decision**, and the one the property test asserts: +One invariant governs every pruning decision, and it is the one the property test asserts: > Pruning may never eliminate a row that full materialization would have returned. The converse, pruning too little, only costs time. That asymmetry is what makes the invariant -testable: every filter test runs with pruning and indexes on and off, and the row sets must be -equal, including over deliberately loose but legal `Inexact` statistics. +testable: every filter test runs with pruning and indexes on and off, and the row sets must be equal, +including over deliberately loose but legal `Inexact` statistics. ## 2. NaN -The reference computes `min`/`max` with `NumericalAggregateOpts::skip_nans()` +The reference computes `min` and `max` with `NumericalAggregateOpts::skip_nans()` (`vortex-array/src/stats/expr.rs`), and tracks NaN separately in the `nan_count` statistic. So: -* **Zone `min`/`max` exclude NaN.** A zone containing NaN still reports the min/max of its +* Zone `min` and `max` exclude NaN. A zone containing NaN still reports the min and max of its non-NaN values. -* Filter expressions follow **IEEE 754**, which is more precise than "NaN never matches". The five - ordering-and-equality predicates — `==`, `<`, `<=`, `>`, `>=` — are all **false** when either - operand is NaN, `NaN == NaN` included. **`!=` is the exception**: it is defined as the negation of - `==`, so `NaN != x` is **true**, for every `x`, NaN included. C, C#, Rust and SQL all agree on - this; a reader that makes `!=` false for NaN "for consistency" drops rows every other - implementation returns. -* This is all distinct from the row encoding, which deliberately defines a *total* order over floats - where NaN is ordered by raw bit pattern ([06-row-encoding.md](06-row-encoding.md) §3). The two - orderings serve different purposes and must not be unified — an implementer who reuses the - row-encoding comparator for filter evaluation introduces a correctness bug. -* Therefore a zone with `max <= 10` may be pruned for `x > 10` **even when `nan_count > 0`**, - because the NaN rows would not have matched anyway. This is only sound given the two facts - above; it is written down here so it is not rediscovered by guesswork. +* Filter expressions follow IEEE 754, which is more precise than "NaN never matches". The five + ordering and equality predicates (`==`, `<`, `<=`, `>`, `>=`) are all false when either operand is + NaN, `NaN == NaN` included. `!=` is the exception: it is defined as the negation of `==`, so + `NaN != x` is true for every `x`, NaN included. C, C#, Rust and SQL all agree on this, and a reader + that makes `!=` false for NaN "for consistency" drops rows every other implementation returns. +* All of this is distinct from the row encoding, which deliberately defines a total order over + floats where NaN is ordered by its raw bit pattern (see [per-type + encodings](06-row-encoding.md#3-per-type-encodings)). The two orderings serve different purposes + and must not be unified. An implementer who reuses the row-encoding comparator to evaluate filters + introduces a correctness bug. +* Therefore a zone with `max <= 10` may be pruned for `x > 10` even when `nan_count > 0`, because the + NaN rows would not have matched anyway. This is only sound given the two facts above, and it is + written down here so it is not rediscovered by guesswork. ## 3. Three-valued logic Predicates evaluate over `{true, false, unknown}`, SQL-style: * A comparison with a null operand yields `unknown`. -* `unknown AND false = false`; `unknown OR true = true`; `NOT unknown = unknown`. -* A row is returned only when the filter evaluates to `true` — `unknown` does not match. -* `IS NULL` / `IS NOT NULL` never yield `unknown`. -* `ListContains(list, v)` is `unknown` on a null list and under a null `v`; a null element matches - nothing and leaves the row `false` ([12-index-reads.md](12-index-reads.md) §6). - -A literal of a type the column cannot compare to is **not a silent non-match**: it is refused -before a row is read. On the typed path it does not compile, because `Sym` compares only with a -`T` or with another column of the same `T`. On the tool path `Where` throws +* `unknown AND false = false`, `unknown OR true = true`, and `NOT unknown = unknown`. +* A row is returned only when the filter evaluates to `true`, so `unknown` does not match. +* `IS NULL` and `IS NOT NULL` never yield `unknown`. +* `ListContains(list, v)` is `unknown` on a null list and under a null `v`. A null element matches + nothing and leaves the row `false` (see [predicates on text and + lists](12-index-reads.md#6-predicates-on-text-and-lists)). + +A literal of a type the column cannot be compared with is not a silent non-match. It is refused +before a row is read. On the typed path it does not compile, because `Sym` only compares with a +`T` or with another column of the same `T`. On the tool path, `Where` throws a `VortexSchemaException` naming the column, its type and the literal, and a hole of an interpolated -filter throws as it is appended; two columns compared with each other must be of the same type, or -`Where` throws the same exception. What looks like a mismatch and is a conversion: text compared -with a date, time, timestamp, uuid or decimal column is parsed in the column's type, and refused -when it does not parse; a number compared with a decimal column is scaled exactly from its digits, -and a value the column's scale cannot hold makes an equality false rather than rounding into a -match. On the typed path a comparison with `null` is nullity, never `unknown`: `== null` is -`IS NULL`, `!= null` is `IS NOT NULL`, an ordering against `null` matches nothing, and a `null` among -the values of `In` is ignored. - -Pruning consequence: a zone where `null_count == row_count` can be skipped for any predicate that -is not satisfiable by nulls — that is, anything except `IS NULL` and expressions reducible to it. -This is one of the highest-value pruning rules in practice and it depends entirely on the 3VL -choice above being fixed. +filter throws as it is appended. Two columns compared with each other must have the same type, or +`Where` throws the same exception. + +Some apparent mismatches are conversions. Text compared with a date, time, timestamp, uuid or +decimal column is parsed in the column's type, and refused when it does not parse. A number compared +with a decimal column is scaled exactly from its digits, and a value the column's scale cannot hold +makes an equality false rather than rounding into a match. On the typed path a comparison with +`null` tests nullity and is never `unknown`: `== null` is `IS NULL`, `!= null` is `IS NOT NULL`, an +ordering against `null` matches nothing, and a `null` among the values of `In` is ignored. + +For pruning, a zone where `null_count == row_count` can be skipped for any predicate that nulls +cannot satisfy, which means anything except `IS NULL` and expressions reducible to it. This is one of +the most valuable pruning rules in practice, and it depends entirely on the three-valued logic above +being fixed. ## 4. Unknown components: resolve lazily, fail on use -[02-format.md](02-format.md) §5.3 covers the *contract* rule (a frozen ID means a frozen payload -shape). This section covers the *resolution* rule, which decides how long a reader survives format -evolution. +[Per-encoding metadata](02-format.md#53-per-encoding-metadata) covers the contract rule: a frozen id +means a frozen payload shape. This section covers the resolution rule, which decides how long a +reader survives the format's evolution. -**Unknown component IDs are resolved at open time into a known/unknown-inert classification, but -only fail when the component is actually required.** +Unknown component ids are classified at open time as known or unknown and inert, but only fail when +the component is actually needed. | Situation | Behavior | |---|---| -| Unknown array encoding in a column that is not projected | scan succeeds | +| Unknown array encoding in a column that is not projected | the scan succeeds | | Unknown array encoding in a projected column | `VortexUnsupportedException(id, ComponentKind.Array)` | -| Unknown layout encoding on a subtree never visited | scan succeeds | +| Unknown layout encoding on a subtree never visited | the scan succeeds | | Unknown layout encoding on the path to projected data | throws | -| Unknown extension dtype in the schema | schema exposes it as opaque; throws only if that field is read | -| Unknown zone-map aggregate | pruning for that aggregate is disabled; scan succeeds | +| Unknown extension dtype in the schema | the schema exposes it as opaque, and only reading that field throws | +| Unknown zone-map aggregate | pruning for that aggregate is disabled, and the scan succeeds | The aggregate row is already mandated by the format spec. Extending the same principle to arrays, layouts and dtypes is what lets a reader keep working when upstream freezes a new core edition and -the default writer starts emitting a new ID in *one* column of a fifty-column file. An -open-time hard failure would make that file entirely unreadable for no reason. +the default writer starts emitting a new id in one column of a fifty-column file. A hard failure at +open would make that file entirely unreadable for no reason. -Lazy resolution is unconditional, and there is no option to fail at open instead: the inspection -such an option would give is always there. `VortexFile.ArrayEncodings` and `LayoutEncodings` list -what the footer declares, each with whether this build supports it, and `vxdump` prints them. +Lazy resolution is unconditional, and there is no option to fail at open instead, because the +inspection such an option would give is always available. `VortexFile.ArrayEncodings` and +`LayoutEncodings` list what the footer declares, each with whether this build supports it, and +`vxdump` prints them. -Every such exception must name the component ID and kind, because that is exactly the input the -upstream troubleshooting procedure requires ("which edition, which minimum library version"). +Every such exception must name the component id and its kind, because that is exactly the input the +upstream troubleshooting procedure asks for ("which edition, which minimum library version"). ### Binary preservation on rewrite Preserving unknown nodes byte for byte through a read-modify-write cycle is a separate feature with invariants of its own: which segments are copied verbatim, what happens when the surrounding layout -changes, how edition enforcement applies to a component nobody can name. It is not offered; a +changes, how edition enforcement applies to a component nobody can name. It is not offered, and a rewrite fails on the component it cannot decode. ## 5. Untrusted hints: a three-class policy -Offsets and lengths are validated ([03-architecture.md](03-architecture.md) §6). The subtler class -is *semantic hints* the file asserts and a decoder might believe. Every field falls into exactly -one class: - -**Class I — memory safety depends on it. Validated unconditionally, whatever the cost.** -`DictMetadata.values_len` vs the actual values length and every code in `[0, values_len)`; -`BitPackedMetadata.bit_width` ≤ the ptype width; `BitPackedMetadata.offset` < 1024; -`RLEMetadata.offset` < 1024; `PatchesMetadata` indices in range; ALP `exp_e`/`exp_f` within the -target float's domain; `BoolMetadata.offset` < 8; a `vortex.listview` row's `offset + size` ≤ -`elements_len`, added **unsigned** so two legal i64 operands cannot wrap past the bound; a -`vortex.varbinview` reference view's 4-byte prefix equal to the value it points at; every buffer -index in `ArrayNode.buffers`; every segment index in `Layout.segments`; `alignment_exponent` ≤ 6 -(see §6). - -**Class II — only result correctness depends on it.** Monotonicity of `RunEndMetadata` ends, -`is_sorted`, `is_strict_sorted`, zone-map min/max. By default they are **not** validated, and the -threat model says so ([09-contracts.md](09-contracts.md) §4): a well-formed file that lies produces -wrong results, as in every format with embedded statistics, Parquet included. -`VortexOpenOptions.VerifyStatistics` validates them in O(n) at first decode, for files from +Offsets and lengths are always validated (see [parser safety](03-architecture.md#6-parser-safety)). +The subtler class is the semantic hints a file asserts and a decoder might believe. Every such field +falls into exactly one of three classes. + +Class I holds the fields memory safety depends on, and they are validated unconditionally, whatever +the cost: + +- `DictMetadata.values_len` against the actual values length, and every code in `[0, values_len)` +- `BitPackedMetadata.bit_width` ≤ the ptype width, and `BitPackedMetadata.offset` < 1024 +- `RLEMetadata.offset` < 1024 +- `PatchesMetadata` indices in range +- ALP `exp_e` and `exp_f` within the target float's domain +- `BoolMetadata.offset` < 8 +- a `vortex.listview` row's `offset + size` ≤ `elements_len`, added unsigned so two legal i64 + operands cannot wrap past the bound +- a `vortex.varbinview` reference view's 4-byte prefix, which must equal the value it points at +- every buffer index in `ArrayNode.buffers` and every segment index in `Layout.segments` +- `alignment_exponent` ≤ 6 (see [resource caps](#6-resource-caps)) + +Class II holds the fields only result correctness depends on: the monotonicity of `RunEndMetadata` +ends, `is_sorted`, `is_strict_sorted`, and zone-map min and max. By default they are not validated, +and the threat model says so (see [the threat model](09-contracts.md#4-threat-model)). A well-formed +file that lies produces wrong results, as in every format with embedded statistics, Parquet +included. `VortexOpenOptions.VerifyStatistics` validates them in O(n) at first decode, for files from producers a caller does not trust. -The distinction matters because a lying `is_sorted` cannot corrupt memory but a lying +The distinction matters because a lying `is_sorted` cannot corrupt memory, while a lying `values_len` can. Conflating the two either costs throughput everywhere or leaves a hole. -**Class III — pure performance hints.** `is_constant`, `uncompressed_size_in_bytes`, -`null_count` used only for fast paths. Ignored unless free to verify; never a correctness input. +Class III holds pure performance hints: `is_constant`, `uncompressed_size_in_bytes`, and +`null_count` when used only for fast paths. They are ignored unless free to verify, and are never a +correctness input. -A dedicated fuzzing mutation targets Class I fields specifically — a generic bit-flip fuzzer -almost never produces a *structurally valid* file with an out-of-range `values_len`, which is -precisely the input that finds these bugs. +A dedicated fuzzing mutation targets class I fields specifically. A generic bit-flip fuzzer almost +never produces a structurally valid file with an out-of-range `values_len`, which is precisely the +input that finds these bugs. ## 6. Resource caps -`VortexFormatException` on violation. Every cap is a constant, documented here so it is not +A violation throws `VortexFormatException`. Every cap is a constant, documented here so it is not invented per call site: | Cap | Value | Why | |---|---|---| -| `alignment_exponent` | ≤ 6 (64 bytes) | the field is a `u8`; `2^255` is expressible. 64 covers every legitimate alignment including 16-byte views | +| `alignment_exponent` | ≤ 6 (64 bytes) | the field is a `u8`, so `2^255` is expressible. 64 covers every legitimate alignment, including 16-byte views | | DType nesting depth | 64 | a 10 000-deep nested struct blows the stack during schema parsing, before any data is touched | | Layout tree depth | 64 | same | | Array tree depth | 64 | same | | Postscript length | 65527 (format-mandated) | enforced on write too, with an early, clear error | -| Decompressed segment size | configurable, default 256 MiB | a 1 KiB Zstd segment can claim to expand to 100 GiB | -| Metadata segment count | 16 (format-mandated) | — | -| FlatBuffers tables per traversal | 1 000 000 | depth cannot bound work: forward-only uoffsets exclude cycles but not *sharing*, so a few hundred bytes of shared children describe a DAG with 2^depth paths. Matches the reference verifiers' `max_tables` | +| Decompressed segment size | configurable, 256 MiB by default | a 1 KiB Zstd segment can claim to expand to 100 GiB | +| Metadata segment count | 16 (format-mandated) | | +| FlatBuffers tables per traversal | 1 000 000 | depth cannot bound work. Forward-only uoffsets exclude cycles but not sharing, so a few hundred bytes of shared children describe a DAG with 2^depth paths. Matches the reference verifiers' `max_tables` | -The decompression cap shapes the *design*, not just a constant: `ZstdDecompressor.Decompress` -decodes a whole frame into a pre-sized destination, so the declared size is validated against the -cap **before** allocating. A `vortex.zstd` frame must declare its content size, which has to match -its metadata; a `vortex.zstd_buffers` buffer is sized by the node's metadata. +The decompression cap shapes the design, not just a constant. `ZstdDecompressor.Decompress` decodes +a whole frame into a pre-sized destination, so the declared size is validated against the cap before +anything is allocated. A `vortex.zstd` frame must declare its content size, which has to match its +metadata, and a `vortex.zstd_buffers` buffer is sized by the node's metadata. ## 7. Buffer-level LZ4 -`Buffer.compression` admits `LZ4` ([02-format.md](02-format.md) §5.2), and the segment table -reserves a compression byte too. **A compressed buffer is refused**, with -`VortexUnsupportedException` naming the id `lz4` and the kind `compression`, and nothing is written -compressed. - -There is nothing to be conformant with. In the whole of Vortex 0.86.1, `lz4` appears in the two -schemas that declare the enum and the generated code that mirrors them, and nowhere else: the -reference writes `Compression::None`, never reads the field, and depends on no LZ4 implementation. -The schema names an algorithm and stops: it does not say whether the bytes would be a raw LZ4 block -or a frame, and it records no decompressed length anywhere, so a decoder could not even size its -output without inventing a rule. Writing one would mean choosing a framing and calling it the -format, the same mistake as inventing an order for `List` in the row encoder -([06-row-encoding.md](06-row-encoding.md) §6). - -Refusing is also the safer behaviour: the reference, which never inspects the field, would read -compressed bytes as data and return wrong values without a word. The decision changes the day -upstream implements it and a real file shows the framing. +`Buffer.compression` admits `LZ4` (see [the array schema](02-format.md#52-schema-arrayfbs)), and +the segment table reserves a compression byte too. A compressed buffer is refused with a +`VortexUnsupportedException` naming the id `lz4` and the kind `compression`, and nothing is ever +written compressed. + +There is nothing to conform to. In the whole of Vortex 0.86.1, `lz4` appears in the two schemas that +declare the enum and in the generated code that mirrors them, and nowhere else. The reference writes +`Compression::None`, never reads the field, and depends on no LZ4 implementation. The schema names an +algorithm and stops there: it does not say whether the bytes would be a raw LZ4 block or a frame, and +it records no decompressed length anywhere, so a decoder could not even size its output without +inventing a rule. Writing one would mean choosing a framing and calling it the format, the same +mistake as inventing an order for `List` in the row encoder (see [unsupported +types](06-row-encoding.md#6-unsupported-types)). + +Refusing is also the safer behaviour. The reference, which never inspects the field, would read +compressed bytes as data and silently return wrong values. The decision will change the day upstream +implements it and a real file shows the framing. diff --git a/docs/design/09-contracts.md b/docs/design/09-contracts.md index 59fa7684..8de44459 100644 --- a/docs/design/09-contracts.md +++ b/docs/design/09-contracts.md @@ -1,156 +1,157 @@ # Library contracts: concurrency, parallelism, versioning, threat model, observability, licensing -The promises made to a caller, as opposed to the promises made about bytes, written once so that -each is not deduced differently at each call site. +The promises made to a caller, as opposed to the promises made about bytes, written down once so +that each one is not deduced differently at each call site. ## 1. Concurrency and thread safety | type | contract | |---|---| -| `VortexSession` | **Thread-safe**, meant to be shared by every request of a process; its options are frozen when `Create` returns, and `VortexSession.Default` is immutable. Disposing it while a file it opened is still open throws `InvalidOperationException` naming the file | -| `VortexFile` | **Thread-safe.** Concurrent scans of one open file are expected; its footer, layout tree, index directory and the runs it caches are immutable or published once | -| `ISegmentSource` | implementations **must** be thread-safe; the built-in ones are | -| `Scan`, `Scan` | **Single use, one thread.** Composition returns the same builder; build it on one thread, then run one sink. A second sink throws `InvalidOperationException`; `ExplainAsync` may be asked before the sink | -| `GroupedScan`, `Aggregation`, `Aggregation`, `Projection`, `Projection`, the `Scan` of a result | **Single use, one thread**, like the scan they are built on: enumerating one runs its query once, and a second sink throws `InvalidOperationException` | +| `VortexSession` | thread-safe, and meant to be shared by every request of a process. Its options are frozen when `Create` returns, and `VortexSession.Default` is immutable. Disposing it while a file it opened is still open throws `InvalidOperationException` naming the file | +| `VortexFile` | thread-safe. Concurrent scans of one open file are expected, and its footer, layout tree, index directory and the runs it caches are immutable or published once | +| `ISegmentSource` | implementations must be thread-safe, and the built-in ones are | +| `Scan`, `Scan` | single use, one thread. Composition returns the same builder, so build it on one thread, then run one sink. A second sink throws `InvalidOperationException`. `ExplainAsync` may be called before the sink | +| `GroupedScan`, `Aggregation`, `Aggregation`, `Projection`, `Projection`, the `Scan` of a result | single use, one thread, like the scan they are built on. Enumerating one runs its query once, and a second sink throws `InvalidOperationException` | | the scan's enumerator | one consumer, as the language requires | -| `Columns`, `Column`, `BatchView` | **Bound to the loop body**, enforced by the compiler: `ref struct`s valid until the next `MoveNextAsync` or `DisposeAsync`, which cannot be stored, captured or carried across an `await`. The one escape the compiler does not see, a span copied into a variable declared outside the loop, is analyzer VX1001's | -| `RecordBatch` | **Owned by its consumer**, not thread-safe: it may be handed to another thread, through a channel for instance, and disposed there; its spans die with it | +| `Columns`, `Column`, `BatchView` | bound to the loop body, enforced by the compiler. They are `ref struct`s valid until the next `MoveNextAsync` or `DisposeAsync`, and cannot be stored, captured or carried across an `await`. The one escape the compiler does not see, a span copied into a variable declared outside the loop, is caught by analyzer VX1001 | +| `RecordBatch` | owned by its consumer, and not thread-safe. It may be handed to another thread (through a channel, for instance) and disposed there, and its spans become invalid with it | | `KeyCursorBuilder`, `KeyCursor` | not thread-safe, like the scan | -| `ColumnsBuilder`, `ColumnBuilder` | **One thread.** The builder a writer hands out is its own, reused after every `WriteAsync` | -| `VortexFileWriter` (an append included), `VortexFileIndexer` | one writer per file; an append is not atomic, and `VortexFileRepair` truncates a torn one | +| `ColumnsBuilder`, `ColumnBuilder` | one thread. The builder a writer hands out belongs to the writer and is reused after every `WriteAsync` | +| `VortexFileWriter` (an append included), `VortexFileIndexer` | one writer per file. An append is not atomic, and `VortexFileRepair` truncates a torn one | | `ScanPlan`, `CountPlan`, `OrderPlan`, `KeyPlan`, `ScanStatistics`, `WriteReport` | immutable records | -| decoders and kernels | pure functions over borrowed memory, no shared mutable state | +| decoders and kernels | pure functions over borrowed memory, with no shared mutable state | ## 2. Parallelism -A ratio against Rust means something only at equal threading, and a library must not take a host's -thread pool without consent: a server running 200 concurrent requests does not want each scan +A ratio against Rust only means something at equal threading, and a library must not take over a +host's thread pool without consent: a server running 200 concurrent requests does not want each scan fanning out. So: -- **Sequential by default.** `VortexSessionOptions.MaxDegreeOfParallelism` is 1 unless set, and +- The default is sequential. `VortexSessionOptions.MaxDegreeOfParallelism` is 1 unless set, and `VortexSession.Default` is immutable, so a caller who configures nothing gets one thread. -- **Consent is given on the session, once**: its degree is where every scan and every writer of the - session starts, for a host that decides its threading at start-up rather than at a hundred call - sites. Two sessions in one process keep two degrees. -- **The call may say otherwise**: `ScanOptions.DegreeOfParallelism` for one scan and - `VortexWriteOptions.DegreeOfParallelism` for one file, 0 taking the session's. -- **A scan** decodes its splits side by side, each on a context and arenas of its own, nothing - shared, and still delivers its batches in row order. A take, whose splits are many and small, - keeps up to three a lane decoded ahead of its consumer, so the lanes that finish first do not - wait for the slowest; its degree still bounds how many decode at once. **A query** that streams - runs on that ordered stream of batches, its decode parallel; a blocking aggregation keeps a state - per group on each lane and merges the lanes' states part by part of the key space, as tasks it - awaits, never a thread blocked on another. The target, [16-queries.md](16-queries.md) §9.4 and - §9.5, folds each lane's rows into a cache of bounded capacity and holds every group once, in the - sub-tables of 256 parts. Its answers are the same bits at every degree, float sums included, and - stay so at every capacity of the caches and every order the states merge in. **A dataset** opens its objects ahead of the one being read and reads them side by - side. **A writer** summarizes its columns and compresses a - column's zstd frames on its threads, and chooses and writes the encodings on the calling one; the - file is the same bytes at any degree. -- **I/O concurrency is separate and always on**: the batch read of the seam issues overlapping reads - whatever the degree, because latency hiding is the point on an object store. It is bounded per - session by `MaxConcurrentReads`, 16 by default, across every scan of every file. A mapped file and - bytes in memory have no reads to bound. - -[threads.md](../guide/threads.md) shows what each costs and when parallelism pays. +- Consent is given once, on the session. Its degree is where every scan and every writer of the + session starts, which suits a host that decides its threading at start-up rather than at a hundred + call sites. Two sessions in one process keep two separate degrees. +- A single call may override it: `ScanOptions.DegreeOfParallelism` for one scan and + `VortexWriteOptions.DegreeOfParallelism` for one file, where 0 takes the session's. +- A scan decodes its splits side by side, each on a context and arenas of its own with nothing + shared, and still delivers its batches in row order. A take, whose splits are many and small, keeps + up to three per lane decoded ahead of its consumer, so the lanes that finish first do not wait for + the slowest, and its degree still bounds how many decode at once. +- A streaming query runs on that ordered stream of batches, with its decoding in parallel. A blocking + aggregation keeps a state per group on each lane and merges the lanes' states one part of the key + space at a time, as tasks it awaits, never a thread blocked on another. The design (see + [parallelism and the merge](16-queries.md#94-parallelism-and-the-merge) and + [memory](16-queries.md#95-memory)) folds each lane's rows into a cache of bounded capacity and holds + every group once, in sub-tables of 256 parts. Its answers are the same bits at every degree, float + sums included, and stay so at every cache capacity and every merge order. +- A dataset opens its objects ahead of the one being read, and reads them side by side. +- A writer summarizes its columns and compresses a column's zstd frames on its threads, and chooses + and writes the encodings on the calling one. The file is the same bytes at any degree. +- I/O concurrency is separate and always on. The batch read of the I/O interface issues overlapping + reads whatever the degree, because hiding latency is the point on an object store. It is bounded + per session by `MaxConcurrentReads`, 16 by default, across every scan of every file. A mapped file + and bytes in memory have no reads to bound. + +[threads.md](../guide/threads.md) shows what each one costs and when parallelism pays off. ## 3. Versioning -Semantic versioning, with each level spelled out, since a format library's version has two axes: -its API and the format it understands. +The library follows semantic versioning, with each level spelled out, since a format library's +version has two axes: its API and the format it understands. | change | level | |---|---| | reading a newly frozen upstream edition (more ids) | minor | | an encoder for an encoding already decoded | minor | | a case that threw `VortexFormatException` now throwing `VortexUnsupportedException` | minor | -| dropping or adding a target framework | **major** | -| changing the lifetime of borrowed spans | **major** | -| changing the default write edition | **major**: it changes who can read the output | -| the row encoding's bytes following upstream | not applicable, below | - -The default write edition is `VortexEditions.Default`, and it is not the newest by definition: it is -the edition the most deployed Rust reader accepts, chosen per release and named in the release -notes beside the minimum Rust version that reads it (`VortexEditions.MinimumRustVersion`). It is -`core2026.08.3`, read from Vortex Rust 0.85.0: the first edition with `vortex.uuid`, without -which a `Guid` column cannot be written. - -**The row encoding ships apart**, in `Vorticity.RowEncoding`, versioned `0.x` and marked -experimental: upstream reserves the right to change its bytes between releases, and binding it into +| dropping or adding a target framework | major | +| changing the lifetime of borrowed spans | major | +| changing the default write edition | major, since it changes who can read the output | +| the row encoding's bytes following upstream | not applicable, see below | + +The default write edition is `VortexEditions.Default`, and it is not the newest by definition. It is +the edition the most widely deployed Rust reader accepts, chosen per release and named in the release +notes next to the minimum Rust version that reads it (`VortexEditions.MinimumRustVersion`). Today it +is `core2026.08.3`, read by Vortex Rust 0.85.0 and later, the first edition with `vortex.uuid`, +without which a `Guid` column cannot be written. + +The row encoding ships separately, in `Vorticity.RowEncoding`, versioned `0.x` and marked +experimental. Upstream reserves the right to change its bytes between releases, and binding it into a stable core would force a choice between breaking this library's semver and diverging from Rust, -which would lose the byte-exactness that is the feature's point. +which would lose the byte-exactness that is the whole point of the feature. Each release records the Vortex version its corpus was generated with and the highest edition it -reads: that pair, more than the version number, is what a user needs. +reads. That pair, more than the version number, is what a user needs. ## 4. Threat model -Two sentences, because the alternative is an unbounded promise: +It fits in two sentences, because the alternative is an unbounded promise: -1. **Malformed input** — bad offsets, truncation, out-of-range fields memory depends on, exceeded - caps — produces a clean `VortexFormatException` or `VortexUnsupportedException`: no out-of-bounds - access, no unbounded allocation, no hang. Fuzzing enforces it. -2. **Well-formed but lying input** — valid structure over false statistics — may produce wrong - results, as in every format with embedded statistics, Parquet included. +1. Malformed input (bad offsets, truncation, out-of-range fields that memory depends on, exceeded + caps) produces a clean `VortexFormatException` or `VortexUnsupportedException`, with no + out-of-bounds access, no unbounded allocation and no hang. Fuzzing enforces it. +2. Well-formed but lying input (a valid structure over false statistics) may produce wrong results, + as in every format with embedded statistics, Parquet included. `VortexOpenOptions.VerifyStatistics` trades throughput for validation when the producer is not - trusted ([08-semantics.md](08-semantics.md) §5). An index that lies costs pruning, never rows: - its regions carry checksums, and a failed one claims nothing - ([10-indexes.md](10-indexes.md) §3). + trusted (see [untrusted hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). An index + that lies costs pruning, never rows: its regions carry checksums, and a region that fails claims + nothing (see [the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)). Not claimed: protection against resource exhaustion below the configured caps, or any confidentiality property (the format's encryption slot is an empty reserved table). ## 5. Observability -Three channels, each free when nobody listens. +There are three channels, each free when nobody listens. -**A meter and an activity source**, both named `Vorticity` (`VortexDiagnostics`), which +The first is a meter and an activity source, both named `Vorticity` (`VortexDiagnostics`), which OpenTelemetry and `dotnet-counters` read without an adapter. Each instrument is created on its own, -so a listener enabling one pays for that one: +so a listener that enables one only pays for that one: | instrument | diagnoses | |---|---| | `vortex.scan.rows` | rows delivered | -| `vortex.scan.requests`, `vortex.scan.bytes_requested` | what the scans asked their sources for: read amplification, against the plan's `BytesToRead` | +| `vortex.scan.requests`, `vortex.scan.bytes_requested` | what the scans asked their sources for, which shows read amplification against the plan's `BytesToRead` | | `vortex.scan.blocks_decoded`, `vortex.scan.blocks_pruned` | whether pruning works at all | | `vortex.cache.hits`, `vortex.cache.misses` | whether the session's segment cache does anything | -| `vortex.write.bytes` | bytes the writers handed their sinks | +| `vortex.write.bytes` | bytes the writers handed to their sinks | -A scan is one activity, `vortex.scan.typed` or `vortex.scan.tool`, tagged when it ends with its -rows, batches, requests, bytes requested and blocks decoded and pruned; a write is one activity, -`vortex.write`, tagged with its rows, bytes and whether it completed, and marked as an error when it -is abandoned. +A scan is one activity, `vortex.scan.typed` or `vortex.scan.tool`, tagged when it ends with its rows, +batches, requests, bytes requested, and blocks decoded and pruned. A write is one activity, +`vortex.write`, tagged with its rows, its bytes and whether it completed, and marked as an error when +it is abandoned. -**An event source**, `Vorticity`, whose polling counters say whether an index earns its bytes: -`segments-requested`, `bytes-requested`, `zones-pruned`, `zones-total`, `index-runs-read`, +The second is an event source, `Vorticity`, whose polling counters say whether an index earns its +bytes: `segments-requested`, `bytes-requested`, `zones-pruned`, `zones-total`, `index-runs-read`, `cursor-seeks`, `cursor-steps`, `count-blocks-proven`, `count-blocks-decoded`, `key-order-windows` and `key-order-window-splits`. -**Per query**: a scan's `ExplainAsync` is its plan before it runs, and its `Statistics` the same -quantities measured after ([observability.md](../guide/observability.md)). +The third works per query: a scan's `ExplainAsync` is its plan before it runs, and its `Statistics` +the same quantities measured afterwards ([observability.md](../guide/observability.md)). ## 6. Licensing and attribution -Vorticity is licensed under Apache-2.0 (`LICENSE`, the canonical text). +Vorticity is licensed under Apache-2.0 (`LICENSE` holds the canonical text). -Upstream material arrives under **two** licenses, which Vortex's `REUSE.toml` splits: +Upstream material arrives under two licenses, which Vortex's `REUSE.toml` splits: | upstream material | license | what this repository does with it | |---|---|---| -| source tree, including the `.fbs` and `.proto` schemas and the format constants | Apache-2.0 | schemas vendored verbatim under `spec/`; constants transcribed into `docs/` | -| documentation at docs.vortex.dev and upstream `docs/` | **CC BY 4.0** | condensed and adapted throughout `docs/` | +| source tree, including the `.fbs` and `.proto` schemas and the format constants | Apache-2.0 | schemas vendored verbatim under `spec/`, constants transcribed into `docs/` | +| documentation at docs.vortex.dev and upstream `docs/` | CC BY 4.0 | condensed and adapted throughout `docs/` | -Two obligations bind, and both are met in `NOTICE`: +Two obligations apply, and both are met in `NOTICE`: -1. **Apache-2.0 §4(c)**: the vendored schemas keep their SPDX copyright and license headers, and +1. Apache-2.0, section 4(c): the vendored schemas keep their SPDX copyright and license headers, and `NOTICE` lists each vendored file against its upstream path. -2. **CC BY 4.0**: adapting the upstream documentation requires attribution, a link to the license, - and an **indication that changes were made**, which citing the source alone does not give. +2. CC BY 4.0: adapting the upstream documentation requires attribution, a link to the license, and + an indication that changes were made, which citing the source alone does not provide. -`NOTICE` also records, per component, whether it was **reimplemented from the specification** or -**translated from reference code**; only the second inherits the upstream license's attribution -requirements. Reimplementation from the specification is the rule, with the reference consulted -for semantics. +`NOTICE` also records, per component, whether it was reimplemented from the specification or +translated from reference code. Only the second inherits the upstream license's attribution +requirements. Reimplementation from the specification is the rule, with the reference consulted for +semantics. "Vortex" is a trademark of LF Projects, LLC. Vorticity is an independent implementation, not affiliated with or endorsed by the Vortex project, and says so at the top of `NOTICE` and of the diff --git a/docs/design/10-indexes.md b/docs/design/10-indexes.md index 26e4bb5a..4aef39fb 100644 --- a/docs/design/10-indexes.md +++ b/docs/design/10-indexes.md @@ -1,326 +1,334 @@ # Skipping and locating indexes What an index is in a Vorticity file: its kinds, its bytes, where it lives, how it survives an -append, and how it stays invisible to every other reader. What a caller can *ask* of an index — a -cursor, a count, key-ordered rows — is [12-index-reads.md](12-index-reads.md); how the writer builds -one in its single pass is [11-write-strategy.md](11-write-strategy.md). - -An index is an **optional accelerator**. A reader that ignores it returns the same rows as a reader -that uses it, and [08-semantics.md](08-semantics.md) §1's invariant — pruning never removes a row a -full scan returns — holds for every kind. It is also **ahead of upstream**: Vortex 0.86.1 ships a -Bloom filter aggregate its authors call unstable and no edition declares, and its index designs are -proposals. This design takes their structure, and adds one rule they have not fixed: **a file with -an index stays readable by a strict Rust 0.86.1 reader, with the index simply invisible to it.** -That rule decides where an index lives (§2), and the rest follows. +append, and how it stays invisible to every other reader. What a caller can ask of an index (a +cursor, a count, key-ordered rows) is covered in [12-index-reads.md](12-index-reads.md), and how the +writer builds one in its single pass in [11-write-strategy.md](11-write-strategy.md). + +An index is an optional accelerator. A reader that ignores it returns the same rows as a reader that +uses it, and the pruning invariant (pruning never removes a row a full scan returns, see [statistic +precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)) holds for every kind. It is +also ahead of upstream. Vortex 0.86.1 ships a Bloom filter aggregate its authors call unstable and no +edition declares, and its index designs are still proposals. This design takes their structure and +adds one rule they have not fixed: a file with an index stays readable by a strict Rust 0.86.1 +reader, with the index simply invisible to it. That rule decides [where an index +lives](#2-where-an-index-lives), and the rest follows. ## 1. Terms | term | meaning | |---|---| -| **skipping index** | a conservative summary per region, "no row here can match": it clears bits of a block mask | -| **locating index** | a value-keyed structure, "the rows or blocks where this value occurs": it may answer a predicate outright | -| **exactness** | `Exact`: the rows it names are exactly the matching ones. `Superset`: what it clears is proven, what it keeps is "maybe" | -| **granularity** | a row, a block (the zone length, 8 192 rows by default, counted from row 0), a node of 16 blocks, the file | -| **run** | an immutable payload over a contiguous range of blocks; an index is a sequence of runs | +| skipping index | a conservative summary per region, "no row here can match", which clears bits of a block mask | +| locating index | a value-keyed structure, "the rows or blocks where this value occurs", which may answer a predicate outright | +| exactness | `Exact` means the rows it names are exactly the matching ones. `Superset` means what it clears is proven and what it keeps is "maybe" | +| granularity | a row, a block (the zone length, 8 192 rows by default, counted from row 0), a node of 16 blocks, or the file | +| run | an immutable payload over a contiguous range of blocks. An index is a sequence of runs | -An index is a **kind** (a versioned string), a **column path**, a **block length** and its -**runs**, which are ordinary Vortex arrays. Everything a reader needs to decide whether it can use -an index is in its directory entry; everything it needs to probe it is in the runs. +An index is a kind (a versioned string), a column path, a block length and its runs, which are +ordinary Vortex arrays. Everything a reader needs to decide whether it can use an index is in its +directory entry, and everything it needs to probe it is in the runs. ## 2. Where an index lives -**The rule.** A file with an index opens and scans, with no configuration, in Rust 0.86.1 with a -strict session (the default, and what DuckDB, DataFusion, Spark and the Python bindings use), in -any earlier Vorticity, and in this one, which then uses the index. +The rule is that a file with an index opens and scans, with no configuration, in Rust 0.86.1 with a +strict session (the default, and what DuckDB, DataFusion, Spark and the Python bindings use), in any +earlier Vorticity, and in this one, which then uses the index. -**The carrier is a postscript metadata entry.** The postscript already carries up to sixteen -`(key, segment)` pairs that every reader loads only on request ([02-format.md](02-format.md) §2). -One entry, `vorticity.index`, holds the **index directory** (§3). The directory points at **run -segments**: regions written after the data and before the footer, 64-byte aligned, referenced by -neither the layout tree nor the footer's segment table. A reader that does not ask for the key never -touches them; Rust validates that the directory's own segment lies inside the file, which it does, -and does not know the runs exist. A second entry, `vorticity.identity`, is the file's identity -([13-dataset.md](13-dataset.md) §7), which leaves fourteen for the caller. +The carrier is a postscript metadata entry. The postscript already carries up to sixteen +`(key, segment)` pairs that every reader only loads on request (see [the +postscript](02-format.md#2-postscript-footerfbs)). One entry, `vorticity.index`, holds the index +directory ([below](#3-the-directory-and-the-runs)). The directory points at run segments: regions +written after the data and before the footer, 64-byte aligned, referenced by neither the layout tree +nor the footer's segment table. A reader that does not ask for the key never touches them. Rust +checks that the directory's own segment lies inside the file, which it does, and does not know the +runs exist. A second entry, `vorticity.identity`, is the file's identity (see [identity and +integrity](13-dataset.md#7-identity-and-integrity)), which leaves fourteen entries for the caller. -**The carriers not used:** +The carriers not used: | carrier | in a strict Rust 0.86.1 reader | verdict | |---|---|---| -| a new aggregate in `vortex.zoned` | the open fails: unknown aggregate | only once an edition declares one, with upstream's id and bytes (§8) | -| a wrapper layout, upstream's `vortex.indexed` | the open fails, and the data beneath is unreachable | the migration target when it freezes (§8) | -| a new field in the postscript table | ignored | refused: it would take a field id upstream may assign | -| a second object beside the file | unaffected | used, for stores that cannot append, as a fragment (§7) | +| a new aggregate in `vortex.zoned` | the open fails on an unknown aggregate | only once an edition declares one, with upstream's id and bytes ([the road upstream](#8-compatibility-and-the-road-upstream)) | +| a wrapper layout, upstream's `vortex.indexed` | the open fails, and the data beneath is unreachable | the migration target once it freezes ([the road upstream](#8-compatibility-and-the-road-upstream)) | +| a new field in the postscript table | ignored | refused, since it would take a field id upstream may assign | +| a second object next to the file | unaffected | used, for stores that cannot append, as a fragment ([indexing after the fact](#7-building-appending-and-indexing-after-the-fact)) | ## 3. The directory and the runs -**The directory** is a Protobuf message of this library's, preceded by a version byte as +The directory is a Protobuf message of this library's own, preceded by a version byte the way `vortex.zoned`'s metadata is, and followed by an XXH3-64 of both (`Indexes/IndexDirectory.cs`). It holds the file's row count when it was written, the write policy (so an append needs no options), -the index budget, and one entry per index: its kind, column path, block length, kind-defined -options, and runs. A run names its first block and block count, its payload regions — each with its -offset, length, alignment, **XXH3-64 checksum** and array dtype — its entry count for a locating -kind, and kind-defined options. +the index budget, and one entry per index: its kind, column path, block length, kind-defined options +and runs. A run names its first block and block count, its payload regions (each with its offset, +length, alignment, XXH3-64 checksum and array dtype), its entry count for a locating kind, and +kind-defined options. -**What a reader checks** before it believes an entry, under the hint policy of -[08-semantics.md](08-semantics.md) §5: +A reader checks the following before it believes an entry, under the policy for [untrusted +hints](08-semantics.md#5-untrusted-hints-a-three-class-policy): -- the directory's checksum, and that its row count is the file's: otherwise the whole directory is - ignored, and the file reads without it; +- the directory's checksum, and that its row count is the file's. Otherwise the whole directory is + ignored, and the file reads without it. - that the column path resolves to a column whose dtype the kind supports, that the kind and its - options are known, and that the runs are disjoint, in order and inside the file: otherwise the - **entry** is ignored, never the file; -- every region it reads, against its checksum: a region that fails claims nothing in a pruner and + options are known, and that the runs are disjoint, in order and inside the file. Otherwise the + entry is ignored, never the file. +- every region it reads, against its checksum. A region that fails claims nothing in a pruner, and makes a key source refuse. -**An index can only ever cost pruning, never correctness.** A block no run covers is live; a zeroed -filter is caught by its checksum rather than dropping rows. +An index can therefore only ever cost pruning, never correctness. A block no run covers stays live, +and a zeroed filter is caught by its checksum rather than dropping rows. -**Runs are ordinary arrays.** Each payload is an array blob, the form of a `vortex.flat` segment, +Runs are ordinary arrays. Each payload is an array blob, the same form as a `vortex.flat` segment, written with the file's encoding table and compressed like data under the same target edition. An index therefore inherits every encoding: a sorted key column bit-packs or FSST-compresses like any -column, and no index defines a binary format of its own, which is upstream's own design decision -for locating indexes. The footer may list an encoding used only by an index run, which is harmless, +column, and no index defines a binary format of its own, which matches upstream's own design decision +for locating indexes. The footer may list an encoding used only by an index run, which is harmless since every id is in the target edition. -**A run is immutable and covers a contiguous block range.** A write produces **one run per entry**: -chunk runs are spilled as they close and merged when the file completes, plus a run of its own for -the last chunk when the rows are not whole blocks, since an append re-opens that chunk. An append -adds runs, and merges an entry's runs once it holds more than **K = 4**, so a lookup reads at most K -runs. A rewrite writes one. +A run is immutable and covers a contiguous range of blocks. A write produces one run per entry. Chunk +runs are spilled as they close and merged when the file completes, plus a separate run for the last +chunk when the rows do not fill whole blocks, since an append reopens that chunk. An append adds runs, +and merges an entry's runs once it holds more than four, so a lookup reads at most four runs. A +rewrite writes a single run. -**Long runs are paged.** A locating run of at most 64 segments of 65 536 entries lists its segments -inline. A longer one writes its segment table as **fence pages** of at most 64 KiB, a tree whose root -is inline in the run's options and whose pages carry their children's checksums; two levels cover -more than 10¹² entries, so a lookup reads at most two fence pages before its payload +Long runs are paged. A locating run of at most 64 segments of 65 536 entries lists its segments +inline. A longer one writes its segment table as fence pages of at most 64 KiB, forming a tree whose +root is inline in the run's options and whose pages carry their children's checksums. Two levels +cover more than 10¹² entries, so a lookup reads at most two fence pages before its payload (`Indexes/FenceTable.cs`). -**Bloom filters form a tree.** A run's filters are laid out with a fan-out of 16: the leaves are -the per-block filters, an internal node filters the union of its 16 children and lists their sizes, -and the children of a node are contiguous behind it, so a probe reads the root and then one region -per node that still says "maybe", level by level, in one coalesced read per level -(`Indexes/BloomTree.cs`). A node whose union exceeds what `max_blocks` holds at the target rate is +Bloom filters form a tree. A run's filters are laid out with a fan-out of 16: the leaves are the +per-block filters, an internal node filters the union of its 16 children and lists their sizes, and +the children of a node sit contiguously behind it. A probe reads the root and then one region per +node that still says "maybe", level by level, in one coalesced read per level +(`Indexes/BloomTree.cs`). A node whose union exceeds what `max_blocks` can hold at the target rate is built without a filter, and the descent starts below it. ## 4. Skipping indexes -### 4.1 `vorticity.bloom.sbbf.v1` — a split-block Bloom filter per block +### 4.1 `vorticity.bloom.sbbf.v1`: a split-block Bloom filter per block -**Structure.** `n` blocks of 256 bits (eight 32-bit words). Insert: `h = XXH3-64(bytes, seed 0)`; -block `((h >> 32) × n) >> 32`; in it, set one bit per word with the eight Parquet salts applied to -`(uint)h`. Probe: the same eight bits all set. The layout and salts are Parquet's and the hash is -upstream's, so a filter is **bit-identical to Rust's `BloomPartial` of 0.86.1**, tested against -vectors it built, and becomes upstream's by a change of id the day they freeze it. A variant with -Parquet's xxHash64 is **bit-identical to a Parquet SBBF**, tested against the `parquet` crate's own -filters, for a caller who exchanges filters with Parquet tooling. +A filter is `n` blocks of 256 bits (eight 32-bit words). To insert, compute `h = XXH3-64(bytes, seed +0)`, pick block `((h >> 32) × n) >> 32`, and in it set one bit per word using the eight Parquet salts +applied to `(uint)h`. A probe checks that the same eight bits are all set. The layout and salts are +Parquet's and the hash is upstream's, so a filter is bit-identical to Rust's `BloomPartial` of +0.86.1, tested against vectors it built, and becomes upstream's by a change of id the day they freeze +it. A variant using Parquet's xxHash64 is bit-identical to a Parquet SBBF, tested against the +`parquet` crate's own filters, for a caller who exchanges filters with Parquet tooling. -**What is hashed**, so that equal values hash equal whatever encoding they came from: +What is hashed is chosen so that equal values hash equal whatever encoding they came from: | dtype | bytes hashed | |---|---| -| integers, floats, decimals | the value's little-endian bytes at the column's width; floats by bit pattern, so `-0.0` and `+0.0` are two values | +| integers, floats, decimals | the value's little-endian bytes at the column's width. Floats hash by bit pattern, so `-0.0` and `+0.0` are two values | | text, binary | the bytes | | extension | its storage value's bytes | -| the elements of a list | each element, into the **row's** block: "does any element of a row in this block equal v", the question `ListContains` asks | -| bool | not indexed: a two-value domain has nothing to skip | -| null | not inserted: `x = v` and `x IN (…)` never match a null | +| the elements of a list | each element, into the row's block, which answers "does any element of a row in this block equal v", the question `ListContains` asks | +| bool | not indexed, since a two-value domain has nothing to skip | +| null | not inserted, since `x = v` and `x IN (…)` never match a null | -**Size.** Built when a block closes, from its **exact** distinct count: the smallest power of two +A filter is built when a block closes, sized from its exact distinct count: the smallest power of two of blocks that gives the target false-positive rate (1 % by default) by Parquet's sizing formula, -clamped to `max_blocks` (4 096 blocks, 128 KiB). A block under `min_distinct` (8) distinct values -gets no filter: the zone map or the dictionary answers equality there. A tree node is sized from the -exact union of its blocks' values, never clamped into a saturated filter. The file-level filter is -the root of the tree when the policy asks for three resolutions (§4.4). +clamped to `max_blocks` (4 096 blocks, 128 KiB). A block with fewer than `min_distinct` (8) distinct +values gets no filter, since the zone map or the dictionary answers equality there. A tree node is +sized from the exact union of its blocks' values, never clamped into a saturated filter. The +file-level filter is the root of the tree when the policy asks for three resolutions (see [the +file-level filter](#44-the-file-level-filter)). -**Probe rules**, as a pruner after the zone maps in the scan's block mask -([11-write-strategy.md](11-write-strategy.md) §6): +The filters act as a pruner after the zone maps in the scan's block mask (see [reading what the +writer produced](11-write-strategy.md#6-reading-what-the-writer-produced)), with these rules: | predicate | proof | |---|---| | `x = v` | a block whose filter does not contain `v` is pruned | -| `x IN (v1 … vn)` | pruned if it contains none; the literals are hashed once per query | -| `AND`, `OR` | a conjunction of proofs; pruned under `OR` only if every arm prunes | +| `x IN (v1 … vn)` | pruned if the filter contains none of them. The literals are hashed once per query | +| `AND`, `OR` | a conjunction of proofs, and under `OR` a block is pruned only if every arm prunes it | | `ListContains(x, v)` | as `x = v`, over the element filters | -| `!=`, `<`, `LIKE` | no claim (§4.2 for `LIKE`) | +| `!=`, `<`, `LIKE` | no claim (see [trigrams](#42-vorticitybloomngram3v1-trigrams-for-text-predicates) for `LIKE`) | -A literal is hashed only when it converts exactly into the column's type — an integer past 2⁵³ from -a double, a non-integral float or an out-of-range integer claims nothing — a zero asks for both -zeros, and a NaN asks for nothing. +A literal is only hashed when it converts exactly into the column's type. An integer beyond 2⁵³ from +a double, a non-integral float or an out-of-range integer claims nothing, a zero asks for both zeros, +and a NaN asks for nothing. -### 4.2 `vorticity.bloom.ngram3.v1` — trigrams for text predicates +### 4.2 `vorticity.bloom.ngram3.v1`: trigrams for text predicates -The same structure over every byte **trigram** of every value of a text or binary column. +This is the same structure over every byte trigram of every value of a text or binary column. `StartsWith`, `Contains` and `LIKE` require the trigrams of each literal run of three bytes or more between their wildcards, and one absent trigram prunes the block. An option folds ASCII case on both -sides; a folded index still answers a case-sensitive predicate, since a value containing `Foo` -contains `foo` once folded. `Superset`: the predicate is always evaluated on the survivors. +sides. A folded index still answers a case-sensitive predicate, since a value containing `Foo` +contains `foo` once folded. It is a `Superset` index, so the predicate is always evaluated on the +survivors. -### 4.3 `vorticity.dict.probe.v1` — the dictionary is already an index +### 4.3 `vorticity.dict.probe.v1`: the dictionary is already an index -When the writer dictionary-encoded a chunk, the chunk's values child *is* the set of its distinct -values, exact. This kind has **no payload**: its entry says which chunks of the column are +When the writer dictionary-encoded a chunk, the chunk's values child already is the exact set of its +distinct values. This kind therefore has no payload. Its entry says which chunks of the column are dictionaries, and the pruner answers `x = v` and `x IN (…)` from those values without decoding the -codes, last in the chain since it reads the column's own bytes. A chunk claims only the blocks it -covers whole. It costs nothing to write. What it costs to read is the chunk's segment, since a flat -chunk is one segment: it pays when the probed column is one of several — on a 1M-row table of six -columns, an absent value reads 0.073× the bytes of the scan — and breaks even on a file of one +codes, last in the chain since it reads the column's own bytes. A chunk only claims the blocks it +covers whole. It costs nothing to write. Reading it costs the chunk's segment, since a flat chunk is +one segment, so it pays off when the probed column is one of several (on a 1M-row table of six +columns, an absent value reads 0.073× the bytes of the scan) and breaks even on a file with a single column. ### 4.4 The file-level filter -The root of each run's tree filters the whole run. A scan's plan reads the roots with the file -statistics, and `ScanPlan.MayMatch` is false when either proves the scan empty; `VortexFile.MayMatch` +The root of each run's tree filters the whole run. A scan's plan reads the roots along with the file +statistics, and `ScanPlan.MayMatch` is false when either proves the scan empty. `VortexFile.MayMatch` answers from the statistics alone, without a read, for an engine that prunes many files before -opening any. An append does not extend a tree, since filters of different sizes do not merge: it -adds a tree for its own blocks. +opening any. An append does not extend a tree, since filters of different sizes do not merge. +Instead it adds a tree for its own blocks. ## 5. Locating indexes -### 5.1 `vorticity.postings.blocks.v1` — value to blocks +### 5.1 `vorticity.postings.blocks.v1`: value to blocks -A run holds the distinct values of its range, sorted, and for each the blocks it occurs in: three -arrays, `keys`, `offsets` and `blocks`, compressed like data. A lookup locates `v` in the keys — -the run's segment table skips to the one segment that can hold it — and keeps exactly the blocks -listed. `Exact` at block granularity: a kept block holds the value, though not in every row. The -keys alone serve a walk of distinct values ([12-index-reads.md](12-index-reads.md) §2). +A run holds the distinct values of its range, sorted, and for each one the blocks it occurs in, as +three arrays (`keys`, `offsets` and `blocks`) compressed like data. A lookup locates `v` in the keys, +where the run's segment table skips straight to the one segment that can hold it, and keeps exactly +the blocks listed. It is `Exact` at block granularity: a kept block holds the value, though not in +every row. The keys alone serve a walk of distinct values (see [sources of +order](12-index-reads.md#2-sources-of-order)). -### 5.2 `vorticity.sorted.runs.v1` — value to rows, exact +### 5.2 `vorticity.sorted.runs.v1`: value to rows, exact -A run holds its range's keys sorted, with a row position per entry (`u32` relative to the run's -first row, `u64` when a run spans 2³² rows or more). It answers `x = v`, `x IN (…)` and ranges -**exactly**: a scan takes its rows as a selection and does not evaluate the predicate again. It is +A run holds its range's keys sorted, with a row position per entry (a `u32` relative to the run's +first row, or a `u64` when a run spans 2³² rows or more). It answers `x = v`, `x IN (…)` and ranges +exactly, so a scan takes its rows as a selection and does not evaluate the predicate again. It is also the key source of a cursor and of key-ordered delivery ([12-index-reads.md](12-index-reads.md)). -Keys are in the total order of [12-index-reads.md](12-index-reads.md) §3.4, and the run is built by a -sort of each chunk's distinct keys, a counting sort of its rows by rank, and a k-way merge of the -chunk runs: memory is one chunk and the merge's buffers, never the column. +Keys follow the total order described in [key order per +dtype](12-index-reads.md#34-key-order-per-dtype). The run is built by sorting each chunk's distinct +keys, counting-sorting its rows by rank, and k-way merging the chunk runs, so memory is one chunk plus +the merge's buffers, never the whole column. -### 5.3 `vorticity.postings.ngram3.v1` — trigrams to blocks +### 5.3 `vorticity.postings.ngram3.v1`: trigrams to blocks -§5.1 with the trigrams of a text column as keys, typed `binary` since a trigram may cut a UTF-8 -character: the locating form of §4.2, larger than the filter and exact at block granularity for the -conjunction of a pattern's trigrams. +This is the [postings index](#51-vorticitypostingsblocksv1-value-to-blocks) with the trigrams of a +text column as keys, typed `binary` since a trigram may cut a UTF-8 character. It is the locating +form of the [trigram filter](#42-vorticitybloomngram3v1-trigrams-for-text-predicates), larger than +the filter and exact at block granularity for the conjunction of a pattern's trigrams. ### 5.4 Composite keys, through the row encoding A sorted run over `(a, b, …)` keys each row by the row encoding of the tuple ([06-row-encoding.md](06-row-encoding.md)): bytes whose `memcmp` order is the tuple order, so a -prefix on the leading columns is a range. The entry keeps the kind, leaves its column path empty, -and lists its key columns in its options with the name of the encoding its bytes follow -(`vortex-row 0.86.1 …`), since that format is experimental upstream. The core ships no encoder: +prefix on the leading columns is a range. The entry keeps the kind, leaves its column path empty, and +lists its key columns in its options together with the name of the encoding its bytes follow +(`vortex-row 0.86.1 …`), since that format is experimental upstream. The core ships no encoder. `IndexPolicy.ForKey(columns, kind, encoder)` takes an `IKeyEncoder`, and the row-encoding package's -`RowKeyEncoder` is one. A tuple holding a null is no entry. +`RowKeyEncoder` is one. A tuple holding a null produces no entry. ### 5.5 Considered and not built -- **Hash to rows.** Sorted runs whose keys are 64-bit hashes: smaller than sorted keys only on - incompressible keys, where a Bloom filter is three times smaller again for the equality it would - answer; where keys have structure, the sorted keys compress below the hashes and stay exact. - Measured at four key shapes, it was never the best answer. A workload of row-level lookups on long - incompressible keys would reopen it. -- **A Bloom filter over column pairs.** A tuple's cardinality is the product of its columns', which - is exactly the regime where filters cost more than they save, and a composite sorted run answers - the same question exactly. +- Hash to rows: sorted runs whose keys are 64-bit hashes. They are smaller than sorted keys only on + incompressible keys, where a Bloom filter is three times smaller again for the equality they would + answer. Where keys have structure, the sorted keys compress below the hashes and stay exact. + Measured on four key shapes, it was never the best answer. A workload of row-level lookups on long + incompressible keys would reopen the question. +- A Bloom filter over column pairs. A tuple's cardinality is the product of its columns', which is + exactly the regime where filters cost more than they save, and a composite sorted run answers the + same question exactly. ## 6. The policy -`VortexWriteOptions.Indexes` is an `IndexPolicy`, **`None` by default**: a file written with the -defaults carries no index. A caller names indexes by column path, nested paths included — -`IndexPolicy.None.Bloom("id").SortedRuns("ts")` — or starts from `IndexPolicy.Auto` +`VortexWriteOptions.Indexes` is an `IndexPolicy`, `None` by default, so a file written with the +defaults carries no index. A caller names indexes by column path, nested paths included +(`IndexPolicy.None.Bloom("id").SortedRuns("ts")`), or starts from `IndexPolicy.Auto` (`Indexes/IndexPolicy.cs`, [indexes.md](../guide/indexes.md)). -**`Auto`** records the dictionary probe wherever it applies, since it is free, and starts a Bloom -filter on every top-level column, which it gives up — whole, with its reason, and always before its -payload is written, so an abandoned index leaves no byte — at the earliest fact that condemns it: +`Auto` records the dictionary probe wherever it applies, since it is free, and starts a Bloom filter +on every top-level column. It gives a filter up, whole and with its reason, at the earliest fact that +condemns it, and always before its payload is written, so an abandoned index leaves no byte behind: -- the column's first block climbs: the zone map already prunes it; +- the column's first block climbs, so the zone map already prunes it - its filters would pass 2 % of the column's raw bytes, checked at every block, or 2 % of its written - bytes, checked at every chunk; -- a full first node holds fewer distinct values than the floor, or its first four blocks hold one - and the same set, so that every filter says "maybe" for every value; -- its first node holds more values than a filter of `max_blocks` can, where a sorted run is the - structure. - -The hash per row is the whole cost of a live filter, which is why the verdicts come early: with them, -`Auto` adds 0.6 % to the write of a million-row table of six columns. Postings and sorted runs cost -a key table per row and are never `Auto`'s. - -**The file budget**, `IndexPolicy.WithBudgetPerMille`, is 100 by default: indexes may add a tenth of -the data's bytes. It is judged before payloads are written, once the data passes 1 MiB, and drops -optional indexes first; an index whose payloads are already written is kept rather than left as -dead bytes. - -**A required index** (`required: true`) is never dropped by the budget to make room. If it cannot -be built — its builder gave it up, or the required indexes alone exceed the budget — + bytes, checked at every chunk +- a full first node holds fewer distinct values than the floor, or its first four blocks hold one and + the same set, so every filter would say "maybe" for every value +- its first node holds more values than a filter of `max_blocks` can, which is where a sorted run is + the right structure + +The hash per row is the whole cost of a live filter, which is why the verdicts come early. With them, +`Auto` adds 0.6 % to the write of a million-row table of six columns. Postings and sorted runs cost a +key table per row and are never chosen by `Auto`. + +The file budget, `IndexPolicy.WithBudgetPerMille`, is 100 by default, meaning indexes may add a tenth +of the data's bytes. It is judged before payloads are written, once the data passes 1 MiB, and drops +optional indexes first. An index whose payloads are already written is kept rather than left as dead +bytes. + +A required index (`required: true`) is never dropped by the budget to make room. If it cannot be +built, because its builder gave it up or the required indexes alone exceed the budget, `CompleteAsync` throws a `VortexException` naming it and the reason before the file's tail is written. The flag is stored in the directory, so an append that reuses the policy keeps it. A -dataset's clustering key is such an index ([13-dataset.md](13-dataset.md) §5.3), which the dataset -writes spared by the budget: kept when it alone is over it, and reported with what it cost, since a -run on a narrow table is as large as the column it orders and no object is big enough to bring it -under a share of the data. +dataset's clustering key is such an index (see [what a compaction +does](13-dataset.md#53-what-a-compaction-does)), and the dataset writes it exempt from the budget: it +is kept even when it alone is over the budget, and reported with what it cost, since a run on a +narrow table is as large as the column it orders and no object is big enough to bring it under a +share of the data. The `WriteReport` names every index built, with its bytes, and every index abandoned, with its reason. ## 7. Building, appending, and indexing after the fact -**Building.** The builders are fed by the writer's pass over each block: a Bloom builder hashes its -rows into an exact set per block and per open node, a key builder keeps its own table of the chunk's -keys and their positions. Filter runs are written between chunks as their nodes close; the merged -key runs, fence pages and upper tree levels after the last chunk, before the zone maps; the -directory last. Nothing is buffered across chunks but the open nodes and the spilled chunk runs, -which go to pooled memory and then to a scratch file. +The builders are fed by the writer's pass over each block. A Bloom builder hashes its rows into an +exact set per block and per open node, and a key builder keeps its own table of the chunk's keys and +their positions. Filter runs are written between chunks as their nodes close, then the merged key +runs, fence pages and upper tree levels after the last chunk, before the zone maps, and the directory +last. Nothing is buffered across chunks except the open nodes and the spilled chunk runs, which go to +pooled memory and then to a scratch file. -**Appending** ([11-write-strategy.md](11-write-strategy.md) §3.8). Runs that end before the -append's first block are kept and listed again beside the new ones; a run reaching into a re-opened -chunk is dropped and rebuilt; past K runs an entry's runs are merged. +When appending (see [appending to a file](11-write-strategy.md#38-appending-to-a-file)), runs that end +before the append's first block are kept and listed again next to the new ones, a run reaching into a +reopened chunk is dropped and rebuilt, and once an entry has more than four runs they are merged. -**Indexing a file that exists**, without touching a data byte: +A file that already exists can be indexed without touching a data byte: -- **In place**, on a file system: `VortexFileIndexer.AppendIndexesAsync(path, policy)` reads the - file, builds the runs, and appends them with a new directory, footer and postscript over the same - data. The old file is a byte prefix of the new one. -- **Outside the file**, on a store that cannot append: `VortexFileIndexer.BuildFragmentAsync(file, +- In place, on a file system: `VortexFileIndexer.AppendIndexesAsync(path, policy)` reads the file, + builds the runs, and appends them with a new directory, footer and postscript over the same data. + The old file is a byte prefix of the new one. +- Outside the file, on a store that cannot append: `VortexFileIndexer.BuildFragmentAsync(file, policy, rows?)` returns an `IndexFragment`, a container of runs and a directory over a range of - blocks, which a caller keeps where it likes and hands back at open through - `VortexOpenOptions.IndexFragments`. A fragment is **bound to the file's identity**, or to the - store's token for a file written without one, from bytes the open reads anyway; a content hash is - recorded, and checked only offline by `VortexFile.VerifyIndexesAsync` (`vxdump --fragment P - --verify`). Fragments add entries beside the file's own; an entry of the same identity joins its - runs when their blocks are disjoint. In a dataset, fragments live in commit objects - ([13-dataset.md](13-dataset.md) §6.4). - -**A key source needs complete coverage.** A pruner uses whatever blocks the runs cover and leaves -the others live; a cursor over partial runs would miss keys, so a key source refuses an entry whose + blocks, which a caller keeps where it likes and passes back at open through + `VortexOpenOptions.IndexFragments`. A fragment is bound to the file's identity, or to the store's + token for a file written without one, using bytes the open reads anyway. A content hash is also + recorded, and only checked offline by `VortexFile.VerifyIndexesAsync` (`vxdump --fragment P + --verify`). Fragments add entries next to the file's own, and an entry with the same identity joins + its runs when their blocks are disjoint. In a dataset, fragments live in commit objects (see + [fragments](13-dataset.md#64-fragments-in-commit-objects-transient)). + +A key source needs complete coverage. A pruner uses whatever blocks the runs cover and leaves the +others live, but a cursor over partial runs would miss keys, so a key source refuses an entry whose runs do not cover every block, and names the blocks it has. ## 8. Compatibility, and the road upstream | reader | a file with an index directory | |---|---| -| Rust 0.86.1, strict or permissive session | opens, scans, prunes with the zone maps; never fetches the entry | -| an earlier Vorticity | the same: the key is listed, never fetched | -| this Vorticity | uses the index; ignores an entry that fails a check (§3) | - -The file stays a pure core-edition file: every array in it, index runs included, carries an id of -the target edition, and the kinds are versioned strings inside the directory, which no edition -needs to know. When upstream freezes a Bloom aggregate in an edition, a file targeting it may carry -filters in the zone map as well, under upstream's id and bytes; when `vortex.indexed` freezes, a -locating index becomes its auxiliary child, the runs the same arrays and the directory entry its +| Rust 0.86.1, strict or permissive session | opens, scans and prunes with the zone maps, and never fetches the entry | +| an earlier Vorticity | the same: the key is listed but never fetched | +| this Vorticity | uses the index, and ignores an entry that fails a check ([the directory and the runs](#3-the-directory-and-the-runs)) | + +The file stays a pure core-edition file. Every array in it, index runs included, carries an id of the +target edition, and the kinds are versioned strings inside the directory, which no edition needs to +know. When upstream freezes a Bloom aggregate in an edition, a file targeting it may also carry +filters in the zone map, under upstream's id and bytes. When `vortex.indexed` freezes, a locating +index becomes its auxiliary child, with the runs as the same arrays and the directory entry as its metadata. Neither migration changes a payload byte. ## 9. How it is tested -- **Rust reads it**: the cross-check writes the whole corpus under rotating policies — every kind, - filters at three resolutions, runs cut into small segments, appends, indexing after the fact, a - dataset's compacted objects — and a strict Rust reader compares every value. -- **Equivalence**: every supported predicate with indexes on and off, on a file written once and on - the same rows appended in two, three and eleven pieces, returns the same rows. -- **Lying indexes**: a zeroed filter with its framing kept is caught by its checksum and drops no - row; torn fence pages, tree nodes and regions claim nothing (`LyingIndexTests`, `BloomTreeTests`, +- Rust reads it. The cross-check writes the whole corpus under rotating policies (every kind, filters + at three resolutions, runs cut into small segments, appends, indexing after the fact, a dataset's + compacted objects), and a strict Rust reader compares every value. +- Equivalence: every supported predicate, with indexes on and off, on a file written once and on the + same rows appended in two, three and eleven pieces, returns the same rows. +- Lying indexes: a zeroed filter with its framing kept is caught by its checksum and drops no row, + and torn fence pages, tree nodes and regions claim nothing (`LyingIndexTests`, `BloomTreeTests`, `FenceTreeTests`). -- **Bit-exactness**: 66 filters against Rust's `BloomPartial`, 26 against Parquet's `Sbbf`. -- **Allocation**: a ceiling per kind on the write (`IndexAllocationTests`). +- Bit-exactness: 66 filters against Rust's `BloomPartial`, and 26 against Parquet's `Sbbf`. +- Allocation: a ceiling per kind on the write (`IndexAllocationTests`). -The upstream designs this follows: the Bloom aggregate (vortex PR #9398, marked unstable in #9753), -the skip-index interface (#9413), the skipping and locating index epics (#8900, #8948, #9024), and -multi-resolution zone maps (#7939); Parquet's `BloomFilter.md` for the filter; O'Neil et al., *The -log-structured merge-tree*, 1996, for the runs. +The upstream designs this follows are the Bloom aggregate (vortex PR #9398, marked unstable in +#9753), the skip-index interface (#9413), the skipping and locating index epics (#8900, #8948, +#9024), and multi-resolution zone maps (#7939). Parquet's `BloomFilter.md` describes the filter, and +O'Neil et al., *The log-structured merge-tree*, 1996, the runs. diff --git a/docs/design/11-write-strategy.md b/docs/design/11-write-strategy.md index 4f2472e5..10d51ba3 100644 --- a/docs/design/11-write-strategy.md +++ b/docs/design/11-write-strategy.md @@ -1,284 +1,301 @@ # The writer: one pass per block -How a file is written: rows arrive in batches of any size, and each value is summarized once, -encoded once, and written once, with every pruning structure exact and every index built by the same -pass. Four constraints shape it, in this order: **throughput on one thread**, with the rest of the -session's threads used where the work splits cleanly; **pruning structures that are exact**, never -sampled; **indexes as outputs of the same pass**, across appends too -([10-indexes.md](10-indexes.md)); **no allocation per row, and no chunk-sized scratch per column**. +How a file is written. Rows arrive in batches of any size, and each value is summarized once, encoded +once and written once, with every pruning structure exact and every index built by the same pass. +Four constraints shape the writer, in this order: + +1. throughput on one thread, with the rest of the session's threads used where the work splits + cleanly +2. pruning structures that are exact, never sampled +3. indexes produced by the same pass, across appends too ([10-indexes.md](10-indexes.md)) +4. no allocation per row, and no chunk-sized scratch buffer per column ## 1. The principle -A value is **read twice and copied never** between its arrival and its emission. The statistics -pass reads it on the batch where the caller put it; the encoder of the scheme that won reads the -same rows again, from cache, while the chunk is still warm. One pass for everything is not possible -without guessing: every wire form but the dictionary carries a parameter known only once the chunk -is whole — the bit width, the frame of reference, ALP's exponents — and a guess from the previous -chunk would have to be checked and, when wrong, redone. The dictionary is the exception: it assigns -its codes in the first pass, and its encoder never reads the values again. - -Everything that must be exact — minimum, maximum, null count, the Bloom insertions, the keys behind -an index — is computed in the first pass over every row. Everything the chooser needs — runs, -order, progressions, bit-width histograms, distinct counts, byte lengths — is computed in the same -pass. The chooser then decides by **exact formulas** for every scheme whose cost is a function of -those statistics, and runs a **trial** only for the four whose cost is not (ALP, ALP-RD, FSST and -zstd), with the best cost in hand as its abort bound. The winner encodes into outputs sized exactly, -which become the segment's buffers. +Between its arrival and its emission, a value is read twice and never copied. The statistics pass +reads it in the batch where the caller put it, and the encoder of the winning scheme reads the same +rows again, from the CPU cache, while the chunk is still warm. A single pass for everything is not +possible without guessing: every wire form except the dictionary carries a parameter that is only +known once the chunk is complete (the bit width, the frame of reference, ALP's exponents), and a +guess from the previous chunk would have to be checked and, when wrong, redone. The dictionary is +the exception, because it assigns its codes in the first pass and its encoder never reads the values +again. + +Everything that must be exact (minimum, maximum, null count, the Bloom insertions, the keys behind an +index) is computed in the first pass over every row. Everything the chooser needs (runs, order, +progressions, bit-width histograms, distinct counts, byte lengths) is computed in the same pass. The +chooser then decides by exact formulas for every scheme whose cost is a function of those +statistics, and only runs a trial for the schemes whose cost is not: ALP, ALP-RD, FSST, OnPair and +zstd, plus pco under the `Smallest` profile, each with the best cost so far as its abort bound. The +winner encodes into outputs sized exactly, which become the segment's buffers. ## 2. Blocks and chunks -A **block** is 8 192 rows counted from row 0 of the file, whatever the caller's batching +A block is 8 192 rows counted from row 0 of the file, whatever the caller's batching (`VortexWriteOptions.BlockRows`). It is the zone of the zone map, upstream's row block, eight -FastLanes blocks of 1 024, and the unit of an index. A batch that straddles blocks produces -**partials** per column and block that merge exactly, so a block's statistics are final when its -last row arrives, whichever batch brings it. +FastLanes blocks of 1 024, and the unit of an index. A batch that straddles blocks produces partial +summaries per column and block that merge exactly, so a block's statistics are final when its last +row arrives, whichever batch brings it. -A **chunk** is a run of whole blocks, written as one segment per column. It stays a multiple of the +A chunk is a run of whole blocks, written as one segment per column. It stays a multiple of the block length, because a chunk edge inside a block would put a zone edge inside a segment, where a -pruned zone saves no I/O. Its size weighs four costs: a read pays a fixed cost per chunk, which stops -mattering between 65 536 and 131 072 rows of an eight-byte column; a dictionary pays only when a -chunk holds many more rows than distinct values; a filtered or indexed read fetches the whole chunk -of every column it keeps; and an encoding that rides on local ranges loses when a chunk spans -several. So by default a chunk holds as many whole blocks as keep its **widest column within a -megabyte** of values, between one block and 128, with all its columns within 64 MiB. -`ChunkTargetBytes` fixes a byte target instead, and `ColumnChunkTargetBytes` gives one column -chunks of its own, spanning whole chunks of the file: a dictionary paid once for more rows, while -the other columns keep what a selective read fetches (§8). - -A batch that is a multiple of the block length and fills a chunk is encoded where it lies; any other +pruned zone saves no I/O. Its size balances four costs. A read pays a fixed cost per chunk, which +stops mattering somewhere between 65 536 and 131 072 rows of an eight-byte column. A dictionary only +pays off when a chunk holds many more rows than distinct values. A filtered or indexed read fetches +the whole chunk of every column it keeps. And an encoding that relies on local ranges loses when a +chunk spans several. So by default a chunk holds as many whole blocks as keep its widest column +within a megabyte of values, between one block and 128, with all its columns within 64 MiB. +`ChunkTargetBytes` sets a byte target instead, and `ColumnChunkTargetBytes` gives one column chunks +of its own, spanning whole chunks of the file, which pays for a dictionary once over more rows while +the other columns keep what a selective read fetches (see [the advisor](#8-choosing-encodings-ahead-the-advisor)). + +A batch that is a multiple of the block length and fills a chunk is encoded where it lies. Any other batch waits in the builder's buffers for the rows that complete its blocks. `FlushAsync` seals the -whole blocks into a chunk; `CompleteAsync` writes the rest as the tail. A caller who writes in -multiples of `BlockRows` gets a file whose every chunk ends on a block, which appends without -rewriting anything (§3.8). +whole blocks into a chunk, and `CompleteAsync` writes the rest as the tail. A caller who writes in +multiples of `BlockRows` gets a file in which every chunk ends on a block, which can be appended to +without rewriting anything (see [appending to a file](#38-appending-to-a-file)). ## 3. The pipeline, stage by stage ### 3.1 One column writer per leaf -The writer keeps the segment list, a small shared scratch, and a tree of column writers mirroring -the schema, one per physical leaf: a struct field, a list's elements, an extension's storage. A -column writer holds the open block's partials, one 56-byte summary per closed block until the zone -map is written, its plan memory (§3.4), its index builders, and while it runs the chunk's distinct -table. A nested column's blocks are defined by its parent's rows: block `i` of a list's elements -covers the elements of the parent's rows of block `i`, so a filter over the elements answers "does -any element of a row in this block match". +The writer keeps the segment list, a small shared scratch area, and a tree of column writers that +mirrors the schema, one per physical leaf: a struct field, a list's elements, an extension's storage. +A column writer holds the open block's partial summaries, one 56-byte summary per closed block until +the zone map is written, its plan memory (see [choosing](#34-choose-exact-verdicts-then-bounded-trials)), +its index builders, and, while it runs, the chunk's distinct table. A nested column's blocks are +defined by its parent's rows. Block `i` of a list's elements covers the elements of the parent's rows +in block `i`, so a filter over the elements answers "does any element of a row in this block match". ### 3.2 The statistics pass -Per column and block, one loop, resolved on the physical type before it starts and walking the -validity as bitmap words, never a row at a time. It keeps: +Per column and block there is one loop, resolved on the physical type before it starts, that walks +the validity as bitmap words, never a row at a time. It keeps: | what | exact | for | |---|---|---| -| minimum and maximum; NaN skipped, `-0.0` and `+0.0` told apart by their bits | yes | the zone map, frame of reference, progressions, constants | +| minimum and maximum, NaN skipped, `-0.0` and `+0.0` told apart by their bits | yes | the zone map, frame of reference, progressions, constants | | null count | yes | the zone map, all-null chunks, every cost formula | | whether the values ascend, strictly or not | yes | the file statistics, progressions, run-end | | a progression's step, and whether it broke | yes | `vortex.sequence` | | the count of runs | yes | run-end | | bit-width histograms, raw and zigzag | yes | bit-packing with patches, kept only where such a packing is held | | distinct values, through the distinct table while it runs | yes | the dictionary | -| text and binary byte totals; bounded minimum and maximum | yes, the bounds as bounds | varbin against views, the trials' ceilings, the zones of a text column | +| text and binary byte totals, bounded minimum and maximum | yes, the bounds as bounds | varbin against views, the trials' ceilings, the zones of a text column | -It runs on the batch's own memory, while the decode that produced a rewrite's batch is still in -cache, with no copy before it. +It runs on the batch's own memory, with no copy before it, and when the batch comes from decoding a +file being rewritten, while that decode is still in the CPU cache. ### 3.3 The distinct table -One open-addressing table per column, **running over a chunk**, maps a value to its code, assigned -in first-seen order, and writes each row's code as it arrives. Equality is on bytes, never on the -hash alone, and the table **owns** its keys: a block straddles batches, and the previous batch's -memory is recycled the moment the next one is decoded, so a string is copied once, the first time it -is seen, into a heap that is, in code order, the dictionary's values child. +There is one open-addressing table per column, running over a chunk, that maps a value to its code, +assigned in first-seen order, and writes each row's code as it arrives. Equality is checked on bytes, +never on the hash alone, and the table owns its keys. A block straddles batches, and the previous +batch's memory is recycled the moment the next one is decoded, so a string is copied once, the first +time it is seen, into a heap that is, in code order, the dictionary's values child. -Probing every row of a column whose values are all distinct costs a few nanoseconds a row for -nothing, so the table runs **only on a chunk whose remembered plan is a dictionary that held**. A -column's first chunk, and a chunk whose memory sends it back to full pricing, count their distinct -values in the chooser's own walk, which gives up early on a column where a dictionary loses; the -table then runs for the next chunk if the dictionary won. +Probing every row of a column whose values are all distinct costs a few nanoseconds per row for +nothing, so the table only runs on a chunk whose remembered plan is a dictionary that held. A +column's first chunk, and a chunk whose plan memory sends it back to full pricing, count their +distinct values in the chooser's own walk, which gives up early on a column where a dictionary loses. +The table then runs for the next chunk if the dictionary won. ### 3.4 Choose: exact verdicts, then bounded trials -In order, per column chunk: - -1. **Degenerate cases**, from the statistics alone: a chunk of one value, or of nulls only, of any - dtype -- a struct's and a list's included, whose validity would otherwise be a bitmap of zeros - -- as `vortex.constant`, a scalar and the length; an integer progression as `vortex.sequence`, - about thirty bytes whatever the rows. The constant comes first, integers included: a sequence - of step zero decodes to its rows, where a constant stays one value through a scan. -2. **Nulls on nine rows in ten or more**, as `vortex.sparse` with a null fill: the valid rows' - positions and values, and no validity, priced as a frame of reference packs them against the - runs the nulls make. It comes before run-end's rule below, which would take every such chunk as - runs, and before a decimal's parts. A **decimal** whose values fit 64 bits is then written as - `vortex.decimal_byte_parts` over the integers, which the integer schemes price in turn. -3. **Exact candidates**, each priced by a formula that is what its encoder will write: the plain +Per column chunk, in order: + +1. Degenerate cases are decided from the statistics alone. A chunk of a single value, or of nulls + only, of any dtype (a struct's and a list's included, whose validity would otherwise be a bitmap + of zeros) is written as `vortex.constant`, a scalar and a length. An integer progression is + written as `vortex.sequence`, about thirty bytes whatever the row count. The constant comes first, + integers included, because a sequence of step zero decodes to all its rows, whereas a constant + stays a single value through a scan. +2. A chunk with nulls on nine rows in ten or more is written as `vortex.sparse` with a null fill: the + valid rows' positions and values, and no validity, priced the way a frame of reference packs them + against the runs the nulls make. It comes before run-end's rule below, which would otherwise take + every such chunk as runs, and before a decimal's parts. A decimal whose values fit in 64 bits is + then written as `vortex.decimal_byte_parts` over the integers, which the integer schemes price in + turn. +3. Exact candidates are each priced by a formula equal to what their encoder will write: the plain form (for text, `vortex.varbin` or views, whichever is smaller), run-end (which wins outright - inside a quarter as many runs as rows), bit-packing in the raw, zigzag or frame-of-reference - domain with the cheapest width and its patches, a **timestamp's** instants split into days, - seconds and subseconds (`vortex.datetimeparts`), priced by each part's packed width and taken - when it comes a tenth under the best so far, and the dictionary, priced by its own layer at the - codes' width plus its values child. A 256-row sample bounds the timestamp split from below, so - instants at full resolution are refused without the pass that splits them. -4. **Trials**, only when no exact verdict won, each carrying its encoded result so that a winner is - never encoded twice: for floats, ALP, then ALP-RD against a zstd frame; for text, a zstd frame - priced first, then FSST, which must save a tenth on the plain form and not lose to the frame by - more than a tenth, then OnPair, which must also come a tenth under FSST: a dictionary of up to - 4 096 tokens trained on a random sixth of the chunk's bytes, whose own codes on that sample - decide before the pass over every row. Zstd is tried only on chunks of at least 16 KiB. - -`CompressionProfile` changes the arithmetic, not the pass: `Auto` weighs bytes and decode speed as -above; `Smallest` prices every candidate by its bytes alone and tries every trial under the bytes -the best exact plan writes, pco among them for numbers of 16 bits and more, on the column and on -every array a scheme makes of it, as the reference's compact compressor cascades; `Fastest` writes -the cheapest encodings and no index; `None` writes every column plain. - -**pco under `Smallest`.** A numeric chunk is compressed as pco does it at its default level: the -latents in chunks of up to 2^18 and pages of 8 192 as Vortex's compact compressor pages them, an -integer's common divisor found by pco's own test on pco's own sample (`IntMult`), the consecutive -delta order a sample prices best, then equal-count bins merged by pco's bit-cost dynamic program -and entropy-coded by its four-way tANS. What upstream's automatic choice also tries and this one -does not is the lookback delta and the float modes, whose values ALP and ALP-RD already take. The -bytes match upstream's own within the framing on every shape measured -- event times, a random -walk, multiples of a hundred, skewed counts, doubles in full -- and the plan is priced with the -framing its pages bring, a buffer and a count each, since a column that compresses to little has -as much there as in the pages. Before the pass that bins and writes a chunk, the sample that chose -its delta prices it: a chunk a quarter over the bytes it must beat is refused unwritten. The trial -costs what pco costs, a partition of the latents into bins rather than a sort, and a count in their -place where their range is narrower than their number, which is what deltas leave. - -**Plan memory.** A column keeps its last plan, the bytes the formulas predicted for it and the bytes -its encoder produced. The next chunk re-prices only the remembered plan, from its own statistics, -as long as the last chunk's bytes stayed within 5 % of the prediction; otherwise it prices -everything again. A progression and a run count are checked before memory, since they are -arithmetic on the summary: a plan that held never costs a progression its 32 bytes. What memory + below a quarter as many runs as rows), bit-packing in the raw, zigzag or frame-of-reference domain + at the cheapest width with its patches, a timestamp's instants split into days, seconds and + subseconds (`vortex.datetimeparts`), priced by each part's packed width and taken when it comes a + tenth under the best so far, and the dictionary, priced by its own layer at the codes' width plus + its values child. A 256-row sample bounds the timestamp split from below, so instants at full + resolution are refused without the pass that splits them. +4. Trials only run when no exact verdict won, and each carries its encoded result so a winner is + never encoded twice. For floats, ALP is tried, then ALP-RD against a zstd frame. For text, a zstd + frame is priced first, then FSST, which must save a tenth on the plain form and not lose to the + frame by more than a tenth, then OnPair, which must also come a tenth under FSST. OnPair trains a + dictionary of up to 4 096 tokens on a random sixth of the chunk's bytes, and its codes on that + sample decide before the pass over every row. Zstd is only tried on chunks of at least 16 KiB. + +`CompressionProfile` changes the arithmetic, not the pass. `Auto` weighs bytes and decode speed as +above. `Smallest` prices every candidate by its bytes alone and tries every trial under the bytes the +best exact plan writes, pco among them for numbers of 16 bits and more, on the column and on every +array a scheme makes of it, the way the reference's compact compressor cascades. `Fastest` writes the +cheapest encodings and no index, and `None` writes every column plain. + +Under `Smallest`, a numeric chunk is compressed the way pco does it at its default level. The latents +are split into chunks of up to 2^18 and pages of 8 192, as Vortex's compact compressor pages them, an +integer's common divisor is found by pco's own test on pco's own sample (`IntMult`), the consecutive +delta order is the one a sample prices best, and equal-count bins are merged by pco's bit-cost +dynamic program and entropy-coded by its four-way tANS. What upstream's automatic choice also tries +and this one does not is the lookback delta and the float modes, whose values ALP and ALP-RD already +handle. The bytes match upstream's own, framing aside, on every shape measured (event times, a random +walk, multiples of a hundred, skewed counts, full doubles), and the plan is priced with the framing +its pages bring, a buffer and a count each, since a column that compresses to little has as much +there as in the pages. Before the pass that bins and writes a chunk, the sample that chose its delta +prices it, and a chunk a quarter over the bytes it must beat is refused unwritten. The trial costs +what pco costs: a partition of the latents into bins rather than a sort, and a count in their place +where their range is narrower than their number, which is what deltas leave. + +A column also keeps a plan memory: its last plan, the bytes the formulas predicted for it and the +bytes its encoder produced. As long as the last chunk's bytes stayed within 5 % of the prediction, +the next chunk only prices the remembered plan again, from its own statistics. Otherwise it prices +everything again. A progression and a run count are checked before the memory, since they are +arithmetic on the summary, so a plan that held never costs a progression its 32 bytes. What memory saves is the walks and the trials, never a candidate the statistics already decide. An FSST symbol -table is still trained per chunk (sharing one costs 4.3 % of the output); what is remembered is -the decision. - -**Hints.** `VortexWriteOptions.Hints` pins a column's scheme by path (`EncodingHint`: `Canonical`, -`RunEnd`, `Dictionary`, `BitPacked`, `Fsst`, `Alp`, `AlpRd`, `Sequence`, `Zstd`, `DecimalByteParts`, -`Constant`, `DateTimeParts`, `Sparse`, `OnPair`, `Pco`): plan memory with an infinite tolerance, priced on every -chunk and written when it applies, under every profile; only a constant and a progression, which -cost nothing per row, are written before it. `Sequence` and `Constant` pin nothing the writer would -not do anyway; they name what a report says a chunk was written as. A chunk the scheme cannot describe -is priced in full, and the next chunk is offered the hint again. A hint on a path the schema does -not have throws at `CreateWriter`. - -**The cascade has context.** A child never enters the chooser blind: a dictionary's codes are dense -in `[0, entries)` and packed at their width; run-end ends are strictly increasing; offsets and sizes -are monotone or bounded, and a list of fixed-width rows has offsets that are an exact progression -and sizes that are constant, so both cost nothing. A dictionary's or a run's values are a column -like any other, chosen in full on a small input. A timestamp's storage knows its unit, which is -what lets it price its split; a part of one value -- the subseconds of instants to the second -- -is handed on as a constant rather than discovered by a walk. +table is still trained per chunk (sharing one costs 4.3 % more output), and only the decision is +remembered. + +`VortexWriteOptions.Hints` pins a column's scheme by path (`EncodingHint`: `Canonical`, `RunEnd`, +`Dictionary`, `BitPacked`, `Fsst`, `Alp`, `AlpRd`, `Sequence`, `Zstd`, `DecimalByteParts`, +`Constant`, `DateTimeParts`, `Sparse`, `OnPair`, `Pco`). A hint acts as plan memory with an infinite +tolerance, priced on every chunk and written whenever it applies, under every profile. Only a +constant and a progression, which cost nothing per row, are written before it. `Sequence` and +`Constant` pin nothing the writer would not do anyway, and simply name what a report says a chunk was +written as. A chunk the scheme cannot describe is priced in full, and the next chunk is offered the +hint again. A hint on a path the schema does not have throws at `CreateWriter`. + +The cascade has context. A child never enters the chooser blind: a dictionary's codes are dense in +`[0, entries)` and packed at their width, run-end ends are strictly increasing, offsets and sizes are +monotone or bounded, and a list of fixed-width rows has offsets that form an exact progression and +sizes that are constant, so both cost nothing. A dictionary's or a run's values are a column like any +other, chosen in full on a small input. A timestamp's storage knows its unit, which is what lets it +price its split, and a part that is the same for every value (the subseconds of instants rounded to +the second) is handed on as a constant rather than discovered by a walk. ### 3.5 Encode once -The winner reads the chunk's rows block by block, from the batch or the retained batches, and -writes outputs sized exactly: patches found inside the pack, frame of reference and zigzag applied -on load, a dictionary's codes packed from the codes the first pass wrote, ALP's exceptions as -patches. A value is ALP-encoded only when decoding the integer gives back its **exact bit pattern**, -compared by bits and never with `==`, since `-0.0 == 0.0` and NaN payloads are invisible to `==`. -An FSST symbol table is ordered by symbol length, two to eight bytes then one, as the reference -requires. +The winner reads the chunk's rows block by block, from the batch or the retained batches, and writes +outputs sized exactly: patches found inside the pack, frame of reference and zigzag applied on load, +a dictionary's codes packed from the codes the first pass wrote, and ALP's exceptions as patches. A +value is only ALP-encoded when decoding the integer gives back its exact bit pattern, compared by +bits and never with `==`, since `-0.0 == 0.0` and NaN payloads are invisible to `==`. An FSST symbol +table is ordered by symbol length, two to eight bytes then one, as the reference requires. -A zstd column is compressed as **one frame per block**, so a take inflates only the frames that hold -the rows it wants, and the frames of a large column are compressed on the writer's threads (§3.7): -each into a room of its own, closed up in order, so the bytes are the same on any number of -threads. +A zstd column is compressed as one frame per block, so a take only inflates the frames that hold the +rows it wants. The frames of a large column are compressed on the writer's threads (see +[threads](#37-memory-allocation-and-threads)), each into its own buffer and then closed up in order, +so the bytes are the same on any number of threads. ### 3.6 Emit -A chunk goes out when enough whole blocks are pending, and a Bloom filter's runs as their tree nodes -close, between chunks. When the file completes: the key runs, fence pages and upper filter levels; -then the zone maps, from the closed blocks' summaries; the file statistics; the index directory; -the file's identity ([13-dataset.md](13-dataset.md) §7); the footer and the postscript. +A chunk goes out when enough whole blocks are pending, and a Bloom filter's runs go out as their tree +nodes close, between chunks. When the file completes, the writer emits the key runs, fence pages and +upper filter levels, then the zone maps from the closed blocks' summaries, the file statistics, the +index directory, the file's identity (see [identity and +integrity](13-dataset.md#7-identity-and-integrity)), and finally the footer and the postscript. -**The file statistics** are one `ArrayStats` per top-level column: the exact minimum and maximum of -a numeric column, the null count, and whether the column ascends, in the reference's semantics — a -null below every value, equal neighbours allowed by `is_sorted` and refused by `is_strict_sorted`, a -NaN claiming nothing. The Rust cross-check holds every one against its own recomputation. A text -column's zones carry **bounded** minimum and maximum, 16 bytes by default (`StringBoundBytes`), so -that a text filter prunes. +The file statistics are one `ArrayStats` per top-level column: the exact minimum and maximum of a +numeric column, the null count, and whether the column ascends, in the reference's semantics (a null +below every value, equal neighbours allowed by `is_sorted` and refused by `is_strict_sorted`, a NaN +claiming nothing). They carry no sum. The Rust cross-check compares every one with its own +recomputation. A text column's zones carry bounded minimum and maximum values, 16 bytes by default +(`StringBoundBytes`), so that a text filter can prune. -`CompleteAsync` returns a **`WriteReport`**: the file's bytes by kind, the chunk sizes, per column -the scheme of every chunk and how often plan memory held, and per index what was built, its bytes, -or why it was abandoned. +`CompleteAsync` returns a `WriteReport`: the file's bytes by kind, the chunk sizes, per column the +scheme of every chunk and how often plan memory held, and per index what was built and its bytes, or +why it was abandoned. ### 3.7 Memory, allocation and threads | what | owner | size | lifetime | |---|---|---|---| -| the open block's partials and 56 bytes per closed block | the column | small | the file | -| the distinct table, its key heap and its codes | the column | at most a small multiple of the column's share of the pending chunk | the chunk; only while it runs | +| the open block's partial summaries, and 56 bytes per closed block | the column | small | the file | +| the distinct table, its key heap and its codes | the column | at most a small multiple of the column's share of the pending chunk | the chunk, and only while it runs | | index builders and spilled chunk runs | the column | pooled pages, then a scratch file | the file | | encoder outputs | the writer's arena | the segment's exact size | until the segment is written | -| per row | — | **nothing** | — | +| per row | none | nothing | none | -A schema of a thousand columns costs a thousand small states, 1 892 bytes each, not a thousand -scratches ([05-benchmarks.md](05-benchmarks.md) §5). +A schema of a thousand columns costs a thousand small states of 1 892 bytes each, not a thousand +scratch buffers ([allocations](05-benchmarks.md#5-allocations)). -**Threads.** At a degree above one (`VortexWriteOptions.DegreeOfParallelism`, or the session's), a -batch of a block or more is summarized column by column side by side: each column's statistics, and -apart from them a text column's bounds and the distinct values a dictionary is priced from, each -into state of its own. A column's zstd frames are compressed side by side too. Choosing and writing -the encodings stays on the calling thread, and the file is the same bytes at any degree. A write -that builds indexes summarizes on one thread. +At a degree above one (`VortexWriteOptions.DegreeOfParallelism`, or the session's), a batch of a +block or more is summarized column by column, side by side: each column's statistics, and separately +a text column's bounds and the distinct values a dictionary is priced from, each into its own state. +A column's zstd frames are compressed side by side too. Choosing and writing the encodings stays on +the calling thread, and the file is the same bytes at any degree. A write that builds indexes +summarizes on one thread. ### 3.8 Appending to a file -`VortexSession.AppendAsync(path)` returns a writer that continues a file this writer's shape can -extend — a struct of columns, each a chunked layout of flat segments, chunked alike — and refuses any -other with the advice to rewrite it. The old segments stay where they are; new blocks are numbered from the old -row count, so their boundaries are the ones a single write would have made; new segments, runs, -zone maps, statistics, directory, footer and postscript are appended, and the old postscript -becomes dead bytes. Three rules make it exact: - -- **A last chunk that is not whole blocks is re-opened.** A zone map allows only its last zone to be - short, and the last block's filter and runs were built on a short block. So the last chunk is - read, dropped from the layout, and written again as the append's first rows, with the runs that - covered it: one chunk, whatever the file's size. Every block before it becomes a summary taken - from the old zone map, so the zone map is written again over the whole file without reading them. -- **Plan memory is seeded from the last chunk's encodings**, one segment read per column, so the - appended rows keep the file's encodings unless their statistics say otherwise. -- **The index policy and budget are read from the directory**, so an append needs no options; the - old runs are kept and merged past K ([10-indexes.md](10-indexes.md) §7). +`VortexSession.AppendAsync(path)` returns a writer that continues a file whose shape this writer can +extend (a struct of columns, each a chunked layout of flat segments, all chunked alike), and refuses +any other file with the advice to rewrite it. The old segments stay where they are. New blocks are +numbered from the old row count, so their boundaries are the ones a single write would have made. +New segments, runs, zone maps, statistics, directory, footer and postscript are appended, and the old +postscript becomes dead bytes. Three rules make it exact: + +- A last chunk that is not made of whole blocks is reopened. A zone map only allows its last zone to + be short, and the last block's filter and runs were built on a short block. So the last chunk is + read, dropped from the layout, and written again as the append's first rows, together with the runs + that covered it. That is one chunk, whatever the file's size. Every block before it becomes a + summary taken from the old zone map, so the zone map is written again over the whole file without + reading those blocks. +- Plan memory is seeded from the last chunk's encodings, at the cost of one segment read per column, + so the appended rows keep the file's encodings unless their statistics say otherwise. +- The index policy and budget are read from the directory, so an append needs no options. The old + runs are kept, and merged once there are more than four (see [indexing after the + fact](10-indexes.md#7-building-appending-and-indexing-after-the-fact)). An append is not atomic. It never truncates while it writes, so a torn append leaves the previous -version intact before garbage: the open finds it by walking back to the last end-of-file record, -reports the tear in `VortexFile.TornTail`, and reads the previous version, unless -`VortexOpenOptions.TornTail` says `Refuse`; `VortexFileRepair.RepairAsync` truncates the file to -it. An append, and an indexing pass, refuse a torn file until it is repaired. `Abandon()` on an +version intact, followed by garbage. The open finds that version by walking back to the last +end-of-file record, reports the tear in `VortexFile.TornTail`, and reads the previous version, unless +`VortexOpenOptions.TornTail` says `Refuse`. `VortexFileRepair.RepairAsync` truncates the file back +to it. An append, and an indexing pass, refuse a torn file until it is repaired. `Abandon()` on an append truncates the file back to the length it had. ## 4. Kernels -`Vector128` everywhere, wider vectors where the hardware has them, and a scalar twin whose tests run -again with intrinsics disabled ([03-architecture.md](03-architecture.md) §1). In the statistics pass: -minimum and maximum per lane, with validity folded in as a mask; null counts by population count -per 64-bit word; run boundaries by comparing a vector with itself shifted by one; order and -progressions by lane-wise subtraction against a broadcast step, exact because the progression's -endpoint is checked to fit the type first; bit widths by lane leading-zero counts where the -instruction set has them; string runs by one 16-byte comparison per pair of views. In the encoders: -frame of reference and zigzag applied on load, patch detection by a comparison mask per 1 024-block, -the FastLanes pack, and ALP's multiply, round and convert with the exception mask. - -Some kernels were priced before being written and were not, because the saving was not there: an -eight-lane Bloom probe saves 0.2 µs of a 7 ms filtered scan, since a scan probes about 140 times per -literal; a vectorized zone-map pass has no register's worth of work in front of the decode that -feeds it; and a Swiss-table group probe would save at most five per cent of a dictionary's write. -The one that was written is the merge-join of an `IN` of many keys against a sorted run's keys, -which took a thousand-key `IN` from 117 to 51 ms. +`Vector128` is used everywhere, wider vectors where the hardware has them, plus a scalar twin whose +tests run again with intrinsics disabled (see [the founding +constraints](03-architecture.md#1-founding-constraints)). + +The statistics pass vectorizes: + +- minimum and maximum per lane, with the validity folded in as a mask +- null counts, by population count per 64-bit word +- run boundaries, by comparing a vector with itself shifted by one +- order and progressions, by lane-wise subtraction against a broadcast step, exact because the + progression's endpoint is first checked to fit the type +- bit widths, by lane leading-zero counts where the instruction set has them +- string runs, by one 16-byte comparison per pair of views + +The encoders apply frame of reference and zigzag on load, detect patches with a comparison mask per +1 024-value block, run the FastLanes pack, and do ALP's multiply, round and convert with the +exception mask. + +Some kernels were priced before being written and then not written, because the saving was not +there. An eight-lane Bloom probe would save 0.2 µs of a 7 ms filtered scan, since a scan probes +about 140 times per literal. A vectorized zone-map pass has less than a register's worth of work +ahead of the decode that feeds it. A Swiss-table group probe would save at most five per cent of a +dictionary's write. The one that was written is the merge-join of an `IN` of many keys against a +sorted run's keys, which took a thousand-key `IN` from 117 to 51 ms. ## 5. What it guarantees, and what holds it -1. **Nothing a contract rests on is sampled.** Minimum, maximum, null count, Bloom insertions and - the keys behind an index cover every row. ALP's exponent search and FSST's symbol training - sample, and neither is a statistic anything prunes by. -2. **Equality is on bytes.** A table hit is confirmed on the key's bytes; hashes only route. -3. **An index is whole or absent**, and the report says which ([10-indexes.md](10-indexes.md) §6). -4. **Merges are exact or conservative.** Statistics merge exactly; a filter node is sized from the - exact union of its blocks; a bounded string minimum or maximum is marked as a bound. -5. **Pruning never removes a row a full scan returns** ([08-semantics.md](08-semantics.md) §1). +1. Nothing a contract rests on is sampled. Minimum, maximum, null count, Bloom insertions and the + keys behind an index cover every row. ALP's exponent search and FSST's symbol training do sample, + but neither is a statistic anything prunes by. +2. Equality is on bytes. A table hit is confirmed on the key's bytes, and hashes only route. +3. An index is whole or absent, and the report says which (see [the policy](10-indexes.md#6-the-policy)). +4. Merges are exact or conservative. Statistics merge exactly, a filter node is sized from the exact + union of its blocks, and a bounded string minimum or maximum is marked as a bound. +5. Pruning never removes a row a full scan returns (see [statistic + precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)). | what | how it is held | |---|---| @@ -286,115 +303,116 @@ which took a thousand-key `IN` from 117 to 51 ms. | Rust reads it | the cross-check, appended files and chunked-apart columns among them | | statistics | property tests against a naive per-row computation, at block and batch edges, for every dtype and validity shape | | formulas | the formula's cost equals the encoder's bytes, for every scheme on the corpus | -| plan memory | a column whose data changes shape re-prices within a chunk; a stable column never | -| append | a file written once and the same rows appended in two, three and eleven pieces answer the same, zone maps and indexes included | +| plan memory | a column whose data changes shape is priced again within a chunk, and a stable column never is | +| append | a file written once and the same rows appended in two, three and eleven pieces give the same answers, zone maps and indexes included | | threads | the file is byte for byte the same at every degree | -| allocations and dispatch | ratchets per written file and per column; no per-row dispatch in `Writing/` beyond the calls named in `PerRowDispatchTests` | +| allocations and dispatch | ratchets per written file and per column, and no per-row dispatch in `Writing/` beyond the calls named in `PerRowDispatchTests` | ## 6. Reading what the writer produced The writer's structures are only worth what the scan does with them. Every pruning structure -implements one operation: **refine a mask of live blocks** for the scan's predicate, a bit per block. -The scan builds the mask once per query and runs the structures **cheapest first**: the file -statistics, the zone maps (minimum and maximum, null counts, the bounded string bounds), the Bloom -filters from the root down, the postings and sorted runs, and last the dictionary probe, the one -structure that reads the column's own bytes. It stops as soon as the mask is empty. Then, per split: - -- a split whose blocks are all dead is skipped before its read: the I/O saving; -- a split with dead blocks becomes a **row selection** over its live blocks, which the decoders - serve through `DecodeSelected`, so only live blocks are decoded: the decode saving; +implements one operation: refine a mask of live blocks, one bit per block, for the scan's predicate. +The scan builds the mask once per query and runs the structures cheapest first: the file statistics, +the zone maps (minimum and maximum, null counts, the bounded string bounds), the Bloom filters from +the root down, the postings and sorted runs, and last the dictionary probe, the one structure that +reads the column's own bytes. It stops as soon as the mask is empty. Then, per split: + +- a split whose blocks are all dead is skipped before it is read, which saves the I/O +- a split with some dead blocks becomes a row selection over its live blocks, which the decoders + serve through `DecodeSelected`, so only live blocks are decoded, which saves the decode - an `Exact` index covering the whole predicate turns its rows into that selection, and the - predicate is not evaluated again; under `OR`, `NOT` or a partial cover it only narrows, and the - filter runs on the survivors. + predicate is not evaluated again. Under `OR`, `NOT` or a partial cover, it only narrows, and the + filter runs on the survivors -Two more promises close the contract: +Two more promises complete the contract: -- **One lease per segment per scan.** The live splits register their segments in one request, each - distinct segment is read once in as few requests as coalescing allows, and every block of it +- A scan takes one lease per segment. The live splits register their segments in one request, each + distinct segment is read once in as few requests as coalescing allows, and every block in it decodes from that lease. `ScanStatistics.Requests` is the plan's distinct segments, and - `BytesRequested` its bytes, within the coalescing gap. -- **A selection instead of a compaction.** A filtered batch is compacted to its survivors by - default; with `ScanOptions.Compact = false` each live block is delivered whole with the - `Selection` of the rows that passed, and nothing is copied. Either way `StartRow` is the block's + `BytesRequested` their bytes, within the coalescing gap. +- A selection can replace a compaction. A filtered batch is compacted to its surviving rows by + default. With `ScanOptions.Compact = false`, each live block is delivered whole with the + `Selection` of the rows that passed, and nothing is copied. Either way, `StartRow` is the block's first row in the file. -**Granularity, honestly.** I/O is per segment, and a segment is a chunk of several blocks: a block -pruned inside a live chunk saves decode, not bytes read. That is why chunks stay whole blocks, and -why a file meant for selective reads wants smaller chunks. +The granularity has to be stated honestly. I/O is per segment, and a segment is a chunk of several +blocks, so a block pruned inside a live chunk saves decode, not bytes read. That is why chunks stay +whole blocks, and why a file meant for selective reads wants smaller chunks. The index directory is read lazily, on the first filter that could use it, and usually lies inside -the tail the open already read; the runs are read on the first query that needs them and cached on -the file, shared by every scan. `ExplainAsync` returns the plan without executing it — blocks, blocks -pruned by each structure with what consulting it cost, segments and bytes to read — and the scan's +the tail the open already read. The runs are read on the first query that needs them and cached on +the file, shared by every scan. `ExplainAsync` returns the plan without executing it (blocks, blocks +pruned by each structure with what consulting it cost, segments and bytes to read), and the scan's `Statistics` report the same quantities once it has run, counted by the same sink. ## 7. Determinism, and its limit -The file is a pure function of the **sequence of batches**: no seed, no sample, plan memory a -function of the chunks before. The one exception is the file's sixteen-byte identity, random unless -`VortexWriteOptions.Identity` pins it. +The file is a pure function of the sequence of batches: no seed, no sample, and plan memory is a +function of the chunks before. The one exception is the file's sixteen-byte identity, which is random +unless `VortexWriteOptions.Identity` pins it. -It is **not** a function of the rows alone. Feed the same rows in another batching and the file is -valid, reads back value for value, and is a different size: rewriting an 8 193-row binary column at -forced batch counts costs 65 980 bytes in 2 chunks, 92 212 in 3, 146 076 in 5 and 169 860 in 9, -about 26 kB per extra chunk, because each chunk builds and writes its own dictionary of the same +The file is not a function of the rows alone. Feed the same rows with a different batching and the +file is valid and reads back value for value, but its size changes. Rewriting an 8 193-row binary +column at forced batch counts costs 65 980 bytes in 2 chunks, 92 212 in 3, 146 076 in 5 and 169 860 +in 9, about 26 kB per extra chunk, because each chunk builds and writes its own dictionary of the same distinct values. The reference shares one values child across a column's chunks through the -`vortex.dict` layout, which this writer does not emit. `BatchingSizeTests` pins those numbers, as a -ratchet on a cost known to be bad. A caller who wants the smallest file writes the largest batches -it can, or gives the column chunks of its own. +`vortex.dict` layout, which this writer does not emit. `BatchingSizeTests` pins those numbers as a +ratchet on a cost known to be bad. A caller who wants the smallest file writes the largest batches it +can, or gives the column chunks of its own. ## 8. Choosing encodings ahead: the advisor The writer optimizes one thing per chunk, bytes, under rules that keep decoding fast. The -measurements of [the encodings, column by column](../guide/benchmarks.md#encodings-column-by-column) -show where that is not what a caller wants: text that does not repeat goes to zstd, which a scan -decodes several times slower than FSST and a take far slower; a column whose values repeat across the -file more than within a chunk cannot take a dictionary, which larger chunks would give it; and which -is right turns on the storage under the file — below a few hundred MB/s the smaller file reads -faster end to end — and on whether the file is scanned or read by row. None of that is the writer's -to know; all of it is the caller's, or can be measured on the caller's data -([choose-encodings.md](../guide/choose-encodings.md)). - -`VortexSession.AdviseAsync` measures it (`Advice/`): it takes a sample of the data, writes it under +measurements in [the encodings, column by +column](../guide/benchmarks.md#encodings-column-by-column) show where that is not what a caller +wants. Text that does not repeat goes to zstd, which a scan decodes several times slower than FSST, +and a take far slower. A column whose values repeat across the file more than within a chunk cannot +take a dictionary, which larger chunks would give it. And the right choice depends on the storage +under the file (below a few hundred MB/s the smaller file reads faster end to end) and on whether the +file is scanned or read by row. None of that is the writer's to know. All of it is the caller's, or +can be measured on the caller's data ([choose-encodings.md](../guide/choose-encodings.md)). + +`VortexSession.AdviseAsync` measures it (`Advice/`). It takes a sample of the data, writes it under each choice that applies, reads it back, and ranks the choices by the cost of the reads the caller -declares in an `EncodingGoal` — a storage throughput, lookups per scan, or size. It returns per column -the numbers, the recommendation and why, and `ToWriteOptions` turns them into hints and chunk -targets that the writer then applies exactly, as any options. - -- **The cost of a candidate**, per value: to minimize read time, `scan + bytes ÷ throughput`, plus - `lookupsPerScan × (lookup + chunkBytes ÷ throughput) ÷ rows`; to minimize size, its bytes, the scan - breaking a tie. `scan` and `lookup` are timed in memory, on the machine the advisor runs on, so one - sample prices any throughput; each time is the least of several passes, and under the JIT the - advisor first waits until the compiler has settled. -- **The sample**: eight windows of contiguous rows spread over the data, 1 048 576 rows in all by - default. Contiguous, because a chunk's encoding depends on the rows that are together; spread, +declares in an `EncodingGoal`: a storage throughput, lookups per scan, or size. It returns, per +column, the numbers, the recommendation and the reason, and `ToWriteOptions` turns them into hints and +chunk targets that the writer then applies exactly, like any other options. + +- The cost of a candidate is computed per value. To minimize read time, it is + `scan + bytes ÷ throughput`, plus `lookupsPerScan × (lookup + chunkBytes ÷ throughput) ÷ rows`. To + minimize size, it is the candidate's bytes, with the scan breaking a tie. `scan` and `lookup` are + timed in memory, on the machine the advisor runs on, so one sample prices any throughput. Each time + is the best of several passes, and under the JIT the advisor first waits until the compiler has + settled. +- The sample is eight windows of contiguous rows spread over the data, 1 048 576 rows in all by + default. Contiguous, because a chunk's encoding depends on the rows that sit together, and spread, because data drifts. -- **The candidates**: per kind, the plain form, `Auto`, and the hints that apply, at the writer's - chunk size, and at 4 MiB and 16 MiB chunks when the sample says the chunk decides. -- **The choice**: the cheapest candidate, unless `Auto` is within 5 % of it, since advice should not - move with a measurement's noise; a larger chunk replaces the writer's own only when it costs the - column 5 % less. - -The advice is a measurement, run once for a kind of data rather than for every file: a -million-row sample costs a fraction of a second for a column of numbers and a few seconds for text. -Held outside CI to the twenty column shapes of `--tradeoffs`, it names the choice the guide's -tables measured as best on every one. Taking the advice by default would change what a write -produces and its determinism, and is not done. +- The candidates, per kind, are the plain form, `Auto` and the hints that apply, at the writer's + chunk size, and also at 4 MiB and 16 MiB chunks when the sample says the chunk size decides. +- The choice is the cheapest candidate, unless `Auto` is within 5 % of it, since advice should not + move with measurement noise. A larger chunk only replaces the writer's own when it costs the column + 5 % less. + +The advice is a measurement, run once for a kind of data rather than for every file. A million-row +sample costs a fraction of a second for a column of numbers and a few seconds for text. Checked +outside CI against the twenty column shapes of `--tradeoffs`, it names the choice the guide's tables +measured as best on every one. Taking the advice by default would change what a write produces and +its determinism, so it is not done. ## 9. Deliberately not done | approach | why not | |---|---| -| pricing from a sample, as BtrBlocks does | it prices what exact formulas price for free, and costs determinism; plan memory takes most of what it would save on drifting text | -| `fastlanes.rle` | its per-row indices need `fastlanes.delta`, which no edition carries, to come under run-end's ends; without it run-end is smaller at every run length | -| a sparse fill other than null | a dominant value that is the chunk's minimum is already a bit-packing of width zero with patches, and one that is not makes runs; finding it would take a frequency count on every chunk | -| emitting a chunk at every batch | a chunk edge inside a block saves no I/O on a pruned zone; a caller who wants no transit writes in multiples of `BlockRows` | -| choosing and writing encodings on several threads | the summaries and zstd frames are parallel; the choice and emission stay ordered on one thread, which keeps the bytes independent of the degree | -| rewriting without decoding, passing encoded arrays through | a separate feature: an order of magnitude on rewrites and compactions, nothing on first writes | -| buffering a column for global decisions, or a global sorted index | it breaks streaming and bounds memory by the file; sorted runs per chunk, merged, give the same answers ([10-indexes.md](10-indexes.md) §5.2) | +| pricing from a sample, as BtrBlocks does | it prices what exact formulas price for free, and costs determinism. Plan memory takes most of what it would save on drifting text | +| `fastlanes.rle` | its per-row indices need `fastlanes.delta`, which no edition carries, to come under run-end's ends. Without it, run-end is smaller at every run length | +| a sparse fill other than null | a dominant value that is the chunk's minimum is already a bit-packing of width zero with patches, and one that is not makes runs. Finding it would take a frequency count on every chunk | +| emitting a chunk at every batch | a chunk edge inside a block saves no I/O on a pruned zone. A caller who wants no buffering writes in multiples of `BlockRows` | +| choosing and writing encodings on several threads | the summaries and zstd frames are parallel, and the choice and emission stay ordered on one thread, which keeps the bytes independent of the degree | +| rewriting without decoding, passing encoded arrays through | a separate feature, worth an order of magnitude on rewrites and compactions and nothing on first writes | +| buffering a column for global decisions, or a global sorted index | it breaks streaming and bounds memory by the file. Sorted runs per chunk, merged, give the same answers (see [sorted runs](10-indexes.md#52-vorticitysortedrunsv1-value-to-rows-exact)) | | one FSST symbol table shared across chunks | measured at 4.3 % more bytes | -| the `vortex.dict` layout, one values child per column | what §7's limit would need; it moves the bytes of every dictionary column, and is weighed on the size ratchet and the cross-check together | +| the `vortex.dict` layout, one values child per column | what the [determinism limit](#7-determinism-and-its-limit) would need. It changes the bytes of every dictionary column, and has to be weighed on the size ratchet and the cross-check together | The reference writer this is measured against is `vortex-file`'s strategy and `vortex-btrblocks`' -schemes at 0.86.1; the kernels follow FastLanes (Afroozeh and Boncz, VLDB 2023) and ALP (Afroozeh, +schemes at 0.86.1. The kernels follow FastLanes (Afroozeh and Boncz, VLDB 2023) and ALP (Afroozeh, Kuiper and Boncz, SIGMOD 2024). diff --git a/docs/design/12-index-reads.md b/docs/design/12-index-reads.md index 816b8360..c1ada0c3 100644 --- a/docs/design/12-index-reads.md +++ b/docs/design/12-index-reads.md @@ -2,268 +2,277 @@ What a caller can ask of the read path beyond a scan: a cursor that seeks and steps in key order, counts and extremes computed without materializing rows, and rows delivered in key order. Every -structure these read is one [10-indexes.md](10-indexes.md) defines or one the file statistics -already carry: **nothing on disk changes** for any of it. +structure these read is one that [10-indexes.md](10-indexes.md) defines, or one the file statistics +already carry, so nothing on disk changes for any of it. -Three more constraints, in order: +Three more constraints apply, in order: -1. **Memory is bounded by a batch, never by the file.** An operation that would need a whole column - in memory — a sort, a distinct table over an unindexed column — is **refused**, naming the index - kind that would serve it, never emulated. -2. **Every answer equals a full materialization's**: a cursor walk equals a sort of the column, a - count equals the length of the scan's output, with indexes on or off. -3. **No allocation per step or per batch** in steady state - ([03-architecture.md](03-architecture.md) §4). +1. Memory is bounded by a batch, never by the file. An operation that would need a whole column in + memory (a sort, a distinct table over an unindexed column) is refused, naming the index kind that + would serve it, and never emulated. +2. Every answer equals what a full materialization would give. A cursor walk equals a sort of the + column, and a count equals the length of the scan's output, with indexes on or off. +3. No allocation per step or per batch in steady state (see [performance + invariants](03-architecture.md#4-performance-invariants)). ## 1. Two models over one payload -**The set model** is the scan: a predicate becomes a block mask, the mask a row selection, the -selection batches in file order ([11-write-strategy.md](11-write-strategy.md) §6). It answers *which -rows*, and it is the right engine for a filter: each touched split is read once and only live blocks -are decoded. +The set model is the scan: a predicate becomes a block mask, the mask a row selection, and the +selection batches in file order (see [reading what the writer +produced](11-write-strategy.md#6-reading-what-the-writer-produced)). It answers which rows match, and +it is the right engine for a filter, since each touched split is read once and only live blocks are +decoded. -**The order model** is a cursor: position at a key, step to the neighbour. It answers *what is -next*, which no set can. A streaming writer produces sorted runs, never a globally sorted column, so -key order exists inside each run; a cursor over the union is a **k-way merge** over the runs the -key range reaches, the merging iterator every log-structured store has. It is a reader, not a -format. +The order model is a cursor: position at a key, then step to the neighbour. It answers what comes +next, which no set can. A streaming writer produces sorted runs, never a globally sorted column, so +key order exists inside each run, and a cursor over their union is a k-way merge over the runs the +key range reaches, the merging iterator every log-structured store has. It is a reader, not a format. | model | entry point | answers | |---|---|---| -| set | `file.Scan()` | which rows match; how many; whether any; the smallest and largest | -| order | `Scan().Keys(r => r.Column)` | what is at, after or before a key; the first, the last, the i-th; the next distinct key; how many share a key | +| set | `file.Scan()` | which rows match, how many, whether any, the smallest and largest | +| order | `Scan().Keys(r => r.Column)` | what is at, after or before a key, the first, the last, the i-th, the next distinct key, how many share a key | | set driven by order | `Scan().OrderBy(r => r.Column)` | the matching rows, in key order | -Neither model mutates. Insertion into a key range, deletion and update belong to a storage engine -that owns a memtable and tombstones; the format carries no tombstone, and this library provides the -immutable half: bulk load, append, and every read below. +Neither model changes a file. Inserting into a key range, deleting and updating belong to a storage +layer that owns a memtable and tombstones, and the file format carries no tombstone. This library +provides the immutable half (bulk load, append, and every read below), and its dataset builds row +deletes and updates on top of whole objects ([deleting and updating +rows](13-dataset.md#12-deleting-and-updating-rows)). ## 2. Sources of order -A cursor needs a **key source**: the entries of a column in key order, with a row per entry, able to -locate a key in `O(log n)`. The cheapest one the file offers is used, and the builder's -`ExplainAsync` returns a `KeyPlan` naming it and why each other was rejected. +A cursor needs a key source: the entries of a column in key order, with a row per entry, able to +locate a key in `O(log n)`. The cheapest source the file offers is used, and the builder's +`ExplainAsync` returns a `KeyPlan` naming it and saying why each other one was rejected. | source | exists when | entries | seek | |---|---|---|---| -| **sorted column** | the file statistics say the column ascends, and it has a zone map | `(value, row)` per non-null row, in row order, which *is* key order | the zone bounds in memory, then one zone decoded and bisected | -| **sorted runs** | the column carries `vorticity.sorted.runs.v1` | `(key, row)` per non-null row, per run | the runs' bounds exclude runs, then a binary search in each survivor, through its fences | -| **postings** | `vorticity.postings.blocks.v1` | the distinct keys of each run; no rows | the same, on keys only | -| **dictionary** | `vorticity.dict.probe.v1` | the values of each dictionary chunk, sorted at open; no rows | in memory | -| none | an unsorted, unindexed column | — | **refused**: `VortexUnsupportedException` naming `IndexPolicy.SortedRuns` | +| sorted column | the file statistics say the column ascends, and it has a zone map | `(value, row)` per non-null row, in row order, which is key order | the zone bounds in memory, then one zone decoded and bisected | +| sorted runs | the column carries `vorticity.sorted.runs.v1` | `(key, row)` per non-null row, per run | the runs' bounds exclude runs, then a binary search in each survivor, through its fences | +| postings | `vorticity.postings.blocks.v1` | the distinct keys of each run, without rows | the same, on keys only | +| dictionary | `vorticity.dict.probe.v1` | the values of each dictionary chunk, sorted at open, without rows | in memory | +| none | an unsorted, unindexed column | none | refused with a `VortexUnsupportedException` naming `IndexPolicy.SortedRuns` | -**Cheapest first**: a sorted column costs one zone-map read and one zone decode, so it beats sorted -runs whenever both exist. **Rows when rows are wanted**: a cursor opened without `Distinct()` takes -only the first two, because a postings or dictionary source cannot say where a key's rows are, and -answering with a block's first row would make `Row` a lie. +The cheapest source wins: a sorted column costs one zone-map read and one zone decode, so it beats +sorted runs whenever both exist. A cursor opened without `Distinct()` only takes the first two +sources, because a postings or dictionary source cannot say where a key's rows are, and answering +with a block's first row would make `Row` a lie. The sorted-column source serves the files this writer produces, whose statistics carry the order -flags in the reference's semantics; the reference writer drops those flags from its file statistics. +flags in the reference's semantics. The reference writer drops those flags from its file statistics. An `Inexact` zone bound still brackets correctly, since the true minimum is at or above a stated one -and the true maximum at or below ([08-semantics.md](08-semantics.md) §1). A lying `is_sorted` gives a -wrong order, never a memory fault ([08-semantics.md](08-semantics.md) §5). +and the true maximum at or below (see [statistic +precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)). A lying `is_sorted` gives a +wrong order, never a memory fault (see [untrusted +hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). -**Nulls are in no source**: no index inserts a null, and a sorted column's null rows carry no key. A -cursor never visits one; `IS NULL` is the scan's question, answered from the zone maps' null counts. +No source holds nulls. No index inserts a null, and a sorted column's null rows carry no key, so a +cursor never visits one. `IS NULL` is the scan's question, answered from the zone maps' null counts. ## 3. The cursor -`KeyCursor` (`Keys/KeyCursor.Typed.cs`), `TKey` inferred from the record member, so a key of the +`KeyCursor` (`Keys/KeyCursor.Typed.cs`) infers `TKey` from the record member, so a key of the wrong type does not compile. Positioning is asynchronous because it may read a run segment or a zone -not yet loaded; inside a loaded run a step completes synchronously. A cursor is not thread-safe; the -runs it reads are cached on the file and shared by every cursor and scan of it. +not loaded yet, while a step inside a loaded run completes synchronously. A cursor is not +thread-safe, but the runs it reads are cached on the file and shared by every cursor and scan of it. ### 3.1 Positioning The entries of a source are totally ordered by `(key, row)`, and `SeekAsync(k, op)` positions the -cursor on that order: +cursor in that order: | `op` | positions on | is | |---|---|---| -| `Exact` | the first entry whose key is `k` | a point lookup; invalid when `k` is absent | -| `AtOrAfter` | the first entry with key `≥ k` | a lower bound: the start of a range | -| `After` | the first entry with key `> k` | an upper bound: the successor | +| `Exact` | the first entry whose key is `k` | a point lookup, invalid when `k` is absent | +| `AtOrAfter` | the first entry with key `≥ k` | a lower bound, the start of a range | +| `After` | the first entry with key `> k` | an upper bound, the successor | | `AtOrBefore` | the last entry with key `≤ k` | the end of a closed range, walking backwards | | `Before` | the last entry with key `< k` | the predecessor | `SeekFirstAsync` and `SeekLastAsync` are the minimum and the maximum. `Exact` on a key with duplicates -lands on its lowest row, `AtOrBefore` on its highest. A range `[a, b)` is `AtOrAfter(a)`, then +lands on its lowest row, and `AtOrBefore` on its highest. A range `[a, b)` is `AtOrAfter(a)`, then `NextAsync` while `Key < b`. A null seek key throws, since no entry has one. ### 3.2 Stepping and direction `NextAsync` and `PrevAsync` move one entry in `(key, row)` order and return false past the ends. -`NextKeyAsync` moves to the first entry of the next **distinct** key by seeking `After(Key)` rather -than stepping through the duplicates: a loose index scan, one seek per group, so a `GROUP BY` over a -key with a million rows costs one seek per value. `PrevKeyAsync` is `Before(Key)`. +`NextKeyAsync` moves to the first entry of the next distinct key by seeking `After(Key)` rather than +stepping through the duplicates. That is a loose index scan, one seek per group, so a `GROUP BY` over +a key with a million rows costs one seek per value. `PrevKeyAsync` is `Before(Key)`. -A cursor has a direction. Stepping the other way re-seeks at the current entry, `O(r log n)` for `r` -runs, because a heap of run positions does not run backwards; that is what every merging iterator -charges, and it is documented rather than optimized. On a sorted column, one contiguous run, a flip -costs nothing. +A cursor has a direction. Stepping the other way seeks again at the current entry, which costs +`O(r log n)` for `r` runs, because a heap of run positions does not run backwards. Every merging +iterator charges that, and it is documented rather than optimized. On a sorted column, which is one +contiguous run, a change of direction costs nothing. ### 3.3 Keys and rows `Key` is the entry's key as the member's .NET type ([07-dotnet-mapping.md](07-dotnet-mapping.md)), so it goes straight back into a seek, a filter or a comparison. `Row` is the entry's row in the file, the -coordinate a scan's `Rows` accepts: that is how a cursor's answer becomes rows, and what §5 does for -the consumer. `HasRows` is false for a `Distinct()` cursor over a keys-only source, where `Row` throws. -A step that reads no key allocates nothing. +coordinate a scan's `Rows` accepts. That is how a cursor's answer becomes rows, and what [rows in key +order](#5-rows-in-key-order) does for the consumer. `HasRows` is false for a `Distinct()` cursor over +a keys-only source, where `Row` throws. A step that reads no key allocates nothing. ### 3.4 Key order per dtype -A cursor is an ordinal structure and needs a **total** order; a filter follows IEEE 754, where NaN is -unordered ([08-semantics.md](08-semantics.md) §2). The two are kept apart by stating the total order -once, for the writer of sorted runs and for this reader: +A cursor is an ordinal structure and needs a total order, whereas a filter follows IEEE 754, where +NaN is unordered (see [NaN](08-semantics.md#2-nan)). The two are kept apart by stating the total +order once, for the writer of sorted runs and for this reader: | dtype | key order | notes | |---|---|---| -| integers | numeric | the run's key column is the column's own type | -| floats | negative NaN, −∞, negatives, −0.0, +0.0, positives, +∞, positive NaN: the row encoding's total order ([06-row-encoding.md](06-row-encoding.md) §3) | −0.0 and +0.0 are two keys. A range from a filter maps onto the slice with its IEEE meaning, `x > v` never reaching a NaN: the two orders meet by construction of the bounds, never by a comparison | +| integers | numeric | the run's key column has the column's own type | +| floats | negative NaN, −∞, negatives, −0.0, +0.0, positives, +∞, positive NaN, the row encoding's total order (see [per-type encodings](06-row-encoding.md#3-per-type-encodings)) | −0.0 and +0.0 are two keys. A range from a filter maps onto the slice with its IEEE meaning, so `x > v` never reaches a NaN. The two orders meet by the way the bounds are built, never by a comparison | | text, binary | bytewise, which is code-point order for UTF-8 | | | extensions | their storage's order | a timestamp orders as its integer | | bool | not a key source | the dictionary or the postings answer a two-value domain | -| composite `(a, b, …)` | bytewise over the row encoding of the tuple | §3.6 | +| composite `(a, b, …)` | bytewise over the row encoding of the tuple | see [composite keys](#36-composite-keys) | -Within one key, entries are in **row order**, which makes a walk deterministic, a `Distinct()` -cursor's row the key's first occurrence, and the reverse walk the exact reverse. The source owns its -comparator: runs are walked in the total order, a sorted column in the IEEE order its flag was -computed in, where −0.0 and +0.0 are one key and a column holding a NaN is not sorted at all. +Within one key, entries are in row order, which makes a walk deterministic, makes a `Distinct()` +cursor's row the key's first occurrence, and makes the reverse walk the exact reverse. The source +owns its comparator. Runs are walked in the total order, and a sorted column in the IEEE order its +flag was computed in, where −0.0 and +0.0 are one key and a column holding a NaN is not sorted at +all. ### 3.5 Rank, select, and the count of a key -`RankAsync(k)` is the number of entries with a key `< k`; `SeekRankAsync(i)` positions on the entry of -rank `i`; `KeyCountAsync()` is the number of entries sharing the current key. All three are exact on -the two sources with rows, and refused on the others. A sorted array is its own counter: rank is the -sum over runs of each run's lower bound, `O(r log n)`; select is a binary search over positions with a -rank computed at each probe, `O(r log² n)`. +`RankAsync(k)` is the number of entries with a key `< k`, `SeekRankAsync(i)` positions on the entry +of rank `i`, and `KeyCountAsync()` is the number of entries that share the current key. All three +are exact on the two sources with rows, and refused on the others. A sorted array is its own +counter: rank is the sum over runs of each run's lower bound, in `O(r log n)`, and select is a binary +search over positions with a rank computed at each probe, in `O(r log² n)`. ### 3.6 Composite keys -A sorted run over the row encoding of `(a, b, …)` ([10-indexes.md](10-indexes.md) §5.4) orders -tuples bytewise, and the encoding of `(a)` is a byte prefix of the encoding of `(a, b)`, so a prefix -on the leading columns is a range. The public cursor walks one column: a composite key's runs serve -the engine's own key-ordered reads, which a dataset's clustering key drives -([13-dataset.md](13-dataset.md) §6.6), and the row-encoding package's `RowEncoder.EncodeKey` gives -one tuple's bytes. +A sorted run over the row encoding of `(a, b, …)` (see [composite +keys](10-indexes.md#54-composite-keys-through-the-row-encoding)) orders tuples bytewise, and the +encoding of `(a)` is a byte prefix of the encoding of `(a, b)`, so a prefix on the leading columns is +a range. The public cursor walks one column. A composite key's runs serve the engine's own key-ordered +reads, which a dataset's clustering key drives (see [order beyond the +lookup](13-dataset.md#66-order-beyond-the-lookup)), and the row-encoding package's +`RowEncoder.EncodeKey` gives one tuple's bytes. ## 4. Answers without rows -Four terminals on a scan, each computed without a batch and bounded by one split of memory, whatever -the file's size. Each honours the scan's filter, row range, and pruning and index switches exactly as -its enumeration would: a terminal is the scan with a different output, never a different scan. +Four terminal operations on a scan are each computed without a batch and bounded by one split of +memory, whatever the file's size. Each honours the scan's filter, row range, and pruning and index +switches exactly as its enumeration would. A terminal is the same scan with a different output, +never a different scan. ### 4.1 `AnyAsync` -Whether the scan would return a row. Cheapest proof first, stopping at the first that decides: an -empty block mask (no data read); an exact source covering the predicate, with an entry in its slice; -a block the zone maps prove full; otherwise the live splits in order, stopping at the first row that -passes. It is also the public form of a point probe: `Where(x == v).AnyAsync()` has the whole pruning -chain in front of it. +`AnyAsync` says whether the scan would return a row, trying the cheapest proof first and stopping at +the first one that decides: an empty block mask (no data read), an exact source covering the +predicate with an entry in its slice, a block the zone maps prove full, and otherwise the live splits +in order, stopping at the first row that passes. It is also the public form of a point probe: +`Where(x == v).AnyAsync()` has the whole pruning chain in front of it. ### 4.2 `CountAsync` -The number of rows the scan would return, exactly, taking per block the cheapest of three tiers: +`CountAsync` returns the exact number of rows the scan would return, taking the cheapest of three +tiers per block: | tier | proof | cost | |---|---|---| -| **exact cover** | a comparison, `IN`, `StartsWith`, or a conjunction of those on one column with a sorted source: the sum of the slices' lengths | `O(r log n)`, no data read | -| **full-block proof** | the zone maps prove every non-null row of the block matches, and its null count is known | the zone maps, already read | -| **count-only decode** | the block decoded, the filter evaluated, its true rows counted | one decode, no gather, no batch | +| exact cover | a comparison, `IN`, `StartsWith`, or a conjunction of those on one column with a sorted source: the sum of the slices' lengths | `O(r log n)`, no data read | +| full-block proof | the zone maps prove every non-null row of the block matches, and its null count is known | the zone maps, already read | +| count-only decode | the block is decoded, the filter evaluated, and its true rows counted | one decode, no gather, no batch | -The full-block proof is the dual of pruning, with the asymmetry reversed: it proves only when the -statistics point the safe way. For `x ≥ v` a block matches whole when its stated minimum is `≥ v`, -and an `Inexact` minimum still proves it; `x = v` only when the minimum and maximum are both `v` and -both `Exact`. A zone's bounds exclude NaN, so a float block's NaN count is subtracted or the block -decoded. Predicates combine as three counts per range — true, false and unknown, any two deciding -the third — so that `NOT` stays exact and two predicates on one nullable column share their unknowns, -its nulls. What that algebra cannot see falls to the third tier. +The full-block proof is the dual of pruning, with the asymmetry reversed: it only proves when the +statistics point the safe way. For `x ≥ v`, a block matches whole when its stated minimum is `≥ v`, +and an `Inexact` minimum still proves it. For `x = v`, only when the minimum and maximum are both `v` +and both `Exact`. A zone's bounds exclude NaN, so a float block's NaN count is subtracted or the block +is decoded. Predicates combine as three counts per range (true, false and unknown, any two deciding +the third), so that `NOT` stays exact and two predicates on one nullable column share their unknowns, +which are its nulls. Whatever that algebra cannot see falls to the third tier. -**A wrong proof is a wrong count**, which is what makes it testable: every count is compared with -the materialized scan with each tier forced off in turn. +A wrong proof gives a wrong count, which is what makes it testable: every count is compared with the +materialized scan with each tier forced off in turn. ### 4.3 `MinAsync` and `MaxAsync` -The smallest and largest non-null value of a column among the scan's rows, in the filter's IEEE -order, so a NaN is never an extreme, as in the statistics. In order: the file statistic, when the -scan is the whole file and the statistic `Exact`; the zone maps, where an `Exact` bound answers a -split and an `Inexact` one is a candidate, decoded best bound first, usually once; otherwise a decode -of the live blocks with a running extreme, one split at a time. +These return the smallest and largest non-null value of a column among the scan's rows, in the +filter's IEEE order, so a NaN is never an extreme, as in the statistics. They try, in order, the file +statistic when the scan covers the whole file and the statistic is `Exact`, then the zone maps, where +an `Exact` bound answers a split and an `Inexact` one is a candidate decoded best bound first (usually +just once), and otherwise a decode of the live blocks with a running extreme, one split at a time. ### 4.4 Distinct keys -`Keys(r => r.Column).Distinct()` opens a cursor that steps by `NextKey`: every distinct non-null value, -in key order, once. It admits the keys-only sources, so a `GROUP BY status` over a billion rows reads -the postings keys of each run and never a data segment. A column with no source is refused naming -`IndexPolicy.Postings`, the cheapest structure that would serve, rather than filled into a hash set -bounded by the column. Distinct is a property of the cursor, not a terminal returning a list, because -it is an ordered walk with an early exit, and a list would buffer it. +`Keys(r => r.Column).Distinct()` opens a cursor that steps by `NextKey`, visiting every distinct +non-null value once, in key order. It accepts the keys-only sources, so a `GROUP BY status` over a +billion rows reads the postings keys of each run and never a data segment. A column with no source is +refused, naming `IndexPolicy.Postings`, the cheapest structure that would serve it, rather than +filled into a hash set bounded by the column. Distinct is a property of the cursor, not a terminal +returning a list, because it is an ordered walk with an early exit, and a list would buffer it. ## 5. Rows in key order -`OrderBy(r => r.Column, descending)` makes the scan deliver its rows in the key order of a column -instead of file order. It composes with the projection and the filter, and excludes `Rows`. +`OrderBy(r => r.Column)` and `OrderByDescending(r => r.Column)` make the scan deliver its rows in the +key order of a column instead of file order. They compose with the projection and the filter, and +exclude `Rows`. -**How.** The scan is driven by a cursor instead of the split planner. The key range is the -intersection of the slices of the filter's top-level conjuncts on the key, so an `OR` or an `IN` on -the key narrows to its exact slices. The cursor walks a **window** of entries, a batch's worth; the -entries whose block the mask pruned are skipped, since the filter's other conjuncts prune as in any -scan; the window's rows are taken as a selection, the filter evaluated on them, and the batch -permuted into the window's key order. Then the next window. Memory is one window and one split's +The scan is then driven by a cursor instead of the split planner. The key range is the intersection +of the slices of the filter's top-level conjuncts on the key, so an `OR` or an `IN` on the key +narrows to its exact slices. The cursor walks a window of entries, a batch's worth at a time. Entries +whose block the mask pruned are skipped, since the filter's other conjuncts prune as in any scan, the +window's rows are taken as a selection, the filter is evaluated on them, and the batch is permuted +into the window's key order. Then comes the next window. Memory is one window plus one split's decode, never the result. -**What it delivers.** Batches in key order, ties in row order, the descending order the exact -reverse. Rows whose key is null come **last in both directions**, read after the walk as a filtered -scan the zone maps' null counts prune. A consumer that stops early is a `LIMIT`: the windows past it -are never read. A degree of parallelism applies within a window, whose splits are independent; -windows stay in order. - -**What it costs**, which a caller should read before using it. A window of `W` entries touches as many -splits as its rows are scattered over. On a sorted column the rows of consecutive keys are -consecutive, so the walk is a plain scan. On sorted runs over a column **uncorrelated with file -order**, a window can touch a split per row, and a scattered take costs what it costs: 64 rows of an -uncorrelated column read in key order take longer than Rust's reader takes for 64 rows scattered the -same way whose positions it is given, a take over 64 splits (the `against a take` axis of -`--ratio-check`, a gate the benchmark page leaves out), and less than Rust's filtered scan of the -same band, which is what a reader without the index has to do (the key-order axes of -[05-benchmarks.md](05-benchmarks.md) §3, measured on -[the benchmark page](../guide/benchmarks.md#in-one-process-after-warm-up)). So it is the tool for a -selective range, a top-k or a merge, and not for a whole uncorrelated column, where sorting the -scan's output costs less. The plan's `OrderPlan` states the source, the runs the range reaches and -the entries it admits before the read; the scan's statistics count the windows and the splits they -touched after it. - -The dataset's key-ordered reads and its compaction merge are built on this, one call per object -([13-dataset.md](13-dataset.md) §5.3, §6.6). +What it delivers is batches in key order, ties in row order, with the descending order the exact +reverse. Rows whose key is null come last in both directions, read after the walk as a filtered scan +that the zone maps' null counts prune. A consumer that stops early acts as a `LIMIT`, and the windows +past it are never read. A degree of parallelism applies within a window, whose splits are +independent, while the windows stay in order. + +What it costs should be read before using it. A window of `W` entries touches as many splits as its +rows are scattered over. On a sorted column, the rows of consecutive keys are consecutive, so the +walk is a plain scan. On sorted runs over a column uncorrelated with file order, a window can touch +one split per row, and a scattered take costs what it costs. Reading 64 rows of an uncorrelated +column in key order takes longer than Rust's reader takes for 64 rows scattered the same way whose +positions it is given, a take over 64 splits (the `against a take` axis of `--ratio-check`, a gate +the benchmark page leaves out). It takes less than Rust's filtered scan of the same band, which is +what a reader without the index has to do (the key-order axes [in one +process](05-benchmarks.md#3-in-one-process-after-warm-up), measured on [the benchmark +page](../guide/benchmarks.md#in-one-process-after-warm-up)). So it is the tool for a selective range, +a top-k or a merge, not for a whole uncorrelated column, where sorting the scan's output costs less. +The plan's `OrderPlan` states the source, the runs the range reaches and the entries it admits +before the read, and the scan's statistics count the windows and the splits they touched afterwards. + +The dataset's key-ordered reads and its compaction merge are built on this, one call per object (see +[what a compaction does](13-dataset.md#53-what-a-compaction-does) and [order beyond the +lookup](13-dataset.md#66-order-beyond-the-lookup)). ## 6. Predicates on text and lists -Four predicates beyond comparisons, `IN` and nullity, each with its pruning: +Four predicates go beyond comparisons, `IN` and nullity, each with its own pruning: | predicate | true when | null | pruning | |---|---|---|---| -| `StartsWith(p)` | the value's bytes begin with `p` | unknown | as a range `[p, succ(p))`, where `succ` increments the last byte after dropping trailing `0xFF`s: the zone bounds and a sorted run answer it, a count by exact cover | -| `Contains(p)` | `p` occurs in the value | unknown | the trigram filter and postings when `p` has three bytes or more ([10-indexes.md](10-indexes.md) §4.2, §5.3) | -| `Like(pattern)` | SQL `LIKE`: `%` any run, `_` any one character of a utf8 column and one byte of a binary one, an escape quoting either | unknown | a leading literal as `StartsWith`, every literal run of three bytes or more through the trigram structures; always re-evaluated on survivors | -| `ListContains(list, v)` | an element of the row's list equals `v`, under the comparison kernels' equality | unknown on a null list or under a null `v`; a null element matches nothing | the list's null counts; the Bloom filter over the elements ([10-indexes.md](10-indexes.md) §4.1) | +| `StartsWith(p)` | the value's bytes begin with `p` | unknown | as a range `[p, succ(p))`, where `succ` increments the last byte after dropping trailing `0xFF`s. The zone bounds and a sorted run answer it, and a count by exact cover | +| `Contains(p)` | `p` occurs in the value | unknown | the trigram filter and postings when `p` has three bytes or more (see [trigram filters](10-indexes.md#42-vorticitybloomngram3v1-trigrams-for-text-predicates) and [trigram postings](10-indexes.md#53-vorticitypostingsngram3v1-trigrams-to-blocks)) | +| `Like(pattern)` | SQL `LIKE`: `%` matches any run, `_` any one character of a utf8 column or one byte of a binary one, and an escape quotes either | unknown | a leading literal as `StartsWith`, and every literal run of three bytes or more through the trigram structures. Always evaluated again on the survivors | +| `ListContains(list, v)` | an element of the row's list equals `v`, under the comparison kernels' equality | unknown on a null list or under a null `v`. A null element matches nothing | the list's null counts, and the Bloom filter over the elements (see [the Bloom filter](10-indexes.md#41-vorticitybloomsbbfv1-a-split-block-bloom-filter-per-block)) | -Matching is bytewise, but for `_` over utf8, and case-sensitive; `NOT StartsWith` claims nothing +Matching is bytewise, except for `_` over utf8, and case-sensitive. `NOT StartsWith` claims nothing from a zone map, which is the safe answer. On the typed path these are members of `Sym` and -of a list member's symbol; on the tool path, expressions of `VortexExpr`. +of a list member's symbol, and on the tool path, expressions of `VortexExpr`. -## 7. What is not on the surface, on purpose +## 7. What is not in the public API, on purpose -A key-range type (the filter says it), a `Probe(column, value)` (`AnyAsync` is it), a distinct -terminal returning a list (the cursor walks it), a key-ordered take (the window inside `OrderBy` is -the only reorder), a sort (constraint 1), and a comparator on the cursor (keys arrive typed and in -order). Each would be a second way to say something one member already says, or a way to buffer a -file. +A key-range type (the filter already says it), a `Probe(column, value)` (`AnyAsync` is it), a +distinct terminal returning a list (the cursor walks it), a key-ordered take (the window inside +`OrderBy` is the only reordering), a sort (ruled out by the first constraint), and a comparator on the +cursor (keys arrive typed and in order). Each would be a second way to say something one member +already says, or a way to buffer a whole file. ## 8. How it is tested - A cursor's walk, forwards and backwards, with and without `Distinct()`, equals a sort of the materialized column, on every source, with windows and lanes. -- `rank(select(i)) ≤ i < rank(select(i)) + KeyCount(select(i))` over every entry. +- `rank(select(i)) ≤ i < rank(select(i)) + KeyCount(select(i))` holds over every entry. - Every count and extreme equals the materialized scan's, with pruning, indexes and each count tier forced off in turn. - A key-ordered scan equals a sort of the filtered scan's rows, nulls last, in both directions. diff --git a/docs/design/13-dataset.md b/docs/design/13-dataset.md index c9a662ac..7fc78979 100644 --- a/docs/design/13-dataset.md +++ b/docs/design/13-dataset.md @@ -1,55 +1,56 @@ # Dataset: a log-structured tree of Vortex files -`Vorticity.Dataset`, experimental (`VX0001`), keeps a versioned table over an object store: rows +`Vorticity.Dataset`, experimental under `VX0001`, keeps a versioned table over an object store: rows appended over time into plain Vortex files, compacted in the background, indexed progressively, and read as one table at any version. [datasets.md](../guide/datasets.md) and [dataset-maintenance.md](../guide/dataset-maintenance.md) show it in use. -**The requirement: no read-path cost proportional to any N.** Not the bytes of data, not the number -of data objects, commits, appends since the last compaction, blocks of an object or index fragments. -Every bound is a constant of the design — a fan-out, a page size, a level count — times a logarithm, +The requirement is that no read-path cost grows with any N: not the bytes of data, not the number of +data objects, commits, appends since the last compaction, blocks of an object or index fragments. +Every bound is a constant of the design (a fan-out, a page size, a level count) times a logarithm, plus the size of the answer. The write path pays instead, once and amortized, in compaction, and the -price is stated (§5.4). +price is stated (see [the price](#54-the-price)). -**What does not bend:** every data object is a plain Vortex file a strict Rust 0.86.1 reader opens, -with this library's structures invisible to it; the core takes no third-party dependency; an index is -a hint unless it is proven complete ([10-indexes.md](10-indexes.md) §7); no interleaving of readers, -writers, indexers, compactions and vacuums produces a wrong answer; and **the S3 client is not this -library's**: another library implements the store seam (§11), and this one never names S3 in code. +Some things do not bend. Every data object is a plain Vortex file that a strict Rust 0.86.1 reader +opens, with this library's structures invisible to it. The core takes no third-party dependency. An +index is a hint unless it is proven complete (see [indexing after the +fact](10-indexes.md#7-building-appending-and-indexing-after-the-fact)). No interleaving of readers, +writers, indexers, compactions and vacuums produces a wrong answer. And the S3 client is not this +library's: another library implements the store interface ([the store +abstraction](#11-the-store-abstraction)), and this one never names S3 in code. ## 1. Where the N's hide, and the bound that replaces each | N | the bound | mechanism | |---|---|---| -| bytes of data | nothing on the read path | identity from bytes the open already reads; the content hash is the writer's, checked only offline (§7) | -| data objects | O(log N) pages read, at most 8 + L objects touched by a key lookup | a copy-on-write tree of objects with summaries per node (§4), leveled compaction on a clustering key (§5) | -| commits | one `List` finds the latest; a commit writes O(depth) pages in one object | newest-first keys (§8.3); path copying into one commit object (§4.3) | -| appends since compaction | at most K runs per index entry per object, at most 8 objects in level 0 | one rule inside a file and across files (§5.2, §6.1) | -| blocks of an object | `1 + 16 · log₁₆ blocks` filter reads, output-sensitive | the filter tree ([10-indexes.md](10-indexes.md) §3) | -| entries of a run | two fence pages to 10¹² entries | fence pages ([10-indexes.md](10-indexes.md) §3) | -| fragments | one ranged read each, inside the commit object that wrote it | §6.4 | -| distinct keys, on a Bloom filter | the structure is chosen by cardinality | §6.5 | -| rows deleted since compaction | a binary search over at most 1 KiB of runs per batch or step, and at most an eighth of an object's rows stepped over | a deletion vector per leaf entry, folded by compaction (§12) | +| bytes of data | nothing on the read path | identity from bytes the open already reads. The content hash is the writer's, checked only offline ([identity](#7-identity-and-integrity)) | +| data objects | O(log N) pages read, at most 8 + L objects touched by a key lookup | a copy-on-write tree of objects with summaries per node ([the tree](#4-the-tree)), leveled compaction on a clustering key ([levels](#5-levels-and-compaction)) | +| commits | one `List` finds the latest, and a commit writes O(depth) pages in one object | newest-first keys ([the latest version](#83-the-latest-version-in-one-request)), path copying into one commit object ([commits](#43-commits-are-paths-written-as-one-object)) | +| appends since compaction | at most four runs per index entry per object, at most 8 objects in level 0 | one rule inside a file and across files ([the invariant](#52-the-invariant), [runs](#61-one-run-per-entry-per-object)) | +| blocks of an object | `1 + 16 · log₁₆ blocks` filter reads, output-sensitive | the filter tree ([the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)) | +| entries of a run | two fence pages up to 10¹² entries | fence pages ([the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)) | +| fragments | one ranged read each, inside the commit object that wrote it | [fragments](#64-fragments-in-commit-objects-transient) | +| distinct keys, on a Bloom filter | the structure is chosen by cardinality | [which structure](#65-which-structure-for-which-column) | +| rows deleted since compaction | a binary search over at most 1 KiB of runs per batch or step, and at most an eighth of an object's rows stepped over | a deletion vector per leaf entry, folded by compaction ([deleting rows](#12-deleting-and-updating-rows)) | ## 2. Principles -1. **Immutable objects, one commit object per version.** Nothing committed is rewritten; a reader - holding a version's header holds a snapshot. -2. **Metadata is a tree, not a list.** The set of data objects is a copy-on-write tree whose nodes - carry summaries: reading descends, committing copies a path, and neither is O(objects). -3. **Bounded fan-in everywhere**: K runs per index entry per object, 8 objects in level 0, L levels, - one object per level for a clustering-key lookup. Compaction holds the bounds; a plan reports - when it lags. -4. **Cost is output-sensitive or logarithmic**, never proportional to the dataset: a predicate that +1. Objects are immutable, with one commit object per version. Nothing committed is rewritten, and a + reader holding a version's header holds a snapshot. +2. Metadata is a tree, not a list. The set of data objects is a copy-on-write tree whose nodes carry + summaries. Reading descends and committing copies a path, and neither is O(objects). +3. Fan-in is bounded everywhere: four runs per index entry per object, 8 objects in level 0, L levels, + and one object per level for a clustering-key lookup. Compaction holds the bounds, and a plan + reports when it lags. +4. Cost is output-sensitive or logarithmic, never proportional to the dataset. A predicate that matches everything reads everything, which is the answer's own size. -5. **The write path pays**: merges when a file completes, filter trees, fences, compaction, each +5. The write path pays: merges when a file completes, filter trees, fences and compaction, each stated as a write amplification. -6. **Identity from bytes already read; integrity by checksums of what is read.** Pages are +6. Identity comes from bytes already read, and integrity from checksums of what is read. Pages are referenced by their XXH3-128, commit headers and index regions carry an XXH3-64, and nothing on the read path hashes data. -7. **Derived data folds back into the data**: a compaction that rewrites an object embeds its - indexes, so external fragments are a transient layer between "written without an index" and - "next rewritten". +7. Derived data folds back into the data. A compaction that rewrites an object embeds its indexes, so + external fragments are a transient layer between "written without an index" and "next rewritten". ## 3. Objects and keys @@ -57,264 +58,284 @@ A dataset is a store prefix holding two kinds of immutable object: | object | key | content | |---|---|---| -| data object | `data/.vortex` | a plain Vortex file, its indexes embedded, its identity in its postscript (§7) | -| commit object | `commit/.vxc` | everything else one commit produced: a header, the tree pages it wrote, the index fragments it wrote, a table of both, a trailer | - -**The commit object** opens by its head, where a Vortex file opens by its tail: magic `VXCOMMIT`, a -format version, the header's length, the header (Protobuf), then pages and fragments in the order -they were added, the table, and a 32-byte trailer — the table's offset and length, **the object's own -length**, an XXH3-64 over the header and the table, and magic `VXCT`. Recording the length turns any -truncation into a stated error rather than a shorter object. The **header** holds the version, its -parent, the schema and the names its columns gave up (§13), the clustering key, the write policy, -the tree's chunking parameters and seed, -the compaction and retention settings, and per level where its last round-robin compaction stopped -and its top page **inlined**, with the pages below it while the header stays under 256 KiB. One ranged read of the first 256 KiB opens a commit. The -pages region follows the header, so a page the commit wrote itself that ends inside those 256 KiB is -not inlined: the read that opens the commit brings it back once, where inlined it would be written -twice. Nor is a leaf that holds an object with marked rows (§12), unless this commit wrote it: its -bytes grow with every mark, and carried by every header they would cost each commit what only those -that change its level have to pay. For every page it names without carrying it — a level's top, or -a child of a page it carries — the header records where that page's version keeps its pages, as far -as the writer knows it, so a reader reads such a page in one request rather than two. - -**A page reference** is a fixed 36 bytes: the version that wrote the page, its offset relative to that +| data object | `data/.vortex` | a plain Vortex file, its indexes embedded, its identity in its postscript ([identity](#7-identity-and-integrity)) | +| commit object | `commit/.vxc` | everything else one commit produced: a header, the tree pages it wrote, the index fragments it wrote, a table of both, and a trailer | + +The commit object opens by its head, where a Vortex file opens by its tail: magic `VXCOMMIT`, a format +version, the header's length, the header (Protobuf), then pages and fragments in the order they were +added, the table, and a 32-byte trailer. The trailer holds the table's offset and length, the +object's own length, an XXH3-64 over the header and the table, and magic `VXCT`. Recording the length +turns any truncation into a stated error rather than a shorter object. + +The header holds the version, its parent, the schema and the names its columns gave up (see [schema +evolution](#13-schema-evolution)), the clustering key, the write policy, the tree's chunking +parameters and seed, the compaction and retention settings, and, per level, where its last +round-robin compaction stopped and its top page inlined, with the pages below it as long as the +header stays within 192 KiB. One ranged read of the first 256 KiB opens a commit. The pages region +follows the header, so a page the commit wrote itself that ends inside those 256 KiB is not inlined, +since the read that opens the commit brings it back once, where inlined it would be written twice. +Nor is a leaf holding an object with marked rows (see [deleting rows](#12-deleting-and-updating-rows)) +inlined, unless this commit wrote it: its bytes grow with every mark, and carried by every header they +would cost each commit what only the commits that change its level should pay. For every page it +names without carrying it (a level's top, or a child of a page it carries), the header records where +that page's version keeps its pages, as far as the writer knows, so a reader fetches such a page in +one request rather than two. + +A page reference is a fixed 36 bytes: the version that wrote the page, its offset relative to that object's pages region, its length, and its XXH3-128. A commit references every page it did not change -where it already lies, in an older commit object, and checks the hash on read; fixed-width, -relative references are what let a header name offsets that depend on its own length, and what make -two identical trees identical page for page. There is no separate page object, no orphan page after -a crash, and one conditional creation per commit. A reference names its page in every version that -shares it, so a handle keeps what it has from one version to the next: the pages it read from the -store, least recently used first out within `DatasetOptions.PageCacheBytes` (32 MiB by default); what -the reads opening its last sixteen versions brought back of their pages regions; and where each -version's pages start. A refresh or a commit asks the store again only for the pages that changed. -Pages are kept page by page only when read from the store, and by version when a read brought them -back with their commit: most of what a version wrote, the next one rewrites, and a budget in bytes -would fill with those. - -**``** is `10²⁰ − 1 − version` in twenty digits, so the **newest commit sorts -first** and the first key of a listing of `commit/` is the latest version (§8.3). There is no mutable -object anywhere. - -An existing Vortex file becomes a leaf by import, **without a copy**: the store grows by the commit -object alone. A process that lists `data/` and reads every file sees uncommitted and superseded -objects, as in every table format: the latest commit object is the only definition of the content. +where it already lies, in an older commit object, and checks the hash on read. Fixed-width, relative +references are what let a header name offsets that depend on its own length, and what make two +identical trees identical page for page. There is no separate page object, no orphan page after a +crash, and one conditional creation per commit. + +A reference names its page in every version that shares it, so a handle keeps what it has from one +version to the next: the pages it read from the store, evicted least recently used first within +`DatasetOptions.PageCacheBytes` (32 MiB by default), what the reads that opened its last sixteen +versions brought back of their pages regions, and where each version's pages start. A refresh or a +commit only asks the store again for the pages that changed. Pages are kept page by page only when +read from the store, and kept by version when a read brought them back with their commit, since most +of what a version wrote the next one rewrites, and a budget in bytes would fill up with those. + +`` is `10²⁰ − 1 − version` in twenty digits, so the newest commit sorts first and +the first key of a listing of `commit/` is the latest version (see [the latest +version](#83-the-latest-version-in-one-request)). There is no mutable object anywhere. + +An existing Vortex file becomes a leaf by import, without a copy: the store only grows by the commit +object. A process that lists `data/` and reads every file sees uncommitted and superseded objects, as +in every table format. The latest commit object is the only definition of the content. ## 4. The tree ### 4.1 A prolly tree per level -Each level (§5) is a **copy-on-write search tree over data objects**, ordered by the clustering key -when the dataset declares one and by first row position otherwise. It is a **prolly tree**, the -probabilistic B-tree of Noms and Dolt. Once a commit is a sorted batch of changes per level, a prolly -tree and a B+tree with path copying are one algorithm — load the touched leaves, merge the changes, -re-emit them, rebuild the nodes above up to the top — and only the rule that places page boundaries -differs. From the prolly rule alone follow history independence, deduplication across writers and -the strongest test oracle there is (§14). - -**The boundary rule.** A boundary is decided per entry from the XXH3-64 of the entry's key alone, -seeded with sixteen random bytes the dataset draws once, so values never move a boundary and an -indexer that updates an object's descriptor rewrites exactly `depth` pages. It is normalized by -size: no boundary before 64 KiB of entries, a probability proportional to the room left after that, -`k · entryBytes / (max − bytes)` with `k = 2`, which puts the mean page at 128 KiB by construction, -and a forced boundary at 256 KiB, the page cap. After an edit, chunking continues until a new -boundary meets an old one, so **the tree is a pure function of the key set and the parameters**: the -same objects committed in any order give byte-identical pages and one root hash. The rule is behind a -seam, whose other implementation, a B+tree's fill factor, is measured on the same bench. +Each level ([levels](#5-levels-and-compaction)) is a copy-on-write search tree over data objects, +ordered by the clustering key when the dataset declares one and by first row position otherwise. It +is a prolly tree, the probabilistic B-tree of Noms and Dolt. Once a commit is a sorted batch of +changes per level, a prolly tree and a B+tree with path copying are the same algorithm (load the +touched leaves, merge the changes, emit them again, rebuild the nodes above up to the top), and only +the rule that places page boundaries differs. The prolly rule alone gives history independence, +deduplication across writers and the strongest test oracle available ([how it is +tested](#15-how-it-is-tested)). + +A boundary is decided per entry from the XXH3-64 of the entry's key alone, seeded with sixteen random +bytes the dataset draws once, so values never move a boundary and an indexer that updates an object's +descriptor rewrites exactly `depth` pages. It is normalized by size: no boundary before 64 KiB of +entries, then a probability proportional to the room left, `k · entryBytes / (max − bytes)` with +`k = 2`, which puts the mean page at 128 KiB by construction, and a forced boundary at 256 KiB, the +page cap. After an edit, chunking continues until a new boundary meets an old one, so the tree is a +pure function of the key set and the parameters: the same objects committed in any order give +byte-identical pages and one root hash. The rule sits behind an interface whose other +implementation, a B+tree's fill factor, is measured on the same bench. With entries of about 180 bytes the fan-out is about 680. The header inlines a level's top page and, -while it fits, the pages below, so a lookup reads **0 dependent pages** up to about 650 objects, -**1** up to about 400 000, **2** up to about 280 million; a leaf with marked rows that another commit -wrote is one more, once (§3). +while they fit, the pages below it, so a lookup reads 0 dependent pages up to about 650 objects, 1 up +to about 400 000, and 2 up to about 280 million. A leaf with marked rows that another commit wrote +costs one more read, once (see [objects and keys](#3-objects-and-keys)). ### 4.2 What a node carries -A **leaf entry** describes one data object: +A leaf entry describes one data object: -- **its key**: the row encoding of the object's smallest clustering key followed by its first row +- its key: the row encoding of the object's smallest clustering key followed by its first row position, or the position alone without a clustering key, then the object's uid, which makes the key - a function of the object rather than of the tree it was committed into (§8.2); + a function of the object rather than of the tree it was committed into (see + [rebase](#82-rebase-by-re-applying-operations)) - its size, row count and identity: the uid in its postscript, the store's token when the store - gives one, and the XXH3-128 the writer computed (§7); -- **summaries**: per summarized column its minimum, maximum and null count, taken from the object's - own file statistics before its put, so they cost the append no request. The first 32 top-level - columns are summarized by default (`DatasetOptions.SummaryColumns`), and the clustering key's - always; -- its index descriptor: embedded in the object, or per entry the fragments that cover it (§6.4); -- **its deletion vector**, when a delete marked rows in it: their positions as sorted runs, after - everything else the entry holds, so that an entry without one is the bytes it always was. Its row - count is then the live rows, and its summaries bounds only (§12). A vector takes at most 1 KiB, - which the fan-out of the page holding it pays until the object is next rewritten: a page of - entries all at the cap holds about a hundred. - -An **internal entry** holds a page reference to a child, the sum of its rows, the **union** of its -children's summaries, and the **tally** of the objects under it: their bytes, the largest of them, the -rows marked deleted in them, the most index fragments one of them carries, and the most any one of -them has marked, as a share of its rows and as the bytes of its vector, which a compaction plans by -(§5.3). The union is intersection-shaped, which is the one thing about it a reader must -get right: a column travels up only when every child carries it, and a bound only when every child -has one, since keeping a bound because another child was silent would prune a subtree that holds the -answer. One inexact child makes the union inexact. - -**What the walk skips.** A subtree whose union refutes the predicate is skipped unread; an object whose -own summaries refute it is not opened; an object outside a row range is neither read nor counted, by -the row sums at every level. The question is asked by the core's own pruner, the one a file's -statistics use, so there is one implementation of [08-semantics.md](08-semantics.md) §1 and two -callers. Measured: a filter inside one of eight objects opens one object with the summaries and eight -without, for the same rows. + provides one, and the XXH3-128 the writer computed ([identity](#7-identity-and-integrity)) +- summaries: per summarized column, its minimum, maximum and null count, taken from the object's own + file statistics before its put, so they cost the append no request. The first 32 top-level columns + are summarized by default (`DatasetOptions.SummaryColumns`), and the clustering key's always are +- its index descriptor: indexes embedded in the object, or per entry the fragments that cover it (see + [fragments](#64-fragments-in-commit-objects-transient)) +- its deletion vector, when a delete marked rows in it: their positions as sorted runs, placed after + everything else the entry holds, so an entry without one keeps exactly the bytes it always had. Its + row count is then the live rows, and its summaries become bounds only ([deleting + rows](#12-deleting-and-updating-rows)). A vector takes at most 1 KiB, which reduces the fan-out of + the page holding it until the object is next rewritten: a page of entries all at the cap holds + about a hundred + +An internal entry holds a page reference to a child, the sum of its rows, the union of its children's +summaries, and the tally of the objects under it: their bytes, the largest of them, the rows marked +deleted in them, the most index fragments any one of them carries, and the most any one of them has +marked, both as a share of its rows and as the bytes of its vector, which a compaction plans by (see +[what a compaction does](#53-what-a-compaction-does)). The union behaves like an intersection, which +is the one thing about it a reader must get right. A column only travels up when every child carries +it, and a bound only when every child has one, since keeping a bound because another child was silent +would prune a subtree that holds the answer. One inexact child makes the union inexact. + +The walk skips a subtree whose union refutes the predicate without reading it. An object whose own +summaries refute it is not opened, and an object outside a row range is neither read nor counted, +thanks to the row sums at every level. The question is asked by the core's own pruner, the same one +a file's statistics use, so there is one implementation of [statistic +precision](08-semantics.md#1-statistic-precision-what-inexact-licenses) and two callers. Measured: a +filter inside one of eight objects opens one object with the summaries, and eight without, for the +same rows. ### 4.3 Commits are paths, written as one object -A commit is a sorted batch of changes per level: entries added, removed, or updated. Adding an object -re-chunks its leaf and the pages above it, O(depth) pages, laid out inside the commit object with the -header and the fragments; every page the commit did not change is referenced where it lies, and a page -whose new content equals one the writer has already read is referenced rather than rewritten, which -the prolly rule makes common after a rebase. One conditional creation commits it (§8), its cost does -not depend on the number of objects, and a crash leaves nothing to clean up but data objects. +A commit is a sorted batch of changes per level: entries added, removed or updated. Adding an object +chunks its leaf again along with the pages above it, O(depth) pages, laid out inside the commit object +with the header and the fragments. Every page the commit did not change is referenced where it lies, +and a page whose new content equals one the writer has already read is referenced rather than +rewritten, which the prolly rule makes common after a rebase. One conditional creation commits it +([the commit protocol](#8-the-commit-protocol)), its cost does not depend on the number of objects, +and a crash leaves nothing to clean up except data objects. ## 5. Levels and compaction ### 5.1 Why levels are part of the read bound -Without a merge policy, the objects a lookup touches are the appends. The bound of §2 holds only if -compaction holds an invariant, so the invariant is specified, and a plan (`PlanCompactionAsync`) -reports every violation of it as a **lag** without opening a data object. A lag is reported, never -refused: compaction is the caller's background job, and **a library never stalls a writer**. +Without a merge policy, the objects a lookup touches are the appends themselves. The bound of the +[principles](#2-principles) only holds if compaction maintains an invariant, so the invariant is +specified, and a plan (`PlanCompactionAsync`) reports every violation of it as a lag without opening +a data object. A lag is reported, never refused: compaction is the caller's background job, and a +library never stalls a writer. ### 5.2 The invariant -- **Level 0** holds appended objects as they came, overlapping in key: at most **8**. -- **Levels 1 to L**, when the dataset declares a clustering key, hold objects with **disjoint key - ranges** within a level, so a lookup by key touches at most one object per level: **8 + L** in all. - This is leveled compaction as RocksDB does it, with Vortex files as the sorted tables. Level `i` - holds `128 KiB × F^i`, `F` = 10, and an object written into it targets `128 KiB × F^(i−1)`, **capped - at 4 MiB**: the cap is what a delete rewrites (§12), and the capacities keep growing past it, so - that a level holds `F` times the one below whatever the size of its objects and a dataset of `N` - bytes spans about `log_F(N / 128 KiB)` levels. A level holding a single object is never over its - size. -- Without a clustering key, levels are **size tiers**: a lookup touches every object the summaries do - not refute, which is output-sensitive rather than bounded, and the plan says so. -- **Inside every object, at most K = 4 runs per index entry**, the same rule as a single file - appended in place ([10-indexes.md](10-indexes.md) §3). - -`CompactionOptions` carries the fan-out, the level-0 ceiling, the level-1 target, the object cap and -a cap on levels: under a cap, the top level has no size bound and only grows, since merging it into -itself would rewrite key-disjoint objects into the same objects. +- Level 0 holds appended objects as they came, overlapping in key, and at most 8 of them. +- Levels 1 to L, when the dataset declares a clustering key, hold objects with disjoint key ranges + within a level, so a lookup by key touches at most one object per level, 8 + L in all. This is + leveled compaction as RocksDB does it, with Vortex files as the sorted tables. Level `i` holds + `128 KiB × F^i`, with `F` = 10, and an object written into it targets `128 KiB × F^(i−1)`, capped + at 4 MiB. The cap is what a delete rewrites ([deleting rows](#12-deleting-and-updating-rows)), and + the capacities keep growing past it, so a level holds `F` times the one below whatever the size of + its objects, and a dataset of `N` bytes spans about `log_F(N / 128 KiB)` levels. A level holding a + single object is never over its size. +- Without a clustering key, levels are size tiers. A lookup touches every object the summaries do not + refute, which is output-sensitive rather than bounded, and the plan says so. +- Inside every object, an index entry has at most four runs, the same rule as for a single file + appended in place (see [the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)). + +`CompactionOptions` carries the fan-out, the level 0 ceiling, the level 1 target, the object cap and a +cap on the number of levels. Under a level cap, the top level has no size bound and only grows, since +merging it into itself would rewrite key-disjoint objects into the same objects. ### 5.3 What a compaction does -It reads `k` objects and writes one or more at the destination level's target: a k-way merge of the -inputs on the clustering key when there is one, a concatenation otherwise, and **the writer builds the -outputs' embedded indexes once**, dropping the inputs' fragments. Compaction is therefore also the -index compaction of everything it touches. - -**Level 0 is not sorted, and nobody sorts it.** An append arrives in any order and the writer streams -with bounded memory ([11-write-strategy.md](11-write-strategy.md)). So the write policy **requires a -sorted run on the clustering key** in every object whose rows may be out of order -([10-indexes.md](10-indexes.md) §6), and the compaction reads each input **in key order through that -run** ([12-index-reads.md](12-index-reads.md) §5): the merge's inputs are ordered, its outputs are -sorted by construction, and the writer never learns to sort. The run is the object's structure rather -than a hint, so the index budget spares it: on a narrow table it is as large as the column it orders, -and a budget of a tenth of the data would refuse it on every object past a mebibyte, the merge's -outputs included. - -**An object whose rows come in key order carries no run** when the key is one integer column that -holds no null: a merge's output, or the rewrite of one (§12). Its statistics say the column is -sorted, and a sorted column is the source a key cursor and a key-ordered read take before any run, -its zone map as the index, so a run beside it is bytes nobody reads. The seal checks that a key cursor -opens on the column alone, so that an object whose order was lost is refused rather than read as one -without the key. - -**Level 0 empties into the first level whose capacity holds it**, not always into level 1. A few -small commits, a few kilobytes, merge into level 1, rewriting a level a few times their size rather -than the levels that hold the dataset; a load of gigabytes goes past the levels it would only -overflow, and is written once rather than once per level on its way up. Nothing orders the levels -among themselves but their sizes: a row in any level is a row of the dataset, and each level above 0 +A compaction reads `k` objects and writes one or more at the destination level's target: a k-way +merge of the inputs on the clustering key when there is one, and a concatenation otherwise. The +writer builds the outputs' embedded indexes once, dropping the inputs' fragments, so compaction is +also the index compaction of everything it touches. + +Level 0 is not sorted, and nothing ever sorts it. An append arrives in any order, and the writer +streams with bounded memory ([11-write-strategy.md](11-write-strategy.md)). So the write policy +requires a sorted run on the clustering key in every object whose rows may be out of order (see [the +policy](10-indexes.md#6-the-policy)), and the compaction reads each input in key order through that +run (see [rows in key order](12-index-reads.md#5-rows-in-key-order)). The merge's inputs are ordered, +its outputs are sorted by construction, and the writer never has to learn to sort. The run is the +object's structure rather than a hint, so the index budget spares it. On a narrow table it is as large +as the column it orders, and a budget of a tenth of the data would refuse it on every object past a +mebibyte, the merge's outputs included. + +An object whose rows come in key order carries no run when the key is a single integer column with +no nulls, as is the case for a merge's output or its rewrite ([deleting +rows](#12-deleting-and-updating-rows)). Its statistics say the column is sorted, and a sorted column +is the source a key cursor and a key-ordered read take before any run, with its zone map as the index, +so a run next to it would be bytes nobody reads. Sealing the object checks that a key cursor opens on +the column alone, so an object whose order was lost is refused rather than read as one without the +key. + +Level 0 empties into the first level whose capacity can hold it, not always into level 1. A few +small commits of a few kilobytes merge into level 1, rewriting a level a few times their size rather +than the levels that hold the dataset. A load of gigabytes skips the levels it would only overflow, +and is written once rather than once per level on its way up. Nothing orders the levels among +themselves except their sizes: a row in any level is a row of the dataset, and each level above 0 only has to stay key-disjoint. -The merge emits **windows, not rows**: at each step the input holding the smallest key emits every -row at or below the smallest key the others hold, a contiguous window of its current batch, so the +The merge emits windows, not rows. At each step the input holding the smallest key emits every row at +or below the smallest key the others hold, which is a contiguous window of its current batch, so the cost is `k` comparisons per window. Rows are compared by their row encoding, the order the tree, the -runs and the seeks already use. An output is sealed at the first key change past its target, **never -inside a key**, since two outputs sharing a boundary value would overlap; the destination inputs are -chosen against the union of the source range. Rows with a null key come last. A composite key merges -on its tuple; a composite key whose column holds nulls is refused, since its run holds no such tuple. -The rows written are checked against the inputs' count: a rewrite that loses rows is the one failure -it must not have. Measured on four interleaved level-0 objects: 42 808 bytes in, one sorted object of -16 596 out. - -Triggers: level 0 above its ceiling, a level above its size, an object whose marks are due (§12), -then an entry above K fragments, a compaction of index bytes only (§6.4). - -**A plan descends the trees rather than reading them.** A level's bytes are the sum of the -tallies its top page carries, and the header carries that page, or the handle keeps it when it is a -leaf with marks (§3). The object a job pushes down from a level over its size is at the end of one -descent: the largest, a leveled dataset's default, following the largest tally at each page, or -under a round robin (`CompactionOptions.Pick`) the first past the key the level's last job stopped -at, which the header records with the level, following the keys. The objects of the level below -that the job's key range meets, the objects over their fragments and the one whose marks are the -most due are found by walks that skip every subtree whose summary or tally rules it out. A cold plan -over 125 000 objects asks the store for two pages, where reading every leaf asks for 124. A tiered -job concatenates a run of one level's objects that nothing else sits between. Under a round robin, -a tiered dataset's default, it takes the run that starts past the level's pointer, which ends -before the first object another level holds past that start, one descent per level: a plan over -100 000 objects by arrival asks for one page, where reading every leaf asks for 71. The longest run -(`CompactionPick.Largest`) is a fact of every level's order in full, and a plan that takes it still -reads every leaf. So does a plan over a level whose pages were written before tallies, until a -commit rewrites them. Both choose the job the other would. When many processes append, one coordinator batching their -rows into one commit answers the contention; without it the protocol of §8 stays correct, only -slower. +runs and the seeks already use. An output is sealed at the first key change past its target, never +inside a key, since two outputs sharing a boundary value would overlap, and the destination inputs +are chosen against the union of the source range. Rows with a null key come last. A composite key +merges on its tuple, and a composite key whose columns hold nulls is refused, since its run holds no +such tuple. The rows written are checked against the inputs' count, because a rewrite that loses rows +is the one failure it must not have. Measured on four interleaved level 0 objects: 42 808 bytes in, +and one sorted object of 16 596 out. + +The triggers, in order, are level 0 above its ceiling, a level above its size, an object whose marks +are due ([deleting rows](#12-deleting-and-updating-rows)), and then an entry with more than four +fragments, which is a compaction of index bytes only (see +[fragments](#64-fragments-in-commit-objects-transient)). + +A plan descends the trees rather than reading them. A level's bytes are the sum of the tallies its top +page carries, and the header carries that page, or the handle keeps it when it is a leaf with marks. +The object a job pushes down from a level over its size sits at the end of one descent: the largest, +the default for a leveled dataset, found by following the largest tally at each page, or, under a +round robin (`CompactionOptions.Pick`), the first one past the key where the level's previous job +stopped, which the header records with the level, found by following the keys. The objects of the +level below that the job's key range meets, the objects with too many fragments and the one whose +marks are most due are found by walks that skip every subtree whose summary or tally rules it out. A +cold plan over 125 000 objects asks the store for two pages, where reading every leaf would ask for +124. + +A tiered job concatenates a run of one level's objects with nothing else between them. Under a round +robin, which is a tiered dataset's default, it takes the run that starts past the level's pointer and +ends before the first object another level holds past that start, one descent per level: a plan over +100 000 objects by arrival asks for one page, where reading every leaf would ask for 71. The longest +run (`CompactionPick.Largest`) depends on every level's order in full, so a plan that takes it still +reads every leaf, as does a plan over a level whose pages were written before tallies existed, until a +commit rewrites them. Both choose the job the other would. When many processes append, one +coordinator batching their rows into one commit solves the contention. Without it, the [commit +protocol](#8-the-commit-protocol) stays correct, only slower. ### 5.4 The price -Leveled compaction with `F` = 10 rewrites each row about `F/2` times per level it crosses: about -**20 to 25×** the appended bytes over four levels. A small commit crosses every level, a load only -those above the one it lands in. Tiered rewrites each row about once per level, about **L×**. The -default is leveled when a clustering key is declared and tiered otherwise. A dataset that appends -1 TiB a day, leveled, writes about 25 TiB a day in compaction: the honest cost of the -one-object-per-level read bound, which every LSM store pays. +Leveled compaction with `F` = 10 rewrites each row about `F/2` times per level it crosses, about 20 to +25× the appended bytes over four levels. A small commit crosses every level, and a bulk load only +those above the one it lands in. Tiered compaction rewrites each row about once per level, about L×. +The default is leveled when a clustering key is declared and tiered otherwise. A dataset that appends +1 TiB a day, leveled, writes about 25 TiB a day in compaction. That is the honest cost of the +one-object-per-level read bound, and every LSM store pays it. -Who pays it, and when, is the caller's: compaction runs when `CompactAsync` is called, in a loop -any host runs, `RunCompactionAsync`, which plans, runs the job due for it at a rate it is given, and -sleeps when nothing is due, or, for level 0 under a byte budget, in the writer's own commit -(`DatasetOptions.InlineCompactionBytes`). [15-compaction.md](15-compaction.md) weighs the drivers — inline in a -commit, on demand, in the background — planning in the depth of the tree, and what a store that -cannot delete does to the price. +Who pays it, and when, is the caller's decision. Compaction runs when `CompactAsync` is called, in a +loop any host runs (`RunCompactionAsync`, which plans, runs the job that is due at the rate it is +given, and sleeps when nothing is due), or, for level 0 under a byte budget, in the writer's own +commit (`DatasetOptions.InlineCompactionBytes`). [15-compaction.md](15-compaction.md) weighs those +drivers (inline in a commit, on demand, in the background), planning from the top of the tree, and +what a store that cannot delete does to the price. ## 6. Indexes at scale ### 6.1 One run per entry per object An index entry has one run per object, merged when the object completes, and an append in place keeps -it at K = 4 runs by merging past that ([10-indexes.md](10-indexes.md) §3). A lookup on a key -uncorrelated with row order therefore reads at most K fence pages per object, never one per chunk. +it at no more than four runs by merging past that (see [the directory and the +runs](10-indexes.md#3-the-directory-and-the-runs)). A lookup on a key uncorrelated with row order +therefore reads at most four fence pages per object, never one per chunk. ### 6.2 The filter tree and the fences -Bloom filters form a tree of fan-out 16 and long runs page their segment tables into fences -([10-indexes.md](10-indexes.md) §3): a point probe reads `1 + 16 · log₁₆ blocks` filters at most, a -lookup two fence pages before its payload, whatever the object's size. +Bloom filters form a tree with a fan-out of 16, and long runs page their segment tables into fences +(see [the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)). A point probe reads at +most `1 + 16 · log₁₆ blocks` filters, and a lookup reads two fence pages before its payload, whatever +the object's size. ### 6.3 Checksums -Every page, fragment and index region carries its checksum (§7), and a region that fails claims -nothing: a reader answers exactly without it, and `VerifyAsync` names it. +Every page, fragment and index region carries its checksum ([identity](#7-identity-and-integrity)), +and a region that fails claims nothing. A reader still answers exactly without it, and `VerifyAsync` +names it. ### 6.4 Fragments, in commit objects, transient -A **fragment** is one entry's runs over a block range of one object, in the container the core -builds for a file it cannot append to ([10-indexes.md](10-indexes.md) §7), with its own encoding table +A fragment is one entry's runs over a block range of one object, in the container the core builds for +a file it cannot append to (see [indexing after the +fact](10-indexes.md#7-building-appending-and-indexing-after-the-fact)), with its own encoding table and dtypes so that it decodes without the object's footer. An indexer works by object and block -range, and each batch is one commit whose **commit object carries the fragment's bytes**; the leaf -entry references it by `(version, offset, length, XXH3-64)`, one ranged read. Coverage may be partial: -a pruner uses the covered blocks, and a key source waits for complete coverage, which the entry -states, so an object can be indexed progressively without ever answering wrong. - -A fragment is its content: two indexers of one range write one fragment, and a drop removes the -fragment of those bytes wherever it lies. Past K fragments an entry's fragments are **bundled** — the -containers copied as they are, one after another, behind a table — rather than merged, since a Bloom -tree's upper filters cannot be rebuilt from its leaves and a run's payloads hold offsets into their -own container. A data compaction that rewrites the object embeds the index and drops every fragment, -so at steady state the external layer holds only the indexes of objects nobody has rewritten. +range, and each batch is one commit whose commit object carries the fragment's bytes. The leaf entry +references it by `(version, offset, length, XXH3-64)`, one ranged read. Coverage may be partial: a +pruner uses the covered blocks, and a key source waits for complete coverage, which the entry states, +so an object can be indexed progressively without ever answering wrong. + +A fragment is identified by its content. Two indexers of the same range write one fragment, and +dropping a fragment removes the fragment with those bytes wherever it lies. Past four fragments, an +entry's fragments are bundled (the containers copied as they are, one after another, behind a table) +rather than merged, since a Bloom tree's upper filters cannot be rebuilt from its leaves and a run's +payloads hold offsets into their own container. A data compaction that rewrites the object embeds the +index and drops every fragment, so at steady state the external layer only holds the indexes of +objects nobody has rewritten yet. ### 6.5 Which structure for which column @@ -323,198 +344,208 @@ so at steady state the external layer holds only the indexes of objects nobody h | few, or dictionary-encoded | zone map and dictionary probe | one values child per chunk the zone map keeps | | up to about 10⁵ per 16 blocks | Bloom filter tree | `1 + 16 · log₁₆ blocks` filters, output-sensitive | | many, or a range or an order | one sorted run per object, with fences | two fence pages and two segments per object touched | -| the dataset's lookup key | the **clustering key** | one object per level, then its run: the only structure that bounds the objects | +| the dataset's lookup key | the clustering key | one object per level, then its run: the only structure that bounds the objects | -`Auto` gives up a Bloom filter on a column whose first node passes a filter's capacity, where a sorted -run is the structure ([10-indexes.md](10-indexes.md) §6). +`Auto` gives up a Bloom filter on a column whose first node exceeds a filter's capacity, where a +sorted run is the right structure (see [the policy](10-indexes.md#6-the-policy)). ### 6.6 Order beyond the lookup -The trees are ordered by the clustering key, so the order of **that one key** is bounded everywhere: +The trees are ordered by the clustering key, so the order of that one key is bounded everywhere: | need | mechanism | cost | |---|---|---| -| walk the objects by key | an in-order walk of a level's tree, the next pages prefetched eight ahead | O(log N + objects touched), about one dependent round trip per window of eight pages | -| rows in key order, `OrderBy` | a merge of each object's key-ordered read; inside a level above 0 the objects are key-disjoint | at most 8 + L objects open at once | +| walk the objects by key | an in-order walk of a level's tree, with the next pages prefetched eight ahead | O(log N + objects touched), about one dependent round trip per window of eight pages | +| rows in key order, `OrderBy` | a merge of each object's key-ordered read. Inside a level above 0 the objects are key-disjoint | at most 8 + L objects open at once | | rows by position, `Rows(a, b)` | the row sums of every node | O(log N) plus the objects the range touches | -| count and rank on the clustering key | objects whose summaries lie wholly inside the range counted unopened; the core's exact cover in the boundary objects | at most level 0's objects plus two per level opened | +| count and rank on the clustering key | objects whose summaries lie wholly inside the range are counted unopened, and the core's exact cover handles the boundary objects | at most level 0's objects plus two per level opened | | an order on another column | no tree helps | one cursor per object the summaries cannot refute | -**An object opens when it could hold the next row, not before.** Objects are offered in the order of a -lower bound on their keys — on the clustering key, upward, a leaf key is the object's exact minimum — -and one is opened only once that bound is at or below the smallest key an open input holds. So a -consumer that stops after `k` rows opens no object whose minimum lies past the k-th key: `LIMIT 10` -opens one object of four. Measured on the same rows: four cursors over four interleaved level-0 -objects, one over the key-disjoint level a compaction made of them. **A float has no such bound**, -since a zone's minimum and maximum exclude NaN while the key order places NaNs at the ends; float -objects open before the first row, and an object whose first row lies below the bound it was ordered -by raises `VortexFormatException` rather than deliver a wrong order. Ties between objects follow the -dataset's order, so a descending read is the exact reverse of an ascending one. - -**A key cursor walks both ways.** Walking down is the same merge with a max-heap. The objects of a -level above 0 are key-disjoint, so the minimum of the next object of the level is a strict upper -bound on an object's keys, and the object opens only once the walk passes below it; level 0's -objects, which nothing but their summaries bounds from above, open at the seek, at most eight and the -lag. A step against the walk's direction seeks every object again just past the current entry, whose -place in the order is its key and then its row in the dataset: an object before it in the dataset's -order lands at or before the key, one after it strictly before, and its own object steps. That is the -flip every merging iterator pays ([12-index-reads.md](12-index-reads.md) §3.2), and the walk down is -the walk up reversed, ties included. - -**A key cursor reads the trees only where it walks.** In each level the walk stands between two -objects, a place found by one path down the level's tree and moved an object at a time, a page read -only when the walk crosses into it; the pages read, and the objects' entries in them, are parsed once -per version and shared by its cursors. An object's first row is counted when the walk first takes one -of its entries: the rows of every level's objects before its tree key, along one path of each. A seek -on a handle that holds nothing yet so reads the depth of each level, whatever the number of objects: -over 20 000 objects in one level, one page, where a listing of the objects read every leaf, 23. - -Any other order costs, at best, one cursor per object the summaries cannot refute: `ORDER BY x LIMIT -k` prunes by the summaries, which is good in practice and output-sensitive, never bounded. A second -bounded order is a second copy of the data clustered by that key, a dataset of its own; a global -secondary index is excluded, since every compaction would rewrite it. +An object opens when it could hold the next row, and not before. Objects are offered in the order of +a lower bound on their keys (on the clustering key, ascending, a leaf key is the object's exact +minimum), and one is only opened once that bound is at or below the smallest key an open input holds. +So a consumer that stops after `k` rows opens no object whose minimum lies past the k-th key, and +`LIMIT 10` opens one object of four. Measured on the same rows: four cursors over four interleaved +level 0 objects, and one over the key-disjoint level a compaction made of them. A float has no such +bound, since a zone's minimum and maximum exclude NaN while the key order places NaNs at the ends, so +float objects open before the first row, and an object whose first row lies below the bound it was +ordered by raises `VortexFormatException` rather than deliver a wrong order. Ties between objects +follow the dataset's order, so a descending read is the exact reverse of an ascending one. + +A key cursor walks both ways. Walking down is the same merge with a max-heap. The objects of a level +above 0 are key-disjoint, so the minimum of the next object in the level is a strict upper bound on an +object's keys, and the object only opens once the walk passes below it. Level 0's objects, which +nothing but their summaries bounds from above, open at the seek, at most eight plus the lag. A step +against the walk's direction seeks every object again just past the current entry, whose place in the +order is its key and then its row in the dataset: an object before it in the dataset's order lands at +or before the key, one after it strictly before, and its own object steps. That is the change of +direction every merging iterator pays for (see [stepping and +direction](12-index-reads.md#32-stepping-and-direction)), and the walk down is the walk up reversed, +ties included. + +A key cursor only reads the trees where it walks. In each level the walk stands between two objects, +a place found by one path down the level's tree and moved one object at a time, with a page read only +when the walk crosses into it. The pages read, and the objects' entries in them, are parsed once per +version and shared by its cursors. An object's first row is counted when the walk first takes one of +its entries, from the rows of every level's objects before its tree key, along one path of each. A +seek on a handle that holds nothing yet therefore reads the depth of each level, whatever the number +of objects: over 20 000 objects in one level it reads one page, where a listing of the objects would +read every leaf, 23 pages. + +Any other order costs, at best, one cursor per object the summaries cannot refute. `ORDER BY x LIMIT +k` prunes by the summaries, which is good in practice and output-sensitive, but never bounded. A +second bounded order requires a second copy of the data clustered by that key, a dataset of its own. +A global secondary index is excluded, since every compaction would rewrite it. ## 7. Identity and integrity -- **Every postscript this library writes mints a uid**: sixteen random bytes in the metadata entry - `vorticity.identity`, written right after the footer, so the open's tail read covers it and - reading it costs no request. It identifies a **version of the bytes**, not a file: an append in - place mints a new one, and a fragment bound to the old uid is refused. It costs 72 to 96 bytes a - file; `VortexWriteOptions.Identity` pins it for a reproducible write. -- **The store's token** — a version id or an entity tag, or on a file system the length and the - modification time — is recorded when the store gives one and compared as an opaque string. It is the - only binding for a foreign file without a uid; on a file system it is a heuristic, and the entry - says so. `VortexOpenOptions.IndexFragmentsNeedIdentity` refuses a fragment bound so. -- **The content hash is the writer's.** The writer sees every byte it emits, so an XXH3-128 of the - object costs nothing then and is recorded in the leaf entry. **No reader computes it**: `verify` - does, offline. -- **Pages are addressed by their XXH3-128**, and a commit object carries an XXH3-64 over its header - and table: integrity, deduplication and idempotent retries in one mechanism. A torn page or header - makes its version unreadable, with the reason; a corrupt fragment or index region claims nothing; a - corrupt data segment is a `VortexFormatException`. XXH3 guards against accident, not forgery. +- Every postscript this library writes mints a uid: sixteen random bytes in the metadata entry + `vorticity.identity`, written right after the footer, so the open's tail read covers it and reading + it costs no request. It identifies a version of the bytes, not a file: an append in place mints a + new one, and a fragment bound to the old uid is refused. It costs 72 to 96 bytes per file, and + `VortexWriteOptions.Identity` pins it for a reproducible write. +- The store's token (a version id or an entity tag, or, on a file system, the length and the + modification time) is recorded when the store provides one and compared as an opaque string. It is + the only binding for a foreign file without a uid. On a file system it is a heuristic, and the entry + says so. `VortexOpenOptions.IndexFragmentsNeedIdentity` refuses a fragment bound that way. +- The content hash is the writer's. The writer sees every byte it emits, so an XXH3-128 of the object + costs nothing at that point and is recorded in the leaf entry. No reader computes it, and only + verification does, offline. +- Pages are addressed by their XXH3-128, and a commit object carries an XXH3-64 over its header and + table, which gives integrity, deduplication and idempotent retries in one mechanism. A torn page or + header makes its version unreadable, with the reason. A corrupt fragment or index region claims + nothing. A corrupt data segment raises `VortexFormatException`. XXH3 guards against accidents, not + forgery. ## 8. The commit protocol ### 8.1 A commit is one conditional creation -A writer reads the latest commit `N`, uploads its data objects under fresh uid keys, in parallel, -then creates `commit/` — header, pages and fragments in one object — with the store's -**put-if-absent**, streamed from a pipe whose length is given up front, so the store chooses between -one request and a multipart upload. Put-if-absent linearizes commits: no lease, no lock, no external -service. Uncontended, a commit is **three dependent requests**: the listing, the read of `N`'s header, -the creation; the data objects cost the commit nothing more. A handle that knows its version was -the latest less than half a retention window ago (§8.3) builds on it as it holds it, so its commit is -**one request**, the creation: a refusal is what tells it another writer went first, and it then asks -for the versions after the number it lost and rebases. Past that half window it lists first. Either -way it reads the version it created as it wrote it, with nothing read back. A handle whose commit -found nothing to change reads the version that commit was decided on, which is asked for first when -the batch was decided on the version held, since a no-op names the latest version. +A writer reads the latest commit `N`, uploads its data objects under fresh uid keys, in parallel, then +creates `commit/` (header, pages and fragments in one object) with the store's +put-if-absent, streamed from a pipe whose length is given up front, so the store can choose between +one request and a multipart upload. Put-if-absent linearizes commits with no lease, no lock and no +external service. + +Uncontended, a commit is three dependent requests: the listing, the read of `N`'s header, and the +creation. The data objects add nothing to the commit itself. A handle that knows its version was the +latest less than half a retention window ago (see [the latest +version](#83-the-latest-version-in-one-request)) builds on that version directly, so its commit is a +single request, the creation. A refusal tells it another writer went first, and it then asks for the +versions after the number it lost and rebases. Past that half window it lists first. Either way, it +reads the version it created as it wrote it, with nothing read back. A handle whose commit found +nothing to change reads the version that decision was made on, which is fetched first when the batch +was decided on the version it held, since a no-op names the latest version. ### 8.2 Rebase by re-applying operations -If `N+1` exists, the writer reads it and **re-applies its logical operations** to that version's trees -— add objects, add fragments, drop fragments, replace objects by a compaction's outputs — and never -merges two trees. Each operation carries its own answer to a moved ground: +If `N+1` already exists, the writer reads it and applies its logical operations again on that +version's trees (add objects, add fragments, drop fragments, replace objects by a compaction's +outputs), and never merges two trees. Each operation carries its own answer to a moved ground: | concurrent operations | result | |---|---| | append, append | both objects land in level 0 | -| append, compaction of level 0 | the compaction took a snapshot; the new object is not among its inputs and stays | +| append, compaction of level 0 | the compaction took a snapshot, so the new object is not among its inputs and stays | | indexer, indexer on the same range | the second finds the same fragment already there and writes nothing | -| indexer, compaction of that object | the object is gone; the fragment is dropped and never written | -| compaction, compaction on overlapping inputs | the second finds an input missing and abandons; its outputs are garbage | -| vacuum, anything | vacuum deletes only what no retained version references and what is older than the window (§10) | -| delete or update, append | the appended object is not among those the change read: it stays, and its rows are not touched | -| delete or update, anything that rewrites or marks an object it read | an input is gone or its entry changed: the replacement is abandoned, and the change is worked out again on the version that won (§12) | -| compaction, delete or update that marks one of its inputs | the entry the compaction read changed: it abandons, since its outputs would bring the marked rows back, and they are garbage | -| schema change, schema change | the second finds another schema than the one it was checked against, and is dropped (§13) | -| append, schema change | the object lands as it was written, and reads as the new schema like any earlier one (§13) | -| reader, anything | the reader holds a version; everything it references is immutable | - -An object's key ends with its uid, which makes it unique and a function of the object: two appenders -that read the same version would otherwise compute one key, and the second would replace the first's -entry, which an interleaving fuzzer caught. Because a commit header carries the pages it wrote and the -pages above them it read, a rebase iteration is also three dependent requests while the tree fits the -header. A writer that loses too many times (`DatasetOptions.MaxAttempts`) is told to use a -coordinator, in the exception's own words. +| indexer, compaction of that object | the object is gone, so the fragment is dropped and never written | +| compaction, compaction on overlapping inputs | the second finds an input missing and abandons, and its outputs are garbage | +| vacuum, anything | vacuum only deletes what no retained version references and what is older than the window ([lifecycle](#10-lifecycle)) | +| delete or update, append | the appended object is not among those the change read, so it stays and its rows are not touched | +| delete or update, anything that rewrites or marks an object it read | an input is gone or its entry changed, so the replacement is abandoned and the change is worked out again on the version that won ([deleting rows](#12-deleting-and-updating-rows)) | +| compaction, delete or update that marks one of its inputs | the entry the compaction read changed, so it abandons, since its outputs would bring the marked rows back, and they are garbage | +| schema change, schema change | the second finds a different schema from the one it was checked against, and is dropped ([schema evolution](#13-schema-evolution)) | +| append, schema change | the object lands as it was written, and reads as the new schema like any earlier one ([schema evolution](#13-schema-evolution)) | +| reader, anything | the reader holds a version, and everything it references is immutable | + +An object's key ends with its uid, which makes it unique and a function of the object. Without it, +two appenders that read the same version would compute the same key, and the second would replace the +first's entry, a case the interleaving fuzzer found. Because a commit header carries the pages it +wrote and the pages above them it read, one rebase iteration is also three dependent requests while +the tree fits in the header. A writer that loses too many times (`DatasetOptions.MaxAttempts`) is told +to use a coordinator, in the exception's own words. ### 8.3 The latest version in one request Commit keys sort newest first, so the first key a listing of `commit/` yields is the whole discovery, -and the listing stops after one page. That needs a strongly consistent listing, which S3 has had since -2020 and which §11 requires of any store. A reader that wants to read its own writes is handed the -version by its writer. There is no hint object. +and the listing stops after one page. That needs a strongly consistent listing, which S3 has offered +since 2020 and which [the store abstraction](#11-the-store-abstraction) requires of any store. A +reader that wants to read its own writes is handed the version by its writer. There is no hint +object. A handle that knows its version was the latest less than half a retention window ago asks for the -versions after it instead, a head request each: every one of them was created after that moment, and -vacuum takes a commit only a window after the next one superseded it, so they are all still there -and the first missing one ends the run. Past that half window the handle lists. Below the window the -commits do not run without a gap: vacuum keeps an old commit object whose pages a retained version -still names and takes the unnamed ones after it, so a handle that took the gap above its own version -for the end would read it as the latest, and commit on it. The handle's clock must therefore agree -with the store's to within the other half window, as a vacuum's already must. +versions after it instead, with a head request each. Every one of them was created after that moment, +and vacuum only takes a commit a full window after the next one superseded it, so they are all still +there and the first missing one ends the run. Past that half window, the handle lists. Below the +window the commits are not contiguous: vacuum keeps an old commit object whose pages a retained +version still names and removes the unnamed ones after it, so a handle that took the gap above its +own version for the end would read it as the latest and commit on it. The handle's clock therefore +has to agree with the store's to within the other half window, as a vacuum's already must. ## 9. The read-path budget ### 9.1 Dependent round trips, not bytes On an object store a request costs 10 to 100 ms whatever its size below a megabyte, so the budget is -**dependent round trips**. Cold, a point lookup on the clustering key through a sorted run: +counted in dependent round trips. Cold, a point lookup on the clustering key through a sorted run +costs: | step | dependent requests | cached by | |---|---|---| | list the latest commit, read its header | 2 | version | -| descend a level's tree | 0 up to ~650 objects, 1 up to ~400 000, 2 up to ~280 million | page, across versions: pages are immutable | +| descend a level's tree | 0 up to ~650 objects, 1 up to ~400 000, 2 up to ~280 million | page, across versions, since pages are immutable | | open the object's tail | 1, in parallel across the ≤ 8 + L objects touched | object | | its fences, then the two payload segments | 1 + 1 | run | -| the data segments | 1, coalesced | — | +| the data segments | 1, coalesced | none | -About **6 to 8 cold and 2 to 3 warm**, for any number of rows and objects. Latency does not choose the -structure; it sets the page cap, the inlining, the one-object commit, the in-process cache of -immutable headers, pages, tails and fences, and the parallel requests. Hedged requests belong to the -store library. +That is about 6 to 8 requests cold and 2 to 3 warm, for any number of rows and objects. Latency does +not choose the structure. It sets the page cap, the inlining, the one-object commit, the in-process +cache of immutable headers, pages, tails and fences, and the parallel requests. Hedged requests belong +to the store library. A scan in the tree's order opens the next object ahead once its consumer reads past the first batch -([16-queries.md](16-queries.md) §2.6): an object's tail and its first data request wait on each -other, not on the object before it, and a scan that stops at its first batch opens one object. +(see [the first batch of each source](16-queries.md#26-the-first-batch-of-each-source)), so an +object's tail and its first data request wait on each other rather than on the object before it, and +a scan that stops at its first batch opens a single object. ### 9.2 Held by a counting store -- **One file, any size**: opening a file and answering a count through a sorted run in fence pages, a - scan through a filter tree, a key seek and the roots costs the same **13 requests and 415 811 bytes** - on a sparse 1.31 GiB file and a sparse 13.1 GiB one (`ReadBudgetTests`). A content hash on the read +- For one file of any size, opening it and answering a count through a sorted run in fence pages, a + scan through a filter tree, a key seek and the roots costs the same 13 requests and 415 811 bytes on + a sparse 1.31 GiB file and on a sparse 13.1 GiB one (`ReadBudgetTests`). A content hash on the read path would turn that into 1 157 against 11 525 requests, and the test would fail. -- **Any number of objects**: a cold clustering-key lookup costs **2, 2 and 3** requests over datasets - of 1, 1 000 and 100 000 objects, and one object added costs **3, 3 and 4** dependent steps +- For any number of objects, a cold clustering-key lookup costs 2, 2 and 3 requests over datasets of + 1, 1 000 and 100 000 objects, and adding one object costs 3, 3 and 4 dependent steps (`DatasetBudgetTests`). -- **The critical path**: with 20 ms of injected latency, the lookup waits for its round trips one after +- On the critical path, with 20 ms of injected latency, the lookup waits for its round trips one after another, which no count of requests can show. ## 10. Lifecycle -- **Vacuum** (`VacuumAsync`, explicit, never automatic) marks everything the retained versions - reference — a subtree already seen is not walked again, so the versions cost their distinct pages — - then deletes the unmarked commit objects and data objects older than the **retention window**, - seven days by default (`DatasetOptions.RetentionWindow`), keeping the latest version always. What - ages is the moment a version was superseded, by the store's own clock. An unreferenced object - younger than the window is a writer in flight and is kept: **a writer must commit within the - window**. Only `commit/`, `data/` and the compaction loops' `leases/` are swept; an imported file - is its owner's. On a store under a retention lock, what the lock or a legal hold still keeps is left - and reported with the date the first of it may go (`DatasetOptions.LockedStore`, - [15-compaction.md](15-compaction.md) §5). A reader that - outlives the window gets an `ObjectNotFoundException` naming the version it was reading, never a - partial answer. A dry run reports without deleting. -- **Repack**: a commit object kept alive by a few live pages among dead ones is reported as sparse, and - a metadata-only commit copies those pages and fragments into itself, so the old object can go at the - next vacuum. -- **Verify** (`VerifyAsync`, offline) checks every object's length, content hash, rows and identity; - every fragment against its reference and with its object; every commit object whole; and it reads - the pages from the store, not from a header's inlined copies, which is the only way to see a torn - stored page a reader never reads. It reports every problem by name and throws none. Given an older - version, it walks the two trees' difference only, O(changed pages). -- **Rebuilding an index** drops an object's fragments and attaches one over the whole object in the - same commit, so no version holds the object with neither; rewriting the object, which a compaction - does, embeds it instead. +- Vacuum (`VacuumAsync`, explicit, never automatic) marks everything the retained versions reference + (a subtree already seen is not walked again, so the versions cost their distinct pages), then + deletes the unmarked commit objects and data objects older than the retention window, seven days by + default (`DatasetOptions.RetentionWindow`), always keeping the latest version. What ages is the + moment a version was superseded, by the store's own clock. An unreferenced object younger than the + window may belong to a writer in flight and is kept, which means a writer must commit within the + window. Only `commit/`, `data/` and the compaction loops' `leases/` are swept, and an imported file + belongs to its owner. On a store with retention locks, whatever the lock or a legal hold still keeps + is left and reported with the date the first of it may go (`DatasetOptions.LockedStore`, see + [stores that cannot delete](15-compaction.md#5-stores-that-cannot-delete)). A reader that outlives + the window gets an `ObjectNotFoundException` naming the version it was reading, never a partial + answer. A dry run reports without deleting. +- Repacking is internal for now. A commit object kept alive by a few live pages among dead ones is + detected as sparse during vacuum, and a metadata-only commit can copy those pages and fragments + into itself so the old object can go at the next vacuum, but neither the detection nor the repack + is exposed in the public API yet. +- Verification (`VerifyAsync`, offline) checks every object's length, content hash, rows and + identity, every fragment against its reference and its object, and every commit object whole. It + reads the pages from the store, not from a header's inlined copies, which is the only way to see a + torn stored page that a reader never reads. It reports every problem by name and throws none. Given + an older version, it only walks the difference between the two trees, O(changed pages). +- Rebuilding an index drops an object's fragments and attaches one over the whole object in the same + commit, so no version holds the object with neither. Rewriting the object, as a compaction does, + embeds the index instead. ## 11. The store abstraction @@ -522,195 +553,214 @@ other, not on the object before it, and a scan that stops at its first batch ope | operation | used by | |---|---| -| `GetRangeAsync(key, offset, length)` → bytes and the object's token, in one answer | every read; a data object's segment source plans its reads with the core's coalescer and issues them together | -| `HeadAsync(key)` → size, token, the store's creation time, and the retention date and legal hold of a store under a lock | opening; vacuum's ages, and what a lock still keeps | -| `PutIfAbsentAsync(key, pipe, length)` → created or exists | every write, streamed | -| `DeleteAsync(keys)` | vacuum, a batch at a time; an absent key is not an error | -| `ListAsync(prefix, startAfter)` | the latest version, one page (§8.3); vacuum | +| `GetRangeAsync(key, offset, length)` returns the bytes and the object's token in one answer | every read. A data object's segment source plans its reads with the core's coalescer and issues them together | +| `HeadAsync(key)` returns the size, the token, the store's creation time, and the retention date and legal hold on a locked store | opening, vacuum's ages, and what a lock still keeps | +| `PutIfAbsentAsync(key, pipe, length)` returns created or exists | every write, streamed | +| `DeleteAsync(keys)` | vacuum, a batch at a time, where an absent key is not an error | +| `ListAsync(prefix, startAfter)` | the latest version, in one page ([the latest version](#83-the-latest-version-in-one-request)), and vacuum | A store must document an atomic put-if-absent, a strongly consistent listing, a token that changes -whenever the bytes under a key change, and ranged reads on objects of any size. Three shapes belong to -the seam: a put **streams** and creates nothing when its content ends early or runs long; a delete -takes a batch and reports nothing; a listing pages by itself as it is enumerated. Retries, hedging, -connection pools and credentials are the store library's and never this one's. +whenever the bytes under a key change, and ranged reads on objects of any size. Three behaviours are +part of the interface: a put streams and creates nothing when its content ends early or runs long, a +delete takes a batch and reports nothing, and a listing pages by itself as it is enumerated. Retries, +hedging, connection pools and credentials belong to the store library, never to this one. -Three stores ship: `FileObjectStore`, whose put-if-absent is the kernel's create-new, and whose -content a crash may leave short, which the format detects; `MemoryObjectStore`, with injectable -latency, failures and crashes after a put; and `CountingObjectStore`, a decorator counting requests, -bytes and the **dependent steps** of the critical path. The contract is a test suite, -`ObjectStoreContractTests`, which an S3 library runs against its own store. +Three stores ship. `FileObjectStore` uses the kernel's create-new for put-if-absent, and a crash may +leave its content short, which the format detects. `MemoryObjectStore` has injectable latency, +failures and crashes after a put. `CountingObjectStore` is a decorator that counts requests, bytes and +the dependent steps of the critical path. The contract is a test suite, `ObjectStoreContractTests`, +which an S3 library runs against its own store. ## 12. Deleting and updating rows -**A delete marks rows, and rewrites no object it can mark.** A delete takes the rows its filter is -true for — a row it is false or unknown for stays, as a scan would not return it — and records them in -the leaf entry of each object that holds one: a **deletion vector**, the sorted runs of the object's -own row positions it takes, two varints a run, after everything else the entry holds (§4.2). The -object's bytes do not change, so the commit writes the pages on the path to the entries it changed and -no data object. An object every row of which goes is removed from the tree without being read, and one -whose summaries refute the filter is not opened. An update is a delete and an insert applied together: -the rows it takes are marked where they were, read as records, passed through the caller's function -and written into new objects of level 0, where appended rows go, because a changed row may carry -another key. One `ReplaceObjects` commits the whole change, so a reader sees all of it or none. - -**When a delete rewrites instead.** An object is rewritten without the rows, rather than marked, when -the marks would cost its reads more than they save the write: under `MarkedObjectBytes`, a mebibyte, -where a rewrite is a few requests and leaves the object exact; when the rows marked in it would pass an -eighth of its rows (`MarkedShare`), since every read of it steps over them; and when its vector would -pass 1 KiB (`MarkedVectorBytes`), a few hundred runs. The vector is counted as the runs it will form: -at a run a row when that already fits, and otherwise from the rows' places, read before anything is -written, since the rows a range takes are one run whatever their number. The vector lives in a leaf page: a commit that -marks a row in any object of a page writes the page again, so the bound weighs the rewrites the marks -save against the bytes they add to each of those commits; a commit that does not change the page -leaves it out of its header (§3). A vector past 256 bytes lies out of line instead: written once into -the commit object that made it, as a fragment is (§6.4), and named by its entry by a reference, so -the page carries 37 bytes for it whatever the marks. A rent of the object reads it beside the -object's open, where the version's pages lie or the handle kept it; vacuum keeps the commit object it -lies in, and a repack moves it. The rows a rewritten object keeps are a subset of -its rows, so its keys lie inside its old range: a level above 0 stays key-disjoint and the tree's order -holds; without a clustering key the rewritten object keeps the old one's position. Either way the rows -are judged by the scan's own evaluator, so a delete and a scan with the same filter agree on every row, -nulls included, and a rewrite checks that it kept exactly the rows the count left. - -**What a read does with a vector.** An entry's row count is its live rows, so a count without a filter -and every position stay arithmetic: a position maps to the object's own through the runs before it. -The rest is the object's own read with its marked rows left out, for a binary search over the runs per -batch or per step: +A delete marks rows, and does not rewrite any object it can mark. It takes the rows its filter is true +for (a row it is false or unknown for stays, as a scan would not return it) and records them in the +leaf entry of each object that holds one, as a deletion vector: the sorted runs of the object's own +row positions it takes, two varints per run, placed after everything else the entry holds (see [what a +node carries](#42-what-a-node-carries)). The object's bytes do not change, so the commit writes the +pages on the path to the entries it changed and no data object. An object whose rows all go is removed +from the tree without being read, and one whose summaries refute the filter is not opened. + +An update is a delete and an insert applied together. The rows it takes are marked where they were, +read as records, passed through the caller's function, and written into new level 0 objects, where +appended rows go, because a changed row may carry a different key. One `ReplaceObjects` commits the +whole change, so a reader sees all of it or none of it. + +### When a delete rewrites instead + +An object is rewritten without the rows, rather than marked, when the marks would cost its reads more +than they save the write. That happens under `MarkedObjectBytes` (a mebibyte), where a rewrite is a +few requests and leaves the object exact, when the rows marked in it would pass an eighth of its rows +(`MarkedShare`), since every read of it steps over them, and when its vector would pass 1 KiB +(`MarkedVectorBytes`), a few hundred runs. The vector is counted as the runs it will form: one run per +row when that already fits, and otherwise from the rows' positions, read before anything is written, +since the rows a range takes form one run whatever their number. + +The vector lives in a leaf page, so a commit that marks a row in any object of a page writes that page +again, and the bounds weigh the rewrites the marks save against the bytes they add to each of those +commits. A commit that does not change the page leaves it out of its header (see [objects and +keys](#3-objects-and-keys)). A vector past 256 bytes lies out of line instead: it is written once into +the commit object that made it, like a fragment ([fragments](#64-fragments-in-commit-objects-transient)), +and its entry names it by reference, so the page carries 37 bytes for it whatever the marks. A read of +the object fetches it alongside the object's open, from where the version's pages lie or from what the +handle kept. Vacuum keeps the commit object it lies in. + +The rows a rewritten object keeps are a subset of its rows, so its keys stay inside its old range: a +level above 0 stays key-disjoint and the tree's order holds, and without a clustering key the +rewritten object keeps the old one's position. Either way, rows are judged by the scan's own +evaluator, so a delete and a scan with the same filter agree on every row, nulls included, and a +rewrite checks that it kept exactly the rows the count left. + +### What a read does with a vector + +An entry's row count is its live rows, so a count without a filter and every position stay +arithmetic: a position maps to the object's own through the runs before it. The rest is the object's +own read with its marked rows left out, at the cost of a binary search over the runs per batch or per +step: | read | with marked rows | |---|---| -| a filtered count, an any | the object's count with the rows left out, tier by tier, in one pass: on a sorted column's slices, arithmetic; on an index's rows, a test of each; on a zone the maps decide whole, its rows less the marked ones; on a decode, the marked rows cleared before the count | -| rows in file order | a batch without a marked row as it came, renumbered; one with some, gathered without them; an aggregation's batches whole, the marked rows deselected, their columns still encoded | -| rows in key order, a key cursor | the key source steps over the marked rows' entries as it walks, and a rank takes away the entries marked below it: arithmetic when the ranks are rows, a sorted column's or the clustering run of a level above 0, and otherwise against the marked rows' keys, read by one take when the cursor opens | -| a minimum, a maximum | on a column the object's statistics say is sorted and holds no null, its first or last live row: the statistic while that row is the column's end, one row read once a delete took it; otherwise the object's extreme when a count finds a live row holding it, and its live rows read when none does | -| the summaries | an entry with marks keeps its bounds, which its live rows still lie inside, and drops what they no longer prove: exactness, a null count other than zero, a column left with no bound. They prune; they no longer answer | - -**Compaction folds the marks.** It reads each input's live rows and writes objects without marks, so -a vector lasts until its object is next compacted. A level no merge reaches, the top one above all, is -purged instead. An object whose marks reach half a delete's bounds, a sixteenth of its rows or half a -kilobyte of vector, is rewritten alone, in its level and inside its range, by a job of its own -(`CompactionTrigger.Marks`), the most marked first; the plan finds it by descending the shares and -vectors the tallies carry (§5.3). A delete that would pass the bounds rewrites the object itself only -when compaction lags. A compaction that read an object before a delete marked -rows in it would bring those rows back: its `ReplaceObjects` names the entries it read — uid, key and -vector — and is abandoned when one of them changed (§8.2). - -**What it costs.** Measured by the churn (§15) on ten million rows and thirty thousand commits of ten -rows at random keys: an update takes 1.9 to 2.9 ms and a delete 0.9 to 1.6 ms, where rewriting the -4 MiB objects that hold the rows takes 19 and 14 ms. A commit writes 12 KiB at first and 16 to 55 KiB -once the vectors have filled, where a rewrite writes 3.3 MiB: the rows' own objects, and a commit -object of 8 to 10 KiB throughout, whose leaf page carries a reference for each vector past 256 bytes. -Compaction, merges and purges together, writes 61 to 111 KiB a commit. After a thousand commits the store holds 137 MiB rather than 449 MiB, since the retention -window keeps every object a rewrite replaced (§10). The reads cost what they cost on the rewritten -dataset: a filter over every object makes the same 220 requests, and a point lookup, a seek, a count -under a key range and the largest key stay where they were. What the marks keep is space: up to an -eighth of an object's rows, until its compaction. - -**Concurrent writers.** The change is worked out on the version the handle holds. An append that lands -first adds objects the change never read, whose rows it does not touch; a commit that removes, rewrites -or marks an object the change read — a compaction, another delete — abandons the replacement (§8.2), -and the change is worked out again on the version that won, up to `MaxAttempts` times. What a lost -attempt wrote is left for vacuum. +| a filtered count, an any | the object's count with the rows left out, tier by tier, in one pass: arithmetic on a sorted column's slices, a test of each row on an index's rows, the zone's rows minus the marked ones on a zone the maps decide whole, and the marked rows cleared before counting on a decode | +| rows in file order | a batch without a marked row is delivered as it came, renumbered, and one with some is gathered without them. An aggregation's batches stay whole, with the marked rows deselected and their columns still encoded | +| rows in key order, a key cursor | the key source steps over the marked rows' entries as it walks, and a rank subtracts the entries marked below it. That is arithmetic when the ranks are rows (a sorted column's, or the clustering run of a level above 0), and otherwise uses the marked rows' keys, read by one take when the cursor opens | +| a minimum, a maximum | on a column the object's statistics say is sorted and that holds no null, its first or last live row: the statistic while that row is the column's end, and one row read once a delete took it. Otherwise the object's extreme when a count finds a live row holding it, and its live rows read when none does | +| the summaries | an entry with marks keeps its bounds, which its live rows still lie inside, and drops what they no longer prove: exactness, a null count other than zero, a column left with no bound. They prune, but no longer answer | + +### Compaction folds the marks + +Compaction reads each input's live rows and writes objects without marks, so a vector only lasts until +its object is next compacted. A level no merge reaches, the top one above all, is purged instead. An +object whose marks reach half a delete's limits (a sixteenth of its rows or half a kilobyte of vector) +is rewritten alone, in its level and inside its range, by a job of its own +(`CompactionTrigger.Marks`), the most marked first, and the plan finds it by descending the shares and +vectors the tallies carry (see [what a compaction does](#53-what-a-compaction-does)). A delete that +would pass the limits only rewrites the object itself when compaction lags. A compaction that read an +object before a delete marked rows in it would bring those rows back, so its `ReplaceObjects` names +the entries it read (uid, key and vector) and is abandoned when one of them changed (see +[rebase](#82-rebase-by-re-applying-operations)). + +### What it costs + +The churn bench ([how it is tested](#15-how-it-is-tested)) measures it on ten million rows and thirty +thousand commits of ten rows at random keys. An update takes 1.9 to 2.9 ms and a delete 0.9 to 1.6 ms, +where rewriting the 4 MiB objects that hold the rows takes 19 and 14 ms. A commit writes 12 KiB at +first and 16 to 55 KiB once the vectors have filled, where a rewrite writes 3.3 MiB (the rows' own +objects), with a commit object of 8 to 10 KiB throughout, whose leaf page carries a reference for each +vector past 256 bytes. Compaction, merges and purges together, writes 61 to 111 KiB per commit. After +a thousand commits the store holds 137 MiB rather than 449 MiB, since the retention window keeps every +object a rewrite replaced. Reads cost what they cost on the rewritten dataset: a filter over every +object makes the same 220 requests, and a point lookup, a seek, a count under a key range and the +largest key stay where they were. What the marks keep is space, up to an eighth of an object's rows, +until its next compaction. + +### Concurrent writers + +The change is worked out on the version the handle holds. An append that lands first adds objects the +change never read, and their rows are not touched. A commit that removes, rewrites or marks an object +the change read (a compaction, another delete) makes the replacement abandon (see +[rebase](#82-rebase-by-re-applying-operations)), and the change is worked out again on the version +that won, up to `MaxAttempts` times. Whatever a lost attempt wrote is left for vacuum. ## 13. Schema evolution A change of schema is a commit that changes the header's schema and rewrites no object. Per top-level -column, a change is one of: added, nullable; dropped, unless the clustering key holds it; renamed; -widened within its kind, a signed integer to a wider signed one, an unsigned to a wider unsigned, a -float to a wider float; made nullable. Each keeps every value already written readable, and anything -else is refused before the commit. The clustering key's columns keep their names and types in every -object: they order the objects and the trees. - -**Names are identities.** An object's field belongs to the column of its name, or to the column a -rename moved that name to. The header records every name a column gave up, with the column carrying it -now, or none when it was dropped, and no column takes one of them again. That is what makes names -enough: an object written while a name was in use never lends its values to another column, and the -summaries its entry carries, keyed by the names it had, never describe another column. A summary under -a former name prunes nothing for the new one, which is conservative and ends at the next rewrite. - -**Reading an object of an earlier schema.** When an object's dtype is the version's — every object -written since the last change — its read is its own scan, after one comparison of dtypes -whose hashes are cached. Otherwise its columns are mapped when it opens, and each batch its own scan -delivers is reshaped in the batch's arena: a struct over its fields in the dataset's order, a missing -column filled with nulls, a column written non-nullable declared nullable over the same buffers, -narrower numbers widened into a buffer of the arena. The filter goes down to the object in its own -terms: renamed, and each predicate over a column the object lacks folded to what a null makes it, a -comparison unknown and `IS NULL` true, so that a filter over an added column refutes every earlier -object without a read. Under a negation an unknown is not false; a predicate that cannot be stated -over the object's columns there is evaluated on the reshaped batches instead. Key cursors, key-ordered -reads, compaction, deletes and updates read through the same mapping, and an object that lacks a -cursor's column holds no entry. - -**Compaction folds the change into the data**, as it folds indexes (§2): every object it writes is in -the current schema. Until then, reading an earlier object costs the reshaping — records per batch, and -a copy for a widened column — and reading any other costs nothing more. - -**Concurrent changes.** The operation carries the schema it was checked against. A rebase onto a -version whose schema is another drops it, since what it checked no longer holds, and the caller is told -to refresh; one that finds its own schema already there is done. An append prepared under the earlier -schema lands after the change, its object read as the new schema like any other. +column, a change is one of the following: a nullable column added, a column dropped (unless the +clustering key uses it), a column renamed, a number widened within its kind (a signed integer to a +wider signed one, an unsigned to a wider unsigned, a float to a wider float), or a column made +nullable. Each keeps every value already written readable, and anything else is refused before the +commit. The clustering key's columns keep their names and types in every object, since they order the +objects and the trees. + +Names are identities. An object's field belongs to the column with its name, or to the column a +rename moved that name to. The header records every name a column gave up, along with the column +that carries it now, or none when it was dropped, and no column ever takes one of those names again. +That is what makes names enough: an object written while a name was in use never lends its values to +another column, and the summaries its entry carries, keyed by the names it had, never describe +another column. A summary under a former name prunes nothing for the new one, which is conservative +and ends at the next rewrite. + +When an object's dtype is the version's, as for every object written since the last change, its read +is its own scan, after one comparison of dtypes whose hashes are cached. Otherwise its columns are +mapped when it opens, and each batch its own scan delivers is reshaped in the batch's arena: a struct +over its fields in the dataset's order, a missing column filled with nulls, a column written +non-nullable declared nullable over the same buffers, and narrower numbers widened into a buffer of +the arena. The filter reaches the object in its own terms. It is renamed, and each predicate over a +column the object lacks is folded to what a null makes it (a comparison unknown, `IS NULL` true), so a +filter over an added column refutes every earlier object without a read. Under a negation an unknown +is not false, so a predicate that cannot be stated over the object's columns there is evaluated on the +reshaped batches instead. Key cursors, key-ordered reads, compaction, deletes and updates read through +the same mapping, and an object that lacks a cursor's column holds no entry. + +Compaction folds the change into the data, as it folds indexes ([principles](#2-principles)): every +object it writes is in the current schema. Until then, reading an earlier object costs the reshaping +(records per batch, and a copy for a widened column), and reading any other object costs nothing +more. + +The operation carries the schema it was checked against. A rebase onto a version with a different +schema drops it, since what it checked no longer holds, and the caller is told to refresh, while one +that finds its own schema already there is done. An append prepared under the earlier schema lands +after the change, and its object reads as the new schema like any other. ## 14. Alternatives weighed | question | chosen | rejected, and why | |---|---|---| -| metadata structure | a prolly tree per level | a flat manifest per version (O(N) per open and per commit); checkpoint and deltas (the checkpoint is still the list); a manifest list of manifests, as Iceberg (fine to 10⁶ files, not a bound); an external catalog (a service dependency) | -| prolly or B+tree | prolly | same lookup and commit costs; the prolly adds history independence, deduplication across writers, a one-hash equality of datasets, an incremental diff across lineages, and a rebuild-from-scratch oracle, for a chunker of about a hundred lines | -| the latest version | newest-first keys and one listing | a hint object and probes (O(lag), a mutable object); a pointer updated by compare-and-swap (one more mutable thing) | -| concurrency | put-if-absent, rebase by operations | leases and lock files; three-way merges of trees | -| data across objects | leveled on a clustering key, tiered without | no compaction (O(appends)); a global secondary index (rewritten at every compaction) | -| index placement | in the file, external fragments transient | in-file only (adding an index rewrites data); external only (the index no longer travels with the file) | -| deleting rows | a deletion vector in the leaf entry, bounded, folded by compaction | copy on write of every object a row is taken from (an object per delete, 3 MiB a commit of ten rows, each kept through the retention window); positional delete files, as Iceberg (a read joins them, and they pile up until a rewrite); equality deletes (a read evaluates each of them against every row) | -| binding | uid and store token; the writer's content hash for verify | a content hash on the read path (reads the data) | +| metadata structure | a prolly tree per level | a flat manifest per version (O(N) per open and per commit), a checkpoint plus deltas (the checkpoint is still the list), a manifest list of manifests as in Iceberg (fine up to 10⁶ files, but not a bound), an external catalog (a service dependency) | +| prolly or B+tree | prolly | the same lookup and commit costs, but the prolly tree adds history independence, deduplication across writers, a one-hash equality of datasets, an incremental diff across lineages, and a rebuild-from-scratch oracle, for a chunker of about a hundred lines | +| the latest version | newest-first keys and one listing | a hint object and probes (O(lag), and a mutable object), a pointer updated by compare-and-swap (one more mutable thing) | +| concurrency | put-if-absent, rebase by operations | leases and lock files, three-way merges of trees | +| data across objects | leveled on a clustering key, tiered without | no compaction (O(appends)), a global secondary index (rewritten at every compaction) | +| index placement | in the file, with transient external fragments | in-file only (adding an index rewrites data), external only (the index no longer travels with the file) | +| deleting rows | a bounded deletion vector in the leaf entry, folded by compaction | copy-on-write of every object a row is taken from (an object per delete, 3 MiB per commit of ten rows, each kept through the retention window), positional delete files as in Iceberg (a read joins them, and they pile up until a rewrite), equality deletes (a read evaluates each of them against every row) | +| binding | uid and store token, with the writer's content hash for verification | a content hash on the read path (it reads the data) | | metadata objects | one commit object per version | an object per page or per pack (sequential reads, orphans after a crash, two creations per commit) | -| Iceberg or Delta as the table layer | — | their data formats are Parquet, ORC and Avro; a Vortex data file would make the table unreadable to every engine that reads them, which is the only reason to adopt them | +| Iceberg or Delta as the table layer | neither | their data formats are Parquet, ORC and Avro, and a Vortex data file would make the table unreadable to every engine that reads them, which is the only reason to adopt them | -**Not in this design:** a clustering key picked automatically, since it decides the write amplification -and is the user's to state; evolving the fields of a nested struct, or a column's type otherwise than -by widening a number or making it nullable (§13). +Not in this design: a clustering key picked automatically, since it decides the write amplification +and is the user's to state, and evolving the fields of a nested struct, or changing a column's type +other than by widening a number or making it nullable ([schema evolution](#13-schema-evolution)). ## 15. How it is tested -- **The counting matrix** of §9.2. -- **An interleaving fuzzer** over the in-memory store: readers, writers, indexers, compactions and - vacuums in seeded schedules, commits prepared on a stale version and crashes after a put; every read +- The counting matrix of [the read-path budget](#92-held-by-a-counting-store). +- An interleaving fuzzer over the in-memory store: readers, writers, indexers, compactions and vacuums + in seeded schedules, with commits prepared on a stale version and crashes after a put. Every read must equal a scan with indexes and summaries off on the same version, and every version a vacuum retains must verify whole. -- **The rebase matrix**, every row of §8.2. -- **The invariant** of §5.2 after every step of a randomized append stream drained at random moments. -- **Progressive indexing**: at every intermediate commit, the answers equal those without indexes. -- **The tree's oracles**: incremental edits equal a rebuild from scratch, byte for byte, and the same +- The rebase matrix, covering every row of the [rebase table](#82-rebase-by-re-applying-operations). +- [The invariant](#52-the-invariant), checked after every step of a randomized append stream drained + at random moments. +- Progressive indexing: at every intermediate commit, the answers equal those without indexes. +- The tree's oracles: incremental edits equal a rebuild from scratch, byte for byte, and the same operations in two orders give one root hash. -- **Order**: key-ordered reads across levels equal a sort of the rows, in both directions, with and - without summaries; a key cursor's walk down is its walk up reversed, a step against its direction +- Order: key-ordered reads across levels equal a sort of the rows, in both directions, with and + without summaries. A key cursor's walk down is its walk up reversed, a step against its direction lands next to the entry before it, and its seeks below a key land where one sorted file's would. -- **Deletes and updates**: the rows left equal the rows written filtered in C# under three-valued - logic, across levels and without a clustering key, and a delete that loses its objects to a - compaction is worked out again. -- **The plan**: a descent chooses the job a read of every leaf chooses, on random shapes and options - that meet every trigger; every page's tally is the one its leaves add up to; a cold plan asks for +- Deletes and updates: the rows left equal the rows written, filtered in C# under three-valued logic, + across levels and without a clustering key, and a delete that loses its objects to a compaction is + worked out again. +- The plan: a descent chooses the job a read of every leaf chooses, on random shapes and options that + reach every trigger, every page's tally is the one its leaves add up to, and a cold plan asks for the same one or two pages over 3 000, 31 000 and 125 000 objects. -- **Marks**: a dataset that marks reads as a second one taking the same changes by rewrites, through - every read — rows by position, rows in key order both ways with null keys around the marks, key - cursors, their ranks and seeks, counts, extremes, aggregates — under a seeded fuzzer of deletes and - updates; a compaction that read an object before rows were marked in it is abandoned; a count told - to leave rows out counts none of them on any tier, each switched off in turn; an object whose marks - are due by share or by vector is purged alone in its level; a range too long for a vector's worst - case is marked as the run it is, an update's rows written once. -- **Schema evolution**: objects of each earlier schema read as the current one through scans, counts, +- Marks: a dataset that marks reads like a second one taking the same changes by rewrites, through + every read (rows by position, rows in key order both ways with null keys around the marks, key + cursors with their ranks and seeks, counts, extremes, aggregates), under a seeded fuzzer of deletes + and updates. A compaction that read an object before rows were marked in it is abandoned, a count + told to leave rows out counts none of them on any tier, each switched off in turn, an object whose + marks are due by share or by vector is purged alone in its level, and a range too long for a + vector's worst case is marked as the single run it is, with an update's rows written once. +- Schema evolution: objects of each earlier schema read as the current one through scans, counts, extremes, key order, key cursors, deletes, updates and compaction, filters over added columns - included; every change that would lose a value is refused. -- **Tampering**: an object replaced at equal size, a page, a root and a fragment torn at every byte, a - fragment of another object: readers answer exactly, and verify names each. -- **Rust**: compaction outputs are among the files the cross-check hands to Rust 0.86.1. -- **The churn** (`bench/Vorticity.Benchmarks.Churn`): ten million rows, then thirty thousand commits of - ten rows — appends, updates and deletes at random keys — with compaction drained and vacuum run - between them. Every cost stays flat, or levels off once the marks have filled: an append 0.5 to - 0.7 ms, an update 1.9 to 2.9 ms, a delete 0.9 to 1.6 ms, five to six requests an operation with - its commit's one among them, compaction about a millisecond a commit, a commit's bytes 12 KiB and - then 16 to 55 KiB, of which its commit object is 8 to 10 KiB; a point lookup 0.7 to 1.0 ms in three + included, and every change that would lose a value is refused. +- Tampering: an object replaced at equal size, and a page, a root and a fragment torn at every byte, or + a fragment of another object. Readers answer exactly, and verification names each one. +- Rust: compaction outputs are among the files the cross-check hands to Rust 0.86.1. +- The churn bench (`bench/Vorticity.Benchmarks.Churn`): ten million rows, then thirty thousand commits + of ten rows (appends, updates and deletes at random keys), with compaction drained and vacuum run + between them. Every cost stays flat, or levels off once the marks have filled: an append takes 0.5 + to 0.7 ms, an update 1.9 to 2.9 ms and a delete 0.9 to 1.6 ms, each making five to six requests, one + of them its commit. Compaction costs about a millisecond per commit, and a commit writes 12 KiB, then + 16 to 55 KiB, of which its commit object is 8 to 10 KiB. A point lookup takes 0.7 to 1.0 ms in three to four requests, a seek and ten steps either way 0.3 to 0.7 ms, a hundred rows by position 0.2 to - 0.4 ms, a scan of every row 30 ms; the heap between 60 and 92 MiB. Rewriting instead of marking, an update takes 19 ms and - a commit writes 3.3 MiB. Before the level-0 destination, the cap on an object and the chunked buffer, - an update cost 820 ms, compaction 100 ms a commit and the heap grew to 2.2 GiB over the first 150 - commits. + 0.4 ms, and a scan of every row 30 ms, with the heap between 60 and 92 MiB. Rewriting instead of + marking, an update takes 19 ms and a commit writes 3.3 MiB. diff --git a/docs/design/14-public-api.md b/docs/design/14-public-api.md index c5ec5cc6..ee6cabaf 100644 --- a/docs/design/14-public-api.md +++ b/docs/design/14-public-api.md @@ -1,107 +1,110 @@ -# The public surface +# The public API -Why the surface a caller programs against has the shape it has, and where each part lives. The -complete listing of every public type and member is +Why the API a caller programs against has the shape it has, and where each part lives. The complete +listing of every public type and member is [tests/Vorticity.Tests/Api/PublicSurface.txt](../../tests/Vorticity.Tests/Api/PublicSurface.txt), -which `PublicSurfaceTests` holds equal to the assemblies; the XML documentation is in the source; the -[guide](../guide/README.md) shows each use in a program that runs. This document does not repeat -them: it gives the rules they follow and the decisions behind them. The engine under the surface is -[03-architecture.md](03-architecture.md), the type mapping [07-dotnet-mapping.md](07-dotnet-mapping.md), -the meaning of a predicate [08-semantics.md](08-semantics.md). +which `PublicSurfaceTests` keeps equal to the assemblies. The XML documentation is in the source, and +the [guide](../guide/README.md) shows each use in a program that runs. This document does not repeat +them. It gives the rules they follow and the decisions behind them. The engine under the API is +described in [03-architecture.md](03-architecture.md), the type mapping in +[07-dotnet-mapping.md](07-dotnet-mapping.md), and the meaning of a predicate in +[08-semantics.md](08-semantics.md). Seven rules decide every signature: -1. **A type is a schema, not a row.** A record declares which columns exist and how they are typed; - reading it yields columns. Rows are a sink a caller asks for, and pays for. A query's result of - several values is a record's schema too: the record a scan reads from, the record a result is +1. A type is a schema, not a row. A record declares which columns exist and how they are typed, and + reading it yields columns. Rows are a sink a caller asks for, and pays for. A query result made of + several values is a record's schema too: the record a scan reads from, and the record a result is read as. -2. **The compiler holds the lifetimes.** Everything borrowed from a scan is a `ref struct`: no - `Dispose` on the hot path, nothing to forget. -3. **The plan lives under the API.** A query is described by ordinary lambdas over a symbolic probe, - run once, in the shape of LINQ: its method names, and its query syntax. What cannot be pushed - down does not compile. No expression tree, no string column name and no untyped value on the +2. The compiler enforces the lifetimes. Everything borrowed from a scan is a `ref struct`, so there is + no `Dispose` on the hot path and nothing to forget. +3. The plan lives under the API. A query is described by ordinary lambdas over a symbolic probe, run + once, in the shape of LINQ (its method names and its query syntax). What cannot be pushed down + does not compile. There is no expression tree, no string column name and no untyped value on the typed path. -4. **Generic, specialized, BCL first**: `INumber`, static abstract members, intrinsics, - `System.IO.Pipelines`, `System.Diagnostics.Metrics`; nothing virtual per value. -5. **A session, not a static.** Pools, caches, I/O concurrency, parallelism and extension registries - live on a `VortexSession`. -6. **`Async` names what runs.** A method that starts the work — an awaitable, or a stream that ends - the query — ends in `Async`: `CountAsync`, `AverageAsync`, `ToRecordsAsync`. A method that only +4. Generic, specialized, BCL first: `INumber`, static abstract members, intrinsics, + `System.IO.Pipelines`, `System.Diagnostics.Metrics`, and nothing virtual per value. +5. A session, not a static. Pools, caches, I/O concurrency, parallelism and extension registries live + on a `VortexSession`. +6. `Async` names what runs. A method that starts the work (an awaitable, or a stream that ends the + query) ends in `Async`: `CountAsync`, `AverageAsync`, `ToRecordsAsync`. A method that only describes the query does not: `Where`, `GroupBy`, `Select`, `Take`, whose result is run by - `await foreach`. A builder never reads: what it checks is in the schema the open read. -7. **Batches flow.** Every source, operator and sink exchanges batches, pulled by its consumer and - decoded ahead within a bounded window. A query answers as soon as its first batch is final, holds - what is still open rather than what it read, stops reading when its consumer stops, and - allocates nothing per batch; a result is a stream of batches like a file's, and a scan of its - own. Time to first batch, peak memory and allocations are promises with gates (§9). - -What the surface is not: a DataFrame, a SQL engine, a join, a bridge to another columnar library. A -query over one table has the shape of LINQ ([16-queries.md](16-queries.md)); what is not one, a -`join` or a `let`, does not compile. An owned `RecordBatch` is where a bridge would attach, and the -plan a scan builds is what a query provider would target. + `await foreach`. A builder never reads, and what it checks is in the schema the open already read. +7. Batches flow. Every source, operator and sink exchanges batches, pulled by its consumer and decoded + ahead within a bounded window. A query answers as soon as its first batch is final, holds what is + still open rather than what it read, stops reading when its consumer stops, and allocates nothing + per batch. A result is a stream of batches like a file's, and a scan in its own right. Time to + first batch, peak memory and allocations are promises backed by gates (see [the performance + contract](#9-the-performance-contract-and-the-gates-that-hold-it)). + +The API is not a DataFrame, a SQL engine, a join, or a bridge to another columnar library. A query +over one table has the shape of LINQ ([16-queries.md](16-queries.md)), and what is not part of that +shape, a `join` or a `let`, does not compile. An owned `RecordBatch` is where a bridge would attach, +and the plan a scan builds is what a query provider would target. ## 1. Packages and namespaces | package | what it holds | depends on | |---|---|---| -| `Vorticity` | the session, files, the writer, schema, the typed scan and its columns, the tool scan, the I/O seam, options, plans, exceptions | `System.IO.Hashing`, `Vorticity.Zstd` | +| `Vorticity` | the session, files, the writer, schema, the typed scan and its columns, the tool scan, the I/O interface, options, plans, exceptions | `System.IO.Hashing`, `Vorticity.Zstd` | | `Vorticity.Generators` | the `[VortexRecord]` generator and the analyzers | build time only | | `Vorticity.Dataset` | the dataset over an object store, `[Experimental("VX0001")]` | `Vorticity` | | `Vorticity.RowEncoding` | the row encoding, `[Experimental("VX0002")]` | `Vorticity` | | `Vorticity.Zstd` | Zstandard compression and decompression in managed C#: `ZstdCompressor`, `ZstdDecompressor` | `System.IO.Hashing` | -The core has two public namespaces: `Vorticity`, for everything a caller uses, and -`Vorticity.IO`, for the segment seam. The attributes the generator reads live in the core, so a -record can implement its interface by hand without referencing the generator. +The core has two public namespaces: `Vorticity`, for everything a caller uses, and `Vorticity.IO`, +for the segment interface. The attributes the generator reads live in the core, so a record can +implement its interface by hand without referencing the generator. ## 2. The session -`VortexSession` (`Session/`) owns what must not be global: +`VortexSession` (`Session/`) owns everything that must not be global: | owned | why | |---|---| | the memory pool, 64-byte aligned by default (`AlignedMemoryPool`) | every batch, segment and builder buffer comes from it, and disposing the session returns everything | | the segment cache | one budget for every file of the session, for sources that do I/O | | the bound on reads in flight, `MaxConcurrentReads` | an object store sees at most that many requests from the process | -| the degree of parallelism, `MaxDegreeOfParallelism`, 1 by default | where every scan and writer of the session starts ([09-contracts.md](09-contracts.md) §2) | -| the index cache, `IndexCacheBytes` | what each open file keeps of decoded index runs | +| the degree of parallelism, `MaxDegreeOfParallelism`, 1 by default | where every scan and writer of the session starts (see [parallelism](09-contracts.md#2-parallelism)) | +| the query memory and scratch budgets, `MemoryBudget` and `ScratchBudget` | what the session's group bys may hold in memory and spill to disk, shared with every session given the same budget | +| the index cache, `IndexCacheBytes` | how much of the decoded index runs each open file keeps | | the files kept mapped once closed, `MappedFileCacheCount` | opening a file again takes over its mapping and every page already mapped in it | -| whether a path is mapped at all, `MapFiles`, true by default | a service that reads files others may truncate reads them positionally, so that a cut file fails a read rather than the process | -| the extension registry | dtype ids this process knows beyond the editions, so a registered type binds to a record member or a `Column`; binding checks the storage type the registration declares | +| whether a path is mapped at all, `MapFiles`, true by default | a service that reads files others may truncate reads them positionally, so a truncated file fails a read rather than the process | +| the extension registry | dtype ids this process knows beyond the editions, so a registered type binds to a record member or a `Column`. Binding checks the storage type the registration declares | `VortexSession.Default` is immutable and is what `VortexFile.OpenAsync(path)` uses. A session is thread-safe and meant to be shared by every request of a process, and its options are frozen when it is created. It opens files from a path or from any `ISegmentSource`, creates writers over a path or a -`PipeWriter`, and appends; how a path is read is [03-architecture.md](03-architecture.md) §3.5. +`PipeWriter`, and appends. How a path is read is described in [I/O](03-architecture.md#35-io). ## 3. Schema and types The schema is a value: `VortexSchema`, a list of `VortexField(Name, VortexType)` that a collection expression builds, and `VortexType`, immutable and interned by value, with a factory per dtype -(`Schema/`). `Map`, `Union` and `Variant` have no factory: a file may declare them, so the schema -shows them, but no .NET type maps to them. The arena the reader decodes into is built once at open -and never appears. +(`Schema/`), `Map(key, value)` included. `Union` and `Variant` have no factory: a file may declare +them, so the schema shows them, but no .NET type maps to them. The arena the reader decodes into is +built once at open and never appears in the API. ## 4. Records: the generated contract -A `[VortexRecord]` type — a `partial` record struct, record class, struct or class — declares -columns by its public members, in declaration order and under their names; `[VortexColumn]` renames -one and carries what a .NET type does not say (a decimal's precision and scale, a time unit, a time -zone), and `[VortexIgnore]` skips one. The generator (`src/Vorticity.Generators/RecordGenerator.cs`) -implements `IVortexRecord` — the schema, and a column-at-a-time read and write of rows — and +A `[VortexRecord]` type (a `partial` record struct, record class, struct or class) declares columns +through its public members, in declaration order and under their names. `[VortexColumn]` renames one +and carries what a .NET type does not say (a decimal's precision and scale, a time unit, a time zone), +and `[VortexIgnore]` skips one. The generator (`src/Vorticity.Generators/RecordGenerator.cs`) +implements `IVortexRecord` (the schema, and a column-at-a-time read and write of rows) and emits extension members on the probe, the columns and the builder, so that `r.Day`, `batch.Day` and -`builder.Day` are typed properties. Everything it emits can be written by hand +`builder.Day` are typed properties. Everything it emits can also be written by hand ([records.md](../guide/records.md)). -A record binds to a file once, at the first sink of a scan or at `CreateWriter`: each member +A record binds to a file once, at the first sink of a scan or at `CreateWriter`. Each member matches a column by exact name, then by a unique case-insensitive match, and a missing column, an -ambiguous match or a type that does not fit throws `VortexSchemaException` naming both. A file with -more columns than the record is fine: **a scan reads the columns its query names** — the record's -members for batches and records, the elements of a `Select`, the keys and inputs of an aggregate — -so the record is the projection of a scan that delivers it. Where the mapping leaves room, -the generator decides: an `enum` is its underlying integer on the columns and an enum symbol on the -probe; a `DateTimeOffset` member without a declared zone is UTC; a list of records and a nullable -registered extension are refused with VX1005. +ambiguous match or a type that does not fit throws a `VortexSchemaException` naming both. A file with +more columns than the record is fine, because a scan only reads the columns its query names (the +record's members for batches and records, the elements of a `Select`, the keys and inputs of an +aggregate), so the record is the projection of a scan that delivers it. Where the mapping leaves +room, the generator decides: an `enum` is its underlying integer in the columns and an enum symbol on +the probe, a `DateTimeOffset` member without a declared zone is UTC, a list of records is a list of +structs, and a nullable registered extension is refused with VX1005. ## 5. Reading @@ -110,68 +113,69 @@ registered extension are refused with VX1005. `file.Scan()` reads the columns the record names and nothing else. `Where` adds a predicate, `Rows` a range or a set of row indices, `OrderBy` or `OrderByDescending` a key order, and `With(ScanOptions)` the scan's batch cap, prefetch, degree, compaction and switches -(`Scan/Scan.Typed.cs`, `Scan/ScanOptions.cs`); `Select` and `GroupBy` turn it into a query of values -or of groups (§5.5). A -scan is consumed with `await foreach`, whatever the source: there is no synchronous enumeration. The -batch is **borrowed** — valid until the next `MoveNextAsync`, the contract of a `PipeReader`'s buffer — -and the loop body runs synchronously between two batches, so a `ref struct` in it is legal. Under -the loop, the decode of the next batch overlaps the caller's work on the current one, by `Prefetch` -batches, with buffers that alternate rather than accumulate. +(`Scan/Scan.Typed.cs`, `Scan/ScanOptions.cs`). `Select` and `GroupBy` turn it into a query of values +or of groups (see [queries](#55-queries-select-group-aggregate)). A scan is consumed with +`await foreach`, whatever the source, and there is no synchronous enumeration. The batch is borrowed, +valid until the next `MoveNextAsync`, which is the contract of a `PipeReader`'s buffer, and the loop +body runs synchronously between two batches, so a `ref struct` inside it is legal. Under the loop, +decoding the next batch overlaps the caller's work on the current one, by `Prefetch` batches, with +buffers that alternate rather than accumulate. ### 5.2 The symbolic algebra The lambda given to `Where` has the syntax of LINQ and none of its machinery (`Symbolic/`). `Probe` has a `Sym` per member, whose operators record a predicate instead of evaluating -one; the lambda runs once, when the scan is built, and its result is the plan. **What compiles is -exactly what is pushed down**: +one. The lambda runs once, when the scan is built, and its result is the plan. What compiles is +exactly what is pushed down: | written | meaning | |---|---| | `r.Day >= 900`, `r.Day.In(1, 2, 3)` | comparison and membership, against a literal of the column's type | -| `r.Celsius > 45.0 && r.City == "Paris"` | conjunction; `\|\|` and `!` likewise | -| `r.Celsius == null`, `r.Celsius.IsNull` | nullity, never unknown ([08-semantics.md](08-semantics.md) §3) | -| `r.City.StartsWith("Par")`, `r.City.Like("P_r%")`, `r.Pages.Contains(42)` | text and list predicates ([12-index-reads.md](12-index-reads.md) §6) | -| `r.High > r.Low` | column against column of one type | -| `r.At.Truncate(CalendarUnit.Day) == day`, `r.Price.Bucket(10m) >= 100m` | a function of a column; non-decreasing, so the zone maps still prune ([16-queries.md](16-queries.md) §3) | -| `r.Day >= "900"`, `r.Day % 2 == 0`, `Math.Abs(r.Celsius) > 1` | **does not compile**: a literal of the wrong type, or no pushdown | - -The same symbols are every lambda's: the keys of a `GroupBy`, the inputs of an aggregate, the -elements of a `Select`; and the results of a group compare in the `Where` that follows a `GroupBy` -as columns do in a filter ([16-queries.md](16-queries.md)). - -A captured local is read once, when the lambda runs, and an optional filter is two `Where` calls, -since the second joins with `AND`. A literal is compared **exactly**, never rounded into the column's -unit: a `DateTime` finer than a timestamp column's unit lies strictly between two stored values, so -`==` holds for no row and an ordering keeps or drops the boundary row correctly; a decimal literal -follows the same rule against the column's scale. A breakpoint inside the lambda sees symbols, not -values, and hits once. +| `r.Celsius > 45.0 && r.City == "Paris"` | conjunction, and `\|\|` and `!` likewise | +| `r.Celsius == null`, `r.Celsius.IsNull` | nullity, never unknown (see [three-valued logic](08-semantics.md#3-three-valued-logic)) | +| `r.City.StartsWith("Par")`, `r.City.Like("P_r%")`, `r.Pages.Contains(42)` | text and list predicates (see [predicates on text and lists](12-index-reads.md#6-predicates-on-text-and-lists)) | +| `r.High > r.Low` | one column against another of the same type | +| `r.At.Truncate(CalendarUnit.Day) == day`, `r.Celsius.Bucket(5.0) >= 40.0` | a function of a column, non-decreasing, so the zone maps still prune (see [value expressions](16-queries.md#3-value-expressions)) | +| `r.Day >= "900"`, `r.Day % 2 == 0`, `Math.Abs(r.Celsius) > 1` | does not compile: a literal of the wrong type, or no pushdown | + +The same symbols serve every lambda: the keys of a `GroupBy`, the inputs of an aggregate, the +elements of a `Select`. The results of a group compare in the `Where` that follows a `GroupBy` the +way columns do in a filter ([16-queries.md](16-queries.md)). + +A captured local is read once, when the lambda runs, and an optional filter is just two `Where` +calls, since the second joins with `AND`. A literal is compared exactly, never rounded into the +column's unit. A `DateTime` finer than a timestamp column's unit lies strictly between two stored +values, so `==` holds for no row, and an ordering keeps or drops the boundary row correctly. A decimal +literal follows the same rule against the column's scale. A breakpoint inside the lambda sees +symbols, not values, and is hit once. ### 5.3 `Columns` and `Column` A batch is a `Columns`, with a typed property per member and a deconstruction, and each -column a `Column` (`Columns/`): `Values` as a span for fixed-width types, a span per value for text -and binary, a range per row for a list, and the accessors of [07-dotnet-mapping.md](07-dotnet-mapping.md) -§1. `Values` is **64-byte aligned** whatever the file's segment alignment: a buffer laid out on a -smaller boundary is copied once into an aligned block at its first read. On a nullable column `Values` -is the raw buffer, meaningless where the validity bit is 0, and `ValidityWords` gives the validity as -64-bit words; the indexer returns `T?` at the cost of a branch per value, and says so. `StartRow` is -the batch's first row in the file, and `Selection` the rows a filtered or taken block keeps when it is -delivered whole (§5.4). +column is a `Column` (`Columns/`): `Values` as a span for fixed-width types, a span per value for +text and binary, a range per row for a list, and the accessors listed in [the +mapping](07-dotnet-mapping.md#1-the-mapping). `Values` is 64-byte aligned whatever the file's segment +alignment, because a buffer laid out on a smaller boundary is copied once into an aligned block at +its first read. On a nullable column, `Values` is the raw buffer, meaningless where the validity bit +is 0, and `ValidityWords` gives the validity as 64-bit words. The indexer returns `T?` at the cost of +a branch per value, and says so. `StartRow` is the batch's first row in the file, and `Selection` the +rows a filtered or taken block keeps when it is delivered whole (see [encoded views and the +selection](#54-encoded-views-and-the-selection)). ### 5.4 Encoded views and the selection A column says what its block holds before any decode (`Encoding`). A dictionary block has not been -decoded: `Values` forces the decode, while `AsDictionary()` gives its codes and values, so a consumer -that groups by it groups by code and never touches its strings; `AsRunEnd()` gives runs and their +decoded. `Values` forces the decode, while `AsDictionary()` gives its codes and values, so a consumer +that groups by it groups by code and never touches its strings. `AsRunEnd()` gives runs and their lengths ([encoded-forms.md](../guide/encoded-forms.md)). With `ScanOptions.Compact = false`, a filtered scan delivers each live block whole with its `Selection` instead of copying the surviving -rows: a filter that keeps half a block copies nothing ([selection.md](../guide/selection.md)). +rows, so a filter that keeps half a block copies nothing ([selection.md](../guide/selection.md)). ### 5.5 Queries: select, group, aggregate A projection, a group by and an aggregate are operators of the scan, not loops the caller writes -(`Aggregation/`, `Scan/Scan.Agg*.cs`, `Scan/Scan.Grouping.cs`), and they read as LINQ does, in -method or query syntax: +(`Aggregation/`, `Scan/Scan.Agg*.cs`, `Scan/Scan.Grouping.cs`), and they read as LINQ does, in method +or query syntax: ```csharp Scan hourly = @@ -184,33 +188,33 @@ Scan hourly = .As(); ``` -`GroupBy` takes one symbol or a tuple of them, whose names become the key's: `g.Key.Hour`. The -group's aggregates take a lambda over the rows, as LINQ's do, and the rows' probe exists only inside -it: a column that is neither a key nor aggregated does not compile. `Where`, `OrderBy`, `ThenBy`, -`Skip` and `Take` on the groups are a filter, an order and a top-k the engine applies. `Select` of -one value delivers an `Aggregation`, one `T` per group; `Select` of several delivers an -`Aggregation` that a record reads, as many values as the record has members, through -`As()`. `Select` on a scan delivers a `Projection` or a `Projection` of computed values -the same way. `As()` turns any of them into a `Scan` over the result, which the -result's columns fill, and on which every operator and sink of a scan runs again: a filter, a group -by of the groups, a write. `AggAsync` answers a whole scan into a record the same way. - -A query runs block by block on the encoded form — a dictionary key by code, a run-end key by run, a -sorted key by run detection, a constant one whole — and never hands a batch to the caller's code. It -**flows**: its result is a stream of batches, delivered as soon as each is final. A group by on a -key the statistics prove ordered, or a function of one that keeps the order, closes its groups as -the key moves on and delivers them then, holding the open ones only; on any other key it holds one -state per group, at any degree, and delivers at the end of its input. The whole semantics, from the -flow and the catalog of aggregates to the order of groups and the engine's strategies, is +`GroupBy` takes one symbol or a tuple of them, whose names become the key's: `g.Key.Hour`. The group's +aggregates take a lambda over the rows, as LINQ's do, and the rows' probe only exists inside it, so a +column that is neither a key nor aggregated does not compile. `Where`, `OrderBy`, `ThenBy`, `Skip` and +`Take` on the groups are a filter, an order and a top-k the engine applies. `Select` of one value +delivers an `Aggregation`, one `T` per group. `Select` of several delivers an `Aggregation` that a +record reads, with as many values as the record has members, through `As()`. `Select` on a +scan delivers a `Projection` or a `Projection` of computed values the same way. `As()` +turns any of them into a `Scan` over the result, filled by the result's columns, on which +every operator and sink of a scan runs again: a filter, a group by of the groups, a write. +`AggAsync` answers a whole scan into a record the same way. + +A query runs block by block on the encoded form (a dictionary key by code, a run-end key by run, a +sorted key by run detection, a constant one whole) and never hands a batch to the caller's code. It +flows: its result is a stream of batches, each delivered as soon as it is final. A group by on a key +the statistics prove ordered, or on a function of one that keeps the order, closes its groups as the +key moves on and delivers them then, holding only the open ones. On any other key it holds one state +per group, at any degree, and delivers at the end of its input. The full semantics, from the flow and +the catalog of aggregates to the order of groups and the engine's strategies, is in [16-queries.md](16-queries.md). -A caller's aggregator implements `IAggregator` and receives the plain form, or -`IEncodedAggregator` to receive runs and dictionaries too. A sum is exact whatever the order of rows -or the cut of a parallel aggregation: an integer sum is delivered as a 64-bit integer, or as its own -type when it is one already, throwing only when the total does not fit it; a float sum is the same -bits at every degree and under every cut of the data into chunks, files or objects. Floats skip -NaN, as the statistics do; a null key is a group of its own. Groups come in the order a query asks -for, and in no promised order without one. +A caller's aggregator implements `IAggregator` and receives the plain form, or `IEncodedAggregator` +to receive runs and dictionaries too. A sum is exact whatever the order of rows or the split of a +parallel aggregation. An integer sum is delivered as a 64-bit integer, or as its own type when it is +one already, and only throws when the total does not fit. A float sum is the same bits at every +degree and under every split of the data into chunks, files or objects. Floats skip NaN, as the +statistics do, and a null key is a group of its own. Groups come in the order a query asks for, and +in no promised order without one. ### 5.6 The sinks @@ -218,106 +222,109 @@ for, and in no promised order without one. |---|---|---|---| | `await foreach` | borrowed `Columns` | nothing per batch | the first split | | `ToBatchesAsync()` | owned `RecordBatch`es | one copy per batch into pooled buffers | the first split | -| `ToRecordsAsync()` | `IAsyncEnumerable` | a copy per row; an allocation per row for text, lists and nested classes | the first split | +| `ToRecordsAsync()` | `IAsyncEnumerable` | a copy per row, and an allocation per row for text, lists and nested classes | the first split | | `Select(r => e)` | `Projection`, a value computed per row | the same as `ToRecordsAsync`, for the columns its element names | the first split | | `Select(r => (…)).As()` | a `Scan` of values computed per row | nothing per batch, for the columns its elements name | the first split | -| `CountAsync`, `AnyAsync`, `MinAsync`, `MaxAsync`, `SumAsync`, `AverageAsync`, `CountDistinctAsync`, `AggregateAsync`, `AggAsync(a => e)` | one value | the statistics when they settle it, blocks otherwise ([12-index-reads.md](12-index-reads.md) §4) | at once when the statistics settle it, the end of the pass otherwise | -| `AggAsync(a => (…))` | several answers, one pass, into the record `TResult` | the same | the same | -| `GroupBy(…).Select(g => r)` | `Aggregation`, one value per group | one state per group, the open groups only on a key that streams; `k` groups for a top-k | the first group closed on a key that streams, the end of the pass otherwise | -| `GroupBy(…).Select(g => (…)).As()` | a `Scan` over the groups, its batches borrowed | the same, and nothing per batch nor per group | the same | -| `writer.WriteAsync(scan)` | the batches of any scan, written as they come | the writer's own work; a rollup builds no record | — | -| `ExplainAsync()` | the `ScanPlan` | statistics and zone maps, never data | — | - -**The plan ends at the sink.** A query has LINQ's names and syntax up to its last operator, and -everything up to there is pushed. Past it are values: `ToRecordsAsync()` and `ToValuesAsync()` hand -them out as an `IAsyncEnumerable`, to which `System.Linq.AsyncEnumerable` applies, on -materialized values. A query is enumerated, like a scan, by the pattern of `await foreach` rather -than as an `IAsyncEnumerable`, since what returns a stream to await carries `Async`; its own -`Skip`, `Take`, `ToListAsync`, `ToArrayAsync`, `As` and `Distinct` stay in the plan. A -query of several values is not enumerable: `As` hands it, like any result, to a scan, where -the plan goes on ([16-queries.md](16-queries.md) §1, [read-rows.md](../guide/read-rows.md)). +| `CountAsync`, `AnyAsync`, `MinAsync`, `MaxAsync`, `SumAsync`, `AverageAsync`, `CountDistinctAsync`, `AggregateAsync`, `AggAsync(a => e)` | one value | the statistics when they settle it, blocks otherwise (see [answers without rows](12-index-reads.md#4-answers-without-rows)) | at once when the statistics settle it, at the end of the pass otherwise | +| `AggAsync(a => (…))` | several answers in one pass, into the record `TResult` | the same | the same | +| `GroupBy(…).Select(g => r)` | `Aggregation`, one value per group | one state per group, only the open groups on a key that streams, and `k` groups for a top-k | the first group closed on a key that streams, the end of the pass otherwise | +| `GroupBy(…).Select(g => (…)).As()` | a `Scan` over the groups, its batches borrowed | the same, and nothing per batch or per group | the same | +| `writer.WriteAsync(scan)` | the batches of any scan, written as they come | the writer's own work, and a rollup builds no record | | +| `ExplainAsync()` | the `ScanPlan` | statistics and zone maps, never data | | + +The plan ends at the sink. A query has LINQ's names and syntax up to its last operator, and +everything up to there is pushed down. Past it are values: `ToRecordsAsync()` and `ToValuesAsync()` +hand them out as an `IAsyncEnumerable`, to which `System.Linq.AsyncEnumerable` applies, on +materialized values. A query is enumerated, like a scan, through the `await foreach` pattern rather +than as an `IAsyncEnumerable`, since whatever returns a stream to await carries `Async`. Its own +`Skip`, `Take`, `ToListAsync`, `ToArrayAsync`, `As` and `Distinct` stay in the plan. A query +of several values is not enumerable: `As` hands it, like any result, to a scan, where the plan +continues (see [the shape](16-queries.md#1-the-shape) and [read-rows.md](../guide/read-rows.md)). ### 5.7 The key cursor -`Scan().Keys(r => r.Column)` builds a `KeyCursorBuilder`, `TKey` inferred from the -member, and `OrderBy(r => r.Column)` delivers rows in key order: [12-index-reads.md](12-index-reads.md). +`Scan().Keys(r => r.Column)` builds a `KeyCursorBuilder`, with `TKey` inferred from +the member, and `OrderBy(r => r.Column)` delivers rows in key order. Both are described in +[12-index-reads.md](12-index-reads.md). ### 5.8 The tool path -For a file whose schema is not known when the program is compiled — `vxdump`, an ad hoc query, a -test — `file.Scan(columns)` reads columns by name, and `Where` takes a filter as text with typed holes -(`Scan/Scan.Tool.cs`, `Expressions/VortexExpr.Surface.cs`). It is the one place in the surface where -names are strings, because there is no other name. A batch is a `BatchView` holding the named columns -in the file's order; a number in a filter is the number the text wrote, scaled exactly against a -decimal column, and text compared with a date, time, timestamp, uuid or decimal column is parsed in +For a file whose schema is not known when the program is compiled (`vxdump`, an ad hoc query, a +test), `file.Scan(columns)` reads columns by name, and `Where` takes a filter as text with typed holes +(`Scan/Scan.Tool.cs`, `Expressions/VortexExpr.Surface.cs`). It is the one place in the API where names +are strings, because there is no other way to name them. A batch is a `BatchView` holding the named +columns in the file's order. A number in a filter is the number the text wrote, scaled exactly against +a decimal column, and text compared with a date, time, timestamp, uuid or decimal column is parsed in that type and refused when it does not parse ([untyped-files.md](../guide/untyped-files.md)). ## 6. Writing -The shape of a `PipeWriter`: the caller fills builders synchronously, and the writer encodes and hands -the bytes to its sink asynchronously (`Writing/`). A writer comes from the session, over a path or a -pipe, typed by a record or by a schema; `Builder()` hands out its `ColumnsBuilder`, whose -`ColumnBuilder` gives spans to fill in place (`GetSpan`, `Advance`, as `IBufferWriter`), +Writing has the shape of a `PipeWriter`: the caller fills builders synchronously, and the writer +encodes and hands the bytes to its sink asynchronously (`Writing/`). A writer comes from the session, +over a path or a pipe, typed by a record or by a schema. `Builder()` hands out its `ColumnsBuilder`, +whose `ColumnBuilder` gives spans to fill in place (`GetSpan`, `Advance`, as `IBufferWriter`), appends values, nulls, text and lists, and is reused after every `WriteAsync`. A primitive column is -encoded from the caller's bytes without a copy. A list appended in one call that fails part way -leaves nothing behind. A nullable column costs nothing until its first null. +encoded from the caller's bytes without a copy. A list appended in one call that fails part way leaves +nothing behind. A nullable column costs nothing until its first null. `WriteAsync` takes a builder, a span or a stream of records, or a batch read from another file. `FlushAsync` seals the whole blocks into a chunk and is the only I/O before `CompleteAsync`, which writes the tail and returns the `WriteReport`. `Abandon()` gives the file up: a new file is deleted, -an append truncated back to its length, and a caller's pipe completed with an error. What the writer -does with the rows is [11-write-strategy.md](11-write-strategy.md); the guide's writing pages show each -form. +an append is truncated back to its length, and a caller's pipe is completed with an error. What the +writer does with the rows is described in [11-write-strategy.md](11-write-strategy.md), and the +guide's writing pages show each form. ## 7. Options, plans, reports, diagnostics, exceptions | type | what it says | |---|---| | `VortexOpenOptions` | the initial tail read, a known length or schema, the decompression cap, statistics verification, a torn tail's policy, index fragments to attach | -| `ScanOptions` | batch cap, prefetch, degree, `Compact`, and the `Pruning` and `UseIndexes` switches, which exist to check the structures, never to change a result | +| `ScanOptions` | batch cap, prefetch, degree, `Compact`, and the `Pruning` and `UseIndexes` switches, which exist to check the structures and never change a result | | `VortexWriteOptions` | blocks and chunk targets, the compression profile and per-column hints, the target edition, statistics, string bounds, the index policy, the identity, user metadata, the degree | -| `IndexPolicy`, `EncodingHint`, `CompressionProfile` | [10-indexes.md](10-indexes.md) §6, [11-write-strategy.md](11-write-strategy.md) §3.4 | -| `ScanPlan`, `PruningStep`, `CountPlan`, `OrderPlan`, `GroupPlan`, `GroupKeyPlan`, `GroupOrdering` | the plan before a read: blocks, live blocks, segments and bytes to read, what each structure pruned and what consulting it cost, how a count, an order and a group by will be answered ([16-queries.md](16-queries.md) §10) | -| `ScanStatistics`, `GroupStatistics` | the same quantities, measured after the scan; for a group by, its groups, the most held at once and the bytes, what its core did, how each key block was grouped, and the time to its first batch | +| `IndexPolicy`, `EncodingHint`, `CompressionProfile` | see [the index policy](10-indexes.md#6-the-policy) and [choosing encodings](11-write-strategy.md#34-choose-exact-verdicts-then-bounded-trials) | +| `ScanPlan`, `PruningStep`, `CountPlan`, `OrderPlan`, `GroupPlan`, `GroupKeyPlan`, `GroupOrdering` | the plan before a read: blocks, live blocks, segments and bytes to read, what each structure pruned and what consulting it cost, and how a count, an order and a group by will be answered (see [plan and statistics](16-queries.md#10-plan-and-statistics)) | +| `ScanStatistics`, `GroupStatistics` | the same quantities, measured after the scan, and for a group by its groups, the most held at once and their bytes, what its core did, how each key block was grouped, and the time to its first batch | | `WriteReport` | what the writer chose, per column and per chunk, and every index built or abandoned | -| `VortexDiagnostics` | the names of the meter and the activity source ([09-contracts.md](09-contracts.md) §5) | -| `VortexException` and its three kinds | a malformed file, an unsupported component with its id and `ComponentKind`, a schema that does not fit ([03-architecture.md](03-architecture.md) §5) | +| `VortexDiagnostics` | the names of the meter and the activity source (see [observability](09-contracts.md#5-observability)) | +| `VortexException` and its four kinds | a malformed file, an unsupported component with its id and `ComponentKind`, a schema that does not fit, and a query that outgrows its memory or scratch budget (see [error handling](03-architecture.md#5-error-handling)) | | `VortexEditions` | the default, newest and floor editions, and the Rust version that reads each | -**The plan and the statistics count in the same units**: a plan's `Segments` and `BytesToRead` are what +The plan and the statistics count in the same units. A plan's `Segments` and `BytesToRead` are what the source will be asked for, each segment once, and `Requests` and `BytesRequested` are what it was asked for, so on a file with statistics the two agree. `BlocksDecoded` counts the blocks that reached -the plain form: a block answered from a dictionary's codes or a run's lengths is not decoded. +the plain form, so a block answered from a dictionary's codes or a run's lengths is not counted as +decoded. -## 8. The I/O seam +## 8. The I/O interface `Vorticity.IO` holds `ISegmentSource` and its three built-in implementations, whose contract is -[03-architecture.md](03-architecture.md) §3.5, and a writer's sink is any `PipeWriter`. What a scan -promises its source: **a segment is requested at most once per scan**, and the session's cache adds -reuse across scans. [object-store.md](../guide/object-store.md) implements the seam for a remote store. +described in [I/O](03-architecture.md#35-io), and a writer's sink is any `PipeWriter`. A scan promises +its source that a segment is requested at most once per scan, and the session's cache adds reuse +across scans. [object-store.md](../guide/object-store.md) implements the interface for a remote +store. ## 9. The performance contract, and the gates that hold it -Each promise is a test (`tests/Vorticity.Tests/Api/*ContractTests.cs` and their neighbours), and -the build fails when one stops holding: +Each promise is a test (`tests/Vorticity.Tests/Api/*ContractTests.cs` and their neighbours), and the +build fails when one stops holding: | promise | gate | |---|---| -| no allocation per batch on every sink but rows | allocation counted across a full scan after one warm-up batch | -| `Values` contiguous and 64-byte aligned | the address of every `Values` span of the conformance corpus | -| a segment requested at most once per scan, and the plan is the execution | `Requests` equals the plan's distinct segments, `BytesRequested` equals `BytesToRead`, on every query of the corpus | +| no allocation per batch on every sink except rows | allocations counted across a full scan after one warm-up batch | +| `Values` contiguous and 64-byte aligned | the address of every `Values` span over the conformance corpus | +| a segment requested at most once per scan, and the plan is the execution | `Requests` equals the plan's distinct segments, and `BytesRequested` equals `BytesToRead`, on every query of the corpus | | a pruned block is not decoded | `BlocksDecoded` equals `LiveBlocks` | | a question the statistics answer reads nothing | no request after the open for a count, a minimum or a maximum on a file with statistics | -| a dictionary or run-end aggregate does not decode | `BlocksDecoded` is 0 for a group by on a dictionary column and a sum over runs | +| a dictionary or run-end aggregate does not decode | `BlocksDecoded` is 0 for a group by on a dictionary column and for a sum over runs | | a chunk is decoded once per scan, whatever runs it | the values decoded equal rows × columns at every prefetch and degree, and for a parallel aggregation | -| a query's answers and order do not depend on its syntax, its degree or the cut of its data | method and query syntax build one plan; every answer, float sums included, the same bits at every degree, over two row orders, and over a dataset cut into several objects, before and after compaction ([16-queries.md](16-queries.md) §13) | -| the first batch waits for the first split | the time to first batch of a scan, a projection, a `Distinct` and a group by that streams, flat over two files sixteen times apart, locally and over the HTTP source with latency | -| memory is the window and the open state | `LiveMemoryTests`: a streaming group by's peak flat as its groups grow a hundredfold, a top-k's proportional to `k`, a high-cardinality group by's at most its groups' | +| a query's answers and order do not depend on its syntax, its degree or how its data is split | method and query syntax build one plan, and every answer, float sums included, is the same bits at every degree, over two row orders, and over a dataset split into several objects, before and after compaction (see [how queries are tested](16-queries.md#13-how-it-is-tested)) | +| the first batch only waits for the first split | the time to first batch of a scan, a projection, a `Distinct` and a group by that streams stays flat over two files sixteen times apart, locally and over the HTTP source with latency | +| memory is the window plus the open state | `LiveMemoryTests`: a streaming group by's peak stays flat as its groups grow a hundredfold, a top-k's is proportional to `k`, and a high-cardinality group by's is at most its groups' | | a `Take` reads what it uses | `Requests` and `BlocksDecoded` bounded by the splits that hold its rows and the window | -| a group by is no slower than the loop by hand | a throughput axis per key form, the hand-written loops of the guide as floor | -| what cannot be pushed down does not compile | `tests/MustNotCompile` builds with exactly the expected diagnostics, a column outside an aggregate, `let` and `join` included | -| the surface does not drift | `PublicSurfaceTests` against `PublicSurface.txt`, and rule 6: a member that returns an awaitable or a terminal stream ends in `Async`, a builder does not | -| `vxdump` uses the public surface only | it compiles without access to internals | +| a group by is no slower than the hand-written loop | a throughput axis per key form, with the guide's hand-written loops as the floor | +| what cannot be pushed down does not compile | `tests/MustNotCompile` builds with exactly the expected diagnostics, including a column outside an aggregate, `let` and `join` | +| the API does not drift | `PublicSurfaceTests` against `PublicSurface.txt`, and rule 6: a member that returns an awaitable or a terminal stream ends in `Async`, and a builder does not | +| `vxdump` uses the public API only | it compiles without access to internals | The analyzers ship in `Vorticity.Generators` ([diagnostics.md](../guide/diagnostics.md)): @@ -331,29 +338,30 @@ The analyzers ship in `Vorticity.Generators` ([diagnostics.md](../guide/diagnost | VX1006 | a record type, or a type containing it, that is not `partial` | | VX1007 | a type that cannot be a record: generic, `ref struct`, static, abstract, file-local, or without a usable constructor | | VX1008 | a member the reader cannot fill | -| VX1009 | a component of a `GroupBy` key that is not a symbol of the scan: a literal, a captured value | +| VX1009 | a component of a `GroupBy` key that is not a symbol of the scan, such as a literal or a captured value | | VX1010 | an `As()` or an `AggAsync` whose record members do not take the selected values, position by position | -| VX1011 | several selected values read without a record: enumerated, or awaited without `TResult`; it names `As()` and the record to declare | +| VX1011 | several selected values read without a record (enumerated, or awaited without `TResult`). It names `As()` and the record to declare | -VX1001 to VX1004 are warnings; VX1005 to VX1008 are errors, and the generator emits nothing for a type -that has one; VX1009 and VX1010 are errors, since what they flag throws when the query is built, and -VX1011 goes with the compiler's own error, to name the fix. +VX1001 to VX1004 are warnings. VX1005 to VX1008 are errors, and the generator emits nothing for a type +that has one. VX1009 and VX1010 are errors, since what they flag throws when the query is built, and +VX1011 accompanies the compiler's own error, to name the fix. ## 10. The satellites -- **`Vorticity.Dataset`**: `VortexDataset.Scan()` returns the same `Scan` as a - file, over every object of the version the handle read, which a later commit does not move; a - batch's `StartRow` and a cursor's `Row` are positions in the dataset, and a cursor walks both ways - as a file's does. Its objects are opened ahead of the one being read and read side by side at a - degree above one, so the first batch waits for the first object and no object boundary stalls the - stream; every query of [16-queries.md](16-queries.md) runs over it, an object a range of a parallel +- `Vorticity.Dataset`: `VortexDataset.Scan()` returns the same `Scan` as a file, + over every object of the version the handle read, which a later commit does not move. A batch's + `StartRow` and a cursor's `Row` are positions in the dataset, and a cursor walks both ways like a + file's. Its objects are opened ahead of the one being read, and read side by side at a degree above + one, so the first batch only waits for the first object and no object boundary stalls the stream. + Every query of [16-queries.md](16-queries.md) runs over it, with an object as a range of a parallel aggregation. `DeleteAsync` and `UpdateAsync` take the same lambda as `Where`, and - `EvolveSchemaAsync` a `VortexSchema` and the renamed columns. The store seam, the maintenance calls, - what each costs and how a change of rows or of schema commits are [13-dataset.md](13-dataset.md). -- **`Vorticity.RowEncoding`**: `RowEncoder`, `RowSortField`, `RowKeys` and `RowKeyEncoder` + `EvolveSchemaAsync` takes a `VortexSchema` and the renamed columns. The store interface, the + maintenance calls, what each costs and how a change of rows or of schema commits are described in + [13-dataset.md](13-dataset.md). +- `Vorticity.RowEncoding`: `RowEncoder`, `RowSortField`, `RowKeys` and `RowKeyEncoder` ([06-row-encoding.md](06-row-encoding.md)). `[Experimental]` is set on the assembly of both experimental packages, so every type reports its diagnostic to a caller, and the packages' own code needs no suppression. -- **`vxdump`** is written against the tool path, the inspection members of `VortexFile` and the plans - only, and published ahead of time: a section it cannot print from the public surface is a gap in the - surface, not in the tool ([vxdump.md](../guide/vxdump.md)). +- `vxdump` is written against the tool path, the inspection members of `VortexFile` and the plans + only, and published ahead of time. A section it could not print from the public API would be a gap + in the API, not in the tool ([vxdump.md](../guide/vxdump.md)). diff --git a/docs/design/15-compaction.md b/docs/design/15-compaction.md index 27ec35ef..86782a9e 100644 --- a/docs/design/15-compaction.md +++ b/docs/design/15-compaction.md @@ -1,322 +1,341 @@ # Compaction: when it runs, and what a store that cannot delete does to it -Compaction is [13-dataset.md](13-dataset.md) §5 together with what this document describes: one -planner, whose jobs run one per `CompactAsync` call, per turn of a background loop, -or in the commit that takes level 0 past its ceiling (§2); a plan that descends the trees rather -than reading them (§3); purges of marks, and long vectors out of line (§4); and a profile for a -store under a retention lock (§5). This document weighs the ways to drive the same work: inline in a -writer's commit, on demand, or in the background. It also weighs what a store under a retention lock -changes, when it refuses to delete or overwrite an object before a date: S3 Object Lock, Azure -immutable blob storage, a GCS bucket lock. Nothing here changes the commit protocol. The format -carries the tallies in a page's summary and a pointer in each level of a header (§3), a vector's -reference in an entry (§4) and the locked store in a header (§5). A page without tallies still -reads, and a reader that knows no vector out of line refuses such an entry rather than reading it -wrong. +Compaction is the [levels and compaction](13-dataset.md#5-levels-and-compaction) section of the +dataset design, plus what this document describes: + +- one planner, whose jobs run one per `CompactAsync` call, one per turn of a background loop, or in + the commit that takes level 0 past its ceiling ([three drivers](#2-three-drivers-for-one-job)) +- a plan that descends the trees rather than reading them + ([planning](#3-planning-in-the-depth-of-the-tree)) +- purges of marks, and long vectors stored out of line + ([folding marks](#4-folding-marks-where-no-merge-goes)) +- a profile for a store under a retention lock ([stores that cannot + delete](#5-stores-that-cannot-delete)) + +The document weighs the ways to drive the same work (inline in a writer's commit, on demand, or in +the background), and what changes on a store under a retention lock, one that refuses to delete or +overwrite an object before a date: S3 Object Lock, Azure immutable blob storage, a GCS bucket lock. +Nothing here changes the commit protocol. The format carries the tallies in a page's summary and a +pointer in each level of a header, a vector's reference in an entry, and the locked-store flag in a +header. A page without tallies still reads, and a reader that does not know about vectors out of line +refuses such an entry rather than reading it wrong. ## 1. What compaction buys, and what it costs -Compaction holds the read bounds of [13-dataset.md](13-dataset.md) §1: +Compaction holds the read bounds of the dataset (see [where the N's +hide](13-dataset.md#1-where-the-ns-hide-and-the-bound-that-replaces-each)): -- **Level 0 holds at most eight objects.** A lookup touches at most eight objects there, rather than +- Level 0 holds at most eight objects, so a lookup touches at most eight objects there, rather than one per append since the last merge. -- **The levels above 0 are key-disjoint.** A lookup touches one object in each of them. -- **The marks a delete leaves are folded.** Their rows are no longer stepped over, and their bytes are - freed (§12 of the same document). +- The levels above 0 are key-disjoint, so a lookup touches one object in each of them. +- The marks a delete leaves are folded, so their rows are no longer stepped over and their bytes are + freed (see [deleting rows](13-dataset.md#12-deleting-and-updating-rows)). It pays for these in bytes written. A row appended in a small commit moves up through every level, and each merge rewrites the part of the next level it overlaps. That is about `F / 2` rewrites per -level at a fan-out `F`: about 5 at the default of 10. A load of gigabytes goes straight to the level -that holds it and is written once (§5.2). The levels grow by `F`. With level 1 at 1.25 MiB, a -tebibyte of data needs seven levels, so a byte that came in with a small commit is written about 35 -times before it settles. +level at a fan-out `F`, about 5 at the default of 10. A load of gigabytes goes straight to the level +that holds it and is written once (see [the invariant](13-dataset.md#52-the-invariant)). The levels +grow by `F`. With level 1 at 1.25 MiB, a tebibyte of data needs seven levels, so a byte that came in +with a small commit is written about 35 times before it settles. -Three costs are left over, and this document is about them: +Three costs remain, and this document is about them: -1. **Who pays, and when.** On demand, whoever calls `CompactAsync`, when they do; until then a +1. Who pays, and when. On demand, whoever calls `CompactAsync` pays, when they call it. Until then a dataset reports its lag, and lookups touch the extra objects. -2. **Planning.** With a level's bytes recorded nowhere but in its leaves, a plan would walk every - level's tree: nothing at the 40 objects of the churn, but at a tebibyte of 4 MiB objects 262 144 - entries of a few hundred bytes each, about a thousand pages and a hundred megabytes, on every - plan. A plan descends instead (§3). -3. **Marks in levels no merge reaches.** Level 4 of a ten-million-row dataset is rewritten only when +2. Planning. If a level's bytes were only recorded in its leaves, a plan would walk every level's + tree. That is nothing at the 40 objects of the churn bench, but at a tebibyte of 4 MiB objects it + is 262 144 entries of a few hundred bytes each, about a thousand pages and a hundred megabytes, on + every plan. A plan descends instead ([planning](#3-planning-in-the-depth-of-the-tree)). +3. Marks in levels no merge reaches. Level 4 of a ten-million-row dataset is only rewritten when level 3 overflows into it, and the top level never is. The marks there last until a delete would pass an object's share or its vector's cap, and that delete then rewrites the object itself. Meanwhile each commit that marks a row writes again the leaf page holding the object's entry, and - that page grows with the vectors it holds (§4). + that page grows with the vectors it holds ([folding marks](#4-folding-marks-where-no-merge-goes)). ## 2. Three drivers for one job -A **job** stays what it is: one plan's inputs read in key order, the outputs written, and one -`ReplaceObjects` committed. That commit names the entries it read, and is abandoned when one of -them changed (13-dataset.md §8.2). Correctness never depends on who runs the job, so every driver -below is safe against every other one and against every writer. What they differ in is who waits -for the work, and how much of it is wasted. +A job is always the same thing: one plan's inputs read in key order, the outputs written, and one +`ReplaceObjects` committed. That commit names the entries it read, and is abandoned when one of them +changed (see [rebase](13-dataset.md#82-rebase-by-re-applying-operations)). Correctness never depends +on who runs the job, so every driver below is safe against every other one and against every writer. +What they differ in is who waits for the work, and how much of it is wasted. | driver | who pays | the bounds | what goes wrong | |---|---|---|---| -| inline, in the commit | the writer whose commit crosses a bound | held after every commit | the writer's latency jumps by a merge; a writer does work for everyone; a crash leaves the outputs for vacuum | -| on demand, `CompactAsync` | the application's scheduler | held when it runs; `Lag` says by how much they are not | a scheduler that stops leaves lookups touching every object since | +| inline, in the commit | the writer whose commit crosses a bound | held after every commit | the writer's latency jumps by a merge, a writer does work for everyone, and a crash leaves the outputs for vacuum | +| on demand, `CompactAsync` | the application's scheduler | held when it runs, and `Lag` says by how much they are not | a scheduler that stops leaves lookups touching every object since | | in the background | a loop in some process | held within the loop's reaction time | two loops on one dataset race for the same inputs, and the loser's outputs are garbage | -**Inline.** A commit that takes level 0 past its ceiling runs one job before it returns, so that no -version the writer publishes leaves level 0 over its bound for long. The price is the merge's own -latency, added to that one commit: level 0 into level 1 costs 0.3 to 1 ms a commit in the churn, -spread over the commits between merges. A merge into a deep level is seconds. So inline mode is -bounded to level 0, and to inputs under a byte budget: past it, the job is left to another driver -and the writer returns at once. Inline mode suits a dataset with one writer and no process to spare. -The writer is the one party that knows level 0 just grew. - -**Implementation** (`DatasetOptions.InlineCompactionBytes`, 0 by default): an append, an import, a -replacement or a change that adds to level 0 runs the level-0 job before it returns, when the job -reads no more than the budget. The commit stands whatever befalls the job: one that fails, or that -the caller cancels, leaves level 0 for the next commit or another driver, and the caller is told the -commit's outcome. On the churn the same bytes are written either way; inline, the merges move into -the writers' commits, and an append takes 0.9 to 1.1 ms on average rather than 0.5 to 0.7, its 99th -percentile 6 to 9 ms rather than 3. - -**On demand.** The default. `PlanCompactionAsync` says what is due and what it costs, and -`CompactAsync` runs one job. The application loops, from a scheduled task or after a batch of -commits. It is the right default for a library, because the library never starts a thread of its own. -It stays the primitive the other two drivers are made of. - -**In the background.** A long-running call, `RunCompactionAsync(schedule, cancellationToken)`, that -any host runs: a hosted service, a worker, a console. It asks for the latest version, one head -request for a handle that knows its own is recent (13-dataset.md §8.3). It plans, runs the job, and -sleeps when nothing is due. Two limits shape it: a budget of bytes written per second, so that -compaction never takes the store's bandwidth from the writers, and one job at a time per loop. A -hosting integration, such as `IHostedService`, belongs in a separate package, since the dataset -package takes no dependency. - -**Two loops on one dataset** are safe and wasteful: both plan the same job, and one commit loses. -The waste is avoided without any object that coordinates: - -- **Deterministic job choice.** The planner ranks the due jobs, and a loop that knows it is one of - `n` takes the job whose rank is its index modulo `n`, which spreads `n` loops over `n` jobs. -- **Or a lease.** Before a job, a loop creates `leases//