From 502b1ce157d5d1d000f2d198eb77125c975ca971 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20DUCOM?= Date: Sat, 10 Oct 2026 13:11:19 +0200 Subject: [PATCH 1/4] docs(guide): every page checked against the code and written for a newcomer The guide had drifted from the code in a few dozen places, and its prose read as working notes. Each page now matches what the library does today, with its sample outputs taken from a fresh run, and says the essential first in whole sentences, without dashes, semicolons or internal references. Among the facts that changed, the writer stores a chunk of one value as a constant rather than a progression, and the demonstration file is 1 508 316 bytes in 54 segments. Groups come in no promised order without an OrderBy, KeyCountAsync counts the entries under the current key, and owned batches keep their selection. Lists of records, char, arrays and dictionaries all map, an untyped non-nullable column over a nullable one is refused, the file statistics carry no sum, and vxdump exits with 2 on an unknown option. The errors page gains the memory, torn-commit and integrity exceptions, and the observability page gains the event counters and the grouping statistics. Two sample labels follow the code. The key cursor prints the entries under a key, and the required-index sample says the path still holds the file written before it. --- docs/guide/README.md | 112 ++++---- docs/guide/aggregates.md | 350 ++++++++++++----------- docs/guide/append-and-repair.md | 129 ++++----- docs/guide/benchmarks-duckdb.md | 143 +++++---- docs/guide/benchmarks-x64.md | 97 ++++--- docs/guide/benchmarks.md | 98 +++---- docs/guide/blocks-and-chunks.md | 116 ++++---- docs/guide/cancel-work.md | 63 ++-- docs/guide/choose-encodings.md | 229 +++++++-------- docs/guide/copy-a-file.md | 71 ++--- docs/guide/dataset-maintenance.md | 208 +++++++------- docs/guide/datasets.md | 242 ++++++++-------- docs/guide/diagnostics.md | 190 ++++++------ docs/guide/editions.md | 67 ++--- docs/guide/encoded-forms.md | 83 +++--- docs/guide/errors.md | 100 ++++--- docs/guide/filter-rows.md | 105 +++---- docs/guide/getting-started.md | 79 ++--- docs/guide/how-it-works.md | 120 ++++---- docs/guide/indexes.md | 156 +++++----- docs/guide/keys-in-order.md | 95 +++--- docs/guide/limits.md | 112 ++++---- docs/guide/native-aot.md | 100 +++---- docs/guide/nested-records.md | 49 ++-- docs/guide/nullable-columns.md | 76 ++--- docs/guide/object-store.md | 120 ++++---- docs/guide/observability.md | 110 ++++--- docs/guide/open-a-file.md | 149 +++++----- docs/guide/owned-batches.md | 89 +++--- docs/guide/project-columns.md | 58 ++-- docs/guide/queries.md | 94 +++--- docs/guide/read-rows-by-index.md | 84 +++--- docs/guide/read-rows.md | 59 ++-- docs/guide/records.md | 162 ++++++----- docs/guide/row-keys.md | 103 +++---- docs/guide/scan-a-table.md | 112 ++++---- docs/guide/selection.md | 62 ++-- docs/guide/statistics-and-pruning.md | 137 ++++----- docs/guide/stream-to-an-object.md | 67 ++--- docs/guide/text-columns.md | 66 ++--- docs/guide/threads.md | 152 +++++----- docs/guide/untyped-files.md | 72 ++--- docs/guide/vxdump.md | 133 +++++---- docs/guide/write-a-file.md | 103 ++++--- docs/guide/write-lists-and-records.md | 63 ++-- docs/guide/write-nulls.md | 59 ++-- docs/guide/write-rows.md | 58 ++-- docs/guide/write-text.md | 45 +-- docs/guide/write-without-a-record.md | 49 ++-- docs/guide/writer-options.md | 178 ++++++------ samples/Vorticity.Samples/Indexes.cs | 2 +- samples/Vorticity.Samples/KeysInOrder.cs | 2 +- 52 files changed, 2862 insertions(+), 2716 deletions(-) diff --git a/docs/guide/README.md b/docs/guide/README.md index 60ce62a3..96aa8e39 100644 --- a/docs/guide/README.md +++ b/docs/guide/README.md @@ -1,20 +1,20 @@ # The guide -Pages named by what you are trying to do. Each one is short, opens with a working example, and says -what happens, what to watch out for, and what it costs. +These pages are named after what you are trying to do. Each one opens with a working example, then +explains what happens, what to watch out for, and what it costs. -Every example is a program in [samples/Vorticity.Samples](../../samples/Vorticity.Samples), -compiled by the build and run by the page's name, and the figures a page quotes are what its program -measured: +Every example is a program in [samples/Vorticity.Samples](../../samples/Vorticity.Samples). The build +compiles it, and you can run it by the page's name: ``` dotnet run -c Release --project samples/Vorticity.Samples -- filter-rows ``` -With no argument it runs every case, and the numbers a page quotes come from such a Release run: a -page's case runs after the others, on code they have already compiled, where a case run alone also -pays for compiling what it runs first. The run writes two demonstration files and deletes them when -it ends: a million readings and a hundred thousand visits, of these three records. +With no argument it runs every case. The figures quoted in the pages come from such a Release run, +where each case runs after the others on code they have already compiled. A case run alone also +pays for compiling what it runs first, so its first timings are higher. The run writes two +demonstration files, a million readings and a hundred thousand visits, and deletes them when it +ends. They use these three records: ```csharp [VortexRecord] public partial record struct Reading(int Day, double? Celsius, string City); @@ -22,93 +22,93 @@ it ends: a million readings and a hundred thousand visits, of these three record [VortexRecord] public partial record struct Address(string Country, string? City); ``` -The design documents are a different thing and live beside this directory: they say why the format -and this implementation are shaped as they are. A page here points at one when you want the depth, -and never repeats it. +The [design documents](../design/README.md) live beside this directory. They explain why the format +and this implementation are shaped the way they are. A page here links to one when you want the +depth, and does not repeat it. ## Start here | page | what you get | |---|---| -| [getting-started.md](getting-started.md) | reference the library and its generator, declare a record, write a file, read a column back | +| [getting-started.md](getting-started.md) | reference the packages, declare a record, write a file, read a column back | ## Reading | page | what you get | |---|---| -| [open-a-file.md](open-a-file.md) | open from a path, which maps the file, or from a source you choose, and the session every open runs in | -| [scan-a-table.md](scan-a-table.md) | every batch of a file with `await foreach`, as borrowed columns that cost no allocation | -| [project-columns.md](project-columns.md) | three columns of fifty, paying for three: the record is the projection | -| [filter-rows.md](filter-rows.md) | a predicate written as a lambda and pushed down, so that whole blocks are never read; what does not push down does not compile | -| [nullable-columns.md](nullable-columns.md) | the values and the validity of a column with nulls, read word by word, or summed by the engine | -| [text-columns.md](text-columns.md) | text as UTF-8 spans without a `string`, and a distinct count that is not a loop | -| [encoded-forms.md](encoded-forms.md) | a dictionary or run-end column read as it is stored, grouped by code or by run with nothing decoded | -| [selection.md](selection.md) | a filtered block delivered whole with the rows that passed, instead of a copy of them | -| [aggregates.md](aggregates.md) | several answers in one pass, a group by, and an aggregator of your own | -| [queries.md](queries.md) | a group by as a query or a chain, its groups filtered, ordered and cut, and the groups that go out as they close | +| [open-a-file.md](open-a-file.md) | open a file from a path or from a source you choose, and the session every open runs in | +| [scan-a-table.md](scan-a-table.md) | read every batch with `await foreach`, as borrowed columns that cost no allocation | +| [project-columns.md](project-columns.md) | read three columns out of fifty and pay for three: the record is the projection | +| [filter-rows.md](filter-rows.md) | a predicate written as a lambda and pushed down, so whole blocks are never read | +| [nullable-columns.md](nullable-columns.md) | the values and the validity of a column with nulls, read word by word or summed by the engine | +| [text-columns.md](text-columns.md) | text as UTF-8 spans without a `string`, and distinct counts without a loop | +| [encoded-forms.md](encoded-forms.md) | a dictionary or run-end column read as it is stored, grouped by code or by run | +| [selection.md](selection.md) | a filtered block delivered whole with the rows that passed, instead of a copy | +| [aggregates.md](aggregates.md) | several answers in one pass, group bys, and an aggregator of your own | +| [queries.md](queries.md) | group bys as queries, their groups filtered, ordered and cut, and groups delivered as soon as they are final | ## Going further on reading | page | what you get | |---|---| -| [keys-in-order.md](keys-in-order.md) | batches in the order of a key, and a cursor that seeks, steps, ranks and counts keys | -| [read-rows.md](read-rows.md) | one record per row when rows are what you need, and where LINQ starts | +| [keys-in-order.md](keys-in-order.md) | batches in key order, and a cursor that seeks, steps, ranks and counts | +| [read-rows.md](read-rows.md) | one record per row when rows are what you need, and where LINQ takes over | | [read-rows-by-index.md](read-rows-by-index.md) | row 4 and row 900 000 without decoding what lies between | -| [threads.md](threads.md) | what a session shares, a local file against a remote one, and when parallelism pays | -| [owned-batches.md](owned-batches.md) | batches you keep, hand to another thread and dispose there, and what the copy costs | -| [nested-records.md](nested-records.md) | a struct column as a nested record, and a list column as spans of its elements | -| [statistics-and-pruning.md](statistics-and-pruning.md) | the plan before a scan and its numbers after: which blocks were skipped, and by what | -| [cancel-work.md](cancel-work.md) | how cancellation reaches a scan, a write and a cursor, and where it takes effect | +| [threads.md](threads.md) | what a session shares, local and remote sources, and when parallelism pays | +| [owned-batches.md](owned-batches.md) | batches you keep, hand to another thread and dispose there | +| [nested-records.md](nested-records.md) | struct columns as nested records, and list columns as spans of their elements | +| [statistics-and-pruning.md](statistics-and-pruning.md) | the plan before a scan, its numbers after, and what makes pruning work | +| [cancel-work.md](cancel-work.md) | how cancellation reaches a scan, a write and a cursor | ## Writing | page | what you get | |---|---| -| [write-a-file.md](write-a-file.md) | columns in, one file out: fill the builder's buffers in place, write, complete | +| [write-a-file.md](write-a-file.md) | fill the builder's buffers in place, write, complete | | [write-rows.md](write-rows.md) | write an array or an asynchronous stream of records | | [write-text.md](write-text.md) | text from UTF-8 bytes, from strings, formatted in place, or written straight into the buffer | -| [write-nulls.md](write-nulls.md) | nulls one at a time or as a bitmap, and why a column with none costs nothing | -| [write-lists-and-records.md](write-lists-and-records.md) | a list in one call or element by element, and a nested record | -| [blocks-and-chunks.md](blocks-and-chunks.md) | what a block and a chunk are, and the write sizes that make a file append-friendly | -| [append-and-repair.md](append-and-repair.md) | add rows to a file, give a write up, and read or repair a file whose tail was torn | -| [writer-options.md](writer-options.md) | compression, encoding hints, text bounds, metadata, a pinned identity, and the report that says what the writer chose | -| [choose-encodings.md](choose-encodings.md) | what each profile and hint costs on ten million rows of twenty column shapes, and the storage speeds at which the choice turns | -| [indexes.md](indexes.md) | the indexes you can ask for, what each costs, the budget, and an index the write must not lose | -| [editions.md](editions.md) | target an edition so that older readers, and Vortex Rust, can read what you write | -| [stream-to-an-object.md](stream-to-an-object.md) | write to any `PipeWriter`, a socket or a multipart upload, with its backpressure | -| [copy-a-file.md](copy-a-file.md) | copy, filter, project or re-encode a file, each column decoded once and encoded once | +| [write-nulls.md](write-nulls.md) | nulls one at a time or as a bitmap, and why a column without any costs nothing | +| [write-lists-and-records.md](write-lists-and-records.md) | lists in one call or element by element, and nested records | +| [blocks-and-chunks.md](blocks-and-chunks.md) | what blocks and chunks are, and how to write a file that appends cheaply | +| [append-and-repair.md](append-and-repair.md) | add rows to a file, give a write up, and repair a file whose tail was torn | +| [writer-options.md](writer-options.md) | compression, encoding hints, text bounds, metadata, identity, and the report of what the writer chose | +| [choose-encodings.md](choose-encodings.md) | what each profile and hint costs, and the storage speeds at which the best choice changes | +| [indexes.md](indexes.md) | the indexes you can ask for, what each costs, and the index budget | +| [editions.md](editions.md) | target an edition so that older readers, Vortex Rust among them, can read your files | +| [stream-to-an-object.md](stream-to-an-object.md) | write to any `PipeWriter`, a socket or a multipart upload, with backpressure | +| [copy-a-file.md](copy-a-file.md) | copy, filter, project or re-encode a file, each column decoded and encoded once | ## Records and the tool path | page | what you get | |---|---| -| [records.md](records.md) | what `[VortexRecord]` generates, the attributes that shape a column, and a record written by hand | -| [diagnostics.md](diagnostics.md) | what the analyzers flag, VX1001 to VX1008, and how to fix the code each one points at | -| [untyped-files.md](untyped-files.md) | read a file whose schema you learn at run time: columns by name, filters as text | -| [write-without-a-record.md](write-without-a-record.md) | write from a schema built at run time, through the same builders under string names | +| [records.md](records.md) | what `[VortexRecord]` generates, the attributes that shape a column, and records written by hand | +| [diagnostics.md](diagnostics.md) | the analyzer diagnostics VX1001 to VX1011 and how to fix the code each one points at | +| [untyped-files.md](untyped-files.md) | read a file whose schema you only learn at run time: columns by name, filters as text | +| [write-without-a-record.md](write-without-a-record.md) | write from a schema built at run time, with builders asked for by name | | [vxdump.md](vxdump.md) | look inside a file: schema, layout tree, encodings, segments, statistics, indexes | ## Experimental packages -Both are marked `[Experimental]` on the assembly: every use reports VX0001 for the dataset or -VX0002 for the row encoding, which a project acknowledges before it builds. +Both packages are marked `[Experimental]`. Using them reports VX0001 for the dataset and VX0002 for +the row encoding, which a project acknowledges once before it builds. | page | what you get | |---|---| -| [datasets.md](datasets.md) | a versioned dataset over an object store: create, append, import, scan, commit, delete and update rows, change the schema | -| [dataset-maintenance.md](dataset-maintenance.md) | compact, vacuum and verify, and what each costs in requests | -| [object-store.md](object-store.md) | implement the store seam for the service you use, and open a single file out of one | +| [datasets.md](datasets.md) | a versioned dataset over an object store: append, import, scan, delete and update rows, change the schema | +| [dataset-maintenance.md](dataset-maintenance.md) | compaction, vacuum and verification, and what each costs in requests | +| [object-store.md](object-store.md) | implement the store interface for your service, and open a single file from a store | | [row-keys.md](row-keys.md) | encode a tuple into bytes whose `memcmp` order is the tuple's order | ## Reference | page | what you get | |---|---| -| [errors.md](errors.md) | the exceptions this library raises, what each one means, and which ones are your fault | +| [errors.md](errors.md) | the exceptions the library raises, what each one means, and which ones are your fault | | [limits.md](limits.md) | the caps that protect a reader from a hostile file, and the options for input you do not trust | -| [observability.md](observability.md) | the meter, the activities and the per-scan statistics, and what each number diagnoses | -| [native-aot.md](native-aot.md) | publish an application that uses this library ahead of time | +| [observability.md](observability.md) | metrics, activities, event counters and per-scan statistics, and what each number tells you | +| [native-aot.md](native-aot.md) | publish an application that uses the library ahead of time | | [how-it-works.md](how-it-works.md) | the shape of the library in two pages, and where to read further | -| [benchmarks.md](benchmarks.md) | every figure the bench publishes: what this costs against the Rust implementation, and each kernel against its baseline | -| [benchmarks-x64.md](benchmarks-x64.md) | the sections that compare Vorticity with itself, from an x64 machine, a Zen 4 processor under Windows | -| [benchmarks-duckdb.md](benchmarks-duckdb.md) | our group bys against DuckDB's on the same files, through its Vortex reader and from its own table: where we lead and where we trail | +| [benchmarks.md](benchmarks.md) | every figure the bench publishes: Vorticity against the Rust implementation, and each kernel against its baseline | +| [benchmarks-x64.md](benchmarks-x64.md) | the same measurements on an x64 machine, a Zen 4 processor under Windows, against a Rust built there | +| [benchmarks-duckdb.md](benchmarks-duckdb.md) | our group bys against DuckDB's on the same files, through its Vortex reader and from its own table | diff --git a/docs/guide/aggregates.md b/docs/guide/aggregates.md index ff55d73c..372fb57b 100644 --- a/docs/guide/aggregates.md +++ b/docs/guide/aggregates.md @@ -18,23 +18,28 @@ AggAsync, four answers 1.9 ms min 10.1, max 49.9, 100000 rows, 8 ci ## Several answers in one pass -`AggAsync` takes a lambda over an `Aggregates` and computes its answers in a single pass -over the rows the scan keeps: one answer comes as itself, `await scan.AggAsync(a => a.Count())`, -and several, of any number, go into the record `AggAsync` names, whose members take them -in order, each of its answer's type or of its nullable form. The members of `a` are `Count()`, -`CountDistinct`, `Sum`, `Min`, `Max`, `Average`, `Variance`, `StandardDeviation` and `Aggregate`, -each over a column named as in a filter, and `Count`, `Any` and `All` of a predicate, and `Where`, -which filters the aggregates that follow it ([a filtered group](#a-filtered-group)). `Variance` and -`StandardDeviation` are of the sample, over `n − 1`, and null below two values. Like a filter, the lambda runs once and describes the work; nothing is evaluated per row in -your code. Here the `Where` let the zone maps skip 109 blocks, and the four answers came from the -14 left, of which one had a column brought to the canonical form; the others were read as runs and +`AggAsync` takes a lambda over an `Aggregates` and computes all its answers in a single pass +over the rows the scan keeps. One answer comes back as itself, as in +`await scan.AggAsync(a => a.Count())`. Several answers, any number of them, go into the record named +by `AggAsync`, whose members take them in order. Each member has the type of its answer or +its nullable form. + +The members of `a` are `Count()`, `CountDistinct`, `Sum`, `Min`, `Max`, `Average`, `Variance`, +`StandardDeviation` and `Aggregate`, each over a column named as in a filter. There are also `Count`, +`Any` and `All` of a predicate, and `Where`, which filters the aggregates that follow it (see +[a filtered group](#a-filtered-group)). `Variance` and `StandardDeviation` are the sample statistics, +divided by `n − 1`, and null below two values. + +As with a filter, the lambda runs once and describes the work. Nothing is evaluated per row in your +code. Here the `Where` let the zone maps skip 109 blocks, and the four answers came from the 14 left. +Only one of them had a column brought to its plain form, and the others were read as runs and dictionaries. -For one answer there are the sinks of the scan itself: `CountAsync`, `AnyAsync`, `MinAsync`, +For a single answer, the scan also has its own sinks: `CountAsync`, `AnyAsync`, `MinAsync`, `MaxAsync`, `SumAsync`, `AverageAsync`, `CountDistinctAsync` and `AggregateAsync`. An aggregate runs -block by block on the encoded form, never hands a batch to your code, keeps one state per chunk, -and merges the states at the end; §5.5 of [14-public-api.md](../design/14-public-api.md) tables -what each encoding lets it skip. +block by block on the encoded form, never hands a batch to your code, keeps one state per chunk and +merges the states at the end. [The query design](../design/16-queries.md#92-aggregates-block-by-block) +lists what each encoding lets it skip. ## Group by @@ -59,18 +64,19 @@ public partial record struct CitySpread(string City, double? Mean, WelfordState Lille mean 30.0009 variance 133.6092 ``` -`Welford` is the aggregator of the next section, a mean and a variance in one pass. -`g.Key` is the key, and the other members of `g` are those of `Aggregates`. A selection of one -value, `Select(g => g.Count())`, is an `Aggregation`, enumerated with `await foreach`, a -`long` per group. Several values have no .NET type until a record gives them one: the selection is -an `Aggregation`, and `As()` makes it a `Scan` of the result, whose members take -the values in order. It is read as a file's scan is: as batches, the default, whose columns are -borrowed and allocate nothing per group, or as records through `ToRecordsAsync()`, a record per -group. A state is a column like the others: `WelfordState` is a `[VortexRecord]`, three columns -in one. +`Welford` is the aggregator of [the section below](#an-aggregator-of-your-own): a mean and a +variance in one pass. `g.Key` is the key, and the other members of `g` are those of `Aggregates`. -A composite key is a tuple of columns, of any length, whose names are the key's, `g.Key.City` and -`g.Key.Day`: +A selection of one value, such as `Select(g => g.Count())`, is an `Aggregation` that you +enumerate with `await foreach`, one `long` per group. Several values have no .NET type until a record +gives them one. The selection is then an `Aggregation`, and `As()` turns it into a +`Scan` of the result, whose members take the values in order. You read it like a file's +scan: as batches by default, whose columns are borrowed and allocate nothing per group, or as +records through `ToRecordsAsync()`, one record per group. A state is a column like any other. +`WelfordState` is a `[VortexRecord]`, three columns in one. + +A composite key is a tuple of columns of any length, and its names become the key's, as in +`g.Key.City` and `g.Key.Day`: ```csharp await foreach (var (city, day, total, state) in file.Scan() @@ -80,7 +86,7 @@ await foreach (var (city, day, total, state) in file.Scan() .ToRecordsAsync()) ``` -The same query in query syntax is the same plan: +The same query in query syntax builds the same plan: ```csharp (from r in file.Scan() @@ -89,20 +95,22 @@ The same query in query syntax is the same plan: .As() ``` -The result's scan is a scan: a `Where` filters its batches, a `GroupBy` aggregates it again, the -daily from the hourly, `CountAsync` and the other single answers read it, and a writer writes it, -`writer.WriteAsync(result)`, a batch at a time. +The result's scan is a scan like any other. A `Where` filters its batches, a `GroupBy` aggregates it +again (the daily totals from the hourly ones), `CountAsync` and the other single answers read it, and +a writer writes it with `writer.WriteAsync(result)`, a batch at a time. -Groups arrive in key order, nulls last, when the key comes from a dictionary or from a column the -statistics say is sorted. Otherwise their order is not defined: on one thread they come in the order -their keys were first met, on more, part by part of the key space, cut the same way by every run of a -query in one process; an order to rely on is asked for with `OrderBy`. A null key is a group of its -own. Memory follows the number of groups, 8 000 for this composite key, not the number of rows. +Without an `OrderBy`, the order of the groups is not defined. On one lane they come in the order +their keys were first met, and on several lanes part by part of the key space, cut the same way by +every run of a query in one process. A group by on a key the statistics say is sorted delivers its +groups in key order as they close ([queries.md](queries.md)). When your code relies on an order, ask +for it with `OrderBy`. A null key is a group of its own. Memory follows the number of groups, 8 000 +for this composite key, not the number of rows. ## A filtered group -`g.Where(r => p)` is the group's rows where `p` is true, with every aggregate: SQL's -`FILTER (WHERE …)`, computed beside the aggregates of the whole group in the same pass. +`g.Where(r => p)` is the subset of the group's rows where `p` is true, with every aggregate +available. It is SQL's `FILTER (WHERE …)`, computed beside the aggregates of the whole group in the +same pass. ```csharp await foreach (CityHeat city in file.Scan() @@ -127,23 +135,24 @@ public partial record struct CityHeat(string City, long Rows, long Hot, double? ... ``` -`g.Count(r => p)` is `g.Where(r => p).Count()`; `g.Any(r => p)` asks whether `p` is true for a row -of the group, `g.All(r => p)` whether it is true for every one. The predicate is in the language of -a scan's `Where`, and a row counts where it is true, not where it is unknown: every temperature -present is above 10, but one in fifty is missing, so `All` is false. Two `Where` keep the rows both -keep. In `AggAsync`, `a.Where`, `a.Count(p)`, `a.Any` and `a.All` do the same over the scan's rows. +`g.Count(r => p)` is shorthand for `g.Where(r => p).Count()`. `g.Any(r => p)` asks whether `p` holds +for some row of the group, and `g.All(r => p)` whether it holds for every row. The predicate uses the +language of a scan's `Where`, and a row counts where it is true, not where it is unknown. Every +temperature present is above 10, but one in fifty is missing, so `All` is false. Two `Where` calls +keep the rows both keep. In `AggAsync`, `a.Where`, `a.Count(p)`, `a.Any` and `a.All` do the same +over the scan's rows. -Each predicate is evaluated once a batch, however many aggregates read it, and its columns join the -pass. On a million requests grouped by endpoint, the count, the failures and their mean latency took -3.4 ms in one pass, and the failures alone, filtered before the group by, 4.6 ms. A filter written -before the group by is still the one to write when the other rows are not wanted and the zone maps -can skip blocks with it. +Each predicate is evaluated once per batch, however many aggregates read it, and its columns join +the pass. On a million requests grouped by endpoint, the count, the failures and their mean latency +took 3.4 ms in one pass, against 4.6 ms for the failures alone filtered before the group by. A +filter before the group by is still the right choice when the other rows are not wanted and the zone +maps can skip blocks with it. ## Chosen rows -`g.First()` and `g.Last()` are the group's first and last row in file order, `g.MinBy(r => e)` and -`g.MaxBy(r => e)` the row holding the smallest and the largest value of `e`. Each is a probe whose -columns are results of the group: +`g.First()` and `g.Last()` are the group's first and last rows in file order. `g.MinBy(r => e)` and +`g.MaxBy(r => e)` are the rows holding the smallest and the largest value of `e`. Each one is a probe +whose columns are results of the group: ```csharp .Select(g => (g.Key, g.First().Celsius, g.Max(r => r.Celsius), g.MaxBy(r => r.Celsius).Day, g.Last().Celsius)) @@ -159,21 +168,21 @@ public partial record struct CityHottest(string City, double? First, double? Hot ... ``` -File order is the order the rows the scan keeps lie in the file, whatever its `OrderBy`, and it is -the same at every degree: of equal values, `MinBy` and `MaxBy` take the first row, and a null or a -NaN is never chosen. Paris's first row has no temperature, so its `First` is null: `First()` is a -row, not the first value. Columns read from one chosen row are one row: `g.MaxBy(r => r.Celsius).Day` -and `.City` come from the same reading. A group with no candidate, every value null, reads null -where the member is nullable and a value type's default where it is not. - -The pass keeps each group's row as its position and the value it is chosen by; the columns read -from the rows are fetched once the groups are known, by position, which decodes only those rows -where the encoding allows: for the groups the result delivers, or, when a `Where` or an `OrderBy` -after the group by compares one, for the groups that reach it. An `OrderByDescending` with a -`Take(10)` reads ten rows whatever the number of groups, and the first and last rows of a day share -one read. A group by that streams fetches them for each batch of groups it closes. A chosen row's -column is a result: a `Where` or an `OrderBy` after the group by compares it, and an aggregate does -not read it. +File order is the order in which the rows the scan keeps lie in the file, whatever its `OrderBy`, and +it is the same at every degree of parallelism. Among equal values, `MinBy` and `MaxBy` take the first +row, and a null or a NaN is never chosen. Paris's first row has no temperature, so its `First` is +null: `First()` is a row, not the first value. Columns read from one chosen row come from the same +row, so `g.MaxBy(r => r.Celsius).Day` and `.City` describe the same reading. A group with no +candidate, every value null, reads null where the member is nullable and the type's default where it +is not. + +The pass keeps each group's chosen row as a position and the value it was chosen by. The columns +read from those rows are fetched once the groups are known, by position, and only those rows are +decoded where the encoding allows it. This happens for the groups the result delivers, or, when a +`Where` or an `OrderBy` after the group by compares one of these columns, for the groups that reach +it. So an `OrderByDescending` with `Take(10)` reads ten rows whatever the number of groups, and the +first and last rows of a day share one read. A chosen row's column is a result: a `Where` or an +`OrderBy` after the group by can compare it, but an aggregate cannot read it. ## Buckets of time and of numbers @@ -198,31 +207,32 @@ GroupBy(StartedAt, by hour) 0.7 ms 24 hours; 5 blocks decoded ... ``` -`Truncate(unit)` is the start of the `Minute`, `Hour`, `Day`, `Week` (from Monday), `Month`, -`Quarter` or `Year` holding an instant or a date, `Truncate(unit, zone)` the same on a zone's -calendar, and `Bucket(width)` the bucket of a `TimeSpan` holding an instant, counted from -1970-01-01. Each is a column of the column's type, valid wherever a column is: a key, a filter, an -aggregate's input, a selection. A `DateTime` is truncated as it is stored, here in UTC, so the day -is the UTC day and its hours, on Paris's calendar, start at 02:00 in Paris. On a zone's calendar each -instant takes its interval's offset: the autumn's repeated hour is two hours, and its day one of 25 -hours. A `DateTimeOffset` read from a zoned column is truncated on its column's calendar. A null -stays null. - -The filter is not evaluated through the function. A comparison of a function with a constant is the -range of the column it stands for, the day `StartedAt >= 2026-09-02 && StartedAt < 2026-09-03`, so -the zone maps, the statistics and a sorted column prune, count and locate it as they would the range -written by hand: the scan decoded the 5 blocks of 13 that hold the day. The instants being sorted, -their hours are too, and the group by streams, an hour at a time. - -When every aggregate is a count, a minimum or a maximum, a block whose rows all fall in one bucket -is not read at all: its zone map gives the bucket, the block's rows and the extremes of its columns, -and the group by decodes only the blocks that straddle a boundary. On a log of a million requests, -one a second, the days with their count, their fastest and their slowest request took 1.3 ms, against -4.1 ms reading every block. Under a filter, a block is left to its zone map when the zone maps prove -the filter keeps it whole. - -On a number, `Bucket(width)` is `⌊v / w⌋ × w`, in integers for an integer column and rounded to the -column's type for a float: +`Truncate(unit)` returns the start of the `Minute`, `Hour`, `Day`, `Week` (starting on Monday), +`Month`, `Quarter` or `Year` that holds an instant or a date. `Truncate(unit, zone)` does the same on +a time zone's calendar. `Bucket(width)` returns the bucket of a `TimeSpan` that holds an instant, +counted from 1970-01-01. Each of these is a column of the column's type, usable anywhere a column is: +as a key, in a filter, as an aggregate's input, in a selection. + +A `DateTime` is truncated as it is stored, here in UTC, so the day is the UTC day, and its hours on +Paris's calendar start at 02:00 in Paris. On a zone's calendar, each instant takes the offset of its +interval, so the hour repeated in autumn appears as two hours, and that day has 25. A +`DateTimeOffset` read from a zoned column is truncated on its column's calendar. A null stays null. + +The filter is not evaluated through the function. A comparison of a function with a constant becomes +the range of the underlying column, here `StartedAt >= 2026-09-02 && StartedAt < 2026-09-03`, so the +zone maps, the statistics and a sorted column prune, count and locate it exactly as they would the +range written by hand. The scan decoded only the 5 blocks out of 13 that hold that day. The instants +are sorted, so their hours are too, and the group by streams an hour at a time. + +When every aggregate is a count, a minimum or a maximum, a block whose rows all fall in one bucket is +not read at all. Its zone map gives the bucket, the row count and the extremes of its columns, and +the group by decodes only the blocks that straddle a boundary. On a log of a million requests, one +per second, the days with their count, their fastest and their slowest request took 1.3 ms, against +4.1 ms when every block is read. Under a filter, a block is answered from its zone map when the zone +maps prove the filter keeps it whole. + +On a number, `Bucket(width)` computes `⌊v / w⌋ × w`, in integers for an integer column and rounded to +the column's type for a float: ```csharp .GroupBy(v => v.DurationMs.Bucket(15_000)) @@ -237,10 +247,10 @@ column's type for a float: from 75000 ms 15000 visits ``` -A comparison with a float's bucket, an `In` over a function and two functions compared are evaluated -through the function, and prune by the zones' bounds taken through it. A decimal's bucket is not -there yet. A function has no key source of its own, so it cannot be the key of a scan's `OrderBy` or -of `Keys`. +A comparison with a float's bucket, an `In` over a function and a comparison of two functions are +evaluated through the function, and prune with the zone bounds mapped through it. Buckets of a +decimal column are not supported yet and throw `NotSupportedException`. A function has no key source +of its own, so it cannot be the key of a scan's `OrderBy` or of `Keys`. ## An aggregator of your own @@ -290,16 +300,16 @@ public readonly struct Welford : IEncodedAggregator ``` The sample holds the rest: `StepDictionary`, which counts the rows per code and adds each distinct -value once with its weight, the `Selected` popcount helper, and `WelfordState`, a count, a mean and -a sum of squared deviations with a weighted `Add`. +value once with its weight, the `Selected` popcount helper, and `WelfordState`, a count, a mean and a +sum of squared deviations with a weighted `Add`. -Every member is static, so the scan calls them monomorphised, without a virtual call per value. -`Step` receives a block in canonical form: the values, the validity words and the rows that passed -the filter, and it must honour both. An `IAggregator` has only `Step`, `Seed` and `Merge`, and -forces the decode of the column it reads; an `IEncodedAggregator` adds a step per encoded form, -called with the nulls already folded out of `rows`. `Merge` joins the states of two chunks, which is -what makes the parallel run correct. `T` is the column's storage primitive, exactly: `int` for an -`i32` column, `double` for `f64`; another type is refused with `VortexSchemaException`. +Every member is static, so the scan calls them monomorphised, without a virtual call per value. `Step` +receives a block in its plain form: the values, the validity words and the rows that passed the +filter, and it must honour both. An `IAggregator` only has `Step`, `Seed` and `Merge`, and forces the +decode of the column it reads. An `IEncodedAggregator` adds a step per encoded form, called with the +nulls already removed from `rows`. `Merge` joins the states of two chunks, which is what makes a +parallel run correct. `T` must be exactly the column's storage type, `int` for an `i32` column and +`double` for `f64`, and any other type is refused with `VortexSchemaException`. What the encoded steps are worth, against `CanonicalWelford`, the same fold with `Step` only: @@ -310,11 +320,11 @@ Welford(Day), encoded 0.2 ms 1000000 values, mean 499.5000, varian Welford(Day), canonical 3.8 ms 1000000 values, mean 499.5000, variance 83333.3333, 123 blocks decoded ``` -`Day` is stored as runs: `StepRunEnd` sees 1 121 runs instead of a million values, more than ten -times faster. `Celsius` is a dictionary whose distinct values include the null, and such a block is -handed to `Step` decoded, so the two are equal ([encoded-forms.md](encoded-forms.md)). -`BlocksDecoded` counts the blocks where a column reached the canonical form: every block of -`Celsius`, and none of `Day`, every block going to `StepRunEnd` as runs. +`Day` is stored as runs, so `StepRunEnd` sees 1 121 runs instead of a million values and runs more +than ten times faster. `Celsius` is a dictionary whose distinct values include the null, and such a +block is handed to `Step` decoded, so the two variants cost the same +([encoded-forms.md](encoded-forms.md)). `BlocksDecoded` counts the blocks where a column reached its +plain form: every block of `Celsius`, and none of `Day`. ## On more cores @@ -329,29 +339,29 @@ GroupBy(City, Day), degree 14 6.3 ms 8000 groups, the widest spread Lille Welford(Celsius), degree 14 1.5 ms 980000 values, mean 30.0000, variance 133.2501, 123 blocks decoded ``` -Parallelism is the session's, 1 by default: a library does not take a host's cores without being -asked. With it, chunks aggregate concurrently, one state per group per chunk, and `Merge` joins -them: the same answers, 3 times faster for the composite group by on 14 cores, 4.5 times for the -Welford fold. A group by whose key streams streams on every lane too: ranges of rows are grouped -side by side and each follows the one before it in the order of the rows, so the groups still come -out in key order as they close, the first ones once the first block is grouped, and the memory is -that of the ranges in flight. `ScanOptions.DegreeOfParallelism` overrides the session for one scan -([threads.md](threads.md)). - -The first groups of an order on the key cost what they keep. `OrderBy(g => g.Key).Take(10)` keeps, -on each lane, the ten best keys it has met, trimmed back as others come, and a row whose integer key -lies past them is not grouped at all: a key of a million values is held ten groups a lane at a time. -On a sorted key, `OrderByDescending(g => g.Key).Take(7)` reads the file's chunks from the last one -back and stops once the seventh group is out. A top on an aggregate, -`OrderByDescending(g => g.Count()).Take(10)`, holds every group to the end, since a count is known -only then. - -## Decimals, at any precision - -A decimal column sums exactly, whatever its precision and its row count: the total is kept in 320 -bits, spilled from a 128-bit or 256-bit total only when that would overflow, so a column holding -the largest value of 76 digits and its opposite in turn sums to what it should. The sum is -converted once, at the end: +Parallelism comes from the session and is 1 by default, because a library should not take a host's +cores without being asked. With it, chunks aggregate concurrently, one state per group per chunk, +and `Merge` joins them. The answers are identical, 3 times faster for the composite group by on 14 +cores and 4.5 times for the Welford fold. A group by whose key streams keeps streaming on every lane: +ranges of rows are grouped side by side, each following the one before it in row order, so the +groups still come out in key order as they close, the first ones as soon as the first block is +grouped, and memory stays at the ranges in flight. `ScanOptions.DegreeOfParallelism` overrides the +session for one scan ([threads.md](threads.md)). + +The first groups of an order on the key cost only what they keep. `OrderBy(g => g.Key).Take(10)` +keeps, on each lane, the ten best keys met so far and trims the rest as others arrive, and a row +whose integer key lies beyond them is not grouped at all. A key of a million values is held ten +groups per lane at a time. On a sorted key, `OrderByDescending(g => g.Key).Take(7)` reads the file's +chunks from the last one backwards and stops once the seventh group is out. A top on an aggregate, +such as `OrderByDescending(g => g.Count()).Take(10)`, holds every group until the end, because a +count is only known then. + +## Decimals at any precision + +A decimal column sums exactly, whatever its precision and its row count. The total is kept in 320 +bits, spilled from a 128-bit or 256-bit total only when that would overflow, so a column alternating +the largest 76-digit value and its opposite sums to exactly what it should. The sum is converted +once, at the end: ```csharp decimal total = await file.Scan().SumAsync(i => i.Amount); // decimal(18, 2): as decimal @@ -359,52 +369,46 @@ VortexDecimal wide = await file.Scan().SumAsync(l => l.Balance); // double? mean = await file.Scan().AverageAsync(l => l.Balance); ``` -A sum as `VortexDecimal` carries 38 digits while it fits them, as a SQL sum of a narrower decimal -does, and 76 beyond; one past 76 digits throws `OverflowException`, as a `decimal` sum past 96 bits -does. An `Int128`, a `UInt128` or a `BigInteger` member sums as its own type, the last never -throwing. `MinAsync` and `MaxAsync` order decimals as the numbers they are, the group keys of a -`GroupBy` may be decimals of 256 bits, and every aggregate reads the column where it lies, a -dictionary or run-end block included. +A sum returned as `VortexDecimal` carries 38 digits while it fits them, like a SQL sum of a narrower +decimal, and 76 beyond that. Past 76 digits it throws `OverflowException`, just as a `decimal` sum +does past 96 bits. An `Int128`, a `UInt128` or a `BigInteger` member sums as its own type, and the +`BigInteger` sum never throws. `MinAsync` and `MaxAsync` order decimals as the numbers they are, +group keys may be 256-bit decimals, and every aggregate reads the column where it lies, dictionary +and run-end blocks included. ## Watch out -* **A sum widens what could overflow.** An `sbyte`, `short` or `int` column sums as a `long`, a - `byte`, `ushort` or `uint` one as a `ulong`, a `Half` or a `float` one as a `double`: - `SumAsync(v => v.DurationMs)` over the `int` column of the visits file is a `long`, where it used - to overflow an `int`. A `long`, a `ulong` or a `decimal` column sums as itself, exactly, and - throws `OverflowException` only when the exact total does not fit it. The engine keeps 64 bits a - group where the file statistics prove the rows times the column's largest value fit them, and 128 - otherwise. -* **The file carries no sum.** `MinAsync`, `MaxAsync` and `CountAsync` without a filter answer from - the file statistics; `SumAsync` and `AverageAsync` read the column ([scan-a-table.md](scan-a-table.md)). -* **NaN is skipped** by a float sum, mean, minimum and maximum, and counted as one value by a - distinct count. An aggregation ignores the scan's `OrderBy`. -* **A float sum is the same bits** at every degree, in every order of the rows and however the - data is cut into files or objects, and closer to the exact sum than a plain one: each value is - split into three integer parts on a grid of exponents, summed exactly, and the total is rounded - once. It takes 69 billion values in one group or one scan, and throws `OverflowException` past - them. Grouped row by row, it costs about a fifth more than a plain sum would; over runs and - sorted keys, nothing measurable. -* **Encoded steps need generic instantiation at run time.** Under Native AOT the scan calls `Step` - with the canonical form instead ([native-aot.md](native-aot.md)). -* **The first passes of a process run on code the JIT has not optimized yet.** Tiered compilation - starts every method unoptimized and recompiles the ones called often, so the best of three passes - of a query on one lane takes a fifth to half as long again as with every method optimized from - the start: 12.4 ms against 10.2 for `GroupBy(City)`, 19.2 against 12.7 for `GroupBy(City, Day)`. - A library cannot choose this for its host. The host can: with - `false` in its project, every method is optimized on its - first call, at the cost of a slower start, and in this sample the passes are then those 10.2 and - 12.7 ms, and 4.8 for Welford against 6.7. Under Native AOT, nothing is left to warm. The cost grows with the lanes: on group - bys of a million to ten million keys over 4 to 20 million rows, measured on 2026-10-08, the first - three passes at fourteen lanes took 1.6 to 3.8 times what the same code compiled ahead of time - took (205 to 218 ms against 76 to 78 from the second pass, for ten million keys). Once warm, at one - lane, the tiered code is between 16 % slower and 22 % faster than the native. With tiered - compilation off, the first pass pays for compiling everything, three to five times the native one, - and the second already runs as fast. - -The figures come from the sample on the demonstration file of a million rows, on a machine of 14 -cores: each timing is the best of three passes in a process, and the median of seven processes -(2026-10-09); from one process to the next they vary by a fifth or more. +* A sum widens what could overflow. An `sbyte`, `short` or `int` column sums as a `long`, a `byte`, + `ushort` or `uint` column as a `ulong`, and a `Half` or `float` column as a `double`. So + `SumAsync(v => v.DurationMs)` over the `int` column of the visits file returns a `long`. A `long`, + a `ulong` or a `decimal` column sums as itself, exactly, and throws `OverflowException` only when + the exact total does not fit. Internally the engine keeps 64 bits per group when the file + statistics prove the rows times the column's largest value fit, and 128 bits otherwise. +* The file carries no sum. `MinAsync`, `MaxAsync` and `CountAsync` without a filter are answered + from the file statistics, while `SumAsync` and `AverageAsync` read the column + ([scan-a-table.md](scan-a-table.md)). +* NaN is skipped by a float sum, mean, minimum and maximum, and counts as one value in a distinct + count. An aggregation ignores the scan's `OrderBy`. +* A float sum gives the same bits at every degree, in every order of the rows and however the data + is cut into files or objects, and it is closer to the exact sum than a plain loop. Each value is + split into three integer parts on a grid of exponents, those parts are summed exactly, and the total + is rounded once. It accepts 2^36 values, about 69 billion, in one group or one scan, and throws + `OverflowException` beyond that. Grouped row by row it costs about a fifth more than a plain sum, + and nothing measurable over runs and sorted keys. +* Encoded steps need generic instantiation at run time. Under Native AOT, the scan calls `Step` with + the plain form instead ([native-aot.md](native-aot.md)). +* The first passes of a process run on code the JIT has not optimized yet. Tiered compilation starts + every method unoptimized and recompiles the busy ones, so the best of three passes on one lane is a + fifth to a half slower than with everything optimized from the start: 12.4 ms against 10.2 for + `GroupBy(City)`, 19.2 against 12.7 for `GroupBy(City, Day)`. The gap grows with the lanes. On + group bys of one to ten million keys at fourteen lanes, the first three passes took 1.6 to 3.8 + times as long as the same code compiled ahead of time. The library cannot choose this for its host, + but the host can: `false` optimizes every method on its + first call at the cost of a slower start, and Native AOT leaves nothing to warm up. + +The figures come from the sample on the demonstration file of a million rows, on a 14-core machine. +Each timing is the best of three passes in a process and the median of seven processes. From one +process to the next they vary by a fifth or more. ## Run it diff --git a/docs/guide/append-and-repair.md b/docs/guide/append-and-repair.md index 702838e5..0e20db94 100644 --- a/docs/guide/append-and-repair.md +++ b/docs/guide/append-and-repair.md @@ -18,62 +18,62 @@ appended 5000 rows to 20000: resumed at 16384, RowCount 25000 after the write, t ## Where an append resumes -`AppendAsync` opens the file, keeps its chunks, and positions the writer after its rows. The new -rows always follow the file's last row. What `appender.RowCount` says right after the open is where -the rewrite starts: - -* a file that ends on a block, 16 384 rows here, is continued as it stands: `RowCount` is its row - count; -* a file that does not, 20 000 rows here, has its last chunk read and written again with the new - rows: `RowCount` is 16 384, where that chunk starts. The 3 616 rows of the old tail are the file's - already and you do not write them again; the writer counts them back in at the first write, - flush or completion, so `RowCount` after writing 5 000 rows is 25 000, what the file holds, and - so is `report.RowCount`. - -An append holds the file alone until it completes or is given up: a second append, or a reader, -meanwhile fails its open with an `IOException`. On Linux and macOS the lock is advisory. A process -that does not take it, or a file system shared between machines that does not pass it on, is not -held off, so keep to one writer per file. +`AppendAsync` opens the file, keeps its chunks, and positions the writer after its rows. New rows +always follow the file's last row. What `appender.RowCount` says right after the open is where the +rewrite starts: + +* A file that ends on a block boundary, 16 384 rows here, is continued as it stands, and `RowCount` + is its row count. +* A file that does not, 20 000 rows here, has its last chunk read and written again together with the + new rows, and `RowCount` is 16 384, where that chunk starts. The 3 616 rows of the old tail already + belong to the file, so you do not write them again. The writer counts them back in at the first + write, flush or completion, so `RowCount` after writing 5 000 rows is 25 000, what the file holds, + and so is `report.RowCount`. + +An append holds the file exclusively until it completes or is given up. A second append, or a reader, +fails its open with an `IOException` in the meantime. On Linux and macOS the lock is advisory, so a +process that does not take it, or a file system shared between machines that does not propagate it, +is not held off. Keep to one writer per file. ## What an append costs -An append never truncates while it writes: the rewritten chunk's old bytes stay where they were, +An append never truncates while it writes. The old bytes of the rewritten chunk stay where they were, which is what a reader falls back to when the append is torn. So the question is how much gets -rewritten. Four appends each way, against the same rows written once: +rewritten. Four appends each way, against the same rows written in one go: | | resumes at | after four appends | the same rows written once | |---|---|---|---| -| appends of 8 192 onto 16 384 | 16 384, 24 576, 32 768, 40 960 | 49 152 rows, 101 485 bytes | 77 468 bytes | -| appends of 5 000 onto 20 000 | 16 384, four times | 40 000 rows, 153 621 bytes | 67 892 bytes | +| appends of 8 192 onto 16 384 | 16 384, 24 576, 32 768, 40 960 | 49 152 rows, 101 475 bytes | 77 476 bytes | +| appends of 5 000 onto 20 000 | 16 384, four times | 40 000 rows, 153 611 bytes | 67 900 bytes | -Appends that end on a block rewrite nothing, and the file is 1.31 times the size of the same rows -written once: every version's footer stays in the file, and each append's rows form chunks of their -own. Appends that do not end on a block keep resuming -at 16 384: the tail chunk grows by each append's rows and is rewritten whole every time, with its -old copy left behind, and the file reaches 2.26 times the size. Write appends in multiples of -`BlockRows`, and rewrite a file that has taken many appends ([copy-a-file.md](copy-a-file.md)), -since nothing else reclaims what they leave behind. +Appends that end on a block boundary rewrite nothing, and the file ends up 1.31 times the size of +the same rows written once: every version's footer stays in the file, and each append's rows form +chunks of their own. Appends that do not end on a block boundary keep resuming at 16 384. The tail +chunk grows by each append's rows and is rewritten whole every time, with its old copy left behind, +and the file reaches 2.26 times the size. Write appends in multiples of `BlockRows`, and rewrite a +file that has taken many appends ([copy-a-file.md](copy-a-file.md)), since nothing else reclaims what +they leave behind. ## Giving an append up ``` -an append abandoned after a flush: 103836 bytes on disk before Abandon, 29316 after, 29316 before the append -an append disposed without CompleteAsync: 29316 bytes, 29316 before +an append abandoned after a flush: 103836 bytes on disk before Abandon, 29324 after, 29324 before the append +an append disposed without CompleteAsync: 29324 bytes, 29324 before a created file abandoned: exists False a created file disposed without CompleteAsync: exists False ``` -`Abandon()` on an append truncates the file back to the length it had, flushed bytes included; on a -created file it deletes what it wrote. A created file is written beside its path and renamed over it -by `CompleteAsync`, so a file that was already there stays as it was until then, and after an -abandon. A writer disposed without `CompleteAsync` does the same. Over a -caller's `PipeWriter` there is nothing to delete, and the pipe is completed with an error instead +`Abandon()` on an append truncates the file back to its original length, flushed bytes included. On +a created file it deletes what it wrote. A created file is written next to its path and renamed over +it by `CompleteAsync`, so a file that was already there stays as it was until then, and after an +abandon too. A writer disposed without `CompleteAsync` does the same. Over a caller's `PipeWriter` +there is nothing to delete, so the pipe is completed with an error instead ([stream-to-an-object.md](stream-to-an-object.md)). ## A torn tail -A crash in the middle of an append leaves bytes after the last complete version. The sample makes one -by cutting the last 100 bytes of a completed append: +A crash in the middle of an append leaves bytes after the last complete version. The sample creates +one by cutting the last 100 bytes of a completed append: ```csharp await using (VortexFile file = await VortexFile.OpenAsync(path)) @@ -87,22 +87,22 @@ await using (VortexFile file = await VortexFile.OpenAsync(path)) ``` ``` -16384 rows in 29316 bytes, 24576 after the append in 47029, cut to 46929 -reading the version before the torn append: 16384 rows, valid to 29316 of 46929 - why: Malformed file: the EOF marker's magic is 0x696e6465, expected 'VTXF'. +16384 rows in 29324 bytes, 24576 after the append in 47019, cut to 46919 +reading the version before the torn append: 16384 rows, valid to 29324 of 46919 + why: Malformed file: the EOF marker's magic is 0x08001400, expected 'VTXF'. ``` -The file opens as the version before the append, and everything it answers is that version's: rows, -statistics, indexes. `VortexTornTailPolicy.Refuse` in `VortexOpenOptions.TornTail` throws -`VortexFormatException` instead. An append refuses a torn file, with the same exception and a message +The file opens as the version before the append, and everything it answers comes from that version: +rows, statistics, indexes. `VortexTornTailPolicy.Refuse` in `VortexOpenOptions.TornTail` throws +`VortexFormatException` instead. An append refuses a torn file with the same exception and a message that says what to do. -`ReadPrevious`, the default, looks for that version from the end of the file backwards, a mebibyte at -a time, and tries at most 16 end records on the way. Two consequences. A file whose only footer is -damaged, rather than torn, is read to its first byte before the open fails. And a file whose last -version was damaged opens at the one before without an error: `TornTail` is the only sign. A reader -that must never read an older version without knowing it, or that opens files it did not write, sets -`Refuse`. +`ReadPrevious`, the default, looks for that version by walking backwards from the end of the file, a +mebibyte at a time, trying at most 16 end-of-file records on the way. This has two consequences. A +file whose only footer is damaged, rather than torn, is read all the way to its first byte before the +open fails. And a file whose last version was damaged opens at the previous one without an error, +with `TornTail` as the only sign. A reader that must never read an older version without knowing it, +or that opens files it did not write, should set `Refuse`. ## Repairing @@ -112,30 +112,31 @@ VortexRepairResult repaired = await VortexFileRepair.RepairAsync(path, ct); // ``` ``` -valid length 29316; repaired: truncated True, 46929 -> 29316 +valid length 29324; repaired: truncated True, 46919 -> 29324 the repaired file appends again: 24576 rows ``` -`ValidLengthAsync` finds the longest prefix that opens without changing the file; `RepairAsync` -truncates to it and leaves a valid file alone. After a repair `TornTail` is null and the file appends -like any other. +`ValidLengthAsync` finds the longest prefix that opens, without changing the file. `RepairAsync` +truncates the file to it, and leaves a valid file alone. After a repair, `TornTail` is null and the +file accepts appends like any other. -**A file with no complete version has nothing to fall back to.** One cut short rather than appended -to, a download that stopped or a disk that filled, throws `VortexFormatException` on open, and so do +A file with no complete version has nothing to fall back to. A file cut short rather than appended +to, such as an interrupted download or a full disk, throws `VortexFormatException` on open, and so do `ValidLengthAsync` and `RepairAsync`. ## Watch out -* **Nothing else may hold the file.** An append opens it for writing alone, so a `VortexFile` still - open over the same path makes `AppendAsync` throw `IOException`. -* **The identity is drawn anew by each append**: every version of the bytes has its own. Pin it - with `VortexWriteOptions.Identity` for a reproducible write ([writer-options.md](writer-options.md)). -* **An append takes the file's own settings** when its options are null: the block size, the index - policy, whether it carries statistics, its metadata. The block size is the file's whatever the - options say. A composite key's encoder is not stored, so give it again. -* **An append is not atomic.** A reader of a file being appended to sees the old version or the new - one, never half of each, and a crash leaves the torn tail above. -* [11-write-strategy.md](../design/11-write-strategy.md), section 3.8, is the design of the append. +* Nothing else may hold the file. An append opens it exclusively for writing, so a `VortexFile` still + open on the same path makes `AppendAsync` throw `IOException`. +* Each append draws a new identity, so every version of the bytes has its own. Pin it with + `VortexWriteOptions.Identity` for a reproducible write ([writer-options.md](writer-options.md)). +* An append takes the file's own settings when its options are null: the block size, the index + policy, whether it carries statistics, and its metadata. The block size is always the file's, + whatever the options say. A composite key's encoder is not stored, so pass it again. +* An append is not atomic. A reader that opens the file sees the old version or the new one, never + half of each, and a crash leaves the torn tail described above. +* [The write strategy](../design/11-write-strategy.md#38-appending-to-a-file) describes the design + of the append. ## Run it diff --git a/docs/guide/benchmarks-duckdb.md b/docs/guide/benchmarks-duckdb.md index 8c29f3cd..4ffd5e21 100644 --- a/docs/guide/benchmarks-duckdb.md +++ b/docs/guide/benchmarks-duckdb.md @@ -1,51 +1,54 @@ # Benchmarks against DuckDB -Our group bys against DuckDB's, on the same files: DuckDB reads them through its Vortex extension, -and again from a table of its own it loaded from them first. Both sides answer the same query, and +Our group bys against DuckDB's, on the same files. DuckDB reads them through its Vortex extension, +and again from a table of its own loaded from them beforehand. Both sides answer the same query, and the answers are compared. [benchmarks.md](benchmarks.md) compares Vorticity with Vortex's Rust -implementation; this page has a reference of its own, so that neither blurs the other. +implementation, and this page keeps its own reference so that neither comparison blurs the other. Vortex™ is a trademark of LF Projects, LLC. DuckDB is a trademark of the DuckDB Foundation. Vorticity is an independent implementation, not affiliated with or endorsed by the Vortex project, LF Projects, LLC, DuckDB or the DuckDB Foundation. -* **The two sides.** Ours: the bench's runner (`bench/Vorticity.Benchmarks.Runner`) compiled with - Native AOT, a process a block of runs. DuckDB 1.5.2 (`8a5851971f`) and its `vortex` extension - (`6ea8bd7`), a process a file kept open for the whole session, at the same number of threads - (`SET threads`). -* **Two bases.** Each query is timed twice. Warm: our runner with its file kept mapped, as a session +How the comparison is run: + +* Our side is the bench's runner (`bench/Vorticity.Benchmarks.Runner`) compiled with Native AOT, one + process per block of runs. DuckDB's side is DuckDB 1.5.2 (`8a5851971f`) with its `vortex` + extension (`6ea8bd7`), one process per file, kept open for the whole session, at the same number + of threads (`SET threads`). +* Each query is timed on two bases. Warm pits our runner, with its file kept mapped the way a session keeps its 64 most recent files by default, against DuckDB's own table, loaded from the file before - anything is timed. Cold: our runner mapping the file anew at every run against DuckDB's Vortex + anything is timed. Cold pits our runner mapping the file anew at every run against DuckDB's Vortex reader, which reads the file at every query. A table already in memory against a file mapped anew would compare page faults rather than engines. -* **The same work on both sides.** Each query is a group by whose result is aggregated once more: +* Both sides do the same work. Each query is a group by whose result is aggregated once more: DuckDB's outer query sums every column the group by computes, and the runner sums the same columns - as it reads the groups. The two sums are the answer, and every row of the tables below gave the - same one on both sides, to a part in 10⁹ (a float's sum depends on its order). -* **The files** are the bench's own, written by its generators, canonical, at edition `core2025.10`, - the one the extension reads (it does not know `vortex.zoned`); both sides read the same file, read - once before anything is timed so that its pages are in the cache. 40 million rows of keys at random - of 10³, 10⁶ and 10⁷ values, and the same number of keys spread by a stride, which no span bounds and - every table hashes. And db-benchmark's group by at 10⁷ rows, the benchmark H2O.ai started and DuckDB - Labs has kept since 2023: its data drawn by the laws of its `groupby-datagen.R` from the bench's own - stream, not R's, and the queries our native aggregates cover, q1 to q5, q7 and q10. q6 (a median), - q8 (the two largest of each group) and q9 (a correlation) are not covered, and not run. -* **In turns, in pairs.** A block of DuckDB's runs and a process of ours alternate, DuckDB then us, us - then DuckDB (ABBA), so that a drift of the machine weighs on both. A block is 10 runs of which the - first 3 are dropped, or 5 and 2 once the query passes 100 ms; a side's figure is the median of its - runs. A pair's speedup is the ratio of its two medians, a query's the median of its pairs': 3 pairs, - then more while the 95 % interval of that median is wider than ±5 %, 8 at most. Fixed canary runs - before and after each query, and the machine's busiest process, swap and power, decide whether it - runs again. -* **DuckDB's time** is the latency its JSON profile gives, to the nanosecond, for a query under 20 ms, - and its timer's past that, which counts whole milliseconds: the profile cost 2 % on q10. -* **Two sessions**, one after the other: a figure is published when they agree within 10 %, a third - session deciding otherwise. The tables give the mean of the two that agreed, the wider of their 95 % - intervals and their pairs. -* **Speedup** is DuckDB's time over ours: above 1.00x, Vorticity took less. - -These figures inform; they gate nothing. Each change to the group by is judged against the -engine's own earlier runs ([05-benchmarks.md](../design/05-benchmarks.md)). + as it reads the groups. Those sums are the answer, and every row of the tables below gave the same + one on both sides, to a part in 10⁹ (a float sum depends on its order). +* The files are the bench's own, written by its generators, canonical, at edition `core2025.10`, + which is the one the extension reads (it does not know `vortex.zoned`). Both sides read the same + file, read once before timing so its pages are in the cache. There are 40 million rows with keys + drawn at random from 10³, 10⁶ and 10⁷ values, and the same number of keys spread by a stride, which + no span bounds and every table has to hash. There is also db-benchmark's group by at 10⁷ rows (the + benchmark H2O.ai started and DuckDB Labs has maintained since 2023), with data drawn by the laws of + its `groupby-datagen.R` from the bench's own random stream rather than R's. The queries run are + those our native aggregates cover: q1 to q5, q7 and q10. q6 (a median), q8 (the two largest of each + group) and q9 (a correlation) are not covered and not run. +* Runs alternate in pairs. A block of DuckDB runs and a process of ours take turns, DuckDB then us, + us then DuckDB (ABBA), so any drift of the machine weighs on both. A block is 10 runs with the + first 3 dropped, or 5 runs with 2 dropped once a query passes 100 ms, and a side's figure is the + median of its runs. A pair's speedup is the ratio of its two medians, and a query's speedup is the + median over its pairs: 3 pairs, then more while the 95 % interval of that median is wider than + ±5 %, up to 8. Fixed canary runs before and after each query, plus the machine's busiest process, + swap and power state, decide whether a query runs again. +* DuckDB's time is the latency from its JSON profile, to the nanosecond, for a query under 20 ms, and + its timer's past that, which counts whole milliseconds (the profile cost 2 % on q10). +* Two sessions run one after the other, and a figure is published when they agree within 10 %, + with a third session deciding otherwise. The tables give the mean of the two that agreed, the wider + of their 95 % intervals, and their pairs. +* Speedup is DuckDB's time over ours. Above 1.00x, Vorticity took less. + +These figures inform and gate nothing. Changes to the group by are judged against the engine's own +earlier runs ([05-benchmarks.md](../design/05-benchmarks.md)). ## One thread @@ -123,34 +126,30 @@ Cold: our file mapped anew at every run, DuckDB's Vortex reader. | db q7: max v1 − min v2 by id3 | 10M | 24.4 | 60.0 | 2.47x | 2.36–2.52 | 6 | | db q10: sum v3, count by id1:id6 | 10M | 91.5 | 129 | 1.39x | 1.31–1.42 | 6 | -## Where Vorticity leads, and where it trails - -* **On one thread it leads every query on both bases**: 1.43x to 6.15x over DuckDB's own table, 1.58x - to 5.65x over its reader. db-benchmark's q10, six keys of which three are texts and a group for - nearly every one of 10⁷ rows, leads by the least: 1.43x and 1.58x. -* **On fourteen threads it leads every query on both bases**: 1.18x to 3.50x over DuckDB's own table - and 1.39x to 4.47x over its reader. Integer keys of 10³ to 10⁷ values lead by 2.06x to 3.50x warm, - hashed ones by 1.75x and 1.87x, and db-benchmark's q1 to q7 by 1.35x to 2.56x warm, 1.60x to 3.44x - cold. -* **db-benchmark's q10 leads by the least on fourteen threads**: 1.18x over DuckDB's own table, its - interval 1.15x to 1.20x, and 1.39x over its reader. Its six keys are held as one tuple of their - values, 64 bytes, which the lanes hand to a shared table of groups by parts; DuckDB packs the same - keys into 38 bytes before it groups them, and writes rows nearly all unique straight to its - partitions. It trailed at 0.51x on 2026-10-09, as 10⁷ hashed keys did at 0.96x (1.75x now): the - memory of the groups is now kept from one query to the next rather than made anew, each part of the - shared table is held whole rather than split as it grows, a key is hashed once on its way, and the - keys are written into the result a part at a time. - -Measured on 2026-10-10 on an Apple M4 Pro (14 cores: 10 performance, 4 efficiency) under macOS, at -commit `7734c472`: two sessions one after the other, from 05:45 to 05:57 and from 05:57 to 06:07, and a -third for the one row they did not agree on (10³ keys on fourteen threads, cold, which took 3.85x and -4.64x, then 4.31x). The machine was near rest: the canaries strayed up to 9.8 % and 10.3 % from their -first values, another process held a query back 8 s and 6 s in all, and one row of the first session -ran again. - -Until 2026-10-09 this page timed every DuckDB query, then every one of ours: a perturbation of the -machine weighed on one side unseen, and DuckDB's q10 on fourteen threads read 257 ms where it took 102 -the same day. It also set our runner mapping its file anew against DuckDB's table already in memory. +## Reading the results + +Vorticity leads every query on both bases, on one thread and on fourteen. + +On one thread the lead runs from 1.43x to 6.15x over DuckDB's own table, and from 1.58x to 5.65x +over its Vortex reader. The smallest lead is db-benchmark's q10, with six keys of which three are +text, and a group for nearly every one of its 10⁷ rows. + +On fourteen threads the lead runs from 1.18x to 3.50x over DuckDB's own table, and from 1.39x to +4.47x over its reader. Integer keys of 10³ to 10⁷ values lead by 2.06x to 3.50x warm, hashed keys by +1.75x and 1.87x, and db-benchmark's q1 to q7 by 1.35x to 2.56x warm and 1.60x to 3.44x cold. + +q10 on fourteen threads is the narrowest margin, 1.18x warm (interval 1.15x to 1.20x) and 1.39x +cold. Its six keys are held as one 64-byte tuple of their values, which the lanes hand to a shared +table of groups split into parts. DuckDB packs the same keys into 38 bytes before grouping them, and +writes rows that are nearly all unique straight to its partitions. Narrower tuples are the open lead +for this query. + +The figures were measured on 2026-10-10 on an Apple M4 Pro (14 cores: 10 performance, 4 efficiency) +under macOS, at commit `7734c472`. Two sessions ran back to back, from 05:45 to 05:57 and from 05:57 +to 06:07, and a third settled the one row they disagreed on (10³ keys on fourteen threads, cold, +which gave 3.85x and 4.64x, then 4.31x). The machine was close to idle: the canaries strayed up to +9.8 % and 10.3 % from their first values, another process held queries back 8 s and 6 s in total, +and one row of the first session ran again. ## Run it again @@ -160,11 +159,11 @@ bench/duckdb.sh --set all --threads 1,14 bench/duckdb.sh --publish bench/.runs/duckdb- bench/.runs/duckdb- ``` -`duckdb` on the `PATH` (or `DUCKDB` naming it) with its extension installed (`INSTALL vortex`), as -`cargo` is for the comparison with Rust: a workstation's dependency, never CI's. Each session writes -any file it lacks first, through `bench/Vorticity.Benchmarks.Queries --fixture`, keeps the runner of -the commit it measures (`--runner`, `HEAD` by default), and keeps every run under -`bench/.runs/duckdb-/`; `--publish` builds the tables above from two sessions, or three when two -do not agree. `--set hc` or `--set db` runs one half, `--rows` and `--db-rows` change the sizes, -`--only db-q4,total-k7` keeps the rows named by the runner's scenario. A session took 11 to 17 minutes -here. +This needs `duckdb` on the `PATH` (or `DUCKDB` pointing at it) with its extension installed +(`INSTALL vortex`), the same way `cargo` is needed for the comparison with Rust. It is a +workstation's dependency, never CI's. Each session first writes any file it lacks through +`bench/Vorticity.Benchmarks.Queries --fixture`, keeps the runner of the commit it measures +(`--runner`, `HEAD` by default), and stores every run under `bench/.runs/duckdb-/`. `--publish` +builds the tables above from two sessions, or three when two do not agree. `--set hc` or `--set db` +runs one half, `--rows` and `--db-rows` change the sizes, and `--only db-q4,total-k7` keeps only the +rows named by the runner's scenario. A session took 11 to 17 minutes here. diff --git a/docs/guide/benchmarks-x64.md b/docs/guide/benchmarks-x64.md index 7476d95e..149552b4 100644 --- a/docs/guide/benchmarks-x64.md +++ b/docs/guide/benchmarks-x64.md @@ -1,44 +1,47 @@ # Benchmarks on x64 -[The benchmark page](benchmarks.md) on an x64 machine: an AMD Ryzen 9 7950X (Zen 4, -16 cores and 32 threads, AVX-512) under Windows 11, against the same Vortex Rust 0.86.1 built the -same way for that machine. The instruments, the files, the rows and the rules are that page's own, -section for section; what changes is the processor, the operating system and the file system under -the page cache. [05-benchmarks.md](../design/05-benchmarks.md) says what is compared and how each -instrument measures; [bench/README.md](../../bench/README.md) how to run them. +This is [the benchmark page](benchmarks.md) on an x64 machine: an AMD Ryzen 9 7950X (Zen 4, 16 +cores and 32 threads, AVX-512) under Windows 11, against the same Vortex Rust 0.86.1 built the same +way for that machine. The instruments, files, rows and rules are the same as on that page, section +for section. What changes is the processor, the operating system and the file system under the page +cache. [05-benchmarks.md](../design/05-benchmarks.md) explains what is compared and how each +instrument measures, and [bench/README.md](../../bench/README.md) how to run them. Vortex™ is a trademark of LF Projects, LLC. Vorticity is an independent implementation, not affiliated with or endorsed by the Vortex project or LF Projects, LLC. -**Rust's ratios here are this machine's.** Its `vxbench` is built here, and the ceilings that -`--ratio-check` and `--throughput --check` hold the ratios to are set on the arm64 machine, against -its own build: the in-process axes, the takes and the writes on this page are measured and held to -none. The per-encoding corpus is the generator's current one, 61 files where the arm64 page has 57: -`pco_f32`, `pco_i16`, `pco_u32` and `pco_nullable` are on this page alone. The kernel tables' -speedups are computed from the Mean column of their run. - -* **The two sides.** Vorticity as a Native AOT binary built for the machine's instruction set, - with the workstation garbage collector, where a section says so, and on the JIT otherwise. Vortex - Rust 0.86.1 built as upstream builds its own benchmarks (mimalloc, `-C target-cpu=native`, one - codegen unit, no LTO), through [tools/vxbench-rs](../../tools/vxbench-rs): `target-cpu=native` - gives its compiler this machine's instruction set, AVX-512 included, as .NET's JIT has it. Both - map the file and read it where it lies, decode every value they return to its plain form, a - constant column kept as one value on both, and must return the same rows or the run fails. -* **Rust at its faster setting.** On one core every Rust figure is the faster of its two ways of - splitting a scan, its default and one split per chunk, measured file by file; on all cores it +Rust's ratios on this page belong to this machine. Its `vxbench` is built here, while the ceilings +that `--ratio-check` and `--throughput --check` enforce are set on the arm64 machine against its own +build, so the in-process axes, the takes and the writes on this page are measured but held to no +ceiling. The per-encoding corpus is the generator's current one, with 61 files where the arm64 page +has 57: `pco_f32`, `pco_i16`, `pco_u32` and `pco_nullable` appear on this page only. The kernel +tables' speedups are computed from the Mean column of their run. + +The ground rules: + +* Vorticity runs as a Native AOT binary built for the machine's instruction set, with the + workstation garbage collector, where a section says so, and on the JIT otherwise. Vortex Rust + 0.86.1 is built the way upstream builds its own benchmarks (mimalloc, `-C target-cpu=native`, one + codegen unit, no LTO), through [tools/vxbench-rs](../../tools/vxbench-rs). `target-cpu=native` + gives its compiler this machine's instruction set, AVX-512 included, just as .NET's JIT has it. + Both sides map the file and read it where it lies, decode every value they return to its plain + form (a constant column kept as one value on both), and must return the same rows or the run + fails. +* Rust gets its faster setting. On one core, every Rust figure is the faster of its two ways of + splitting a scan (its default, and one split per chunk), measured file by file. On all cores it keeps its default, as on [the benchmark page](benchmarks.md). -* **Speedup** is Rust's time over Vorticity's: above 1.00x, Vorticity took less. -* **Throughput** is the plain size of the rows returned over the time, as on [the benchmark +* Speedup is Rust's time over Vorticity's. Above 1.00x, Vorticity took less. +* Throughput is the plain size of the rows returned divided by the time, as on [the benchmark page](benchmarks.md). -* **All cores** are the 32 hardware threads, simultaneous multithreading included, where the arm64 - page's are 14 cores of two kinds. -* **The page cache is warm** for every run, on both sides, and every call maps its file anew, on - both sides, as on [the benchmark page](benchmarks.md). +* All cores means the 32 hardware threads, simultaneous multithreading included, where the arm64 + page has 14 cores of two kinds. +* The page cache is warm for every run, and every call maps its file anew, on both sides, as on + [the benchmark page](benchmarks.md). Each section is regenerated on an x64 machine by `dotnet run -c Release --project -bench/Vorticity.Benchmarks --` and the arguments below, which rewrite that section of this page and -leave the others. The Native AOT runner that `--report` times is published first (`dotnet publish -c -Release bench/Vorticity.Benchmarks.Runner`): an older one times older code. +bench/Vorticity.Benchmarks --` with the arguments below, which rewrite that section of this page and +leave the others alone. Publish the Native AOT runner that `--report` times first (`dotnet publish -c +Release bench/Vorticity.Benchmarks.Runner`), since an older runner times older code. | section | arguments | |---|---| @@ -886,12 +889,12 @@ It keeps the writer's choice under every goal on the 9 other columns. ## Kernels, against their baselines -Each hot loop of the library against a baseline, the plain loop it stands in for or the floor of the work it -does, measured by BenchmarkDotNet in one process on one clock, both checked to give the same result -before any is timed. A **Speedup** column is the mean time of the row's baseline, the row whose -`MannWhitney(5%)` cell reads `Baseline`, over the row's own: above 1.00x, the row took less. These -are the fast profile's figures, which tell a direction: `--full` before a class's name runs the -reference profile on that class, the one to quote a small difference from. +Each hot loop of the library is measured against a baseline, either the plain loop it replaces or +the floor of the work it does. BenchmarkDotNet times both in one process on one clock, after +checking that they give the same result. The Speedup column is the mean time of the baseline (the +row whose `MannWhitney(5%)` cell reads `Baseline`) over the row's own, so above 1.00x the row took +less. These figures come from the fast profile and show a direction. To quote a small difference, +run the reference profile on that class with `--full` before its name. ### `FastLanesKernelBenchmarks` @@ -959,13 +962,13 @@ Benchmarks with issues: ## What this does not measure -* **A cold cache.** The page cache is warm for every run, on both sides; upstream's own benchmarks - flush it before each query. -* **Pinned cores.** Windows pins no process, and two hardware threads share each core's execution - units; both sides see all 32. Upstream measures on 94 pinned cores of one kind. -* **Another operating system on this processor.** The file system, the page cache and the thread - scheduler are Windows' here, and macOS' on [the benchmark page](benchmarks.md): a difference - between the two pages is the machine's as much as the processor's. -* **Your data and your machine.** A handful of tables and one machine: a column the compressor likes - less, or a filter the zone maps cannot prune, moves these numbers more than either implementation - does. +* Nothing here runs on a cold cache. The page cache is warm for every run, on both sides, while + upstream's own benchmarks flush it before each query. +* Cores are not pinned. Windows pins no process, and two hardware threads share each core's + execution units, so both sides see all 32. Upstream measures on 94 pinned cores of a single kind. +* The operating system differs between the two pages. The file system, the page cache and the + thread scheduler are Windows' here and macOS' on [the benchmark page](benchmarks.md), so a + difference between the two pages comes from the whole machine as much as from the processor. +* Your data and your machine will differ. This is a handful of tables on one machine, and a column + the compressor likes less, or a filter the zone maps cannot prune, moves these numbers more than + either implementation does. diff --git a/docs/guide/benchmarks.md b/docs/guide/benchmarks.md index ef4148fd..b5b4c99d 100644 --- a/docs/guide/benchmarks.md +++ b/docs/guide/benchmarks.md @@ -1,37 +1,39 @@ # Benchmarks -Every figure the bench publishes, on one page: Vorticity against Vortex's Rust implementation on -the same files, and Vorticity's kernels against their baselines. Each section is written by -the instrument that measures it and ends with the machine, the build, the commit and the date it was -measured on. [05-benchmarks.md](../design/05-benchmarks.md) says what is compared and how each -instrument measures; [bench/README.md](../../bench/README.md) how to run them. [Benchmarks on +Every figure the bench publishes, on one page: Vorticity against Vortex's Rust implementation on the +same files, and Vorticity's kernels against their baselines. Each section is written by the +instrument that measures it, and ends with the machine, the build, the commit and the date of the +measurement. [05-benchmarks.md](../design/05-benchmarks.md) explains what is compared and how each +instrument measures, and [bench/README.md](../../bench/README.md) how to run them. [Benchmarks on x64](benchmarks-x64.md) is the same page from an x64 machine. Vortex™ is a trademark of LF Projects, LLC. Vorticity is an independent implementation, not affiliated with or endorsed by the Vortex project or LF Projects, LLC. -* **The two sides.** Vorticity as a Native AOT binary built for the machine's instruction set, - with the workstation garbage collector, where a section says so, and on the JIT otherwise. Vortex - Rust 0.86.1 built as upstream builds its own benchmarks (mimalloc, `-C target-cpu=native`, one +The ground rules for every comparison on this page: + +* Vorticity runs as a Native AOT binary built for the machine's instruction set, with the + workstation garbage collector, where a section says so, and on the JIT otherwise. Vortex Rust + 0.86.1 is built the way upstream builds its own benchmarks (mimalloc, `-C target-cpu=native`, one codegen unit, no LTO), through [tools/vxbench-rs](../../tools/vxbench-rs). -* **The same work on both sides.** Both open the file and map it anew at every call, Rust through - `open_buffer` over a `memmap2` mapping, and read it where it lies; neither keeps a segment cache. - Both decode every value they return to its plain form, a constant column kept as one value and a - length on both; a write hands its bytes to a sink that counts them and keeps none, on both. Both - must return the same rows, or the run fails. -* **Rust at its faster setting.** On one core every Rust figure is the faster of its two ways of - splitting a scan, its default and one split per chunk, measured file by file: its default decodes - a chunk again for every 100,000 rows of it, which the per-encoding files, one chunk of a million - rows each, made ten times. On all cores it keeps its default - ([05-benchmarks.md](../design/05-benchmarks.md) §1). -* **Speedup** is Rust's time over Vorticity's: above 1.00x, Vorticity took less. -* **Throughput** is the plain size of the rows returned over the time. Plain is Arrow's layout: 8 - bytes an `i64` or an `f64`, a string's bytes and a 4-byte offset, a bit a boolean and a bit a row - for a column with nulls. -* **The page cache is warm** for every run, on both sides. - -Each section is regenerated by `dotnet run -c Release --project bench/Vorticity.Benchmarks --` -and the arguments below, which rewrite that section of this page and leave the others: +* Both sides do the same work. Both open the file and map it anew at every call (Rust through + `open_buffer` over a `memmap2` mapping) and read it where it lies, and neither keeps a segment + cache. Both decode every value they return to its plain form, with a constant column kept as one + value and a length on both. A write hands its bytes to a sink that counts them and keeps none, on + both. Both must return the same rows, or the run fails. +* Rust gets its faster setting. On one core, every Rust figure is the faster of its two ways of + splitting a scan (its default, and one split per chunk), measured file by file. Its default + decodes a chunk again for every 100,000 rows of it, which on the per-encoding files, one chunk of a + million rows each, meant ten decodes. On all cores it keeps its default (see [the comparison + rules](../design/05-benchmarks.md#1-the-comparison)). +* Speedup is Rust's time over Vorticity's. Above 1.00x, Vorticity took less. +* Throughput is the plain size of the rows returned divided by the time. Plain means Arrow's layout: + 8 bytes for an `i64` or an `f64`, a string's bytes plus a 4-byte offset, a bit for a boolean, and a + bit per row for a column with nulls. +* The page cache is warm for every run, on both sides. + +Each section is regenerated by `dotnet run -c Release --project bench/Vorticity.Benchmarks --` with +the arguments below, which rewrite that section of this page and leave the others alone: | section | arguments | |---|---| @@ -44,7 +46,8 @@ and the arguments below, which rewrite that section of this page and leave the o ## On x64 [Benchmarks on x64](benchmarks-x64.md) is this page's twin on an AMD Ryzen 9 7950X (Zen 4, -AVX-512) under Windows 11, every section measured there, against a Rust built on that machine. +AVX-512) under Windows 11, with every section measured there against a Rust build made on that +machine. ## Reading and writing a table, process against process @@ -871,12 +874,12 @@ It keeps the writer's choice under every goal on the 10 other columns. ## Kernels, against their baselines -Each hot loop of the library against a baseline, the plain loop it stands in for or the floor of the work it -does, measured by BenchmarkDotNet in one process on one clock, both checked to give the same result -before any is timed. A **Speedup** column is the mean time of the row's baseline, the row whose -`MannWhitney(5%)` cell reads `Baseline`, over the row's own: above 1.00x, the row took less. These -are the fast profile's figures, which tell a direction: `--full` before a class's name runs the -reference profile on that class, the one to quote a small difference from. +Each hot loop of the library is measured against a baseline, either the plain loop it replaces or +the floor of the work it does. BenchmarkDotNet times both in one process on one clock, after +checking that they give the same result. The Speedup column is the mean time of the baseline (the +row whose `MannWhitney(5%)` cell reads `Baseline`) over the row's own, so above 1.00x the row took +less. These figures come from the fast profile and show a direction. To quote a small difference, +run the reference profile on that class with `--full` before its name. ### `FastLanesKernelBenchmarks` @@ -939,18 +942,17 @@ reference profile on that class, the one to quote a small difference from. ## What this does not measure -* **A cold cache.** The page cache is warm for every run, on both sides; upstream's own benchmarks - flush it before each query. -* **Pinned cores.** macOS pins no process, and an Apple M-series processor has cores of two kinds; - both sides see all of them. Upstream measures on 94 pinned cores of one kind. -* **Rust's default way of reading a file.** Upstream's `open_path` reads the segments it needs - into buffers of its own; both sides here map the file instead, Vorticity's default. On a column - stored in its plain form, the mapping spares Rust a copy; on a compressed one it costs Rust page - faults where the copy cost it a read, which on the per-encoding corpus makes its scans of `fsst` - and `alp` 7 and 10 % slower, and moves the others by less. -* **x64 on this page.** Every figure here is from one arm64 machine; [Benchmarks on - x64](benchmarks-x64.md) has the same sections from one x64 machine. - CI builds and tests on x64 and arm64 and runs no benchmark. -* **Your data and your machine.** A handful of tables and one machine: a column the compressor likes - less, or a filter the zone maps cannot prune, moves these numbers more than either implementation - does. +* Nothing here runs on a cold cache. The page cache is warm for every run, on both sides, while + upstream's own benchmarks flush it before each query. +* Cores are not pinned. macOS pins no process, and an Apple M-series processor has two kinds of + cores, so both sides see all of them. Upstream measures on 94 pinned cores of a single kind. +* Rust does not read files its default way. Upstream's `open_path` reads the segments it needs into + buffers of its own, whereas both sides here map the file, which is Vorticity's default. On a column + stored in plain form the mapping spares Rust a copy. On a compressed one it costs Rust page faults + where the copy cost it a read, which makes its scans of `fsst` and `alp` on the per-encoding corpus + 7 and 10 % slower, and moves the others less. +* Every figure on this page comes from one arm64 machine. [Benchmarks on x64](benchmarks-x64.md) has + the same sections from one x64 machine. CI builds and tests on x64 and arm64 but runs no benchmark. +* Your data and your machine will differ. This is a handful of tables on one machine, and a column + the compressor likes less, or a filter the zone maps cannot prune, moves these numbers more than + either implementation does. diff --git a/docs/guide/blocks-and-chunks.md b/docs/guide/blocks-and-chunks.md index dbaff026..dd94493d 100644 --- a/docs/guide/blocks-and-chunks.md +++ b/docs/guide/blocks-and-chunks.md @@ -22,41 +22,45 @@ wrote 5 000 rows: RowCount 5000, UnflushedBytes 140625, nothing at the path yet flushed: RowCount 5000, UnflushedBytes 140625, nothing at the path yet wrote 5 000 more: RowCount 10000, UnflushedBytes 281250, nothing at the path yet flushed: RowCount 10000, UnflushedBytes 50850, nothing at the path yet -completed: chunk rows 8192, 1808, 22708 bytes +completed: chunk rows 8192, 1808, 22716 bytes ``` -The first flush leaves `UnflushedBytes` as it was; the second hands out a block of 8 192 rows, and -only the 1 808 after it stay pending. The flushed bytes go to a file written beside the path, which -`CompleteAsync` renames over it, so the path shows nothing, or the file it held before, until the -file is whole ([append-and-repair.md](append-and-repair.md)). +The first flush leaves `UnflushedBytes` unchanged. The second hands out a block of 8 192 rows, and +only the 1 808 rows after it stay pending. Flushed bytes go to a file written next to the path, which +`CompleteAsync` renames over it, so the path shows nothing, or the file it held before, until the new +file is complete ([append-and-repair.md](append-and-repair.md)). ## Two units -* **A block** is `BlockRows` rows, 8 192 by default. It is the unit of pruning, of a take, and of - the batches of a filtered or ordered scan; a scan that only reads delivers up to sixteen blocks - per batch, never more than a chunk ([scan-a-table.md](scan-a-table.md)), and so does a filter - over the blocks its zone maps prove whole, every row of which passes. -* **A chunk** is a run of whole blocks encoded together: each column chooses its encoding once per - chunk. A chunk is also the seam of an append, which keeps the chunks before it and rewrites the - last one when the file does not end on a block. +A block is `BlockRows` rows, 8 192 by default. It is the unit of pruning, of a take, and of the +batches of a filtered or ordered scan. A scan that only reads delivers up to sixteen blocks per batch, +never more than a chunk ([scan-a-table.md](scan-a-table.md)), and a filter does the same over the +blocks its zone maps prove whole. + +A chunk is a run of whole blocks encoded together, and each column chooses its encoding once per +chunk. A chunk is also the seam of an append, which keeps the chunks before it and rewrites the last +one when the file does not end on a block boundary. ## What each call does -* `WriteAsync` accepts any number of rows and keeps them in the builder's buffers. Once they hold a - chunk, the whole blocks among them are encoded; the rest stays pending. It writes nothing to the - file. By default the writer sizes a chunk by what its rows hold: as many whole blocks as keep the - widest column within a megabyte, between one block and 128, the rows within 64 MiB. A chunk is - what a read fetches of a column, so that bounds what a selective read brings in, and at eight bytes - a value it is sixteen blocks, enough for a scan to pay the cost of a chunk seldom. - `ChunkTargetBytes` sets a fixed target in bytes of all the columns instead. - `ColumnChunkTargetBytes` gives one column a target of its own: its rows wait across the file's - chunks until they hold that many bytes, and go out as one chunk of that column, which a dictionary - then pays for once. The other columns keep the file's chunks, and so what a selective read of them - fetches. `WriteReport.ChunkRowsOf` gives each column's chunks. -* `FlushAsync` seals every whole block pending into a chunk and hands the encoded chunks to the file. - A partial block stays pending: the first flush above, with 5 000 rows, wrote nothing. -* `CompleteAsync` seals the whole blocks, writes what remains as the tail, a chunk shorter than a - block, and then the statistics, zone maps and footer: 8 192 rows and 1 808 above. +`WriteAsync` accepts any number of rows and keeps them in the builder's buffers. Once they fill a +chunk, the whole blocks among them are encoded and the rest stays pending. It writes nothing to the +file. By default the writer sizes a chunk by what its rows hold: as many whole blocks as keep the +widest column within a megabyte, between one block and 128, with all columns within 64 MiB. A chunk +is what a read fetches of a column, so this bounds what a selective read brings in, and at eight +bytes per value it makes sixteen blocks, enough for a scan to pay the fixed cost of a chunk rarely. + +`ChunkTargetBytes` sets a fixed target in bytes for all the columns instead. +`ColumnChunkTargetBytes` gives a single column a target of its own: its rows wait across the file's +chunks until they reach that many bytes, then go out as one chunk of that column, so a dictionary is +paid for once. The other columns keep the file's chunks, and with them what a selective read fetches. +`WriteReport.ChunkRowsOf` gives each column's chunks. + +`FlushAsync` seals every pending whole block into a chunk and hands the encoded chunks to the file. A +partial block stays pending, which is why the first flush above, with 5 000 rows, wrote nothing. + +`CompleteAsync` seals the whole blocks, writes what remains as the tail (a chunk shorter than a +block), then the statistics, the zone maps and the footer: 8 192 rows and 1 808 above. ## What makes a file append-friendly @@ -65,46 +69,48 @@ it would resume: | written as | chunks | bytes | an append resumes at | |---|---|---|---| -| 98 304 rows, writes of 8 192 | 2 (65 536, 32 768) | 154 612 | 98 304 of 98 304 | -| 98 304 rows, writes of 8 192, a flush after each | 12 (8 192 x12) | 177 628 | 98 304 of 98 304 | -| 98 304 rows, one write | 2 (65 536, 32 768) | 154 612 | 98 304 of 98 304 | -| 100 000 rows, writes of 5 000 | 3 (65 536, 32 768, 1 696) | 160 036 | 98 304 of 100 000 | -| 100 000 rows, writes of 5 000, a flush after each | 13 (8 192 x12, 1 696) | 183 052 | 98 304 of 100 000 | -| 98 304 rows, `ChunkTargetBytes` 64 KiB | 12 (8 192 x12) | 177 628 | 98 304 of 98 304 | -| 98 304 rows, `ChunkTargetBytes` 16 MiB | 1 (98 304) | 152 452 | 98 304 of 98 304 | -| 98 304 rows, `BlockRows` 1 024 | 2 (65 536, 32 768) | 159 788 | 98 304 of 98 304 | +| 98 304 rows, writes of 8 192 | 2 (65 536, 32 768) | 154 620 | 98 304 of 98 304 | +| 98 304 rows, writes of 8 192, a flush after each | 12 (8 192 x12) | 177 636 | 98 304 of 98 304 | +| 98 304 rows, one write | 2 (65 536, 32 768) | 154 620 | 98 304 of 98 304 | +| 100 000 rows, writes of 5 000 | 3 (65 536, 32 768, 1 696) | 160 044 | 98 304 of 100 000 | +| 100 000 rows, writes of 5 000, a flush after each | 13 (8 192 x12, 1 696) | 183 060 | 98 304 of 100 000 | +| 98 304 rows, `ChunkTargetBytes` 64 KiB | 12 (8 192 x12) | 177 636 | 98 304 of 98 304 | +| 98 304 rows, `ChunkTargetBytes` 16 MiB | 1 (98 304) | 152 460 | 98 304 of 98 304 | +| 98 304 rows, `BlockRows` 1 024 | 2 (65 536, 32 768) | 159 796 | 98 304 of 98 304 | What decides where an append resumes is the file's row count, not how the rows arrived. A file of -98 304 rows ends on a block, and an append continues after it without rewriting anything. A file of -100 000 rows ends with a tail of 1 696 rows, and an append rewrites that tail chunk: it resumes at -98 304. Both are correct files. [append-and-repair.md](append-and-repair.md) measures what repeated -appends cost each way. +98 304 rows ends on a block boundary, so an append continues after it without rewriting anything. A +file of 100 000 rows ends with a tail of 1 696 rows, and an append rewrites that tail chunk and +resumes at 98 304. Both are correct files. [append-and-repair.md](append-and-repair.md) measures what +repeated appends cost in each case. ## What it costs -Chunks are not free. Each one chooses its encodings again and carries its own framing, so a flush -after every block made the file 15 % larger than one write (177 628 bytes against 154 612), and a -16 MiB target, one chunk, made it 1.4 % smaller. Blocks eight times smaller cost 3 % here (159 788 -bytes against 154 612 for the same writes) and buy pruning eight times finer, which -[filter-rows.md](filter-rows.md) is about. Chunks are not free for a reader either: a selective read -fetches the chunk of every column it reads for each block it keeps, which is why the writer bounds -them by the widest column rather than growing them as far as a scan would like. +Chunks are not free. Each one chooses its encodings again and carries its own framing, so flushing +after every block made the file 15 % larger than a single write (177 636 bytes against 154 620), +while a 16 MiB target, a single chunk, made it 1.4 % smaller. Blocks eight times smaller cost 3 % +here (159 796 bytes against 154 620 for the same writes) and buy pruning eight times finer, which is +what [filter-rows.md](filter-rows.md) is about. Chunks are not free for a reader either: a selective +read fetches the chunk of every column it reads for each block it keeps, which is why the writer +bounds chunks by the widest column rather than growing them as large as a scan would like. -A write of one block at a time comes out as one call does: the chunk is sized by the rows' widest -column, here the city's sixteen-byte views, a megabyte of which is eight blocks, and the rows are -cut there however they arrive. +Writing one block at a time gives the same result as one big write. The chunk is sized by the rows' +widest column, here the city's sixteen-byte views, a megabyte of which is eight blocks, and the rows +are cut there however they arrive. ## Watch out -* **A flush is the only I/O before completion**, and it writes nothing while less than a block is +* A flush is the only I/O before completion, and it writes nothing while less than a block is pending. `UnflushedBytes` counts the encoded chunks waiting for the file and the rows waiting for their chunk. -* **Flush when you need the bytes out**, to bound memory or to feed a slow sink +* Flush when you need the bytes out, to bound memory or to feed a slow sink ([stream-to-an-object.md](stream-to-an-object.md)), not after every write. -* **An append keeps the file's block size**, whatever its options say, and chunks its columns - alike: a file whose columns have chunks of their own is rewritten, not appended to. -* [11-write-strategy.md](../design/11-write-strategy.md) is the design of the ingest, the chunking - and the append. +* An append keeps the file's block size, whatever its options say, and chunks its columns the same + way the file does. `AppendAsync` refuses a file whose columns were given chunks of their own + (`ColumnChunkTargetBytes`) with `VortexUnsupportedException`. Rewrite such a file instead + ([copy-a-file.md](copy-a-file.md)). +* [11-write-strategy.md](../design/11-write-strategy.md) describes the design of the ingest, the + chunking and the append. ## Run it diff --git a/docs/guide/cancel-work.md b/docs/guide/cancel-work.md index bc9b2e8d..5bb67379 100644 --- a/docs/guide/cancel-work.md +++ b/docs/guide/cancel-work.md @@ -30,34 +30,35 @@ using (CancellationTokenSource cts = new CancellationTokenSource()) borrowed columns: cancelled after 2 batches ``` -Every call that reads or writes takes a token as its last argument, defaulted. What comes out when -it fires is `OperationCanceledException`; catch that, since `TaskCanceledException` derives from it. +Every call that reads or writes takes a token as its last argument, with a default value. When the +token fires, what comes out is an `OperationCanceledException`. Catch that type, since +`TaskCanceledException` derives from it. ## A scan of borrowed columns -`await foreach` passes no token by itself, and a scan whose batches are borrowed `ref struct` -columns is not an `IAsyncEnumerable`, so the scan carries the token: `WithCancellation(ct)`, as -above, in the builder chain like `With(options)`. `MoveNextAsync` checks it at each batch boundary -and hands it to the reads under it, and the enumerator, which returns the scan's buffers, is -disposed on the way out, cancelled or not. A loop written out by hand passes the token to -`GetAsyncEnumerator(ct)` instead, which takes precedence. +`await foreach` passes no token by itself, and a scan whose batches are borrowed `ref struct` columns +is not an `IAsyncEnumerable`, so the scan carries the token: call `WithCancellation(ct)` in the +builder chain, like `With(options)`. `MoveNextAsync` checks it at each batch boundary and passes it to +the reads underneath. The enumerator, which returns the scan's buffers, is disposed on the way out, +cancelled or not. A loop written out by hand passes the token to `GetAsyncEnumerator(ct)` instead, +which takes precedence. -Breaking out of an `await foreach` is the other clean way to stop: the enumerator is disposed, the +Breaking out of an `await foreach` is the other clean way to stop. The enumerator is disposed, the scan stops, and nothing throws. ## Rows and owned batches -`ToRecordsAsync(ct)` and `ToBatchesAsync(ct)` take the token directly, and `WithCancellation` works -on them too. They check it at a batch boundary: +`ToRecordsAsync(ct)` and `ToBatchesAsync(ct)` take the token directly, and `WithCancellation` works on +them too. They check it at batch boundaries: ``` rows: cancelled after 65536 rows owned batches: cancelled after 3, each disposed by its using ``` -The rows of the batch in hand are still yielded, so the cancellation above, asked at the 10 000th -row, surfaced at the end of the batch that held it, the first, of 65 536 rows. An owned batch -already handed out stays yours to dispose ([owned-batches.md](owned-batches.md)). +The rows of the batch in hand are still yielded. So the cancellation above, requested at the +10 000th row, surfaced at the end of the batch that held it, the first one, of 65 536 rows. An owned +batch already handed out stays yours to dispose ([owned-batches.md](owned-batches.md)). ## A token cancelled before the call @@ -66,23 +67,23 @@ A call that has nothing to read does not look at the token: | call | with a cancelled token | |---|---| | `OpenAsync` | throws | -| `CountAsync`, no filter | **answers**: the row count is in the footer | +| `CountAsync`, no filter | answers, since the row count is in the footer | | `CountAsync`, with a filter | throws | -| `AnyAsync`, no filter | **answers** | -| `MaxAsync` | **answers**: the file's statistics hold it | -| `SumAsync`, with or without a filter | throws: this library's files record no sum | +| `AnyAsync`, no filter | answers | +| `MaxAsync` | answers, since the file's statistics hold it | +| `SumAsync`, with or without a filter | throws, since this library's files record no sum | | `ExplainAsync` | throws | -| `GetIndexesAsync` | **answers**: the directory is already read | +| `GetIndexesAsync` | answers, since the index directory is already read | | borrowed columns, `ToRecordsAsync` | throw | | `Keys(...).OpenAsync`, `KeyCursor.SeekAsync` | throw | -This is not a bug to work around: a call that reads nothing has nothing to abandon. It does mean a -cancelled token is not an assertion that nothing ran; check the token yourself if you need one. +This is not a bug to work around. A call that reads nothing has nothing to abandon. It does mean that +a cancelled token does not guarantee nothing ran, so check the token yourself if you need that. ## A cursor -A cancelled move throws and leaves the cursor unpositioned: `IsValid` says false, and `Key`, `Row` -and `KeyCountAsync` throw `InvalidOperationException` until a move succeeds. Seek again: +A cancelled move throws and leaves the cursor without a position. `IsValid` returns false, and `Key`, +`Row` and `KeyCountAsync` throw `InvalidOperationException` until a move succeeds. Seek again: ``` the cursor after it: valid False @@ -94,22 +95,22 @@ and `KeyCountAsync` throw `InvalidOperationException` until a move succeeds. See | call | what a cancellation leaves | |---|---| -| `WriteAsync` | throws before taking anything when the token is already cancelled: `RowCount` said 50 000 after a cancelled second write of 50 000, and a builder keeps its rows; cancelled while the rows are being written, the rows are taken | +| `WriteAsync` | throws before taking anything when the token is already cancelled. `RowCount` still said 50 000 after a cancelled second write of 50 000 rows, and a builder keeps its rows. Cancelled while the rows are being written, the rows are taken | | `FlushAsync` | throws | | `CompleteAsync` | throws, and the writer is done: a second `CompleteAsync` throws `ObjectDisposedException` | -A writer that is disposed without completing abandons the file. A created file is deleted — it did -not exist afterwards — and an append is truncated back to what the file was: 4 532 bytes and 50 000 -rows before, the same after a cancelled `CompleteAsync` of an append that had flushed 50 000 more. -`Abandon()` says the same explicitly. A file whose tail a crash tore is another case, +A writer disposed without completing abandons its file. A created file is deleted, and an append is +truncated back to what the file was: 4 220 bytes and 50 000 rows before, and the same after a +cancelled `CompleteAsync` of an append that had flushed 50 000 more rows. `Abandon()` does the same +explicitly. A file whose tail was torn by a crash is a different case, covered in [append-and-repair.md](append-and-repair.md). ## Watch out -* **The token is not stored.** One given to `OpenAsync` cancels the open and nothing after it; each +* The token is not stored. A token given to `OpenAsync` cancels the open and nothing after it. Each later call takes its own. -* A scan is single-use, cancelled or not: build another one to start again. -* Cancelling does not dispose the file, the cursor or the writer: `await using` still does. +* A scan is single-use, cancelled or not. Build another one to start again. +* Cancelling does not dispose the file, the cursor or the writer. `await using` still does that. ## Run it diff --git a/docs/guide/choose-encodings.md b/docs/guide/choose-encodings.md index f40a4b18..5d26f2f4 100644 --- a/docs/guide/choose-encodings.md +++ b/docs/guide/choose-encodings.md @@ -1,10 +1,10 @@ # Choose encodings -What each compression profile and hint trades on common column shapes, and where the storage under a -file, or the way a column is read, makes another choice the better one. The measurements are on the -benchmark page: [every profile and hint on twenty column shapes](benchmarks.md#encodings-column-by-column) +What each compression profile and hint trades on common column shapes, and when the storage under a +file, or the way a column is read, makes another choice the better one. The measurements themselves +are on the benchmark page: [every profile and hint on twenty column shapes](benchmarks.md#encodings-column-by-column) and [what the advice picks on them](benchmarks.md#what-the-advice-picks). This page says what they -mean for a choice. +mean for your choice. ```csharp // The default: each chunk of each column takes the smallest encoding that decodes fast. @@ -30,97 +30,99 @@ VortexWriteOptions repeating = new() ## Reading the measurements -Each shape is one column of ten million rows, written under each configuration, then opened and -read: a **scan** decodes every value to its plain form and reads it once, a **take** reads a thousand -rows spread over the file. Each read opens the file anew and maps it again, so a larger file pays for -its pages every time, as a file opened once does. A column chooses its encoding chunk by chunk, so the -volume only scales the times. +Each shape is one column of ten million rows, written under each configuration, then opened and read. +A scan decodes every value to its plain form once, and a take reads a thousand rows spread over the +file. Each read opens the file anew and maps it again, so a larger file pays for its pages every +time, as a file opened once would. A column chooses its encoding chunk by chunk, so the volume only +scales the times. -**Crosses at** is the storage throughput at which a configuration and `Auto` read the column whole -in the same time, counting its bytes at that throughput and then its scan. Below it the smaller file -reads faster end to end; above it the faster decode does. For reference: the page cache delivers -over 10 GB/s, a local NVMe drive 3 to 7, a SATA SSD 0.5, a gigabit link 0.125, and one stream from an -object store about 0.1. +The "crosses at" figure is the storage throughput at which a configuration and `Auto` read the whole +column in the same time, counting the bytes at that throughput plus the decode. Below it the smaller +file reads faster end to end, above it the faster decode wins. For reference, the page cache delivers +over 10 GB/s, a local NVMe drive 3 to 7 GB/s, a SATA SSD 0.5, a gigabit link 0.125, and one stream +from an object store about 0.1. ## Integers -Bit-packing, runs, a progression and a dictionary each scan in about the time of the plain column, -within a fifth either way and a third faster for runs and a progression, the plain column's eight -bytes a value having to be paged in; and they take a row as cheaply. Zstd -frames are the exception, an order of magnitude slower to scan and far slower to take from where the -values do not come in runs: where zstd is not the smallest form, `Auto` never writes it. On -increasing timestamps Pco is the smallest, 1.35 bytes a value against bit-packing's 3.38, and -`Smallest` takes it, for a scan more than ten times as long: worth it only from storage slower than -its crossing, about 300 MB/s. - -A column whose values repeat across the file more than within a chunk, 100 003 values over ten -million rows, takes a dictionary only in larger chunks: at 16 MiB it halves the bytes for the same -scan. `Smallest` writes it with Pco instead, a hundredth of the bytes for a scan ten times as long. -On the nullable integers a `Dictionary` hint saves a few percent of the bytes bit-packing leaves, for -a slower scan. +Bit-packing, runs, sequences and dictionaries all scan in about the time of the plain column, within +a fifth either way, and runs and sequences even a third faster, since the plain column's eight bytes +per value have to be paged in. They also read a single row cheaply. Zstd frames are the exception: +an order of magnitude slower to scan, and far slower to take from unless the values come in runs. So +`Auto` never writes zstd where it is not the smallest form. + +On increasing timestamps, pco is the smallest, 1.35 bytes per value against 3.38 for bit-packing, +and `Smallest` takes it, for a scan more than ten times as long. That is only worth it from storage +slower than its crossing point, about 300 MB/s. + +A column whose values repeat across the file more than within a chunk, here 100 003 values over ten +million rows, only takes a dictionary in larger chunks: at 16 MiB it halves the bytes for the same +scan. `Smallest` writes it with pco instead, at a hundredth of the bytes for a scan ten times as long. +On nullable integers, a `Dictionary` hint saves a few percent of the bytes bit-packing leaves, for a +slower scan. ## Floating point -Prices take ALP, and a few distinct values a dictionary, each within a few times the plain scan. -Uniform doubles take ALP-RD, which the plain form beats end to end from storage faster than about -2 GB/s. +Prices take ALP, and columns with few distinct values take a dictionary, each scanning within a few +times the plain form. Uniform doubles take ALP-RD, which the plain form beats end to end from storage +faster than about 2 GB/s. -The repeating column is the one to look at twice. Its values come back every 100 003 rows, about as -many as a chunk of the default size holds: a dictionary would store nearly every value once per -chunk, so zstd, which finds the repeats anyway, is the smallest. Chunks of 16 MiB hold enough repeats -for the dictionary to pay: a scan and a take then cost a small fraction of the frames', for a file a -third larger. +The repeating column deserves a second look. Its values come back every 100 003 rows, about as many +as a default chunk holds, so a dictionary would store nearly every value once per chunk. Zstd finds +the repeats anyway and is the smallest. In chunks of 16 MiB there are enough repeats for the +dictionary to pay: a scan and a take then cost a small fraction of what the zstd frames cost, for a +file a third larger. ## Text Text is where the choice matters most. On values that do not repeat, `Auto` writes zstd frames, the -smallest form, and FSST then scans in a fraction of their time; a take costs the frames about what a -scan costs them, because each row it wants sits in a frame that is inflated whole, where FSST reads -a row at a time. FSST is the larger file here: from storage faster than its crossing, under 100 MB/s -on the UUIDs and a few hundred on the log lines, it is also the faster file to scan. A column read by -row, or from a local drive, wants the `Fsst` hint. +smallest form, and FSST scans in a fraction of their time. A take costs the frames about as much as a +scan, because each row it wants sits in a frame that must be inflated whole, while FSST reads one row +at a time. FSST makes the larger file here, but from storage faster than its crossing point, about +115 MB/s on the UUIDs and a few hundred MB/s on the log lines, it is also the faster file to scan. A +column read by row, or from a local drive, wants the `Fsst` hint. The ids show the other lever. A dictionary of 10 000 entries stored again in every chunk is most of -what the column costs to read; in chunks of 16 MiB it is stored far fewer times, and the column is +what the column costs to read. In chunks of 16 MiB it is stored far fewer times, and the column gets smaller, faster to scan and faster to write. ## Booleans -Booleans half true stay a bitmap. Rare trues take runs, smaller than the bitmap and slower to scan: -from anything but slow storage the bitmap, which `None` keeps, reads faster. +Booleans that are half true stay a bitmap. Rare trues take runs, smaller than the bitmap but slower to +scan, so from anything but slow storage the bitmap, which `None` keeps, reads faster. ## The profiles side by side `Fastest` chooses what `Auto` chooses on these shapes, and writes no index. `Smallest` tries every -scheme on every chunk and writes several times slower than `Auto`, up to fifteen, for a smaller file -where one exists: Pco on the timestamps and the repeating integers, a chunk here and there. `None` -writes the plain form, the largest file every time, 8 bytes a value for a number and as many as the -text holds; it scans fastest only where decoding costs more than paging those bytes in, on most -doubles and on text that does not repeat, about as fast as the packed integers, and slower than a -dictionary of a few values. The benchmark page counts them under -[the encodings, column by column](benchmarks.md#encodings-column-by-column). +scheme on every chunk and writes several times slower than `Auto`, up to about eighteen times, for a +smaller file where one exists: pco on the timestamps and the repeating integers, and a chunk here and +there elsewhere. `None` writes the plain form, the largest file every time, eight bytes per number and +as many bytes as the text holds. It scans fastest only where decoding costs more than paging those +bytes in, which is the case for most doubles and for text that does not repeat. It scans about as +fast as packed integers, and slower than a dictionary of a few values. The benchmark page counts them +under [the encodings, column by column](benchmarks.md#encodings-column-by-column). ## What to steer, and with what -* **Where the bytes come from.** Local drives and the page cache favour `Auto`, with the `Fsst` - hint on text that does not repeat. An object store read one stream at a time favours `Smallest`. -* **How a column is read.** A column read by row, through a take, a key or a join, wants neither - zstd nor a large dictionary: `Fsst` for its text, and chunks of the default size. -* **How often values repeat, against the chunk.** A column whose values come back across the file - more often than within a chunk takes a dictionary only with larger chunks. `ColumnChunkTargetBytes` - sets them for that column alone, and the others keep theirs, which is what a selective read of - them fetches; `ChunkTargetBytes` sets them for the whole file +* The storage the bytes come from matters. Local drives and the page cache favour `Auto`, with the + `Fsst` hint on text that does not repeat, while an object store read one stream at a time favours + `Smallest`. +* So does the way a column is read. A column read by row, through a take, a key or a join, wants + neither zstd nor a large dictionary: use `Fsst` for its text, and chunks of the default size. +* So does how often values repeat compared with the chunk. A column whose values come back across + the file more often than within a chunk only takes a dictionary with larger chunks. + `ColumnChunkTargetBytes` sets them for that column alone while the others keep theirs, which is + what a selective read of the other columns fetches. `ChunkTargetBytes` sets them for the whole file ([blocks-and-chunks.md](blocks-and-chunks.md)). -* **What a hint cannot do.** A hint is tried first and falls back when it does not apply: a - `Dictionary` hint on the repeating floats still gives zstd, because a chunk holds too few repeats - for a dictionary to pay ([writer-options.md](writer-options.md)). -* **All of it at once, on your data**: the advice below measures these levers for your columns and - your reads, and returns the options that pull them. +* A hint cannot force what does not pay. It is tried first and falls back when it does not apply: a + `Dictionary` hint on the repeating floats still gives zstd, because a chunk holds too few repeats for + a dictionary to pay ([writer-options.md](writer-options.md)). +* All of it at once, on your data: the advice below measures these levers for your columns and your + reads, and returns the options that set them. ## Let your data choose The benchmark page measures twenty shapes on one machine. `VortexSession.AdviseAsync` makes the same -measurements on your data and your machine, and ranks every way to write each column for the reads +measurements on your data and your machine, and ranks every way of writing each column for the reads you describe: ```csharp @@ -139,64 +141,67 @@ foreach (ColumnEncodingAdvice column in advice.Columns) VortexWriteOptions options = advice.ToWriteOptions(); ``` -Rows in memory are advised on the same way, `session.AdviseAsync(events)`, which writes only -the rows it samples. Per column of numbers, booleans, text or binary, the advice: - -* **samples** a million rows, `SampleRows`, in eight windows spread over the data; -* **writes** them under the writer's own choice and under each hint the column's kind takes, and - at 4 and 16 MiB chunks when its values come back across the data more than within a chunk, on - rows enough for whole chunks of that size; -* **reads** each back from memory: a full scan, and single rows spread over its chunks; -* **ranks** them by the time a scan of the data would take: the decode, the bytes at - `StorageBytesPerSecond`, and for each lookup its decode and the chunk it reads. `Objective.Size` - ranks by bytes alone, under `Smallest`. The writer's own choice stands unless another is 5 % - cheaper, and a larger chunk is taken only when it saves the column 5 %: the writer gives it to - that column alone. - -`Reason` says why in one sentence with the numbers. Each candidate carries its bytes, its scan and -lookup times, its cost, and the throughput at which it and the recommended one cross. -`ToWriteOptions` returns the hints, each column's chunk target and the profile, over a baseline of -yours. +Rows in memory are advised on the same way, with `session.AdviseAsync(events)`, which only +writes the rows it samples. For each column of numbers, booleans, text or binary, the advice: + +* samples about a million rows (`SampleRows`, 1 048 576 by default), in eight windows spread over the + data, +* writes them under the writer's own choice and under each hint the column's kind accepts, and also + with 4 MiB and 16 MiB chunks when its values come back across the data more than within a chunk, +* reads each version back from memory, as a full scan and as single rows spread over its chunks, +* ranks them by the time a scan of the data would take: the decode, plus the bytes at + `StorageBytesPerSecond`, plus for each lookup its decode and the chunk it reads. Setting + `Objective` to `EncodingObjective.Size` ranks by bytes alone, under `Smallest`. The writer's own + choice stands unless another is at least 5 % cheaper, and a larger chunk is taken only when it saves + the column 5 %, in which case the writer gives it to that column alone. + +`Reason` explains the choice in one sentence with the numbers. Each candidate carries its bytes, its +scan and lookup times, its cost, and the throughput at which it and the recommended one cross. +`ToWriteOptions` returns the hints, each column's chunk target and the profile, on top of a baseline +you provide. [What the advice picks](benchmarks.md#what-the-advice-picks) on the benchmark page's twenty shapes -falls where the tables' crossings put it, and compares every candidate with every other, where the -tables compare each with `Auto`. So it can find the plain form ahead of FSST for text scanned from -fast storage, the plain form being larger and faster to decode; under the bytes alone, a dictionary -in 16 MiB chunks for the columns that repeat across the file; and for rows read one at a time, zstd, -which reaches a row by inflating only the frame that holds it, or OnPair on text. - -* **It is a measurement.** A column costs seconds: run it once for a kind of data, keep the - options, and run it again when the data or the machine changes. Under the JIT, it first waits for - the runtime to finish compiling the decoders it times, up to 3 s a column; a process whose other - threads keep compiling is measured on the code it runs then. -* **The decode is timed on one thread.** A scan decoding on several shares the storage between - them: give `StorageBytesPerSecond` divided by the threads. -* **A chunk target is the column's.** A column that wants 16 MiB chunks gets them alone, through - `ColumnChunkTargetBytes`, and a selective read of that column pays for them in larger reads. +matches where the tables' crossing points put it. Since it compares every candidate with every other, +while the tables compare each one with `Auto`, it can find the plain form ahead of FSST for text +scanned from fast storage, the plain form being larger but faster to decode. Under the bytes alone, it +finds a dictionary in 16 MiB chunks for the columns that repeat across the file, and for rows read one +at a time it finds zstd, which reaches a row by inflating only the frame that holds it, or OnPair on +text. + +Keep in mind that the advice is a measurement: + +* A column takes seconds to measure. Run it once for a kind of data, keep the options, and run it + again when the data or the machine changes. Under the JIT it first waits, up to 3 s per column, for + the runtime to finish compiling the decoders it times, so a process whose other threads keep + compiling is measured on the code it actually runs. +* The decode is timed on one thread. A scan decoding on several threads shares the storage between + them, so pass `StorageBytesPerSecond` divided by the number of threads. +* A chunk target belongs to its column. A column that wants 16 MiB chunks gets them alone, through + `ColumnChunkTargetBytes`, and a selective read of that column pays for them with larger reads. ## Watch out -* **The benchmark page's times are one machine's**, with the file in the page cache and mapped anew - by each read. The bytes carry to any machine, and so do the ratios between the scans; where a - choice crosses scales with the machine's decode speed. A process that opens the same file again - keeps its mapping (`MappedFileCacheCount`), and there the plain form, whose pages are already in, - scans faster than these tables say. -* **A take pays per block it touches.** The thousand rows measured touch every chunk of the file, - which is the worst case for a frame and a large dictionary alike; rows that sit together cost far - less. -* **The report says what each chunk became**: `WriteReport.Columns[i].Encodings`, as the *written - as* column of the benchmark page's tables. +* The benchmark page's times come from one machine, with the file in the page cache and mapped anew + by each read. The bytes carry over to any machine, and so do the ratios between scans, but the + crossing points scale with the machine's decode speed. A process that opens the same file again + keeps its mapping (`MappedFileCacheCount`), and there the plain form, whose pages are already in + memory, scans faster than the tables say. +* A take pays for every block it touches. The thousand rows measured touch every chunk of the file, + which is the worst case for both a zstd frame and a large dictionary. Rows that sit together cost + far less. +* The report says what each chunk became: `WriteReport.Columns[i].Encodings`, like the *written as* + column of the benchmark page's tables. ## Run it -The benchmark page's two sections are regenerated by +The benchmark page's two sections are regenerated by: ``` DOTNET_TieredCompilation=0 dotnet run -c Release --project bench/Vorticity.Benchmarks -- --tradeoffs --out docs/guide/benchmarks.md DOTNET_TieredCompilation=0 dotnet run -c Release --project bench/Vorticity.Benchmarks -- --tradeoffs --advise --out docs/guide/benchmarks.md ``` -`--rows N` changes the volume, and words after it pick columns by name: `-- --tradeoffs --rows -1000000 utf8` runs the text columns on a million rows and prints their tables without touching the -page. `--advise` runs the advice on each column under the five goals, with the choice and its -reason. +`--rows N` changes the volume, and words after it select columns by name. For example, +`-- --tradeoffs --rows 1000000 utf8` runs the text columns on a million rows and prints their tables +without touching the page. `--advise` runs the advice on each column under the five goals and prints +the choice and its reason. diff --git a/docs/guide/copy-a-file.md b/docs/guide/copy-a-file.md index 593057ee..a75a102b 100644 --- a/docs/guide/copy-a-file.md +++ b/docs/guide/copy-a-file.md @@ -1,7 +1,7 @@ # Copy a file -Copy a file batch by batch, keep only the rows or the columns you want, or write it again under -other options. +Copy a file batch by batch, keep only the rows or columns you want, or write it again under other +options. ```csharp await using VortexFile source = await session.OpenAsync(input); @@ -13,13 +13,13 @@ await using (VortexFileWriter target = session.CreateWriter(output, source.Schem } ``` -The source is the demonstration file of a million readings. `source.Scan()` with no column names -reads every column, and `WriteAsync(BatchView)` takes each batch as it comes: a columnar -pass-through, with no rows and no builder in between. +The source is the demonstration file of a million readings. `source.Scan()` with no column names reads +every column, and `WriteAsync(BatchView)` takes each batch as it comes: a columnar pass-through, with +no rows and no builder in between. ## The variants -The same loop with a filter is a filtered copy, with a record a projected one, and under other +The same loop with a filter makes a filtered copy, with a record a projected copy, and under other options a rewrite: ```csharp @@ -44,45 +44,46 @@ await using (VortexFileWriter target = session.CreateWriter(outpu } ``` -`DayAndCelsius` is `[VortexRecord] public partial record struct DayAndCelsius(int Day, double? Celsius)`: -the record is the projection, so the scan reads two columns and the target has two. +`DayAndCelsius` is `[VortexRecord] public partial record struct DayAndCelsius(int Day, double? Celsius)`. +The record is the projection, so the scan reads two columns and the target has two. ## What it costs ``` -source: 1000000 rows, 1508212 bytes -a copy: 1000000 rows, 1507228 bytes, 16 chunks, in 21 ms -re-encoded under Smallest: 245628 bytes, in 49 ms; Day RunEnd x16, Celsius Zstd x16, City Zstd x16 -a filtered copy, Day >= 900: 100000 rows, 157452 bytes, in 3 ms -a typed filtered copy, Day >= 900 and City == Paris: 12502 rows, 19932 bytes, in 2 ms -a projected copy, Day and Celsius: 1000000 rows, 1153396 bytes, in 15 ms +source: 1000000 rows, 1508316 bytes +a copy: 1000000 rows, 1507236 bytes, 16 chunks, in 19 ms +re-encoded under Smallest: 242780 bytes, in 71 ms; Day RunEnd x16, Celsius Zstd x16, City Zstd x16 +a filtered copy, Day >= 900: 100000 rows, 157460 bytes, in 3 ms +a typed filtered copy, Day >= 900 and City == Paris: 12502 rows, 19844 bytes, in 4 ms +a projected copy, Day and Celsius: 1000000 rows, 1153404 bytes, in 12 ms the projected copy reads back: struct{Day: i32, Celsius: f64?}, 1000000 rows, mean 30.0000 ``` -The times are from a run in which earlier samples had already compiled the code paths. Each column -is decoded once and encoded once; the target chooses its encodings afresh and cuts its own chunks, -by the width of the rows it is handed ([blocks-and-chunks.md](blocks-and-chunks.md)). The copy came -out within a kilobyte of its source, in 16 chunks where the source has 17. A copy is also how a file that has -taken many appends gets its space back ([append-and-repair.md](append-and-repair.md)). `Smallest` -made it six times smaller than the source, for more than twice the time spent writing: the -temperatures and the cities went to zstd frames, which a read then inflates block by block. A -filtered copy reads only the blocks the filter keeps, by the same pruning as any scan -([filter-rows.md](filter-rows.md)). +These times come from a run in which earlier samples had already compiled the code paths. Each +column is decoded once and encoded once. The target chooses its encodings afresh and cuts its own +chunks, based on the width of the rows it is handed ([blocks-and-chunks.md](blocks-and-chunks.md)). +The copy came out within a kilobyte of its source, in 16 chunks where the source has 17. Copying is +also how a file that has taken many appends gets its space back +([append-and-repair.md](append-and-repair.md)). `Smallest` made the file six times smaller than the +source, for more than three times the write time: the temperatures and the cities went to zstd +frames, which a read then inflates block by block. A filtered copy only reads the blocks the filter +keeps, with the same pruning as any scan ([filter-rows.md](filter-rows.md)). ## Watch out -* **The target must not be the source.** The source is open, and a writer over the same path throws +* The target must not be the source. The source is open, and a writer over the same path throws `IOException`: *The process cannot access the file … because it is being used by another process.* Write elsewhere and move the file once the copy completes. -* **A record written as columns covers every column of the target.** Three columns into a writer of +* A record written as columns must cover every column of the target. Three columns into a writer of two throws `VortexSchemaException`: *Member 'City' of Reading has no column to write; the struct is struct{Day: i32, Celsius: f64?}.* -* **The types must be the target's.** A batch is refused when a column's type differs from the - target's; a non-nullable column may go into a nullable one. -* **A batch is borrowed** until the task `WriteAsync` returns completes: await it before the next - batch, as the loop does. -* The copy takes the target's options, not the source's: indexes, statistics, edition and metadata - are what you pass to `CreateWriter` ([writer-options.md](writer-options.md), [indexes.md](indexes.md)). +* The types must match the target's. A batch is refused when a column's type differs from the + target's, although a non-nullable column may go into a nullable one. +* A batch is borrowed until the task returned by `WriteAsync` completes, so await it before moving to + the next batch, as the loop does. +* The copy uses the target's options, not the source's. Indexes, statistics, edition and metadata + are whatever you pass to `CreateWriter` ([writer-options.md](writer-options.md), + [indexes.md](indexes.md)). ## Run it @@ -90,7 +91,7 @@ filtered copy reads only the blocks the filter keeps, by the same pruning as any dotnet run -c Release --project samples/Vorticity.Samples -- copy-a-file ``` -The sizes above come from that run, which prints them the same whichever way it is run; the times -from a run of every sample, `dotnet run -c Release --project samples/Vorticity.Samples`, in which -this one comes after the others. On its own, compiling the copy's code paths as it goes, the first -copy takes about 130 ms. +The sizes above come from that run, and they are the same however it is run. The times come from a +run of every sample, `dotnet run -c Release --project samples/Vorticity.Samples`, in which this one +comes after the others. Run on its own, compiling the copy's code paths as it goes, the first copy +takes about 130 ms. diff --git a/docs/guide/dataset-maintenance.md b/docs/guide/dataset-maintenance.md index 87b35225..6147ba4f 100644 --- a/docs/guide/dataset-maintenance.md +++ b/docs/guide/dataset-maintenance.md @@ -1,6 +1,6 @@ # Dataset maintenance -Compact, vacuum and verify a dataset, and what each costs in requests. +Compact, vacuum and verify a dataset, and see what each operation costs in store requests. ```csharp await using CountingObjectStore store = new CountingObjectStore(new MemoryObjectStore(), ownsInner: true); @@ -28,21 +28,22 @@ plan: work True, style Leveled, clustered True, lag 2, objects by level [10], by cost: 0 requests (0 get, 0 head, 0 put, 0 delete, 0 list), 0 dependent steps, 0 bytes read, 0 written ``` -Appending is cheap and leaves a dataset in poor shape: many small objects, and every old version -still holding its own. The three operations here put it back in order. None of them runs by -itself: they are the caller's background job, and the dataset only reports how far behind it is. -Like everything in `Vorticity.Dataset`, they are experimental: see [datasets.md](datasets.md). +Appending is cheap, but it leaves a dataset in poor shape: many small objects, and every old version +still holding on to its own. The three operations on this page put it back in order. None of them +runs by itself. They are the caller's background job, and the dataset only reports how far behind +it is. Like everything in `Vorticity.Dataset`, they are experimental (see [datasets.md](datasets.md)). ## Compaction -Level 0 is where appends land, and it holds eight objects before compaction has work to do. `Lag` -is how far past that ceiling it is: here ten objects, a lag of 2, and every lookup by key touches -the two extra objects until compaction catches up. Ask before doing: `PlanCompactionAsync` reads -the top page of each level, which the version's header carries or the handle already holds, and -descends from it only to the objects a job would take: no request at all here, and two over 125 000 -objects. `HasWork` -is the question; `Job` is the one job it would run: the levels it moves between, what triggered -it, the objects, rows and bytes it would read, and the size of the objects it would write. +Appends land in level 0, which can hold eight objects before compaction has work to do. `Lag` is how +far past that ceiling it is. Here ten objects give a lag of 2, and every lookup by key touches the +two extra objects until compaction catches up. + +You can ask before doing anything. `PlanCompactionAsync` reads the top page of each level, which the +version's header carries or the handle already holds, and only descends to the objects a job would +take. Here that costs no request at all, and two requests over 125 000 objects. `HasWork` answers +the question, and `Job` describes the one job it would run: the levels it moves between, what +triggered it, the objects, rows and bytes it would read, and the size of the objects it would write. ```csharp CompactionResult? compacted = await dataset.CompactAsync(); @@ -54,42 +55,39 @@ compacted: version 12, Applied, level 0 to 1, 10 objects in and 1 out, 128862 by now: 1 objects, lag 0; again: nothing to do ``` -**Ten objects became one, and 128 862 bytes became 79 036**: the same 50 000 rows in two thirds of -the space, because a column encoder does more with 50 000 rows than with 5 000 at a time. Small -appends cost size, and compaction is what gets it back. - -`CompactAsync` runs one job and returns `null` when none is due, so a caller that wants the -dataset back within its bounds loops until it does, rather than having one call rewrite -gigabytes. With a clustering key the style is `Leveled`: the levels above 0 hold key-disjoint -objects, so a lookup by key touches at most one object per level. Without one it is `Tiered`, -which rewrites each row less often and bounds lookups less. `CompactionOptions` moves the level 0 -ceiling, the fan-out between levels, the size of a level 1 object and the cap on an output object. -`Outcome` says whether the commit applied, or found that a concurrent writer had already done it. - -Level 0 goes to the first level that holds it: the ten appends here, 126 KiB, fit level 1, which -holds ten objects of 128 KiB; a load of gigabytes goes past the levels it would only overflow and -is written once. A dataset that takes many small commits then pays for each a merge into a level a -few times its size, never one into the levels that hold the bulk of its rows. An output object is -at most `MaxObjectBytes`, 4 MiB by default. - -A level above 0 over its size gives up one object, merged into the level above with the objects -there its keys meet: by default the largest, the one holding the level over its size the most. -`CompactionOptions.Pick = CompactionPick.RoundRobin` takes instead the object past the one the -level's last job took, and the first past the end, so that the rewrites go over every key in turn, -as LevelDB does; each level of a header records where its last job stopped. On ten million rows -and 30 000 small commits it wrote 6 % more than the largest first, which is why a leveled dataset -does not default to it. A tiered dataset does: its job concatenates a run of one level's objects -that nothing else sits between, and the run past the pointer is found by descending the trees, one -page over 100 000 objects, where the longest run is found only by reading every leaf, 71 pages. A -dataset ordered by arrival keeps each level's objects together, and the two take the same runs. - -Compaction writes the live rows only: the rows a delete marked in an input object end with it, and -its outputs carry no marks. An object whose marks reach half a delete's bounds, a sixteenth of its -rows or half a kilobyte of positions, is rewritten alone in its level by a job whose `Trigger` is -`Marks`: after the bounds of the levels and before the fragments, the most marked object first. A -compaction that read an object before a delete marked rows in it -finds the object's entry changed at its commit, and is `Abandoned` rather than bringing those rows -back. +Ten objects became one, and 128 862 bytes became 79 036. The same 50 000 rows take a third less +space, because the column encoders do more with 50 000 rows at once than with 5 000. Small appends +cost size, and compaction wins it back. + +`CompactAsync` runs one job and returns `null` when none is due. A caller that wants the dataset +back within its bounds calls it in a loop, so no single call rewrites gigabytes. With a clustering +key the style is `Leveled`: the levels above 0 hold objects with disjoint keys, so a lookup by key +touches at most one object per level. Without a key it is `Tiered`, which rewrites each row less +often but bounds lookups less. `CompactionOptions` sets the level 0 ceiling, the fan-out between +levels, the size of a level 1 object and the cap on an output object. `Outcome` says whether the +commit applied, or whether a concurrent writer had already done the work. + +Level 0 goes to the first level large enough to hold it. The ten appends here, 126 KiB, fit in level +1, which holds ten objects of 128 KiB, while a load of gigabytes skips the levels it would only +overflow and is written once. A dataset that takes many small commits therefore pays for each one a +merge into a level a few times its size, never a merge into the levels that hold most of its rows. +An output object is at most `MaxObjectBytes`, 4 MiB by default. + +When a level above 0 grows past its size, it gives up one object, which is merged into the next +level with the objects there whose keys it overlaps. By default it picks the largest object, the one +that pushes the level furthest over its size. `CompactionOptions.Pick = CompactionPick.RoundRobin` +takes instead the object after the one the level's previous job took, wrapping around at the end, +so that rewrites sweep every key in turn as LevelDB does. On ten million rows and 30 000 small +commits it wrote 6 % more than the largest-first default, which is why leveled datasets do not use +it. Tiered datasets do, because their job concatenates a run of one level's objects, and finding the +run after the pointer costs a page where finding the longest run means reading every leaf. + +Compaction writes only the live rows. Rows a delete marked in an input object disappear with it, and +the outputs carry no marks. An object whose marks reach half a delete's limits (a sixteenth of its +rows or half a kilobyte of positions) is rewritten alone in its level by a job whose `Trigger` is +`Marks`. Those jobs come after the level limits and before the fragments, the most marked object +first. A compaction that read an object before a delete marked rows in it finds the object's entry +changed when it commits, and ends `Abandoned` rather than bringing those rows back. ## Verify @@ -104,16 +102,16 @@ verify since 12: holds True, 1 objects, 1 pages; cost: 12 requests (9 get, 3 hea ``` `VerifyAsync` checks the version's tree pages, objects and index fragments against what their -references promise, and names every problem instead of throwing: `Problems` is a list of -sentences, empty when `Holds` is true. An object is hashed against the hash its entry records; -`Unhashed` counts the imported objects whose entry records none, whose length is checked instead. -`since` skips what an earlier, trusted version shares with this one: after one more append, it -read 28 830 bytes where a full verification read 146 049. +references promise, and reports every problem instead of throwing. `Problems` is a list of +sentences, empty when `Holds` is true. Each object is hashed and compared with the hash its entry +records. `Unhashed` counts the imported objects whose entry records no hash, and for those only the +length is checked. `since` skips what an earlier, trusted version shares with this one: after one +more append it read 28 178 bytes, where a full verification read 145 609. ## Vacuum -Old versions keep their objects readable. Vacuum deletes what no version inside the retention -window references: +Old versions keep their objects readable. Vacuum deletes what no version inside the retention window +still references: ```csharp VacuumResult today = await dataset.VacuumAsync(new VacuumOptions { DryRun = true }); @@ -130,26 +128,28 @@ vacuum in eight days: 21 deleted (11 commit objects), latest version 13; cost: 3 and the data: 55000 rows, 2 objects ``` -Today nothing goes: every version is younger than the window, seven days unless -`DatasetOptions.RetentionWindow` said otherwise when the dataset was created, and +Today nothing goes, because every version is younger than the window. The window is seven days +unless `DatasetOptions.RetentionWindow` set another when the dataset was created, and `RetainedVersions` keeps a number of versions whatever their age. Eight days later only the latest -version is kept, and 21 objects go: the ten data objects compaction replaced, and eleven of the -twelve superseded commit objects, the twelfth still holding pages the latest version reads. `VacuumOptions.TimeProvider` is the clock vacuum measures ages against, which is -how the sample crosses the window and how you test your own retention; it must agree with the -store's clock, because the ages it compares are the store's timestamps. The deletes go out in -batches, two requests for twenty-one keys here. The fourth listing is of the leases a compaction -loop may have left (below); each lease's key says when it ended, so none costs a head request. - -**Run a dry run first.** `Deleted` lists what would go, commit objects first, `Young` the +version is kept and 21 objects go: the ten data objects compaction replaced, and eleven of the +twelve superseded commit objects. The twelfth still holds pages the latest version reads. + +`VacuumOptions.TimeProvider` is the clock vacuum measures ages against. That is how the sample +jumps past the window, and how you can test your own retention. It must agree with the store's +clock, since the ages it compares are the store's timestamps. Deletes go out in batches, two +requests for twenty-one keys here. The fourth listing covers the leases a compaction loop may have +left behind (see below), and since each lease's key says when it ended, none needs a head request. + +Run a dry run first. `Deleted` lists what would go, commit objects first. `Young` lists the unreferenced objects kept because they may belong to a writer still in flight, and `Retained` the versions kept. On a store under a retention lock, `Locked` lists what is past the window but still -under the store's lock or a legal hold, and `NextUnlock` says when the first of it may go: the next -vacuum after that date takes it. +under the store's lock or a legal hold, and `NextUnlock` says when the first of it may go, so the +next vacuum after that date takes it. ## In the background -A loop that compacts until it is cancelled, which any host runs: a hosted service, a worker, a -console. The library starts no thread of its own. +The library starts no thread of its own. Compaction in the background is a loop that runs until it +is cancelled, hosted wherever you like: a hosted service, a worker, a console. ```csharp using CancellationTokenSource stop = new CancellationTokenSource(); @@ -163,31 +163,31 @@ Task loop = dataset.RunCompactionAsync( stopped: 1 compactions, 2 objects, lag 0 ``` -Each turn refreshes the handle, one head request, plans, and runs the job due for it; when nothing -is due it sleeps for `Idle`, a minute by default, and asks again. `BytesPerSecond` paces it: after a -job it waits until what the job wrote fits the rate, so that compaction leaves the store's bandwidth -to the writers. The task ends with `OperationCanceledException` once the token is cancelled, or with -the exception a job raised, which a host catches before running the loop again. - -A job is the one `CompactAsync` runs, so a loop is safe against every writer and every other loop; -what several loops on one dataset can waste is a job two of them run, which a commit then abandons. -Loops that know one another say so with `Loops` and `Loop`: each takes the due job ranked at its -index, and no two ranked jobs read or write one level. Loops that cannot count one another set -`Leases`: before a job a loop creates `leases//` for each of its levels by -put-if-absent, holds them until the end of the current `LeaseSpan`, a minute by default, and moves -to the next job when another loop holds one. The loop remembers the leases it created, so it runs -the next job on its levels within the same span without asking the store again; a loop that leased -some of a job's levels before finding another holding the rest keeps those for the span all the -same. Nothing releases a lease, so a store that refuses deletes still works; vacuum deletes the +Each turn refreshes the handle (one head request), plans, and runs the job that is due. When nothing +is due it sleeps for `Idle`, a minute by default, and asks again. `BytesPerSecond` paces the loop: +after a job it waits until what the job wrote fits the rate, which leaves the store's bandwidth to +the writers. The task ends with `OperationCanceledException` once the token is cancelled, or with +the exception a job raised, which the host catches before starting the loop again. + +A loop runs the same jobs as `CompactAsync`, so it is safe against every writer and every other +loop. What several loops on one dataset can waste is a job that two of them run at once, which one +commit then abandons. Loops that know about one another say so with `Loops` and `Loop`: each takes +the due job ranked at its index, and no two ranked jobs read or write the same level. Loops that +cannot coordinate set `Leases` instead. Before a job, a loop creates `leases//` for each +of its levels with put-if-absent, holds them until the end of the current `LeaseSpan` (a minute by +default), and moves to the next job when another loop holds one. A loop remembers the leases it +created, so it can run the next job on the same levels within the span without asking the store +again. Nothing releases a lease, so a store that refuses deletes still works, and vacuum deletes the leases that ended a window ago. ## In the writer's commit -A dataset with one writer and no process to spare can have the writer hold level 0 itself: -`DatasetOptions.InlineCompactionBytes` makes a commit that takes level 0 past its ceiling merge it -into the level above before it returns, when the job reads no more than that many bytes. The merge's -latency lands in that one commit, and a larger job is left to another driver. The commit stands -whatever befalls the merge, which, failed or cancelled, leaves level 0 for the next commit. +A dataset with a single writer and no spare process can let the writer keep level 0 in check. +`DatasetOptions.InlineCompactionBytes` makes a commit that pushes level 0 past its ceiling merge it +into the next level before returning, as long as the job reads no more than that many bytes. The +merge's latency lands on that one commit, and larger jobs are left to another driver. The commit +stands whatever happens to the merge, and a failed or cancelled merge leaves level 0 to the next +commit. ## What each costs @@ -201,25 +201,25 @@ Measured above with `CountingObjectStore`, on a dataset of one object after comp | `VerifyAsync(since)` | 12 | what the earlier version does not share | | `VacuumAsync` | 29 to 31 | a listing, a head per commit object, the retained trees, the leases' listing, the deletes in batches | -`DependentSteps` is the number that decides latency: the round trips that waited for the one -before. On a store where a request costs 30 ms, it is what a job takes. See +`DependentSteps` is the number that sets latency: the round trips that had to wait for the previous +one. On a store where a request costs 30 ms, it is what a job takes. See [object-store.md](object-store.md). ## Watch out -* **Compaction and vacuum are separate.** Compaction leaves the replaced objects in place for the - versions that still name them; vacuum removes them once those versions are past the window. -* **A reader that outlives the window loses its objects.** Vacuum marks from the store's latest - version, not from yours: a handle still on an old version then finds an object missing, and - `ObjectNotFoundException` says to refresh. -* Compaction commits like any writer, so a concurrent writer can win the race; look at `Outcome`. - Vacuum deletes only what no retained version references, and deletes nothing until every - retained version is marked. -* **A writer that dies mid-commit on `FileObjectStore` leaves a commit that is not whole.** Every - open and every commit then throws `TornCommitException`, which names the version; +* Compaction and vacuum are separate. Compaction leaves the replaced objects in place for the + versions that still name them, and vacuum removes them once those versions are past the window. +* A reader that outlives the window loses its objects. Vacuum works from the store's latest version, + not from yours, so a handle still on an old version may find an object missing. + `ObjectNotFoundException` then tells it to refresh. +* Compaction commits like any writer, so a concurrent writer can win the race. Check `Outcome`. + Vacuum deletes only what no retained version references, and deletes nothing until every retained + version has been walked. +* A writer that dies mid-commit on `FileObjectStore` leaves an incomplete commit. Every open and + every commit then throws `TornCommitException`, which names the version. `VortexDataset.RemoveTornCommitAsync(store)` removes it once no writer holds it, and the dataset reads as the version before. -* Nothing here changes the data: after all three, the same rows read back. +* None of this changes the data. After all three operations, the same rows read back. ## Run it diff --git a/docs/guide/datasets.md b/docs/guide/datasets.md index 90ef0bd9..fc1152fe 100644 --- a/docs/guide/datasets.md +++ b/docs/guide/datasets.md @@ -1,7 +1,7 @@ # Datasets -Keep a table that grows as a versioned set of Vortex files in an object store, and read it with -the same scans as a file. +A dataset keeps a growing table as a versioned set of Vortex files in an object store. You read it +with the same scans as a single file. ```csharp await using IObjectStore store = new FileObjectStore(Demo.Path("dataset")); @@ -19,42 +19,41 @@ await using (ObjectDraft draft = dataset.StartObject()) ``` created: version 1, 0 rows, clustered by Day -appended data/7f4f12630bb346098a7d785a6bed85bb.vortex: version 2, 50000 rows +appended data/3ee81023c25245108e5b2cfcf0978769.vortex: version 2, 50000 rows ``` -This is `Vorticity.Dataset`, a separate package, and it is **experimental**: the assembly is -marked `[Experimental("VX0001")]`, so the compiler reports VX0001, as an error, wherever your code -uses one of its types, until you opt in. Opt in once for the project, with +Datasets live in `Vorticity.Dataset`, a separate package, and it is experimental. The assembly is +marked `[Experimental("VX0001")]`, so the compiler reports VX0001 as an error wherever your code uses +one of its types, until you opt in. You can opt in once for the whole project with `$(NoWarn);VX0001`, or mark your own assembly or type `[Experimental]`, which -makes its uses of experimental code legal and passes the warning on to its callers; the samples do -the latter. The diagnostic says what the attribute means: the dataset's commit format and API may -change between releases. A release reads the formats of the releases before it, and a release -before a format refuses a dataset written in it rather than read it wrongly: version 0.2.0 reads -format 1, and refuses the format 2 this one writes. The files inside a dataset are ordinary Vortex -files; the tree of commits -over them is this repository's own format, which no other Vortex implementation reads. A single -file needs none of this. +allows its uses and passes the warning on to its callers. The samples do the latter. + +The warning means what it says: the dataset's commit format and API may change between releases. A +release reads the formats of the releases before it, and an older release refuses a newer format +rather than misread it. Version 0.2.0 reads format 1 and refuses the format 2 written today. The +files inside a dataset are ordinary Vortex files, but the tree of commits over them is this +project's own format, and no other Vortex implementation reads it. A single file needs none of this. ## What a dataset is -A set of immutable Vortex files, the **data objects**, and one commit object per version that says -which objects the version holds. Every change is a commit, and every commit is a new version; -nothing is overwritten. A commit is a conditional creation in the store, so two writers need no -lock: the one that loses the race re-applies its change on top of the winner's version, up to -`DatasetOptions.MaxAttempts` times. +A dataset is a set of immutable Vortex files, the data objects, plus one commit object per version +that lists the objects the version holds. Every change is a commit and every commit is a new +version, so nothing is ever overwritten. A commit is a conditional create in the store, which means +two writers need no lock. The one that loses the race applies its change again on top of the +winner's version, up to `DatasetOptions.MaxAttempts` times (8 by default). -`ClusteringKey` is the decision that matters most. The objects are ordered by the smallest key -each holds, every object this handle writes carries a sorted-runs index on the key, and -compaction keeps the levels above 0 free of overlap on it. That is what lets a scan skip whole -objects, the way ordering rows lets a scan skip blocks; `SummaryColumns` chooses which other -columns each object summarises for the same purpose, the first 32 by default. +`ClusteringKey` is the decision that matters most. Objects are ordered by the smallest key each +holds, every object this handle writes carries a sorted-runs index on the key, and compaction keeps +the levels above 0 free of overlap on it. That lets a scan skip whole objects, the same way ordered +rows let a scan skip blocks. `SummaryColumns` picks which other columns each object summarises for +the same purpose, the first 32 by default. ## Adding rows -`StartObject()` returns an `ObjectDraft`: a key in the store and a `VortexFileWriter` with the -dataset's schema, whose rows you write as you would to any file. `AppendAsync(draft)` completes the -writer, puts the object in the store and commits it; disposing a draft that was not committed -abandons it, and nothing reaches the store. `AppendAsync(batches)` does the same with an +`StartObject()` returns an `ObjectDraft`, which is a key in the store plus a `VortexFileWriter` with +the dataset's schema. You write rows to it as you would to any file. `AppendAsync(draft)` completes +the writer, puts the object in the store and commits it. Disposing a draft that was never committed +abandons it, and nothing reaches the store. `AppendAsync(batches)` does the same from an `IAsyncEnumerable`, such as another scan's `ToBatchesAsync()`. A file that is already in the store joins for the price of a commit, without being rewritten: @@ -70,20 +69,20 @@ ulong imported = await dataset.ImportAsync("imports/days-100-119.vortex"); ``` imported: version 4, 120000 rows, 3 objects, lag 0 - data/7f4f12630bb346098a7d785a6bed85bb.vortex: level 0, rows 0 to 50000, 82457 bytes - data/6fd130088886482683ebdd6485d5530d.vortex: level 0, rows 50000 to 100000, 82457 bytes + data/3ee81023c25245108e5b2cfcf0978769.vortex: level 0, rows 0 to 50000, 82457 bytes + data/73531b788ee34599aa8fc9e7e167e372.vortex: level 0, rows 50000 to 100000, 82457 bytes imports/days-100-119.vortex: level 0, rows 100000 to 120000, 37836 bytes ``` -`ImportAsync` takes a key in the store, not a path on disk: putting the bytes there is yours to -do, and the file's columns must read as the dataset's: the same ones, or those of an earlier schema -of the dataset ([Changing the schema](#changing-the-schema)). `ObjectsAsync()` lists the objects of -the version, reading the tree and never an object. +`ImportAsync` takes a key in the store, not a path on disk, so putting the bytes there is up to you. +The file's columns must read as the dataset's, either the current ones or those of an earlier +schema of the dataset (see [Changing the schema](#changing-the-schema)). `ObjectsAsync()` lists the +objects of the version from the commit tree, without opening any object. ## Reading -`dataset.Scan()` is the `Scan` of a file, over every object of the version, and -`dataset.Scan(columns)` is the tool scan. Everything in the guide's read pages applies: +`dataset.Scan()` is the same `Scan` as on a file, over every object of the +version, and `dataset.Scan(columns)` is the tool scan. Everything in the guide's read pages applies: ```csharp Scan recent = dataset.Scan().Where(r => r.Day >= 100); @@ -98,26 +97,29 @@ key cursor: seek 75 found True, key 75 at row 75000; 1000 entries walking down: last key 119 at row 119999; the key before it 118, its last entry at row 118999 ``` -The plan's first pruning step is the objects' summaries: the two objects whose days end before 100 -are not opened, and their 13 blocks are pruned without a read. `MayMatch` answers from the -summaries the handle already holds, the version's header and the pages it kept from earlier -versions, so it reads nothing, and `false` is a proof. -Positions are the dataset's: a batch's `StartRow`, `Rows(…)` and a key cursor's `Row` count the -version's rows, object after object, in the order a scan delivers them. A key cursor over the -clustering key merges the objects' cursors; its `KeyCountAsync` is the number of entries under the -current key, here the thousand rows of day 75. It walks both ways, as a file's does: `SeekLastAsync`, -`PrevAsync`, `PrevKeyAsync`, `AtOrBefore` and `Before`, and a step against the direction of the -last one seeks every object again at the current entry. Walking down, an object of a level above 0 -opens only once the walk could reach its keys, and level 0's all open at the seek, since nothing -bounds them from above. See [keys-in-order.md](keys-in-order.md). +Pruning starts with the object summaries. The two objects whose days all fall before 100 are never +opened, and their 13 blocks are pruned without a read. `MayMatch` answers from the summaries the +handle already holds (the version's header and the pages it kept from earlier versions), so it reads +nothing, and `false` is a proof. + +Row positions are the dataset's. A batch's `StartRow`, `Rows(…)` and a key cursor's `Row` count the +version's rows object after object, in the order a scan delivers them. A key cursor over the +clustering key merges the cursors of the objects, and its `KeyCountAsync` is the number of entries +under the current key, here the thousand rows of day 75. It walks both ways like a file's cursor +(`SeekLastAsync`, `PrevAsync`, `PrevKeyAsync`, `AtOrBefore`, `Before`), and a step against the +direction of the previous one seeks every object again at the current entry. Walking down, an object +above level 0 opens only once the walk could reach its keys, while the level 0 objects all open at +the seek since nothing bounds them from above. See [keys-in-order.md](keys-in-order.md). A scan reads the objects one after another in the tree's order. Once the consumer reads past the -first batch, the next object is opened, and its first batch read and decoded, while the one before -it is read: two objects at most are open, and a scan that stops at its first batch opens one. -`ScanOptions.Prefetch` governs it, 1 by default; 0 opens each object only when the scan reaches it. -A group by at a degree above one reads the objects side by side, each a range of the lanes' queue -and a large one cut between its chunks. On the first column of the clustering key, or a function -of it, a group by streams: the dataset reads its rows in the key's order. See +first batch, the next object is opened and its first batch read and decoded while the current one +is still being read. At most two objects are open at a time, and a scan that stops at its first +batch opens only one. `ScanOptions.Prefetch` controls this, 1 by default, and 0 opens each object +only when the scan reaches it. + +A group by at a degree above one reads the objects side by side, each one a range in the lanes' +queue, with large objects cut between their chunks. On the first column of the clustering key, or a +function of it, a group by streams, because the dataset delivers its rows in key order. See [aggregates.md](aggregates.md). ## Versions @@ -126,13 +128,13 @@ of it, a group by streams: the dataset reads its rows in the key's order. See after another append: the handle reads version 5, 130000 rows; a scan built before it counts 120000; another handle reads version 4 until it refreshes to 5 ``` -A handle is pinned to one version: it moves only when it commits or when you call `RefreshAsync`. -A scan is pinned to the version its handle held when the scan was built, so a commit in the middle -of a query does not change its answer. `Version`, `RowCount`, `ObjectCount` and `Lag` describe the -version the handle holds; `Lag` is how many objects level 0 holds above its ceiling of eight, -which compaction brings back to 0 ([dataset-maintenance.md](dataset-maintenance.md)). +A handle is pinned to one version. It moves only when it commits or when you call `RefreshAsync`. A +scan is pinned to the version its handle held when the scan was built, so a commit in the middle of +a query does not change the answer. `Version`, `RowCount`, `ObjectCount` and `Lag` describe the +version the handle holds. `Lag` is how many objects level 0 holds beyond its ceiling of eight, and +compaction brings it back to 0 ([dataset-maintenance.md](dataset-maintenance.md)). -## Removing and replacing +## Removing and replacing objects ```csharp await using (ObjectDraft rewritten = dataset.StartObject()) @@ -146,17 +148,17 @@ await using (ObjectDraft rewritten = dataset.StartObject()) ``` ``` -replaced data/7f4f12630bb346098a7d785a6bed85bb.vortex by the rows with a temperature: version 6, Applied, 129000 rows +replaced data/3ee81023c25245108e5b2cfcf0978769.vortex by the rows with a temperature: version 6, Applied, 129000 rows removed the import: version 7, Applied, 109000 rows, 3 objects removing it again: Abandoned, version 7 ``` -`ReplaceAsync(removed, added)` swaps objects in one commit: a reader sees the old ones or the new -ones, never both and never neither. `RemoveAsync(objects)` is the same with nothing added. Both +`ReplaceAsync(removed, added)` swaps objects in one commit. A reader sees either the old ones or the +new ones, never both and never neither. `RemoveAsync(objects)` is the same with nothing added. Both take `DataObject`s from `ObjectsAsync()`. The removed objects stay in the store, readable by the -versions that name them, until vacuum deletes them. Removing an object the version no longer -holds is `Abandoned`: no version is created, the result names the latest one, and the objects a -replace would have added are left for vacuum. +versions that still name them, until vacuum deletes them. Removing an object the version no longer +holds returns `Abandoned`: no version is created, the result names the latest one, and any objects +a replace would have added are left for vacuum. ## Deleting and updating rows @@ -174,31 +176,32 @@ warmed Nice from day 60: version 9, 6251 rows changed, 2 object(s) in, 3 out ``` `DeleteAsync` removes the rows its filter is true for, in one commit. A row the filter is false or -unknown for, a null compared, stays. `UpdateAsync` reads the matching rows as records, passes each -through the lambda, and writes the results into new objects of level 0, as an append would, since a -change may move a row's key: an update is a delete and an insert, applied together. Its record must -have a member for every column. `DeleteAsync(VortexExpr)` is the delete for a caller without a -record type. - -A delete marks the rows it takes in the objects that hold them rather than rewriting those objects: -each object's entry in the dataset records their positions, and every read steps over them, so a -delete of ten rows in a 4 MiB object writes a few kilobytes of metadata and no data. Counts stay -exact, and a scan or a count of a marked object reads what it would of a rewritten one; a walk in -key order pays a little for the marks, since a cursor over an object of level 0 reads the keys of -its deleted rows once to rank around them, and a step that lands in a run of them steps past it. -An object is rewritten without the rows -instead when it is under a mebibyte, where a rewrite costs little and keeps the object exact, and -when the rows marked in it would pass an eighth of its rows or a kilobyte of positions; an object left with no row is removed -whole, and one whose summaries refute the filter is not opened. `ObjectsMarked` counts the objects a -change marked, `ObjectsIn` and `ObjectsOut` those it rewrote, and `BytesIn` and `BytesOut` what the -rewrites read and wrote. `DataObject.DeletedRows` says how many rows an object holds marked, which -compaction drops when it next rewrites the object: in a merge, or alone once its marks reach half of -those bounds. +unknown for (a comparison with a null, for instance) stays. `UpdateAsync` reads the matching rows as +records, passes each one through the lambda and writes the results into new level 0 objects, as an +append would, because a change may move a row's key. An update is a delete and an insert applied +together. Its record must have a member for every column. `DeleteAsync(VortexExpr)` is the delete +for a caller without a record type. + +A delete usually marks rows instead of rewriting objects. Each object's entry in the dataset records +the positions of its deleted rows and every read steps over them, so deleting ten rows from a 4 MiB +object writes a few kilobytes of metadata and no data. Counts stay exact, and a scan of a marked +object reads the same bytes it would read from a rewritten one. A walk in key order pays a little +for the marks: a cursor over a level 0 object reads the keys of its deleted rows once to rank around +them, and a step that lands in a run of them skips past it. + +An object is rewritten without the rows instead in two cases: when it is under a mebibyte, where a +rewrite is cheap and keeps the object exact, and when its marked rows would pass an eighth of its +rows or a kilobyte of positions. An object left with no rows is removed whole, and one whose +summaries refute the filter is not opened. In the result, `ObjectsMarked` counts the objects a change +marked, `ObjectsIn` and `ObjectsOut` those it rewrote, and `BytesIn` and `BytesOut` what the rewrites +read and wrote. `DataObject.DeletedRows` says how many marked rows an object holds. Compaction drops +them the next time it rewrites the object, either in a merge or on its own once the marks reach half +of those limits. A reader sees the change whole or not at all. Rows appended by another writer while the change is -worked out are not touched; if another writer rewrites an object the change read, a compaction for -instance, the change is worked out again on the version that won, and `Attempts` counts how many -times. +being worked out are not touched. If another writer rewrites an object the change read (a compaction, +for instance), the change is worked out again on the winning version, and `Attempts` says how many +times that took. ## Changing the schema @@ -223,40 +226,45 @@ evolved: version 10, columns Day, Celsius, Town, Humidity a row written before: Observation { Day = 10, Celsius = 10.1, Town = Nice, Humidity = } ``` -`EvolveSchemaAsync` gives the dataset new columns in one commit that rewrites no object. A column is -added, nullable, under a name no column ever had; dropped, unless the clustering key holds it; -renamed, the new name mapped to the old one; its numbers widened within their kind, `int` to `long`, -`uint` to `ulong`, `float` to `double`; or made nullable. Anything else would lose a value already -written, and is refused before anything is committed, with the column and the reason. +`EvolveSchemaAsync` gives the dataset a new schema in one commit, without rewriting any object. The +allowed changes are these: + +* add a nullable column, under a name no column has ever had +* drop a column, unless the clustering key uses it +* rename a column, mapping the new name to the old one +* widen a number within its kind: `int` to `long`, `uint` to `ulong`, `float` to `double` +* make a column nullable + +Anything else could lose a value already written, and is refused before anything is committed, with +the column and the reason. -The objects written before read as the new schema: a column they lack as nulls, a renamed one -through the name it had, a widened one converted batch by batch. A filter is pushed down to such an -object in its own terms, and one over a column it lacks is settled without reading it: `Humidity > -50.0` skips every earlier object, `Humidity.IsNull` takes them whole. Compaction writes each object -it rewrites in the current schema, so the conversion lasts until an object's level is next -compacted. A dropped or renamed name is never used again, so an object written with it cannot lend -its values to another column. +Objects written earlier read as the new schema. A column they lack reads as nulls, a renamed column +is read through its old name, and a widened one is converted batch by batch. A filter reaches such an +object in its own terms, and a filter over a column the object lacks is settled without reading it: +`Humidity > 50.0` skips every earlier object and `Humidity.IsNull` takes them whole. Compaction +writes every object it rewrites in the current schema, so the conversion only lasts until the +object's level is next compacted. A dropped or renamed name is never reused, so an old object can +never lend its values to an unrelated column. ## Watch out -* **Experimental, and this repository's own format.** Pin the package version, and do not expect - another Vortex implementation to read the tree. -* **Marked rows keep their bytes until compaction.** A delete that marks rows frees no space: the - object holds them, up to an eighth of its rows, until compaction rewrites it or a delete would pass - that share. An object under a mebibyte is rewritten by every delete that touches it, so batch the - rows you change into few calls: two deletes of one row each in a small object rewrite it twice. -* **Two writers changing the schema at once do not both succeed**: the second is refused with - `InvalidOperationException` and changes the schema the dataset has once it has refreshed. -* **The store must create objects atomically and list them consistently**: see - [object-store.md](object-store.md). A store that cannot promise `PutIfAbsent` cannot host a - dataset safely. +* The dataset format is experimental and specific to this project. Pin the package version, and do + not expect another Vortex implementation to read the commit tree. +* Marked rows keep their bytes until compaction. A delete that marks rows frees no space, since the + object holds them (up to an eighth of its rows) until compaction rewrites it or a later delete + passes that share. An object under a mebibyte is rewritten by every delete that touches it, so + group the rows you change into few calls: two one-row deletes in a small object rewrite it twice. +* Two writers changing the schema at the same time do not both succeed. The second gets an + `InvalidOperationException`, and once it has refreshed it sees the schema the first one set. +* The store must create objects atomically and list them consistently. A store that cannot promise + `PutIfAbsent` cannot host a dataset safely. See [object-store.md](object-store.md). * Every version keeps its objects alive until a vacuum removes them, and a handle that outlives the - retention window loses its objects to vacuum: + retention window can lose its objects to vacuum. See [dataset-maintenance.md](dataset-maintenance.md). -* An object is created before the commit that names it, so a crash between the two leaves an - object no version references, which vacuum collects once it is older than the window. -* `DatasetOptions.Session` is the session whose pool, cache and parallelism the dataset's scans - use, and `DatasetOptions.TimeProvider` the clock a commit reads its creation time from. +* An object is created before the commit that names it, so a crash between the two leaves an object + no version references. Vacuum collects it once it is older than the window. +* `DatasetOptions.Session` is the session whose pool, cache and parallelism the dataset's scans use, + and `DatasetOptions.TimeProvider` is the clock a commit reads its creation time from. ## Run it diff --git a/docs/guide/diagnostics.md b/docs/guide/diagnostics.md index e80c5f0a..ac32ba9c 100644 --- a/docs/guide/diagnostics.md +++ b/docs/guide/diagnostics.md @@ -1,31 +1,35 @@ # Diagnostics -What the compiler tells you about your use of the library, and how to fix each one. +What the compiler tells you about the way you use the library, and how to fix each case. -The `Vorticity.Generators` package ships the `[VortexRecord]` generator and six analyzers. -Together they report eleven diagnostics. VX1001 to VX1004 are warnings about code that compiles -but would misbehave or throw; VX1005 to VX1008 are errors about a `[VortexRecord]` type the -generator cannot implement, and the generator emits nothing for a type that has one; VX1009 to -VX1011 are errors about a query that would throw when it is built, or that the compiler refuses -without naming the fix. Every diagnostic's help link points at its section below. +The `Vorticity.Generators` package ships the `[VortexRecord]` generator and six analyzers, which +report eleven diagnostics together: + +* VX1001 to VX1004 are warnings about code that compiles but would misbehave or throw. +* VX1005 to VX1008 are errors about a `[VortexRecord]` type the generator cannot implement. The + generator emits nothing for a type that has one. +* VX1009 to VX1011 are errors about a query that would throw when it is built, or that the compiler + refuses without saying how to fix it. + +Each diagnostic's help link names its section on this page. | id | severity | reported when | |---|---|---| | [VX1001](#vx1001) | warning | a span borrowed from a batch is kept in a variable that outlives the loop body | | [VX1002](#vx1002) | warning | an owned `RecordBatch` is never disposed | | [VX1003](#vx1003) | warning | a hole of an interpolated `Where` has a type no column maps to | -| [VX1004](#vx1004) | warning | one scan selects rows by range and by indices | +| [VX1004](#vx1004) | warning | one scan selects rows both by range and by indices | | [VX1005](#vx1005) | error | a record member has no column type | | [VX1006](#vx1006) | error | a record type, or a type containing it, is not `partial` | | [VX1007](#vx1007) | error | a type cannot be a record at all | | [VX1008](#vx1008) | error | a record member cannot be filled when rows are read | | [VX1009](#vx1009) | error | a component of a group key is not a column | -| [VX1010](#vx1010) | error | a record does not take the values of a selection | +| [VX1010](#vx1010) | error | a record does not match the values of a selection | | [VX1011](#vx1011) | error | several values are read without a record | ## VX1001 -**A borrowed span outlives its batch.** Warning. +A borrowed span outlives its batch. This is a warning. Everything a scan delivers in the body of `await foreach` points into buffers that the next `MoveNextAsync` reuses. The compiler already keeps `Columns` and `Column` inside the body, @@ -45,18 +49,18 @@ await foreach (var (_, celsius, _) in file.Scan()) warning VX1001: 'celsius.Values' is borrowed from a column of the current batch and is valid only inside the body of the enumeration that produced it; 'last' is declared outside that body ``` -**Fix:** declare the variable inside the body, or keep a copy that you own: `celsius.ToArray()` -for one column, `columns.ToOwned()` for the batch. See [scan-a-table.md](scan-a-table.md) and +To fix it, declare the variable inside the body, or keep a copy you own: `celsius.ToArray()` for one +column, `columns.ToOwned()` for the whole batch. See [scan-a-table.md](scan-a-table.md) and [owned-batches.md](owned-batches.md). ## VX1002 -**An owned batch is never disposed.** Warning. +An owned batch is never disposed. This is a warning. -A `RecordBatch` from `ToBatchesAsync()` or `ToOwned()` holds buffers from the session's pool -until it is disposed. The analyzer flags one that nothing disposes: no `using`, no `Dispose`, and -it is not returned, passed to a method, stored in a field or a collection, or captured by a -lambda, any of which hands it to a new owner. +A `RecordBatch` from `ToBatchesAsync()` or `ToOwned()` holds buffers from the session's pool until it +is disposed. The analyzer flags a batch that nothing disposes: no `using`, no `Dispose`, and it is not +returned, passed to a method, stored in a field or a collection, or captured by a lambda, any of which +hands it to a new owner. ```csharp await foreach (RecordBatch batch in file.Scan().ToBatchesAsync()) @@ -76,18 +80,18 @@ warning VX1002: The RecordBatch from 'ToBatchesAsync' is never disposed; dispose warning VX1002: The RecordBatch from 'ToOwned' is never disposed; dispose it, or hand it on ``` -**Fix:** `using (batch) { … }`, `using RecordBatch owned = columns.ToOwned();`, or pass the batch -to whoever will dispose it, as a channel does in [owned-batches.md](owned-batches.md). +To fix it, write `using (batch) { … }` or `using RecordBatch owned = columns.ToOwned();`, or pass the +batch to whoever will dispose it, as a channel does in [owned-batches.md](owned-batches.md). ## VX1003 -**A filter hole has no column type.** Warning. +A filter hole has no column type. This is a warning. -The tool scan takes a filter as an interpolated string, `Where($"Day >= {threshold}")`, and each -hole is compared with a column. Which column only the open file knows, so the analyzer cannot -check a hole against it; what it can check is that the hole's type maps to some column type at -all: a number, a `bool`, a `string`, a `decimal`, a date, a time, a timestamp, a `Guid`, bytes or a -list. A hole of any other type throws when the filter is built. +The tool scan takes a filter as an interpolated string, as in `Where($"Day >= {threshold}")`, and each +hole is compared with a column. Which column only the open file knows, so the analyzer cannot check a +hole against it. What it can check is that the hole's type maps to some column type at all: a number, +a `bool`, a `string`, a `decimal`, a date, a time, a timestamp, a `Guid`, bytes or a list. A hole of +any other type throws when the filter is built. ```csharp object threshold = 900; @@ -102,18 +106,18 @@ warning VX1003: The hole 'threshold' is of type 'object', which no column type m warning VX1003: The hole 'span' is of type 'System.TimeSpan', which no column type maps to; the filter throws when it is built ``` -**Fix:** give the hole the type of the column it is compared with, `int threshold = 900`. A hole -of a mapped type that does not fit its column, a `string` against an `i32` column, is not -reported here: `Where` throws `VortexSchemaException` for it, naming the column and both types -(see [errors.md](errors.md) and [untyped-files.md](untyped-files.md)). +To fix it, give the hole the type of the column it is compared with, as in `int threshold = 900`. A +hole of a mapped type that does not fit its column, such as a `string` against an `i32` column, is not +reported here: `Where` throws `VortexSchemaException` for it, naming the column and both types (see +[errors.md](errors.md) and [untyped-files.md](untyped-files.md)). ## VX1004 -**Rows by range and by indices on one scan.** Warning. +Rows are selected both by range and by indices on one scan. This is a warning. -`Rows(RowRange)` and `Rows(params ReadOnlySpan)` exclude each other: the second call on one -scan throws `InvalidOperationException`. The analyzer finds both in one fluent chain, or on a -local that holds the scan and is not reassigned, on the typed scan and the tool scan alike. +`Rows(RowRange)` and `Rows(params ReadOnlySpan)` exclude each other, and the second call on a +scan throws `InvalidOperationException`. The analyzer finds both in one fluent chain, or on a local +that holds the scan and is not reassigned, on the typed scan and the tool scan alike. ```csharp await file.Scan().Rows(new RowRange(0, 10)).Rows(4, 5).CountAsync(); @@ -127,26 +131,30 @@ warning VX1004: This scan already selects rows by range; a scan selects rows by warning VX1004: This scan already selects rows by indices; a scan selects rows by range or by indices, not both, and throws here ``` -**Fix:** keep one of the two. A range is also a list of indices, if you need to add single rows to -it. See [read-rows-by-index.md](read-rows-by-index.md). +To fix it, keep one of the two. A range can always be expressed as a list of indices if you also need +single rows. See [read-rows-by-index.md](read-rows-by-index.md). ## VX1005 -**A record member has no column type.** Error. +A record member has no column type. This is an error. Every member of a `[VortexRecord]` type becomes a column, so its type must map to a column type -under the table in [records.md](records.md#the-mapping). The message says why this one does not, -and usually what to declare instead: +according to the table in [records.md](records.md#the-mapping). Most .NET types do, including `char`, +arrays, `List`, dictionaries and lists of records. The message says why a member does not, and +usually what to declare instead: ```csharp [VortexRecord] -public partial record struct Grade(int Id, char Letter); // a char +public partial record struct Grid(int Id, int[,] Cells); // a two-dimensional array + +[VortexRecord] +public partial record struct Route(int Id, ReadOnlyMemory Stops); // a list of nullable records [VortexRecord] -public partial record struct Scores(int Id, int[] Values); // an array +public partial record struct Tags(int Id, Dictionary ByName); // a dictionary of lists [VortexRecord] -public partial record struct Batch(int Id, ReadOnlyMemory Readings); // a list of records +public partial record struct Site(int Id, Uri Home); // a type with no column [VortexRecord] public partial record struct Event(int Id, [VortexColumn(TimeZone = "Europe/Paris")] DateTime At); @@ -164,27 +172,28 @@ public partial class Node ``` ``` -error VX1005: Member 'Letter' of 'Grade' is of type 'char', which does not map to a column: a char is not a dtype; use string -error VX1005: Member 'Values' of 'Scores' is of type 'int[]', which does not map to a column: an array is not a column type; declare the member ReadOnlyMemory -error VX1005: Member 'Readings' of 'Batch' is of type 'System.ReadOnlyMemory', which does not map to a column: a list of records has no typed column to read it through; declare a list of a scalar type +error VX1005: Member 'Cells' of 'Grid' is of type 'int[*,*]', which does not map to a column: an array of more than one dimension is not a column type; declare an array of arrays +error VX1005: Member 'Stops' of 'Route' is of type 'System.ReadOnlyMemory', which does not map to a column: a list of records holds records, not nulls; declare its element type without '?' +error VX1005: Member 'ByName' of 'Tags' is of type 'System.Collections.Generic.Dictionary', which does not map to a column: a map's keys and values are scalars; hold anything deeper in a record +error VX1005: Member 'Home' of 'Site' is of type 'System.Uri', which does not map to a column: the type has no mapping to a dtype error VX1005: Member 'At' of 'Event' is of type 'System.DateTime', which does not map to a column: a DateTime reads a naive or UTC timestamp, not one in 'Europe/Paris'; declare the member DateTimeOffset error VX1005: Member 'Amount' of 'Price' is of type 'decimal', which does not map to a column: a decimal member reads a column of at most 28 digits and a scale of 0 to 28; declare it VortexDecimal for a wider one error VX1005: Member 'Parent' of 'Node' is of type 'Node?', which does not map to a column: a record cannot hold itself, directly or through another record ``` -The same diagnostic covers a `List` or a `Memory` member, a nullable extension type (an -extension is read and written without nulls), a time in days, and a decimal precision or scale -outside what the format allows. +The same diagnostic covers a list of lists of records, a map with nullable keys, a nullable extension +type (an extension is read and written without nulls), a time in days, and a decimal precision or +scale outside what the format allows. -**Fix:** change the member's type as the message says, or mark the member `[VortexIgnore]` if it -is not meant to be a column. +To fix it, change the member's type as the message says, or mark the member `[VortexIgnore]` if it is +not meant to be a column. ## VX1006 -**A record type is not partial.** Error. +A record type is not partial. This is an error. -The generator adds the schema and the row copies to the type itself, which C# allows only for a -`partial` type, nested in `partial` types. +The generator adds the schema and the row copies to the type itself, which C# only allows for a +`partial` type nested in `partial` types. ```csharp [VortexRecord] @@ -202,14 +211,15 @@ error VX1006: 'Plain' must be declared partial, as must every type that contains error VX1006: 'Outer.Inner' must be declared partial, as must every type that contains it, for the generator to implement IVortexRecord ``` -**Fix:** add `partial` to the type and to every type it is nested in. +To fix it, add `partial` to the type and to every type it is nested in. ## VX1007 -**A type cannot be a record.** Error. +A type cannot be a record. This is an error. A record is a concrete, non-generic class or struct whose schema is known when the program is -compiled, and which reading can construct. The message names which of these the type is not: +compiled, and which reading can construct. The message names which of these conditions the type +fails: ```csharp [VortexRecord] @@ -253,18 +263,18 @@ error VX1007: 'Local' cannot be a [VortexRecord]: a file-local type cannot be co error VX1007: 'Seeded' cannot be a [VortexRecord]: no constructor takes only members as its parameters, and none takes no parameter ``` -A static class is refused the same way: it has no rows. +A static class is refused the same way, since it has no rows. -**Fix:** for a generic or abstract type, declare a concrete record for each shape you store. For -the constructor, add one whose parameters are all members (matched by name, ignoring case), or a -parameterless one; a positional record always has one. +To fix it, declare a concrete record for each shape you store instead of a generic or abstract type. +For the constructor, add one whose parameters are all members (matched by name, ignoring case), or a +parameterless one. A positional record always has one. ## VX1008 -**A record member cannot be filled from a column.** Error. +A record member cannot be filled from a column. This is an error. Reading builds each row through a constructor whose parameters are members, then sets the other -members. A member that is a column but that neither the constructor nor a setter reaches cannot be +members. A member that is a column but that neither the constructor nor a setter can reach cannot be filled: ```csharp @@ -310,20 +320,20 @@ error VX1008: Member 'Id' of 'Derived' cannot be filled when rows are read: its error VX1008: Member 'Note' of 'Noted' cannot be filled when rows are read: it is required, and a member marked [VortexIgnore] has no value to set it to ``` -A private setter on the record's own member is fine: the generated code is part of the type. +A private setter on the record's own member is fine, since the generated code is part of the type. -**Fix:** give the member a setter or an `init` accessor, make it a constructor parameter, or mark -a computed member `[VortexIgnore]`. For a `required` member you ignore, remove `required`, or put +To fix it, give the member a setter or an `init` accessor, make it a constructor parameter, or mark a +computed member `[VortexIgnore]`. For a `required` member you ignore, remove `required`, or put `[SetsRequiredMembers]` on the constructor that reading uses. ## VX1009 -**A component of a group key is not a column.** Error. +A component of a group key is not a column. This is an error. -A group by takes a column of the scan, or a tuple of them, and each row falls into the group of its -values. A literal, or a value captured from outside the lambda, is the same for every row: it -groups nothing, and `GroupBy` refuses it with `ArgumentException` when it is built. The analyzer -flags it in the tuple, in method and in query syntax. +A group by takes a column of the scan, or a tuple of columns, and each row falls into the group of its +values. A literal, or a value captured from outside the lambda, is the same for every row, so it groups +nothing, and `GroupBy` refuses it with `ArgumentException` when it is built. The analyzer flags it in +the tuple, in both method and query syntax. ```csharp file.Scan().GroupBy(r => (r.City, 42)); // VX1009: component 2, '42', is an int @@ -333,18 +343,18 @@ group r by (r.Day, "all") into g // VX1009: component 2 select (g.Key.Day, g.Count()); ``` -**Fix:** group by the columns alone, `r => (r.City, r.Day)`. A constant that should appear in every -row of the result belongs to C# after the query. +To fix it, group by the columns alone, as in `r => (r.City, r.Day)`. A constant that should appear in +every row of the result belongs in C# code after the query. ## VX1010 -**A record does not take the values of a selection.** Error. +A record does not match the values of a selection. This is an error. A selection of several values is read through a record: `As()` after a `Select`, and -`AggAsync` for the answers of a whole scan. The record's members, in declaration order, -take the elements in order, one each, and a member is of its element's type or of its nullable -form. `As` and `AggAsync` check it when they are built and throw `VortexSchemaException`; the -analyzer reads the types of the tuple's elements in the lambda and flags it at compile time. +`AggAsync` for the answers of a whole scan. The record's members, in declaration order, take +the elements in order, one each, and each member has its element's type or its nullable form. `As` and +`AggAsync` check this when they are built and throw `VortexSchemaException`, and the analyzer reads +the types of the tuple's elements in the lambda to flag the problem at compile time. ```csharp [VortexRecord] @@ -359,23 +369,22 @@ file.Scan().GroupBy(r => r.City) .As(); // VX1010: CityDays has 2 members and the selection 3 elements ``` -A selection kept in a variable and read in another method is checked when `As` runs: the analyzer -sees only a `Select` it is called on. +A selection kept in a variable and read in another method is checked when `As` runs, because the +analyzer only sees a `Select` it is called on. -**Fix:** declare the member of the element's type, `long Days`, or of its nullable form, `long?`: -a count is a `long`, an average a `double?`, a key component or a minimum the column's type. One -member per element, in the order of the tuple. +To fix it, declare each member with its element's type, `long Days`, or its nullable form, `long?`. +A count is a `long`, an average a `double?`, and a key component or a minimum has the column's type. +Declare one member per element, in the order of the tuple. ## VX1011 -**Several values are read without a record.** Error, beside the compiler's. +Several values are read without a record. This is an error, reported beside the compiler's own. -Several values have no .NET type until a record gives them one, so a `Select` of several values -returns an `Aggregation`, which is not enumerable, and `AggAsync` of several answers needs the -record they go into. The compiler refuses both, CS8411 for an `await foreach`, CS0411 for an -`AggAsync` whose type it cannot infer or for the operators of `System.Linq` asked of an -`Aggregation`, with a message that does not say what to do; the analyzer reports this one beside -it. +Several values have no .NET type until a record gives them one. So a `Select` of several values +returns an `Aggregation`, which is not enumerable, and `AggAsync` of several answers needs the record +they go into. The compiler refuses both cases, with CS8411 for an `await foreach`, and CS0411 for an +`AggAsync` whose type it cannot infer or for `System.Linq` operators applied to an `Aggregation`, but +its message does not say what to do. The analyzer reports this diagnostic beside it. ```csharp await foreach (var (city, count) in file.Scan() @@ -386,6 +395,7 @@ var (min, max) = await file.Scan() .AggAsync(a => (a.Min(r => r.Day), a.Max(r => r.Day))); // CS0411 and VX1011 ``` -**Fix:** declare a `[VortexRecord]` whose members take the values in order, and read the selection +To fix it, declare a `[VortexRecord]` whose members take the values in order, and read the selection through it: `.As()`, enumerated as batches or with `ToRecordsAsync()`, or -`AggAsync(a => (…))`. A selection of one value, `Select(g => g.Count())`, comes as itself. +`AggAsync(a => (…))`. A selection of one value, such as `Select(g => g.Count())`, comes back as +itself. diff --git a/docs/guide/editions.md b/docs/guide/editions.md index 4f91179f..fd47f83a 100644 --- a/docs/guide/editions.md +++ b/docs/guide/editions.md @@ -21,7 +21,7 @@ foreach (VortexEdition edition in Enum.GetValues()) ``` An edition is a frozen set of components (encodings, layouts, dtypes, aggregates) that every -implementation claiming it must understand. Writing to an edition is the promise that a reader of +implementation claiming it must understand. Writing to an edition is a promise that any reader of that vintage can read the file. ## What this build knows @@ -40,13 +40,13 @@ Default core2026.08.3, Newest core2026.08.3, ReadFloor core2025.05.0 | `Core20260802` | core2026.08.2 | 0.85.0 and later | | `Core20260803` | core2026.08.3 | 0.85.0 and later | -`VortexEditions.Default` is what a write targets unless told otherwise: the edition the most -deployed Rust reader accepts, and the first that carries `vortex.uuid`, so that a `Guid` member -writes. It equals `Newest` today, and will not stay equal. `ReadFloor` is the oldest edition this -library reads and will keep reading. `VortexEditions.Name` and `MinimumRustVersion` give the other -two columns of the table. +`VortexEditions.Default` is what a write targets unless told otherwise. It is the edition the most +deployed Rust reader accepts, and the first one that carries `vortex.uuid`, so that a `Guid` member +can be written. It equals `Newest` today, but that will not always be the case. `ReadFloor` is the +oldest edition this library reads and will keep reading. `VortexEditions.Name` and +`MinimumRustVersion` give the other two columns of the table. -`IntroducedIn` and `Contains` say where a component comes from: +`IntroducedIn` and `Contains` tell you where a component comes from: | component | introduced in | in the read floor | |---|---|---| @@ -57,8 +57,8 @@ two columns of the table. | dtype `vortex.uuid` | core2026.08.3 | no | | array `vortex.patched` | no edition | no | -A component in no edition can be read only by something that knows it, which is why the target -exists. +A component that belongs to no edition can only be read by a reader that happens to know it, which +is why the target exists. ## What an older target costs @@ -66,20 +66,20 @@ The same million rows, written to each edition: | target | bytes | zone maps | the file reads as | `Day >= 900` reads | |---|---|---|---|---| -| core2025.05.0 | 1 544 292 | 0 | core2025.05.0 | 123 of 123 blocks | -| core2025.06.0 | 1 499 396 | 0 | core2025.06.0 | 123 of 123 blocks | -| core2025.10.0 | 1 499 396 | 0 | core2025.06.0 | 123 of 123 blocks | -| core2026.08.0 to core2026.08.3 | 1 508 212 | 8 264 | core2026.08.0 | 14 of 123 blocks | +| core2025.05.0 | 1 544 268 | 0 | core2025.05.0 | 123 of 123 blocks | +| core2025.06.0 | 1 499 500 | 0 | core2025.06.0 | 123 of 123 blocks | +| core2025.10.0 | 1 499 500 | 0 | core2025.06.0 | 123 of 123 blocks | +| core2026.08.0 to core2026.08.3 | 1 508 316 | 8 264 | core2026.08.0 | 14 of 123 blocks | -**Before core2026.08.0 a file carries no zone maps.** The zoned layout arrived in that edition, and -a file written for an older one prunes nothing by block: the filter that reads 14 blocks of the -default file reads all 123 of the old one. The encodings barely change: the file written for -core2025.05.0, which lacks the progression and zstd, is 45 KB larger, and from core2025.06.0 on the -only difference is the zone maps. +Before core2026.08.0, a file carries no zone maps. The zoned layout arrived in that edition, so a +file written for an older one cannot prune by block: the filter that reads 14 blocks of the default +file reads all 123 of the old one. The encodings barely change. The file written for core2025.05.0, +which lacks sequences and zstd, is 45 KB larger, and from core2025.06.0 on the only difference is the +zone maps. -`file.Edition` is the oldest edition that contains every component the file declares, so the -default write of these rows opens in any reader of core2026.08.0, Vortex 0.84.0, although it -targeted core2026.08.3: nothing in it needed more. +`file.Edition` is the oldest edition that contains every component the file declares. So the default +write of these rows opens in any reader of core2026.08.0, Vortex 0.84.0, even though it targeted +core2026.08.3: nothing in it needed more. ## What refuses to write @@ -90,23 +90,24 @@ a Guid member under core2026.08.2: Kind DType, ComponentId vortex.uuid Unsupported Vortex component: dtype 'vortex.uuid'. The write targets edition core2026.08.2, which does not contain it; it was introduced in core2026.08.3. ``` -`VortexUnsupportedException.Kind` says which table the id belongs to, and so which edition -introduced it: `Array` or `Layout` for an encoding, `DType`, `Aggregate`, `Compression`, -`Encryption`, `Index` for an index kind, and `Feature` for something the library does not offer for -this input, such as an append over a layout it cannot continue. `ComponentId` is the id as the file -spells it. An encoding the target lacks is not refused: the writer chooses among what the edition -contains, and the report says what it chose. +`VortexUnsupportedException.Kind` says which table the id belongs to, and so which edition introduced +it: `Array` or `Layout` for an encoding, `DType`, `Aggregate`, `Compression`, `Encryption`, `Index` +for an index kind, and `Feature` for something the library does not offer on this input, such as an +append over a layout it cannot continue. `ComponentId` is the id as the file spells it. An encoding +the target lacks is not refused: the writer simply chooses among what the edition contains, and the +report says what it chose. ## Watch out -* **The default is chosen per release**, and changing it is a major version change for this library, +* The default is chosen per release, and changing it is a major version change for this library because it changes who can read the output. Set `TargetEdition` when that matters to you. -* **An edition constrains the writer, not the reader.** This library reads everything from the floor - upwards, and refuses a component it does not know with `VortexUnsupportedException` naming it, +* An edition constrains the writer, not the reader. This library reads everything from the floor + upwards, and refuses a component it does not know with `VortexUnsupportedException`, naming it, when a scan first needs it ([limits.md](limits.md)). -* A schema may name an extension registered on the session: it belongs to no edition, and a reader - needs the same registration. -* [90-registry.md](../design/90-registry.md) lists every component and the edition that brought it. +* A schema may name an extension registered on the session. Such an extension belongs to no edition, + and a reader needs the same registration. +* [90-registry.md](../design/90-registry.md) lists every component and the edition that introduced + it. ## Run it diff --git a/docs/guide/encoded-forms.md b/docs/guide/encoded-forms.md index 894a6341..52654397 100644 --- a/docs/guide/encoded-forms.md +++ b/docs/guide/encoded-forms.md @@ -1,7 +1,7 @@ # Encoded forms -Read a column the way the file stores it, as a dictionary, as runs or as one constant, without -decoding it; and know when `GroupBy` does that for you. +Read a column the way the file stores it, as a dictionary, as runs or as a single constant, without +decoding it. This page also shows when `GroupBy` does that for you. ```csharp double[] totalByCity = new double[Demo.Cities.Length]; @@ -35,17 +35,17 @@ await foreach (var (_, celsius, city) in file.Scan()) } ``` -The city is looked up once per run instead of once per row. `IndexOf` compares a UTF-8 span with -the eight city names. +The city is looked up once per run instead of once per row. `IndexOf` compares a UTF-8 span with the +eight city names. ## What the demonstration files hold -`Column.Encoding` says how a delivered column holds its values after the scan's pruning and -before anything decodes them. The sample counts it for every batch of both files: +`Column.Encoding` says how a delivered column holds its values, after the scan's pruning and before +anything decodes them. The sample counts it for every batch of both files: | batches of | column | canonical | dictionary | run-end | constant | |---|---|---|---|---|---| -| readings | `Day` | 1 | 0 | 16 | 0 | +| readings | `Day` | 0 | 0 | 16 | 1 | | readings | `Celsius` | 1 | 16 | 0 | 0 | | readings | `City` | 0 | 0 | 17 | 0 | | visits | `Referrer` | 0 | 5 | 0 | 0 | @@ -53,27 +53,29 @@ before anything decodes them. The sample counts it for every batch of both files | readings, `Where(r => r.Day == 500)` | `Day`, `City` | 1 each | 0 | 0 | 0 | | readings, `Where(r => r.Day == 500)` | `Celsius` | 0 | 1 | 0 | 0 | -`Day` runs for a thousand rows and `City` for seven, so both come out as runs; `Celsius` takes 400 -values, and its dictionary per batch holds 393 of them, one being the null. `Referrer` is a -dictionary of thirteen URLs. No column of these files is constant. Only these four forms are -surfaced: bit-packing, frame of reference, ALP, FSST and the general-purpose compressors are decoded -by the scan and arrive as `Canonical`. The writer chooses per column and per chunk, and its -`WriteReport` lists what it chose ([writer-options.md](writer-options.md)). +`Day` keeps the same value for a thousand rows and `City` for seven, so both come out as runs. The last +576 rows of `Day` all hold day 999, and that small tail chunk is a constant. `Celsius` takes 400 +values, and its dictionary holds 393 of them per batch, one of which is the null. `Referrer` is a +dictionary of thirteen URLs. Only these four forms are surfaced. Bit-packing, frame of reference, +ALP, FSST and the general-purpose compressors are decoded by the scan and arrive as `Canonical`. The +writer chooses per column and per chunk, and its `WriteReport` lists what it chose +([writer-options.md](writer-options.md)). ## The views | call | returns | on any other encoding | |---|---|---| -| `AsDictionary()` | `DictionaryView`: `Codes`, one `uint` per row; `Values`, the distinct values as a `Column` in code order; `Cardinality` | `InvalidOperationException` | -| `AsRunEnd()` | `RunEndView`: `RunEnds`, the exclusive end of each run in the batch; `Values`, one per run; `RunCount` | `InvalidOperationException` | +| `AsDictionary()` | `DictionaryView`: `Codes`, one `uint` per row, `Values`, the distinct values as a `Column` in code order, and `Cardinality` | `InvalidOperationException` | +| `AsRunEnd()` | `RunEndView`: `RunEnds`, the exclusive end of each run in the batch, `Values`, one per run, and `RunCount` | `InvalidOperationException` | | `AsConstant()` | the one value | `InvalidOperationException` | -| `Canonical()` | the column decoded, once, into contiguous memory | returns the column itself | +| `Canonical()` | the column decoded once into contiguous memory | returns the column itself | -The views are those of §5.4 of [14-public-api.md](../design/14-public-api.md). Read `Encoding` -before asking for one. `Values`, the indexer and `GetString` on an encoded -column decode it first, once per batch, which is what a consumer that does not care gets. A null -row of a dictionary column has code 0 and its validity bit clear, and the dictionary's own values -may hold the null too: that one of `Celsius` does. +These views are described in +[the public API design](../design/14-public-api.md#54-encoded-views-and-the-selection). Check +`Encoding` before asking for one. `Values`, the indexer and `GetString` decode an encoded column +first, once per batch, which is what a consumer that does not care about encodings gets. A null row +of a dictionary column has code 0 and its validity bit cleared, and the dictionary's own values may +hold the null too, as they do for `Celsius`. Grouping by code, over the dictionary of `Referrer`: @@ -93,8 +95,8 @@ for (int i = 0; i < codes.Length; i++) } ``` -An array indexed by code, one increment per row, no hash; the sample then names each code once per -batch with `dict.Values.GetString(code)`. +This is an array indexed by code, one increment per row and no hashing. The sample then names each +code once per batch with `dict.Values.GetString(code)`. ## And why you rarely write it @@ -105,36 +107,35 @@ Scan byCity = file.Scan() .As(); ``` -The group by does the dictionary case by code, the run-end case by run and the canonical case by -hash, and merges the chunks by value, so the answer is one row per city whatever the batches held. -It is one line, it treats a null key as a group of its own, and it takes the session's parallelism +The group by handles the dictionary case by code, the run-end case by run and the canonical case by +hash, then merges the chunks by value, so the answer has one row per city whatever the batches held. +It is one line, it treats a null key as a group of its own, and it uses the session's parallelism ([aggregates.md](aggregates.md)). Measured against the loops above: ``` -City by run, by hand 8 groups, 18.2 ms -GroupBy(r => r.City) 8 groups, 55.9 ms +City by run, by hand 8 groups, 14.9 ms +GroupBy(r => r.City) 8 groups, 7.8 ms 8 of 8 cities agree, Paris 3659129.4; the GroupBy decoded 1 blocks -Referrer by code, by hand 14 groups, 0.9 ms -GroupBy(v => v.Referrer) 14 groups, 0.7 ms +Referrer by code, by hand 14 groups, 1.1 ms +GroupBy(v => v.Referrer) 14 groups, 0.5 ms 14 of 14 groups agree, null referrers 25000; the GroupBy decoded 13 blocks ``` -Both give the same answers. On the dictionary key the operator wins; on the run-end key, runs of -seven rows, the loop by hand won by a factor of three in this run. Write the loop when a -measurement says so, not before. "Decoded" counts the blocks where a column reached the canonical -form: the one block of readings whose `Celsius` arrives canonical, and all 13 of the visits, whose +Both give the same answers, and the operator wins on both keys. Write the loop yourself only when a +measurement says it pays. "Decoded" counts the blocks where a column reached the plain form: the one +block of readings whose `Celsius` arrives canonical, and all 13 blocks of the visits, whose `DurationMs` always does. ## Watch out -* **The forms are per batch.** A dictionary's codes, and a run's values, mean something only in - the batch that holds them: map them through that batch's `Values`, as both loops do. -* **A filter changes the form.** Compacting a batch decodes its run-end columns, which is why - `Day` and `City` come back canonical under `Day == 500` while `Celsius` stays a dictionary. - With `ScanOptions.Compact = false` the blocks arrive whole ([selection.md](selection.md)). -* **A view is borrowed** like the column it comes from: valid inside the loop body only. +* The forms are per batch. A dictionary's codes and a run's values only mean something in the batch + that holds them, so map them through that batch's `Values`, as both loops do. +* A filter changes the form. Compacting a batch decodes its run-end columns, which is why `Day` and + `City` come back canonical under `Day == 500` while `Celsius` stays a dictionary. With + `ScanOptions.Compact = false` the blocks arrive whole ([selection.md](selection.md)). +* A view is borrowed like the column it comes from, and is valid inside the loop body only. -The figures come from one run of the sample on the demonstration files; each timing is the best of +The figures come from one run of the sample on the demonstration files. Each timing is the best of three passes. ## Run it diff --git a/docs/guide/errors.md b/docs/guide/errors.md index c0cd12d6..ffb8d4db 100644 --- a/docs/guide/errors.md +++ b/docs/guide/errors.md @@ -1,6 +1,6 @@ # Errors -The exceptions this library throws, what each one means, and which are your fault. +The exceptions this library throws, what each one means, and which ones are your fault. ```csharp await Show("512 zero bytes", async () => @@ -25,8 +25,8 @@ a record member the file has no column for -> VortexSchemaException: Member 'Per a key cursor over a column nothing orders -> VortexUnsupportedException (Index, vorticity.sorted.runs.v1): Unsupported Vortex component: index 'vorticity.sorted.runs.v1'. 'Celsius' has no key source (SortedColumn: the file statistics say the column is not sorted; SortedRuns: the file carries no index directory; …). A cursor over an unindexed column would have to hold the column to sort it, which this library refuses. Write the file with IndexPolicy.SortedRuns for that column, … ``` -`Show` runs the call and prints the exception's type and message; the sample runs sixteen cases -this way. +`Show` runs the call and prints the exception's type and message. The sample runs sixteen cases this +way. ## The hierarchy @@ -34,31 +34,45 @@ Every exception the library throws on purpose derives from `VortexException`: | | means | whose fault | |---|---|---| -| `VortexFormatException` | the bytes are not what they claim to be, or a limit says no | the file's, or yours for pointing at the wrong bytes; never worth retrying | -| `VortexUnsupportedException` | the input is well formed and asks for something this build does not do | nobody's: a newer library, another edition, or an index the file lacks | +| `VortexFormatException` | the bytes are not what they claim to be, or a limit says no | the file's, or yours for pointing at the wrong bytes. Never worth retrying | +| `VortexUnsupportedException` | the input is well formed but asks for something this build does not do | nobody's: it needs a newer library, another edition, or an index the file lacks | | `VortexSchemaException` | a record, a builder, a column request or a literal does not fit the schema | yours, and the message says what to change | +| `VortexMemoryException` | a query needs more memory than its `QueryMemoryBudget` grants and nothing it holds can spill, or more scratch than its `ScratchBudget` or the disk allows | the workload's: raise the budget, lower the degree, or narrow the query | -What stays outside it is plain .NET: an argument out of range is an `ArgumentException`, misuse is -an `InvalidOperationException`, a missing file is a `FileNotFoundException`. The object store of -[object-store.md](object-store.md) adds two of its own, `ObjectStoreException` and -`ObjectNotFoundException`, both `VortexException`s too. +A `VortexMemoryException` names the operator, its groups and the bytes it asked for, and the query +has already given back everything it held, scratch files included ([threads.md](threads.md) +covers the budgets). A writer whose policy requires an index that could not be built throws a plain +`VortexException` naming the index and the reason, before the file is completed +([indexes.md](indexes.md)). + +Everything else is plain .NET. An argument out of range is an `ArgumentException`, misuse is an +`InvalidOperationException` and a missing file is a `FileNotFoundException`. + +`Vorticity.Dataset` adds four exceptions of its own, all derived from `VortexException`: + +| | means | +|---|---| +| `ObjectStoreException` | the object store refused or failed a request ([object-store.md](object-store.md)) | +| `ObjectNotFoundException` | a key is not in the store, often because a handle outlived the retention window and should refresh | +| `TornCommitException` | a writer died mid-commit on `FileObjectStore`, and `VortexDataset.RemoveTornCommitAsync` clears it ([dataset-maintenance.md](dataset-maintenance.md)) | +| `DatasetIntegrityException` | a safety check found that a change would lose rows, double them or break their order, so nothing was committed. Running the same change again will hit it again | ## `VortexFormatException` -A file that does not parse: bad offsets, a truncation, a field out of range, a nesting deeper than -a cap allows, a block that would decode to more than `MaxDecompressedSize`. It is the half of the -promise about hostile input that says a malformed file produces this exception, or -`VortexUnsupportedException`, and never an out-of-bounds read, an unbounded allocation or a hang. +This is a file that does not parse: bad offsets, a truncation, a field out of range, nesting deeper +than a cap allows, or a block that would decode to more than `MaxDecompressedSize`. It is half of +the library's promise about hostile input: a malformed file produces this exception or a +`VortexUnsupportedException`, never an out-of-bounds read, an unbounded allocation or a hang. [limits.md](limits.md) lists the caps and shows a thousand damaged files producing nothing else. -A torn append is one too, when you ask for it: by default an open falls back to the last whole -version of the file, and `VortexTornTailPolicy.Refuse` makes it throw instead. See +A torn append raises it too, if you ask. By default an open falls back to the last whole version of +the file, and `VortexTornTailPolicy.Refuse` makes it throw instead. See [append-and-repair.md](append-and-repair.md). ## `VortexUnsupportedException` -The file, or the request, needs a component this build does not implement. The exception names -it, in two parts that together say which edition introduced it: +The file, or the request, needs a component this build does not implement. The exception names it +in two parts, which together tell you which edition introduced it: ```csharp catch (VortexUnsupportedException e) @@ -72,14 +86,15 @@ a uuid column written to an edition older than uuid -> VortexUnsupportedExceptio ``` `Kind` is a `ComponentKind`: `Array` or `Layout` for an encoding, `DType` for a column type, -`Aggregate` for a zone-map statistic, `Compression` or `Encryption` for a segment scheme, `Index` -for an index kind or an index a request needs, and `Feature` for something the library does not -offer on this input, such as an append over a layout it cannot resume. `ComponentId` is the id as -the file spells it. `VortexEditions.IntroducedIn(kind, id)` answers the edition question in code; -[editions.md](editions.md) says how to write for older readers. +`Aggregate` for a zone-map statistic, `Compression` or `Encryption` for a segment scheme, `Index` for +an index kind or an index a request needs, and `Feature` for something the library does not offer on +this input, such as an append over a layout it cannot resume. `ComponentId` is the id as the file +spells it. `VortexEditions.IntroducedIn(kind, id)` answers the edition question in code, and +[editions.md](editions.md) explains how to write for older readers. -An unknown encoding does not stop a file from opening: the refusal comes from the first block that -needs it, so a scan that does not read that column succeeds. [limits.md](limits.md) shows one. +An unknown encoding does not stop a file from opening. The refusal comes from the first block that +needs it, so a scan that does not read that column still succeeds. [limits.md](limits.md) shows an +example. ## `VortexSchemaException` @@ -93,11 +108,11 @@ a column the file does not have, by name -> VortexSchemaException: The file has a nullable builder over a non-nullable column -> VortexSchemaException: Column 'Day' is i32, which is not nullable; ask for a builder of int instead of int?. ``` -On the typed path most of these cannot be written at all: a literal of the wrong type does not +On the typed path most of these cannot even be written: a literal of the wrong type does not compile, and a record binds once, at the first sink of a scan. On the tool path, where names and -literals are data, the check happens when `Where` is called or the column is asked for, before -anything is read. [records.md](records.md#binding-a-record-to-a-file) says how members find their -columns. +literals are data, the check happens when `Where` is called or the column is requested, before +anything is read. [records.md](records.md#binding-a-record-to-a-file) explains how members find +their columns. ## The .NET ones @@ -112,26 +127,27 @@ a session disposed while one of its files is open -> InvalidOperationException: | | when | |---|---| -| `FormatException` | a filter string for the tool path, `VortexExpr.Parse`, that does not parse | -| `FileNotFoundException`, `IOException` | no file at the path, or one the operating system will not open | -| `ArgumentOutOfRangeException` | a row index below zero or past the last row; a negative option | +| `FormatException` | a filter string for the tool path (`VortexExpr.Parse`) does not parse | +| `FileNotFoundException`, `IOException` | there is no file at the path, or the operating system will not open it | +| `ArgumentOutOfRangeException` | a row index below zero or past the last row, or a negative option | | `InvalidOperationException` | a scan run twice, `Rows` by range and by indices on one scan, a key cursor on a filtered scan, a session disposed before its files, an option set after `VortexSession.Create` returned | -| `ObjectDisposedException` | an owned batch used after `Dispose`, a disposed file asked for what it no longer holds | -| `OperationCanceledException` | a token you passed was cancelled: see [cancel-work.md](cancel-work.md) | +| `ObjectDisposedException` | an owned batch used after `Dispose`, or a disposed file asked for something it no longer holds | +| `OperationCanceledException` | a token you passed was cancelled (see [cancel-work.md](cancel-work.md)) | ## Watch out -* **A row range past the end is clamped, a row index past the end throws.** +* A row range past the end is clamped, but a row index past the end throws. `Rows(new RowRange(0, rows + 10))` counts 1 000 000 rows on the demonstration file, without an exception. -* **A disposed `VortexFile` answers what its open captured and refuses the rest.** `RowCount`, - `Schema` and `Statistics` go on answering; `Identity` and `SegmentMap` throw - `ObjectDisposedException`, because they read the tail buffer the file returned to the pool. - `Scan()` still builds a scan, which throws `ObjectDisposedException` at its first read. - Treat a disposed file as gone. -* **Each scan is single-use.** Build a new one per question; building one costs nothing that is - worth caching. -* VX1003 and VX1004 catch some of these at compile time: see [diagnostics.md](diagnostics.md). +* A disposed `VortexFile` still answers what its open captured and refuses the rest. `RowCount`, + `Schema` and `Statistics` keep answering, while `Identity` and `SegmentMap` throw + `ObjectDisposedException` because they read the tail buffer the file returned to the pool. + `Scan()` still builds a scan, which throws `ObjectDisposedException` at its first read. Treat a + disposed file as gone. +* Each scan is single-use. Build a new one per question, since building one costs nothing worth + caching. +* The analyzers VX1003 and VX1004 catch some of these at compile time (see + [diagnostics.md](diagnostics.md)). ## Run it diff --git a/docs/guide/filter-rows.md b/docs/guide/filter-rows.md index 1530ea86..61b55318 100644 --- a/docs/guide/filter-rows.md +++ b/docs/guide/filter-rows.md @@ -1,6 +1,6 @@ # Filter rows -Push a predicate down so that whole blocks are never read, and see what it will cost before it +Push a predicate down so that whole blocks are never read, and see what a filter will cost before it runs. ```csharp @@ -26,23 +26,25 @@ ScanStatistics stats = scan.Statistics; ``` ``` -plan: 14 of 123 blocks, 15 segments, 217028 bytes +plan: 14 of 123 blocks, 15 segments, 217052 bytes zone map pruned 109 blocks, reading 8172 bytes to decide 6116 rows, from day 900 -ran: 14 blocks decoded, 109 pruned, 12 requests, 208856 bytes +ran: 14 blocks decoded, 109 pruned, 12 requests, 208880 bytes ``` -**The filter is exact.** What comes back is the matching rows and nothing else, compacted into -the batches; it is not a hint to check again. +The filter is exact. What comes back is the matching rows and nothing else, compacted into the +batches. It is not a hint you need to check again. ## The lambda is not a delegate over rows -`r` is a `Probe`, and `r.Day` a `Sym`: its operators record a predicate instead of -comparing values. The lambda runs once, when `Where` is called, and what it returns is the plan; -the sample counts the calls of a lambda over a million rows and prints `the lambda ran 1 time`. A -captured local such as `max` is read then. An optional filter is a second `Where`, joined to the -first by `&`; a filter built from optional parts starts from `Predicate.All`, the neutral element -of `&`, and `ToString()` shows what was built: +`r` is a `Probe` and `r.Day` is a `Sym`, whose operators record a predicate instead of +comparing values. The lambda runs once, when `Where` is called, and what it returns is the plan. The +sample counts the calls of a lambda over a million rows and prints `the lambda ran 1 time`. A +captured local such as `max` is read at that moment. + +Each call to `Where` adds a condition, joined to the previous ones with `&`, which is how the +optional filter above works. For a filter built from optional parts in one lambda, start from +`Predicate.All`, the neutral element of `&`. `ToString()` shows what was built: ```csharp Scan optional = file.Scan().Where(r => @@ -73,21 +75,21 @@ built from optional parts: Day >= 500 and City = 'Lyon' ``` What compiles is exactly what the scan can push down. [08-semantics.md](../design/08-semantics.md) -says what each form means, and §5.2 of [14-public-api.md](../design/14-public-api.md) why the -algebra is built this way: +defines what each form means, and [the symbolic algebra](../design/14-public-api.md#52-the-symbolic-algebra) +explains why it is built this way. | written | meaning | |---|---| | `r.Day >= 900`, `r.City == "Paris"` | comparison with a literal of the column's type | -| `a && b`, `a \|\| b`, `!a` | logic; `&&` and `\|\|` evaluate both sides, as `&` and `\|` | -| `r.Day.In(1, 2, 3)`, `r.Celsius.Between(10.0, 20.0)` | membership, an inclusive range | +| `a && b`, `a \|\| b`, `!a` | logic, where `&&` and `\|\|` always evaluate both sides, like `&` and `\|` | +| `r.Day.In(1, 2, 3)`, `r.Celsius.Between(10.0, 20.0)` | membership, and an inclusive range | | `r.Celsius == null`, `r.Celsius.IsNull`, `IsNotNull` | nullity | | `r.City.StartsWith("L")`, `Contains`, `Like("P_r%")` | text | | `v.Pages.Contains(7)` | a list holds a value | | `v.Origin.Country.In("FR", "BE")` | a field of a nested record | | `r.High > r.Low`, on a record with two such columns | two columns of the same type | -And what does not compile, with the compiler's own words: +And here is what does not compile, in the compiler's own words: ``` r.Day >= "900" CS0019: Operator '>=' cannot be applied to operands of type 'Sym' and 'string' @@ -98,12 +100,12 @@ r.City.Length > 3 CS1061: 'Sym' does not contain a definition fo ## What happens, in order -The file statistics may settle the predicate without a read. The zone maps, a minimum, maximum and -null count per block, prune whole blocks; text columns carry truncated bounds too -(`StringBoundBytes`, 16 by default). An index, if the writer built one, prunes more -([indexes.md](indexes.md)). The kernel decides the rest value by value, on the encoded form where -the encoding allows it, and the batch is compacted to the rows that passed, or handed over whole -with a selection ([selection.md](selection.md)). +The file statistics may settle the predicate without reading anything. Then the zone maps, which +hold a minimum, a maximum and a null count per block, prune whole blocks. Text columns carry +truncated bounds too (`StringBoundBytes`, 16 by default). An index, if the writer built one, prunes +further ([indexes.md](indexes.md)). The kernel decides the remaining rows value by value, on the +encoded form when the encoding allows it. The batch is then compacted to the rows that passed, or +handed over whole with a selection ([selection.md](selection.md)). ## When pruning does nothing @@ -111,53 +113,54 @@ Counts from the sample, each with its plan: | predicate | rows | live blocks | requests | bytes | how the count was answered | |---|---|---|---|---|---| -| `r.Day >= 900` | 100 000 | 14 of 123 | 1 | 428 | exact from the structures | -| `r.Day.In(1, 2, 3)` | 3 000 | 1 of 123 | 1 | 412 | exact from the structures | -| `r.Celsius == null` | 20 000 | 123 of 123 | 0 | 0 | every block proven from its null count | +| `r.Day >= 900` | 100 000 | 14 of 123 | 1 | 428 | exactly, from the structures | +| `r.Day.In(1, 2, 3)` | 3 000 | 1 of 123 | 1 | 412 | exactly, from the structures | +| `r.Celsius == null` | 20 000 | 123 of 123 | 0 | 0 | every block proven by its null count | | `r.Celsius > 45.0` | 122 500 | 123 of 123 | 17 | 1 150 428 | 123 blocks evaluated | | `r.City == "Paris"` | 125 006 | 123 of 123 | 17 | 336 956 | 123 blocks evaluated | | `r.City.StartsWith("L")` | 249 999 | 123 of 123 | 17 | 336 956 | 123 blocks evaluated | -The rows are in `Day` order, so a block's bounds settle `Day >= 900` for 109 blocks of 123, and the -count reads one small segment of `Day`, the one that places the boundary. The zone maps it decides -on are already on the file, kept there by the scans before it, which is also why `== null` reads +The rows are in `Day` order, so the bounds of 109 blocks out of 123 settle `Day >= 900` on their own, +and the count reads one small segment of `Day`, the one that holds the boundary. The zone maps it +relies on are already in memory, kept by the scans before it, which is also why `== null` reads nothing at all. -`Celsius` walks its whole range inside every block and every city appears in every block, so no -block can be excluded and each value is compared. **This is a property of the data, not of the -predicate**: write the rows in the order of the column you filter on, or at least clustered by it. + +`Celsius` covers its whole range inside every block, and every city appears in every block, so no +block can be excluded and each value has to be compared. This is a property of the data, not of the +predicate. Write the rows in the order of the column you filter on, or at least cluster them by it. ## Before and after -`ExplainAsync` reads the statistics and zone maps, and of the data at most the few segments of a -sorted column it searches, and returns a `ScanPlan`: the live blocks, the segments and bytes they -need, and a `PruningStep` per structure consulted with what it pruned and what consulting it cost. -`Statistics`, read after the sink, says what the scan did. Above, the plan counts 15 segments and -217 028 bytes, the 8 172 bytes of zone maps it consulted among them, and the scan that followed read -the other 12 segments, 208 856 bytes: the file keeps the zone maps a plan or a scan has read, and -the scan reads each segment once, however many live blocks share it +`ExplainAsync` reads the statistics and zone maps, plus at most the few segments of a sorted column it +searches, and returns a `ScanPlan`: the live blocks, the segments and bytes they need, and one +`PruningStep` per structure consulted, with what it pruned and what consulting it cost. `Statistics`, +read after the sink, says what the scan actually did. Above, the plan counts 15 segments and +217 052 bytes, including the 8 172 bytes of zone maps it consulted. The scan that followed read the +other 12 segments, 208 880 bytes. The file keeps the zone maps a plan or a scan has read, and the +scan reads each segment once, however many live blocks share it ([statistics-and-pruning.md](statistics-and-pruning.md)). -The cheapest questions need no scan at all: `CountAsync` and `AnyAsync` answer from the structures -wherever they suffice, and `file.MayMatch(r => r.Day >= 5000)` compares the predicate with -the file statistics alone and answers `False` here, reading nothing. +The cheapest questions need no scan at all. `CountAsync` and `AnyAsync` answer from the structures +whenever they are enough, and `file.MayMatch(r => r.Day >= 5000)` compares the predicate +with the file statistics alone. It answers `False` here and reads nothing. ## Watch out -* **Nulls answer unknown, and only true keeps a row.** `!(r.Celsius > 45.0)` keeps 857 500 rows, +* Nulls answer unknown, and only a true answer keeps a row. `!(r.Celsius > 45.0)` keeps 857 500 rows, not 877 500: the 20 000 nulls are kept by neither the predicate nor its negation. `== null`, `IsNull` and `IsNotNull` never answer unknown. -* **Do not branch on a predicate.** `r.Day >= 900 ? r.City == "Paris" : r.City == "Lyon"` compiles - and always takes the second branch (125 000 rows, all of Lyon), and so does an `if` on a - predicate inside the lambda: a predicate is never true while the lambda runs. Branch on captured +* Do not branch on a predicate. `r.Day >= 900 ? r.City == "Paris" : r.City == "Lyon"` compiles and + always takes the second branch (125 000 rows, all of Lyon), and so does an `if` on a predicate + inside the lambda, because a predicate is never true while the lambda runs. Branch on captured values, as the optional parts above do, and combine predicates with `&`, `|` and `!`. -* **A literal is compared as it is, not rounded to the column.** `r.At <= DateTime.UtcNow` on a +* A literal is compared as it is, never rounded to the column. `r.At <= DateTime.UtcNow` on a microsecond timestamp compares against the stored microsecond at or below the instant, so the boundary row is neither gained nor lost, and `r.At == DateTime.UtcNow` matches nothing unless the - instant falls on a whole microsecond. A decimal with more digits than the column's scale is held - the same way. -* **A filter names members of the record.** To filter on a column, the record you scan with names - it ([project-columns.md](project-columns.md)). -* **A scan is single-use**, but `ExplainAsync` may be called before its sink, as here. + instant falls exactly on a microsecond. A decimal with more digits than the column's scale is + handled the same way. +* A filter names members of the record. To filter on a column, the record you scan with must name it + ([project-columns.md](project-columns.md)). +* A scan is single-use, but `ExplainAsync` may be called before its sink, as here. The figures come from one run of the sample on the demonstration file of a million rows. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 8d31f1f3..986f74be 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -1,31 +1,31 @@ # Getting started -Declare a record, write a hundred thousand rows, read them back, take a mean and count a filter. -One file of code, a few minutes, and the only page that shows both directions. +This page declares a record, writes a hundred thousand rows, reads them back, computes a mean and +counts the rows of a filter. It takes one file of code and a few minutes, and it is the only page +that shows both directions at once. ## Reference the library -The library ships on nuget.org. Reference its packages from a project that targets .NET 11; the -generator is a build-time dependency, which `PrivateAssets` keeps out of whatever your project -packs: +The packages ship on nuget.org. Reference them from a project that targets .NET 11. The generator +only runs at build time, and `PrivateAssets` keeps it out of whatever your project packs: ```xml - - + + ``` -`Vorticity` is the whole format: opening, scanning, filtering, aggregating, indexes and the -writer, with `System.IO.Hashing` and `Vorticity.Zstd`, this repository's managed Zstandard, as its -only dependencies. `Vorticity.Generators` runs at build time only: it turns a `[VortexRecord]` -type into the code that reads and writes it, and ships the analyzers. The two other packages are -separate subjects, both experimental, and referenced the same way at the same version as the core: -`Vorticity.Dataset` for a versioned dataset over an object store ([datasets.md](datasets.md)), -`Vorticity.RowEncoding` for byte-sortable keys ([row-keys.md](row-keys.md)). +`Vorticity` is the whole format: opening, scanning, filtering, aggregating, indexes and the writer. +Its only dependencies are `System.IO.Hashing` and `Vorticity.Zstd`, the managed Zstandard of this +repository. `Vorticity.Generators` turns a `[VortexRecord]` type into the code that reads and writes +it, and ships the analyzers. Two more packages cover separate subjects. Both are experimental and +referenced the same way, at the same version as the core: `Vorticity.Dataset` for a versioned +dataset over an object store ([datasets.md](datasets.md)), and `Vorticity.RowEncoding` for +byte-sortable keys ([row-keys.md](row-keys.md)). -Everything that touches bytes is asynchronous: a file is opened with `await`, a scan is consumed -with `await foreach`, a writer is fed with `WriteAsync`. There is no synchronous path to look for. +Everything that touches bytes is asynchronous. A file is opened with `await`, a scan is consumed with +`await foreach`, and a writer is fed with `WriteAsync`. There is no synchronous variant to look for. ## Declare a record @@ -34,11 +34,11 @@ with `await foreach`, a writer is fed with `WriteAsync`. There is no synchronous public partial record struct Reading(int Day, double? Celsius, string City); ``` -A record is a schema, not a row: its members are the columns, in declaration order, under their -own names. `double?` makes `Celsius` a nullable column; `int` and `string` are not nullable. The -type must be `partial`, because the generator adds the `IVortexRecord` implementation and -the members the examples below use (`r.Celsius`, the deconstruction of a batch). -[records.md](records.md) says what else a record can hold. +A record is a schema rather than a row. Its members are the columns, in declaration order and under +their own names. `double?` makes `Celsius` a nullable column, while `int` and `string` are not +nullable. The type must be `partial`, because the generator adds the `IVortexRecord` +implementation and the members the examples below rely on (`r.Celsius`, the deconstruction of a +batch). [records.md](records.md) lists everything a record can hold. ## Write a file @@ -60,8 +60,8 @@ await using (VortexFileWriter writer = VortexSession.Default.CreateWriter().Where(r => r.Celsius > 45.0 && r.City == " It prints: ``` -wrote 100000 rows in 157980 bytes +wrote 100000 rows in 157988 bytes struct{Day: i32, Celsius: f64?, City: utf8}, 100000 rows 3 batches, 100000 rows, 2000 without a temperature mean 30.00 degrees @@ -95,24 +95,27 @@ mean 30.00 degrees ## What happened -* **The rows became columns.** About 1.85 MB of values became a file of 157 980 bytes, because the - writer chose an encoding per column and per chunk instead of storing what it was handed. -* **The scan came back in batches, not rows**: 3 of them, one per chunk the writer made of the - file's 8 192-row blocks. `day`, `celsius` and `city` are `Column` values over the decoded - batch. They are borrowed: valid inside the loop body, and the compiler refuses to let one outlive - it ([scan-a-table.md](scan-a-table.md)). -* **`AverageAsync` ran inside the scan.** No batch reached the caller; the nulls were skipped for you. - An aggregate is an operator, not a loop you write ([aggregates.md](aggregates.md)). -* **The filter is not a delegate.** The lambda given to `Where` runs once, when the scan is built, - over a symbolic record: `r.Celsius` is a `Sym`, and `>` records a predicate instead of - comparing. A breakpoint inside the lambda sees `Sym`, not values, and hits once. What - compiles is exactly what the scan can push down ([filter-rows.md](filter-rows.md)). +The rows became columns. About 1.85 MB of values turned into a file of 157 988 bytes, because the +writer picked an encoding for each column and each chunk instead of storing what it was handed. + +The scan came back in batches, not rows: 3 of them, one per chunk the writer made out of the file's +8 192-row blocks. `day`, `celsius` and `city` are `Column` values over the decoded batch. They are +borrowed, valid inside the loop body only, and the compiler refuses to let one outlive it +([scan-a-table.md](scan-a-table.md)). + +`AverageAsync` ran inside the scan. No batch reached your code, and the nulls were skipped for you. +An aggregate is an operator of the scan, not a loop you write ([aggregates.md](aggregates.md)). + +The filter is not a delegate. The lambda given to `Where` runs once, when the scan is built, over a +symbolic record: `r.Celsius` is a `Sym`, and `>` records a predicate instead of comparing +values. A breakpoint inside the lambda sees `Sym` rather than values, and it hits once. +Whatever compiles is exactly what the scan can push down ([filter-rows.md](filter-rows.md)). ## Next -* [open-a-file.md](open-a-file.md): the ways in, and what an open reads. +* [open-a-file.md](open-a-file.md): the ways to open a file, and what an open reads. * [scan-a-table.md](scan-a-table.md): the loop above, and who owns what. -* [filter-rows.md](filter-rows.md): how not to read the rows you do not want. +* [filter-rows.md](filter-rows.md): how to avoid reading the rows you do not want. * [write-a-file.md](write-a-file.md): the writer, column by column. ## Run it diff --git a/docs/guide/how-it-works.md b/docs/guide/how-it-works.md index 184ceeea..d9caab82 100644 --- a/docs/guide/how-it-works.md +++ b/docs/guide/how-it-works.md @@ -25,35 +25,36 @@ await using VortexFile file = await VortexFile.OpenAsync(path); ``` ``` -written: 1000000 rows in blocks of 8192, 17 chunks of 65536, 16384, 576 rows; 1508212 bytes: data 1495924, statistics 200, zone maps 8264, indexes 0, footer 3824 - Day: RunEnd, Sequence +written: 1000000 rows in blocks of 8192, 17 chunks of 65536, 16384, 576 rows; 1508316 bytes: data 1495988, statistics 200, zone maps 8264, indexes 0, footer 3864 + Day: RunEnd, Constant Celsius: Dictionary, Alp City: RunEnd -opened: struct{Day: i32, Celsius: f64?, City: utf8}, 1000000 rows, 1508212 bytes, 54 segments, edition core2026.08.0 +opened: struct{Day: i32, Celsius: f64?, City: utf8}, 1000000 rows, 1508316 bytes, 54 segments, edition core2026.08.0 layout: vortex.struct of vortex.zoned(123 zones of 8192), vortex.zoned(123 zones of 8192), vortex.zoned(123 zones of 8192) -plan: 14 of 123 blocks live, 13 segments, 210980 bytes; ran: 100000 rows, 14 blocks decoded, 109 pruned; the lambda ran 1 time +plan: 14 of 123 blocks live, 13 segments, 211004 bytes; ran: 100000 rows, 14 blocks decoded, 109 pruned; the lambda ran 1 time ``` ## The session -Everything runs in a `VortexSession`: one memory pool that every batch, segment and builder buffer -comes from, one optional segment cache for all its files, one bound on the reads in flight, one -degree of parallelism, and the extension types it knows. It is thread-safe and meant to be shared -by a whole process. `VortexSession.Default`, which `VortexFile.OpenAsync(path)` and the samples -use, is immutable and takes no cores it was not given: its parallelism is 1. See +Everything runs inside a `VortexSession`. It owns one memory pool that every batch, segment and +builder buffer comes from, an optional segment cache shared by all its files, a bound on the reads +in flight, a degree of parallelism, and the extension types it knows. A session is thread-safe and +meant to be shared by the whole process. `VortexSession.Default`, which `VortexFile.OpenAsync(path)` +and the samples use, is immutable and takes no cores it was not given: its parallelism is 1. See [threads.md](threads.md). ## A file -A Vortex file is read from the end. Its last bytes are a marker and a postscript; before them the -footer, which holds the schema, the layout tree, the statistics of each column and the map of every -segment. One read of the tail, 64 KiB by default, gets all of it for an ordinary file, so opening -costs one round trip whatever the file's size; a path, where a read costs no round trip, reads its -last 8 KiB first and a larger footer next. A path is memory-mapped by the first scan that reads -data, so that every read after it is a page fault, unless the open read it whole, a file of 64 KiB -or less, whose scans that read serves. Everything else is **segments**: aligned -runs of bytes, 153 in this file, which the reader fetches only when something it needs lies in -them. +A Vortex file is read from the end. Its last bytes are a marker and a postscript, and before them +sits the footer, which holds the schema, the layout tree, the statistics of each column and the map +of every segment. For an ordinary file, one read of the tail (64 KiB by default) gets all of it, so +opening costs one round trip whatever the file's size. A local path, where a read costs no round +trip, reads its last 8 KiB first and a larger footer next. + +A path is memory-mapped by the first scan that reads data, so every later read is a page fault, +unless the open already read the whole file (64 KiB or less), in which case scans are served from +that read. Everything else is segments: aligned runs of bytes, 54 in this file (51 of data and 3 of +zone maps), which the reader fetches only when something it needs lies in them. ## A layout tree, and its encodings @@ -66,66 +67,70 @@ vortex.struct the columns vortex.flat bytes, in one segment, under one encoding ``` -`struct` splits the columns, which is what makes a projection cheap: a column no record member -asks for is a branch not walked. `zoned` holds the bounds of each block, which is what makes a -filter cheap: a block whose bounds exclude the predicate is never read. `chunked` is where appends -join. `flat` is a leaf, and its bytes are not raw values but a nest of reversible transforms, -chosen by the writer per chunk after pricing candidates on the data it held: `Day` became run ends, -`Celsius` a dictionary of ALP-encoded doubles, and the whole million readings 1.5 MB. +`struct` splits the columns, which makes projection cheap: a column no record member asks for is a +branch never walked. `zoned` holds the bounds of each block, which makes filtering cheap: a block +whose bounds exclude the predicate is never read. `chunked` is where appends join. `flat` is a leaf, +and its bytes are not raw values but a nest of reversible transforms, chosen per chunk by the writer +after pricing candidates on the data it held. Here `Day` became run ends (and a constant for the +last 576 rows), `Celsius` a dictionary of ALP-encoded doubles, and the whole million readings fit in +1.5 MB. ## A scan -`file.Scan()` is a builder. The lambda given to `Where` runs **once**, when the scan is -built, over a probe whose members are symbols: `r.Day >= 900` records a predicate instead of -comparing anything, which is why the sample counts one call for a million rows, and why what -cannot be pushed down does not compile. The predicate is planned before a byte of data is read: -the file's statistics may settle it, the zone maps prune blocks, an index prunes more, and the -plan lists the segments the live blocks need: here 14 of 123 blocks, 13 segments. - -Then a pipeline reads those segments, decodes one block per batch into buffers from the session's -pool, overlapping the next decode with your work on the current one, and hands you a -`Columns`: a `ref struct` over the decoded block, valid until the next `MoveNextAsync`, -which the compiler keeps inside the loop body. Nothing is copied for you and nothing is left to -dispose. What you want beyond borrowed columns is a sink you choose, and pay for: `ToBatchesAsync` -copies each batch once into buffers you own, `ToRecordsAsync` builds rows, and `CountAsync`, -`SumAsync`, `GroupBy(…).Select(…)` run as operators of the scan, on the encoded blocks, reading -from the statistics when those suffice. `ExplainAsync` is the plan before; `Statistics` is what -the scan did, after. See [scan-a-table.md](scan-a-table.md) and +`file.Scan()` is a builder. The lambda given to `Where` runs once, when the scan is built, +over a probe whose members are symbols. `r.Day >= 900` records a predicate instead of comparing +anything, which is why the sample counts one call for a million rows, and why a filter that cannot +be pushed down does not compile. The predicate is planned before a byte of data is read: the file's +statistics may settle it outright, the zone maps prune blocks, an index can prune more, and the plan +lists the segments the remaining blocks need. Here that is 14 of 123 blocks and 13 segments. + +A pipeline then reads those segments and decodes them into buffers from the session's pool, a chunk +at a time for a plain read or a block at a time under a filter, overlapping the next decode with +your work on the current batch. You get a `Columns`, a `ref struct` over the decoded rows +that stays valid until the next `MoveNextAsync`, and that the compiler keeps inside the loop body. +Nothing is copied for you and nothing is left to dispose. + +Anything beyond borrowed columns is a sink you choose and pay for. `ToBatchesAsync` copies each batch +once into buffers you own, `ToRecordsAsync` builds rows, and `CountAsync`, `SumAsync` and +`GroupBy(…).Select(…)` run as operators of the scan, on the encoded blocks, answering from the +statistics when those are enough. `ExplainAsync` shows the plan before the run, and `Statistics` +shows what the scan did after it. See [scan-a-table.md](scan-a-table.md) and [statistics-and-pruning.md](statistics-and-pruning.md). ## A write -A writer is fed through builders, as a `PipeWriter` is fed through buffers. `Builder()` -hands out the writer's own column builders; you append, or fill spans in place, and `WriteAsync` -takes the rows. The writer keeps them until it holds whole blocks of `BlockRows`, encodes each -column from where its bytes lie, and seals the blocks into a chunk at `FlushAsync` or when a chunk -reaches its target size; `CompleteAsync` writes what is pending as the tail block, then the -statistics, the zone maps, any index, and the footer. The `WriteReport` says what it chose, down to -each chunk's encoding. See [write-a-file.md](write-a-file.md). +A writer is fed through builders, the way a `PipeWriter` is fed through buffers. `Builder()` +hands out the writer's own column builders. You append values, or fill spans in place, and +`WriteAsync` takes the rows. The writer holds them until it has whole blocks of `BlockRows`, encodes +each column straight from where its bytes lie, and seals the blocks into a chunk at `FlushAsync` or +when a chunk reaches its target size. `CompleteAsync` writes whatever is pending as the tail block, +then the statistics, the zone maps, any index and the footer. The `WriteReport` says what the writer +chose, down to each chunk's encoding. See [write-a-file.md](write-a-file.md). ## The engine under it -Under the surface sits the engine the scans and the writers compile to: a type arena built once at -open, arenas of canonical arrays that a batch borrows, a decoder per encoding registered in a -static table, a reader per layout, the pruners, the index readers, and the encoders and the -compressor of the writer. None of it is public: a caller names only the types of the surface, each -on purpose, and the engine can change under them. +Below the public API sits the engine that scans and writers compile down to: a type arena built once +at open, arenas of canonical arrays that a batch borrows, a decoder per encoding registered in a +static table, a reader per layout, the pruners, the index readers, and the writer's encoders and +compressor. None of it is public. A caller only names the types of the public API, each one there on +purpose, and the engine can change underneath them. ## The packages | | | |---|---| -| `Vorticity` | the format: session, files, scans, columns, writers, options, plans, the I/O seam | +| `Vorticity` | the format: session, files, scans, columns, writers, options, plans, the I/O interface | | `Vorticity.Generators` | the `[VortexRecord]` generator and the analyzers, at build time only: [records.md](records.md), [diagnostics.md](diagnostics.md) | | `Vorticity.Dataset` | a versioned dataset over an object store, experimental: [datasets.md](datasets.md) | | `Vorticity.RowEncoding` | the byte-sortable row encoding, experimental: [row-keys.md](row-keys.md) | +| `Vorticity.Zstd` | the managed Zstandard codec the core uses, also usable on its own | -Only the first is needed to read or write a file, and it depends on nothing beyond the shared -framework but `System.IO.Hashing` and `Vorticity.Zstd`, this repository's managed Zstandard. +Only `Vorticity` is needed to read or write a file. Beyond the shared framework it depends on +nothing but `System.IO.Hashing` and `Vorticity.Zstd`. ## Where to read further -The design documents beside this guide say why, in far more depth than a page here should. +The design documents next to this guide explain the why, in far more depth than a guide page should. | | | |---|---| @@ -138,7 +143,8 @@ The design documents beside this guide say why, in far more depth than a page he | [11-write-strategy.md](../design/11-write-strategy.md) | how the writer chooses an encoding, block by block | | [12-index-reads.md](../design/12-index-reads.md) | the key cursor, and answers that need no rows | | [13-dataset.md](../design/13-dataset.md) | the versioned dataset | -| [14-public-api.md](../design/14-public-api.md) | the rules the public surface follows, and where each part lives | +| [14-public-api.md](../design/14-public-api.md) | the rules the public API follows, and where each part lives | +| [16-queries.md](../design/16-queries.md) | filters, aggregates, group by and ordering | | [90-registry.md](../design/90-registry.md) | every encoding, layout and column type, and its state | | [05-benchmarks.md](../design/05-benchmarks.md) | how performance is measured against the Rust implementation, whose figures are on [benchmarks.md](benchmarks.md) | diff --git a/docs/guide/indexes.md b/docs/guide/indexes.md index 1648fdbe..f9444671 100644 --- a/docs/guide/indexes.md +++ b/docs/guide/indexes.md @@ -1,6 +1,6 @@ # Indexes -Ask the writer for an index, read in the report whether it was built, and check that a filter uses +Ask the writer for an index, check in the report whether it was built, and verify that a filter uses it. ```csharp @@ -14,26 +14,27 @@ VortexWriteOptions options = new() ``` `Hit` is `[VortexRecord] public partial record struct Hit(string Session, string Path, int Status, int Score)`, -and the sample writes 400 000 of them: a twelve-character session identifier, a path, an HTTP status, -a score, none of them in order. The zone maps and the string bounds every file carries can prune -nothing on such columns; that is where an index earns its bytes. [statistics-and-pruning.md](statistics-and-pruning.md) -is the free kind of pruning. +and the sample writes 400 000 of them: a twelve-character session identifier, a path, an HTTP status +and a score, none of them in any particular order. The zone maps and string bounds that every file +carries can prune nothing on such columns, and that is where an index earns its bytes. +[statistics-and-pruning.md](statistics-and-pruning.md) covers the pruning you get for free. ## What the report says ``` -Bloom on Session, NgramBloom on Path, budget 300 per mille: 5310908 bytes, 932416 of them indexes +Bloom on Session, NgramBloom on Path, budget 300 per mille: 5310896 bytes, 932412 of them indexes Session vorticity.bloom.sbbf.v1: Built, 820056 bytes Path vorticity.bloom.ngram3.v1: Built, 111448 bytes ``` -`report.Indexes` has one entry per index the policy asked for: the column, the kind as the file -names it, `Built` or `Abandoned`, the reason when it was abandoned, and the bytes it takes. It is the -only place that says what was given up and why, and the reason is a sentence. +`report.Indexes` has one entry per index the policy asked for: the column, the kind as the file names +it, `Built` or `Abandoned`, the reason when it was abandoned, and the bytes it takes. It is the only +place that tells you what was given up and why, and the reason is a full sentence. ## What it buys -The sample asks each filter for its plan with and without the indexes (`ScanOptions.UseIndexes = false`): +The sample asks each filter for its plan with and without the indexes +(`ScanOptions.UseIndexes = false`): | filter | rows | blocks read, with | bytes read, with | without | |---|---|---|---|---| @@ -41,16 +42,17 @@ The sample asks each filter for its plan with and without the indexes (`ScanOpti | `Session == ` a value that does not | 0 | 0 of 49 | 803 596 | 49 blocks, 4 365 728 bytes | | `Path.Contains("checkout")` | 10 | 10 of 49 | 3 681 912 | 49 blocks, 4 365 728 bytes | -A skipping index does not find rows; it proves that a block cannot hold them, so the block is never +A skipping index does not find rows. It proves that a block cannot hold them, so the block is never read. The Bloom filters on `Session` cost 820 KB against 4.4 MB of data and cut the read almost four -times for a hit and five times for a miss; the filters are read whole, 804 KB of the 1.16 million +times for a hit and five times for a miss. The filters are read whole, 804 KB of the 1.16 million bytes. The n-gram filters are cheaper and serve `Contains` and `Like`. `plan.Pruning` names the -structure that pruned each block: `bloom filter pruned 48 reading 803596 bytes`. +structure that pruned each block, as in `bloom filter pruned 48 reading 803596 bytes`. -A block an index keeps is read by its chunk: the writer seals whole blocks into chunks, four here, -and a column's chunk is one segment. The ten blocks `Contains` keeps fall in ten of the file's -thirteen chunks, which is why it reads most of the file. A file read mostly for a few rows at a time -is one to write with smaller chunks, `ChunkTargetBytes` ([blocks-and-chunks.md](blocks-and-chunks.md)). +A block that an index keeps is read by its chunk. The writer seals whole blocks into chunks, four +here, and a column's chunk is one segment. The ten blocks that `Contains` keeps fall in ten of the +file's thirteen chunks, which is why it reads most of the file. A file that is mostly read a few rows +at a time is worth writing with smaller chunks, through `ChunkTargetBytes` +([blocks-and-chunks.md](blocks-and-chunks.md)). ## The kinds @@ -59,30 +61,30 @@ is one to write with smaller chunks, `ChunkTargetBytes` ([blocks-and-chunks.md]( | `Bloom(column, falsePositiveRate)` | a split-block Bloom filter per block | equality and `In` | | `NgramBloom(column)` | a Bloom filter of trigrams per block | `Contains` and `Like` on a text column | | `Postings(column)` | value to blocks, exact | equality and `In` on a column of few values | -| `SortedRuns(column)` | value to rows, sorted | equality, and the key cursor over an unsorted column | +| `ForKey([column], IndexKind.NgramPostings)` | trigrams to blocks, exact | `Contains` and `Like` on a text column, larger than the n-gram filter | +| `SortedRuns(column)` | value to rows, sorted | equality, ranges, and the key cursor over an unsorted column | | `ForKey(columns, kind, encoder)` | sorted runs over a tuple's row encoding | a composite key | The locating kinds, measured on the same rows: ``` -Postings on Status, SortedRuns on Score, budget 3000 per mille: 6232421 bytes, 1853905 of them indexes +Postings on Status, SortedRuns on Score, budget 3000 per mille: 6232409 bytes, 1853901 of them indexes Status == 301: 16 rows; 16 of 49 blocks and 4287212 bytes with the indexes (… locating index pruned 33 reading 1184 bytes), 49 blocks and 4365728 bytes without Score == a value that exists: 1 rows; 1 of 49 blocks and 692068 bytes with the indexes (… locating index pruned 48 reading 333200 bytes), 49 blocks and 4365728 bytes without Score between 1000 and 1100: 38 rows; 36 of 49 blocks and 4365728 bytes with the indexes (… sorted runs pruned 13 reading 0 bytes), 49 blocks and 4365728 bytes without a key cursor on Score: the first key at or after 1000 is 1001, at row 280415; 382 rows hold a smaller one ``` -Postings on four statuses take 1 184 bytes and find the sixteen rows of a rare one, in sixteen -blocks that spread over nearly every chunk. Sorted runs take 1.85 MB, two fifths of the data: they -list every row. They prune an equality, and they are what lets -`Scan().Keys(r => r.Score)` open a cursor over a column that is not sorted; without them -`OpenAsync` throws `VortexUnsupportedException` naming the index to build -([keys-in-order.md](keys-in-order.md)). They prune a range filter too: `Between` kept the 36 blocks -that hold its 38 rows. The bytes did not drop, since every segment holds one of those blocks, and -the step reports 0 bytes because sorted runs are read through the index's own source and the -file's run cache, which the plan does not count. +Postings on four statuses take 1 184 bytes and find the sixteen rows of a rare one, in sixteen blocks +spread over nearly every chunk. Sorted runs take 1.85 MB, two fifths of the data, because they list +every row. They prune an equality, and they are what lets `Scan().Keys(r => r.Score)` open a +cursor over a column that is not sorted. Without them, `OpenAsync` throws +`VortexUnsupportedException`, naming the index to build ([keys-in-order.md](keys-in-order.md)). They +prune a range filter too: `Between` kept the 36 blocks that hold its 38 rows. The bytes did not drop, +since every segment holds one of those blocks, and the step reports 0 bytes because sorted runs are +read through the index's own source and the file's run cache, which the plan does not count. -An index can also be added to a file already written, in place: +An index can also be added to a file that is already written, in place: ```csharp IReadOnlyList added = await VortexFileIndexer.AppendIndexesAsync( @@ -90,19 +92,22 @@ IReadOnlyList added = await VortexFileIndexer.AppendIndexesAsy ``` ``` - added afterwards: vorticity.sorted.runs.v1 Built, 1850040 bytes; the file grew from 4378452 to 6234823 bytes; the cursor finds 1001 at row 280415 + added afterwards: vorticity.sorted.runs.v1 Built, 1850040 bytes; the file grew from 4378460 to 6234813 bytes; the cursor finds 1001 at row 280415 ``` -The data stays where it is; the file's tail is written again with the index regions after the last -chunk, and the file takes a new identity. The index is the one the write would have built, 1 850 040 -bytes, and the file grew by 1 856 371: the directory and the tail written again are the difference. -`VortexFileIndexer.BuildFragmentAsync` -builds the same index into an `IndexFragment` instead and leaves the file untouched: a reader passes -it with `VortexOpenOptions.IndexFragments`, which is how an index reaches a file that cannot be -rewritten. A fragment is bound to the file's identity, which every file this library writes carries; -a file from another writer has none, and its fragment is bound by the file's length and modification -time, which a copy that keeps both can fool. `VortexOpenOptions.IndexFragmentsNeedIdentity` leaves -such a fragment out, with the reason in `IndexFragmentRefusals`. +The data stays where it is. The file's tail is written again with the index regions after the last +chunk, and the file takes a new identity. The index is the same one the write would have built, +1 850 040 bytes, and the file grew by 1 856 353: the difference is the directory and the rewritten +tail. + +`VortexFileIndexer.BuildFragmentAsync` builds the same index into an `IndexFragment` instead and +leaves the file untouched. A reader passes it through `VortexOpenOptions.IndexFragments`, which is how +an index reaches a file that cannot be rewritten. A fragment is bound to the file's identity, which +every file written by this library carries. A file from another writer has none, so its fragment is +bound by the file's length and modification time, which a copy that preserves both can fool, and it +only works when the file is opened from a path. `VortexOpenOptions.IndexFragmentsNeedIdentity` makes +the open ignore such fragments. A fragment the open does not accept is simply ignored, and the open +does not fail. A composite key names its encoder, from the row-encoding package: @@ -112,11 +117,11 @@ Indexes = IndexPolicy.None .WithBudgetPerMille(3_000), ``` -It was built, 1.40 MB. No scan of the typed surface reads it: `Status == 301 & Score >= 1000` read -49 of 49 blocks, and `Keys` takes one column. It is how a dataset's clustering key is written -([datasets.md](datasets.md)). Without an encoder the key is abandoned: *no key encoder: declare the -key with IndexPolicy.ForKey(columns, kind, encoder), e.g. with the RowKeyEncoder of -Vorticity.RowEncoding*. [row-keys.md](row-keys.md) is the encoding. +It was built, 1.40 MB. No scan of the typed surface reads it: `Status == 301 & Score >= 1000` read 49 +of 49 blocks, and `Keys` takes a single column. It is how a dataset's clustering key is written +([datasets.md](datasets.md)). Without an encoder the key is abandoned, with the reason *no key +encoder: declare the key with IndexPolicy.ForKey(columns, kind, encoder), e.g. with the RowKeyEncoder +of Vorticity.RowEncoding*. [row-keys.md](row-keys.md) describes the encoding. ## Auto, and the budget @@ -124,38 +129,42 @@ Vorticity.RowEncoding*. [row-keys.md](row-keys.md) is the encoding. and its share of the budget say they pay. On these rows: ``` -IndexPolicy.Auto: 4378617 bytes, 101 of them indexes +IndexPolicy.Auto: 4378605 bytes, 97 of them indexes + Session vorticity.dict.probe.v1: Abandoned, 0 bytes -- no chunk of this column is dictionary-encoded Session vorticity.bloom.sbbf.v1: Abandoned, 0 bytes -- Auto gave it up: 16384 bytes of filters against 229376 raw bytes of column, over its share of 20‰. An explicit Bloom policy overrides the share + Path vorticity.dict.probe.v1: Built, 0 bytes Status vorticity.dict.probe.v1: Built, 0 bytes Status vorticity.bloom.sbbf.v1: Abandoned, 0 bytes -- Auto gave it up: the first 16 blocks hold 4 distinct values, under the floor of 8 the policy asks before a filter pays + ... ``` -It built dictionary probes, which cost nothing since the dictionary is already in the file; a probe -answers for the chunk its dictionary covers, and with a 301 in all but one of the file's chunks it -pruned one block of 49 for `Status == 301`. It gave up the Bloom filter the session lookup needed. -Ask for that one by name. +It built dictionary probes, which cost nothing since the dictionary is already in the file. A probe +answers for the chunk its dictionary covers, and with a 301 in all but one of the file's chunks, it +pruned one block of 49 for `Status == 301`. It gave up the Bloom filter the session lookup needed, so +ask for that one by name. -`WithBudgetPerMille` bounds the bytes the indexes may take together, per thousand bytes of data, 100 -by default. The same Bloom and n-gram policy under the default budget: +`WithBudgetPerMille` bounds the bytes all the indexes may take together, per thousand bytes of data, +100 by default. The same Bloom and n-gram policy under the default budget: ``` Session vorticity.bloom.sbbf.v1: Abandoned, 0 bytes -- the file's indexes reached 296960 bytes against 1430532 bytes of data, over the budget of 100‰ (IndexPolicy.WithBudgetPerMille) ``` -**An abandoned index leaves no bytes.** The file was 106 bytes larger than one without a policy, the -index directory. The budget drops only an index that has written nothing, so one whose filters are -already in the file is kept even past the budget. +An abandoned index leaves no bytes behind. The file was 90 bytes larger than one written without a +policy, which is the index directory. The budget only drops an index that has not written anything +yet, so an index whose filters are already in the file is kept even past the budget. ## Required ``` a required index over its budget: VortexException: The vorticity.bloom.sbbf.v1 index on 'Session' is required and was not built: the required indexes take 3277420 bytes against 4367852 bytes of data, over the budget of 50‰; raise it with IndexPolicy.WithBudgetPerMille. The file is not completed. - the file is left behind: False + the path still holds the file written before it: True ``` -`required: true` makes an index that cannot be built fail the write: `CompleteAsync` throws -`VortexException` naming the index and the reason, before the tail is written, and the writer's -disposal deletes the partial file. The budget is judged once the data passes 1 MiB. +`required: true` makes an index that cannot be built fail the write. `CompleteAsync` throws +`VortexException`, naming the index and the reason, before the file's tail is written, and the +partial file is discarded when the writer is disposed. The path keeps whatever it held before, here +the file of the previous write in the sample. The budget is only judged once the data passes 1 MiB. ## What a file carries @@ -166,25 +175,26 @@ disposal deletes the partial file. The budget is judged once the data passes 1 M ``` `file.GetIndexesAsync()` lists the index directory: kind, column, runs, blocks, entries and bytes. -`ListedBytes` is what the directory lists, not what the index weighs: a `FilterTree` holds the +`ListedBytes` is what the directory lists, not the weight of the index: a `FilterTree` keeps its filters below those regions. `file.VerifyIndexesAsync()` checks every listed region against its -checksum, reading each once; it is a tool's check, and a scan does not run it. +checksum, reading each one once. It is a tool's check, and a scan does not run it. ## Watch out -* **`AppendIndexesAsync` rewrites the tail even when every index is abandoned**, and the file takes - a new identity all the same. A `required` index it cannot build throws `VortexException` before - anything reaches the file. -* **A fragment built by `BuildFragmentAsync` records no hash of the file**, so `VerifyIndexesAsync` - and `vxdump --verify` cannot tell that it still matches; it is bound to the file by its identity, - or, for a file written by another library, by its length and last write time. -* **A name the schema does not have throws at `CreateWriter`**: *The index policy names 'Referrer', - which is no column of the schema struct{…}.* -* **The composite keys of a file share one encoder**: `ForKey` refuses a second encoder of another +* `AppendIndexesAsync` rewrites the tail even when every index is abandoned, and the file takes a new + identity all the same. A `required` index it cannot build throws `VortexException` before anything + reaches the file. +* A fragment built by `BuildFragmentAsync` records no hash of the file, so `VerifyIndexesAsync` and + `vxdump --verify` cannot tell whether it still matches. It is bound to the file by its identity or, + for a file written by another library, by its length and last write time. +* A column name the schema does not have throws at `CreateWriter`: *The index policy names + 'Referrer', which is no column of the schema struct{…}.* +* The composite keys of a file share one encoder, and `ForKey` refuses a second encoder of another format when the policy is built. -* **A skipping index earns most when the answer is no**, and nothing on a column whose zone maps - already prune: a sorted column needs none. -* [10-indexes.md](../design/10-indexes.md) is the design: the kinds, the directory, the policy. +* A skipping index earns the most when the answer is no, and nothing on a column whose zone maps + already prune. A sorted column needs none. +* [10-indexes.md](../design/10-indexes.md) describes the design: the kinds, the directory and the + policy. ## Run it diff --git a/docs/guide/keys-in-order.md b/docs/guide/keys-in-order.md index 970a770d..42a093a0 100644 --- a/docs/guide/keys-in-order.md +++ b/docs/guide/keys-in-order.md @@ -18,23 +18,23 @@ await using (KeyCursor cursor = await file.Scan().Keys(r => r.Day) } Console.WriteLine($"rank of 900: {await cursor.RankAsync(900)}"); - Console.WriteLine($"{await cursor.KeyCountAsync()} distinct keys"); + Console.WriteLine($"{await cursor.KeyCountAsync()} entries share key {cursor.Key}"); } ``` -A cursor is not a scan. It walks the **keys** of one column in the file's order and tells you which -row each one is at; you read the row afterwards if you want it. It is the shape for a lookup, a -range walk, or "how many values are below this one". +A cursor is not a scan. It walks the keys of one column in order and tells you which row each one is +at. You read the row afterwards if you need it. This is the right tool for a lookup, a range walk, or +a question like "how many values are below this one". `Keys(r => r.Day)` infers the key type from the member, so the cursor is a `KeyCursor` and -`SeekAsync` takes an `int`. On the demonstration file, whose `Day` rises by one every thousand rows: +`SeekAsync` takes an `int`. On the demonstration file, whose `Day` grows by one every thousand rows: ``` first key 0 at row 0 next distinct key 1 at row 1000 seek to 900 landed on 900 at row 900000 rank of 900: 900000 -1000 distinct keys +1000 entries share key 900 entry of rank 123456: key 123 at row 123456 last key 999 at row 999999 an exact seek to a key that is not there: False, cursor valid: False @@ -45,23 +45,22 @@ seek at or before 700: key 700 at row 700999, the last of its run ## The moves -Every move is a `ValueTask`: `false` means there was nowhere to go, and `IsValid` is then -false. +Every move is a `ValueTask`. `false` means there was nowhere to go, and `IsValid` is then false. | move | what it does | |---|---| -| `SeekFirstAsync`, `SeekLastAsync` | the ends | +| `SeekFirstAsync`, `SeekLastAsync` | go to either end | | `SeekAsync(key, op)` | `Exact`, `AtOrAfter`, `After`, `AtOrBefore`, `Before` | -| `SeekRankAsync(n)` | the entry at position *n*, counting from zero | -| `NextAsync`, `PrevAsync` | one entry | -| `NextKeyAsync`, `PrevKeyAsync` | the first entry of the next or previous distinct key | -| `RankAsync(key)` | how many entries have a smaller key | -| `KeyCountAsync()` | how many distinct keys there are | +| `SeekRankAsync(n)` | go to the entry at position *n*, counting from zero | +| `NextAsync`, `PrevAsync` | move by one entry | +| `NextKeyAsync`, `PrevKeyAsync` | move to the first entry of the next or previous distinct key | +| `RankAsync(key)` | count the entries whose key is smaller | +| `KeyCountAsync()` | count the entries that share the current key, without moving | -A key repeated over many rows is many entries, ordered by row: `Exact` lands on the first of them -and `AtOrBefore` on the last, which is why the seek to 700 above stops at row 700 999. Two ranks -count a range without walking it: `RankAsync(102) - RankAsync(100)` is the 2 000 rows whose day is -100 or 101. `Row` is then the argument to hand to [`Rows(...)`](read-rows-by-index.md) on a scan. +A key repeated over many rows is many entries, ordered by row. `Exact` lands on the first of them and +`AtOrBefore` on the last, which is why the seek to 700 above stops at row 700 999. Two ranks count a +range without walking it: `RankAsync(102) - RankAsync(100)` is the 2 000 rows whose day is 100 or +101. `Row` is the value to pass to [`Rows(...)`](read-rows-by-index.md) on a scan. ## What the file needs @@ -79,24 +78,24 @@ day: source SortedColumn, 1 run, 1000000 entries, rows reachable: True turned down Dictionary: it holds keys without rows, which only a Distinct() cursor takes ``` -Two sources serve a cursor with rows: +Two sources can serve a cursor that returns rows: -* **A sorted column.** The file's statistics say the column is sorted, so the column is its own - index. It costs nothing to write: order the rows by the column. A seek is a binary search over - the zone maps and one block decoded; a thousand seeks took 3.6 ms here. -* **A sorted-runs index**, which the writer builds when asked: +* A sorted column: when the file statistics say a column is sorted, the column is its own index, and + it costs nothing to write: just order the rows by that column. A seek is a binary search over the + zone maps followed by one decoded block. A thousand seeks took 3.8 ms here. +* A sorted-runs index, which the writer builds when asked: ```csharp new VortexWriteOptions { Indexes = IndexPolicy.None.SortedRuns(Reading.ColumnNames.City) } ``` - On a 100 000-row copy of the readings whose `City` cycles row by row, that index is two runs - and 100 000 entries, and the cursor walks the eight cities in order, each with the rank of its - first entry. It is not free: it took 212 880 bytes against 154 708 bytes of data. The default - budget of 100 per mille abandoned it, and the write report said so; it was built only once the - budget was raised with `WithBudgetPerMille(20_000)`. The budget weighs the index before it is - encoded, 2 637 485 bytes here, so the figure to raise it by is the report's, not the file's. - [indexes.md](indexes.md) covers the budget. + On a 100 000-row copy of the readings whose `City` changes every row, that index has two runs and + 100 000 entries, and the cursor walks the eight cities in order, each with the rank of its first + entry. It is not free. It took 212 880 bytes against 154 708 bytes of data. The default budget of + 100 per mille dropped it, and the write report said so. It was only built once the budget was + raised with `WithBudgetPerMille(20_000)`. The budget weighs the index before it is encoded, here + 2 637 485 bytes, so use the figure from the report, not the size on disk, when deciding how far to + raise it. [indexes.md](indexes.md) covers the budget. A column with neither has no cursor: @@ -109,18 +108,18 @@ celsius: source None file with VortexFileIndexer.AppendIndexesAsync. ``` -`Distinct()` on the builder asks for one stop per key instead of one per entry. A postings index or -a dictionary can serve it, since it needs keys without rows; on the sorted `Day` column it stops -1 000 times and still knows its rows (`HasRows` is true). +`Distinct()` on the builder asks for one stop per key instead of one per entry. A postings index or a +dictionary can serve it, since it needs keys without rows. On the sorted `Day` column it stops 1 000 +times and still knows its rows (`HasRows` is true). ## Over a result The scan of a query's result, such as a group by read through `As()` -([aggregates.md](aggregates.md)), has no zone map and no index. Its cursor is the one that sorts: +([aggregates.md](aggregates.md)), has no zone map and no index, so its cursor is the one that sorts. `OpenAsync` runs the query, keeps the column's non-null values, and sorts them in memory unless they already arrive in order, as they do from a group by that streams on that key. The plan's source is -`InMemory`. `Row` is a position in the order the result is delivered, which `Rows(...)` on a scan -of the same result reads. A bool or a decimal column has no key order and throws +then `InMemory`. `Row` is a position in the order the result is delivered, which `Rows(...)` on a +scan of the same result can read. A bool or a decimal column has no key order and throws `NotSupportedException`. ## Batches in key order @@ -129,8 +128,8 @@ of the same result reads. A bool or a decimal column has no key order and throws Scan ordered = file.Scan().OrderByDescending(r => r.Day); ``` -The batches then arrive in the key's order rather than the file's, from the same sources a cursor -uses. `ExplainAsync().Order` says which: +The batches then arrive in key order instead of file order, from the same sources a cursor uses. +`ExplainAsync().Order` says which one: ``` ordered scan: source SortedColumn, 1 run, 1000000 entries, descending True @@ -138,24 +137,24 @@ ordered scan: source SortedColumn, 1 run, 1000000 entries, descending True 123 batches ``` -A null key comes last in either direction. An aggregation ignores the order: it gives the same -answer either way, and [aggregates.md](aggregates.md) says what order a group by returns. +A null key comes last in both directions. An aggregation ignores the order and gives the same answer +either way, and [aggregates.md](aggregates.md) explains what order a group by returns. ## Watch out -* **`Keys` takes the whole column.** On a scan that already has a `Where` or a `Rows`, it throws +* `Keys` takes the whole column. On a scan that already has a `Where` or a `Rows`, it throws `InvalidOperationException: A key cursor walks the whole column; build it on a scan without Where or Rows.` * A failed seek leaves the cursor invalid, not where it was. Seek again before reading `Key`. -* The order of keys is the file's order for the type, and there is no comparer on the cursor: - compare keys you got from the cursor, which already arrive in that order. A text key is ordered - by its UTF-8 bytes. -* A cursor is not thread-safe, and it holds its source until disposed: `await using` it. +* Keys follow the file's order for their type, and the cursor takes no comparer. Compare the keys + you get from the cursor, which already arrive in that order. A text key is ordered by its UTF-8 + bytes. +* A cursor is not thread-safe, and it holds its source until it is disposed, so use `await using`. * `SeekAsync` refuses a null key with `ArgumentNullException`. -For the design — sources of order, the cursor's positioning rules, key-ordered delivery — see -[12-index-reads.md](../design/12-index-reads.md), and §5.7 of -[14-public-api.md](../design/14-public-api.md) for the surface. +The design behind this page (the sources of order, how a cursor is positioned, delivery in key +order) is in [12-index-reads.md](../design/12-index-reads.md), and the surface in +[the public API design](../design/14-public-api.md#57-the-key-cursor). ## Run it diff --git a/docs/guide/limits.md b/docs/guide/limits.md index 654c196d..90442792 100644 --- a/docs/guide/limits.md +++ b/docs/guide/limits.md @@ -1,7 +1,7 @@ # Limits -Open a file you did not write: what a hostile file can make the reader do, what it cannot, and the -caps in between. +Opening a file you did not write: what a hostile file can make the reader do, what it cannot, and +the caps in between. ```csharp await using VortexFile file = await VortexSession.Default.OpenAsync(path, new VortexOpenOptions @@ -32,34 +32,34 @@ ceiling 1048576: 1000000 rows ceiling 268435456: 1000000 rows ``` -**A file is untrusted input.** Every structure the reader follows is bounded, so that a malformed -or malicious file makes it refuse rather than read out of bounds, allocate without limit, or loop -forever. A refusal is a `VortexFormatException` for bytes that break the format or a cap, and a -`VortexUnsupportedException` for a component this build does not implement; nothing else. The +A file is untrusted input. Every structure the reader follows is bounded, so a malformed or +malicious file makes it refuse rather than read out of bounds, allocate without limit or loop +forever. A refusal is either a `VortexFormatException`, for bytes that break the format or a cap, or +a `VortexUnsupportedException`, for a component this build does not implement. Nothing else. The reasoning behind each cap is in [08-semantics.md](../design/08-semantics.md). ## What a damaged file does -The sample writes a 38 436-byte file, then opens a thousand copies of it, each with four bits -flipped at random in its last 4 KiB, where the footer, the layout tree and the last chunk live, -and reads every row of each: +The sample writes a 38 444-byte file, then opens a thousand copies of it, each with four bits flipped +at random in its last 4 KiB (where the footer, the layout tree and the last chunk live), and reads +every row of each: ``` -1000 copies of a 38436-byte file, four bits flipped in the last 4 KiB of each: - VortexFormatException: 860 - VortexUnsupportedException: 41 - read whole: 99 +1000 copies of a 38444-byte file, four bits flipped in the last 4 KiB of each: + VortexFormatException: 850 + VortexUnsupportedException: 47 + read whole: 103 ``` -Every failure is one of the two refusals. The 99 copies that read whole are the other half of the -lesson: **the reader guarantees safety, not integrity.** A flipped bit in a data segment decodes -to a different value, and the format carries no checksum over data. The index regions do carry -one, which `VerifyIndexesAsync` and `vxdump --verify` check. +Every failure is one of the two refusals. The 103 copies that read whole carry the other half of the +lesson: the reader guarantees safety, not integrity. A flipped bit in a data segment decodes to a +different value, because the format carries no checksum over data. The index regions do carry one, +which `VerifyIndexesAsync` and `vxdump --verify` check. ## The fixed caps -`VortexLimits` holds them. None is settable, because each bounds a loop or a recursion the file -controls: +`VortexLimits` holds them. None can be changed, because each one bounds a loop or a recursion the +file controls: | | | |---|---| @@ -72,44 +72,43 @@ controls: | `MaxAlignment` | 64 bytes, `MaxAlignmentExponent` 6 | | `MaxPostscriptSize` | 65 527 bytes | -A file past any of them is malformed, and says so with `VortexFormatException`. +A file past any of them is malformed and reported with `VortexFormatException`. ## The ceiling you set -`VortexOpenOptions.MaxDecompressedSize` bounds what one decode may produce, and a decode is at most -a chunk of one column. The default is `VortexLimits.DefaultMaxDecompressedSize`, 268 435 456 bytes. -It is a refusal, not a truncation, and it is the defence against a decompression bomb: a few bytes on -disk that claim to expand to gigabytes. +`VortexOpenOptions.MaxDecompressedSize` bounds what a single decode may produce, and a decode is at +most one chunk of one column. The default is `VortexLimits.DefaultMaxDecompressedSize`, 268 435 456 +bytes. It is a refusal, not a truncation, and it is the defence against a decompression bomb: a few +bytes on disk that claim to expand to gigabytes. The right value is small. A chunk this library writes holds about a megabyte of its widest column -([blocks-and-chunks.md](blocks-and-chunks.md)), so an honest decode is a few megabytes at most; the -first decode the sample makes produces 131 072 bytes, which is why 4 096 is refused above and 1 MiB -is plenty. Lower the ceiling for input you did not write; raise it only for a file you trust whose +([blocks-and-chunks.md](blocks-and-chunks.md)), so an honest decode is a few megabytes at most. The +first decode in the sample produces 131 072 bytes, which is why 4 096 is refused above and 1 MiB is +plenty. Lower the ceiling for input you did not write, and raise it only for a trusted file whose chunks are genuinely large, such as long text values. `VortexOpenOptions.MaxBatchDecompressedSize` bounds what the decodes of one batch produce together, -across its columns. It is off by default: a thousand columns each decoding near the ceiling above -make a batch a thousand times that, which is legitimate for a wide table you wrote and a bomb in a -file you did not. Set it for input you do not trust, at a few times the widest batch you expect; -a batch past it throws `VortexFormatException`. +across all its columns. It is off by default, because a thousand columns each decoding near the +ceiling make a batch a thousand times that, which is legitimate for a wide table you wrote and a bomb +in a file you did not. For untrusted input, set it to a few times the widest batch you expect. A +batch past it throws `VortexFormatException`. ## Statistics are claims -A file's statistics and zone maps are what the writer said about the values, and the reader -prunes by them. A lying file cannot make the reader fault, but it can make it skip blocks it -should have read. `VerifyStatistics = true` has the reader check what a decode can recompute: a -masked array's values carry no nulls of their own, and a key cursor walking a sorted column finds -each zone's null count, order and bounds as stated, and refuses to go on otherwise. It costs a pass -over the values it checks. +A file's statistics and zone maps are what the writer said about the values, and the reader prunes +by them. A lying file cannot make the reader fault, but it can make it skip blocks it should have +read. `VerifyStatistics = true` makes the reader check what a decode can recompute: a masked array's +values carry no nulls of their own, and a key cursor walking a sorted column finds each zone's null +count, order and bounds as stated, and stops otherwise. It costs a pass over the values it checks. -An answer the reader takes from the statistics alone, such as a count, a minimum or a mean that -no block had to be decoded for, is still the file's claim. For input you do not trust, ask the -question over the values, or check it against a scan. +An answer the reader takes from the statistics alone (a count, a minimum or a mean that needed no +block decoded) is still the file's claim. For input you do not trust, compute the answer over the +values, or check it against a scan. ## Components this build does not know -A file may name an encoding, a layout or a type this build does not implement. It still opens, -and the refusal comes when a block needs the component: +A file may name an encoding, a layout or a type this build does not implement. It still opens, and +the refusal comes only when a block needs that component: ```csharp await using VortexFile file = await VortexSession.Default.OpenAsync(patched); @@ -126,25 +125,24 @@ opened, not supported: vortex.alz; Day and City read 1000000 rows; Scan ``` The sample renames the encoding of the `Celsius` column in a copy of the file, so this build no -longer knows it. The columns that do not use it read in full; the scan that needs it throws -`VortexUnsupportedException` naming the kind and the id, at its first block. The open cannot refuse -earlier: a footer may list encodings no chunk uses, and which ones a chunk uses is known only when -its bytes are read. `file.ArrayEncodings` and -`file.LayoutEncodings` list what the footer declares, each with `Supported`, so a caller can ask -before scanning. +longer knows it. The columns that do not use it read in full, and the scan that needs it throws +`VortexUnsupportedException` at its first block, naming the kind and the id. The open cannot refuse +earlier, because a footer may list encodings no chunk uses, and which ones a chunk uses is only known +once its bytes are read. `file.ArrayEncodings` and `file.LayoutEncodings` list what the footer +declares, each with `Supported`, so a caller can check before scanning. ## Watch out -* **The ceiling is per decode, not per scan.** A scan that decodes a thousand blocks of 1 MiB never - approaches a 256 MiB ceiling; one block that claims 300 MiB trips it. The batch ceiling, when - set, adds the decodes of a batch's columns together, and starts again at the next batch. +* The ceiling applies per decode, not per scan. A scan that decodes a thousand blocks of 1 MiB never + comes near a 256 MiB ceiling, while a single block that claims 300 MiB trips it. The batch ceiling, + when set, adds up the decodes of a batch's columns and starts again at the next batch. * The open reads the file's tail in one request of `InitialReadSize` bytes, 64 KiB by default and - never less, but for a local file, which reads its last 8 KiB first since a read costs it no round - trip; a footer larger than that costs a second read, not a refusal. -* `IndexCacheBytes` on the session bounds what each open file keeps of decoded index runs. It is a - memory ceiling rather than a safety one, and 0 keeps nothing. -* A cap refusal is `VortexFormatException`, an unknown component `VortexUnsupportedException`: - [errors.md](errors.md) says which is which. + never less. A local file is the exception: it reads its last 8 KiB first, since a read costs it no + round trip. A footer larger than the first read costs a second read, not a refusal. +* `IndexCacheBytes` on the session bounds how much of the decoded index runs each open file keeps. + It is a memory ceiling rather than a safety one, and 0 keeps nothing. +* A cap refusal is a `VortexFormatException` and an unknown component a + `VortexUnsupportedException`. [errors.md](errors.md) covers both. ## Run it diff --git a/docs/guide/native-aot.md b/docs/guide/native-aot.md index fedef5ed..9cf765e8 100644 --- a/docs/guide/native-aot.md +++ b/docs/guide/native-aot.md @@ -13,74 +13,78 @@ Publish an application that uses this library ahead of time. dotnet publish -c Release -r linux-x64 ``` -Nothing else is required. The three libraries are built with `IsAotCompatible` and `IsTrimmable`, -so the trim and AOT analyzers run over your code when you publish, and an `IL2xxx` or `IL3xxx` -warning names the call of yours that is the problem. `InvariantGlobalization` is an application's -choice, not a library's: set it if your application can live without culture data. vxdump and the -samples set it. +Nothing else is required. The four libraries (`Vorticity`, `Vorticity.Dataset`, +`Vorticity.RowEncoding` and `Vorticity.Zstd`) are built with `IsAotCompatible` and `IsTrimmable`, so +the trim and AOT analyzers run over your code when you publish, and an `IL2xxx` or `IL3xxx` warning +points at the call of yours that causes the problem. `InvariantGlobalization` is the application's +choice, not the library's. Set it if your application can do without culture data, as vxdump and +the samples do. ## What makes it possible -Ahead-of-time compilation needs to see, at publish time, every type and every generic -instantiation the program will use. The surface is built so that it can: +Ahead-of-time compilation needs to see, at publish time, every type and every generic instantiation +the program will use. The API is designed so that it can: -* **A record is compiled, not discovered.** `[VortexRecord]` runs in the compiler and writes +* A record is compiled, not discovered. `[VortexRecord]` runs inside the compiler and writes `Schema`, `ReadRows` and `WriteRows` into the type as static members of `IVortexRecord`. The library calls them through the constraint `where TRecord : IVortexRecord`, so - `file.Scan()` and `writer.WriteAsync(rows)` are ordinary generic code over a type - the compiler knows. Nothing reads a member by name, builds a delegate or emits code at run time. - A record written by hand works the same way ([records.md](records.md)). -* **A filter is data, not an expression tree.** The lambda given to `Where` runs once over a - `Probe`, and its `Sym` operators record a predicate. There is no `Expression` to - compile, so there is nothing an interpreter would have to stand in for. -* **An extension type is a generic registration.** `o.Extensions.Register()` reaches - `Money`'s static members through `IVortexExtension`; no type is activated by name. -* **The decoders are a table.** Every encoding the library reads is registered by a static - constructor that names each decoder, so the trimmer keeps what is reachable and nothing is - resolved from an id string by reflection. -* **The generator is a build-time dependency.** It ships as an analyzer and never loads in your + `file.Scan()` and `writer.WriteAsync(rows)` are ordinary generic code over a type the + compiler knows. Nothing reads a member by name, builds a delegate or emits code at run time. A + record written by hand works the same way ([records.md](records.md)). +* A filter is data, not an expression tree. The lambda given to `Where` runs once over a `Probe`, + and its `Sym` operators record a predicate. There is no `Expression` to compile, so nothing + needs an interpreter in its place. +* An extension type is a generic registration. `o.Extensions.Register()` reaches `Money`'s + static members through `IVortexExtension`, and no type is created by name. +* The decoders are a table. Every encoding the library reads is registered by a static constructor + that names each decoder, so the trimmer keeps what is reachable and nothing is resolved from an id + string by reflection. +* The source generator is a build-time dependency. It ships as an analyzer and never loads in your process. -One path reflects, and it is guarded: a caller's aggregator that implements -`IEncodedAggregator` has its encoded steps reached through `MakeGenericMethod` when -the runtime can compile code, and under Native AOT it takes its canonical `Step` instead, on -decoded values. The answer is the same, the decode is not skipped. See +One path uses reflection, and it is guarded. A custom aggregator that implements +`IEncodedAggregator` has its encoded steps reached through `MakeGenericMethod` when the +runtime can compile code. Under Native AOT it falls back to its canonical `Step` over decoded +values instead. The answer is the same, only the decode is no longer skipped. See [aggregates.md](aggregates.md). -One check runs at load: a module initializer refuses a big-endian host with -`PlatformNotSupportedException`, because the reader casts file buffers directly and would -otherwise return byte-swapped values. +One check runs at load. A module initializer refuses a big-endian host with +`PlatformNotSupportedException`, because the reader casts file buffers directly and would otherwise +return byte-swapped values. ## vxdump is the proof -A library cannot prove it is AOT-clean; an executable that uses it can. vxdump, the inspection -tool in this repository, is written against the public surface alone and is published ahead of -time with `PublishAot`, `PublishTrimmed`, `TrimMode` full and warnings as errors, so a -reflection-shaped mistake fails the publish rather than scrolling past. The CI job then opens -**every file of the conformance corpus** with the published binary, `--all`, and fails if more than -one is refused; the one refusal is a file that embeds no schema, which cannot be opened without -one supplied. That pair, a clean publish and a run over real files, is what lets this page say the -library works under Native AOT rather than believe it. +A library cannot prove it is AOT-clean on its own, but an executable that uses it can. vxdump, the +inspection tool in this repository, uses only the public API and is published ahead of time with +`PublishAot`, `PublishTrimmed`, full trimming and warnings as errors, so a reflection-shaped mistake +fails the publish rather than scrolling past. + +CI then runs the published binary over every file of the conformance corpus twice. With `--all` it +opens each file, and fails if more than one is refused (the one refusal is a file that embeds no +schema and cannot be opened without one supplied). With `--scan` it decodes every batch of every +file, and fails on any crash or if fewer than 800 files scan. A clean publish plus a run over real +files is what lets this page claim the library works under Native AOT rather than hope it does. ``` dotnet publish tools/vxdump -c Release -r linux-x64 tools/vxdump/bin/Release/net11.0/linux-x64/publish/vxdump readings.vortex --all ``` -What it buys is startup. On the shared runtime, vxdump spends about 55 ms from start to exit on -`--schema` over the demonstration file, most of it the runtime starting and the first calls being -compiled. That is nothing inside a server and everything in a tool that runs once per file over a -directory of ten thousand; a native binary starts without either. +What AOT buys is startup. On the shared runtime, vxdump spends about 55 ms from start to exit on +`--schema` over the demonstration file, most of it starting the runtime and compiling the first +calls. That is nothing inside a server and a lot for a tool that runs once per file over a directory +of ten thousand. A native binary pays neither cost. ## Watch out -* **Publish your own executable ahead of time in your CI, and run it over real files.** A trim - warning at publish is cheap; a missing instantiation in production is not. -* **Generic code you write over records stays generic.** A method of yours that takes a - `TRecord : IVortexRecord` is compiled for each record it is called with, which is what - you want, as long as every call site names a concrete record type the compiler can see. -* A timestamp column with a named zone resolves it with `TimeZoneInfo.FindSystemTimeZoneById` - once, at open. A host without time zone data, such as a minimal container image, cannot resolve - it, and the column then binds only as its `long` storage, not as `DateTimeOffset`. -* Trimming and AOT are separate switches: `PublishTrimmed` alone already removes what nothing +* Publish your own executable ahead of time in your CI, and run it over real files. A trim warning + at publish is cheap, a missing instantiation in production is not. +* Generic code you write over records stays generic. A method of yours that takes a + `TRecord : IVortexRecord` is compiled for each record it is called with, which is what you + want, as long as every call site names a concrete record type the compiler can see. +* A timestamp column with a named time zone resolves it once, at open, with + `TimeZoneInfo.FindSystemTimeZoneById`. A host without time zone data, such as a minimal container + image, cannot resolve it, and the column then binds only as its `long` storage, not as + `DateTimeOffset`. +* Trimming and AOT are separate switches. `PublishTrimmed` alone already removes what nothing reaches, and is worth having where AOT is not an option. diff --git a/docs/guide/nested-records.md b/docs/guide/nested-records.md index 92642191..788f941f 100644 --- a/docs/guide/nested-records.md +++ b/docs/guide/nested-records.md @@ -1,7 +1,6 @@ # Nested records -Read a struct column as a nested record, a list column as one buffer of elements, and filter on -both. +Read a struct column as a nested record, a list column as one buffer of elements, and filter on both. ```csharp await foreach (var v in file.Scan()) @@ -39,7 +38,7 @@ await foreach (var v in file.Scan()) origin: 66666 from FR, 9091 without a city ``` -The records are those of the demonstration's visits file: +These are the records of the demonstration's visits file: ```csharp [VortexRecord] public partial record struct Visit(Guid Id, DateTime StartedAt, int DurationMs, string? Referrer, ReadOnlyMemory Pages, Address Origin); @@ -60,30 +59,36 @@ segments are read: [VortexRecord] public partial record struct CountryOnly(string Country); ``` -Every column of `Visit` requested 2 162 496 bytes; `VisitCountry`, which reads `Origin.Country` -alone, 65 580. See [project-columns.md](project-columns.md) for the projection in general. +Reading every column of `Visit` requested 2 132 656 bytes, while `VisitCountry`, which reads +`Origin.Country` alone, requested 65 580. [project-columns.md](project-columns.md) covers projection +in general. ## A list column is its elements and their ranges -`v.Pages` is a `Column>`. It does not hand out one array per list: `Elements` is -a `Column` holding every list's elements back to back, and `pages[i]` is the `Range` of list -`i` inside it. `allPages[pages[i]]` slices it, with no allocation. Here 100 000 lists hold 200 000 +`v.Pages` is a `Column>`. It does not hand out one array per list. `Elements` is +a `Column` holding every list's elements back to back, and `pages[i]` is the `Range` of list `i` +inside it, so `allPages[pages[i]]` slices it without allocating. Here 100 000 lists hold 200 000 elements, none longer than four. Only `ToRecordsAsync` allocates one array per list, because a `Visit` row owns its `ReadOnlyMemory` ([read-rows.md](read-rows.md)). +A list of records, such as a member declared `ReadOnlyMemory
` or `List
`, works the +same way one level up. Its generated member is a `ListColumns
`, whose indexer gives the +`Range` of each row and whose `Elements` is a `Columns
` holding every row's records back to +back. The elements are read in one pass and then cut per row. + ## A nullable nested record -A record member of a nullable record type makes a nullable struct column: +A member of a nullable record type makes a nullable struct column: ```csharp [VortexRecord] public partial record struct Parcel(long Id, Address? Destination); ``` -`p.Destination` is still a `Columns
`, and its own nulls are on it: `IsValid(i)`, -`IsAllValid` and `ValidityWords`, the same bitmap shape as a column's. The fields of a null record -say nothing; check `IsValid` first. On 20 000 parcels, one in five without a destination: +`p.Destination` is still a `Columns
`, and its own nulls are on it: `IsValid(i)`, `IsAllValid` +and `ValidityWords`, with the same bitmap layout as a column's. The fields of a null record say +nothing, so check `IsValid` first. On 20 000 parcels, one in five without a destination: ``` parcels: 4000 without a destination, by IsValid; 0 with one but no city @@ -96,7 +101,7 @@ As a row, a null nested record is a null `Address?`. ## Filtering on nested fields and lists -The probe of a scan nests the same way, so a filter reaches into a record and into a list: +The probe of a scan nests the same way, so a filter can reach into a record and into a list: ``` Where(v => v.Origin.Country == "FR"): 66666 @@ -105,21 +110,23 @@ Where(v => v.Pages.Contains(7)): 5000 Where(v => v.Origin.IsNull), on a member that is never null: 0 ``` -`v.Origin.IsNull` and `IsNotNull` test the nested record itself; on a record member that is not -nullable, `IsNull` is simply never true. `Contains` on a list column is list membership, evaluated -per row by the kernel; zone maps do not prune it. +`v.Origin.IsNull` and `IsNotNull` test the nested record itself. On a record member that is not +nullable, `IsNull` is simply never true. `Contains` on a list column tests list membership and is +evaluated per row by the kernel, and the zone maps do not prune it. No filter compares a list of +records or a map. ## Watch out -* **A list of records is refused** by the generator (VX1005): no typed column reads one. A list of - primitives, strings or dates is fine. +* A list of nullable records, and a list of lists of records, are refused by the generator with + VX1005 ([diagnostics.md](diagnostics.md)). Hold the inner lists in a record instead. * A nullable list member is exposed as `ReadOnlyMemory`, and its nulls are read from the list column's `ValidityWords` or `IsValid`, not from the elements. -* `allPages` and every other span are borrowed from the batch, like any column: they end with the +* `allPages` and every other span are borrowed from the batch, like any column, and they end with the loop body. -The mapping of struct and list types to .NET is [07-dotnet-mapping.md](../design/07-dotnet-mapping.md), -and the record contract §3.1 and §4 of [14-public-api.md](../design/14-public-api.md). +The mapping of struct and list types to .NET is described in +[07-dotnet-mapping.md](../design/07-dotnet-mapping.md), and the record contract in +[the public API design](../design/14-public-api.md#4-records-the-generated-contract). ## Run it diff --git a/docs/guide/nullable-columns.md b/docs/guide/nullable-columns.md index b7200bc3..4ff7d5b6 100644 --- a/docs/guide/nullable-columns.md +++ b/docs/guide/nullable-columns.md @@ -1,7 +1,7 @@ # Nullable columns -Read a column that can hold nulls: the values, the bitmap that says which of them mean anything, -and the call that makes both unnecessary. +Read a column that can hold nulls: the values, the bitmap that says which of them mean anything, and +the call that makes both unnecessary. ```csharp await foreach (var (_, celsius, _) in scan) @@ -38,25 +38,25 @@ await foreach (var (_, celsius, _) in scan) } ``` -`Sum` is ten lines of `System.Numerics.Vector` at the bottom of the sample. +`Sum` is ten lines of `System.Numerics.Vector` at the end of the sample. ## Values and validity `Celsius` is declared `double?`, so its column is a `Column`: -* `Values` is the raw buffer, contiguous and 64-byte aligned, one slot per row. At a null row the - slot holds whatever the decoder left there: never read it without the bitmap. -* `ValidityWords` is the bitmap, 64 rows per `ulong`: bit `i % 64` of word `i / 64` is row `i`, - least significant first, and the bits past `Length` are zero. It is empty when `IsAllValid`, - which is what the first branch above tests. -* `IsValid(i)` asks for one row, and the indexer `celsius[i]` returns a `double?`. -* `NullCount` is counted from the words by `BitOperations.PopCount`, 64 rows per instruction. +* `Values` is the raw buffer, contiguous and aligned on 64 bytes, with one slot per row. A null row's + slot holds whatever the decoder left there, so never read it without the bitmap. +* `ValidityWords` is the bitmap, 64 rows per `ulong`. Bit `i % 64` of word `i / 64` is row `i`, least + significant first, and the bits past `Length` are zero. It is empty when `IsAllValid` is true, which + is what the first branch above tests. +* `IsValid(i)` asks about one row, and the indexer `celsius[i]` returns a `double?`. +* `NullCount` is computed from the words with `BitOperations.PopCount`, 64 rows per instruction. -## Two idioms, by where the nulls are +## Two idioms, depending on where the nulls are -**By the words**, as above: a batch with no null is summed with SIMD and no mask, a word of 64 -valid rows likewise, and only a word that holds a null is walked bit by bit. **Per element**, when -the per-row branch does not matter: +The loop above works by words. A batch with no null is summed with SIMD and no mask, so is a word of +64 valid rows, and only a word that holds a null is walked bit by bit. When the per-row branch does +not matter, a per-element loop is simpler: ```csharp for (int i = 0; i < celsius.Length; i++) @@ -71,17 +71,17 @@ for (int i = 0; i < celsius.Length; i++) What each costs on the demonstration file, where one temperature in fifty is missing: ``` -by the words 6.4 ms sum 29400000.0; 0 batches all valid, 0 words all valid, 15625 words with a null -by the words, nulls filtered out 10.7 ms sum 29400000.0; 123 batches all valid, 0 words all valid, 0 words with a null -per element 10.8 ms sum 29400000.0 -SumAsync 1.9 ms sum 29400000.0 +by the words 5.6 ms sum 29400000.0; 0 batches all valid, 0 words all valid, 15625 words with a null +by the words, nulls filtered out 14.6 ms sum 29400000.0; 123 batches all valid, 0 words all valid, 0 words with a null +per element 10.0 ms sum 29400000.0 +SumAsync 2.0 ms sum 29400000.0 ``` -A null every fifty rows puts one in every word of 64, so the two fast paths never fire here and the -words idiom does its bit-by-bit walk everywhere; it still beats the per-element branch. With -`Where(r => r.Celsius != null)`, the compacted batches come back all valid and take the first -branch, but the filter is paid for. The idiom pays when nulls are rare or clustered, and the words -tell you which case you are in. +With a null every fifty rows, every word of 64 holds one, so the two fast paths never fire and the +words idiom walks bit by bit everywhere. It still beats the per-element branch. With +`Where(r => r.Celsius != null)` the compacted batches come back all valid and take the first branch, +but you pay for the filter. The words idiom pays off when nulls are rare or clustered, and the words +themselves tell you which case you are in. ## Or let the scan do it @@ -89,26 +89,26 @@ tell you which case you are in. double sum = await file.Scan().SumAsync(r => r.Celsius); ``` -`SumAsync`, `AverageAsync` and the other aggregates take a nullable column and skip its nulls, block -by block, inside the scan, on the encoded form where the encoding allows it and in parallel when the -session allows it. Here it is the fastest of the four and has nothing to get wrong -([aggregates.md](aggregates.md)). +`SumAsync`, `AverageAsync` and the other aggregates accept a nullable column and skip its nulls, +block by block, inside the scan. They work on the encoded form where the encoding allows it, and in +parallel when the session allows it. Here it is the fastest of the four, and there is nothing to get +wrong ([aggregates.md](aggregates.md)). ## Watch out -* **`foreach (double v in celsius.Values)` silently counts garbage.** That is the wrong mean of +* `foreach (double v in celsius.Values)` silently counts garbage. That is the wrong mean of [scan-a-table.md](scan-a-table.md): 29.4001 instead of 30.0000. -* **Nullability is checked at binding.** A `double` member over a nullable column is refused with - `VortexSchemaException`; a `double?` member over a non-nullable column is fine, and its +* Nullability is checked at binding. A `double` member over a nullable column is refused with + `VortexSchemaException`. A `double?` member over a non-nullable column is fine, and its `ValidityWords` are always empty. -* **A null nested record has its own bitmap**: `Columns.ValidityWords` and `IsValid`, on - top of the bitmaps of its fields ([nested-records.md](nested-records.md)). -* **Counting nulls needs no scan.** `NullCount` summed over the batches gives 20 000, and so does - `Where(r => r.Celsius == null).CountAsync()`, answered from the null counts of the zone maps with - one read of 3 108 bytes ([filter-rows.md](filter-rows.md)). - -The figures come from one run of the sample on the demonstration file of a million rows; each -timing is the best of three passes. +* A null nested record has its own bitmap, `Columns.ValidityWords` and `IsValid`, on top of + the bitmaps of its fields ([nested-records.md](nested-records.md)). +* Counting nulls needs no scan. `NullCount` summed over the batches gives 20 000, and so does + `Where(r => r.Celsius == null).CountAsync()`, which is answered from the null counts of the zone + maps with a single read of 3 108 bytes ([filter-rows.md](filter-rows.md)). + +The figures come from one run of the sample on the demonstration file of a million rows. Each timing +is the best of three passes. ## Run it diff --git a/docs/guide/object-store.md b/docs/guide/object-store.md index 0650efb1..3035b5d6 100644 --- a/docs/guide/object-store.md +++ b/docs/guide/object-store.md @@ -1,7 +1,7 @@ # Object store -Implement the store seam for the service you use, measure what an operation asks of it, and open -a single file out of one. +Implement the store interface for the service you use, measure what an operation asks of it, and +open a single file straight out of a store. ```csharp await using MemoryObjectStore memory = new MemoryObjectStore(); @@ -20,17 +20,17 @@ await memory.DeleteAsync(["logs/00", "logs/01", "logs/99"], ct); ``` ``` -put: Created, then Exists; the store holds 1 object, 1508212 bytes -head: 1508212 bytes, token 1; a missing key: null +put: Created, then Exists; the store holds 1 object, 1508316 bytes +head: 1508316 bytes, token 1; a missing key: null range: asked 64 bytes 8 before the end, got 8, token 1, contiguous True list under logs/ after logs/01, first two: logs/02, logs/03 deleted a batch of three keys, one absent: 4 objects left ``` -`IObjectStore` is the seam a dataset reads and writes through. It lives in `Vorticity.Dataset`, -which is experimental (see [datasets.md](datasets.md)), and it depends on no cloud SDK: talking -to S3, Azure Blob Storage or Google Cloud Storage is the implementation's business, and so are -retries, pooling and credentials. +`IObjectStore` is the interface a dataset reads and writes through. It lives in `Vorticity.Dataset`, +which is experimental (see [datasets.md](datasets.md)), and depends on no cloud SDK. Talking to S3, +Azure Blob Storage or Google Cloud Storage is the implementation's job, and so are retries, pooling +and credentials. ```csharp public interface IObjectStore : IAsyncDisposable @@ -47,27 +47,27 @@ public interface IObjectStore : IAsyncDisposable | | | |---|---| -| `PutIfAbsentAsync` | **the one that must be exact.** `Created` when the key was free and now holds the bytes, `Exists` when it was taken, and never `Created` for two racing callers: this is what makes a commit atomic. The content is a `PipeReader` of announced `length`, read as it is sent, so an object never has to fit in memory and a store can choose a multipart upload by the length; content that ends early or runs past it is refused and nothing is created. A cancelled put leaves the key created or absent, never half written | -| `GetRangeAsync` | the bytes of a range and the object's token **in one answer**, because a token read in a separate call proves nothing about the bytes. The length is clamped to the object, which is how a reader opens a file by its tail without knowing its length; a missing key throws `ObjectNotFoundException` | -| `HeadAsync` | length, token and creation time, or `null` for a missing key, not an exception; and, on a store that keeps the object under a retention lock, the date before which it refuses to delete it (`RetainUntil`) and whether a legal hold keeps it (`LegalHold`) | -| `ListAsync` | the keys under a prefix, in ordinal order, after `startAfter`. The implementation fetches pages as the enumeration advances, so a caller that stops early pays for one page | -| `DeleteAsync` | a batch of keys, in any order; an absent key is not an error, and a store whose service caps a batch splits it. It reports nothing, since a store cannot say which keys existed | - -An object is never overwritten, and its token changes whenever the bytes under its key change, -which for immutable objects means the key was deleted and created again. Implementations must be +| `PutIfAbsentAsync` | The one that must be exact. It returns `Created` when the key was free and now holds the bytes, `Exists` when it was taken, and never `Created` to two racing callers. That is what makes a commit atomic. The content is a `PipeReader` of the announced `length`, read as it is sent, so an object never has to fit in memory and a store can switch to a multipart upload based on the length. Content that ends early or runs long is refused and nothing is created. A cancelled put leaves the key either created or absent, never half written | +| `GetRangeAsync` | The bytes of a range and the object's token in a single answer, because a token read in a separate call proves nothing about the bytes. The length is clamped to the object, which lets a reader open a file by its tail without knowing its length. A missing key throws `ObjectNotFoundException` | +| `HeadAsync` | Length, token and creation time, or `null` for a missing key (not an exception). On a store with retention locks, it also gives the date before which a delete is refused (`RetainUntil`) and whether a legal hold applies (`LegalHold`) | +| `ListAsync` | The keys under a prefix, in ordinal order, after `startAfter`. The implementation fetches pages as the enumeration advances, so a caller that stops early pays for one page | +| `DeleteAsync` | A batch of keys, in any order. An absent key is not an error, and a store whose service caps batch sizes splits the batch. It reports nothing, since a store cannot always say which keys existed | + +An object is never overwritten, and its token changes whenever the bytes under its key change, which +for immutable objects means the key was deleted and created again. Implementations must be thread-safe and list with strong consistency. S3, Azure Blob Storage and Google Cloud Storage all -offer the conditional put `PutIfAbsentAsync` needs; a store that cannot promise it cannot host a -dataset, because two writers would both believe they committed. +offer the conditional put that `PutIfAbsentAsync` needs. A store that cannot promise it cannot host +a dataset, because two writers would both believe they had committed. -A store under a retention lock, S3 Object Lock, Azure immutable blob storage or a GCS bucket lock, -reports each object's lock in its head and refuses a delete before the date. A dataset on such a store -says so once, `DatasetOptions.LockedStore` at creation, and then writes as few bytes as it can and -vacuums only what the lock has let go; `MemoryObjectStore.RetainFor` and `Hold` play such a store in -tests. +A store with retention locks (S3 Object Lock, Azure immutable blob storage, a GCS bucket lock) +reports each object's lock in its head and refuses a delete before the date. A dataset on such a +store declares it once at creation with `DatasetOptions.LockedStore`. It then writes as few bytes as +it can and vacuums only what the lock has released. `MemoryObjectStore.RetainFor` and `Hold` +simulate such a store in tests. `ObjectRange` holds its bytes as a `SegmentLease`, so a store can hand back a pooled buffer, memory -it keeps, or a response that arrived in pieces, and gets it back when the caller disposes the -range. `Bytes` is always valid; `Memory` only when `IsContiguous`. +it keeps, or a response that arrived in pieces, and gets it back when the caller disposes the range. +`Bytes` is always valid, and `Memory` only when `IsContiguous` is true. ## The three that ship @@ -77,30 +77,32 @@ await using MemoryObjectStore memory = new MemoryObjectStore(); // await using CountingObjectStore counting = new CountingObjectStore(memory); // around any store ``` -`FileObjectStore` maps keys to files under a root directory; its `Durable` option flushes every -put to the device before it returns, for a store that is the system of record. Claiming a key is -atomic there, but writing the content is not: a process that dies mid-put leaves a short object -under a taken key, which the format detects when the object is read. A read that meets a put still -writing waits for it; a commit left short throws `TornCommitException`, which names its version, and -`VortexDataset.RemoveTornCommitAsync` removes it. `MemoryObjectStore` is a test double with the seams to prove your -own error paths: `Fails` refuses a chosen operation, `CrashesAfterPut` throws after a put has -landed, so the object exists and its writer never learned it, `Latency` delays every call, and -`TimeProvider` stamps the objects it creates. +`FileObjectStore` maps keys to files under a root directory. Its `Durable` option flushes every put +to the device before returning, for a store that is the system of record. Claiming a key is atomic +there, but writing the content is not, so a process that dies mid-put leaves a short object under a +taken key, which the format detects when the object is read. A read that meets a put still in +progress waits for it. A commit left short throws `TornCommitException`, which names its version, +and `VortexDataset.RemoveTornCommitAsync` removes it. + +`MemoryObjectStore` is a test double with hooks for exercising your own error paths. `Fails` refuses +a chosen operation, `CrashesAfterPut` throws after a put has landed (so the object exists but its +writer never learned it), `Latency` delays every call, and `TimeProvider` stamps the objects it +creates. ``` a store told to fail: ObjectStoreException: The store was told to fail GetRange on 'files/readings.vortex'. a key that is not there: ObjectNotFoundException, key 'files/nothing' ``` -`ObjectStoreException` is a store that refused or failed, `ObjectNotFoundException` a key that is -not there; both derive from `VortexException`, and both are yours to throw from an implementation. +`ObjectStoreException` means the store refused or failed, and `ObjectNotFoundException` means a key +is not there. Both derive from `VortexException`, and an implementation should throw them too. ## Pricing what you do `CountingObjectStore` wraps any store and counts what passes through: `Requests`, `CountOf` each -`ObjectOperation`, `BytesRead`, `BytesWritten`, `KeysListed`, and `DependentSteps`, a lower bound -on the round trips that waited one after another, which is what decides latency on a store where -a request costs tens of milliseconds. `Reset` starts again. +`ObjectOperation`, `BytesRead`, `BytesWritten`, `KeysListed`, and `DependentSteps`. That last one is +a lower bound on the round trips that had to wait one after another, and it is what sets latency on +a store where each request costs tens of milliseconds. `Reset` starts the counts again. ``` creating a dataset: 5 requests (1 get, 0 head, 1 put, 3 list), 5 dependent steps, 123 bytes read, 123 written @@ -112,8 +114,8 @@ same way. ## Opening one file out of a store -A single file needs no dataset. `ISegmentSource`, the core's I/O seam, is two reads and a length, -and a store object is one in a dozen lines: +A single file needs no dataset. `ISegmentSource`, the core library's I/O interface, is two reads and +a length, and wrapping a store object takes a dozen lines: ```csharp private sealed class StoreSegmentSource(IObjectStore store, string key, long length) : ISegmentSource @@ -140,32 +142,32 @@ await using (VortexFile opened = await VortexSession.Default.OpenAsync(new Store ``` ``` -open from the store: 1000000 rows; 2 requests (1 get, 1 head, 0 put, 0 list), 2 dependent steps, 65588 bytes read, 0 written +open from the store: 1000000 rows; 2 requests (1 get, 1 head, 0 put, 0 list), 2 dependent steps, 65564 bytes read, 0 written mean of Celsius for Day >= 900: 30.000; 4 requests (4 get, 0 head, 0 put, 0 list), 4 dependent steps, 151392 bytes read, 0 written -a full scan, 1000000 rows: 45 requests (45 get, 0 head, 0 put, 0 list), 40 dependent steps, 1465556 bytes read, 0 written -a full scan through a session with a segment cache, 1000000 rows: 45 requests (45 get, 0 head, 0 put, 0 list), 45 dependent steps, 1465556 bytes read, 0 written +a full scan, 1000000 rows: 45 requests (45 get, 0 head, 0 put, 0 list), 35 dependent steps, 1465556 bytes read, 0 written +a full scan through a session with a segment cache, 1000000 rows: 45 requests (45 get, 0 head, 0 put, 0 list), 37 dependent steps, 1465556 bytes read, 0 written ``` -The open costs one ranged read of the tail, plus the head the sample asks for the length. From -there every projection and filter of the guide works, and each saves requests, not just bytes: a -filtered mean fetches 4 of the file's 51 segments. The session's `MaxConcurrentReads` bounds -the reads in flight across every file it opens this way, and its segment cache applies too. A -scan fetches each segment once, even a chunk that spans several blocks, and none that lies in the -tail the open read: 45 requests and 1.47 MB for a 1.51 MB file, with a cache or without. The cache -pays across scans: a second scan through the same session fetches nothing -([open-a-file.md](open-a-file.md)). +The open costs one ranged read of the tail, plus the head the sample uses to learn the length. From +there every projection and filter in the guide works, and each one saves requests as well as bytes: +the filtered mean fetches 4 of the file's 54 segments. The session's `MaxConcurrentReads` bounds the +reads in flight across every file opened this way, and its segment cache applies too. A scan fetches +each segment once, even a chunk that spans several blocks, and never one that sits in the tail the +open already read. That gives 45 requests and 1.47 MB for a 1.51 MB file, with or without a cache. +The cache pays off across scans: a second scan through the same session fetches nothing (see +[open-a-file.md](open-a-file.md)). The source receives the ranges of a batch together, so a source over a network store can coalesce -neighbouring ranges into one request; the sample's does not, to stay short. Because the reads of a -scan overlap, `DependentSteps` moves a little from one run to the next; the request and byte counts -do not. +neighbouring ranges into one request. The sample's source does not, to stay short. Because the reads +of a scan overlap, `DependentSteps` varies a little from one run to the next, while the request and +byte counts do not. ## Watch out -* **The seam is experimental** with the rest of the package: pin the version. -* `ListAsync` must honour `startAfter` and ordinal order, or a dataset's listing of its commits - goes wrong. Test an implementation against `MemoryObjectStore`: it is the reference behaviour. -* An `ObjectRange` owns its buffer. Dispose it once, and do not read its bytes after. +* The interface is experimental along with the rest of the package, so pin the version. +* `ListAsync` must honour `startAfter` and ordinal order, or a dataset's listing of its commits goes + wrong. Test an implementation against `MemoryObjectStore`, which is the reference behaviour. +* An `ObjectRange` owns its buffer. Dispose it once, and do not read its bytes afterwards. ## Run it diff --git a/docs/guide/observability.md b/docs/guide/observability.md index 603e88cb..5365c75a 100644 --- a/docs/guide/observability.md +++ b/docs/guide/observability.md @@ -1,7 +1,7 @@ # Observability -Watch what scans and writes do: the process-wide counters, an activity per operation, and the -numbers of one scan. +Watch what scans and writes do: process-wide counters, an activity per operation, and the figures of +a single scan. ```csharp ConcurrentDictionary totals = new(StringComparer.Ordinal); @@ -28,12 +28,16 @@ using ActivityListener activities = new ActivityListener ActivitySource.AddActivityListener(activities); ``` -The library reports through the two instrumentation APIs of the base class library, -`System.Diagnostics.Metrics` and `System.Diagnostics.Activity`, under one name, `"Vorticity"`, -which `VortexDiagnostics.MeterName` and `VortexDiagnostics.ActivitySourceName` hold. Anything that -listens to those APIs sees it: OpenTelemetry with `AddMeter("Vorticity")` and -`AddSource("Vorticity")`, `dotnet-counters monitor --counters Vorticity`, or a listener of your -own like the one above. No package is involved on the library's side. +The library reports through the standard .NET instrumentation APIs, all under the single name +`"Vorticity"`: + +* a `Meter` (`System.Diagnostics.Metrics`), whose name `VortexDiagnostics.MeterName` holds +* an `ActivitySource` (`System.Diagnostics.Activity`), whose name `VortexDiagnostics.ActivitySourceName` holds +* an `EventSource` with polling counters for the engine's internals (see [below](#the-event-counters)) + +Anything that listens to those APIs sees it: OpenTelemetry with `AddMeter("Vorticity")` and +`AddSource("Vorticity")`, `dotnet-counters monitor --counters Vorticity`, or a listener of your own +like the one above. The library takes no dependency for any of this. ## The meter @@ -48,10 +52,10 @@ instrument vortex.cache.misses ({segment}): Segments a session's cache did not h instrument vortex.write.bytes (By): Bytes written by writers. ``` -Eight counters of `long`, with no tags: they add up every scan and every write of the process. -A scan adds its figures when its sink ends; a writer adds its bytes as they reach the sink, at -each flush and at completion. After the sample's two filtered scans, a count on the tool path and -a small write: +These are eight `long` counters with no tags, adding up every scan and every write in the process. A +scan adds its figures when its sink ends, and a writer adds its bytes as they reach the sink, at each +flush and at completion. After the sample's two filtered scans, a count on the tool path and a small +write: ``` total vortex.cache.hits = 8 @@ -61,33 +65,44 @@ total vortex.scan.blocks_pruned = 218 total vortex.scan.bytes_requested = 690916 total vortex.scan.requests = 27 total vortex.scan.rows = 200000 -total vortex.write.bytes = 3012 +total vortex.write.bytes = 3020 ``` -`vortex.scan.rows` counts rows delivered, so a count, which delivers none, adds requests and -bytes but no rows. +`vortex.scan.rows` counts rows delivered, so a count, which delivers none, adds requests and bytes +but no rows. ## The activities -One activity per scan, named `vortex.scan.typed` or `vortex.scan.tool` after the path it took, -and one per write, `vortex.write`. Each carries what the operation did as tags, set when it ends: +Each scan gets one activity, named `vortex.scan.typed` or `vortex.scan.tool` after the path it took, +and each write gets one named `vortex.write`. Each carries what the operation did as tags, set when it +ends: ``` activity vortex.scan.typed [Unset]: vortex.rows=100000, vortex.batches=5, vortex.requests=6, vortex.bytes_requested=180368, vortex.blocks_decoded=14, vortex.blocks_pruned=109 activity vortex.scan.tool [Unset]: vortex.rows=0, vortex.batches=0, vortex.requests=15, vortex.bytes_requested=330180, vortex.blocks_decoded=123, vortex.blocks_pruned=0 -activity vortex.write [Unset]: vortex.rows=2, vortex.bytes=3012, vortex.completed=True +activity vortex.write [Unset]: vortex.rows=2, vortex.bytes=3020, vortex.completed=True activity vortex.write [Error]: vortex.rows=1, vortex.bytes=0, vortex.completed=False ``` A write that is abandoned, or disposed without `CompleteAsync`, ends its activity with -`vortex.completed=False` and the status `Error`, described `abandoned`. The activities are -children of whatever activity is current when the scan or the writer starts, so a request's trace -shows the scans it ran. `ExplainAsync` reads statistics and zone maps without starting one. +`vortex.completed=False` and the status `Error`, described as `abandoned`. Activities are children of +whatever activity is current when the scan or the writer starts, so a request's trace shows the +scans it ran. `ExplainAsync` reads statistics and zone maps without starting one. + +## The event counters -## The numbers of one scan +The `EventSource` named `Vorticity` publishes eleven incrementing counters about the engine's inner +work: `segments-requested`, `bytes-requested`, `zones-pruned`, `zones-total`, `index-runs-read`, +`cursor-seeks`, `cursor-steps`, `count-blocks-proven`, `count-blocks-decoded`, `key-order-windows` and +`key-order-window-splits`. They answer in production what a single scan's statistics answer in a +test, above all whether an index earns the bytes it costs. Watch them with +`dotnet-counters monitor --counters Vorticity`. Nothing is paid until a listener attaches: every hook +is one flag test, and the counters are only created once a listener asks for them. -The counters are the process; `Statistics` on a scan is that scan alone, read once its sink has -run: +## The figures of one scan + +The counters cover the whole process, while `Statistics` on a scan covers that scan alone and is +valid once its sink has run: ```csharp Scan scan = file.Scan().Where(r => r.Day >= 900); @@ -107,14 +122,18 @@ scan 1: 100000 rows in 5 batches, 6 requests, 180368 bytes, 14 blocks decoded, 1 scan 2: 100000 rows in 5 batches, 6 requests, 180368 bytes, 14 blocks decoded, 109 pruned, 6 cache hits ``` -`ScanStatistics` is a `readonly record struct` of seven `long`s: `Rows`, `Batches`, `Requests`, -`BytesRequested`, `BlocksDecoded`, `BlocksPruned` and `CacheHits`. `Requests` counts the segments -the scan asked for, each once, the ones the cache then served included: the second scan finds all -6 of its requests in the cache, where the first found none. Neither asks for the zone maps it -consults or for the other 6 segments its rows lie in: they are in the tail the open read, which -the file serves itself. The same record is on the tool scan and on a grouped aggregation. What the scan was going to do, before it -runs, is `ExplainAsync`: [statistics-and-pruning.md](statistics-and-pruning.md) puts the two side -by side. +`ScanStatistics` is a `readonly record struct` with seven `long`s (`Rows`, `Batches`, `Requests`, +`BytesRequested`, `BlocksDecoded`, `BlocksPruned` and `CacheHits`) plus `Grouping`, which a group by +fills with a `GroupStatistics`: the groups found and the most held at once, the peak memory, the +lanes, what was spilled and the time to the first batch. `Grouping` is null for a scan without a +group by. + +`Requests` counts the segments the scan asked for, each once, including those the cache then served. +The second scan finds all 6 of its requests in the cache, where the first found none. Neither asks +for the zone maps it consults, nor for the other 6 segments its rows lie in, because those sit in the +tail the open already read and the file serves them itself. The same statistics exist on the tool +scan and on a grouped aggregation. What the scan will do before it runs is `ExplainAsync`, and +[statistics-and-pruning.md](statistics-and-pruning.md) puts the two side by side. ## The segment cache @@ -130,26 +149,25 @@ await using (VortexFile file = await session.OpenAsync(new FileSegmentSource(pat cache: 8 hits, 19 misses, 0 bytes held ``` -A path handed to `OpenAsync` is memory-mapped by its first scan, unless the session's `MapFiles` -is false, and a mapping has nothing to cache: its reads never reach the cache and never touch its -counters. `SegmentCache` exposes `Capacity`, `Size`, `Hits` and -`Misses` for the session as a whole; a file's entries leave the cache when the file is disposed, -which is why nothing is held at the end. [threads.md](threads.md) says when a cache is worth its -memory. +A path passed to `OpenAsync` is memory-mapped by its first scan, unless the session's `MapFiles` is +false, and a mapping has nothing to cache: its reads never reach the cache or its counters. +`SegmentCache` exposes `Capacity`, `Size`, `Hits` and `Misses` for the whole session. A file's +entries leave the cache when the file is disposed, which is why nothing is held at the end. +[threads.md](threads.md) explains when a cache is worth its memory. ## What it costs -Nothing without a listener. A counter with no listener is a check of a flag, and an activity is -created only when some listener samples the source; the scan's own `Statistics` are counted -either way, since they are what the counters are made of. +Without a listener it costs nothing. A counter with no listener costs a flag check, and an activity +is only created when a listener samples the source. A scan's own `Statistics` are counted either way, since +the counters are built from them. ## Watch out -* **The counters carry no tags.** Tell two files or two tenants apart with one session each and - the scan's `Statistics`, or with the activities, which nest under your own. -* The scan counters are added when the scan ends, each for whoever listens to it. -* A scan's `Statistics.CacheHits` counts the segments the cache served to that scan alone, even - while other scans of the session run; the cache's own `Hits` counts the whole session's. +* The meter's counters carry no tags. To tell two files or two tenants apart, use one session each + and the scan's `Statistics`, or the activities, which nest under your own. +* The scan counters are added when the scan ends. +* A scan's `Statistics.CacheHits` counts the segments the cache served to that scan alone, even while + other scans of the session run. The cache's own `Hits` counts the whole session's. ## Run it diff --git a/docs/guide/open-a-file.md b/docs/guide/open-a-file.md index fe961f44..582f915b 100644 --- a/docs/guide/open-a-file.md +++ b/docs/guide/open-a-file.md @@ -1,7 +1,7 @@ # Open a file -A path, a mapped file, positional reads, or bytes you already hold: the ways in differ in who holds -the bytes, and the open reads the same thing whichever you choose. +You can open a file from a path, from a mapping, through positional reads, or from bytes you already +hold. These differ in who holds the bytes. The open itself reads the same thing whichever you choose. ## From a path @@ -16,29 +16,29 @@ await using (VortexFile file = await VortexFile.OpenAsync(path)) ``` struct{Day: i32, Celsius: f64?, City: utf8} -1000000 rows, 1508212 bytes, edition Core20260800 -identity f6595d25-2b99-4160-9071-63ffd454d25f, metadata keys [] +1000000 rows, 1508316 bytes, edition Core20260800 +identity 0fb569a5-6900-41ba-a069-ed5ca7110ead, metadata keys [] ``` -`VortexFile.OpenAsync(path)` opens in `VortexSession.Default`: it reads the file's tail, and the -first scan that reads data maps the file into memory, which every read after it goes through. An -open that asks for nothing but the schema, the statistics or a count they answer maps nothing, -and costs a fifth less. This is the one to use unless you have a reason not to. +`VortexFile.OpenAsync(path)` opens the file in `VortexSession.Default`. It reads the end of the file, +and the first scan that reads data maps the file into memory, which every later read goes through. +An open that only asks for the schema, the statistics, or a count they answer maps nothing and costs +about a fifth less. This is the call to use unless you have a reason not to. -What the file tells you before a scan: +Before any scan, the file tells you: | member | what it is | |---|---| | `Schema` | the columns and their types, a `VortexSchema` you can index, enumerate and print | -| `RowCount`, `Length` | rows, and bytes on disk | -| `Edition` | the oldest edition whose readers can open this file, worked out from what it declares ([editions.md](editions.md)) | -| `Identity` | a `Guid` drawn by every write and append of this library; `Guid.Empty` for a file another writer produced | -| `Metadata` | the user metadata by key; `ReadAsync(key)` returns a copy of a value | -| `Statistics` | per column null count, order flags, and the bounds of a numeric column ([statistics-and-pruning.md](statistics-and-pruning.md)) | -| `TornTail` | not null when the file opened at the version before a torn append ([append-and-repair.md](append-and-repair.md)) | +| `RowCount`, `Length` | the number of rows, and the bytes on disk | +| `Edition` | the oldest edition whose readers can open this file, worked out from the components it declares ([editions.md](editions.md)) | +| `Identity` | a `Guid` drawn by every write and every append of this library, `Guid.Empty` for a file another writer produced | +| `Metadata` | the user metadata by key, where `ReadAsync(key)` returns a copy of a value | +| `Statistics` | per column, the null count, the order flags, and the bounds of a numeric column ([statistics-and-pruning.md](statistics-and-pruning.md)) | +| `TornTail` | not null when the file was opened at its version before a torn append ([append-and-repair.md](append-and-repair.md)) | -The identity is stored as metadata of the library's own, which `Metadata.Keys` leaves out: the keys -it lists are the ones the writer's `VortexWriteOptions.Metadata` gave. +The identity is stored as metadata of the library's own, which `Metadata.Keys` leaves out. The keys +it lists are the ones the writer received through `VortexWriteOptions.Metadata`. ## In a session, from a source @@ -52,42 +52,42 @@ await TimeAsync("positional reads", await session.OpenAsync(new FileSegmentSourc await TimeAsync("bytes in memory", await session.OpenAsync(new MemorySegmentSource(bytes))); ``` -`TimeAsync` scans the whole file three times and prints the best: +`TimeAsync` scans the whole file three times and prints the best time: ``` -a path a full scan in 2.8 ms -a mapped source a full scan in 2.2 ms -positional reads a full scan in 2.4 ms -bytes in memory a full scan in 2.3 ms +a path a full scan in 2.2 ms +a mapped source a full scan in 2.3 ms +positional reads a full scan in 2.2 ms +bytes in memory a full scan in 2.2 ms ``` -A session owns the memory pool, the segment cache, the bound on reads in flight and the degree of +A session owns the memory pool, the segment cache, the limit on reads in flight and the degree of parallelism of every file opened through it ([threads.md](threads.md)). `OpenAsync(path)` on a -session reads a path as the static call does; `OpenAsync(ISegmentSource)` takes any source, and -the file then owns it: disposing the file disposes the source. The three sources of -`Vorticity.IO`: +session reads a path the same way the static call does. `OpenAsync(ISegmentSource)` accepts any +source, and the file then owns it: disposing the file disposes the source. `Vorticity.IO` provides +three sources: -| source | reads by | when | +| source | reads through | use it when | |---|---|---| -| `MemoryMappedSegmentSource` | pointers into one mapping of the whole file, made at the open | a file you want mapped before its first scan; a path gives the same once a scan reads data | -| `FileSegmentSource` | `RandomAccess` positional reads on one handle | a file too large to map, or a host that forbids mappings | -| `MemorySegmentSource` | views into bytes you hold, pinned for the source's life, or into one aligned copy of them | the file came over the wire, out of a cache, or from a test | +| `MemoryMappedSegmentSource` | pointers into one mapping of the whole file, made at open | you want the file mapped before its first scan (a path gives you the same once a scan reads data) | +| `FileSegmentSource` | positional `RandomAccess` reads on one handle | the file is too large to map, or the host forbids mappings | +| `MemorySegmentSource` | views into bytes you hold, or into one aligned copy of them | the file came over the network, out of a cache, or from a test | -A `MemorySegmentSource` keeps its bytes on a 64-byte boundary, the widest alignment a segment -declares, so that every read is a view a decoder can take as it is. Bytes already on one are read -where they lie; bytes that are not, which is where a `byte[]` puts them, are copied once, when the -source is made, into memory the source owns and frees. To spare that copy, rent the memory from an -`AlignedMemoryPool` and read the file into it. +A `MemorySegmentSource` keeps its bytes on a 64-byte boundary, the widest alignment a segment can +declare, so every read is a view a decoder can use as it is. Bytes that already sit on such a +boundary are read where they lie. Bytes that do not, which is the usual case for a `byte[]`, are +copied once into memory the source owns, when the source is created. To avoid that copy, rent the +memory from an `AlignedMemoryPool` and read the file into it. For a file on an object store, implement `ISegmentSource` for your service: a length and two read -methods ([object-store.md](object-store.md)). The demonstration file fits in -the page cache, and the four ways in scan it within a millisecond of each other: the choice is about -who holds the bytes, not about speed on a warm disk. +methods ([object-store.md](object-store.md)). The demonstration file fits in the page cache, so the +four sources scan it within a millisecond of each other. The choice is about who holds the bytes, +not about speed on a warm disk. ## What an open reads -The sample wraps a `FileSegmentSource` in `CountingSource`, an `ISegmentSource` of thirty lines at -the bottom of the sample that counts what it is asked for and passes every call through: +The sample wraps a `FileSegmentSource` in `CountingSource`, a thirty-line `ISegmentSource` at the end +of the sample that counts what it is asked for and forwards every call: ```csharp CountingSource counting = new CountingSource(new FileSegmentSource(path)); @@ -100,16 +100,16 @@ await using (VortexFile file = await session.OpenAsync(counting)) ``` ``` -the open: 1 request, 65588 bytes +the open: 1 request, 65564 bytes CountAsync 1000000, MaxAsync 49.9: still 1 request ``` -One read of the last 64 KiB, from the 64-byte boundary before them so that the segments inside keep -their alignment: 52 bytes more here. It finds the postscript, and through it the schema, the -layout, the footer and the file statistics; one of them that begins before that window costs one -more read, issued once for the whole gap. Metadata values are read when asked for, from the tail -the open already holds when they lie in it. Neither the row count nor `MaxAsync` on a column the -statistics cover asks for anything more. +The open makes one read of the last 64 KiB, starting at the 64-byte boundary just before them so that +the segments inside keep their alignment, which here adds 28 bytes. That read covers the postscript, +and through it the schema, the layout, the footer and the file statistics. When one of them starts +before that window, a second read fetches the whole gap. Metadata values are read on demand, from +the bytes the open already holds when they lie there. Neither the row count nor `MaxAsync` on a +column the statistics cover asks for anything more. `VortexOpenOptions` shapes the open: @@ -120,16 +120,16 @@ await using (VortexFile file = await session.OpenAsync(told, wide)) ``` ``` -InitialReadSize 256 KiB: 1 request, 262196 bytes +InitialReadSize 256 KiB: 1 request, 262172 bytes ``` | option | default | what it changes | |---|---|---| -| `InitialReadSize` | 65 536 | the tail read; raise it for a file whose footer is large, so that the open stays one request, or, over a source that fetches, to hold a small file whole | +| `InitialReadSize` | 65 536 | the size of the tail read. Raise it for a file with a large footer so the open stays one request, or to fetch a small file whole from a remote source | | `Length` | probed | skips asking the source for its length | | `Schema` | the file's | a schema for a file written without one, or to skip reading the embedded one | | `TornTail` | `ReadPrevious` | `Refuse` throws on a torn tail instead of opening the version before it | -| `VerifyStatistics`, `MaxDecompressedSize` | off, 256 MiB | how much the open trusts the bytes ([limits.md](limits.md)) | +| `VerifyStatistics`, `MaxDecompressedSize` | off, 256 MiB | how far the open trusts the bytes ([limits.md](limits.md)) | | `IndexFragments` | none | indexes built for this file elsewhere ([indexes.md](indexes.md)) | ## What a scan then reads @@ -145,34 +145,33 @@ with a segment cache, scan 2: the source served 0 requests, 0 bytes; 45 cache hi ``` The plan of a full scan names 45 segments, 1.47 MB of the file's 1.51 MB, and the source serves -exactly that: 45 requests for 1.47 MB. The file has 51 segments; the six that lie in the window -the open read are served from it, and a file shorter than the window is read by its open alone. A -scan reads each segment once, and a segment that spans several batches is shared by them rather -than read again for each. The cache works across scans: the first scan of a session with a -`SegmentCache` reads the same 45 segments and finds none in the cache, and a second scan of the -same file reads nothing, the 45 segments coming from the cache. Over a source where a read is a -request, that is the reason to give the session one. +exactly that. The file's data lies in 51 segments, and the six that fall inside the window the open +read are served from it. A file shorter than that window is read entirely by its open. A scan reads +each segment once, and a segment that spans several batches is shared by them rather than read +again. A `SegmentCache` works across scans: the first scan of the session reads the 45 segments and +finds none in the cache, and a second scan of the same file reads nothing because all 45 come from +the cache. When every read is a request to a remote service, that is the reason to give the session +a cache. ## Watch out -* **A source handed to `OpenAsync` belongs to the file.** Keep the bytes of a - `MemorySegmentSource` alive until the file is disposed. -* **Dispose the files before the session.** Disposing a session while one of its files is open - throws `InvalidOperationException` naming the file. -* Two files opened over the same path are two independent readers. Opened from a path in one - session they share one mapping, which the session keeps once both are closed, for the next open - of the same file (`MappedFileCacheCount`). A file is immutable once open and safe to scan from - several threads at once. -* **A file deleted once closed keeps its disk space while the session keeps it mapped**, up to 64 - files by default. A process that deletes the files it has read calls `ReleaseMappedFiles()` on - their session, `VortexSession.Default` for a file opened with `VortexFile.OpenAsync(path)`, or - creates its session with `MappedFileCacheCount = 0`. -* **A mapped file must not be cut short under a scan.** On Linux and macOS, a file that another - process truncates while a scan reads it through its mapping faults on the pages past its new - end, and the fault kills the process. A writer of this library cannot do it while the file is - open. A service that reads files other processes may truncate creates its session with - `MapFiles = false`: every read is then positional, and a cut file throws - `VortexFormatException`. +* A source handed to `OpenAsync` belongs to the file. Keep the bytes of a `MemorySegmentSource` + alive until the file is disposed. +* Dispose the files before the session. Disposing a session while one of its files is open throws + `InvalidOperationException`, naming the file. +* Two files opened over the same path are two independent readers. Opened from a path in the same + session, they share one mapping, which the session keeps after both are closed for the next open + of that file (`MappedFileCacheCount`). An open file is immutable and safe to scan from several + threads at once. +* A file deleted after it was closed keeps its disk space for as long as its session keeps it + mapped, which is up to 64 files by default. A process that deletes the files it has read should + call `ReleaseMappedFiles()` on their session (`VortexSession.Default` for files opened with + `VortexFile.OpenAsync(path)`), or create its session with `MappedFileCacheCount = 0`. +* A mapped file must not be truncated during a scan. On Linux and macOS, if another process cuts a + file short while a scan reads it through its mapping, the access to the missing pages faults and + kills the process. A writer of this library cannot do that while the file is open. A service that + reads files other processes may truncate should create its session with `MapFiles = false`: every + read is then positional, and a truncated file throws `VortexFormatException` instead. The figures come from one run of the sample on the demonstration file of a million rows. diff --git a/docs/guide/owned-batches.md b/docs/guide/owned-batches.md index 1829e19f..097ac9d5 100644 --- a/docs/guide/owned-batches.md +++ b/docs/guide/owned-batches.md @@ -49,26 +49,26 @@ await producer; through a channel: 17 batches, 1000000 rows, mean 30.00 degrees ``` -The columns of an ordinary `await foreach` are borrowed: they are valid until the loop moves on, and +The columns of an ordinary `await foreach` are borrowed. They are valid until the loop moves on, and the compiler keeps them inside the loop body ([scan-a-table.md](scan-a-table.md)). A pipeline needs -the opposite, a batch that outlives the loop and crosses threads. `ToBatchesAsync` gives one: each -`RecordBatch` is **owned** by whoever holds it, and whoever holds it last disposes it. +the opposite: a batch that outlives the loop and crosses threads. `ToBatchesAsync` gives you that. +Each `RecordBatch` is owned by whoever holds it, and whoever holds it last disposes it. ## What an owned batch is -A `RecordBatch` has its `Schema`, its `RowCount` and its `StartRow`, and two ways in: +A `RecordBatch` has a `Schema`, a `RowCount` and a `StartRow`, and two ways in: * `batch.As()` gives the same `Columns` a typed scan gives, with the generated members (`cols.Celsius`), valid until the batch is disposed. -* `batch.View` gives a `BatchView`, columns by index or by name, for code that has no record type - ([untyped-files.md](untyped-files.md)). +* `batch.View` gives a `BatchView`, with columns by index or by name, for code that has no record + type ([untyped-files.md](untyped-files.md)). -After `Dispose` both throw `ObjectDisposedException`, and so does every other member. A batch that -is never disposed keeps its buffers until a collection finalizes it, and they then go back to the -system rather than to the pool. So that such batches cannot pile up, the library asks for a -collection in the background once the batches its callers own hold 256 MiB, or an eighth of the -memory the GC may use when that is less, and again each time what they hold doubles. Analyzer VX1002 -flags a batch that nothing disposes ([diagnostics.md](diagnostics.md)). +After `Dispose`, both throw `ObjectDisposedException`, as does every other member. A batch that is +never disposed keeps its buffers until a collection finalizes it, and they then go back to the system +rather than to the pool. To keep such batches from piling up, the library requests a background +collection once the batches held by callers reach 256 MiB, or an eighth of the memory the GC may use +when that is less, and again each time that amount doubles. Analyzer VX1002 flags a batch that +nothing disposes ([diagnostics.md](diagnostics.md)). ## Keeping one batch of many @@ -87,55 +87,56 @@ await foreach (Columns cols in file.Scan()) kept past the loop: 65536 rows from row 458752, days 458 to 524, schema struct{Day: i32, Celsius: f64?, City: utf8} ``` -`ToOwned()` on borrowed columns is the same copy as `ToBatchesAsync` makes, for the one batch you -want to keep. `BatchView.ToOwned()` does the same on the tool path. +`ToOwned()` on borrowed columns makes the same copy `ToBatchesAsync` makes, for the one batch you want +to keep. `BatchView.ToOwned()` does the same on the tool path. ## What the copy costs The batch is copied once, into buffers from the session's pool. The decoded buffers are not handed -over instead, because they live beside the chunks the scan keeps across batches: a batch that took -them would take the chunk with it, and the next batch of that chunk would decode it again. +over directly because they live alongside the chunks the scan keeps across batches: a batch that took +them would take the chunk with it, and the next batch of that chunk would have to decode it again. -Reading every value of the demonstration file, warmed, the four variants run in turn, the best of +Reading every value of the demonstration file, warmed up, the four variants run in turn, the best of twenty rounds: | | time | managed allocation | |---|---|---| -| borrowed columns, typed scan | 6.6 ms | 6 KiB | -| borrowed columns, tool scan | 4.8 ms | 4 KiB | -| `ToBatchesAsync` | 5.6 ms | 123 KiB | -| `ToOwned()` on every borrowed batch | 7.1 ms | 121 KiB | - -Two things differ between these rows, the copy and the form of the columns: - -* A typed scan's borrowed columns arrive **encoded** where the file is: here `Day` and `City` - run-end, `Celsius` dictionary. Reading `Values` or a text length decodes them in your loop - ([encoded-forms.md](encoded-forms.md)). `ToBatchesAsync` and the tool scan decode them in the - scan, and deliver them canonical. -* So the tool scan against `ToBatchesAsync` is the copy alone: **0.7 ms, 16 % of the canonical - scan** in this run; timings this short move from run to run. - -`ToOwned()` copies what it is given as it is, the encoded form included, which is why it adds little -to the typed scan's own cost. An owned batch allocates a few KiB of managed memory beyond the -scan's own, the `RecordBatch` and its bookkeeping; its buffers come from the pool. Its `Schema` is +| borrowed columns, typed scan | 6.0 ms | 6 KiB | +| borrowed columns, tool scan | 4.5 ms | 4 KiB | +| `ToBatchesAsync` | 5.2 ms | 122 KiB | +| `ToOwned()` on every borrowed batch | 6.0 ms | 121 KiB | + +Two things differ between these rows: the copy, and the form of the columns. A typed scan's borrowed +columns arrive encoded the way the file stores them, here `Day` and `City` as runs and `Celsius` as a +dictionary, and reading `Values` or a text length decodes them in your loop +([encoded-forms.md](encoded-forms.md)). `ToBatchesAsync` and the tool scan decode them inside the +scan and deliver them in their plain form. So the difference between the tool scan and +`ToBatchesAsync` is the copy alone, 0.8 ms or 17 % of the plain scan in this run. Timings this short +move from run to run. + +`ToOwned()` copies what it is given as it is, encoded form included, which is why it adds little to +the typed scan's own cost. An owned batch allocates a few KiB of managed memory on top of the scan's +own, for the `RecordBatch` and its bookkeeping, and its buffers come from the pool. Its `Schema` is the scan's, one instance for every batch, so `As()` binds the record once for the whole scan rather than once per batch. A borrowed batch allocates nothing. ## Watch out -* **An owned batch keeps its selection.** With `ScanOptions.Compact = false`, or on a take by - position, the scan delivers whole blocks with a `Selection`, and the copy carries it: on this file +* An owned batch keeps its selection. With `ScanOptions.Compact = false`, or on a take by position, + the scan delivers whole blocks with a `Selection`, and the copy carries it. On this file, `Celsius > 45` under `Compact = false` passes 122 500 rows and the owned batches select the same - 122 500, and `Rows(4, 900_000)` gives owned batches that select 2. Read `Selection` on the batch's - `View` or `As()` as you would on a borrowed one; the whole block is still copied. -* **A batch left in a channel is still owned.** A consumer that stops early drains the channel and - disposes what it finds; a producer that fails to hand a batch over disposes it, as above. -* Holding batches holds pool memory: a bounded channel, as above, is what keeps a slow consumer from - holding the whole file. -* A batch is not thread-safe. It may move to another thread, and be disposed there, but one + 122 500, and `Rows(4, 900_000)` gives owned batches that select 2 rows. Read `Selection` on the + batch's `View` or on `As()` as you would on a borrowed batch. The whole block is still copied. +* A batch left in a channel is still owned. A consumer that stops early drains the channel and + disposes what it finds, and a producer that fails to hand a batch over disposes it, as above. +* Holding batches holds pool memory. A bounded channel, as above, keeps a slow consumer from holding + the whole file. +* A batch is not thread-safe. It may move to another thread and be disposed there, but only one consumer uses it at a time. -`RecordBatch` and the sinks are in §5.6 and §5.8 of [14-public-api.md](../design/14-public-api.md). +`RecordBatch` and the sinks are described in +[the public API design](../design/14-public-api.md#56-the-sinks), along with +[the tool path](../design/14-public-api.md#58-the-tool-path). ## Run it diff --git a/docs/guide/project-columns.md b/docs/guide/project-columns.md index 1df51ee7..9883bff3 100644 --- a/docs/guide/project-columns.md +++ b/docs/guide/project-columns.md @@ -1,6 +1,6 @@ # Project columns -Read two columns of seven and pay for two. There is no projection call: the record you scan with +Read two columns out of seven and pay for two. There is no projection call: the record you scan with is the projection. ```csharp @@ -28,27 +28,27 @@ await foreach (var (startedAt, durationMs) in file.Scan()) longest visit 89999 ms, the last one started at 2026-09-04 11:19:57Z ``` -The visits file has six top-level columns, one of them a nested record of two fields: `Id`, -`StartedAt`, `DurationMs`, `Referrer`, `Pages`, `Origin.Country` and `Origin.City`. `VisitTiming` -names two of them, and the scan reads those two. +The visits file has six top-level columns, one of which is a nested record of two fields, so seven +leaves in all: `Id`, `StartedAt`, `DurationMs`, `Referrer`, `Pages`, `Origin.Country` and +`Origin.City`. `VisitTiming` names two of them, and the scan reads those two. ## What it saves -The sample scans the file with four records and measures each twice: `ExplainAsync` before, from -the layout alone, and `Statistics` after the scan ran. +The sample scans the file with four records and measures each one twice: with `ExplainAsync` before +the scan, from the layout alone, and with `Statistics` after it. | record | columns | segments | bytes to read | requests | bytes read | |---|---|---|---|---|---| -| `Visit` | 7 | 30 | 2 162 496 | 30 | 2 162 496 | +| `Visit` | 7 | 30 | 2 132 656 | 30 | 2 132 656 | | `VisitTiming` | 2 | 10 | 53 096 | 10 | 53 096 | | `VisitOrigin` | 1 | 5 | 65 580 | 5 | 65 580 | | `VisitId` | 1 | 5 | 1 600 620 | 5 | 1 600 620 | Each column lives in its own segments, so a column the record does not name is a branch of the -layout the scan never walks: not read and discarded, not read at all. The saving follows the -columns you drop, weighted by their size, not by their number: the timestamp and the duration of -`VisitTiming` compress to little and cost under 3 % of the whole record, while the sixteen bytes -of each `Id`, stored as they are, make three quarters of it. +layout the scan never walks. It is not read and then discarded, it is not read at all. The saving +follows the size of the columns you drop rather than their number. The timestamp and the duration of +`VisitTiming` compress well and cost under 3 % of the whole record, while the sixteen raw bytes of +each `Id` account for three quarters of it. ## Nested records project too @@ -60,36 +60,36 @@ public partial record struct OriginCountry(string Country); public partial record struct VisitOrigin(OriginCountry Origin); ``` -A struct column is reached through a nested record, and the projection is over leaves: `Origin` in -`VisitOrigin` names only `Country`, so the scan reads that field's 5 segments and not those of -its sibling `City`. +A struct column is reached through a nested record, and the projection works on leaves. `Origin` in +`VisitOrigin` names only `Country`, so the scan reads that field's 5 segments and not those of its +sibling `City`. ## How a record binds to a file At the first sink of a scan, each member is matched to a column by exact name, then by a unique -case-insensitive match; `[VortexColumn("name")]` on the member says which column when the names -differ ([records.md](records.md)). A file with more columns than the record is fine. A member with -no column, an ambiguous match, or a type that does not fit the column's type throws -`VortexSchemaException`, naming the member and the columns: +case-insensitive match. `[VortexColumn("name")]` on the member says which column to use when the +names differ ([records.md](records.md)). A file may have more columns than the record. A member with +no column, an ambiguous match, or a type that does not fit the column throws +`VortexSchemaException`, which names the member and the columns: ``` VisitLength: Member 'Duration' of VisitLength has no column; the columns are 'Id', 'StartedAt', 'DurationMs', 'Referrer', 'Pages', 'Origin'. ``` -The batch delivers the record's columns in the record's order, whatever the file's order: the -deconstruction above gives `StartedAt` first because `VisitTiming` declares it first. +The batch delivers the record's columns in the record's order, whatever the file's order. That is why +the deconstruction above gives `StartedAt` first: `VisitTiming` declares it first. ## Watch out -* **A filter is written against the same record.** `Where` sees the record's members and no - others, so a column you filter on is a column you name, read for the filter and delivered too - ([filter-rows.md](filter-rows.md)). A record per query is the idiom, not a smell. -* **Nullability is part of the fit.** A non-nullable member over a nullable column is refused at - binding; declare `int?` or `string?` where the file says the column can hold a null. -* `ToRecordsAsync()` materialises rows of the same record, one field copy per row - ([read-rows.md](read-rows.md)); the projection is the same, the cost is not. -* For a file whose columns you only know at run time, the untyped scan takes column names instead - of a record ([untyped-files.md](untyped-files.md)). +* A filter is written against the same record. `Where` sees the record's members and nothing else, + so a column you filter on must be a column you name, and it is read and delivered too + ([filter-rows.md](filter-rows.md)). Declaring one record per query is the intended way to work. +* Nullability is part of the fit. A non-nullable member over a nullable column is refused at + binding, so declare `int?` or `string?` wherever the file says the column can hold a null. +* `ToRecordsAsync()` materialises rows of the same record, with a field copy per row + ([read-rows.md](read-rows.md)). The projection is the same, the cost is not. +* When you only know the columns at run time, the untyped scan takes column names instead of a + record ([untyped-files.md](untyped-files.md)). The figures come from one run of the sample on the demonstration file of a hundred thousand visits. diff --git a/docs/guide/queries.md b/docs/guide/queries.md index 24f8ae65..add2c2c6 100644 --- a/docs/guide/queries.md +++ b/docs/guide/queries.md @@ -1,7 +1,7 @@ # Queries -A group by, its groups filtered, ordered and cut, written as a query or as a chain of methods: the -same plan either way, and groups that go out as soon as they are final when the key allows it. +A group by whose groups are filtered, ordered and cut, written as a query or as a chain of methods. +Both build the same plan, and when the key allows it the groups go out as soon as they are final. ```csharp Scan hottest = @@ -28,8 +28,8 @@ the query and the chain agree: True ## A query, or the chain it becomes -The compiler turns the query into the methods it names, and both build the same plan; the sample -runs both and compares them: +The compiler turns the query into the methods it names, and both forms build the same plan. The +sample runs both and compares them: ```csharp Scan chained = readings.Scan() @@ -42,13 +42,13 @@ Scan chained = readings.Scan() .As(); ``` -The `where` before the `group` filters rows, and is pushed down as a scan's filter is -([filter-rows.md](filter-rows.md)); the one after filters groups, on their results. A key of several -columns names its components as a tuple does, `g.Key.Day` and `g.Key.City`. The lambdas run once, to -describe the work: nothing is evaluated per row in your code. +The `where` before the `group` filters rows, and is pushed down like any scan filter +([filter-rows.md](filter-rows.md)). The one after it filters groups, on their results. A key of +several columns names its components the way a tuple does, `g.Key.Day` and `g.Key.City`. The lambdas +run once to describe the work, and nothing is evaluated per row in your code. -**One value comes as itself, several into a record.** A `select` of one value is an `Aggregation`, -a value per group: +One value comes back as itself, several go into a record. A `select` of one value is an +`Aggregation`, one value per group: ```csharp Aggregation perCity = from r in readings.Scan() group r by r.City into g select g.Count(); @@ -59,25 +59,25 @@ List counts = await perCity.ToListAsync(); rows per city: 125006, 125000, 124999, 124999, 124999, 124999, 124999, 124999 ``` -A `select` of several values has no .NET type until a record gives it one: `As()` makes the -result a scan of that record, read as batches or as records like a file's, and as many values as the -record has members. A result is a scan: `writer.WriteAsync(result)` writes a rollup as it comes, with -no record built per group. +A `select` of several values has no .NET type until a record gives it one. `As()` turns the +result into a scan of that record, which you read as batches or as records like a file, with as many +values as the record has members. Since a result is a scan, `writer.WriteAsync(result)` writes a +rollup as it comes, without building a record per group. ## What each operator holds -The operators after the group by apply in the order written, as in LINQ, and each holds what its +The operators after the group by apply in the order written, as in LINQ, and each holds only what its answer needs: | after the group by | holds | |---|---| | `Where` on the groups | nothing more: a group is kept or dropped as it is delivered | -| `OrderBy(g => g.Key).Take(k)`, the key's order | `k` groups a lane, the best met so far, and a row whose integer key lies past them is not grouped at all | -| `Take(n)` without an order | `n` groups on one lane, which ones not promised; every group on several | -| `OrderBy(g => g.Count())` or any aggregate | every group to the end, since a group's value is known only then | +| `OrderBy(g => g.Key).Take(k)` | `k` groups per lane, the best met so far. A row whose integer key lies past them is not grouped at all | +| `Take(n)` without an order | `n` groups on one lane, with no promise about which ones, and every group on several lanes | +| `OrderBy(g => g.Count())` or any other aggregate | every group until the end, since a group's value is only known then | | on a key that streams | the groups still open | -The five shortest visit durations of the visits file, against every one of them: +The five shortest visit durations of the visits file, against all of them: ```csharp List shortest = await ToListAsync(visits.Scan() @@ -91,8 +91,8 @@ the five shortest durations, 0 to 4 ms, in 1.05 ms; every one of the 90000 durat ## Groups that go out as they close A group is final once no row still to come can hold its key. On a column the statistics say is -sorted, that is as soon as the key changes: the groups go out in key order while the file is read, -the first after the first batch, and the memory is the groups still open. +sorted, that happens as soon as the key changes. The groups go out in key order while the file is +being read, the first one after the first batch, and memory holds only the groups still open. ```csharp await foreach (long rows in readings.Scan().GroupBy(r => r.Day).Select(g => g.Count())) @@ -102,8 +102,8 @@ await foreach (long rows in readings.Scan().GroupBy(r => r.Day).Select( GroupBy(Day): the first of 1000 groups after 0.07 ms and 1 batch(es), all of them after 0.30 ms ``` -Under a descending order of such a key and a `Take`, the chunks are read from the last one back, -and the read stops once the last group asked for is out: +Under a descending order of such a key followed by a `Take`, the chunks are read from the last one +backwards, and reading stops once the last group asked for is out: ```csharp List seven = await ToListAsync(lastDays.GroupBy(r => r.Day).OrderByDescending(g => g.Key).Take(7) @@ -114,40 +114,42 @@ List seven = await ToListAsync(lastDays.GroupBy(r => r.Day).OrderByDesc the last seven days, 999 down to 993: 16960 rows read of a million ``` -Two more keys stream. A column of integers stored nearly in order, the rows of a log appended a -little late, streams on its zone maps: a group goes out once every block still to read holds keys -above it. It does on one lane, under a `Take`, and under a tight memory budget; on several lanes -without either, the pass blocks, which runs faster there. A dataset's clustering key streams over its -objects, read in the key's order. The plan -says which component streams, and [16-queries.md](../design/16-queries.md) §2.3 says when each one -does. +Two more kinds of keys stream. A column of integers stored nearly in order, like the rows of a log +appended slightly out of order, streams on its zone maps: a group goes out once every block still to +read holds larger keys. It streams on one lane, under a `Take`, and under a tight memory budget. On +several lanes without either, the pass blocks instead, because that runs faster there. A dataset's +clustering key also streams, since the dataset reads its objects in key order. The plan says which +component streams, and [the query design](../design/16-queries.md#23-a-group-by-that-streams) +explains when each one does. -## From `AggAsync` on groups +## Migrating from `AggAsync` on groups | before | now | |---|---| | `GroupBy(…).AggAsync(g => …)` | `GroupBy(…).Select(g => …)` | -| `await foreach (var (city, n) in ….AggAsync(g => (g.Key, g.Count())))`, a tuple a group | `….Select(g => (g.Key, g.Count())).As()`, read as batches or with `ToRecordsAsync()` | +| `await foreach (var (city, n) in ….AggAsync(g => (g.Key, g.Count())))`, a tuple per group | `….Select(g => (g.Key, g.Count())).As()`, read as batches or with `ToRecordsAsync()` | | at most eight values in a selection | as many as the record has members | | `GroupBy(r => (r.City, r.Day))`, then `g.Key.Item1` | `g.Key.City` | | the groups at the end of the pass | as soon as they are final, on a key that streams | ## Watch out -* **Without an order, which groups a `Take` keeps is not promised.** On several lanes the groups - come in the order the lanes met them. `OrderBy(g => g.Key)` makes them the first. -* **A descending order streams only under a `Take`, and with nothing before it that reads an order**: - no window before it, no `First`, `Last`, `MinBy` or `MaxBy`, no aggregator of your own. Backwards, - the rows of a group come last first. Read whole, a descending order is the blocking pass's, on - every lane. -* **A key that streams costs a little throughput for its memory and its first answer.** On a nearly - sorted key of a million values over four million rows, the stream answers first after 2 ms rather - than 30 and holds a fifteenth of the memory, for a pass 5 to 23 % longer than holding every group. -* **The first passes of a process run on code the JIT has not optimized yet**: see the last point of - [aggregates.md](aggregates.md#watch-out). - -The figures come from the sample on the demonstration files, on a machine of 14 cores: each timing -is the best of three passes in a process, and the median of seven processes (2026-10-09). +* Without an order, which groups a `Take` keeps is not promised. On several lanes the groups come in + the order the lanes met them. `OrderBy(g => g.Key)` makes them the first ones. +* A descending order streams only under a `Take`, and only when nothing before the order depends on + the order of rows: no `Skip` or `Take` before it, no `First`, `Last`, `MinBy` or `MaxBy`, and no + aggregator of your own. Read backwards, the rows of a group arrive last first. Without a `Take`, a + descending order runs as a blocking pass on every lane. +* A key that streams trades a little throughput for memory and a fast first answer. On a nearly + sorted key of a million values over four million rows, the stream answers first after 1 to 2 ms + rather than 34. On one lane it holds a fifteenth of the memory of the blocking pass and takes 0.84 + of its time. On fourteen lanes it holds a third of the memory but runs 2.1 times as long, which is + why the pass blocks there unless a `Take` or a tight budget makes streaming worth it. +* The first passes of a process run on code the JIT has not optimized yet. The last point of + [aggregates.md](aggregates.md#watch-out) explains what that costs. + +The figures come from the sample on the demonstration files, on a 14-core machine. Each timing is +the best of three passes in a process and the median of seven processes. ## Run it diff --git a/docs/guide/read-rows-by-index.md b/docs/guide/read-rows-by-index.md index 9d9cb959..5189a7b0 100644 --- a/docs/guide/read-rows-by-index.md +++ b/docs/guide/read-rows-by-index.md @@ -1,6 +1,6 @@ # Read rows by index -Take row 4 and row 900 000 without decoding what lies between. +Take row 4 and row 900 000 without decoding what lies between them. ```csharp Scan take = file.Scan().Rows(4, 900_000); @@ -21,14 +21,15 @@ a block of 7073 rows from row 892928, 1 selected row 900000: day 900, Toulouse ``` -A take delivers the **blocks** that hold the rows, not the rows alone, and marks the rows asked for -in the batch's `Selection`. Enumerating the selection gives their positions in the batch, and -`StartRow + i` their positions in the file. +A take delivers batches around the rows you asked for, not the rows alone, and marks the requested +rows in the batch's `Selection`. Enumerating the selection gives their positions in the batch, and +`StartRow + i` gives their positions in the file. -`StartRow` is the file row of the batch's row 0. The batch is the part of the block the rows asked -for span: it starts at the block's first row, or at the first row asked for when that lies inside -the block, and it ends after the last one. Above, the first batch starts at row 4 itself; the second -starts at row 892 928, where its block begins, and ends at row 900 000. +The take covers the range from the first requested row to the last one. The scan cuts that range +into batches of about one block (8 192 rows), aligned on the file's chunks, and skips every batch +that holds no requested row. Above, the first batch starts at row 4, the first row asked for, and +spans 8 192 rows. The second batch is the end of the range, from row 892 928 up to and including +row 900 000. `StartRow` is always the file row of the batch's row 0. ## A contiguous range @@ -36,32 +37,32 @@ starts at row 892 928, where its block begins, and ends at row 900 000. file.Scan().Rows(RowRange.FromLength(1_000, 10)) ``` -`RowRange` is a half-open `[start, end)`; `new RowRange(1_000, 1_010)` says the same thing. A range -is delivered as it is, with no selection: `10 rows from row 1000, selection all: True`. A range that -runs past the end of the file is cut there, so `RowRange.FromLength(999_990, 100)` gives ten rows. +`RowRange` is a half-open interval `[start, end)`, so `new RowRange(1_000, 1_010)` says the same +thing. A range is delivered as it is, with no selection: `10 rows from row 1000, selection all: True`. +A range that runs past the end of the file is cut there, so `RowRange.FromLength(999_990, 100)` gives +ten rows. ## What it costs -Taking two rows of a million, against reading all of them, on the demonstration file: +Taking two rows out of a million, against reading all of them, on the demonstration file: | | requests | bytes | blocks decoded | |---|---|---|---| | `Rows(4, 900_000)` | 6 | 180 352 | 2 | -| the whole file | 51 | 1 494 044 | 123 | +| the whole file | 51 | 1 494 068 | 123 | -The take reads the segments of the two chunks the rows are in, for the record's three columns, and -decodes the two blocks that hold them; it never touches the 121 between them. That is the shape of -the cost: **a take is priced in blocks, not rows,** and what it reads is the chunk of each block, a -segment per column. Two rows in one block cost one block, two rows far apart cost two, and the -blocks here are 8 192 rows. `ExplainAsync()` gives the same figures before the read, 2 rows, 2 of -2 blocks live, 6 segments and 180 352 bytes to read, and a record with fewer members reads fewer -segments. +The take reads the segments of the two chunks that hold the rows, for the record's three columns, +and decodes the two blocks that hold them. It never touches the 121 blocks in between. That is the +shape of the cost: a take is priced in blocks, not in rows, and for each block it reads the chunk +that contains it, one segment per column. Two rows in one block cost one block, two rows far apart +cost two, and blocks hold 8 192 rows here. `ExplainAsync()` gives the same figures before the read +(2 rows, 2 of 2 blocks live, 6 segments, 180 352 bytes to read), and a record with fewer members +reads fewer segments. ## With a filter -A filter on a take is evaluated on the rows asked for, and then the batch is **compacted** to the -survivors, as any filtered batch is. `Selection` is then all, and row `i` is no longer -`StartRow + i`: +A filter on a take is evaluated on the requested rows, and the batch is then compacted to the +survivors, like any filtered batch. `Selection` is then all, and row `i` is no longer `StartRow + i`: ``` a take with a filter: 3 rows, StartRow 4, selection all: True @@ -70,10 +71,10 @@ a take with a filter: 3 rows, StartRow 4, selection all: True batch row 2: 12 degrees, so file row 20; StartRow + 2 = 6 ``` -That is `Rows(4, 10, 20, 900_000).Where(r => r.Celsius > 10.0)`: row 900 000 holds exactly 10.0 and +That is `Rows(4, 10, 20, 900_000).Where(r => r.Celsius > 10.0)`. Row 900 000 holds exactly 10.0 and is dropped, and the three others come back packed. When the file position of each surviving row -matters, set `ScanOptions.Compact = false`: positions then hold and `Selection` says which rows -passed ([selection.md](selection.md)). +matters, set `ScanOptions.Compact = false`. Positions are then preserved, and `Selection` says which +rows passed ([selection.md](selection.md)). ``` the same under Compact = false: 8192 rows from row 4, selected rows 4 10 20 @@ -81,22 +82,23 @@ the same under Compact = false: 8192 rows from row 4, selected rows 4 10 20 ## Watch out -* **The list is sorted and deduplicated.** `Rows(900_000, 4, 4)` delivers two rows, in file order: - the order of your list is not kept and a repeat is not repeated. -* **A range and a list do not compose.** Both on one scan throws `InvalidOperationException: A scan - selects rows by range or by index, not both.`, and analyzer VX1004 flags it at compile time - ([diagnostics.md](diagnostics.md)). -* A row past the end throws `ArgumentOutOfRangeException: The file has 1000000 rows.` when the - scan starts, not when `Rows` is called. -* The plan of a take or a range counts only the blocks its rows touch: 2 of 2 for the take above, +* The list is sorted and deduplicated. `Rows(900_000, 4, 4)` delivers two rows in file order: the + order of your list is not kept, and a repeated index is not repeated. +* A range and a list do not combine. Using both on one scan throws + `InvalidOperationException: A scan selects rows by range or by index, not both.`, and analyzer + VX1004 flags it at compile time ([diagnostics.md](diagnostics.md)). +* A row past the end throws `ArgumentOutOfRangeException: The file has 1000000 rows.` when the scan + starts, not when `Rows` is called. +* The plan of a take or a range only counts the blocks its rows touch: 2 of 2 for the take above, and 1 of 1 for the range of ten rows, whose `Segments` and `BytesToRead` are 3 and 100 428. -* An owned batch from `ToBatchesAsync` does not carry the selection: a take through it hands over - the whole span of each block, 16 195 rows for these two ([owned-batches.md](owned-batches.md)). - Use the borrowed columns, or `ToRecordsAsync`, which yields the two rows only. - -Blocks and the selection are the read contract of -[11-write-strategy.md](../design/11-write-strategy.md) §6; `StartRow` and `Selection` are in §5.3 -of [14-public-api.md](../design/14-public-api.md). +* Owned batches keep the selection too. Through `ToBatchesAsync`, each batch still covers the same + span of rows, and its `Selection` marks the two requested rows + ([owned-batches.md](owned-batches.md)). `ToRecordsAsync` yields just the two rows. + +Blocks and selections are part of the read contract described in +[the write strategy](../design/11-write-strategy.md#6-reading-what-the-writer-produced), and +`StartRow` and `Selection` in +[the public API design](../design/14-public-api.md#53-columnstrecord-and-columnt). ## Run it diff --git a/docs/guide/read-rows.md b/docs/guide/read-rows.md index 36b8e8ea..c06f0651 100644 --- a/docs/guide/read-rows.md +++ b/docs/guide/read-rows.md @@ -1,6 +1,6 @@ # Read rows -Get records, one per row, when rows are what the code wants, and know what that costs. +Get one record per row when rows are what your code wants, and know what that costs. ```csharp int shown = 0; @@ -30,54 +30,55 @@ Toulouse: 10.2 hottest, sorted on the client: 49.9 in Lyon on day 0 ``` -`ToRecordsAsync` is a sink, like `await foreach` over the scan or `CountAsync`: it runs the scan and -turns each surviving row into a `Reading`, through the `ReadRows` the generator wrote for the record -(see [records.md](records.md)). The record is the projection, the `Where` is pushed down exactly as -in [filter-rows.md](filter-rows.md), and only what survives becomes a row. +`ToRecordsAsync` is a sink, like `await foreach` over the scan or `CountAsync`. It runs the scan and +turns each surviving row into a `Reading` through the `ReadRows` method the generator wrote for the +record ([records.md](records.md)). The record is the projection, the `Where` is pushed down exactly as +in [filter-rows.md](filter-rows.md), and only the rows that survive become records. ## LINQ starts after the sink -`ToRecordsAsync` returns an `IAsyncEnumerable`, so `System.Linq.AsyncEnumerable`, in the -shared framework, applies to it: `OrderByDescending`, `Take`, `ToListAsync` above. They run on the -client, over rows already materialised. Everything to the left of `ToRecordsAsync` is pushed into -the scan; nothing to the right of it is, and the code shows where the line falls. +`ToRecordsAsync` returns an `IAsyncEnumerable`, so `System.Linq.AsyncEnumerable`, which is +part of the shared framework, applies to it: `OrderByDescending`, `Take` and `ToListAsync` above. +Those operators run on the client, over rows that are already materialised. Everything to the left +of `ToRecordsAsync` is pushed into the scan, nothing to the right of it is, and the code shows where +the line falls. ## What it costs -Measured on the demonstration file, a million rows, warmed, the best of five passes: +Measured on the demonstration file, a million rows, warmed up, the best of five passes: | | time | allocated | |---|---|---| -| every row as a `Reading` | 30 ms | 35 MiB | +| every row as a `Reading` | 45 ms | 35 MiB | | the same values as columns, `await foreach` | 5 ms | 6 KiB | -| the ten hottest, sorted on the client | 148 ms | 148 MiB | -| `MaxAsync(r => r.Celsius)` | 0.001 ms | 2 192 bytes, no request, no block decoded | -| `MaxAsync`, then `Where(r => r.Celsius >= max).ToRecordsAsync()` | 14 ms | 362 KiB, 2 500 rows | +| the ten hottest, sorted on the client | 149 ms | 148 MiB | +| `MaxAsync(r => r.Celsius)` | 0.002 ms | 1 888 bytes, no request, no block decoded | +| `MaxAsync`, then `Where(r => r.Celsius >= max).ToRecordsAsync()` | 2 ms | 361 KiB, 2 500 rows | -A row costs a field copy, and an allocation per row for every `string`, list or nested class member: -here the 35 MiB are the million `City` strings. The columns cost neither, because a text column is +A row costs a field copy, plus an allocation per row for every `string`, list or nested class +member. Here the 35 MiB are the million `City` strings. Columns cost neither, because a text column is read as UTF-8 spans ([text-columns.md](text-columns.md)). -The client-side top ten reads the whole file and materialises a million rows to keep ten of them. -The pushed forms ask the file first: `MaxAsync` is answered from the file's statistics without a -read, and a filter on the answer materialises the 2 500 rows that hold it and nothing else. A key -that is sorted has a third form, `OrderBy` on it, which delivers the batches in its order so that -the first rows are the answer ([keys-in-order.md](keys-in-order.md)). +The client-side top ten reads the whole file and materialises a million rows to keep ten. The pushed +version asks the file first. `MaxAsync` is answered from the file's statistics without a read, and a +filter on that answer materialises only the 2 500 rows that hold it. When the key is sorted there is +a third way, `OrderBy` on it, which delivers the batches in key order so that the first rows are the +answer ([keys-in-order.md](keys-in-order.md)). ## Watch out -* **A row is a copy.** Changing a `Reading` changes nothing in the file, and holding a million of - them holds a million copies. -* The scan behind `ToRecordsAsync` is single-use, like every scan: build another one to enumerate +* A row is a copy. Changing a `Reading` changes nothing in the file, and holding a million of them + holds a million copies. +* The scan behind `ToRecordsAsync` is single-use, like every scan. Build another one to enumerate again. -* Cancellation reaches the sink at a batch boundary: the rows of the batch in hand are still +* Cancellation reaches the sink at a batch boundary, so the rows of the batch in hand are still yielded ([cancel-work.md](cancel-work.md)). -* With `ScanOptions.Compact = false`, or on a take by position, the scan delivers whole blocks and - `ToRecordsAsync` yields only their selected rows, in order: `Rows(4, 900_000)` yields 2 records, +* With `ScanOptions.Compact = false`, or on a take by position, the scan delivers whole blocks, and + `ToRecordsAsync` yields only their selected rows, in order. `Rows(4, 900_000)` yields 2 records, and `Celsius > 49` under `Compact = false` yields 22 500, the coldest of them 49.1. -The sinks and what each one pays for are listed in §5.6 of -[14-public-api.md](../design/14-public-api.md). +The sinks and what each one costs are listed in +[the public API design](../design/14-public-api.md#56-the-sinks). ## Run it diff --git a/docs/guide/records.md b/docs/guide/records.md index 5178848d..07bc5182 100644 --- a/docs/guide/records.md +++ b/docs/guide/records.md @@ -42,17 +42,17 @@ Order struct{order_id: i64, Amount: decimal(12,2), PlacedAt: timestamp(ms), D first row: 1000, 5.00, 2026-09-01T08:00:00.0000000, 2026-09-04T10:00:00.0000000+02:00, Placed, leave at the door, [0], FR/null ``` -**A record is a schema, not a row.** The type says which columns exist and how they are typed; -reading it yields columns, and rows only when you ask for them. The `[VortexRecord]` source -generator, which ships in the `Vorticity.Generators` package, turns the declaration into -everything the library needs, and it does so at compile time: nothing is discovered by -reflection when the program runs. +A record is a schema, not a row. The type says which columns exist and how they are typed, and +reading it gives you columns, with rows only when you ask for them. The `[VortexRecord]` source +generator, shipped in the `Vorticity.Generators` package, turns the declaration into everything the +library needs at compile time. Nothing is discovered by reflection while the program runs. ## What can be a record -A `partial` `record struct`, `record class`, `struct` or `class`, nested only in `partial` types. -Its columns are its public instance properties and fields, and a positional record's parameters: -the parameters first, then the other members, base types first, each level in declaration order. +A record can be a `partial` `record struct`, `record class`, `struct` or `class`, nested only in +`partial` types. Its columns are its public instance properties and fields, and a positional record's +parameters. The parameters come first, then the other members, base types first, each level in +declaration order. ```csharp [VortexRecord] @@ -85,13 +85,12 @@ Shipment struct{Id: i64, Slot: time(µs), Carrier: utf8} Reading builds each row through a constructor whose parameters are members, then sets the other members, so every column member needs a way in: a constructor parameter, a setter or an `init` accessor. A member that has none, such as `AmountWithTax` above, is marked `[VortexIgnore]`. The -compiler says what is wrong when a type does not qualify: [diagnostics.md](diagnostics.md) lists -the eight diagnostics and their fixes. +compiler tells you what is wrong when a type does not qualify, and [diagnostics.md](diagnostics.md) +lists the eleven diagnostics and how to fix each one. ## What the generator writes -For `Order`, the generator adds to the type itself the three static members of -`IVortexRecord`: +For `Order`, the generator adds the three static members of `IVortexRecord` to the type itself: | member | what it is | |---|---| @@ -99,16 +98,16 @@ For `Order`, the generator adds to the type itself the three static members of | `ReadRows(Columns, Span)` | fills rows from a batch, one column at a time | | `WriteRows(ColumnsBuilder, ReadOnlySpan)` | appends rows to a builder, one column at a time | -and a nested `Order.ColumnNames` class with one constant per member, for the options that name a -column by string: +It also adds a nested `Order.ColumnNames` class with one constant per member, for the options that +name a column by string: ``` Order.ColumnNames.Id = "order_id" ``` -Beside the type, in an `OrderVortexExtensions` class, it adds C# extension members that name the -columns wherever the library hands you a record's columns (a record declared inside `Shop` gets a -`Shop_OrderVortexExtensions`): +Next to the type, in an `OrderVortexExtensions` class, it adds C# extension members that name the +columns wherever the library hands you a record's columns. A record declared inside `Shop` gets a +`Shop_OrderVortexExtensions` class. | on | each member is | a nested record is | |---|---|---| @@ -116,7 +115,7 @@ columns wherever the library hands you a record's columns (a record declared ins | `Columns`, in the body of `await foreach` | `Column`, plus a `Deconstruct` over all of them | `Columns
` | | `ColumnsBuilder`, from `writer.Builder()` | `ColumnBuilder` | `ColumnsBuilder
` | -That is what makes the rest of the guide read as it does: +That is what makes the rest of the guide read the way it does: ```csharp long shipped = await file.Scan() @@ -146,8 +145,8 @@ shipped and above 100: 2275 of 10000 delivered total 515415.00, 2000 notes ``` -An enum is a `Sym` in a filter, so an enum literal compares, and its underlying -integer on the columns and the builders, which have no enum accessors: `status.Values` above is a +An enum is a `Sym` in a filter, so an enum literal can be compared. On the columns and +the builders, which have no enum accessors, it is its underlying integer: `status.Values` above is a `ReadOnlySpan`. ## `[VortexColumn]` and `[VortexIgnore]` @@ -163,8 +162,8 @@ goes on a property, a field or a positional parameter. | `Unit` | a `DateTime`, `DateTimeOffset` or `TimeOnly` member | `TimeUnit.Microseconds` | | `TimeZone` | a `DateTimeOffset` member | `"UTC"` | -They shape the schema the record writes. When the record reads a file, the file's column type -decides, and the binding only checks that the member's .NET type fits it. `[VortexIgnore]` leaves +These attributes shape the schema the record writes. When the record reads a file, the file's column +type decides, and the binding only checks that the member's .NET type fits it. `[VortexIgnore]` leaves a member out of the columns entirely. ## The mapping @@ -173,52 +172,52 @@ a member out of the columns entirely. |---|---| | `bool` | bool | | `sbyte`, `short`, `int`, `long`, `byte`, `ushort`, `uint`, `ulong` | i8 to i64, u8 to u64 | -| `char` | u16: the UTF-16 code unit, a lone surrogate included | -| `nint`, `nuint` | i64, u64, on a 64-bit host; refused on another | +| `char` | u16, the UTF-16 code unit, a lone surrogate included | +| `nint`, `nuint` | i64, u64, on a 64-bit host, and refused on any other | | `Half`, `float`, `double` | f16, f32, f64 | | an `enum` | its underlying integer type | | `string` | utf8 | | `ReadOnlyMemory`, `Memory`, `byte[]` | binary | | `decimal` | decimal of at most 28 digits and a scale of 0 to 28 | | `VortexDecimal` | decimal of any precision up to 76 | -| `Int128`, `UInt128` | decimal of scale 0: 38 digits by default, 39 for every value of the type | +| `Int128`, `UInt128` | decimal of scale 0, 38 digits by default, 39 for every value of the type | | `BigInteger` | decimal of scale 0, 76 digits by default | | `TimeSpan` | i64, its ticks of 100 ns | | `DateOnly` | `vortex.date` | | `TimeOnly` | `vortex.time` | | `DateTime` | `vortex.timestamp`, naive or UTC | -| `DateTimeOffset` | `vortex.timestamp` with a zone | +| `DateTimeOffset` | `vortex.timestamp` with a time zone | | `Guid` | `vortex.uuid` | -| `ReadOnlyMemory`, `Memory`, `T[]`, `List`, `ImmutableArray`, `IReadOnlyList`, `IList`, `ICollection`, `IReadOnlyCollection`, `IEnumerable` | list of `T`, a record included | -| `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | map, of scalar keys and values | +| `ReadOnlyMemory`, `Memory`, `T[]`, `List`, `ImmutableArray`, `IReadOnlyList`, `IList`, `ICollection`, `IReadOnlyCollection`, `IEnumerable` | list of `T`, records included | +| `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | map of scalar keys and values | | another `[VortexRecord]` type | struct | | a type implementing `IVortexExtension` | the extension it registers | -| `T?` or `string?` of any of these, but an extension | the nullable column | +| `T?` or `string?` of any of these, except an extension | the nullable column | -A list read back is the container the member declares: an array for an array and for an -interface an array implements, a `List` filled in place, an `ImmutableArray` over the array -it was read into, with no copy. A dictionary is read back as a `Dictionary`. A list -of records is read and written through `ListOf`, a map through `MapOf`; a +A list read back is the container the member declares: an array for an array or for an interface an +array implements, a `List` filled in place, or an `ImmutableArray` wrapping the array it was +read into, without a copy. A dictionary is read back as a `Dictionary`. A list of +records is read and written through `ListOf`, and a map through `MapOf`. A filter compares neither. -An integer of 128 bits is a decimal of scale 0, because no dtype is 128 bits wide. Thirty-eight -digits hold every value a 128-bit decimal holds, which every reader of one reads and a column reads -without a copy; a value past them is refused when it is written, with the way out in the message: -`[VortexColumn(Precision = 39)]`, which holds every value of the type in 256 bits. Reading, an -`Int128` binds to a decimal of scale 0 and at most 39 digits, and a value past the type in a -39-digit column is refused as it is read, as a timestamp past `DateTime` is. +A 128-bit integer is a decimal of scale 0, because no dtype is 128 bits wide. Thirty-eight digits +cover every value a 128-bit decimal holds, every reader of such a decimal can read them, and a column +reads them without a copy. A value beyond them is refused when it is written, and the message gives +the way out: `[VortexColumn(Precision = 39)]`, which holds every value of the type in 256 bits. When +reading, an `Int128` binds to a decimal of scale 0 and at most 39 digits, and a value beyond the type +in a 39-digit column is refused as it is read, just as a timestamp beyond `DateTime` is. -A type with no column -- an array of more than one dimension, a dictionary of lists, a list of -lists of records, a `Uri` -- is refused by the generator with VX1005, which says what to declare -instead. The whole contract, and where a naive mapping would lose data, is in +A type with no column equivalent, such as an array of more than one dimension, a dictionary of +lists, a list of lists of records or a `Uri`, is refused by the generator with VX1005, which says +what to declare instead. The full contract, and where a naive mapping would lose data, is in [07-dotnet-mapping.md](../design/07-dotnet-mapping.md). ## Binding a record to a file -A record meets a file at the first sink of a scan, or when a writer is created. Each member is -matched to a column by exact name, then by a unique case-insensitive match, and its type must fit -the column's. A file with more columns than the record is fine: the record is the projection, and -the other columns are not read. A file without one of the record's columns is not: +A record meets a file at the first sink of a scan, or when a writer is created. Each member is matched +to a column by exact name, then by a unique case-insensitive match, and its type must fit the +column's. A file with more columns than the record is fine: the record is the projection, and the +other columns are not read. A file missing one of the record's columns is not: ```csharp [VortexRecord] @@ -233,9 +232,8 @@ bound case-insensitively: Order_Id 1000, amount 5.00 a member with no column: Member 'Id' of OrderById has no column; the columns are 'order_id', 'Amount', 'PlacedAt', 'DeliverBy', 'Status', 'Note', 'Items', 'ShipTo'. ``` -That is a `VortexSchemaException`, and so is a non-nullable member over a nullable column. A -member binds by the column's name, so the renamed `order_id` is not found by a member called -`Id`. +That is a `VortexSchemaException`, and so is a non-nullable member over a nullable column. A member +binds by the column's name, so the renamed `order_id` column is not found by a member called `Id`. ## A record written by hand @@ -290,15 +288,15 @@ long above = await pairFile.Scan().Where(p => p.Column("key") >= 990) hand-written record: 10 keys at or above 990, last (999, 249.75) ``` -It loses the named members on the probe, the columns and the builder, and nothing else: the index -passed to `Column` is the member's position in `Schema`, whatever the file's order. The -attributes live in the core package, so a project that writes its records by hand does not need -the generator at all. +A hand-written record loses the named members on the probe, the columns and the builder, and nothing +else. The index passed to `Column` is the member's position in `Schema`, whatever the file's order. +The attributes live in the core package, so a project that writes all its records by hand does not +need the generator at all. ## An extension type -A column type the format does not define is an extension: an id, and a storage type the values -are kept as. A .NET type becomes one by implementing `IVortexExtension`, whose members are all +A column type the format does not define is an extension: an id, plus a storage type in which the +values are kept. A .NET type becomes one by implementing `IVortexExtension`. All its members are static, so the conversion is compiled per type and costs no virtual call per value: ```csharp @@ -319,7 +317,7 @@ public readonly record struct Money(long Cents) : IVortexExtension public partial record struct Invoice(long Number, Money Total); ``` -A session knows it once it is registered, and only then: +A session only knows the extension once it is registered: ```csharp await using VortexSession session = VortexSession.Create(o => o.Extensions.Register()); @@ -332,38 +330,38 @@ without it, the schema reads struct{Number: i64, Total: ext(acme.money, i64)} and reading Invoice throws VortexUnsupportedException: Unsupported Vortex component: dtype 'acme.money'. This id is not implemented by Vorticity; check the edition that introduced it and the minimum library version required to read it. ``` -The writer and every reader need the same registration. A filter compares a `Money` literal -through its storage, `i.Total > new Money(100_000)`, which is why the storage of an extension you -want to filter on should be a primitive. Registration happens inside `VortexSession.Create`; the -session's registry is frozen when it returns, and an id in the `vortex.` namespace is refused. +The writer and every reader need the same registration. A filter compares a `Money` literal through +its storage, as in `i.Total > new Money(100_000)`, which is why the storage of an extension you want +to filter on should be a primitive. Registration happens inside `VortexSession.Create`, the session's +registry is frozen when it returns, and an id in the `vortex.` namespace is refused. ## Watch out -* **A naive timestamp reads back as `DateTimeKind.Unspecified`**, whatever `Kind` the value had - when it was written: `PlacedAt` above lost its `Utc`. Declare the member with - `TimeZone = "UTC"` to read it back as `Utc`. A `DateTime` member with any other zone is refused - at compile time: declare it `DateTimeOffset`. -* A nullable extension member, a list of nullable records and a list of lists of records are - refused with VX1005: no builder appends a null to the first, and no typed column reads the others. -* **A `DateTime`, a `DateTimeOffset` or a `TimeOnly` is stored in its column's unit**, microseconds - unless `Unit` says otherwise: the tenth of a microsecond a tick adds is floored away. Declare - `Unit = TimeUnit.Nanoseconds` to keep it, within the years 1677 to 2262 a nanosecond count holds. -* **Vortex Rust reads no instant after 9999-12-30T22:00Z**, which leaves room for any offset: a - `DateTime.MaxValue` written as a sentinel is a value this library reads back and the reference - refuses to hand out as a scalar. -* A member called `Schema`, `ReadRows`, `WriteRows` or `ColumnNames` does not stop the generator: - it implements the interface explicitly instead, and leaves `ColumnNames` out when that name is - taken. -* The record's accessibility carries to its extension class: an `internal` record gets `internal` - members. +* A naive timestamp reads back as `DateTimeKind.Unspecified`, whatever `Kind` the value had when it + was written, so `PlacedAt` above lost its `Utc`. Declare the member with `TimeZone = "UTC"` to read + it back as `Utc`. A `DateTime` member with any other time zone is refused at compile time, so + declare it `DateTimeOffset`. +* A nullable extension member, a list of nullable records and a list of lists of records are refused + with VX1005: no builder can append a null to the first, and no typed column can read the others. +* A `DateTime`, a `DateTimeOffset` or a `TimeOnly` is stored in its column's unit, microseconds unless + `Unit` says otherwise, so the tenth of a microsecond a tick adds is dropped. Declare + `Unit = TimeUnit.Nanoseconds` to keep it, within the years 1677 to 2262 that a nanosecond count can + hold. +* Vortex Rust reads no instant after 9999-12-30T22:00Z, which leaves room for any offset. A + `DateTime.MaxValue` written as a sentinel is a value this library reads back but the reference + refuses to return as a scalar. +* A member called `Schema`, `ReadRows`, `WriteRows` or `ColumnNames` does not stop the generator. It + implements the interface explicitly instead, and leaves `ColumnNames` out when that name is taken. +* The record's accessibility carries over to its extension class, so an `internal` record gets + `internal` members. ## What it costs -`ReadRows` and `WriteRows` move one column at a time, so the loop over a column is a loop over a -span. What rows cost on top of the columns is the rows themselves: `ToRecordsAsync` copies every -field, and allocates per row for a `string`, a list (read into a new array, or straight into a new -`List`), a dictionary, a `BigInteger` and a nested record that is a class. `WriteRows` transcodes each `string` once, and -fills primitive columns through `GetSpan`, in place. When the columns are what you want, read the +`ReadRows` and `WriteRows` move one column at a time, so the loop over a column is a loop over a span. +What rows cost on top of the columns is the rows themselves. `ToRecordsAsync` copies every field, and +allocates per row for a `string`, a list (read into a new array, or straight into a new `List`), a +dictionary, a `BigInteger` and a nested record that is a class. `WriteRows` transcodes each `string` +once, and fills primitive columns in place through `GetSpan`. When columns are what you want, read the columns: see [scan-a-table.md](scan-a-table.md) and [read-rows.md](read-rows.md). ## Run it diff --git a/docs/guide/row-keys.md b/docs/guide/row-keys.md index 06e7f390..fb9770ee 100644 --- a/docs/guide/row-keys.md +++ b/docs/guide/row-keys.md @@ -1,6 +1,6 @@ # Row keys -Encode tuples into bytes whose `memcmp` order is the tuple order. +Encode tuples into byte strings whose `memcmp` order is the tuple order. ```csharp [VortexRecord] @@ -26,19 +26,19 @@ await foreach (Columns batch in file.Scan().With(new S row 3: Paris, 10.3 -> 0250617269730000000000000000000000000000000000000000000000000000000501C02499999999999A ``` -The sample's own class is called `RowKeys`, which is why it spells the library's type -`RowEncoding.RowKeys`; with `using Vorticity.RowEncoding;` your code writes `RowKeys`. +The sample has its own class called `RowKeys`, which is why it spells the library's type +`RowEncoding.RowKeys`. With `using Vorticity.RowEncoding;` your code just writes `RowKeys`. -This is `Vorticity.RowEncoding`, a separate package, and it is **experimental**: its assembly is -marked `[Experimental("VX0002")]`, reported as an error wherever you use it until you opt in, as -[datasets.md](datasets.md) describes for VX0001. The diagnostic says why: the encoding follows an -upstream format that may change its bytes between Vortex releases, so keys are comparable only -with keys of the same `RowKeyEncoder.Format`. This build reproduces the bytes of -Vortex 0.86.1, which `RowEncoder.VortexVersion` states. +Row keys are useful whenever the comparison you need is one a B-tree, a sort, a merge or an object +store already does on bytes. Encoding each tuple once and then comparing bytes is both faster and +simpler than comparing field by field. -What it is for: when the comparison you need is the one a B-tree, a sort, a merge or an object -store already does on bytes, encoding each tuple once and comparing bytes is both faster and -simpler than comparing fields. +They live in `Vorticity.RowEncoding`, a separate package, and it is experimental. Its assembly is +marked `[Experimental("VX0002")]`, which is reported as an error wherever you use it until you opt +in, the same way [datasets.md](datasets.md) describes for VX0001. The reason is that the encoding +follows an upstream format whose bytes may change between Vortex releases, so keys are only +comparable with keys of the same `RowKeyEncoder.Format`. This build reproduces the bytes of +Vortex 0.86.1, as `RowEncoder.VortexVersion` states. ## The promise @@ -47,20 +47,20 @@ row 6 against row 7: keys >, bytes > sorted by key: 7, 0, 1, 2, 3, 4, 5, 6 ``` -`keys.Compare(6, 7)` is the tuple comparison and `keys.Row(6).SequenceCompareTo(keys.Row(7))` the -byte comparison, and they agree, always. That is the whole point, and what makes +`keys.Compare(6, 7)` is the tuple comparison and `keys.Row(6).SequenceCompareTo(keys.Row(7))` is the +byte comparison, and they always agree. That is the whole point, and it is what makes `keys.SortIndices(order)` work: it sorts an array of row indices by their encoded keys, with no -comparer, no boxing and no field access. Row 7 is Lyon, the others Paris, and a null sorts before -every value by default, which is why row 0 comes before row 1. +comparer, no boxing and no field access. Row 7 is Lyon and the others are Paris, and by default a +null sorts before every value, which is why row 0 comes before row 1. -`RowKeys` is disposable and owns one pooled buffer for every key of the batch: `Row(i)` is a span -into it, `Elements` the whole buffer, `Offsets` and `Sizes` its layout, `TotalBytes` its length. -The keys cover every row of the batch, selected or not. +`RowKeys` is disposable and owns one pooled buffer holding every key of the batch. `Row(i)` is a +span into it, `Elements` is the whole buffer, `Offsets` and `Sizes` describe its layout and +`TotalBytes` is its length. The keys cover every row of the batch, selected or not. ## Direction and nulls -One `RowSortField` per column, in the record's member order; one field for every column; or none, -for ascending with nulls first: +You pass one `RowSortField` per column in the record's member order, a single field applied to every +column, or none at all for ascending with nulls first: ```csharp Show("descending, nulls first", batch, [RowSortField.Ascending, RowSortField.Ascending.WithDescending(true)]); @@ -72,8 +72,8 @@ Show("ascending, nulls last", batch, [RowSortField.Ascending, RowSortField.Ascen ascending, nulls last: row 0 02506172697300000000000000000000000000000000000000000000000000000005020000000000000000, row 1 0250617269730000000000000000000000000000000000000000000000000000000501C024333333333333 ``` -Each option changes the bytes, not the comparison. Descending inverts the value bytes of that -column, so `memcmp` still yields the order you asked for; a null is one byte, placed before or +Each option changes the bytes, never the comparison. Descending inverts the value bytes of that +column, so `memcmp` still gives the order you asked for. A null is a single byte, placed before or after every value in either direction. ## A key for a tuple you hold @@ -88,11 +88,12 @@ the key of (Paris, 10.5): 025061726973000000000000000000000000000000000000000000 the key of (Paris) is a prefix of row 1: True ``` -`EncodeKey` writes one tuple, given as a record whose members are the key's columns in key order. -The record's schema gives the widths and the nullability the bytes depend on, so a probe encoded -from the same record type as the rows compares with them: to search sorted keys, to look one up, -or to build an object key from it. A record of the leading columns alone, `CityKey` here, encodes -a byte prefix of every key that starts with them, so a prefix query is a seek and a walk. +`EncodeKey` writes a single tuple, given as a record whose members are the key's columns in key +order. The record's schema supplies the widths and nullability the bytes depend on, so a probe +encoded from the same record type as the rows compares correctly with them, whether you search +sorted keys, look one up or build an object key from it. A record of only the leading columns, +`CityKey` here, encodes a byte prefix of every key that starts with them, so a prefix query becomes +a seek followed by a walk. ## From the tool path @@ -106,10 +107,10 @@ await foreach (BatchView batch in file.Scan("City", "Celsius").With(new ScanOpti the tool path, 8 rows: row 1 01C02433333333333302506172697300000000000000000000000000000000000000000000000000000005 ``` -`Encode(BatchView)` encodes the columns of a batch in its schema order, and a batch holds its -columns in the file's order, not in the order `Scan` named them: here `Celsius` comes first, and -the keys order by it. Project exactly the key's columns, in a file whose order is the key's, or use -a record. +`Encode(BatchView)` encodes a batch's columns in its schema order, and a batch holds its columns in +the file's order, not in the order `Scan` named them. Here `Celsius` comes first, so the keys order +by it. Either project exactly the key's columns from a file whose column order matches the key, or +use a record. ## Handing it to the writer @@ -126,27 +127,29 @@ in the file: vorticity.sorted.runs.v1 on (City, Day), 2 runs, 200000 entries, 45 the format it names: vortex-row 0.86.1 asc-nf; descending: vortex-row 0.86.1 asc-nf,desc-nf ``` -The core ships no key encoder and finds none by reflection, so an index over a composite key names -its encoder in `IndexPolicy.ForKey`. `RowKeyEncoder` is one: it implements `IKeyEncoder`, turns -each tuple of the key's columns into one row key, and names its layout in `Format`, which the -writer records in the file so that a reader can tell keys it may compare from keys it may not. The -composite keys of one file share one encoder. A dataset with a clustering key of several columns -writes each object's sorted run this way. [indexes.md](indexes.md) says what an index costs and -when to ask for one. +The core library ships no key encoder and does not look for one by reflection, so an index over a +composite key names its encoder in `IndexPolicy.ForKey`. `RowKeyEncoder` is such an encoder. It +implements `IKeyEncoder`, turns each tuple of the key's columns into one row key, and names its +layout in `Format`. The writer records that format in the file so a reader can tell which keys it +may compare. All composite keys in one file share one encoder. A dataset with a clustering key of +several columns writes each object's sorted run this way. [indexes.md](indexes.md) covers what an +index costs and when to ask for one. ## Watch out -* **The bytes are not stable across releases.** Keep them inside files this library writes, and - compare only keys of the same `Format`. -* **The field list follows the columns you encode**, in their order: members for a record, schema - order for a batch. No name is checked. -* `RowKeys` is disposable and its spans die with it. -* A date, time, timestamp or uuid column orders by its storage, which is its order. A variable-size list, a - map, a variant, a union and a decimal wider than 128 bits have no order the format defines, and - encoding one throws `VortexUnsupportedException`. A NaN is not canonicalised: two NaNs with - different payloads differ. -* Encoding is a pass over the batch that produces a byte string per row: 344 bytes for these eight - rows, 43 per row for a short text and a nullable double. +* The bytes are not stable across releases. Keep them inside files this library writes, and only + compare keys of the same `Format`. +* The fields follow the columns you encode, in their order: member order for a record, schema order + for a batch. No name is checked. +* `RowKeys` is disposable, and its spans become invalid once it is disposed. +* A date, time, timestamp or uuid column orders by its storage, which matches its natural order. + Inside a struct or a fixed-size list, though, such a column is refused with + `VortexUnsupportedException`, so put it at the top level of the key. +* A variable-size list, a map, a variant, a union and a decimal wider than 128 bits have no order the + format defines, and encoding one throws `VortexUnsupportedException`. NaNs are not canonicalised, + so two NaNs with different payloads compare as different. +* Encoding is one pass over the batch that produces a byte string per row: 344 bytes for these + eight rows, 43 per row for a short text and a nullable double. ## Run it diff --git a/docs/guide/scan-a-table.md b/docs/guide/scan-a-table.md index be8702ba..caf67972 100644 --- a/docs/guide/scan-a-table.md +++ b/docs/guide/scan-a-table.md @@ -1,6 +1,7 @@ # Scan a table -Read every row, in batches, without copying, and know when not to write the loop at all. +This page reads every row in batches, without copying, and shows when you should not write the loop +at all. ```csharp await using VortexFile file = await VortexFile.OpenAsync(path); @@ -20,29 +21,28 @@ await foreach (var (_, celsius, _) in scan) ``` ``` -the loop: mean 29.4001 over 1000000 rows, 5.4 ms - 17 batches, 123 blocks decoded, 51 requests, 1494044 bytes +the loop: mean 29.4001 over 1000000 rows, 4.3 ms + 17 batches, 123 blocks decoded, 51 requests, 1494068 bytes ``` -That mean is wrong, and on purpose: `Celsius` is nullable, and `Values` is the raw buffer, which -holds something meaningless at each of the 20 000 null slots. The right mean is 30.0000. -[nullable-columns.md](nullable-columns.md) has the two ways to read around the nulls; the end of -this page has the way that does not need a loop. +That mean is wrong on purpose. `Celsius` is nullable, and `Values` is the raw buffer, which holds +meaningless values at the 20 000 null slots. The right mean is 30.0000. +[nullable-columns.md](nullable-columns.md) shows two ways to read around the nulls, and the end of +this page shows the way that needs no loop. ## What comes back -`file.Scan()` returns a `Scan`, a builder: `Where`, `Rows`, `OrderBy` and `With` -compose it, and nothing is read until a sink runs. `await foreach` is the sink that hands you each -batch as a `Columns`, which deconstructs into one `Column` per member of the record, in -declaration order; `_` skips a member. `Columns` also carries `RowCount`, `StartRow` (the +`file.Scan()` returns a `Scan`, which is a builder. `Where`, `Rows`, `OrderBy` and +`With` shape it, and nothing is read until a sink runs. `await foreach` is the sink that hands you each +batch as a `Columns`. It deconstructs into one `Column` per member of the record, in +declaration order, and `_` skips a member. `Columns` also carries `RowCount`, `StartRow` (the file row of its first row) and `Selection` ([selection.md](selection.md)). -`celsius.Values` is a `ReadOnlySpan` over the decoded buffer itself, 64-byte aligned, so -reading a column copies nothing. A scan that only reads, like this one, delivers up to sixteen of -the file's 8 192-row blocks per batch, the rows it decodes at once, and never goes past the end of -a chunk: each chunk of this file is one batch, 17 for a million rows. A scan with a filter, an -order or a take holds one block per batch. `ScanOptions.BatchRows` asks for smaller ones and never -for more: +`celsius.Values` is a `ReadOnlySpan` over the decoded buffer itself, aligned on 64 bytes, so +reading a column copies nothing. A scan that only reads delivers up to sixteen of the file's +8 192-row blocks per batch and never crosses the end of a chunk. In this file each chunk is one +batch, 17 for a million rows. A scan with a filter, an order or a take delivers one block per batch. +`ScanOptions.BatchRows` asks for smaller batches, never for larger ones: ```csharp await foreach (Columns columns in file.Scan().With(new ScanOptions { BatchRows = 4_096 })) @@ -54,9 +54,10 @@ BatchRows 4096: 245 batches, 1000000 rows, the last starting at row 999424 ## Who owns what -**A batch is borrowed.** `Current` is valid until the next `MoveNextAsync`: the buffers go back to -the scan and the next batch decodes into them. `Columns` and `Column` are `ref struct`s, so -the compiler refuses to let one, or a span taken from one, live across an `await`: +A batch is borrowed. `Current` stays valid until the next `MoveNextAsync`, after which its buffers go +back to the scan and the next batch is decoded into them. `Columns` and `Column` are +`ref struct`s, so the compiler refuses to let one of them, or a span taken from one, live across an +`await`: ```csharp await foreach (var (day, _, _) in file.Scan()) @@ -70,14 +71,15 @@ await foreach (var (day, _, _) in file.Scan()) error CS4007: Instance of type 'Vorticity.Column' cannot be preserved across 'await' or 'yield' boundary. ``` -The body of the loop runs synchronously between two batches, which is what makes a `ref struct` -legal there. Do the columnar work, then await what you must, then let the loop move on. To keep a -batch past its iteration, copy it once with `ToOwned()`, or ask for owned batches with -`ToBatchesAsync()` ([owned-batches.md](owned-batches.md)). +The loop body runs synchronously between two batches, which is what makes a `ref struct` legal there. +Do the columnar work first, then await what you need to, then let the loop move on. To keep a batch +beyond its iteration, copy it once with `ToOwned()`, or ask for owned batches with `ToBatchesAsync()` +([owned-batches.md](owned-batches.md)). -**A scan is single-use.** One sink per builder: a second throws `InvalidOperationException`. -`ExplainAsync` may be asked before the sink, and `Statistics` read after it. The sinks, and what -each one costs, are tabled in §5.6 of [14-public-api.md](../design/14-public-api.md). +A scan is single-use. Each builder accepts one sink, and a second one throws +`InvalidOperationException`. You may call `ExplainAsync` before the sink and read `Statistics` after +it. The sinks and what each one costs are listed in +[the public API design](../design/14-public-api.md#56-the-sinks). ## Or let the scan do it @@ -86,46 +88,46 @@ double? mean = await file.Scan().AverageAsync(r => r.Celsius); ``` ``` -AverageAsync: mean 30.0000, 1.8 ms +AverageAsync: mean 30.0000, 2.1 ms 1 blocks decoded, 17 requests, 1150428 bytes ``` -The right answer, faster, and no batch ever reaches your code: the aggregate runs block by block -inside the scan, skips the nulls, and reads the one column it needs, 17 segments of the 51. The -loop above read three, because the record names three: [project-columns.md](project-columns.md) is -how to name fewer. For an aggregate, `BlocksDecoded` counts the blocks it had to bring to the -canonical form; it folded the other 122 in the form they are stored in -([aggregates.md](aggregates.md)). +You get the right answer, faster, and no batch ever reaches your code. The aggregate runs block by +block inside the scan, skips the nulls, and reads only the column it needs, 17 segments out of 51. +The loop above read three columns because the record names three, and +[project-columns.md](project-columns.md) shows how to name fewer. For an aggregate, +`BlocksDecoded` counts the blocks it had to bring to their plain form. It folded the other 122 in +the form they are stored in ([aggregates.md](aggregates.md)). -A file this library writes carries, per column, a null count, order flags and, for a numeric -column, a minimum and a maximum; it carries no sum. `MinAsync`, `MaxAsync` and `CountAsync` -without a filter answer from those statistics and read nothing; `SumAsync` and `AverageAsync` read -the column. [aggregates.md](aggregates.md) has the rest of the operators. +A file written by this library carries, per column, a null count, order flags and, for a numeric +column, a minimum and a maximum. It does not carry a sum. So `MinAsync`, `MaxAsync` and `CountAsync` +without a filter are answered from those statistics and read nothing, while `SumAsync` and +`AverageAsync` read the column. [aggregates.md](aggregates.md) covers the other operators. ## What it costs -* **Reads.** `Statistics.Requests` counts the segments the scan read: 51 for three columns over - 17 batches, 1.49 MB for a file of 1.51 MB, because a segment that spans several batches is read - once and shared by them. A second scan reads them all again; over a source where a read is a - request, [open-a-file.md](open-a-file.md) shows what a `SegmentCache` on the session saves. -* **Memory.** One batch is decoded ahead of the one you hold (`ScanOptions.Prefetch`, 1 by - default), in buffers that alternate rather than accumulate. -* **Allocations.** A whole scan allocated 7 136 bytes over its 17 batches, and 7 224 bytes over - 123 batches of a block as over 245 of half a block: a scan pays for its start and then nothing - per batch, a little less when each batch holds a whole chunk, which then need not stay decoded - from one batch to the next. Counted process-wide with `GC.GetTotalAllocatedBytes`, so the thread - that decodes ahead is included. +* Reads: `Statistics.Requests` counts the segments the scan read, 51 for three columns over 17 + batches, which is 1.49 MB for a file of 1.51 MB. A segment that spans several batches is read once + and shared. A second scan reads everything again. Over a source where each read is a request, + [open-a-file.md](open-a-file.md) shows what a `SegmentCache` on the session saves. +* Memory: one batch is decoded ahead of the one you hold (`ScanOptions.Prefetch`, 1 by default), in + buffers that alternate rather than pile up. +* Allocations: a whole scan allocated 6 944 bytes over its 17 batches, and 7 032 bytes over 123 + batches of one block or 245 batches of half a block. A scan pays for its start and then nothing per + batch, slightly less when each batch holds a whole chunk because the chunk then does not need to + stay decoded from one batch to the next. These numbers come from `GC.GetTotalAllocatedBytes` for + the whole process, so they include the thread that decodes ahead. ## Watch out -* Batches arrive in file order. [keys-in-order.md](keys-in-order.md) has the key-ordered scan. +* Batches arrive in file order. [keys-in-order.md](keys-in-order.md) covers scans in key order. * Breaking out of the loop ends the scan and returns its buffers. -* A scan does not see rows appended after the file was opened. Open it again for that. -* Parallelism is the session's, and pays on aggregates and wide decodes rather than on a loop like - this one ([threads.md](threads.md)). +* A scan does not see rows appended after the file was opened. Open the file again for that. +* Parallelism comes from the session. It pays on aggregates and wide decodes much more than on a + loop like this one ([threads.md](threads.md)). -The figures come from one run of the sample on the demonstration file of a million rows; the -timings are the third of three passes. +The figures come from one run of the sample on the demonstration file of a million rows. The timings +are the third of three passes. ## Run it diff --git a/docs/guide/selection.md b/docs/guide/selection.md index 6882a53c..010a1ebc 100644 --- a/docs/guide/selection.md +++ b/docs/guide/selection.md @@ -21,66 +21,66 @@ first batch: 8192 rows from file row 0, 5970 selected, the third at file row 103 ## Compacted or whole -By default a filtered scan compacts: the rows of a block that passed are copied into the batch, so -every row you see passed, `Selection.IsAll` is true, and row `i` is no longer file row -`StartRow + i`. With `ScanOptions.Compact = false` the scan hands over the block as it decoded it, -every row of it, and `Columns.Selection` says which rows passed: +By default a filtered scan compacts its batches: the rows of a block that passed are copied into the +batch, so every row you see has passed, `Selection.IsAll` is true, and row `i` is no longer file row +`StartRow + i`. With `ScanOptions.Compact = false`, the scan hands over each block as it decoded it, +every row of it, and `Columns.Selection` tells you which rows passed: | member | what it is | |---|---| -| `IsAll` | every row passed; `Words` is then empty | -| `Count` | how many rows passed: 5 970 of the 8 192 of the first block above | -| `Words` | the bitmap, bit `i % 64` of word `i / 64` for row `i`, as `ValidityWords` lays out validity | +| `IsAll` | every row passed, and `Words` is then empty | +| `Count` | how many rows passed, 5 970 of the 8 192 of the first block above | +| `Words` | the bitmap, bit `i % 64` of word `i / 64` for row `i`, laid out like `ValidityWords` | | `Contains(i)` | whether row `i` passed | | `foreach (int i in selection)` | the rows that passed, in order, found word by word with `TrailingZeroCount` | -Positions hold: row `i` of the batch is file row `StartRow + i`, so the third reading above 20 -degrees is file row 103. The values of the rows that did not pass are real values of the file, not -garbage; they simply did not pass. A block the zone maps prune is never read, in either mode. +Positions are preserved: row `i` of the batch is file row `StartRow + i`, so the third reading above +20 degrees is file row 103. The values of the rows that did not pass are real values from the file, +not garbage. They simply did not pass. A block the zone maps prune is never read, in either mode. ## What the copy costs -The same filtered sum, both ways, with the whole record and with a record of the one column the -sum reads, and with a filter that keeps 2 % of the rows: +The same filtered sum both ways, with the whole record and with a record of the single column the +sum reads, and with a filter that keeps only 2 % of the rows: | record, filter | rows kept | whole blocks | compacted | |---|---|---|---| -| `Reading`, `Celsius > 20` | 735 000 | 11.2 ms | 21.9 ms | -| `Temperature`, `Celsius > 20` | 735 000 | 9.5 ms | 17.3 ms | -| `Reading`, `Celsius > 49` | 22 500 | 9.7 ms | 20.0 ms | +| `Reading`, `Celsius > 20` | 735 000 | 8.3 ms | 14.2 ms | +| `Temperature`, `Celsius > 20` | 735 000 | 6.9 ms | 8.1 ms | +| `Reading`, `Celsius > 49` | 22 500 | 7.1 ms | 6.2 ms | ```csharp [VortexRecord] public partial record struct Temperature(double? Celsius); ``` -The compaction copies every column the record names, whether the loop reads it or not: with -`Reading` it copies `Day` and `City` too, and `City` is text stored as runs, which is why the gap is -widest there. Narrowing the record ([project-columns.md](project-columns.md)) shrinks the copy, and -a filter that keeps few rows copies few, but the compacted scan still walks every live block to -build them. Both modes allocated about 10 KB per scan, the scan's own. +Compaction copies every column the record names, whether the loop reads it or not. With `Reading` it +also copies `Day` and `City`, and `City` is text stored as runs, which is why the gap is widest there. +Narrowing the record ([project-columns.md](project-columns.md)) shrinks the copy, and a filter that +keeps few rows copies little: at 2 % of the rows, the compacted scan even comes out ahead. Both modes +allocated about 10 KB per scan, which is the scan's own. ## Which to use -* **Whole blocks** when the loop reads few columns of many, when the filter keeps much of each +* Whole blocks work best when the loop reads few of many columns, when the filter keeps much of each block, or when you want file positions without arithmetic. -* **Compacted**, the default, when the code downstream wants dense columns: a SIMD sum over +* Compacted batches, the default, suit code that wants dense columns downstream: a SIMD sum over `Values`, or `ToOwned()` to keep a batch of the rows that passed ([owned-batches.md](owned-batches.md)). -* Neither matters for an aggregate: `SumAsync`, `GroupBy` and the rest read the selection +* For an aggregate it makes no difference. `SumAsync`, `GroupBy` and the rest read the selection themselves and never materialise a batch ([aggregates.md](aggregates.md)). ## Watch out -* **Iterate the selection, not the block.** A `foreach` over `Values` of a whole block adds the - rows that did not pass, nulls included. -* **A take is a selection too.** `Rows(4, 900_000)` without a filter delivers the blocks that hold - those rows with a selection of them ([read-rows-by-index.md](read-rows-by-index.md)). -* **Compaction decodes run-end columns.** With whole blocks the encoded forms arrive as the file - holds them ([encoded-forms.md](encoded-forms.md)). +* Iterate the selection, not the block. A `foreach` over `Values` of a whole block also adds the rows + that did not pass, nulls included. +* A take is a selection too. `Rows(4, 900_000)` without a filter delivers the blocks that hold those + rows, with a selection of them ([read-rows-by-index.md](read-rows-by-index.md)). +* Compaction decodes run-end columns. With whole blocks, the encoded forms arrive as the file holds + them ([encoded-forms.md](encoded-forms.md)). -The figures come from one run of the sample on the demonstration file of a million rows; each -timing is the best of three passes. +The figures come from one run of the sample on the demonstration file of a million rows. Each timing +is the best of three passes. ## Run it diff --git a/docs/guide/statistics-and-pruning.md b/docs/guide/statistics-and-pruning.md index 96c2c759..d5657fda 100644 --- a/docs/guide/statistics-and-pruning.md +++ b/docs/guide/statistics-and-pruning.md @@ -1,6 +1,7 @@ # Statistics and pruning -See what a scan will skip before it runs, what it did after, and what makes pruning work. +See what a scan will skip before it runs and what it did afterwards, and learn what makes pruning +work. ```csharp ScanPlan plan = await file.Scan().Where(r => r.Day >= 900).ExplainAsync(); @@ -15,77 +16,81 @@ Print("Day >= 900", scan.Statistics); ``` ``` -plan, Day >= 900: 1000000 rows, 14 of 123 blocks live, 13 segments, 210980 bytes to read, may match True +plan, Day >= 900: 1000000 rows, 14 of 123 blocks live, 13 segments, 211004 bytes to read, may match True zone map: 109 blocks pruned, 1 segments and 2124 bytes read to decide count: exact True, 100000 rows, 109 pruned, 13 proven, 1 decoded -ran, Day >= 900: 100000 rows in 5 batches, 12 requests, 208856 bytes, 14 blocks decoded, 109 pruned +ran, Day >= 900: 100000 rows in 5 batches, 12 requests, 208880 bytes, 14 blocks decoded, 109 pruned ``` -`Print` writes the fields of the two records; the sample has it. 2 124 bytes of zone maps decided -that 109 of the file's 123 blocks hold nothing for this predicate, and they were never decoded. +`Print` writes the fields of the two records, and the sample holds it. Reading 2 124 bytes of zone +maps was enough to decide that 109 of the file's 123 blocks hold nothing for this predicate, and +those blocks were never decoded. ## The plan, before -`ExplainAsync()` reads the file's statistics, its zone maps and its indexes, and of the data at -most the few segments of a sorted column it searches, and returns a `ScanPlan`: +`ExplainAsync()` reads the file's statistics, its zone maps and its indexes, plus at most the few +segments of a sorted column it searches, and returns a `ScanPlan`: | field | what it says | |---|---| -| `Rows` | the rows the scan covers: the file's, those of a range, or the rows a take takes | +| `Rows` | the rows the scan covers: the whole file, a range, or the rows a take asks for | | `Blocks`, `LiveBlocks` | the blocks those rows touch, and those no structure could prove empty | -| `Segments`, `BytesToRead` | the distinct segments the live blocks need, with what consulting the structures reads, and their bytes | +| `Segments`, `BytesToRead` | the distinct segments the live blocks need, including what consulting the structures reads, and their size | | `MayMatch` | false when the file's statistics prove the scan empty | | `Pruning` | one `PruningStep` per structure, cheapest first: `Structure`, `BlocksPruned`, and the `SegmentsRead` and `BytesRead` it took to consult | | `Count` | how a `CountAsync` of the scan would be answered | | `Order` | the key source of an `OrderBy` scan, null otherwise ([keys-in-order.md](keys-in-order.md)) | +| `Grouping` | for a group by, a `GroupPlan`: its key components and whether each is sorted, the component the groups stream on or why none does, the aggregates, the order of the groups and the degree. Null without a group by | `Count` is a `CountPlan`. `Exact` is true when a sorted column or a sorted-runs index answers the -count by itself, and `Rows` is then the count: 100 000 above, with nothing decoded. `Pruned`, -`Proven` and `Decoded` say how the blocks divide otherwise: ruled out by their bounds, decided whole +count on its own, and `Rows` is then the count: 100 000 above, with nothing decoded. Otherwise, +`Pruned`, `Proven` and `Decoded` say how the blocks divide: ruled out by their bounds, decided whole by them, or left to decode. ## The numbers, after -`scan.Statistics` is valid once a sink has run, and is a `ScanStatistics`: `Rows` and `Batches` -delivered, `Requests` and `BytesRequested` made to the source, `BlocksDecoded`, `BlocksPruned`, -and `CacheHits` from the session's segment cache ([threads.md](threads.md)). - -**The plan's bytes and the run's differ by the zone maps.** The plan counts each segment once, 13 of -them and 210 980 bytes with the 2 124 bytes of zone maps it consulted; the run made 12 requests for -208 856 bytes, because the file kept the zone maps the plan had read. A segment here holds several -blocks of one column, and the scan reads it once for all the blocks that need it. A second -scan reads it again; over a `FileSegmentSource` or a remote source, a session `SegmentCache` is -what keeps it from reaching the disk or the network again. `BlocksDecoded` matches `LiveBlocks`: -14. They come in 5 batches rather than 14: the 13 blocks the zone maps prove whole, every row of -which passes, go out up to sixteen at a time, as a scan that only reads delivers them +`scan.Statistics` is valid once a sink has run. It is a `ScanStatistics`: the `Rows` and `Batches` +delivered, the `Requests` and `BytesRequested` made to the source, `BlocksDecoded`, `BlocksPruned`, +and the `CacheHits` from the session's segment cache ([threads.md](threads.md)). For a group by, +`Grouping` adds a `GroupStatistics`: the groups found and the most held at once, the peak memory, the +lanes, what was spilled, and the time to the first batch ([queries.md](queries.md)). + +The plan's bytes and the run's differ by the zone maps. The plan counts each segment once, 13 of them +and 211 004 bytes including the 2 124 bytes of zone maps it consulted. The run made 12 requests for +208 880 bytes, because the file kept the zone maps the plan had already read. A segment here holds +several blocks of one column, and the scan reads it once for all the blocks that need it. A second +scan reads it again, and over a `FileSegmentSource` or a remote source, a session `SegmentCache` is +what saves that second trip to the disk or the network. `BlocksDecoded` matches `LiveBlocks`, 14. +They arrive in 5 batches rather than 14 because the 13 blocks the zone maps prove whole, where every +row passes, are delivered up to sixteen at a time, as in a scan without a filter ([blocks-and-chunks.md](blocks-and-chunks.md)). ## A filter that prunes, and one that cannot -Measured on the demonstration file, 1 508 212 bytes: +Measured on the demonstration file, 1 508 316 bytes: | | live blocks | requests | bytes requested | rows | |---|---|---|---|---| -| no filter | 123 of 123 | 51 | 1 494 044 | 1 000 000 | -| `Day >= 900` | 14 of 123 | 12 | 208 856 | 100 000 | -| `Celsius > 45` | 123 of 123 | 51 | 1 494 044 | 122 500 | +| no filter | 123 of 123 | 51 | 1 494 068 | 1 000 000 | +| `Day >= 900` | 14 of 123 | 12 | 208 880 | 100 000 | +| `Celsius > 45` | 123 of 123 | 51 | 1 494 068 | 122 500 | | `Day >= 5000` | 0 of 123 | 0 | 0 | 0 | -The rows are in `Day` order, so 109 blocks are provably outside `Day >= 900`: a seventh of the -reading for a tenth of the rows, since a column is read by the chunk, eight blocks here, and the -live blocks bring in the chunks they fall in. `Celsius` walks its whole range inside every block, so no block can be -excluded: finding that out cost the 3 108 bytes of its zone maps, which the plan read here, and the -run then read what no filter reads. `Day >= 5000` lies above the file's own maximum, and `MayMatch` says so: the file -statistics are in memory once the file is open, so the plan and the run read nothing at all, and -count every block pruned by the file statistics. +The rows are in `Day` order, so 109 blocks are provably outside `Day >= 900`. The scan reads a +seventh of the file for a tenth of the rows, because a column is read by the chunk, eight blocks +here, and the live blocks bring in the chunks they belong to. `Celsius` covers its whole range in +every block, so no block can be excluded. Finding that out cost the 3 108 bytes of its zone maps, +which the plan read here, and the run then read everything. `Day >= 5000` lies above the file's +maximum, and `MayMatch` says so. The file statistics are in memory once the file is open, so the plan +and the run read nothing at all and count every block as pruned by the file statistics. ## Three levels | level | granularity | cost to consult | |---|---|---| -| file statistics | the whole file | nothing: read at open | +| file statistics | the whole file | nothing, they are read at open | | zone maps | one block | one small read | -| the values | one row | the block | +| the values | one row | decoding the block | `file.MayMatch` asks the first level alone, and a `false` is a proof: @@ -96,8 +101,8 @@ MayMatch(Celsius > 100): False MayMatch(City == "Paris"): True ``` -`file.Statistics` is what it reads, one `FieldStatistics` per top-level column, each value a -`TryGet` that answers false when the file does not record it: +`file.Statistics` is what it reads: one `FieldStatistics` per top-level column, each value behind a +`TryGet` that returns false when the file does not record it: ``` Day: bounds 0 to 999, nulls 0, sorted True, constant unknown @@ -106,47 +111,51 @@ MayMatch(City == "Paris"): True sums: Day not recorded, Celsius not recorded ``` -`TryGetMin` and `TryGetMax` read a bound as the column's type and answer only for an exact -one. A file this library writes records, per column, the bounds of a numeric column, the null count -and whether the column is sorted; it does not record a sum, whether the column is constant, or file -bounds for a text column, whose bounds live in the zone maps instead. A file from another writer may -record more, which is why each is a `TryGet`. They are claims the file makes about itself: open -with `VerifyStatistics = true` to have them checked against what is decoded ([limits.md](limits.md)). +`TryGetMin` and `TryGetMax` read a bound as the column's type, and answer only for an exact +bound. A file written by this library records, per column, the bounds of a numeric column, the null +count and whether the column is sorted. It does not record a sum, whether the column is constant, or +file-level bounds for a text column, whose bounds live in the zone maps instead. A file from another +writer may record more, which is why each value is a `TryGet`. These are claims the file makes about +itself. Open it with `VerifyStatistics = true` to have them checked against what is decoded +([limits.md](limits.md)). ## What makes pruning work -**Clustering.** A block is excluded by its bounds, and a block that holds the whole range of a -column has bounds that exclude nothing. Order the rows by the column you filter on, or group them -by it. +Clustering comes first. A block is excluded by its bounds, and a block that holds the whole range of +a column has bounds that exclude nothing. Order the rows by the column you filter on, or at least +group them by it. -**Bounds for text columns**, which the writer keeps by default, `StringBoundBytes = 16`. They prune -only if the rows are clustered. `City == "Paris"` over 200 000 rows, the same data twice: +Text columns get bounds too, which the writer keeps by default (`StringBoundBytes = 16`). They only +prune when the rows are clustered. Here is `City == "Paris"` over 200 000 rows, the same data twice: ``` City == "Paris" over 200000 rows, cities cycling row by row: 25 of 25 blocks live, 25000 rows City == "Paris" over 200000 rows, clustered by city: 8 of 25 blocks live, 25000 rows ``` -Clustered, Paris fills four blocks, the last shared with the next city. The four other live blocks -each straddle two cities whose names bracket "Paris", Marseille and Toulouse for instance, so that +Clustered, Paris fills four blocks, the last one shared with the next city. The four other live +blocks each straddle two cities whose names bracket "Paris", Marseille and Toulouse for instance, so their bounds cannot exclude it. -**Indexes**, for what bounds cannot do: a value scattered over every block is found by a bloom -filter, a substring by an n-gram one ([indexes.md](indexes.md)). They appear in `Pruning` as steps -of their own, with what they cost to consult. +Indexes handle what bounds cannot. A Bloom filter finds a value scattered over every block, and an +n-gram filter finds a substring ([indexes.md](indexes.md)). They appear in `Pruning` as steps of their +own, with what they cost to consult. ## Watch out -* `ExplainAsync` on a scan with no filter is still worth running: it gives the file's shape. -* `Blocks` and `LiveBlocks` follow `Rows(...)`: a range of ten rows plans 1 block of 1, a take of - two rows 2 of 2, with `Rows` 2 ([read-rows-by-index.md](read-rows-by-index.md)). -* `Statistics` read before the sink has run are zeros, and a scan runs one sink only. -* `ScanOptions.Pruning = false` and `UseIndexes = false` turn the structures off, to check them, - never to change a result. - -What each statistic licenses is [08-semantics.md](../design/08-semantics.md) §1, how the scan uses -them [11-write-strategy.md](../design/11-write-strategy.md) §6, and the plan's records §7 of -[14-public-api.md](../design/14-public-api.md). +* `ExplainAsync` is worth running even on a scan without a filter, because it gives you the shape of + the file. +* `Blocks` and `LiveBlocks` follow `Rows(...)`: a range of ten rows plans 1 block of 1, and a take of + two rows 2 of 2, with `Rows` equal to 2 ([read-rows-by-index.md](read-rows-by-index.md)). +* `Statistics` read before the sink has run are zeros, and a scan runs only one sink. +* `ScanOptions.Pruning = false` and `UseIndexes = false` turn the structures off, to check them, never + to change a result. + +What each statistic allows is defined in +[the semantics design](../design/08-semantics.md#1-statistic-precision-what-inexact-licenses), how the +scan uses them in [the write strategy](../design/11-write-strategy.md#6-reading-what-the-writer-produced), +and the plan's records in +[the public API design](../design/14-public-api.md#7-options-plans-reports-diagnostics-exceptions). ## Run it diff --git a/docs/guide/stream-to-an-object.md b/docs/guide/stream-to-an-object.md index 05b4d311..eae0b529 100644 --- a/docs/guide/stream-to-an-object.md +++ b/docs/guide/stream-to-an-object.md @@ -1,7 +1,7 @@ # Stream to an object -Write a file to anything that takes bytes (an upload, a socket, a stream) through a `PipeWriter`, -and let a slow destination slow the producer down. +Write a file to anything that accepts bytes, such as an upload, a socket or a stream, through a +`PipeWriter`, and let a slow destination slow the producer down. ```csharp Pipe pipe = new Pipe(new PipeOptions(pauseWriterThreshold: 256 << 10, resumeWriterThreshold: 128 << 10)); @@ -25,59 +25,62 @@ await using (VortexFileWriter writer = session.CreateWriter(pipe.Writer, Reading } ``` -`UploadAsync` stands in for a multipart upload: it reads the pipe, takes 64 KiB at a time as a part, +`UploadAsync` stands in for a multipart upload. It reads the pipe, takes 64 KiB at a time as a part, and spends 2 ms on each. `Reading.Schema` is the record's schema, and `WriteAsync` binds the record to it as it would to a file. ``` -through a Pipe: 1603156 bytes in 25 parts of up to 65536 bytes; 61 flushes waited 47 ms for the upload in all +through a Pipe: 1603260 bytes in 25 parts of up to 65536 bytes; 61 flushes waited 49 ms for the upload in all read back from the uploaded bytes: 1000000 rows, mean 30.0000 -through PipeWriter.Create(stream): 1508212 bytes, 1000000 rows +through PipeWriter.Create(stream): 1508316 bytes, 1000000 rows ``` ## What happens -* **The writer writes into the pipe's buffers and flushes only when asked.** `WriteAsync` encodes - chunks into the `PipeWriter` without flushing it; `FlushAsync` calls the pipe's own `FlushAsync`, - which returns once the reader has taken enough bytes to bring the pipe under its pause threshold. - A slow destination therefore slows the producer instead of growing a buffer: here 61 flushes - waited 47 ms in all, about what the 25 parts took to upload. -* **`CompleteAsync` completes the pipe** once the footer is in it, which is how the reader knows the - file is whole. Do not complete it yourself. -* **Any `PipeWriter` will do.** `PipeWriter.Create(stream)` covers a `Stream`, a `FileStream` or a - `MemoryStream`, and the bytes are the same file as one written to a path. An object store's own - pipe gets the same backpressure, and a multipart upload is a pipe that sends a part per threshold; - nothing in the writer knows. +The writer writes into the pipe's buffers and only flushes when asked. `WriteAsync` encodes chunks +into the `PipeWriter` without flushing it, and `FlushAsync` calls the pipe's own `FlushAsync`, which +returns once the reader has taken enough bytes to bring the pipe back under its pause threshold. A +slow destination therefore slows the producer instead of growing a buffer: here the 61 flushes +waited 49 ms in total, about what the 25 parts took to upload. + +`CompleteAsync` completes the pipe once the footer is in it, which is how the reader knows the file +is whole. Do not complete the pipe yourself. + +Any `PipeWriter` will do. `PipeWriter.Create(stream)` covers a `Stream`, a `FileStream` or a +`MemoryStream`, and the bytes form the same file as one written to a path. An object store's own pipe +gets the same backpressure, and a multipart upload is just a pipe that sends a part each time it +fills. Nothing in the writer needs to know. ## Giving up -A writer abandoned, or disposed without `CompleteAsync`, has no file to delete. It completes the -pipe with an error instead, so that whatever it feeds knows the bytes are not a file: +A writer that is abandoned, or disposed without `CompleteAsync`, has no file to delete. It completes +the pipe with an error instead, so that whatever the pipe feeds knows the bytes are not a file: ``` a writer disposed without CompleteAsync: the upload sees OperationCanceledException: The Vortex file was abandoned; the bytes written so far are not a file. ``` -The consumer's `ReadAsync` throws that exception. An upload should abort there, and never publish -what it received. +The consumer's `ReadAsync` throws that exception. An upload should abort at that point and never +publish what it received. ## What it costs -The file through the pipe is 5 % larger than through `PipeWriter.Create(stream)` with one write: -flushing every 256 KiB seals smaller chunks, and each chunk carries its own framing and encoding -choices ([blocks-and-chunks.md](blocks-and-chunks.md)). Flush at the size your destination wants, -not more often. +The file written through the pipe is about 6 % larger than the one written through +`PipeWriter.Create(stream)` in a single write. Flushing every 256 KiB seals smaller chunks, and each +chunk carries its own framing and encoding choices ([blocks-and-chunks.md](blocks-and-chunks.md)). +Flush at the size your destination wants, and no more often. ## Watch out -* **Bytes pile up in the pipe until you flush.** A pause threshold applies at a flush and nowhere - else, so a producer that feeds the builder and never calls `FlushAsync` buffers the whole file in - the pipe. `UnflushedBytes` says how much is waiting. `WriteAsync` over an `IAsyncEnumerable` - flushes by itself once 8 MiB are waiting ([write-rows.md](write-rows.md)). -* **The sink is written once, front to back.** Nothing is sought or rewritten, which is what lets a - socket or an upload be the destination. Appending needs a path ([append-and-repair.md](append-and-repair.md)). -* Reading the uploaded file back from memory is `session.OpenAsync(new MemorySegmentSource(bytes))`; - from an object store, through your own `ISegmentSource` ([object-store.md](object-store.md)). +* Bytes pile up in the pipe until you flush. A pause threshold only applies at a flush, so a producer + that feeds the builder and never calls `FlushAsync` buffers the whole file in the pipe. + `UnflushedBytes` says how much is waiting. `WriteAsync` over an `IAsyncEnumerable` flushes on its + own once 8 MiB are waiting ([write-rows.md](write-rows.md)). +* The sink is written once, front to back. Nothing is sought or rewritten, which is what lets a socket + or an upload be the destination. Appending, on the other hand, needs a path + ([append-and-repair.md](append-and-repair.md)). +* To read the uploaded file back from memory, use `session.OpenAsync(new MemorySegmentSource(bytes))`. + From an object store, use your own `ISegmentSource` ([object-store.md](object-store.md)). ## Run it diff --git a/docs/guide/text-columns.md b/docs/guide/text-columns.md index 5a433e89..03dbd4d8 100644 --- a/docs/guide/text-columns.md +++ b/docs/guide/text-columns.md @@ -1,7 +1,7 @@ # Text columns -Read a text column as the UTF-8 bytes it holds, allocate a `string` only where you need one, and -count distinct values without a loop. +Read a text column as the UTF-8 bytes it holds, allocate a `string` only where you need one, and count +distinct values without a loop. ```csharp long paris = 0; @@ -26,36 +26,36 @@ await foreach (var (_, _, city) in file.Scan()) ## What a text column gives you -A `string` member makes a `Column`, and a `string?` member a `Column`, with the +A `string` member produces a `Column`, and a `string?` member a `Column`, with the same accessors: | member | returns | allocates | |---|---|---| | `city[i]` | the value as a `ReadOnlySpan` of UTF-8, pointing into the decoded batch | nothing | | `city.GetLength(i)` | its length in bytes | nothing | -| `city.GetString(i)` | a `string`, or `null` at a null row | one string per call | -| `city.IsValid(i)` | false at a null row, whose span is empty | nothing | +| `city.GetString(i)` | a `string`, or `null` for a null row | one string per call | +| `city.IsValid(i)` | false for a null row, whose span is empty | nothing | -The span is compared with the `u8` literals of C#, `SequenceEqual`, `StartsWith`, `IndexOf`, +You compare the span with C#'s `u8` literals, `SequenceEqual`, `StartsWith`, `IndexOf`, `SearchValues` and the rest of `System.Memory`, byte for byte, with no culture and no -transcoding. It is borrowed like everything else from a scan: valid inside the loop body only -([scan-a-table.md](scan-a-table.md)). Why spans come first and strings on request is §4 of -[07-dotnet-mapping.md](../design/07-dotnet-mapping.md). +transcoding. Like everything a scan hands out, it is borrowed and only valid inside the loop body +([scan-a-table.md](scan-a-table.md)). Why spans come first and strings on request is explained in +[the .NET mapping](../design/07-dotnet-mapping.md#4-strings-spans-first-string-on-request). ## What a string costs The same two counts, once over spans and once with `city.GetString(i)`: ``` -UTF-8 spans 39.9 ms, 7256 bytes allocated: 125006 rows in Paris, 249999 starting with L -GetString 54.8 ms, 37007216 bytes allocated: 125006 rows in Paris, 249999 starting with L -the scan alone 3.6 ms, 7184 bytes allocated: 1000000 rows -CountDistinctAsync 10.3 ms, 10584 bytes allocated: 8 cities +UTF-8 spans 40.6 ms, 7064 bytes allocated: 125006 rows in Paris, 249999 starting with L +GetString 56.3 ms, 37007024 bytes allocated: 125006 rows in Paris, 249999 starting with L +the scan alone 2.2 ms, 6992 bytes allocated: 1000000 rows +CountDistinctAsync 9.4 ms, 15616 bytes allocated: 8 cities ``` -A million strings are 37 MB of garbage for the collector, where the spans allocate a few bytes more -than the scan itself does. Keep `GetString` for the values that leave the loop: a dictionary key, a -log line, a result. +A million strings are 37 MB of garbage for the collector, while the spans allocate barely more than +the scan itself. Keep `GetString` for values that leave the loop: a dictionary key, a log line, a +result. ## Distinct values, and filters on text @@ -63,32 +63,32 @@ log line, a result. long cities = await file.Scan().CountDistinctAsync(r => r.City); ``` -A distinct count is an operator of the scan, not a loop over the values: it counts non-null values, -counts codes rather than rows on a dictionary-encoded block, and merges the chunks by value. -On the visits file, whose `Referrer` is a nullable text column stored as a dictionary: +A distinct count is an operator of the scan, not a loop over the values. It counts non-null values, +counts codes rather than rows on a dictionary-encoded block, and merges the chunks by value. On the +visits file, whose `Referrer` is a nullable text column stored as a dictionary: ``` referrers: 13 distinct, 25000 null, 75000 starting with https:// ``` -The second and third numbers are `Where(v => v.Referrer == null).CountAsync()` and -`Where(v => v.Referrer.StartsWith("https://")).CountAsync()`: equality, `In`, `StartsWith`, -`Contains` and `Like` on text are pushed down like any predicate, and a file written with string -bounds (the default, `StringBoundBytes = 16`) prunes blocks with them -([filter-rows.md](filter-rows.md)). [encoded-forms.md](encoded-forms.md) shows how to read the -dictionary codes yourself, and why you rarely need to. +The second and third numbers come from `Where(v => v.Referrer == null).CountAsync()` and +`Where(v => v.Referrer.StartsWith("https://")).CountAsync()`. Equality, `In`, `StartsWith`, `Contains` +and `Like` on text are pushed down like any predicate. A file written with string bounds, which is +the default (`StringBoundBytes = 16`), prunes blocks with them ([filter-rows.md](filter-rows.md)). +[encoded-forms.md](encoded-forms.md) shows how to read the dictionary codes yourself, and why you +rarely need to. ## Watch out -* **An empty span is not a null.** A null row gives an empty span and `GetLength(i) == 0`, as an - empty string does; `IsValid(i)`, or `GetString(i)` returning `null`, tells them apart. Of the - first eight visits, two have a null referrer. -* **Comparisons are ordinal.** A text predicate compares UTF-8 bytes; culture-aware comparison is - for the strings you materialise. -* **A `string` member of a row costs the same allocation.** `ToRecordsAsync()` allocates one string - per row for it ([read-rows.md](read-rows.md)). +* An empty span is not a null. A null row gives an empty span and `GetLength(i) == 0`, exactly like + an empty string. `IsValid(i)`, or `GetString(i)` returning `null`, tells them apart. Of the first + eight visits, two have a null referrer. +* Comparisons are ordinal. A text predicate compares UTF-8 bytes, and culture-aware comparison is for + the strings you materialise. +* A `string` member of a row costs the same allocation. `ToRecordsAsync()` allocates one string per + row for it ([read-rows.md](read-rows.md)). -The figures come from one run of the sample on the demonstration files; each timing is the best of +The figures come from one run of the sample on the demonstration files. Each timing is the best of three passes. ## Run it diff --git a/docs/guide/threads.md b/docs/guide/threads.md index e676f220..e5d804ee 100644 --- a/docs/guide/threads.md +++ b/docs/guide/threads.md @@ -26,83 +26,83 @@ Console.WriteLine($"four concurrent scans of one open file: {string.Join(", ", a four concurrent scans of one open file: 1000000, 1000000, 1000000, 1000000 ``` -One open file, many scans: each task builds its own scan and enumerates it, and nothing is shared -but the file and its session. +One open file can serve many scans. Each task builds its own scan and enumerates it, and nothing is +shared but the file and its session. ## The session -A `VortexSession` holds what would otherwise be process-wide state, so that a host decides it once: +A `VortexSession` holds what would otherwise be process-wide state, so a host decides it once: | option | default | what it governs | |---|---|---| -| `MemoryPool` | `AlignedMemoryPool.Shared` | every batch and builder buffer, and the segments a source of yours reads; disposing the session returns them. The file sources of the library read into `AlignedMemoryPool.Shared` | +| `MemoryPool` | `AlignedMemoryPool.Shared` | every batch and builder buffer, and the segments read by a source of yours. The file sources of the library read into `AlignedMemoryPool.Shared` | | `SegmentCache` | none | segments kept across scans, one budget for every file of the session whose reads do I/O | | `MaxConcurrentReads` | 16 | reads in flight across every scan of every file of the session whose reads do I/O | -| `MaxDegreeOfParallelism` | 1 | how many chunks a scan decodes and aggregates at once, and how many threads a writer compresses on | -| `IndexCacheBytes` | 64 MiB | decoded index runs | -| `MappedFileCacheCount` | 64 | files opened from a path kept mapped once closed, so the next open of one takes over its mapping | -| `MapFiles` | true | whether a file opened from a path is mapped by its first scan; false reads it positionally, and a file cut short under a scan then fails the read instead of the process | -| `MemoryBudget` | the process's | the memory the session's queries share with every session given the same `QueryMemoryBudget`: their groups' tables and their merges. Past it, a group by spills to `ScratchDirectory`, or fails with `VortexMemoryException` | -| `ScratchDirectory` | the system's temporary directory | where a group by writes what its memory budget does not hold, and a sort its runs; never a file system in memory | -| `ScratchBudget` | none: a tenth of the directory's free space kept | the scratch the session's queries share with every session given the same `ScratchBudget`; past it, `VortexMemoryException`, no file left | +| `MaxDegreeOfParallelism` | 1 | how many chunks a scan decodes and aggregates at once, and how many threads a writer uses | +| `IndexCacheBytes` | 64 MiB | decoded index runs kept per open file | +| `MappedFileCacheCount` | 64 | files opened from a path that stay mapped after they are closed, so the next open reuses the mapping | +| `MapFiles` | true | whether a file opened from a path is mapped by its first scan. When false, it is read positionally, and a file truncated during a scan fails the read instead of the process | +| `MemoryBudget` | the process's | the memory the session's queries share with every session given the same `QueryMemoryBudget`: their group tables and their merges. Past it, a group by spills to `ScratchDirectory`, or fails with `VortexMemoryException` | +| `ScratchDirectory` | the system's temporary directory | where a group by writes what its memory budget cannot hold, and where a sort writes its runs. A file system in memory is refused | +| `ScratchBudget` | none, a tenth of the directory's free space is kept free | the scratch space the session's queries share with every session given the same `ScratchBudget`. Past it, `VortexMemoryException`, and no file is left behind | | `Extensions` | empty | extension types the session reads beyond the editions | -The options are set inside `Create` and frozen when it returns; setting one afterwards throws +The options are set inside `Create` and frozen when it returns. Setting one afterwards throws `InvalidOperationException: The session is created; its options are frozen. Set them inside -VortexSession.Create.` `VortexFile.OpenAsync(path)` uses `VortexSession.Default`, which is immutable: -the shared pool, no cache, 16 reads in flight, no parallelism, 64 closed files kept mapped. +VortexSession.Create.` `VortexFile.OpenAsync(path)` uses `VortexSession.Default`, which is +immutable: the shared pool, no cache, 16 reads in flight, no parallelism, and 64 closed files kept +mapped. -A session is disposed after its files. Disposing it with one still open throws +A session is disposed after its files. Disposing it while one is still open throws `InvalidOperationException: The session still has 'readings.vortex' open; dispose every file before -the session.` Two `await using` declarations in that order, as above, do it right. +the session.` Two `await using` declarations in that order, as above, get it right. -**The cache and the bound on reads apply to a source that does I/O**: a `FileSegmentSource`, as +The cache and the limit on reads in flight apply to sources that do I/O: a `FileSegmentSource`, as above, or your own `ISegmentSource` ([object-store.md](object-store.md)). A path opened with -`session.OpenAsync(path)` is memory-mapped by its first scan, and a mapping has nothing to bound or -to cache, unless the session sets `MapFiles` to false: the path is then read as a -`FileSegmentSource` would read it, through both. With the -cache, a fifth scan of the file made 45 requests and the cache served all 45; over the five scans -it counted 121 to 138 hits and 87 to 104 misses in seven runs, and held 1 431 KiB, the data -segments of a 1.5 MB file but for the six lying in the tail its open read, which the file serves -itself. The concurrent scans missed more than the 45 segments they ask for: two scans that ask for -one at the same moment both miss it. +`session.OpenAsync(path)` is memory-mapped by its first scan, and a mapping has nothing to limit or +cache, unless the session sets `MapFiles` to false, in which case the path is read like a +`FileSegmentSource`, through both. With the cache in place, a fifth scan of the file made 45 requests +and the cache served all 45. Over the five scans, this run counted 173 hits and 52 misses, and the +cache held 1 431 KiB: the data segments of the 1.5 MB file except the six that lie in the tail read by +the open, which the file serves itself. The concurrent scans miss more often than a single scan +would, because two scans asking for the same segment at the same moment both miss it. ## What is safe to share -| | | +| type | contract | |---|---| -| `VortexSession` | **thread-safe**, and meant to be shared by every request of a process | -| `VortexFile` | **thread-safe**: concurrent scans of one open file are expected; dispose it once none runs | -| `ISegmentSource` | must be thread-safe; the built-in ones are | -| `Scan`, `Scan` | **single-use, one thread**: build it, then run one sink | -| `Columns`, `Column`, `BatchView` | valid inside the loop body only; the compiler holds you to it | -| `RecordBatch` | one consumer at a time; it may be handed to another thread and disposed there | +| `VortexSession` | thread-safe, and meant to be shared by every request of a process | +| `VortexFile` | thread-safe. Concurrent scans of one open file are expected, so dispose it once none is running | +| `ISegmentSource` | must be thread-safe, and the built-in ones are | +| `Scan`, `Scan` | single-use and single-threaded: build it, then run one sink | +| `Columns`, `Column`, `BatchView` | valid inside the loop body only, which the compiler enforces | +| `RecordBatch` | one consumer at a time. It may be handed to another thread and disposed there | | `KeyCursor` | not thread-safe | -| `VortexFileWriter`, `ColumnsBuilder` | one thread, one writer per file | +| `VortexFileWriter`, `ColumnsBuilder` | one thread, and one writer per file | | `ScanPlan`, `KeyPlan`, `WriteReport` | immutable | -A second sink on the same scan throws `InvalidOperationException: A scan is single-use: build -another one for another sink.` A scan is cheap to build; build one per question. +A second sink on the same scan throws `InvalidOperationException: A scan is single-use: build another +one for another sink.` A scan is cheap to build, so build one per question. ## Parallelism -Parallelism is off by default: a library does not take a host's cores without being asked. The host -asks once, with `MaxDegreeOfParallelism` on its session, and one scan can say otherwise: +Parallelism is off by default, because a library should not take a host's cores without being asked. +The host asks once, with `MaxDegreeOfParallelism` on its session, and a single scan can say otherwise: ```csharp ScanOptions four = new ScanOptions { DegreeOfParallelism = 4 }; // 0, the default, is the session's ``` -A writer takes the session's degree too, and `VortexWriteOptions.DegreeOfParallelism` says -otherwise for one file. As a batch of a block or more comes in, it summarizes its columns side by -side: each column's statistics, and apart from them a text column's bounds and the distinct values -a dictionary is priced from. It compresses a column's zstd frames side by side too, a frame to a -block, once the column holds a quarter of a megabyte of values, and chooses and writes the -encodings on the calling thread. The file is the same bytes whatever the degree. A write that -builds indexes summarizes its columns on one thread. +A writer uses the session's degree too, and `VortexWriteOptions.DegreeOfParallelism` overrides it +for one file. When a batch of a block or more comes in, the writer summarizes its columns side by +side: each column's statistics, plus a text column's bounds and the distinct values a dictionary is +priced from. It also compresses a column's zstd frames in parallel, one frame per block, once the +column holds a quarter of a megabyte of values. Choosing and writing the encodings stays on the +calling thread, and the file is byte for byte the same whatever the degree. A write that builds +indexes summarizes its columns on one thread. -Measured on the demonstration file, warmed, each variant run in turn, the best of fifteen rounds in a -process and the median of seven processes (2026-10-09): +Measured on the demonstration file, warmed up, each variant run in turn, the best of fifteen rounds in +a process and the median of seven processes: | | degree 1 | degree 4 | |---|---|---| @@ -110,45 +110,47 @@ process and the median of seven processes (2026-10-09): | `GroupBy(r => r.City)` with an average | 4.5 ms | 2.4 ms | | `Where(r => r.Celsius > 20.0).SumAsync(r => r.Celsius)` | 3.9 ms | 1.6 ms | -An aggregate keeps one state per chunk and merges them at the end, so its chunks run side by side: -about twice as fast at four threads, two and a half times for the filtered sum. A scan that hands batches to your loop gains little: it still -delivers them one at a time and in file order (checked, at degree 4), and most of its cost here is -walking 123 blocks. Measure on your own files; the timings vary from run to run on a busy machine, -and the ratios are what to read. +An aggregate keeps one state per chunk and merges them at the end, so its chunks run side by side. It +is about twice as fast at four threads, and two and a half times for the filtered sum. A scan that +hands batches to your loop gains little: it still delivers them one at a time and in file order +(which the sample checks at degree 4), and most of its cost here is walking 123 blocks. Measure on +your own files. Timings vary from run to run on a busy machine, so read the ratios. ## Prefetch -`ScanOptions.Prefetch` is how many batches are decoded ahead of the loop, 1 by default, so that the -decode of the next batch overlaps your work on this one; the scan holds at most that many batches -more. The batch ahead is decoded on the thread pool, so a scan at the default degree of 1 takes one -pool thread beside yours; `Prefetch = 0` keeps it on your thread. On a mapped file with a loop that -works on every value it changed nothing measurable, 6.1 ms at 0, 1 and 2. It -pays when the source has latency to hide, which a remote one does. +`ScanOptions.Prefetch` is the number of batches decoded ahead of the loop, 1 by default, so that +decoding the next batch overlaps your work on the current one. The scan holds at most that many extra +batches. The batch ahead is decoded on the thread pool, so a scan at the default degree of 1 uses one +pool thread beside yours, and `Prefetch = 0` keeps everything on your thread. On a mapped file with a +loop that works on every value it changed nothing measurable, about 3.1 ms at 0, 1 and 2. It pays +off when the source has latency to hide, as a remote one does. ## Local and remote -The loop is the same for every source: there is no synchronous enumeration to choose. A mapped -segment completes at once, so the `await` costs no suspension; a remote one suspends while it -reads. Inside the body the rule is the compiler's: a `Column` does not live across an `await`. Do -the columnar work, then await what you must, then let the loop move on. +The loop is the same for every source, and there is no synchronous enumeration to choose. A read from +a mapped file completes at once, so the `await` costs no suspension, while a remote read suspends +until the data arrives. Inside the body the compiler's rule applies: a `Column` cannot live across +an `await`. Do the columnar work, then await what you need, then let the loop move on. ## Watch out -* **A batch's columns must not cross threads.** Their spans point into buffers the scan reuses. To - hand rows to another thread, take owned batches ([owned-batches.md](owned-batches.md)). -* A `MemoryPool` other than an `AlignedMemoryPool` serves the builders and the owned batches only; - the engine then decodes into `AlignedMemoryPool.Shared`. -* Two hosts in one process that want different parallelism or caches want two sessions, not a +* A batch's columns must not cross threads. Their spans point into buffers the scan reuses. To hand + rows to another thread, take owned batches ([owned-batches.md](owned-batches.md)). +* A `MemoryPool` other than an `AlignedMemoryPool` only serves the builders and the owned batches, + and the engine then decodes into `AlignedMemoryPool.Shared`. +* Two hosts in one process that want different parallelism or caches need two sessions, not a setting changed between calls. -* **In a container, the free space a disk shows is not the limit.** A Kubernetes pod's writable layer - and its `emptyDir` volumes count against its `ephemeral-storage`, which the kubelet enforces by - evicting the pod, while the disk under them still shows room. Without a `ScratchBudget`, a group by - that spills keeps only a tenth of that free space, and a large one can get its pod evicted. Give - its sessions a `ScratchBudget` under the pod's limit, less what the pod writes besides: a query that - would pass it fails with `VortexMemoryException` instead, its files gone. - -The contracts are in [09-contracts.md](../design/09-contracts.md) §1 and §2, and the session in §2 -of [14-public-api.md](../design/14-public-api.md). +* In a container, the free space a disk reports is not the real limit. A Kubernetes pod's writable + layer and its `emptyDir` volumes count against its `ephemeral-storage`, which the kubelet enforces + by evicting the pod while the disk still shows room. Without a `ScratchBudget`, a group by that + spills only keeps a tenth of that free space, and a large one can get its pod evicted. Give its + sessions a `ScratchBudget` under the pod's limit, minus whatever else the pod writes: a query that + would exceed it then fails with `VortexMemoryException` instead, and its files are removed. + +The contracts are described in +[the concurrency design](../design/09-contracts.md#1-concurrency-and-thread-safety) and +[the parallelism design](../design/09-contracts.md#2-parallelism), and the session in +[the public API design](../design/14-public-api.md#2-the-session). ## Run it diff --git a/docs/guide/untyped-files.md b/docs/guide/untyped-files.md index f80633de..aa6b6006 100644 --- a/docs/guide/untyped-files.md +++ b/docs/guide/untyped-files.md @@ -1,7 +1,7 @@ # Untyped files -Read a file whose schema is not known when the program is compiled: look at its schema, name its -columns as strings, and write filters as text. +Read a file whose schema is unknown when the program is compiled: look at its schema, name its columns +as strings, and write filters as text. ```csharp await using VortexFile file = await VortexFile.OpenAsync(path); @@ -34,15 +34,16 @@ asked for Celsius, Day; the batch holds struct{Day: i32, Celsius: f64?}, first d Day >= 900: 100000 rows, 2940000.0 degrees in all ``` -This is the tool path: vxdump, an ad hoc query, a test. It is the one place where columns are named -by strings, because there is no other name for them. When the schema is known, a record is the -better tool ([records.md](records.md)): the compiler then checks every name and every literal. +This is the tool path, used by vxdump, ad hoc queries and tests. It is the one place where columns are +named by strings, because there is no other way to name them. When the schema is known, a record is +the better tool ([records.md](records.md)), since the compiler then checks every name and every +literal. ## The schema `file.Schema` is a `VortexSchema`, read at open, so asking for it costs nothing. It is a list of -`VortexField`, each a name and a `VortexType`, and it prints in one line, which is what a log or an -error wants. A type is a value, and a tree: +`VortexField`, each a name and a `VortexType`, and it prints on one line, which is what a log or an +error message wants. A type is a value, and a tree: | `VortexType` member | for | |---|---| @@ -51,7 +52,7 @@ error wants. A type is a value, and a tree: | `Fields` | a struct's fields | | `ElementType`, `FixedSize` | a list's elements | | `Precision`, `Scale` | a decimal | -| `ExtensionId`, `StorageType`, `ExtensionMetadata` | an extension: a date, a timestamp, a uuid, or your own | +| `ExtensionId`, `StorageType`, `ExtensionMetadata` | an extension: a date, a timestamp, a uuid, or your own type | The visits file, walked field by field: @@ -64,9 +65,9 @@ The visits file, walked field by field: Origin: struct{Country: utf8, City: utf8?} (Struct, 2 fields) ``` -`schema.IndexOf("Origin.City")` finds a nested field by its dotted path and answers its index -**within its own struct**, 1 here; -1 means there is no such field, and nothing throws. A schema is -also written by hand, as a collection expression, and compares by value: +`schema.IndexOf("Origin.City")` finds a nested field by its dotted path and returns its index within +its own struct, 1 here. It returns -1 when there is no such field, and nothing throws. A schema can +also be written by hand as a collection expression, and schemas compare by value: ```csharp VortexSchema declared = [("Day", VortexType.Int32), ("Celsius", VortexType.Float64.Nullable), ("City", VortexType.Utf8)]; @@ -74,19 +75,20 @@ VortexSchema declared = [("Day", VortexType.Int32), ("Celsius", VortexType.Float ## Columns by name -`file.Scan("Celsius", "Day")` reads those columns; `file.Scan()` reads all of them, and a dotted -path reaches a nested field. **A batch holds its columns in the file's order, not in the order you -named them**, because that is the order the projection reads them in: above, `Celsius` was named -first and the batch holds `Day` first. Read them by name, `batch.Column("Day")`, and the order -does not matter. A nested path keeps its parents: `Scan("Origin.City", "Id")` delivers +`file.Scan("Celsius", "Day")` reads those columns, `file.Scan()` reads all of them, and a dotted path +reaches a nested field. A batch holds its columns in the file's order, not in the order you named +them, because that is the order the projection reads them in. Above, `Celsius` was named first and the +batch holds `Day` first. Read columns by name, as in `batch.Column("Day")`, and the order does +not matter. A nested path keeps its parents: `Scan("Origin.City", "Id")` delivers `struct{Id: uuid, Origin: struct{City: utf8?}}`. `Column` takes the .NET type the column maps to, and throws `VortexSchemaException` when it does -not fit. The columns are the same `Column` as the typed path's, borrowed in the same way. +not fit, including when you ask for a non-nullable type over a nullable column. The columns are the +same `Column` as on the typed path, and they are borrowed in the same way. ## Filters as text -Two forms, joined by `and` when both are given: +There are two forms, joined with `and` when you use both: ```csharp VortexExpr parsed = VortexExpr.Parse("Day >= 900 and City = 'Paris' and Celsius is not null"); @@ -98,14 +100,14 @@ parsed: Day >= 900 and City = 'Paris' and Celsius is not null, 12216 rows combined: City in ('Lyon', 'Nice') and not (Celsius < 40), 61426 rows ``` -The grammar is comparisons of a column with a literal or another column, `and`, `or`, `not`, -parentheses, `in (…)`, `is [not] null`, `like`, `starts with` and `contains`; text that is not in it -throws `FormatException`. `&`, `|` and `!` combine parsed filters, and a `VortexExpr` prints back in -the grammar. +The grammar covers comparisons of a column with a literal or with another column, `and`, `or`, `not`, +parentheses, `in (…)`, `is [not] null`, `like`, `starts with` and `contains`. Text outside it throws +`FormatException`. `&`, `|` and `!` combine parsed filters, and a `VortexExpr` prints back in the +same grammar. -The interpolated form, `Where($"Day >= {threshold}")`, is the one to use with values: each hole -reaches the filter typed, not formatted into the text, and is checked against the column it is -compared with. A hole of the wrong type throws at `Where`, naming both: +The interpolated form, `Where($"Day >= {threshold}")`, is the one to use with values. Each hole +reaches the filter as a typed value rather than being formatted into the text, and it is checked +against the column it is compared with. A hole of the wrong type throws at `Where`, naming both: ``` a string hole against an integer column: Column 'Day' is a column of i32, which string does not map to: a string reads a utf8 column. @@ -114,23 +116,23 @@ a column the file does not have: The file has no column 'Month'; its schema is s A number compared with a decimal column is scaled exactly from its digits, and text compared with a date, time, timestamp, uuid or decimal column is parsed in the column's type. Nulls follow -three-valued logic: `not (Celsius < 40)` does not return a row whose `Celsius` is null -([08-semantics.md](../design/08-semantics.md) §3). +three-valued logic, so `not (Celsius < 40)` does not return a row whose `Celsius` is null +([three-valued logic](../design/08-semantics.md#3-three-valued-logic)). ## Watch out -* **A non-nullable type over a nullable column is not refused here.** `batch.Column("Celsius")` - hands the raw values, which mean nothing at a null slot: row 0 of the demonstration file is null - and reads 10.1. Ask for `Column`, whose indexer answers `null`. -* Names are matched exactly, case included; an unknown one throws `VortexSchemaException` when the - scan or the filter is built. +* A non-nullable type over a nullable column is refused. `batch.Column("Celsius")` throws + `VortexSchemaException`: *Column 'Celsius' is f64?, which is nullable; read it as Column, + whose accessors say which rows are null.* Ask for `Column`, whose indexer returns `null`. +* Names are matched exactly, case included, and an unknown name throws `VortexSchemaException` when + the scan or the filter is built. * The tool scan is single-use, like the typed one. Its sinks are the borrowed batches, - `ToBatchesAsync`, `CountAsync`, `AnyAsync` and `ExplainAsync`; rows, aggregates, `OrderBy` and the + `ToBatchesAsync`, `CountAsync`, `AnyAsync` and `ExplainAsync`. Rows, aggregates, `OrderBy` and the key cursor need a record type. * The same scan exists on a dataset ([datasets.md](datasets.md)). -The tool path is §5.8 of [14-public-api.md](../design/14-public-api.md), and [vxdump.md](vxdump.md) -is a program written on it. +The tool path is described in [the public API design](../design/14-public-api.md#58-the-tool-path), +and [vxdump.md](vxdump.md) is a program built on it. ## Run it diff --git a/docs/guide/vxdump.md b/docs/guide/vxdump.md index b41550f5..b60bb023 100644 --- a/docs/guide/vxdump.md +++ b/docs/guide/vxdump.md @@ -1,7 +1,8 @@ # vxdump -Look inside a file: schema, layout tree, encodings, segments, statistics, indexes, and the plan of -a query. +vxdump looks inside a Vortex file and prints its schema, layout tree, encodings, segments, +statistics and indexes. It can also show the plan of a query before running it, check index +checksums and repair a torn append. ``` dotnet build tools/vxdump -c Release @@ -11,10 +12,10 @@ tools/vxdump/bin/Release/net11.0/vxdump readings.vortex --schema --stats ``` file readings.vortex edition core2026.08.0 -bytes 1508212 +bytes 1508316 rows 1000000 tabular yes -identity 5f97a64e440b4eb993aa283d83870dc4 +identity 960e4de9e1994575838d481d8539e1d8 schema Day: i32 @@ -27,35 +28,35 @@ statistics City nulls=0 ``` -The figures on this page are the demonstration file of a million readings the samples write. The -header is printed whatever you ask for: the oldest edition that holds every component the file -uses, its length, its rows, whether its root is a table of columns, and its identity, the sixteen -bytes that name this write; two files with the same identity are the same write, and a file whose -writer set none says `none`. +The file on this page is the million readings the samples write. The header comes first whatever +you ask for. It gives the oldest edition that holds every component the file uses, the length, the +row count, whether the root is a table of columns, and the identity, sixteen bytes that name this +particular write. Two files with the same identity come from the same write, and a file whose writer +set none prints `none`. -vxdump is written against the public surface of `Vorticity` alone, the tool scan, the options, -and the inspection members of `VortexFile`, and nothing else. That makes it the proof that the -surface is complete: a section vxdump could not print from it would be a gap in the surface, not in -vxdump. +vxdump uses nothing but the public surface of `Vorticity`: the tool scan, the options and the +inspection members of `VortexFile`. That makes it a test of the surface. If vxdump could not print +something, the gap would be in the library, not in the tool. ## The options | | | |---|---| -| `--layout` | the layout tree, each flat node with its array encoding; what vxdump prints when no section is asked for | +| `--layout` | the layout tree, each flat node with its array encoding. This is the default when no section is asked for | | `--schema` | the file's columns and their types | | `--encodings` | the array and layout encodings the footer declares, each marked when this build cannot read it | | `--segments` | every segment's offset, length and alignment | -| `--stats` | the file's statistics, per column: nulls, exact bounds, sorted, constant | +| `--stats` | the file's statistics per column: nulls, exact bounds, sorted, constant | | `--all` | the five sections above | -| `--scan` | read every batch, and report batches and rows | +| `--scan` | read every batch and report batches and rows | | `--indexes` | the index directory: kind, column, block length, runs, blocks, entries, listed bytes, layout | | `--explain E` | the plan of a scan filtered by `E`, then the count | -| `--fragment P` | add the index fragment in file `P` to the file's own indexes; repeatable | +| `--fragment P` | add the index fragment in file `P` to the file's own indexes (repeatable) | | `--verify` | check every index region against its checksum, and a fragment's record of the file's hash against the file | | `--repair` | truncate a torn append back to the last whole version, and print nothing else | -An option vxdump does not know is reported on the standard error and ignored. +`--help` prints the same list. An option vxdump does not know is reported on the standard error +and the run exits with 2. ## The layout tree @@ -68,7 +69,7 @@ layout vortex.flat rows=65536 segments=[3] dtype=i32 encoding=vortex.runend(vortex.primitive,vortex.sequence) … vortex.flat rows=16384 segments=[45] dtype=i32 encoding=vortex.runend(vortex.primitive,vortex.primitive) - vortex.flat rows=576 segments=[48] dtype=i32 encoding=vortex.sequence + vortex.flat rows=576 segments=[48] dtype=i32 encoding=vortex.constant vortex.flat rows=123 segments=[51] dtype=struct{vortex.min(): i32?, vortex.max(): i32?, vortex.null_count(): u64?} encoding=vortex.struct(vortex.primitive,vortex.primitive,vortex.primitive) vortex.zoned rows=1000000 zones=123x8192 dtype=f64? vortex.chunked rows=1000000 dtype=f64? @@ -76,16 +77,18 @@ layout … ``` -Read it downwards: a struct of three columns; each column wrapped in a zone map of 123 zones of -8 192 rows, the minimum, maximum and null count of each block, which are what make pruning work; -under it the chunks, and under those the flat nodes that hold the bytes, each naming its encoding -as a nest of transforms. `vortex.runend(vortex.primitive,vortex.sequence)` is a run-end encoding -of plain run ends and values that form a progression: `Day` changes once every thousand rows, one -day after the other, so a chunk of 65 536 rows is 66 or 67 runs of consecutive days. The last -chunk, 576 rows of an arithmetic progression, is a `vortex.sequence`: two numbers. `Celsius` is a -dictionary whose values are ALP-encoded doubles. The zone map of `City` holds text bounds -truncated to 16 bytes, `vortex.bounded_min(16)`, which is how text columns prune. -[how-it-works.md](how-it-works.md) walks through the same tree. +Read it from the top down. The root is a struct of three columns. Each column sits in a zone map of +123 zones of 8 192 rows, which keeps the minimum, maximum and null count of every block, and that is +what pruning works from. Under the zone map come the chunks, and under those the flat nodes that +hold the bytes, each naming its encoding as a nest of transforms. + +`vortex.runend(vortex.primitive,vortex.sequence)` is a run-end encoding whose values form an +arithmetic progression. `Day` changes once every thousand rows, one day after the other, so a chunk +of 65 536 rows is 66 or 67 runs of consecutive days and the values cost two numbers. The last chunk +holds 576 rows of the same day, so it is a `vortex.constant`. `Celsius` is a dictionary whose values +are ALP-encoded doubles. The zone map of `City` keeps text bounds cut to 16 bytes, +`vortex.bounded_min(16)`, which is how text columns prune. [how-it-works.md](how-it-works.md) walks +through the same tree. ## Explaining a query without running it @@ -99,17 +102,20 @@ explain Day >= 900 rows 1000000 blocks 14 live of 123 zone map: 109 pruned, 1 segments / 2124 bytes read - to read 13 segments, 210980 bytes of 1508212 + to read 13 segments, 211004 bytes of 1508316 count tiers exact (100000), 109 pruned, 13 proven, 1 decoded count 100000 ``` -The figures `ExplainAsync` returns in code, for a file you did not write and an expression you are -still writing; only the last line reads data. The count is settled in tiers: 109 blocks pruned by -the zone map, 13 proven whole by it, and one decoded to count its survivors. The expression is the -grammar the tool path parses: comparisons, `and`, `or`, `not`, `in`, `is null`, `like`, with text in -single quotes. `"Day >= 900 and City = 'Paris'"` reads 14 segments and counts 12 502 rows. See -[statistics-and-pruning.md](statistics-and-pruning.md) and [untyped-files.md](untyped-files.md). +These are the figures `ExplainAsync` returns in code, available here for a file you did not write +and an expression you are still working on. Only the last line reads data. The count is settled in +tiers: the zone map prunes 109 blocks, proves 13 others match in full, and one block is decoded to +count its survivors. + +The expression uses the grammar of the tool path: comparisons, `and`, `or`, `not`, `in`, `is null`, +`like`, with text in single quotes. `"Day >= 900 and City = 'Paris'"` reads 14 segments and counts +12 502 rows. [statistics-and-pruning.md](statistics-and-pruning.md) explains the plan and +[untyped-files.md](untyped-files.md) the grammar. ## Indexes, and checking them @@ -127,7 +133,7 @@ explain City = 'Nice' blocks 123 live of 123 zone map: 0 pruned, 1 segments / 2940 bytes read bloom filter: 0 pruned, 10 segments / 5760 bytes read - to read 62 segments, 1502744 bytes of 1514700 + to read 62 segments, 1502768 bytes of 1514786 count tiers 0 pruned, 0 proven, 123 decoded count 124999 @@ -136,37 +142,38 @@ verify file hash no fragment records one ``` -The same rows written with a Bloom filter on `City`. Every block holds every city, so neither the -zone map nor the filter can prune a block for `Nice`, and the plan says so before a byte of data -is read. `--verify` checks each index region against its checksum; a region that does not hold -exits with 6. [indexes.md](indexes.md) says when an index pays. +This is the same data written with a Bloom filter on `City`. Every block holds every city, so +neither the zone map nor the filter can rule out a block for `Nice`, and the plan says so before a +single byte of data is read. `--verify` checks each index region against its checksum, and a region +that fails makes the run exit with 6. [indexes.md](indexes.md) explains when an index pays off. -`--fragment P` adds an index fragment, a file of index runs built for this file elsewhere, to -what the file carries, exactly as `VortexOpenOptions.IndexFragments` does in code; `--verify` then -also checks the fragment's record of the file's hash against the file. +`--fragment P` adds an index fragment, a file of index runs built for this file elsewhere, exactly as +`VortexOpenOptions.IndexFragments` does in code. `--verify` then also checks the fragment's record of +the file's hash against the file. ## A torn file, and repairing it ``` -file torn-copy.vortex +file torn.vortex edition core2026.08.0 -bytes 1508212 +bytes 1508316 rows 1000000 tabular yes -identity 5f97a64e440b4eb993aa283d83870dc4 -torn 128 bytes after the last whole version, of 1508340 (Malformed file: the EOF marker's magic is 0x00000000, expected 'VTXF'.); --repair truncates them +identity 960e4de9e1994575838d481d8539e1d8 +torn 128 bytes after the last whole version, of 1508444 (Malformed file: the EOF marker's magic is 0x00000000, expected 'VTXF'.); --repair truncates them ``` ``` -vxdump torn-copy.vortex --repair -repaired torn-copy.vortex: 1508340 -> 1508212 bytes (128 torn bytes removed) -vxdump torn-copy.vortex --repair -valid torn-copy.vortex: 1508212 bytes, nothing to repair +vxdump torn.vortex --repair +repaired torn.vortex: 1508444 -> 1508316 bytes (128 torn bytes removed) +vxdump torn.vortex --repair +valid torn.vortex: 1508316 bytes, nothing to repair ``` -A file whose last append did not finish opens at its last whole version, and the header says how -much lies after it. `--repair` is `VortexFileRepair.RepairAsync`: it truncates the file to that -version, and does nothing to a file that is whole. See [append-and-repair.md](append-and-repair.md). +When an append did not finish, the file still opens at its last whole version, and the header says +how many bytes lie after it. `--repair` calls `VortexFileRepair.RepairAsync`, which truncates the +file back to that version and leaves a whole file alone. See +[append-and-repair.md](append-and-repair.md). ## Exit codes @@ -177,13 +184,13 @@ version, and does nothing to a file that is whole. See [append-and-repair.md](ap | 3 | the file needs a component this build does not have: `unsupported array: … 'vortex.alz'` | | 4 | the file is malformed: `malformed: Malformed file: the EOF marker's magic is 0x00000000, expected 'VTXF'.` | | 5 | the file could not be read: `io: Could not find file '…/missing.vortex'.` | -| 6 | `--verify` found a region that does not hold its checksum, or a file hash that is not a fragment's record | +| 6 | `--verify` found a region that does not hold its checksum, or a file hash that does not match a fragment's record | -The header and the sections go to the standard output, the diagnostics to the standard error, and -every number is formatted the same whatever the culture, so `vxdump f.vortex | diff -` works and -a script can run `--all` over a directory and look at every file that exits above 0. A component -this build cannot decode is refused only where it is needed: `--encodings` lists it and exits 0, -and `--scan` exits 3. +The header and the sections go to the standard output and the diagnostics to the standard error. +Numbers print the same way whatever the culture, so `vxdump f.vortex | diff -` works, and a script +can run `--all` over a directory and look at every file that exits above 0. A component this build +cannot decode only fails the run where it is needed: `--encodings` lists it and exits 0, while +`--scan` exits 3. ## Run it @@ -192,5 +199,5 @@ dotnet run --project tools/vxdump -c Release -- --all ``` On the shared runtime, `--schema` on the demonstration file takes about 55 ms from start to exit, -most of it the runtime starting. Published ahead of time it needs no runtime at all, which is what -you want from a tool that runs once per file over a directory: see [native-aot.md](native-aot.md). +most of it spent starting the runtime. Published ahead of time it needs no runtime at all, which is +what a tool run once per file over a whole directory wants. See [native-aot.md](native-aot.md). diff --git a/docs/guide/write-a-file.md b/docs/guide/write-a-file.md index d6032cea..c762d594 100644 --- a/docs/guide/write-a-file.md +++ b/docs/guide/write-a-file.md @@ -32,86 +32,85 @@ await using (VortexFileWriter writer = session.CreateWriter(path)) `Reading` is `[VortexRecord] public partial record struct Reading(int Day, double? Celsius, string City)`. The record's schema becomes the file's, and the generator gives the builder one property per member: -`b.Day` is a `ColumnBuilder`, `b.Celsius` a `ColumnBuilder`, `b.City` a -`ColumnBuilder`. `cities` holds the city names as UTF-8 byte arrays, and `Temperatures` fills -a block of values and clears one bit in fifty in `validity`. [records.md](records.md) says what a -record may hold. +`b.Day` is a `ColumnBuilder`, `b.Celsius` a `ColumnBuilder`, and `b.City` a +`ColumnBuilder`. `cities` holds the city names as UTF-8 byte arrays, and `Temperatures` fills a +block of values and clears one bit in fifty in `validity`. [records.md](records.md) lists what a +record can hold. ## What happens -* `CreateWriter` creates the file, replacing whatever was at the path, and binds the record. - `Builder()` returns the writer's one builder, whose buffers come from the session's pool. - Every call returns the same builder. -* `GetSpan(count)` is the `IBufferWriter` pattern. It hands you exactly `count` slots of the buffer - the encoder will read; you fill them and `Advance` commits them. A primitive column is encoded - from those bytes where they lie, without a copy. -* `Append(values, validity)` takes a block of values and its bitmap in one call: bit `i % 64` of - word `i / 64` belongs to value `i`, and a set bit means the value is present. A bitmap with every - bit set allocates nothing. [write-nulls.md](write-nulls.md) covers the other null appends. +* `CreateWriter` starts the file and binds the record. The file replaces whatever is at the + path once it is complete. `Builder()` returns the writer's single builder, whose buffers + come from the session's pool, and every call returns the same builder. +* `GetSpan(count)` follows the `IBufferWriter` pattern. It hands you exactly `count` slots of the + buffer the encoder will read, you fill them, and `Advance` commits them. A primitive column is + encoded from those bytes where they lie, without a copy. +* `Append(values, validity)` takes a block of values and its bitmap in one call. Bit `i % 64` of word + `i / 64` belongs to value `i`, and a set bit means the value is present. A bitmap with every bit set + allocates nothing. [write-nulls.md](write-nulls.md) covers the other ways to append nulls. * `Append` on a text column takes UTF-8 bytes and does not transcode them. [write-text.md](write-text.md) covers strings, formatted values and long values. -* `WriteAsync(b)` takes the builder's rows and clears it for the next block, keeping its buffers. - Once the rows it holds reach a chunk's worth of bytes, it encodes the whole blocks among them. +* `WriteAsync(b)` takes the builder's rows and clears it for the next block, keeping its buffers. Once + the rows it holds reach a chunk's worth of bytes, it encodes the whole blocks among them. * `FlushAsync` hands the encoded chunks to the file. It is the only I/O before completion, and `UnflushedBytes` says how much is waiting for it. -* `CompleteAsync` writes the pending rows as the last block, then the statistics, the zone maps - and the footer, and returns the `WriteReport`. +* `CompleteAsync` writes the pending rows as the last block, then the statistics, the zone maps and + the footer, and returns the `WriteReport`. ## What the writer decided -The report is the only place that says what was chosen. The sample prints it, grouping the -per-chunk encodings: +The report is the only place that says what was chosen. The sample prints it, grouping the encodings +of each chunk: ``` -1000000 rows in blocks of 8192, 17 chunks, 1508212 bytes, in 117 ms - data 1495924, statistics 200, zone maps 8264, indexes 0, footer 3824 +1000000 rows in blocks of 8192, 17 chunks, 1508316 bytes, in 86 ms + data 1495988, statistics 200, zone maps 8264, indexes 0, footer 3864 chunk rows: 65536 x15, 16384, 576 - Day: RunEnd x16, Sequence + Day: RunEnd x16, Constant Celsius: Dictionary x16, Alp City: RunEnd x17 -read back: 1000000 rows, 1508212 bytes on disk, mean 30.0000 °C +read back: 1000000 rows, 1508316 bytes on disk, mean 30.0000 °C ``` `report.Columns` gives one encoding per chunk and per column. A day lasts a thousand rows, so `Day` -is written as runs; the last 576 rows all fall on day 999, a progression of step zero. Four hundred -distinct temperatures make a dictionary, except in the small tail chunk, which took ALP. The city -changes every seven rows, which is still a run. Nothing was sampled or guessed: the writer priced the -candidates on each chunk it held. [11-write-strategy.md](../design/11-write-strategy.md) describes -how. - -`report.Bytes` sums to the file's length. `report.ChunkRows` says how the rows were cut into chunks, -and [blocks-and-chunks.md](blocks-and-chunks.md) explains why a write of one block at a time comes -out as chunks of 65 536 rows: the writer sizes a chunk by its widest column, here the city's -sixteen-byte views, a megabyte of them. `report.Indexes` lists every index the policy asked for: -none here, since the default is `IndexPolicy.None` ([indexes.md](indexes.md)). +is written as runs, and the last 576 rows all fall on day 999, which makes that small chunk a single +constant. Four hundred distinct temperatures make a dictionary, except in the small tail chunk, which +took ALP. The city changes every seven rows, which is still worth storing as runs. Nothing was +sampled or guessed: the writer priced the candidates on each chunk it held. +[11-write-strategy.md](../design/11-write-strategy.md) explains how. + +`report.Bytes` adds up to the file's length. `report.ChunkRows` says how the rows were cut into +chunks, and [blocks-and-chunks.md](blocks-and-chunks.md) explains why writing one block at a time +still produces chunks of 65 536 rows: the writer sizes a chunk by its widest column, here the city's +sixteen-byte views, and puts a megabyte of them in a chunk. `report.Indexes` lists every index the +policy asked for, none here, since the default is `IndexPolicy.None` ([indexes.md](indexes.md)). ## What it costs -The million rows took 117 ms in this run and 1.51 MB on disk; run alone, compiling the writer as it -goes, the sample takes about 200 ms. The -same rows stored without encoding take 20.8 MB ([writer-options.md](writer-options.md)). The -columns are encoded on the thread that calls `WriteAsync`. The builder holds up to a chunk's worth -of rows, a megabyte of the widest column by default, before whole blocks are encoded and released. +The million rows took 86 ms in this run and 1.51 MB on disk. Run on its own, compiling the writer as +it goes, the sample takes about 200 ms. The same rows stored without any encoding take 20.9 MB +([writer-options.md](writer-options.md)). The columns are encoded on the thread that calls +`WriteAsync`. The builder holds up to a chunk's worth of rows, a megabyte of the widest column by +default, before whole blocks are encoded and released. ## Watch out -* **A span does not cross an `await`.** Fill it and call `Advance` before the next `await`: the +* A span does not cross an `await`. Fill it and call `Advance` before the next `await`, since the compiler refuses a `Span` that lives across one. -* **Every column must hold the same number of rows** when `WriteAsync` is called, and no list may be - left open. Otherwise it throws `VortexSchemaException`, for example: *The builder cannot be - written: field 1 of struct{…} holds 7 rows and field 0 holds 8. Complete every row before writing - it.* -* **A builder belongs to its writer.** `WriteAsync` refuses another writer's builder with +* Every column must hold the same number of rows when `WriteAsync` is called, and no list may be left + open. Otherwise it throws `VortexSchemaException`, for example *The builder cannot be written: + field 1 of struct{…} holds 7 rows and field 0 holds 8. Complete every row before writing it.* +* A builder belongs to its writer. `WriteAsync` refuses another writer's builder with `ArgumentException`. A builder is not thread-safe: one thread fills it and writes it. -* **`CompleteAsync` is what makes the file.** A writer disposed without it gives the file up, and a - created file is deleted. [append-and-repair.md](append-and-repair.md) says what that means for an - append. +* `CompleteAsync` is what makes the file. A writer disposed without it gives the file up, and a + created file is deleted. [append-and-repair.md](append-and-repair.md) explains what that means + for an append. * The options that change the file (compression, hints, block size, statistics, edition, metadata, - identity) are in [writer-options.md](writer-options.md). + identity) are covered in [writer-options.md](writer-options.md). -The other ways in: rows instead of columns ([write-rows.md](write-rows.md)), lists and nested -records ([write-lists-and-records.md](write-lists-and-records.md)), a schema known only at run -time ([write-without-a-record.md](write-without-a-record.md)), and a stream or an upload as the +The other ways in are rows instead of columns ([write-rows.md](write-rows.md)), lists and nested +records ([write-lists-and-records.md](write-lists-and-records.md)), a schema known only at run time +([write-without-a-record.md](write-without-a-record.md)), and a stream or an upload as the destination ([stream-to-an-object.md](stream-to-an-object.md)). ## Run it diff --git a/docs/guide/write-lists-and-records.md b/docs/guide/write-lists-and-records.md index 8e81d731..ab94c94c 100644 --- a/docs/guide/write-lists-and-records.md +++ b/docs/guide/write-lists-and-records.md @@ -36,19 +36,24 @@ and `Address(string Country, string? City)`, both `[VortexRecord]`. Read back: ## Lists -`v.Pages` is a `ColumnBuilder>`. `Append(ReadOnlySpan)` appends one list -in one call, and a list refused part way leaves nothing behind. `BeginList` opens a list, -`Elements` is the `ColumnBuilder` of every list's elements, and `EndList` closes it: whatever -was appended to `Elements` in between is one list. A nullable list column also takes `AppendNull`. +`v.Pages` is a `ColumnBuilder>`. `Append(ReadOnlySpan)` appends one list in +one call, and a list refused part way leaves nothing behind. `BeginList` opens a list, `Elements` is +the `ColumnBuilder` of every list's elements, and `EndList` closes it: whatever was appended to +`Elements` in between forms one list. A nullable list column also accepts `AppendNull`. + +A list of records, such as a member declared `ReadOnlyMemory` or `List`, gets a +`ListColumnsBuilder` instead. It works the same way: `BeginList`, then one row of the nested +record per element through `Elements`, a `ColumnsBuilder`, then `EndList`. Writing rows through +`WriteAsync` does all of this for you. ## Nested records `v.Struct
(5)` returns the builder of the nested record held by member 5, and the generated -`v.Origin` is the same builder. Its columns fill like any other; each row of it is one row of +`v.Origin` is the same builder. Its columns fill like any others. Each of its rows is one row of `Visit`, so every field must get a value per row. -A nullable nested record takes `AppendNull()` on its builder: the row is null and its fields get -placeholder values. With `[VortexRecord] public partial record struct Trip(int Id, Address? Destination)`: +A nullable nested record accepts `AppendNull()` on its builder. The row is then null and its fields +get placeholder values. With `[VortexRecord] public partial record struct Trip(int Id, Address? Destination)`: ```csharp ColumnsBuilder t = writer.Builder(); @@ -67,43 +72,43 @@ t.Destination.AppendNull(); // a null record: its ## What it costs -The sample writes a hundred thousand visits twice. Column by column, with the uuids and timestamps -appended in bulk (`Append(ReadOnlySpan)`, `Append(ReadOnlySpan)`), the durations -through `GetSpan` and the rest row by row: +The sample writes a hundred thousand visits twice. The first time it goes column by column, with the +uuids and timestamps appended in bulk (`Append(ReadOnlySpan)`, `Append(ReadOnlySpan)`), +the durations through `GetSpan`, and the rest row by row: ``` -100000 visits column by column: 2169236 bytes, best of three 36 ms +100000 visits column by column: 2139788 bytes, best of three 36 ms Id: Canonical x5 StartedAt: Sequence x5 DurationMs: Sequence x4, BitPacked Referrer: Dictionary x5 Pages: BitPacked x5 Origin: {Country: Dictionary, City: Dictionary} x5 -the same visits as rows: 2169236 bytes, best of three 24 ms -read back: 100000 visits, 100000 lists holding 200000 pages, 9091 origins without a city, 2169236 bytes +the same visits as rows: 2168748 bytes, best of three 26 ms +read back: 100000 visits, 100000 lists holding 200000 pages, 9091 origins without a city, 2168748 bytes ``` -The report reads each chunk's encoding from what was written: a list by its elements, a timestamp -or a uuid by its storage, and a nested record field by field. +The report reads each chunk's encoding from what was written: a list by its elements, a timestamp or +a uuid by its storage, and a nested record field by field. -The rows won here, and the reason is worth knowing: the generated `WriteRows` fills one column at a +The rows won here, and the reason is worth knowing. The generated `WriteRows` fills one column at a time over the whole span, while the sample's loop appends the referrer, the list and both origin -fields of a row before moving to the next. Filling one column at a time is what paid. +fields of a row before moving on to the next. Filling one column at a time is what paid off. The two +files differ by about 1 % in size. ## Watch out -* **A list left open makes `WriteAsync` throw** `VortexSchemaException`: *…: a list of list(i32) is - open. Complete every row before writing it.* -* **A hint on a timestamp or a uuid column does not reach its storage.** The hint pins the - extension, and the chooser still picks the storage's encoding: a `Canonical` hint on `StartedAt` - is still reported `Sequence`. `vxdump --layout` shows the whole encoding tree - ([vxdump.md](vxdump.md)). -* **A list of records cannot be written**: the generator refuses such a member with VX1005, since - no typed column reads it back ([records.md](records.md)). -* `StartedAt.Append(DateTime)` converts a local time for a UTC column; `Id.Append(Guid)` stores the - uuid in network order. A `Guid` column needs the edition that carries `vortex.uuid`, the default - ([editions.md](editions.md)). -* Reading nested records and lists is [nested-records.md](nested-records.md). +* A list left open makes `WriteAsync` throw `VortexSchemaException`: *…: a list of list(i32) is open. + Complete every row before writing it.* +* A hint on a timestamp or a uuid column does not reach its storage. The hint pins the extension, and + the chooser still picks the storage's encoding, so a `Canonical` hint on `StartedAt` is still + reported as `Sequence`. `vxdump --layout` shows the whole encoding tree ([vxdump.md](vxdump.md)). +* A list of nullable records and a list of lists of records are refused by the generator with VX1005 + ([records.md](records.md)). Hold the inner lists in a record instead. +* `StartedAt.Append(DateTime)` converts a local time for a UTC column, and `Id.Append(Guid)` stores the + uuid in network byte order. A `Guid` column needs an edition that carries `vortex.uuid`, which the + default does ([editions.md](editions.md)). +* Reading nested records and lists back is covered in [nested-records.md](nested-records.md). ## Run it diff --git a/docs/guide/write-nulls.md b/docs/guide/write-nulls.md index 956c22c1..aad70f6b 100644 --- a/docs/guide/write-nulls.md +++ b/docs/guide/write-nulls.md @@ -1,7 +1,7 @@ # Write nulls -Append a missing value one at a time, in bulk with a bitmap, or many at once, and pay for the -bitmap only when there is a null to record. +Append a missing value one at a time, in bulk with a bitmap, or many at once, and pay for the bitmap +only when there is a null to record. ```csharp ColumnsBuilder b = writer.Builder(); @@ -17,7 +17,7 @@ b.Station.AppendNulls(7); // a nullable text co ``` `Sample` is `[VortexRecord] public partial record struct Sample(int Day, double? Celsius, string? Station)`, -`values` is `[18.5, 19.0, 19.5, 20.0, 20.5]` and `validity` is `[0b11011]`: the third value is +`values` is `[18.5, 19.0, 19.5, 20.0, 20.5]` and `validity` is `[0b11011]`, so the third value is missing. The eight temperatures read back as: ``` @@ -26,12 +26,12 @@ missing. The eight temperatures read back as: ## What happens -* `ColumnBuilder` takes `Append(double)`, `Append(double?)`, `AppendNull()`, +* `ColumnBuilder` accepts `Append(double)`, `Append(double?)`, `AppendNull()`, `AppendNulls(count)` and the bulk `Append(values, validity)`. The text builder has the same null appends ([write-text.md](write-text.md)). -* **The bitmap is lazy.** A nullable column has none until its first null. A bulk append whose bitmap - has every bit set, for the rows it covers, allocates nothing either. -* On disk the validity is decided per chunk: a chunk without a null carries none. +* The bitmap is created lazily. A nullable column has none until its first null, and a bulk append + whose bitmap has every bit set for the rows it covers allocates nothing either. +* On disk, validity is decided per chunk, so a chunk without a null carries no bitmap. ## What it costs @@ -39,33 +39,34 @@ A million rows, with `Celsius` written five ways: | `Celsius` | bytes | null count in the statistics | batches carrying a bitmap | |---|---|---|---| -| no null, `Append(values)` | 1 504 820 | 0 | 0 of 17 | -| no null, `Append(values, validity)` with every bit set | 1 504 820 | 0 | 0 of 17 | -| one null, at row 500 000 | 1 504 948 | 1 | 1 of 17 | -| one row in fifty | 1 508 212 | 20 000 | 17 of 17 | -| every row, `AppendNulls` | 358 980 | 1 000 000 | 17 of 17 | +| no null, `Append(values)` | 1 504 924 | 0 | 0 of 17 | +| no null, `Append(values, validity)` with every bit set | 1 504 924 | 0 | 0 of 17 | +| one null, at row 500 000 | 1 505 052 | 1 | 1 of 17 | +| one row in fifty | 1 508 316 | 20 000 | 17 of 17 | +| every row, `AppendNulls` | 356 908 | 1 000 000 | 17 of 17 | -A nullable column with no null costs exactly what a column without nulls does. One null costs 128 -bytes, and only the chunk that holds it carries a bitmap, read here as a single batch. A column -that is entirely null is written as runs, and most of the remaining 359 KB is the other two columns. +A nullable column with no null costs exactly what a non-nullable column costs. One null costs 128 +bytes, and only the chunk that holds it carries a bitmap, read here as a single batch. A column that +is entirely null is written as one constant per chunk, and most of the remaining 357 KB belongs to the +other two columns. -The null count comes back three ways: from the file's statistics (`file.Statistics[1].TryGetNullCount`), -per batch from `Column.NullCount`, and as `IsAllValid` on a batch without a bitmap. Reading -nullable columns is [nullable-columns.md](nullable-columns.md). +The null count comes back three ways: from the file's statistics +(`file.Statistics[1].TryGetNullCount`), per batch from `Column.NullCount`, and as `IsAllValid` on a +batch without a bitmap. Reading nullable columns is covered in [nullable-columns.md](nullable-columns.md). ## Watch out -* **A non-nullable column refuses a nullable builder.** Asking for `ColumnBuilder` over `Day` - throws `VortexSchemaException`: *Column 'Day' is i32, which is not nullable; ask for a builder of - int instead of int?.* The other way round is fine: a `ColumnBuilder` over a nullable - column writes it all valid. -* **The value under a null is not a value.** Whatever the slot holds is written, and a reader goes by - the bitmap. -* **The bitmap must cover the values.** `Append(values, validity)` needs at least one bit per value, - and throws `ArgumentException` otherwise; bits past the last value are ignored. -* **`AppendNull` on a non-nullable text column throws** `VortexSchemaException`; the method exists on - every text builder because nullability is not part of the builder's type there. -* A null nested record is [write-lists-and-records.md](write-lists-and-records.md). +* A non-nullable column refuses a nullable builder. Asking for a `ColumnBuilder` over `Day` + throws `VortexSchemaException`: *Column 'Day' is i32, which is not nullable; ask for a builder of int + instead of int?.* The other way round is fine: a `ColumnBuilder` over a nullable column + writes every row as valid. +* The value under a null is not a value. Whatever the slot holds is written, and a reader goes by the + bitmap. +* The bitmap must cover the values. `Append(values, validity)` needs at least one bit per value and + throws `ArgumentException` otherwise. Bits past the last value are ignored. +* `AppendNull` on a non-nullable text column throws `VortexSchemaException`. The method exists on + every text builder because nullability is not part of the builder's type for text. +* Null nested records are covered in [write-lists-and-records.md](write-lists-and-records.md). ## Run it diff --git a/docs/guide/write-rows.md b/docs/guide/write-rows.md index 1ab54df2..7a283ced 100644 --- a/docs/guide/write-rows.md +++ b/docs/guide/write-rows.md @@ -15,52 +15,52 @@ await streaming.WriteAsync(ReadingsAsync(Rows, ct), ct); report = await streaming.CompleteAsync(ct); ``` -`readings` is a `Reading[]` of a million rows; `ReadingsAsync` is an `async IAsyncEnumerable` +`readings` is a `Reading[]` of a million rows. `ReadingsAsync` is an `async IAsyncEnumerable` that yields the same rows and awaits every ten thousand, as a source reading pages would. ## What happens -* **A span** goes through the record's generated `WriteRows`, which copies it into the writer's - builder one column at a time: the numbers through `GetSpan`, the nullable members through - `Append(T?)`, the strings transcoded to UTF-8. The rows are read before `WriteAsync` returns, so - the array can be reused at once. The builder then behaves as in [write-a-file.md](write-a-file.md). -* **A stream** is gathered into groups of `BlockRows` rows, 8 192 by default, in a pooled array; each - full group is written, and the writer flushes by itself once 8 MiB are waiting (four chunk targets, - when that is more). The last partial group is written when the stream ends. The caller counts - nothing and calls nothing but `CompleteAsync`. +A span goes through the record's generated `WriteRows`, which copies it into the writer's builder +one column at a time: numbers through `GetSpan`, nullable members through `Append(T?)`, and strings +transcoded to UTF-8. The rows are read before `WriteAsync` returns, so you can reuse the array +immediately. From there the builder behaves as in [write-a-file.md](write-a-file.md). + +A stream is gathered into groups of `BlockRows` rows, 8 192 by default, in a pooled array, and each +full group is written. The writer flushes on its own once 8 MiB are waiting, or four chunk targets +when that is more. The last partial group is written when the stream ends. You count nothing and call +nothing but `CompleteAsync`. ## What it costs ``` -a span of 1000000 rows: 1508212 bytes, best of three 79 ms +a span of 1000000 rows: 1508316 bytes, best of three 63 ms chunk rows: 65536 x15, 16384, 576 -a stream of 1000000 rows: 1508212 bytes, best of three 92 ms +a stream of 1000000 rows: 1508316 bytes, best of three 97 ms chunk rows: 65536 x15, 16384, 576 ``` -Rows cost a field copy per row, and a `string` member is transcoded once per row; a list member is -copied element by element, and a nested record goes through a pooled array. The generated code -touches one column at a time, which is also why it can beat a hand-written loop that appends every -column of a row before moving to the next ([write-lists-and-records.md](write-lists-and-records.md) +Rows cost a field copy per row, and a `string` member is transcoded once per row. A list member is +copied element by element, and a nested record goes through a pooled array. The generated code works +on one column at a time, which is also why it can beat a hand-written loop that appends every column +of a row before moving on to the next ([write-lists-and-records.md](write-lists-and-records.md) measures one). -The stream costs the enumeration and nothing else: both calls cut the rows into the same chunks, and -the same bytes, because the writer sizes a chunk by the width of the rows it holds and not by how they -arrived. [blocks-and-chunks.md](blocks-and-chunks.md) shows where that width comes from. +The stream costs the enumeration and nothing else. Both calls cut the rows into the same chunks and +produce the same bytes, because the writer sizes a chunk by the width of the rows it holds, not by how +they arrived. [blocks-and-chunks.md](blocks-and-chunks.md) shows where that width comes from. ## Watch out -* **Rows are a convenience, not the fast path for wide numeric data.** Filling a column with - `GetSpan` or a bulk `Append` skips the per-row copy; [write-a-file.md](write-a-file.md) is that - path. -* **Rows appended to the builder and not yet written are written with these.** The span form shares - the writer's builder. -* **A write that fails part way leaves nothing behind.** If `WriteRows` throws, the builder is cut back - to where it was before the call. -* **The record must cover the file.** `CreateWriter` makes the file's schema the record's. - Over a file whose schema came from elsewhere (an append, an untyped writer), the record's members - are bound to columns by name, and a member with no column throws `VortexSchemaException`. -* Reading the rows back is [read-rows.md](read-rows.md). +* Rows are a convenience, not the fast path for wide numeric data. Filling a column with `GetSpan` or + a bulk `Append` skips the copy per row, and [write-a-file.md](write-a-file.md) shows that path. +* Rows appended to the builder and not yet written go out with these, since the span form shares the + writer's builder. +* A write that fails part way leaves nothing behind. If `WriteRows` throws, the builder is cut back to + where it was before the call. +* The record must cover the file. `CreateWriter` makes the record's schema the file's. When + the schema came from elsewhere (an append, an untyped writer), the record's members are bound to + columns by name, and a member with no column throws `VortexSchemaException`. +* Reading the rows back is covered in [read-rows.md](read-rows.md). ## Run it diff --git a/docs/guide/write-text.md b/docs/guide/write-text.md index 597d56e5..37bc01d5 100644 --- a/docs/guide/write-text.md +++ b/docs/guide/write-text.md @@ -1,7 +1,7 @@ # Write text Append text without allocating a string: UTF-8 bytes as they are, a `string` transcoded once, a -number or a date formatted straight into the column, and a long value written in place. +number or a date formatted straight into the column, or a long value written in place. ```csharp ColumnsBuilder b = writer.Builder(); @@ -32,7 +32,7 @@ b.Note.Commit(length); ``` `Entry` is `[VortexRecord] public partial record struct Entry(DateTime StartedAt, string City, string? Note)`. -`b.City` is a `ColumnBuilder` and `b.Note` a `ColumnBuilder`: the nullable one takes +`b.City` is a `ColumnBuilder` and `b.Note` a `ColumnBuilder`. The nullable one takes the same appends, plus `AppendNull` and `AppendNulls(count)`. Read back, the rows are: ``` @@ -47,14 +47,14 @@ the same appends, plus `AppendNull` and `AppendNulls(count)`. Read back, the row | append | what it does | |---|---| -| `Append(ReadOnlySpan)` | copies the bytes as they are, after checking they are valid UTF-8 | -| `Append(ReadOnlySpan)` | transcodes with `Utf8.FromUtf16`, vectorised, straight into the column's buffer; a lone surrogate becomes U+FFFD. A `string` converts to it | -| `Append(TValue)` | formats an `IUtf8SpanFormattable` (a number, a date, a `Guid`) with `TryFormat` into the column's buffer, with the default format and the invariant culture | -| `GetSpan(n)`, `Commit(length)` | hands out exactly `n` bytes of the column's buffer for one value; `Commit` appends the first `length` of them, after the UTF-8 check | +| `Append(ReadOnlySpan)` | copies the bytes as they are, after checking that they are valid UTF-8 | +| `Append(ReadOnlySpan)` | transcodes with the vectorised `Utf8.FromUtf16`, straight into the column's buffer. A lone surrogate becomes U+FFFD, and a `string` converts to this overload | +| `Append(TValue)` | formats an `IUtf8SpanFormattable` (a number, a date, a `Guid`) into the column's buffer with `TryFormat`, using the default format and the invariant culture | +| `GetSpan(n)`, `Commit(length)` | hands out exactly `n` bytes of the column's buffer for one value. `Commit` appends the first `length` of them, after the UTF-8 check | -A value of twelve bytes or fewer is stored inside its sixteen-byte view; a longer one goes to the -data buffer, with its first four bytes in the view. The builder does both, and the caller never sees -a view. +A value of twelve bytes or fewer is stored inside its sixteen-byte view. A longer one goes to the data +buffer, with its first four bytes kept in the view. The builder handles both, and you never see a +view. ## What it costs @@ -65,24 +65,25 @@ per value, 1000000 appends: UTF-8 bytes 4.4 ns, a string 6.2 ns, an int formatte ``` Keeping the names as UTF-8 byte arrays saves the transcoding, about 30 % of the append here. -Formatting a number in place costs about what copying its bytes would, and allocates nothing. +Formatting a number in place costs about as much as copying its bytes, and allocates nothing. ## Watch out -* **The formatted append uses the default format.** Under the invariant culture a `DateOnly` comes - out as `09/22/2026`, not as an ISO date. For another format, write it yourself through - `GetSpan` and `TryFormat`, as the last row does. -* **A text column refuses bytes that are not UTF-8**, with `VortexSchemaException`: *The column is +* The formatted append uses the default format. Under the invariant culture, a `DateOnly` comes out + as `09/22/2026`, not as an ISO date. For another format, write it yourself through `GetSpan` and + `TryFormat`, as the last row does. +* A text column refuses bytes that are not valid UTF-8 with `VortexSchemaException`: *The column is utf8: the bytes appended to it must be valid UTF-8.* Nothing is appended. A binary column - (`ColumnBuilder>`) takes any bytes. -* **`GetSpan(n)` gives exactly `n` bytes.** Ask for the most the value can take: + (`ColumnBuilder>`) accepts any bytes. +* `GetSpan(n)` gives exactly `n` bytes, so ask for the most the value can take: `Encoding.UTF8.GetMaxByteCount(text.Length)` for a string. -* **`AppendNull` exists on every text builder**, and throws `VortexSchemaException` on a column that - is not nullable. -* A `char` member is refused: it is not a type the format has ([records.md](records.md)). -* A text column's zones carry string bounds by default, so equality and prefix filters on it prune: - `StringBoundBytes` in [writer-options.md](writer-options.md). -* Reading text back without a string is [text-columns.md](text-columns.md). +* `AppendNull` exists on every text builder, and throws `VortexSchemaException` on a column that is + not nullable. +* A `char` member is not text. It maps to a `u16` column holding the UTF-16 code unit + ([records.md](records.md)), so use `string` for text. +* A text column's zones carry string bounds by default, so equality and prefix filters on it prune + (`StringBoundBytes` in [writer-options.md](writer-options.md)). +* Reading text back without allocating strings is covered in [text-columns.md](text-columns.md). ## Run it diff --git a/docs/guide/write-without-a-record.md b/docs/guide/write-without-a-record.md index db07042c..587f39e8 100644 --- a/docs/guide/write-without-a-record.md +++ b/docs/guide/write-without-a-record.md @@ -1,6 +1,6 @@ # Write without a record -Write a file whose columns are known only when the program runs: a schema built as a value, and +Write a file whose columns are only known when the program runs: a schema built as a value, and column builders asked for by position or by name. ```csharp @@ -26,45 +26,46 @@ await using (VortexFileWriter writer = session.CreateWriter(path, schema)) ``` ``` -100000 rows, 222388 bytes; id Sequence x7; payload Zstd x7; origin {country: Dictionary} x7 +100000 rows, 222396 bytes; id Sequence x7; payload Zstd x7; origin {country: Dictionary} x7 schema struct{id: i64, payload: binary?, origin: struct{country: utf8}} read back by name: ids summing to 104999950000, 33334 null payloads, 2666523 payload bytes ``` ## What happens -* **A schema is a value**: a collection expression of `(name, VortexType)` pairs. `VortexType` has a - member per type the format has (`Int64`, `Utf8`, `Binary`, `Decimal(p, s)`, `List(element)`, - `Struct(fields)`, `Timestamp(unit, zone)`, `Uuid` and the rest), and `.Nullable` makes any of them - nullable. -* **`Builder()` is the untyped view of the writer's builder**, the same rows `Builder()` would - show through a record. `Column(index)` and `Column(name)` hand out a column's builder; `T` - is the .NET type the column maps to: `long` for an i64, `ReadOnlyMemory?` for a nullable - binary, `string` for text. The type is checked once, when the builder is asked for. -* **A struct column is a builder of its own**: `Struct(index)` returns a `ColumnsBuilder` over its +* A schema is a value: a collection expression of `(name, VortexType)` pairs. `VortexType` has a + member for every type the format has (`Int64`, `Utf8`, `Binary`, `Decimal(p, s)`, `List(element)`, + `Struct(fields)`, `Map(key, value)`, `Timestamp(unit, zone)`, `Uuid` and the rest), and `.Nullable` + makes any of them nullable. +* `Builder()` is the untyped view of the writer's builder, the same rows `Builder()` shows through + a record. `Column(index)` and `Column(name)` hand out a column's builder, where `T` is the .NET + type the column maps to: `long` for an i64, `ReadOnlyMemory?` for a nullable binary column, + `string` for text. The type is checked once, when the builder is requested. +* A struct column has a builder of its own. `Struct(index)` returns a `ColumnsBuilder` over its fields, whose rows count as the parent's. -* From there everything is as with a record: the appends of [write-a-file.md](write-a-file.md), +* From there everything works as with a record: the appends of [write-a-file.md](write-a-file.md), [write-text.md](write-text.md) and [write-nulls.md](write-nulls.md), then `WriteAsync` and `CompleteAsync`. -Reading such a file back uses the same names: `file.Scan("id", "payload")` yields `BatchView`s whose +Reading such a file back uses the same names. `file.Scan("id", "payload")` yields `BatchView`s whose `Column("id")` and `Column?>("payload")` are the typed columns ([untyped-files.md](untyped-files.md)). ## Watch out -* **A type that does not fit throws when the builder is asked for**, with `VortexSchemaException`: - *Column 'id' is a column of i64, which int does not map to: a int maps to a i32 column exactly; - convert it before a write or after a read.* There is no conversion on the way in. -* **Names match exactly.** `Column(name)` compares ordinally, and a name the schema does not have +* A type that does not fit throws `VortexSchemaException` when the builder is requested: *Column 'id' + is a column of i64, which int does not map to: a int maps to a i32 column exactly; convert it before + a write or after a read.* There is no conversion on the way in. +* Names are matched exactly. `Column(name)` compares ordinally, and a name the schema does not have throws: *The builder has no column 'identifier'; its schema is struct{…}.* A record binds with a - case-insensitive fallback; the untyped builder does not. -* **A binary value can be written in place.** `ColumnBuilder?>` offers - `GetSpan(n)` and `Commit(length)` beside `Append(bytes)`, and `AppendNull()` or - `AppendNulls(count)` for the rows without a payload, as the text builders do. -* The report reads a struct column field by field, `{country: Dictionary}`, the way it reads a list - by its elements ([write-lists-and-records.md](write-lists-and-records.md)). -* The mapping between types and .NET types is [07-dotnet-mapping.md](../design/07-dotnet-mapping.md). + case-insensitive fallback, but the untyped builder does not. +* A binary value can be written in place. `ColumnBuilder?>` offers `GetSpan(n)` + and `Commit(length)` next to `Append(bytes)`, plus `AppendNull()` and `AppendNulls(count)` for rows + without a payload, as the text builders do. +* The report reads a struct column field by field, as in `{country: Dictionary}`, the same way it + reads a list by its elements ([write-lists-and-records.md](write-lists-and-records.md)). +* The mapping between column types and .NET types is described in + [07-dotnet-mapping.md](../design/07-dotnet-mapping.md). ## Run it diff --git a/docs/guide/writer-options.md b/docs/guide/writer-options.md index 0d8ebeb6..416873ac 100644 --- a/docs/guide/writer-options.md +++ b/docs/guide/writer-options.md @@ -1,7 +1,7 @@ # Writer options -Choose how the writer encodes, what it records beside the data, and what the file carries, then -read in the report what it did. +Choose how the writer encodes, what it records beside the data, and what the file carries, then read +in the report what it actually did. ```csharp VortexWriteOptions options = new() @@ -30,103 +30,104 @@ identity 0199f0c4-7d2a-7c3e-9a51-3f6b2c1d4e5f, edition core2026.08.0, 564148 byt written again with the same identity: the same bytes; without one: different bytes ``` -`VortexWriteOptions` is a record with `init` properties: set what you mean to change, and `with` -makes a variant. The options are checked at `CreateWriter`: a hint, an index or a metadata entry that -does not fit throws `ArgumentException` there, before a byte is written. +`VortexWriteOptions` is a record with `init` properties. Set what you want to change, and use `with` +to make a variant. The options are checked by `CreateWriter`: a hint, an index or a metadata entry +that does not fit throws `ArgumentException` there, before a single byte is written. | option | default | what it changes | |---|---|---| -| `BlockRows` | 8 192 | rows per block: the unit of pruning, of a take and of a filtered scan's batch where the zone maps do not prove it whole ([blocks-and-chunks.md](blocks-and-chunks.md)) | -| `ChunkTargetBytes` | 0, the writer's | the bytes gathered before whole blocks are sealed into a chunk; by default a megabyte of the widest column ([blocks-and-chunks.md](blocks-and-chunks.md)) | -| `ColumnChunkTargetBytes` | empty | a chunk target of its own, by top-level column, for a column that wants larger chunks than the file's: its chunks span whole chunks of the file ([blocks-and-chunks.md](blocks-and-chunks.md)) | -| `Compression` | `Auto` | what the encoder optimises for: `Auto`, `Fastest`, `Smallest`, `None` | +| `BlockRows` | 8 192 | rows per block: the unit of pruning, of a take, and of a filtered scan's batch where the zone maps do not prove it whole ([blocks-and-chunks.md](blocks-and-chunks.md)) | +| `ChunkTargetBytes` | 0, the writer decides | the bytes gathered before whole blocks are sealed into a chunk. By default, a megabyte of the widest column ([blocks-and-chunks.md](blocks-and-chunks.md)) | +| `ColumnChunkTargetBytes` | empty | a chunk target of its own for a top-level column that wants larger chunks than the file's. Its chunks span whole chunks of the file ([blocks-and-chunks.md](blocks-and-chunks.md)) | +| `Compression` | `Auto` | what the encoder optimises for: `Auto`, `Fastest`, `Smallest` or `None` | | `Hints` | empty | an encoding to try first, by column path | -| `DegreeOfParallelism` | 0, the session's | the threads the writer summarizes its columns and compresses a column's zstd frames on; the file is the same bytes whatever the degree ([threads.md](threads.md)) | +| `DegreeOfParallelism` | 0, the session's | the threads the writer summarizes its columns and compresses zstd frames on. The file is byte for byte the same whatever the degree ([threads.md](threads.md)) | | `TargetEdition` | `VortexEditions.Default` | the edition every component must belong to ([editions.md](editions.md)) | -| `Statistics` | `true` | the per-column minimum, maximum, sum, null count and order the open reads back | -| `StringBoundBytes` | 16 | the length of the bounds a text or binary column's zones carry; 0 for none | +| `Statistics` | `true` | the per-column statistics the open reads back: null count, order, and the minimum and maximum of a numeric column | +| `StringBoundBytes` | 16 | the length of the bounds a text or binary column's zones carry, 0 for none | | `Indexes` | `IndexPolicy.None` | the indexes to build ([indexes.md](indexes.md)) | -| `Identity` | drawn per write | the sixteen bytes naming this version of the file | +| `Identity` | drawn per write | the sixteen bytes that name this version of the file | | `Metadata` | empty | small values by key, read back through `file.Metadata` | -| `Durable` | `false` | a file created or appended to is on the device when `CompleteAsync` returns, everything its postscript names before the postscript: two flushes to the device per completion | +| `Durable` | `false` | makes a created or appended file reach the device before `CompleteAsync` returns, with everything the postscript names written before the postscript itself. It costs two flushes to the device per completion | ## Compression -The same million rows, under each profile. The write is the best of three; the decode forces every -column of every batch to its plain form with `Canonical()`: +The same million rows under each profile. The write time is the best of three, and the decode forces +every column of every batch to its plain form with `Canonical()`: | profile | bytes | written in | decoded in | encodings | |---|---|---|---|---| -| `Auto` | 1 508 316 | 34 ms | 1 ms | `Day` runs, `Celsius` a dictionary, `City` runs | -| `Fastest` | 1 508 316 | 33 ms | 1 ms | the same | -| `Smallest` | 243 028 | 66 ms | 2 ms | `Day` runs, `Celsius` and `City` zstd, pco on a chunk of `Day` and on `Celsius`'s tail | -| `None` | 20 929 828 | 28 ms | 4 ms | every column `Canonical`, the plain form | +| `Auto` | 1 508 316 | 34 ms | 1 ms | `Day` as runs, `Celsius` as a dictionary, `City` as runs | +| `Fastest` | 1 508 316 | 34 ms | 1 ms | the same | +| `Smallest` | 243 028 | 87 ms | 7 ms | `Day` as runs, `Celsius` and `City` as zstd, and pco on a chunk of `Day` and on the tail of `Celsius` | +| `None` | 20 929 828 | 25 ms | 4 ms | every column in its plain form | Encoding is what makes the file fourteen times smaller than its plain form. `Auto` prices each -column's size and decode speed together; `Fastest` spends less time choosing and builds no index, -and on this data chose the same encodings; `Smallest` prices bytes alone, tries zstd, pco, FSST, -ALP and ALP-RD on every chunk -- and on every array an encoding makes of it, an ALP's integers or -a run's values -- and keeps the smallest: zstd for `Celsius` and `City`, a file six times smaller -than `Auto`'s, which a full decode pays for with a millisecond more. +column's size and decode speed together. `Fastest` spends less time choosing and builds no index, and +on this data it chose the same encodings. `Smallest` prices bytes alone. It tries zstd, pco, FSST, +ALP and ALP-RD on every chunk, and on every array an encoding makes of it (an ALP's integers, a run's +values), and keeps the smallest: zstd for `Celsius` and `City`, a file six times smaller than +`Auto`'s, which a full decode pays for with a few milliseconds more. [choose-encodings.md](choose-encodings.md) measures what each choice costs on ten million rows of -other shapes, and where it turns. +other shapes, and where the best choice changes. ## Hints -A hint names the encoding a column should try first, by path (dotted for a nested field). One hint -at a time, the rest left to the chooser: +A hint names the encoding a column should try first, by path (dotted for a nested field). One hint at +a time, the rest left to the chooser: | hint | bytes | written as | |---|---|---| -| none | 1 508 316 | `Celsius` a dictionary, `City` runs | +| none | 1 508 316 | `Celsius` as a dictionary, `City` as runs | | `City` as `Dictionary` | 1 551 708 | a dictionary | | `City` as `Zstd` | 1 191 484 | zstd (runs on the 576-row tail) | | `City` as `Fsst` | 4 919 828 | FSST | -| `City` as `Canonical` | 9 959 068 | `Canonical`, the plain form | +| `City` as `Canonical` | 9 959 068 | the plain form | | `Celsius` as `Alp` | 1 611 740 | ALP | | `Celsius` as `Zstd` | 564 092 | zstd (ALP on the 576-row tail) | -| `Celsius` as `BitPacked` | 1 508 316 | a dictionary: bit-packing does not apply to floats | -| `Celsius` as `Canonical` | 8 484 100 | `Canonical`, the plain form | -| `Celsius` as `Canonical`, under `Smallest` | 8 164 052 | `Canonical`; `Day` and `City` as `Smallest` writes them | +| `Celsius` as `BitPacked` | 1 508 316 | a dictionary, since bit-packing does not apply to floats | +| `Celsius` as `Canonical` | 8 484 100 | the plain form | +| `Celsius` as `Canonical`, under `Smallest` | 8 164 052 | the plain form, with `Day` and `City` as `Smallest` writes them | A hint is a preference, and the report is where you find out whether it held: -* **A column that is a progression is written as one whatever the hint says**, and so is a chunk - of one value, or of nulls only, written as a constant: nothing costs less per row. Runs are not: - a hint comes before them, since runs cost to decode, and the bitmap of a column one percent true - reads twelve times faster than its runs. -* **Some shapes need no hint.** A decimal whose values fit 64 bits is written as those integers - (`DecimalByteParts`), which then take the integer schemes; a timestamp coarser than its unit -- - instants to the second in microseconds, dates at midnight -- as its days, seconds and subseconds - (`DateTimeParts`); a column nine rows in ten null as its valid rows at their positions - (`Sparse`). A hint for another scheme still holds such a column, as the caller asked. -* **`OnPair` is the hint for text a dictionary of repeated substrings spells** -- URLs, paths, - log lines -- decoded as fast as FSST and, on such text, a fifth to two fifths smaller. `Auto` - takes it where it comes a tenth under FSST and under zstd. -* **`Pco` is the hint for numbers of sixteen bits or more** that bit-packing leaves wide: - timestamps with jitter, counters, skewed amounts, multiples of a common unit. Its bins - entropy-code what a frame of reference packs to one width, after a delta or a divisor where one - pays: two million event times in milliseconds take 1.7 MB as pco against 5.7 MB bit-packed and - 2.6 MB as zstd. `Smallest` tries it on every such chunk; `Auto` never does, as the reference's - default compressor never does, since it decodes slower than bit-packing. -* **A hint that does not apply falls back** to what `Auto` would have chosen, as bit-packing on - floating-point values does. -* **`AlpRd` is the hint for floats with no short decimal form**, measurements or ratios that `Alp` - refuses: their high bits go to a dictionary of eight and the rest are bit-packed. `Auto` keeps - zstd on such a column whenever zstd saves a tenth on it, since the file would otherwise grow; the - split reads and writes several times faster than zstd, for several times its bytes, which pays on - a local disk and not over a slow network. `VortexSession.AdviseAsync` measures both on your data. -* **Every profile takes the hint**, `Smallest` included, and prices the columns without one. -* `Auto` is not the smallest: zstd shrank `Celsius` threefold, and `City` by a fifth. `Auto` weighs - decode speed too, and a zstd block must be inflated whole before one value is read. Take such a - hint for a file written once and read rarely; leave `Auto` for one scanned often, and measure on - your own data: `VortexSession.AdviseAsync` does +* A chunk holding a single value, or only nulls, is written as a constant, and an arithmetic + progression is written as a sequence, whatever the hint says: nothing costs less per row. Runs + are different. A hint comes before them, since runs cost something to decode, and the bitmap of a + boolean column that is one percent true reads twelve times faster than its runs. +* Some shapes need no hint. A decimal whose values fit 64 bits is written as those integers + (`DecimalByteParts`), which then get the integer schemes. A timestamp coarser than its unit + (instants to the second stored in microseconds, dates at midnight) is split into days, seconds and + subseconds (`DateTimeParts`). A column where nine rows in ten are null is stored as its valid rows + at their positions (`Sparse`). A hint for another scheme still takes precedence on such a column, + as you asked. +* `OnPair` is the hint for text made of repeated substrings, such as URLs, paths and log lines. It + decodes as fast as FSST and, on such text, is a fifth to two fifths smaller. `Auto` picks it when it + comes out a tenth smaller than both FSST and zstd. +* `Pco` is the hint for numbers of sixteen bits or more that bit-packing leaves wide: timestamps with + jitter, counters, skewed amounts, multiples of a common unit. It entropy-codes the values after a + delta or a common divisor when one pays. Two million event times in milliseconds take 1.7 MB as pco, + against 5.7 MB bit-packed and 2.6 MB as zstd. `Smallest` tries it on every such chunk. `Auto` never + does, like the reference's default compressor, because it decodes more slowly than bit-packing. +* A hint that does not apply falls back to what `Auto` would have chosen, as bit-packing does on + floating-point values. +* `AlpRd` is the hint for floats with no short decimal form, measurements or ratios that `Alp` + refuses. Their high bits go to a dictionary of eight entries and the rest are bit-packed. `Auto` + keeps zstd on such a column whenever zstd saves a tenth of the bytes, since the file would grow + otherwise. The split reads and writes several times faster than zstd for several times its bytes, + which pays on a local disk and not over a slow network. `VortexSession.AdviseAsync` measures both on + your data. +* Every profile honours the hint, `Smallest` included, and prices the columns that have none. +* `Auto` is not the smallest. Zstd shrank `Celsius` threefold and `City` by a fifth, but `Auto` weighs + decode speed too, and a zstd block must be inflated whole before any of its values can be read. Use + such a hint for a file written once and rarely read, and keep `Auto` for files scanned often. Better + still, measure on your own data with `VortexSession.AdviseAsync` ([choose-encodings.md](choose-encodings.md#let-your-data-choose)). ## String bounds -`StringBoundBytes` sets how many bytes of a text column's minimum and maximum each zone keeps, so -that text filters prune by block. On the same rows with the cities in eight runs of 125 000: +`StringBoundBytes` sets how many bytes of a text column's minimum and maximum each zone keeps, so that +text filters can prune by block. On the same rows with the cities in eight runs of 125 000: | `StringBoundBytes` | bytes | zone maps | `City == "Nice"` reads | |---|---|---|---| @@ -134,36 +135,35 @@ that text filters prune by block. On the same rows with the cities in eight runs | 16 | 1 174 436 | 8 752 | 20 of 123 blocks | | 32 | 1 174 436 | 8 752 | 20 of 123 blocks | -Bounds cost 2 376 bytes here and cut the scan six times. 32 bytes changed nothing because no city -name is longer than 16; longer bounds matter for values that share a long prefix, such as URLs. -Bounds prune only when the values come in runs: with the cities rotating every seven rows, every +The bounds cost 2 376 bytes here and cut the scan six times. 32 bytes changed nothing because no city +name is longer than 16. Longer bounds matter for values that share a long prefix, such as URLs. +Bounds only prune when the values come in runs: with the cities rotating every seven rows, every block holds all of them and no bound can rule one out. -## Statistics, metadata, identity - -* **`Statistics = false`** saved 224 bytes of 1 508 316, and `file.Statistics.Count` is then 0: a - question the statistics would answer at open, such as a minimum or a sum, reads the zone maps or - the data instead. The zone maps are kept, so `Day > 2000` still reads 0 of 123 blocks. -* **`Metadata`** holds at most 14 entries, with keys of at most 64 UTF-8 bytes, besides the library's - own; `CreateWriter` refuses more: *A file carries at most 14 metadata entries besides its own; 15 - were given.* `file.Metadata.Keys` lists the caller's entries only, and `ReadAsync(key)` returns a - copy of the value. An append keeps the file's entries, and an entry named - again takes its new value. -* **`Identity`** pinned makes the write reproducible: the same rows and options gave the same bytes - twice, and a write without it gave different bytes. Each write draws a fresh identity otherwise, +## Statistics, metadata and identity + +* `Statistics = false` saved 224 bytes out of 1 508 316, and `file.Statistics.Count` is then 0. A + question the statistics would have answered at open, such as a minimum or a count, then reads the + zone maps or the data instead. The zone maps are kept, so `Day > 2000` still reads 0 of 123 blocks. +* `Metadata` holds at most 14 entries, with keys of at most 64 UTF-8 bytes, besides the library's + own. `CreateWriter` refuses more: *A file carries at most 14 metadata entries besides its own; 15 + were given.* `file.Metadata.Keys` lists only the caller's entries, and `ReadAsync(key)` returns a + copy of a value. An append keeps the file's entries, and an entry named again takes its new value. +* `Identity`, once pinned, makes the write reproducible: the same rows and options gave the same bytes + twice, and a write without it gave different bytes. Otherwise each write draws a fresh identity, appends included. -* **`TargetEdition`** decides which readers can open the file, and which structures the writer may - use: under `core2025.05.0` this file is 1 544 268 bytes and carries no zone maps. [editions.md](editions.md) - is that subject. +* `TargetEdition` decides which readers can open the file, and which structures the writer may use. + Under `core2025.05.0` this file is 1 544 268 bytes and carries no zone maps. + [editions.md](editions.md) covers that subject. ## The options that are not the writer's -Reading has its own options, apart from these. `VortexOpenOptions` (the tail read at open, the -torn-tail policy, the decompression ceiling, whether statistics are verified, a schema for a file -that embeds none, index fragments) is described in [open-a-file.md](open-a-file.md), -and its caps in [limits.md](limits.md). `ScanOptions` (batch size, pruning, indexes, compaction, -parallelism, read-ahead) belongs to a scan, and the session's options (memory pool, segment cache, -reads in flight, parallelism) to [threads.md](threads.md). +Reading has its own options. `VortexOpenOptions` (the tail read at open, the torn-tail policy, the +decompression ceiling, statistics verification, a schema for a file that embeds none, index +fragments) is described in [open-a-file.md](open-a-file.md), and its caps in [limits.md](limits.md). +`ScanOptions` (batch size, pruning, indexes, compaction, parallelism, read-ahead) belongs to a scan, +and the session's options (memory pool, segment cache, reads in flight, parallelism) are covered in +[threads.md](threads.md). ## Run it diff --git a/samples/Vorticity.Samples/Indexes.cs b/samples/Vorticity.Samples/Indexes.cs index 0bf7b18c..41432166 100644 --- a/samples/Vorticity.Samples/Indexes.cs +++ b/samples/Vorticity.Samples/Indexes.cs @@ -145,7 +145,7 @@ internal static async Task RunAsync() catch (VortexException e) { Console.WriteLine($"a required index over its budget: {e.GetType().Name}: {e.Message}"); - Console.WriteLine($" the file is left behind: {System.IO.File.Exists(path)}"); + Console.WriteLine($" the path still holds the file written before it: {System.IO.File.Exists(path)}"); } try diff --git a/samples/Vorticity.Samples/KeysInOrder.cs b/samples/Vorticity.Samples/KeysInOrder.cs index 6f9e70ca..ebe2eded 100644 --- a/samples/Vorticity.Samples/KeysInOrder.cs +++ b/samples/Vorticity.Samples/KeysInOrder.cs @@ -24,7 +24,7 @@ internal static async Task RunAsync() } Console.WriteLine($"rank of 900: {await cursor.RankAsync(900)}"); - Console.WriteLine($"{await cursor.KeyCountAsync()} distinct keys"); + Console.WriteLine($"{await cursor.KeyCountAsync()} entries share key {cursor.Key}"); await cursor.SeekRankAsync(123_456); Console.WriteLine($"entry of rank 123456: key {cursor.Key} at row {cursor.Row}"); await cursor.SeekLastAsync(); From 06cda70b9a01135d666554028523c2aa40b11bc8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rapha=C3=ABl=20DUCOM?= Date: Sat, 10 Oct 2026 13:11:20 +0200 Subject: [PATCH 2/4] docs(design): the design documents rewritten, and brought back in line with the code The design documents used section numbers as cross-references and read as a journal. Every pointer is now a named link to its section, the numbered headings stay so that existing anchors still resolve, and the prose is whole sentences without dashes or semicolons. Among the facts that changed, a map has a factory and maps to a dictionary, lists of records are supported, a decimal bucket is not implemented, and the exception hierarchy has four kinds. A caller's bytes are copied once when they are not 64-byte aligned, the mapping cache works on Windows, top-level extensions are row-encoded by their storage while nested ones are refused, and the commit header inlines up to 192 KiB of a 256 KiB opening read. Repacking is not public, the per-encoding corpus is 61 files, and the writer also tries OnPair and, under the size-first profile, pco. --- docs/design/01-scope.md | 116 +-- docs/design/02-format.md | 191 ++-- docs/design/03-architecture.md | 361 +++---- docs/design/04-conformance.md | 136 +-- docs/design/05-benchmarks.md | 366 +++---- docs/design/06-row-encoding.md | 157 +-- docs/design/07-dotnet-mapping.md | 156 +-- docs/design/08-semantics.md | 251 ++--- docs/design/09-contracts.md | 181 ++-- docs/design/10-indexes.md | 428 ++++---- docs/design/11-write-strategy.md | 598 ++++++------ docs/design/12-index-reads.md | 323 ++++--- docs/design/13-dataset.md | 1124 +++++++++++---------- docs/design/14-public-api.md | 406 ++++---- docs/design/15-compaction.md | 543 ++++++----- docs/design/16-queries.md | 1561 ++++++++++++++++-------------- docs/design/90-registry.md | 155 +-- docs/design/99-sources.md | 77 +- docs/design/README.md | 28 +- 19 files changed, 3706 insertions(+), 3452 deletions(-) diff --git a/docs/design/01-scope.md b/docs/design/01-scope.md index 26568b5d..74275d96 100644 --- a/docs/design/01-scope.md +++ b/docs/design/01-scope.md @@ -4,80 +4,82 @@ What Vorticity covers, what it deliberately leaves out, and the guarantees it ho ## 1. What Vortex is, and what that implies -Vortex is not another Parquet. It is a **columnar compression framework** whose file format is a +Vortex is not another Parquet. It is a columnar compression framework whose file format is a container, and three of its properties shape this implementation: -1. **The file is self-describing.** How rows are partitioned into chunks, zones and columns is not - fixed by the format: the writer decides, by composing a layout tree that the footer serializes. - A reader is therefore an interpreter of that tree, not a parser of a fixed structure. -2. **Data stays compressed until the last moment.** An array is a tree of encodings, such as +1. The file describes itself. How rows are split into chunks, zones and columns is not fixed by the + format. The writer decides, by composing a layout tree that the footer serializes, so a reader is + an interpreter of that tree rather than a parser of a fixed structure. +2. Data stays compressed until the last moment. An array is a tree of encodings, such as `dict(runend(for(bitpacked)))`, and a filter, a take or a projection can run on the compressed form. Decoding is a progressive canonicalization. -3. **Compatibility is governed by editions.** An edition is a frozen set of component ids. Files - written from Vortex 0.36.0 on, edition `core2025.05.0`, are promised to stay readable. A reader's - scope is a set of editions, not a version of the format, which has stayed at `VTXF` version 1. +3. Compatibility is governed by editions. An edition is a frozen set of component ids. Files written + by Vortex 0.36.0 and later, edition `core2025.05.0`, are promised to stay readable. A reader's + scope is therefore a set of editions, not a version of the format, which has stayed at `VTXF` + version 1. -[02-format.md](02-format.md) condenses the format; [90-registry.md](90-registry.md) lists every +[02-format.md](02-format.md) condenses the format, and [90-registry.md](90-registry.md) lists every component and its state. ## 2. What Vorticity is -A reader and a writer of Vortex files for .NET 11, with no third-party dependency: the FlatBuffers -and Protobuf runtimes are written here, and so is the Zstandard codec, `Vorticity.Zstd`; the one -other package the core references is the first-party `System.IO.Hashing` -([03-architecture.md](03-architecture.md) §1). It is compatible with Native AOT and trimming. +Vorticity is a reader and a writer of Vortex files for .NET 11, with no third-party dependency. The +FlatBuffers and Protobuf runtimes are written here, and so is the Zstandard codec, +`Vorticity.Zstd`. The only other package the core references is Microsoft's `System.IO.Hashing` +(see [the founding constraints](03-architecture.md#1-founding-constraints)). It is compatible with +Native AOT and trimming. | package | what it holds | |---|---| -| `Vorticity` | the session, files, the typed scan and its columns, aggregates, the key cursor, the tool path for schemas known only at run time, the writer and its builders, the I/O seam | -| `Vorticity.Generators` | the `[VortexRecord]` source generator and the analyzers VX1001 to VX1008, build-time only | -| `Vorticity.Dataset` | a versioned dataset over an object store ([13-dataset.md](13-dataset.md)); experimental, `VX0001` | -| `Vorticity.RowEncoding` | the byte-sortable row encoding ([06-row-encoding.md](06-row-encoding.md)); experimental, `VX0002`, because upstream reserves the right to change it | +| `Vorticity` | the session, files, the typed scan and its columns, aggregates, the key cursor, the tool path for schemas known only at run time, the writer and its builders, the I/O interface | +| `Vorticity.Generators` | the `[VortexRecord]` source generator and the analyzers VX1001 to VX1011, build time only | +| `Vorticity.Dataset` | a versioned dataset over an object store ([13-dataset.md](13-dataset.md)), experimental under `VX0001` | +| `Vorticity.RowEncoding` | the byte-sortable row encoding ([06-row-encoding.md](06-row-encoding.md)), experimental under `VX0002` because upstream reserves the right to change it | | `Vorticity.Zstd` | Zstandard compression and decompression in managed C#, which the core's `vortex.zstd` and `vortex.zstd_buffers` go through ([its README](../../src/Vorticity.Zstd/README.md)) | -| `vxdump` | the inspection tool, written against the public surface only | +| `vxdump` | the inspection tool, written against the public API only | ## 3. What it reads -- **Every component a Rust writer emits**: 37 array encodings and 7 layouts. That is every - component of the core editions up to `core2026.08.3`, `vortex.zstd_buffers` from the draft - `zstd2026.02.0` edition, and the three Rust writes outside every edition when an upstream flag - asks for them: `fastlanes.delta`, `vortex.patched` and the experimental `vortex.list` layout. -- **Every file from Rust 0.36.0 to the version the conformance corpus is pinned to**, 0.86.1, - value for value ([04-conformance.md](04-conformance.md)). -- **Lazily.** An id this library does not know fails only the read that needs it: an unprojected - column may use one, and a zone-map aggregate nobody knows only disables its pruning - ([08-semantics.md](08-semantics.md) §4). -- **Refused by name**, with the id and its kind: a legacy two-buffer `vortex.fsst`, a shredded - variant child, and a compressed buffer or segment, which the schema declares and no Vortex release - implements ([08-semantics.md](08-semantics.md) §7). +- Every component a Rust writer emits: 37 array encodings and 7 layouts. That covers every component + of the core editions up to `core2026.08.3`, `vortex.zstd_buffers` from the draft `zstd2026.02.0` + edition, and the three components Rust writes outside every edition when an upstream flag asks for + them: `fastlanes.delta`, `vortex.patched` and the experimental `vortex.list` layout. +- Every file from Rust 0.36.0 up to the version the conformance corpus is pinned to, 0.86.1, value + for value ([04-conformance.md](04-conformance.md)). +- Lazily. An id this library does not know only fails the read that needs it. An unprojected column + may use one, and a zone-map aggregate nobody knows only disables its own pruning (see [unknown + components](08-semantics.md#4-unknown-components-resolve-lazily-fail-on-use)). +- A few things are refused by name, with the id and its kind: a legacy two-buffer `vortex.fsst`, a + shredded variant child, and a compressed buffer or segment, which the schema declares but no + Vortex release implements (see [buffer-level LZ4](08-semantics.md#7-buffer-level-lz4)). ## 4. What it writes -- **Files every Rust reader of the target edition opens.** The target is `VortexEditions.Default`, - `core2026.08.3`, read by Vortex Rust from 0.85.0; lower targets are honoured by dropping - what they cannot carry, and refused where a column needs a component the target lacks - ([90-registry.md](90-registry.md)). The reference reads every file this writer produces, value - for value, in CI. -- **Encodings chosen per chunk**, by exact formulas over statistics computed in one pass, with a - trial only for zstd, pco, FSST, OnPair, ALP and ALP-RD ([11-write-strategy.md](11-write-strategy.md)). -- **Pruning structures**: a zone map per block of 8 192 rows, file statistics, bounded string - bounds, and on request the skipping and locating indexes of [10-indexes.md](10-indexes.md), which - a Rust reader ignores. -- **Appends** that continue a file in place, and a repair for a torn one. +- Files every Rust reader of the target edition opens. The default target is + `VortexEditions.Default`, `core2026.08.3`, read by Vortex Rust 0.85.0 and later. Lower targets are + honoured by dropping what they cannot carry, and refused where a column needs a component the + target lacks ([90-registry.md](90-registry.md)). In CI, the reference reads every file this writer + produces, value for value. +- Encodings chosen per chunk, by exact formulas over statistics computed in one pass, with a trial + only for zstd, pco, FSST, OnPair, ALP and ALP-RD ([11-write-strategy.md](11-write-strategy.md)). +- Pruning structures: a zone map per block of 8 192 rows, file statistics, bounded string bounds, + and on request the skipping and locating indexes of [10-indexes.md](10-indexes.md), which a Rust + reader ignores. +- Appends that continue a file in place, and a repair for a torn one. ## 5. What it answers -A scan reads the columns a record names and nothing else, pushes a filter written as a lambda into -the file statistics, the zone maps and the indexes, and decodes only the blocks that survive. On +A scan reads the columns a record names and nothing else, pushes a filter written as a lambda down +into the file statistics, the zone maps and the indexes, and decodes only the blocks that survive. On top of that: | capability | where | |---|---| | rows by index, decoding only the rows asked for where the encoding allows it | [90-registry.md](90-registry.md) | -| queries in the shape of LINQ — projections, group by, aggregates — on the encoded form, a dictionary by code, a run by its length, flowing as batches as soon as they are final, with sums exact whatever the cut | [16-queries.md](16-queries.md) | -| counts, existence, minimum and maximum answered from the structures before any decode | [12-index-reads.md](12-index-reads.md) §4 | +| queries in the shape of LINQ (projections, group by, aggregates) on the encoded form, a dictionary by its codes and a run by its length, flowing as batches as soon as they are final, with sums exact whatever the split | [16-queries.md](16-queries.md) | +| counts, existence, minimum and maximum answered from the structures before any decode | [answers without rows](12-index-reads.md#4-answers-without-rows) | | a key cursor that seeks, steps, ranks and counts, and batches in key order | [12-index-reads.md](12-index-reads.md) | -| parallel decode on a session's threads, batches still in file order | [09-contracts.md](09-contracts.md) §2 | +| parallel decoding on a session's threads, with batches still in file order | [parallelism](09-contracts.md#2-parallelism) | ## 6. What it leaves out @@ -87,21 +89,21 @@ top of that: | the `tensor`, `spatial` and `json` editions | plugins outside `core` | | encryption | the format's slot is an empty reserved table | | engine integrations (DataFusion, DuckDB, Spark, CUDA) | not a format library's job | -| Apache Arrow interoperability | it would break the zero-dependency rule; an owned `RecordBatch` is where a bridge package would attach | +| Apache Arrow interoperability | it would break the zero-dependency rule. An owned `RecordBatch` is where a bridge package would attach | | a DataFrame, SQL, joins | the plan a scan builds is what such a layer would target | -| deleting or updating rows | a file is immutable and an append adds blocks; mutation belongs to a storage engine above the library | -| an S3 client | the dataset's store is a seam another library implements ([13-dataset.md](13-dataset.md) §11) | +| deleting or updating rows inside a file | a file is immutable and an append only adds blocks. Row deletes and updates live one level up, in the dataset ([deleting and updating rows](13-dataset.md#12-deleting-and-updating-rows)) | +| an S3 client | the dataset's store is an interface another library implements ([the store abstraction](13-dataset.md#11-the-store-abstraction)) | | preserving unknown nodes byte for byte through a rewrite | a separate feature, with its own invariants | ## 7. Guarantees, and what holds each | guarantee | held by | |---|---| -| every corpus file Rust wrote is read value for value | the conformance corpus, 876 files over seven editions ([04-conformance.md](04-conformance.md) §3) | -| Rust reads every file this library writes | the cross-check in CI ([04-conformance.md](04-conformance.md) §4) | -| a scan allocates nothing per batch in steady state | allocation ratchets in the test suite ([05-benchmarks.md](05-benchmarks.md) §5) | -| a full scan within 2× of Rust's time, a decoder within 1.5×, files no larger than 105 % of Rust's | the benchmark gates, and `WrittenSizeTests` for the bytes ([05-benchmarks.md](05-benchmarks.md) §1); where each stands is on [the benchmark page](../guide/benchmarks.md) | -| pruning never removes a row a full scan returns | every filter test run with pruning and indexes on and off ([08-semantics.md](08-semantics.md) §1) | -| malformed input fails with a clean exception, never a crash, a hang or an unbounded allocation | parser fuzzing in CI and the resource caps ([08-semantics.md](08-semantics.md) §6) | -| Native AOT and trimming, no reflection | `vxdump` published ahead of time over the corpus in CI | -| the row encoding is byte-identical to Rust's `vortex-row`, and byte order is tuple order | golden vectors and a randomized order property ([04-conformance.md](04-conformance.md) §7) | +| every corpus file Rust wrote is read value for value | the conformance corpus, 876 files over seven editions ([the golden corpus](04-conformance.md#3-the-golden-corpus)) | +| Rust reads every file this library writes | the cross-check in CI ([the cross-check](04-conformance.md#4-the-cross-check-rust-reads-what-we-write)) | +| a scan allocates nothing per batch in steady state | allocation ratchets in the test suite ([allocations](05-benchmarks.md#5-allocations)) | +| a full scan within 2× of Rust's time, a decoder within 1.5×, files no larger than 105 % of Rust's | the benchmark gates, and `WrittenSizeTests` for the bytes ([the comparison](05-benchmarks.md#1-the-comparison)). Where each stands is on [the benchmark page](../guide/benchmarks.md) | +| pruning never removes a row a full scan returns | every filter test runs with pruning and indexes on and off ([statistic precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)) | +| malformed input fails with a clean exception, never a crash, a hang or an unbounded allocation | parser fuzzing in CI and the resource caps ([resource caps](08-semantics.md#6-resource-caps)) | +| Native AOT and trimming, no reflection | `vxdump` published ahead of time and run over the corpus in CI | +| the row encoding is byte-identical to Rust's `vortex-row`, and byte order is tuple order | golden vectors and a randomized order property ([the row encoding](04-conformance.md#7-the-row-encoding)) | diff --git a/docs/design/02-format.md b/docs/design/02-format.md index beda5adb..c5cfedc8 100644 --- a/docs/design/02-format.md +++ b/docs/design/02-format.md @@ -1,8 +1,8 @@ # The format, condensed The Vortex file format as this library reads and writes it, condensed from the primary sources of -[99-sources.md](99-sources.md); the schemas themselves are vendored in [spec/](../../spec/). -**Everything is little-endian.** +[99-sources.md](99-sources.md). The schemas themselves are vendored in [spec/](../../spec/). +Everything is little-endian. ## 1. File structure @@ -17,11 +17,11 @@ The Vortex file format as this library reads and writes it, condensed from the p ├────────────────────────────┤ │ DType flatbuffer │ optional (may be supplied out of band) ├────────────────────────────┤ -│ Layout flatbuffer │ required — root layout tree +│ Layout flatbuffer │ required: the root layout tree ├────────────────────────────┤ -│ Statistics flatbuffer │ optional — per-field, file-level statistics +│ Statistics flatbuffer │ optional: per-field, file-level statistics ├────────────────────────────┤ -│ Footer flatbuffer │ required — dictionaries (segments, encodings, ...) +│ Footer flatbuffer │ required: dictionaries (segments, encodings, ...) ├────────────────────────────┤ │ Postscript │ ≤ 65527 bytes ├────────────────────────────┤ @@ -29,7 +29,7 @@ The Vortex file format as this library reads and writes it, condensed from the p └────────────────────────────┘ ``` -Frozen constants (these never change): +Frozen constants, which never change: | Constant | Value | |---|---| @@ -40,17 +40,17 @@ Frozen constants (these never change): | `V1_FOOTER_FBS_SIZE` | 32 | | Extension | `.vortex` | -**Open sequence (2 round trips worst case, 1 best case):** +Opening a file takes one round trip at best and two at worst: -1. Read the 64 KiB tail (or the whole file if smaller). By construction this always covers the +1. Read the 64 KiB tail, or the whole file if it is smaller. By construction this always covers the postscript (`MAX_POSTSCRIPT_SIZE` + `EOF_SIZE` ≤ 64 KiB). A local file, whose reads are copies - and cost no round trip, reads its last 8 KiB first; one whose postscript reaches past them reads - the 64 KiB next. -2. Validate magic and version from the last 8 bytes, read the postscript length. -3. Parse the postscript → locate dtype / layout / statistics / footer / metadata. + and cost no round trip, reads its last 8 KiB first, and reads the 64 KiB next only if its + postscript reaches past them. +2. Validate the magic and the version in the last 8 bytes, and read the postscript length. +3. Parse the postscript to locate the dtype, layout, statistics, footer and metadata. 4. If any of those segments falls outside the window already read, issue one targeted second read. -This is exactly why the postscript exists: without it, locating a large DType would cost three +That is exactly why the postscript exists: without it, locating a large DType would cost three round trips. ## 2. Postscript (`footer.fbs`) @@ -70,10 +70,10 @@ table PostscriptSegment { } ``` -Compression and encryption are **inline** in the postscript (rather than indices into the footer) -so a reader does not need the footer before it can decrypt. +Compression and encryption are inline in the postscript, rather than indices into the footer, so a +reader does not need the footer before it can decrypt. -User metadata values are **not** loaded by default: each locator is resolved on demand. +User metadata values are not loaded by default. Each locator is resolved on demand. ## 3. Footer (`footer.fbs`) @@ -100,17 +100,17 @@ table FileStatistics { field_stats: [ArrayStats]; } // one entry per root field ``` `segment_specs` is a vector of fixed-size structs, so it is read as a reinterpreted -`ReadOnlySpan` with no traversal and no allocation, which matters on wide files; so is -`Buffer` (8 bytes) in `array.fbs`. +`ReadOnlySpan` with no traversal and no allocation, which matters on wide files. The +same goes for `Buffer` (8 bytes) in `array.fbs`. ## 4. DType (`dtype.fbs` + `dtype.proto`) -The root DType is a FlatBuffer; the Protobuf variant is used in scalars and statistics. Both -define the same union, with identical and **stable** tags: +The root DType is a FlatBuffer, and the Protobuf variant is used in scalars and statistics. Both +define the same union, with identical and stable tags: | Tag | Type | Fields | |---|---|---| -| 1 | `Null` | — | +| 1 | `Null` | none | | 2 | `Bool` | nullable | | 3 | `Primitive` | ptype, nullable | | 4 | `Decimal` | precision (u8), scale (i8), nullable | @@ -119,18 +119,19 @@ define the same union, with identical and **stable** tags: | 7 | `Struct_` | names[], dtypes[], nullable | | 8 | `List` | element_type, nullable | | 9 | `Extension` | id, storage_dtype, metadata[] | -| 10 | `FixedSizeList` | element_type, size (u32), nullable — *placed after Extension for backward compatibility* | +| 10 | `FixedSizeList` | element_type, size (u32), nullable. Placed after Extension for backward compatibility | | 11 | `Variant` | nullable | | 12 | `Union` | names[], dtypes[], type_ids[], nullable | | 13 | `Map` | key_type, value_type, keys_sorted, nullable | `PType`: `U8=0, U16, U32, U64, I8, I16, I32, I64, F16, F32, F64=10`. -Two parsers therefore share one model, and a property test holds them equivalent: a generated -DType serialized both ways parses back to the same value (`DTypeEquivalenceTests`). +Two parsers therefore share one model, and a property test keeps them equivalent: a generated DType +serialized both ways parses back to the same value (`DTypeEquivalenceTests`). -A file's root DType is **not required to be a Struct**: a file may hold a bare `Float64` or `Bool`, -and a reader must not assume a tabular schema ([07-dotnet-mapping.md](07-dotnet-mapping.md) §5). +A file's root DType does not have to be a struct. A file may hold a bare `Float64` or `Bool`, and a +reader must not assume a tabular schema (see [a root that is not a +struct](07-dotnet-mapping.md#5-a-root-that-is-not-a-struct)). Core-edition extension dtypes: `vortex.date`, `vortex.time`, `vortex.timestamp`, `vortex.uuid`. @@ -142,13 +143,13 @@ Core-edition extension dtypes: `vortex.date`, `vortex.time`, `vortex.timestamp`, [padding] [buffer 0] [padding] [buffer 1] ... [padding] [Array flatbuffer] [u32 flatbuffer length] ``` -Reading: read the last 4 bytes → `fb_length`; the FlatBuffer occupies -`[len-4-fb_length, len-4)`; everything before it is the buffer region. Each buffer's offset is +To read one, take the last 4 bytes as `fb_length`. The FlatBuffer occupies +`[len-4-fb_length, len-4)`, and everything before it is the buffer region. Each buffer's offset is reconstructed by accumulating `padding + length` from 0. The FlatBuffer is 8-byte aligned. Each buffer carries its own required alignment (up to 16 for -`varbinview` views), and the written padding guarantees that a memory-mapped segment is directly -usable — **this is what makes zero-copy possible**. +`varbinview` views), and the written padding guarantees that a memory-mapped segment is usable as +is. This is what makes zero-copy reads possible. ### 5.2 Schema (`array.fbs`) @@ -180,14 +181,14 @@ table ArrayStats { enum Precision : uint8 { Inexact = 0, Exact = 1 } ``` -**Key point:** an `ArrayNode` does **not** carry its DType. The type is inherited or derived from -the parent (the root DType comes from the file). Decoding is therefore a type-guided top-down +The key point is that an `ArrayNode` does not carry its DType. The type is inherited or derived from +the parent, and the root DType comes from the file. Decoding is therefore a type-guided top-down traversal, and every decoder must know how to compute its children's DTypes. ### 5.3 Per-encoding metadata -Each encoding serializes its metadata as **Protobuf** (`prost`) with stable tags. There is no -single `.proto` file upstream: messages are declared beside each encoding. A few examples: +Each encoding serializes its metadata as Protobuf (`prost`) with stable tags. There is no single +`.proto` file upstream: messages are declared next to each encoding. A few examples: ```proto // vortex.dict @@ -215,52 +216,52 @@ message PatchesMetadata { uint64 len = 1; uint64 offset = 2; PType indices_ptype ``` `vortex.primitive`, `vortex.varbinview`, `vortex.struct`, `vortex.chunked`, `vortex.null`, -`vortex.masked`, `vortex.fixed_size_list`, `vortex.ext`, `vortex.bytebool`, `vortex.zigzag` and -`fastlanes.for`: empty metadata. +`vortex.masked`, `vortex.fixed_size_list`, `vortex.ext`, `vortex.bytebool` and `vortex.zigzag` have +empty metadata. -Two traps in that neighbourhood, in opposite directions. `vortex.constant` **is** empty — but its -scalar is not absent, it lives in **buffer 0** as a Protobuf `ScalarValue`, so a decoder that reads -the metadata and stops finds nothing. `fastlanes.for` is **not** empty: its metadata is a bare +Two traps sit nearby, in opposite directions. `vortex.constant` has empty metadata, but its scalar +is not absent: it lives in buffer 0 as a Protobuf `ScalarValue`, so a decoder that reads the +metadata and stops finds nothing. `fastlanes.for` does not have empty metadata: it is a bare `ScalarValue` holding the frame-of-reference value, with zero buffers and one child. And `vortex.zoned`'s layout metadata is a version byte followed by a protobuf, not a bare message. -The complete transcription of every metadata message, with the upstream source path of each, is +The complete transcription of every metadata message, with the upstream source path of each, is in [spec/METADATA.md](../../spec/METADATA.md). Its tag numbers are the contract. #### Unknown Protobuf fields: skip, never reject -The Protobuf runtime **skips fields with unknown numbers**, dispatching on the wire type (varint, -fixed32, fixed64, length-delimited; groups, wire types 3 and 4, are rejected, as proto3 never emits -them). +The Protobuf runtime skips fields with unknown numbers, dispatching on the wire type (varint, +fixed32, fixed64, length-delimited). Groups, wire types 3 and 4, are rejected, since proto3 never +emits them. -This protects the read-forever promise. Upstream may add an *optional* field to a metadata message -without it being a reader-visible evolution: old readers ignore it and the semantics are unchanged, -so upstream need not mint a new id. A parser that rejected unknown fields would fail on legal files, +This protects the read-forever promise. Upstream may add an optional field to a metadata message +without it being a reader-visible change: old readers ignore it and the semantics are unchanged, so +upstream does not need a new id. A parser that rejected unknown fields would fail on legal files, and a corpus pinned to one version would not notice until its next regeneration. The distinction to keep sharp: -* **Reject** = a value outside the contract's domain (a forbidden ptype, `lower_part_count != 0`, - an out-of-range exponent). -* **Tolerate** = a field number the wire format does not recognize. +* Reject a value outside the contract's domain (a forbidden ptype, `lower_part_count != 0`, an + out-of-range exponent). +* Tolerate a field number the wire format does not recognize. -**An encoding id is a frozen contract.** A reader-visible evolution gets a **new id** -(`vortex.foo` → `vortex.foo_v2`), never a silent extension. A decoder therefore rejects a payload -outside the contract of the id it read, even when it could decode the successor's: `vortex.pco` -refuses an 8-bit dtype, which belongs to `vortex.pco.v2`. +An encoding id is a frozen contract. A reader-visible change gets a new id (`vortex.foo` becomes +`vortex.foo_v2`), never a silent extension. A decoder therefore rejects a payload outside the +contract of the id it read, even when it could decode the successor's: `vortex.pco` refuses an 8-bit +dtype, which belongs to `vortex.pco.v2`. ### 5.4 Validity (nullability) -Validity is not a mandatory bitmap. Four states: `NonNullable`, `AllValid`, `AllInvalid`, or a -**boolean array** (itself possibly encoded: constant, dict, runend…). `true` = valid. The validity -array's DType is always non-nullable `Bool`. +Validity is not a mandatory bitmap. It has four states: `NonNullable`, `AllValid`, `AllInvalid`, or +a boolean array, which may itself be encoded (constant, dict, runend…). `true` means valid. The +validity array's DType is always a non-nullable `Bool`. ## 6. Layouts (`layout.fbs`) ```fbs table Layout { encoding: uint16; // index into Footer.layout_specs - row_count: uint64; // rows represented — the basis for pruning + row_count: uint64; // rows represented, the basis for pruning metadata: [ubyte]; // opaque, layout-specific children: [Layout]; segments: [uint32]; // indices into Footer.segment_specs @@ -269,55 +270,55 @@ table Layout { | ID | Role | Children | Segments | Metadata | |---|---|---|---|---| -| `vortex.flat` | one serialized array | 0 | exactly 1 | `optional bytes array_encoding_tree` — when present the array FlatBuffer is inlined in the layout and the segment holds only buffers | -| `vortex.chunked` | row-wise partition | ≥ 1 | 0 | empty (chunk offsets derive from children `row_count`s, whose sum must equal the parent's) | -| `vortex.struct` | one child per field (+ validity) | n fields | 0 | empty | -| `vortex.dict` | dictionary shared with a child | codes + values | — | `PType codes_ptype = 1; optional bool is_nullable_codes = 2; optional bool all_values_referenced = 3;` | -| `vortex.zoned` | statistics zone map for pruning | data + stats table | — | `[u8 version = 1] ++ proto { uint32 zone_len = 1; repeated AggregateSpec aggregate_specs = 2; }` | -| `vortex.stats` | legacy ancestor of `zoned` | same | — | must stay readable (read by the same machinery as `zoned`) | +| `vortex.flat` | one serialized array | 0 | exactly 1 | `optional bytes array_encoding_tree`. When present, the array FlatBuffer is inlined in the layout and the segment holds only buffers | +| `vortex.chunked` | row-wise partition | ≥ 1 | 0 | empty. Chunk offsets derive from the children's `row_count`s, whose sum must equal the parent's | +| `vortex.struct` | one child per field (plus validity) | n fields | 0 | empty | +| `vortex.dict` | dictionary shared with a child | codes + values | | `PType codes_ptype = 1; optional bool is_nullable_codes = 2; optional bool all_values_referenced = 3;` | +| `vortex.zoned` | statistics zone map for pruning | data + stats table | | `[u8 version = 1] ++ proto { uint32 zone_len = 1; repeated AggregateSpec aggregate_specs = 2; }` | +| `vortex.stats` | legacy ancestor of `zoned` | same | | must stay readable, and is read by the same machinery as `zoned` | Aggregates allowed in zone maps (core edition): `vortex.min`, `vortex.max`, `vortex.bounded_min`, -`vortex.bounded_max`, `vortex.nan_count`, `vortex.null_count`. -An unknown aggregate must **not** invalidate the file: it disables the affected pruning only. +`vortex.bounded_max`, `vortex.nan_count`, `vortex.null_count`. An unknown aggregate must not +invalidate the file. It only disables the pruning that depends on it. ## 7. What the Rust writer produces -What a default Rust writer emits is what most files contain. Its pipeline is **split structs → -repartition into 8 192-row blocks → zone maps → dictionary where useful → coalesce toward ~1 MiB → -BtrBlocks compression → flat leaves**. +What a default Rust writer emits is what most files contain. Its pipeline splits structs, +repartitions into 8 192-row blocks, adds zone maps, applies a dictionary where useful, coalesces +toward about 1 MiB, compresses with BtrBlocks and writes flat leaves. -Compression schemes enabled by default, in this order (order breaks ties): -FoR, ZigZag, BitPacking, Sparse, IntDict, RunEnd, Sequence, IntRLE, Delta; ALP, ALPrd, FloatDict, -NullDominatedSparse, FloatRLE; StringDict, FSST, OnPair; BinaryDict, VarBin; Decimal; Temporal. +Compression schemes enabled by default, in this order (the order breaks ties): FoR, ZigZag, +BitPacking, Sparse, IntDict, RunEnd, Sequence, IntRLE, Delta, ALP, ALPrd, FloatDict, +NullDominatedSparse, FloatRLE, StringDict, FSST, OnPair, BinaryDict, VarBin, Decimal, Temporal. Two caveats: -* **Zstd and Pco are not enabled by default** (`with_compact()`, the `zstd` and `pco` features), so - ordinary files need neither. This library reads both, every type, mode and delta pco has, and - writes both: zstd costs it no third-party dependency, since its frames go through - `Vorticity.Zstd`, this repository's managed Zstandard, which writes libzstd's frames byte for - byte, and pco is ported, its encoder tried by the size-first profile as `with_compact()` tries - it. -* **`fastlanes.delta` belongs to no edition.** The Delta scheme is in the list, but the edition - allowlist forbids its id, so it appears only in files written with enforcement turned off. +* Zstd and Pco are not enabled by default (they need `with_compact()` and the `zstd` and `pco` + features), so ordinary files need neither. This library reads both, including every type, mode and + delta pco has, and writes both. Zstd costs it no third-party dependency, since its frames go + through `Vorticity.Zstd`, this repository's managed Zstandard, which writes libzstd's frames byte + for byte. Pco is ported, and its encoder is tried by the size-first profile the way + `with_compact()` tries it. +* `fastlanes.delta` belongs to no edition. The Delta scheme is in the list, but the edition allowlist + forbids its id, so it only appears in files written with enforcement turned off. ## 8. Algorithmic details that are easy to get wrong -* **FastLanes**: 1024-element **transposed** blocks. Bit-packing operates in that order, not - logical order — which is what allows unpacking without cross-lane shuffles, and any access by - logical index must go through the inverse transposition. There are **two** permutations, not one, - and they are easy to conflate: the bit-packing index - `FL_ORDER[row / 8] * 16 + (row % 8) * 128 + lane` (lane over `1024 / bitwidth(T)` lanes), and the - element transposition `lane * 64 + FL_ORDER[order] * 8 + row` (lane always `idx % 16`). Only - `FL_ORDER = [0,4,2,6,1,5,3,7]` is its own inverse; the transposition itself is not, so - `untranspose` is the inverse mapping and not a second `transpose`. Both derivations, their - inverses, and the known-value table to test against are in +* FastLanes uses 1024-element transposed blocks. Bit-packing operates in that order, not in logical + order, which is what allows unpacking without cross-lane shuffles, and any access by logical index + must go through the inverse transposition. There are two permutations, not one, and they are easy + to conflate: the bit-packing index `FL_ORDER[row / 8] * 16 + (row % 8) * 128 + lane` (lane over + `1024 / bitwidth(T)` lanes), and the element transposition `lane * 64 + FL_ORDER[order] * 8 + row` + (lane always `idx % 16`). Only `FL_ORDER = [0,4,2,6,1,5,3,7]` is its own inverse. The transposition + is not, so `untranspose` is the inverse mapping and not a second `transpose`. Both derivations, + their inverses and the known-value table to test against are in [spec/REFERENCE.md](../../spec/REFERENCE.md). -* **FSST**: 255-entry symbol table; code 255 is the escape code. Symbol buffers are padded to that - fixed size. -* **ALP**: the encoded integer is `i32` or `i64`; `exp_e`/`exp_f` must be validated in range for - the target float type. Patches in a modern ALP are read as a `Patched` array wrapping a +* FSST uses a 255-entry symbol table, and code 255 is the escape code. Symbol buffers are padded to + that fixed size. +* In ALP the encoded integer is an `i32` or an `i64`, and `exp_e` and `exp_f` must be validated in + range for the target float type. Patches in a modern ALP are read as a `Patched` array wrapping a patch-free ALP. -* **VarBinView**: 16-byte views (Arrow StringView compatible), `views` buffer aligned to 16. -* **Patches**: one shared structure (indices, values and optional chunk offsets) that BitPacked, - ALP and Sparse all use, implemented once here. +* VarBinView uses 16-byte views (compatible with Arrow's StringView), with the `views` buffer aligned + to 16. +* Patches are one shared structure (indices, values and optional chunk offsets) that BitPacked, ALP + and Sparse all use, implemented once here. diff --git a/docs/design/03-architecture.md b/docs/design/03-architecture.md index 583b1b75..1daf9ad7 100644 --- a/docs/design/03-architecture.md +++ b/docs/design/03-architecture.md @@ -1,254 +1,259 @@ # .NET architecture -The engine under the public surface: the constraints it is built to, where things are, the objects a -scan and a write run on, the I/O seam, and the rules that keep it fast and safe. The surface itself -is [14-public-api.md](14-public-api.md); what each column type becomes in .NET is -[07-dotnet-mapping.md](07-dotnet-mapping.md). +The engine under the public API: the constraints it is built to, where things are, the objects a +scan and a write run on, the I/O interface, and the rules that keep it fast and safe. The public API +itself is described in [14-public-api.md](14-public-api.md), and what each column type becomes in +.NET in [07-dotnet-mapping.md](07-dotnet-mapping.md). ## 1. Founding constraints | constraint | consequence | |---|---| -| **No third-party dependency** | FlatBuffers and Protobuf are read and written by runtimes of this repository, hand-written against the five schemas; no `Google.FlatBuffers`, `protobuf-net` or `Apache.Arrow`. Two packages are referenced: `System.IO.Hashing` from `dotnet/runtime`, for XXH3, which the shared framework does not carry, and `Vorticity.Zstd`, this repository's managed Zstandard, rather than the libzstd the runtime ships | -| **No allocation on the hot path** | no LINQ, no `foreach` over interfaces, no capturing closures, no boxing, no `params` array (only `params ReadOnlySpan`); aligned native buffers, pools for transients, `ref struct` views | -| **SIMD with a scalar twin** | `System.Runtime.Intrinsics`: `Vector128` everywhere, `Vector256` and `Vector512` where the hardware has them, and a scalar path that the suite runs under `DOTNET_EnableHWIntrinsic=0` and compares bit for bit | -| **Async only, where there is I/O** | `ValueTask` and `ConfigureAwait(false)` throughout; no blocking public call and no synchronous twin of an asynchronous one. A source whose reads complete at once, a mapped file or bytes in memory, pays for a completed `ValueTask`, not for a second path. Pure CPU work stays synchronous: filling a builder, row encoding, decode kernels, a merge or a sort cut into parts the caller awaits. A builder never reads | -| **Streaming, bounded** | every source, operator and sink exchanges batches, pulled by its consumer: the async boundary is the batch, and every loop inside one is synchronous. Read-ahead is a bounded window, an operator holds its open state and never its input, a result is a stream of batches like a file's, and a consumer that stops stops the reads. What a query costs before its first answer, and holds while it runs, is measured ([16-queries.md](16-queries.md) §2) | - -**One target framework, `net11.0`.** A single target means no `#if`, no API surface that differs by -target, no test matrix, and no kernel that could diverge between legs. The price is reach: +| no third-party dependency | FlatBuffers and Protobuf are read and written by runtimes in this repository, hand-written against the five schemas. There is no `Google.FlatBuffers`, `protobuf-net` or `Apache.Arrow`. Two packages are referenced: `System.IO.Hashing` from `dotnet/runtime`, for XXH3, which the shared framework does not carry, and `Vorticity.Zstd`, this repository's managed Zstandard, rather than the libzstd the runtime ships | +| no allocation on the hot path | no LINQ, no `foreach` over interfaces, no capturing closures, no boxing, no `params` array (only `params ReadOnlySpan`). Aligned native buffers, pools for transient memory, `ref struct` views | +| SIMD with a scalar twin | `System.Runtime.Intrinsics`: `Vector128` everywhere, `Vector256` and `Vector512` where the hardware has them, and a scalar path that the suite runs under `DOTNET_EnableHWIntrinsic=0` and compares bit for bit | +| async only where there is I/O | `ValueTask` and `ConfigureAwait(false)` throughout, with no blocking public call and no synchronous twin of an asynchronous one. A source whose reads complete at once (a mapped file, bytes in memory) pays for a completed `ValueTask`, not for a second code path. Pure CPU work stays synchronous: filling a builder, row encoding, decode kernels, a merge or a sort split into parts the caller awaits. A builder never reads | +| streaming, bounded | every source, operator and sink exchanges batches, pulled by its consumer. The async boundary is the batch, and every loop inside one is synchronous. Read-ahead is a bounded window, an operator holds its open state and never its input, a result is a stream of batches like a file's, and a consumer that stops also stops the reads. What a query costs before its first answer, and what it holds while it runs, is measured ([the flow](16-queries.md#2-the-flow)) | + +The library has one target framework, `net11.0`. A single target means no `#if`, no API that differs +by target, no test matrix, and no kernel that could diverge between targets. The price is reach: .NET 11 is not a long-term-support release, and a consumer pinned to .NET 10 cannot reference the -library. `netstandard2.1` is out of the question: it lacks `Vector512`, static abstract members, -`INumber`, `SearchValues` and aligned native allocation, which is to say the performance model. -`global.json` pins the SDK, so that a build never changes compiler silently under a library whose -code generation is measured. +library. `netstandard2.1` is out of the question, because it lacks `Vector512`, static abstract +members, `INumber`, `SearchValues` and aligned native allocation, which is to say the whole +performance model. `global.json` pins the SDK, so a build never silently changes compiler under a +library whose code generation is measured. The library enables `AllowUnsafeBlocks` and `IsAotCompatible`. It does not set -`InvariantGlobalization`, an application property with no effect on a library; it avoids +`InvariantGlobalization`, an application property with no effect on a library, and avoids culture-sensitive APIs instead. A module initializer asserts that the process is little-endian and that the inline FlatBuffers structs have their wire sizes, since the zero-copy design reinterprets file bytes in place. -**Cancellation** is taken where it is actionable: an open, a segment read, a scan's enumeration, +Cancellation is accepted wherever it is actionable: an open, a segment read, a scan's enumeration, every answer a scan computes, every write. It is not threaded through decode kernels, which are -short and CPU-bound: the granularity is the batch. +short and CPU-bound. The granularity is the batch. ## 2. Where things are | `src/Vorticity/` | | |---|---| -| `Buffers`, `Serialization` | aligned buffers and leases; the FlatBuffers and Protobuf runtimes and the schema views | -| `Types`, `Schema`, `Editions` | the dtype model and its two serializations; the public schema; the edition registry | -| `File`, `IO`, `Layouts` | the open path, the footer and the file's statistics; the segment seam and its sources; the layout readers | -| `Arrays`, `Compute`, `Expressions` | the canonical arena and one decoder per encoding; kernels, masks, pruning; the filter expressions | -| `Scan`, `Columns`, `Symbolic`, `Records` | the scan and its plan; the borrowed and owned batches; the typed filter algebra; record binding | -| `Aggregation`, `Keys`, `Indexes` | aggregates and group by on encoded forms; the key cursor and its sources; index directory, pruners, policies | -| `Writing`, `Advice` | the writer, its builders and chooser, the index builders; the encoding advisor | -| `Session`, `Diagnostics` | the session, its pool and cache; exceptions, limits, the meter and the event source | - -Beside it: `src/Vorticity.Generators`, `.Dataset` and `.RowEncoding`; `tests/` (unit and property -tests, the conformance corpus, the fuzzer, and code that must not compile); `bench/` -([bench/README.md](../../bench/README.md)); `tools/` (`vxdump`, the Rust corpus generator, the -Rust benchmark shim, the row-encoding vectors, the test-impact selector); `samples/`, the programs -the guide's pages run. +| `Buffers`, `Serialization` | aligned buffers and leases, the FlatBuffers and Protobuf runtimes and the schema views | +| `Types`, `Schema`, `Editions` | the dtype model and its two serializations, the public schema, the edition registry | +| `File`, `IO`, `Layouts` | the open path, the footer and the file's statistics, the segment interface and its sources, the layout readers | +| `Arrays`, `Compute`, `Expressions` | the canonical arena and one decoder per encoding, kernels, masks and pruning, the filter expressions | +| `Scan`, `Columns`, `Symbolic`, `Records` | the scan and its plan, the borrowed and owned batches, the typed filter algebra, record binding | +| `Aggregation`, `Keys`, `Indexes` | aggregates and group by on encoded forms, the key cursor and its sources, the index directory, pruners and policies | +| `Writing`, `Advice` | the writer, its builders and chooser, the index builders, the encoding advisor | +| `Session`, `Diagnostics` | the session, its pool and cache, the exceptions, limits, meter, activity source and event source | + +Alongside it sit `src/Vorticity.Generators`, `.Dataset`, `.RowEncoding` and `.Zstd`, then `tests/` +(unit and property tests, the conformance corpus, the fuzzer, and code that must not compile), +`bench/` ([bench/README.md](../../bench/README.md)), `tools/` (`vxdump`, the Rust corpus generator, +the Rust benchmark shim, the row-encoding vectors, the test-impact selector), and `samples/`, the +programs the guide's pages run. ## 3. Object model -None of this is public. The arenas are built once at open and at `CreateWriter` and never appear; -[14-public-api.md](14-public-api.md) builds the surface over them. +None of this is public. The arenas are built once, at open and at `CreateWriter`, and never appear +in the API. [14-public-api.md](14-public-api.md) builds the public API on top of them. ### 3.1 Buffers A segment is a view over aligned bytes, whatever their origin: a mapped file (zero-copy), pooled -native memory (a positional or remote read), or managed memory the caller already holds. A -`SegmentLease` owns the bytes and releases them when the batch that borrowed them is done. +native memory (a positional or remote read), or memory the caller already holds. A `SegmentLease` +owns the bytes and releases them when the batch that borrowed them is done. ### 3.2 Types A dtype is a `readonly struct` with a tag and a payload, its children in a shared array, rather than -a class hierarchy: a wide schema must not become thousands of objects. Equality and hashing are +a class hierarchy, so a wide schema does not become thousands of objects. Equality and hashing are structural and allocate nothing. ### 3.3 Arrays: an arena, not an object graph A batch of fifty columns, each a tree of three to six encodings rebuilt per chunk, would allocate -hundreds of objects per batch as an object graph. So nodes are structs in a pooled array owned by -the batch (`CanonicalArena`), addressed by index; `CanonicalNode` is a `ref struct` view of one. -This mirrors the format, where `ArrayNode.children` and `buffers` are indices already. The blocks a -batch decodes into are kept for the next batch of the scan, which writes into memory still in the -cache without renting it again, and go back to the pool when the scan ends. The parse of a flat -layout's array blob is kept the same way, in the contexts the scans pool, for the next batch that -holds the same segment bytes. - -A file's encoding ids are resolved **once, at open**, against the registry, into an array indexed -by the file's own `u16`: on the hot path, dispatch is an index, not a string lookup. Kernels are +hundreds of objects per batch as an object graph. Nodes are therefore structs in a pooled array owned +by the batch (`CanonicalArena`), addressed by index, and `CanonicalNode` is a `ref struct` view of +one. This mirrors the format, where `ArrayNode.children` and `buffers` are already indices. The +blocks a batch decodes into are kept for the scan's next batch, which writes into memory that is +still in the CPU cache without renting it again, and go back to the pool when the scan ends. The +parse of a flat layout's array blob is kept the same way, in the contexts the scans pool, for the +next batch that holds the same segment bytes. + +A file's encoding ids are resolved once, at open, against the registry, into an array indexed by +the file's own `u16`, so on the hot path dispatch is an index, not a string lookup. Kernels are generic over the physical type (`where T : unmanaged, INumberBase`) and specialized per type by -the compiler: no `switch` on a type inside a loop, and no virtual call per value. +the compiler. There is no `switch` on a type inside a loop and no virtual call per value. -A decoder canonicalizes a node, and may instead decode only a selection of its rows -(`DecodeSelected`), which is how a take avoids decoding a chunk it needs one row of -([90-registry.md](90-registry.md)). +A decoder canonicalizes a node, or decodes only a selection of its rows (`DecodeSelected`), which is +how a take avoids decoding a whole chunk to get one row of it ([90-registry.md](90-registry.md)). -### 3.4 What the surface sees of it +### 3.4 What the public API sees of it -A scan's batch is borrowed: `Columns`, `Column` and `BatchView` are `ref struct`s over -the scan's arenas, valid until the next `MoveNextAsync`, and the compiler, not a `Dispose`, holds -that lifetime. A batch that must outlive its successor is an owned `RecordBatch`, one copy into -pooled buffers, which the caller disposes. +A scan's batch is borrowed. `Columns`, `Column` and `BatchView` are `ref struct`s over +the scan's arenas, valid until the next `MoveNextAsync`, and the compiler, not a `Dispose`, enforces +that lifetime. A batch that must outlive the next one is an owned `RecordBatch`, one copy into pooled +buffers, which the caller disposes. ### 3.5 I/O -The public seam is `Vorticity.IO.ISegmentSource` (`IO/SegmentSeam.cs`): a length, a single read, -and a batch read of many ranges into many leases. The batch read is the real entry point: a scan -registers every segment a split needs before reading, which lets a source **coalesce** nearby -ranges (1 MiB gap by default) and issue them together, and a scan requests each segment at most -once, every block of it decoding from the one lease. That is what makes or breaks performance on -an object store. - -What an implementer of the seam must honour, since object storage lives outside the core: - -- **All or nothing**: a batch read that fails releases every lease it acquired and throws; retry - policy is the source's. -- **Ownership**: a lease holds its bytes, one block or the pieces a response arrived in, until the - reader disposes it; a caching source therefore reference-counts. -- **Thread safety**: a source is shared by the scans of a session. -- **Cancellation** leaves a source's cache consistent: a segment is present whole or absent. - -The session adds one segment cache and one bound on reads in flight (`MaxConcurrentReads`) for every -file it opens, for sources that do I/O; a mapped file and bytes in memory bypass both. Over a source -that does I/O, a file also serves every segment lying in the tail its open read, which is then -neither asked of the source nor cached: the open reads that tail from a 64-byte boundary, so that a -segment inside keeps the alignment it declares, and a file shorter than the tail is read once. A test-only -HTTP source with injectable latency (`tests/Vorticity.Tests/IO/HttpRangeSegmentSource.cs`) is -the proof that the seam holds for a remote store. - -Three sources ship: `MemoryMappedSegmentSource` (zero-copy), `FileSegmentSource` (positional reads -into aligned buffers) and `MemorySegmentSource` (bytes the caller holds, never copied). A file -opened **from a path** reads positionally until a scan's plan says it will read data, and is mapped -from then on: - -- the open reads the last 8 KiB positionally, or the whole file up to 64 KiB, and a larger footer - whole in a second read: mapping a whole file to read its tail costs more than one read. A file - the open read whole is never mapped: its scans are served from that read; -- the file is opened, locked as `FileShare.Read` locks it, mapped and read with the system's calls - made directly where the platform has them (`NativeFile`): on macOS a third fewer calls than the - framework makes for an open, a scan and a close; -- the first scan whose plan reads any data maps the file, once, and the mapping is the file's until - it is disposed and the last lease on it released. A held mapping reads two to three times faster - than a positional read at every size, and a mapped scan allocates nothing per batch, so no - threshold on the volume is worth its complexity; -- two scans that race to map publish one mapping by compare-and-exchange; a read already under way - finishes positionally; -- the session keeps the mapping once the file is closed (`MappedFileCacheCount`, 64 files by - default): the next open of the same file, known by its device and inode, takes it over with every - page already mapped in it, so a file scanned again pays neither the mapping nor a page fault per - page. A file whose length changed is mapped again, one replaced under its name leaves the cache at - once, and the mapping holds neither the handle nor its lock, so a writer is never kept out. A - file deleted once closed keeps its disk space while it is kept, and `ReleaseMappedFiles` lets - every kept mapping go. On Windows, where a mapped file can be neither deleted nor replaced, - nothing is kept; -- a session created with `MapFiles = false` maps nothing: a path is read by a `FileSegmentSource` - under the session's bound on reads and segment cache, as any source that does I/O. On Linux and - macOS a mapped file cut short by another process faults on the pages past its new end, which - kills the process; a positional read of it throws `VortexFormatException` instead. - -**Coalescing against alignment.** A buffer for a coalesced range `[start, end)` places a segment at -file offset `o` at memory offset `o − start`, arbitrary for an arbitrary `start`. So the start is -rounded down to 64 and the buffer allocated 64-aligned: every segment's offset is a multiple of its -own alignment, at most 64 ([08-semantics.md](08-semantics.md) §6), and so is its distance from the -rounded start. +The public interface is `Vorticity.IO.ISegmentSource` (`IO/SegmentSeam.cs`): a length, a single +read, and a batch read of many ranges into many leases. The batch read is the real entry point. A +scan registers every segment a split needs before reading, which lets a source coalesce nearby +ranges (with a 1 MiB gap by default) and issue them together, and a scan requests each segment at +most once, every block in it decoding from the one lease. That is what makes or breaks performance +on an object store. + +Since object storage lives outside the core, an implementer of the interface must honour four rules: + +- All or nothing: a batch read that fails releases every lease it acquired and throws. Retry policy + belongs to the source. +- Ownership: a lease holds its bytes (one block, or the pieces a response arrived in) until the + reader disposes it, so a caching source has to reference-count. +- Thread safety: a source is shared by all the scans of a session. +- Cancellation leaves a source's cache consistent: a segment is either present whole or absent. + +For sources that do I/O, the session adds one segment cache and one bound on reads in flight +(`MaxConcurrentReads`) shared by every file it opens. A mapped file and bytes in memory bypass both. +Over a source that does I/O, a file also serves every segment lying in the tail its open read, and +those segments are neither requested from the source nor cached. The open reads that tail from a +64-byte boundary, so a segment inside keeps the alignment it declares, and a file shorter than the +tail is read once. A test-only HTTP source with injectable latency +(`tests/Vorticity.Tests/IO/HttpRangeSegmentSource.cs`) proves the interface holds for a remote store. + +Three sources ship. `MemoryMappedSegmentSource` is zero-copy. `FileSegmentSource` does positional +reads into aligned buffers. `MemorySegmentSource` serves bytes the caller holds, and uses them in +place when they start on a 64-byte boundary, otherwise copying them once, when the source is made, +into aligned memory it owns. A file opened from a path reads positionally until a scan's plan says it +will read data, and is mapped from then on: + +- The open reads the last 8 KiB positionally, or the whole file up to 64 KiB, and a larger footer + whole in a second read, because mapping a whole file just to read its tail costs more than one + read. A file the open read whole is never mapped, and its scans are served from that read. +- The file is opened, locked the way `FileShare.Read` locks it, mapped and read with the system calls + made directly where the platform has them (`NativeFile`). On macOS that is a third fewer calls than + the framework makes for an open, a scan and a close. +- The first scan whose plan reads any data maps the file, once, and the mapping belongs to the file + until it is disposed and the last lease on it is released. A held mapping reads two to three times + faster than a positional read at every size, and a mapped scan allocates nothing per batch, so no + threshold on volume is worth its complexity. +- Two scans that race to map publish a single mapping by compare-and-exchange, and a read already + under way finishes positionally. +- The session keeps the mapping after the file is closed (`MappedFileCacheCount`, 64 files by + default). The next open of the same file, recognized by its device and inode (on Windows by its + volume and file id), takes it over with every page already mapped, so a file scanned again pays + neither the mapping nor a page fault per page. A file whose length changed is mapped again, one + replaced under its name leaves the cache at once, and the mapping holds neither the handle nor its + lock, so a writer is never kept out. A file deleted after closing keeps its disk space as long as + its mapping is kept, and `ReleaseMappedFiles` lets every kept mapping go. On Windows a kept file + can be deleted and replaced under its name, but another writer cannot truncate or overwrite it + from its start until the session lets it go, and this library's own writers release it themselves. +- A session created with `MapFiles = false` maps nothing. A path is then read by a + `FileSegmentSource` under the session's read bound and segment cache, like any source that does + I/O. On Linux and macOS, a mapped file truncated by another process faults on the pages past its + new end, which kills the process, whereas a positional read of it throws + `VortexFormatException`. + +Coalescing has to respect alignment. A buffer for a coalesced range `[start, end)` places a segment +at file offset `o` at memory offset `o − start`, which is arbitrary for an arbitrary `start`. So the +start is rounded down to 64 and the buffer allocated 64-aligned. Every segment's offset is a +multiple of its own alignment, at most 64 ([resource caps](08-semantics.md#6-resource-caps)), and so +is its distance from the rounded start. ### 3.6 Two phases: materialize, then execute -Reading a split is two calls: an asynchronous one that reads every segment the split needs, -coalesced, and a synchronous one that decodes. A decoder therefore never awaits: threading -`ValueTask` through every decoder would put state machines on the hot path to serve one case, the -lazily loaded values of a dictionary layout, which belongs to the layout level, where the +Reading a split takes two calls: an asynchronous one that reads every segment the split needs, +coalesced, and a synchronous one that decodes. A decoder therefore never awaits. Threading +`ValueTask` through every decoder would put state machines on the hot path to serve a single case, +the lazily loaded values of a dictionary layout, and that case belongs at the layout level, where the materialization decides what to fetch. ### 3.7 Async enumeration -A compiler-generated `async` enumerator allocates its state machine and possibly more per -`MoveNextAsync`. One allocation per scan is acceptable; one per batch is not. The scan's enumerator +A compiler-generated `async` enumerator allocates its state machine, and possibly more, per +`MoveNextAsync`. One allocation per scan is acceptable, one per batch is not. The scan's enumerator is therefore written by hand over `ManualResetValueTaskSourceCore`, the pattern -`System.Threading.Channels` uses, and so is every enumerator of a query's flow: an operator's -stream of batches, a result's, the scan over a result, a dataset's walk of its objects. A view of -rows over those batches — values, records — steps synchronously within a batch and awaits only at -the next one: an `await` per batch, never per row, which is also what lets a batch be borrowed, a -span never crossing an `await`. +`System.Threading.Channels` uses, and so is every enumerator of a query's flow: an operator's stream +of batches, a result's, the scan over a result, a dataset's walk of its objects. A view of rows over +those batches (values, records) steps synchronously within a batch and only awaits at the next one. +That means one `await` per batch, never per row, which is also what allows a batch to be borrowed, +since a span never crosses an `await`. ### 3.8 The write sink -The writer's seam is a `PipeWriter`: `CreateWriter(path)` builds one over a file handle, +The writer writes to a `PipeWriter`. `CreateWriter(path)` builds one over a file handle, `PipeWriter.Create(stream)` covers a stream, and an object store provides its own. The writer writes -**forward only, in one pass**: segments, then the metadata FlatBuffers, the postscript and the end -record, the order the format is designed for. Padding is computed as each segment is written, so -no file is ever materialized whole; bytes leave the pipe at `FlushAsync` and `CompleteAsync`, so a -sink that applies backpressure slows the producer instead of growing a buffer. A writer that gives -a file up completes the caller's pipe with an error, so that whatever it feeds knows the bytes are -not a file. The postscript's 65 527-byte ceiling is checked when user metadata is given, not at the -last write. +forward only, in one pass: segments, then the metadata FlatBuffers, the postscript and the end +record, the order the format is designed for. Padding is computed as each segment is written, so no +file is ever materialized whole. Bytes leave the pipe at `FlushAsync` and `CompleteAsync`, so a sink +that applies backpressure slows the producer instead of growing a buffer. A writer that gives up on a +file completes the caller's pipe with an error, so whatever the pipe feeds knows the bytes are not a +file. The postscript's 65 527-byte ceiling is checked when user metadata is given, not at the last +write. ## 4. Performance invariants -Each is a test, not a guideline: +Each one is a test, not a guideline: -1. **No managed allocation per batch** in steady state on a full scan (`ScanAllocationTests`), and - a fixed ceiling per operation and per written file ([05-benchmarks.md](05-benchmarks.md) §5). -2. **Zero copy**: a plain column's decode is a view of the segment's bytes +1. No managed allocation per batch in steady state on a full scan (`ScanAllocationTests`), and a + fixed ceiling per operation and per written file ([allocations](05-benchmarks.md#5-allocations)). +2. Zero copy: a plain column's decode is a view of the segment's bytes (`LeafDecoderTests.PrimitiveReadsValuesZeroCopy`). -3. **No per-row dispatch** in decode and write loops: every kernel is generic and specialized +3. No per-row dispatch in decode and write loops: every kernel is generic and specialized (`PerRowDispatchTests`). -4. **Scalar fallback**: CI runs again with hardware intrinsics disabled the test classes that reach - every scalar fallback the suite does (`tests/scalar-pass.txt`), and the conformance corpus. -5. **Native AOT**: CI publishes `vxdump` ahead of time and runs it over the corpus. -6. **The first batch waits for the first split**: the time to first batch of every streaming query - — a scan, a projection, a `Distinct`, a group by on a key that streams — does not grow with the - file or the dataset, locally and over a source with latency - ([16-queries.md](16-queries.md) §13). -7. **Memory is the window and the open state**: a streaming group by's peak stays flat as its - groups grow, a top-k's follows `k`, a blocking group by's follows its groups and not the degree - times them (`LiveMemoryTests`). -8. **Answers do not depend on the cut**: every aggregate that does not depend on the order of rows - by definition, float sums included, is the same bits at every degree, in every order of rows, - and under every cut into chunks, files or objects. +4. Scalar fallback: CI runs again, with hardware intrinsics disabled, the test classes that reach + every scalar fallback the suite covers (`tests/scalar-pass.txt`), plus the conformance corpus. +5. Native AOT: CI publishes `vxdump` ahead of time and runs it over the corpus. +6. The first batch only waits for the first split. The time to first batch of every streaming query + (a scan, a projection, a `Distinct`, a group by on a key that streams) does not grow with the file + or the dataset, locally and over a source with latency ([how queries are + tested](16-queries.md#13-how-it-is-tested)). +7. Memory is the window plus the open state. A streaming group by's peak stays flat as its groups + grow, a top-k's follows `k`, and a blocking group by's follows its groups, not the degree times its + groups (`LiveMemoryTests`). +8. Answers do not depend on how the data is split. Every aggregate whose definition does not depend + on row order, float sums included, gives the same bits at every degree, in every row order, and + under every split into chunks, files or objects. ## 5. Error handling -- On the hot path, `Try` forms and return codes; no exception for control flow. -- At the boundary: `VortexFormatException` for a malformed file; `VortexUnsupportedException` for - a component this build does not implement, which always names the **id and its kind** (`Array`, +- On the hot path, `Try` forms and return codes, never exceptions for control flow. +- At the boundary, `VortexFormatException` for a malformed file, and `VortexUnsupportedException` for + a component this build does not implement, which always names the id and its kind (`Array`, `Layout`, `DType`, `Aggregate`, `Compression`, `Encryption`, `Index` or `Feature`), since that is - what upstream's troubleshooting asks for; `VortexSchemaException` when a record or a column does - not fit the file. -- Unknown components fail lazily, only the read that needs them, and there is no option to make - them fail at open ([08-semantics.md](08-semantics.md) §4). `VortexFile.ArrayEncodings` and - `LayoutEncodings` list what a footer declares, each with whether this build supports it. + what upstream's troubleshooting asks for. `VortexSchemaException` when a record or a column does + not fit the file, and `VortexMemoryException` when a query outgrows its memory or scratch budget. +- Unknown components fail lazily, only in the read that needs them, and there is no option to make + them fail at open ([unknown + components](08-semantics.md#4-unknown-components-resolve-lazily-fail-on-use)). + `VortexFile.ArrayEncodings` and `LayoutEncodings` list what a footer declares, each with whether + this build supports it. ## 6. Parser safety A file is untrusted input. - Every offset and length read from the file is checked against the real size before any access. -- FlatBuffers verification follows the format's actual rules, stated precisely because the obvious - phrasing both over- and under-rejects: - - **uoffsets** (table and vector references) are unsigned and point forward: `uoffset > 0` and the - target in bounds. Cycles are therefore impossible by construction, and no visited set is kept. - - Forward-only offsets exclude cycles but **not sharing**: two slots may resolve to one target, +- FlatBuffers verification follows the format's actual rules, stated precisely here because the + obvious phrasing both over- and under-rejects: + - uoffsets (table and vector references) are unsigned and point forward: `uoffset > 0` and the + target in bounds. Cycles are impossible by construction, so no visited set is kept. + - Forward-only offsets exclude cycles but not sharing. Two slots may resolve to the same target, so the decoded graph is a DAG, and a reader that recurses once per edge is exponential in depth. - A few hundred bytes of shared children pass every bounds and depth rule while describing 2⁶³ + A few hundred bytes of shared children can pass every bounds and depth rule while describing 2⁶³ visits. Total work therefore has its own bound, separate from depth: every table visited is - charged against `VortexLimits.MaxFlatBufferTables`, as the reference verifiers' `max_tables`. - - **soffsets** (vtable references) are signed: a vtable may precede or follow its table, and both + charged against `VortexLimits.MaxFlatBufferTables`, like the reference verifiers' `max_tables`. + - soffsets (vtable references) are signed. A vtable may precede or follow its table, and both directions are bounded. - - **vtable sharing** between tables is legal and routine; a verifier that refused to revisit a + - Vtable sharing between tables is legal and routine, so a verifier that refused to revisit a position would reject valid files. - - Recursion depth is capped for layout trees, array trees and dtypes: a 10 000-deep nested struct - blows the stack while the schema is parsed, before any data is read. -- No allocation is sized by a file-supplied value without a cap; the caps are one table, - [08-semantics.md](08-semantics.md) §6, not a choice per call site. `alignment_exponent` is a `u8`, - so an unchecked file could demand `2²⁵⁵`. -- Semantic hints are classified ([08-semantics.md](08-semantics.md) §5): what memory safety depends - on is validated unconditionally, what only correctness depends on is covered by the threat model - or by the opt-in `VortexOpenOptions.VerifyStatistics`. + - Recursion depth is capped for layout trees, array trees and dtypes. A 10 000-deep nested struct + would otherwise blow the stack while the schema is parsed, before any data is read. +- No allocation is sized by a file-supplied value without a cap, and the caps live in one table + ([resource caps](08-semantics.md#6-resource-caps)), not in a choice per call site. + `alignment_exponent` is a `u8`, so an unchecked file could demand `2²⁵⁵`. +- Semantic hints are classified ([untrusted + hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). What memory safety depends on is + validated unconditionally, and what only correctness depends on is covered by the threat model or + by the opt-in `VortexOpenOptions.VerifyStatistics`. - Children's row counts are checked against their parents, never assumed. diff --git a/docs/design/04-conformance.md b/docs/design/04-conformance.md index 8ddc331c..3d5a95fb 100644 --- a/docs/design/04-conformance.md +++ b/docs/design/04-conformance.md @@ -1,6 +1,6 @@ # Testing and conformance -A format implementation tested only against itself proves nothing: a round trip through its own +A format implementation tested only against itself proves nothing. A round trip through its own writer and reader is self-consistent and can be uniformly wrong. So every claim about bytes is anchored in the Rust reference, in both directions. Tests use xUnit v3 on Microsoft.Testing.Platform. @@ -15,16 +15,17 @@ anchored in the Rust reference, in both directions. Tests use xUnit v3 on Micros ## 2. Unit and property tests -- The FlatBuffers and Protobuf readers on hand-built bytes: a missing vtable field reads its - default, an empty vector, an absent optional table, and a truncated or over-long input fails with - `VortexFormatException` and nothing else. -- Every SIMD kernel against a scalar reference over random inputs; in CI, again with hardware - intrinsics disabled, every test class that reaches a scalar fallback and the conformance corpus. -- The FastLanes transposition: `untranspose(transpose(i)) == i` for every `i` of a block, as the - reference asserts. -- Every operator against the full materialization it replaces: a projection, a filter, a take, a - count or an aggregate equals the same operation applied to every decoded row. -- The allocation ratchets of [05-benchmarks.md](05-benchmarks.md) §5. +- The FlatBuffers and Protobuf readers are tested on hand-built bytes: a missing vtable field reads + its default, an empty vector, an absent optional table, and a truncated or over-long input fails + with `VortexFormatException` and nothing else. +- Every SIMD kernel is checked against a scalar reference over random inputs. In CI, every test class + that reaches a scalar fallback, and the conformance corpus, run again with hardware intrinsics + disabled. +- The FastLanes transposition satisfies `untranspose(transpose(i)) == i` for every `i` of a block, as + the reference asserts. +- Every operator is checked against the full materialization it replaces: a projection, a filter, a + take, a count or an aggregate must equal the same operation applied to every decoded row. +- The allocation ratchets described in [allocations](05-benchmarks.md#5-allocations). ## 3. The golden corpus @@ -33,90 +34,95 @@ anchored in the Rust reference, in both directions. Tests use xUnit v3 on Micros core editions from `core2025.05.0` to `core2026.08.3`. The matrix is deliberate: - every dtype × nullability × row counts of 0, 1, 1 023, 1 024, 1 025, 8 191, 8 192 and 8 193, - because the block edges at 1 024 (FastLanes) and 8 192 (the default block) are where bugs live; -- every encoding forced on its own by an explicit compressor configuration, so that each is - exercised in isolation rather than whichever the sampling happens to pick; -- the default configuration on realistic distributions, which is the production path; -- edge cases: all-null and constant columns, a single-value dictionary, strings over a mebibyte, - deep nesting, a root that is not a struct, a file with no dtype segment, user metadata, and the - layouts a default writer never emits (an inlined array tree, a dictionary layout). - -Beside each file, a **sidecar of expected values** produced by the Rust reader (`SIDECAR.md` gives + because the block edges at 1 024 (FastLanes) and 8 192 (the default block) are where bugs live +- every encoding forced on its own by an explicit compressor configuration, so that each one is + exercised in isolation rather than whichever the sampling happens to pick +- the default configuration on realistic distributions, which is the production path +- edge cases: all-null and constant columns, a single-value dictionary, strings over a mebibyte, deep + nesting, a root that is not a struct, a file with no dtype segment, user metadata, and the layouts + a default writer never emits (an inlined array tree, a dictionary layout) + +Next to each file sits a sidecar of expected values produced by the Rust reader (`SIDECAR.md` gives its format): the dtype, the layout tree, the file statistics, the zone maps and every value, with floats compared by their bits, so a wrong NaN payload or a `-0.0` read as `+0.0` fails. The tests -therefore need no Rust toolchain. A `manifest.json` records the Vortex version, the edition and the -encodings of every file, and its SHA-256. +therefore need no Rust toolchain. A `manifest.json` records the Vortex version, the edition, the +encodings and the SHA-256 of every file. -**The coverage gate**: the union of the corpus's array and layout ids must cover every component +A coverage gate checks that the union of the corpus's array and layout ids covers every component this library claims to read, or the build fails. What no Rust writer can produce is forged by -patching bytes, in `forged/`: the legacy `vortex.stats` layout, and the unknown ids of §6. +patching bytes, in `forged/`: the legacy `vortex.stats` layout, and the unknown ids of the +[negative tests](#6-negative-and-forward-compatibility-tests). ## 4. The cross-check: Rust reads what we write Golden files prove that we read what Rust wrote. They do not prove that Rust reads what we write, -and a writer tested only by its own reader reads back its own mistakes. So a CI job has this -library write the whole corpus out again, and the reference read every file and compare it scalar -by scalar with the source (`bench/crosscheck.sh` locally, `verify_written` in the generator). The -writes rotate the index policies, append in pieces, rewrite files as a dataset's compaction does, -and chunk some columns apart, so that every structure this writer can emit meets a strict Rust -reader. +and a writer tested only by its own reader reads back its own mistakes. So a CI job has this library +write the whole corpus out again, and the reference reads every file and compares it scalar by +scalar with the source (`bench/crosscheck.sh` locally, `verify_written` in the generator). The writes +rotate the index policies, append in pieces, rewrite files the way a dataset's compaction does, and +chunk some columns apart, so every structure this writer can emit meets a strict Rust reader. -The benchmark shim adds an in-process check ([05-benchmarks.md](05-benchmarks.md) §6): both readers -return the same rows and the same checksum of every decoded value on the same files. +The benchmark shim adds an in-process check ([how it is measured](05-benchmarks.md#6-how-it-is-measured)): +both readers must return the same rows and the same checksum of every decoded value on the same +files. ## 5. Fuzzing -- **Target**: open and full scan, over mutated corpus files. -- **Invariant**: only `VortexFormatException` or `VortexUnsupportedException` escapes. An access - violation, an index out of range, an out-of-memory, a hang or a silently wrong value is a bug. -- **Structure-aware mutation**: a naive bit flip on an offset-based format dies at the first bounds - check. The mutator (`tests/Vorticity.Fuzz`) rewrites words in the tail where the offsets live, - truncates, writes plausible Protobuf tags and long varints, and puts extreme values in the - lengths and counts that memory safety depends on ([08-semantics.md](08-semantics.md) §5). -- **Caps are invariants**: a file declaring a `2²⁵⁵` alignment or a segment that expands to 100 GiB - must fail cleanly and fast. -- **Where it runs**: a smoke campaign in every test run (`FuzzSmokeTests`), and 100 000 mutations - with a fixed seed in CI, so a finding reproduces from the log. +- The target is an open followed by a full scan, over mutated corpus files. +- The invariant is that only `VortexFormatException` or `VortexUnsupportedException` escapes. An + access violation, an index out of range, an out-of-memory, a hang or a silently wrong value is a + bug. +- Mutation is structure-aware, because a naive bit flip on an offset-based format dies at the first + bounds check. The mutator (`tests/Vorticity.Fuzz`) rewrites words in the tail where the offsets + live, truncates, writes plausible Protobuf tags and long varints, and puts extreme values in the + lengths and counts that memory safety depends on ([untrusted + hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). +- The caps are invariants too. A file declaring a `2²⁵⁵` alignment, or a segment that expands to + 100 GiB, must fail cleanly and fast. +- A smoke campaign runs in every test run (`FuzzSmokeTests`), and CI runs 100 000 mutations with a + fixed seed, so any finding reproduces from the log. ## 6. Negative and forward-compatibility tests Two fixtures carry an encoding or a layout id that exists in no edition (`forged/negative/unknown_*_id.vortex`): -- reading a **projected** column that uses one fails with `VortexUnsupportedException` naming the - id and its kind; -- scanning **without** projecting it succeeds. This is what locks in lazy resolution - ([08-semantics.md](08-semantics.md) §4): without it, an open path "simplified" into an eager - failure would pass. +- reading a projected column that uses one fails with `VortexUnsupportedException`, naming the id + and its kind +- scanning without projecting it succeeds -On the pruning side, filter tests run with pruning and indexes on and off and must return the same -rows, which is what enforces that pruning never removes a matching row. +The second test is what locks in lazy resolution ([unknown +components](08-semantics.md#4-unknown-components-resolve-lazily-fail-on-use)). Without it, an open +path "simplified" into an eager failure would still pass. + +On the pruning side, filter tests run with pruning and indexes both on and off and must return the +same rows, which is what enforces that pruning never removes a matching row. ## 7. The row encoding -The row encoder's contract is one property, byte order equals tuple order, and a format match -([06-row-encoding.md](06-row-encoding.md)): +The row encoder's contract is one property, byte order equals tuple order, plus a byte-for-byte +format match ([06-row-encoding.md](06-row-encoding.md)): -- **The order property, randomized**: random tuples over a random schema and random sort options; +- The order property is randomized: random tuples over a random schema with random sort options, and sorting by `memcmp` of the encoded rows must give exactly the permutation of a tuple sort. -- **Adversarial values**: `-0.0` and `+0.0`, NaN bit patterns, `i64::MIN`, empty against null - against one-byte strings, strings of 31, 32, 33 and 64 bytes around the block size, prefix pairs, - and nulls under nested structs with garbage in their children. -- **Byte for byte against Rust**: `tools/row-vectors` encodes generated tables with the reference - `vortex-row` and records the bytes in `tests/Vorticity.Conformance/row-vectors`; the tests +- Adversarial values are covered: `-0.0` and `+0.0`, NaN bit patterns, `i64::MIN`, empty against + null against one-byte strings, strings of 31, 32, 33 and 64 bytes around the block size, prefix + pairs, and nulls under nested structs with garbage in their children. +- The bytes are checked against Rust. `tools/row-vectors` encodes generated tables with the reference + `vortex-row` and records the bytes in `tests/Vorticity.Conformance/row-vectors`, and the tests compare every one. `vortex-row` is not published on crates.io, so that crate takes the Vortex repository at tag `0.86.1` as a git dependency, the only one in the repository, and its output is - committed. Being order-compatible is not enough: two encoders can each be consistent and still - disagree, which would break any comparison across languages. + committed. Being order-compatible is not enough, since two encoders can each be consistent and + still disagree, which would break any comparison across languages. ## 8. Read forever Files written under the oldest edition must stay readable, and the corpus holds files written under -each edition from `core2025.05.0` on: the files of the older editions are what catches a reader -that hard-codes the encodings of the newest edition. They are written by the pinned 0.86.1 generator restricted to -each edition, which is the same component set as the 0.36.0 floor with a modern writer; files from -a 0.36.0 writer itself would need a second, older generator. +each edition from `core2025.05.0` on. The files of the older editions are what catches a reader that +hard-codes the encodings of the newest edition. They are written by the pinned 0.86.1 generator +restricted to each edition, which gives the same component set as the 0.36.0 floor with a modern +writer. Files from a 0.36.0 writer itself would need a second, older generator. -A CI job re-vendors the upstream schemas and reports any drift; a change there is a format change, +A CI job vendors the upstream schemas again and reports any drift. A change there is a format change, reviewed against [90-registry.md](90-registry.md), and the corpus is regenerated with the generator pinned to the new version. diff --git a/docs/design/05-benchmarks.md b/docs/design/05-benchmarks.md index 047519b8..6810120c 100644 --- a/docs/design/05-benchmarks.md +++ b/docs/design/05-benchmarks.md @@ -1,226 +1,238 @@ # Benchmarks -How Vorticity is measured against Vortex's Rust implementation, and its kernels against their -baselines: what is compared (§1), what a caller sees (§2), the ratios the gates hold in one -process (§3), each decoder (§4), allocations (§5), and the instruments (§6). The figures are on one -page, [the benchmark page](../guide/benchmarks.md), each section written by the instrument that -measures it; [bench/README.md](../../bench/README.md) says how to run each one. +How Vorticity is measured against Vortex's Rust implementation, and how its kernels are measured +against their baselines. This page covers [what is compared](#1-the-comparison), [what a caller +sees](#2-what-a-caller-sees), [the ratios the gates hold in one +process](#3-in-one-process-after-warm-up), [each decoder](#4-decoders-per-encoding), +[allocations](#5-allocations) and [the instruments](#6-how-it-is-measured). The figures are all on +[the benchmark page](../guide/benchmarks.md), each section written by the instrument that measures +it, and [bench/README.md](../../bench/README.md) explains how to run each one. ## 1. The comparison -* **Against Rust.** The reference is Vortex's own implementation, the one the format was designed - around. A .NET Parquet library would measure a difference of format, not of implementation. -* **The same work.** Both sides read the same bytes and must return the same rows, or the run fails. - Every call opens its file and maps it anew, on both sides, and reads it where it lies: Rust through - `open_buffer` over a `memmap2` mapping, ours through a session that keeps no mapping once a file is - closed. Both decode every value they return to its plain form, except a constant column, which both - keep as one value and a length: our reader's form, and upstream's `Columnar`. A write hands its - bytes to a sink that counts them and keeps none, on both sides. A count is a count on both sides, - and a read the reference has no index for is asked of it as the rows it would have to find. - Neither side keeps a segment cache. -* **Rust's figure is its faster setting.** By default Rust cuts a chunk of more than 100 000 rows - into splits of 100 000, and its reader decodes the whole chunk again for every split: on the - per-encoding corpus, files of one chunk of a million rows, that is ten decodes of each, and the - string-heavy files took ten times as long. Asked for one split per chunk it decodes each once, but - builds a chunk made of smaller arrays in one piece, three to six times slower on those. Neither - wins everywhere, so on one core every Rust figure is the faster of the two, measured on the spot, - file by file and axis by axis; on all cores Rust keeps its default, whose splits spread its work. -* **Each side built for speed.** Vorticity as a Native AOT binary for the machine's instruction - set. Vortex 0.86.1 built as upstream builds its own benchmarks: mimalloc, `-C target-cpu=native`, - one codegen unit, no LTO, upstream's `release_debug` profile and the `RUSTFLAGS` of its benchmark - workflows. -* **The same threads.** Every scenario runs on one core — Rust's single-threaded runtime, our scans - at one lane — and on all of them: a Tokio worker per processor for Rust, a lane per processor for - our scans and as many threads for our writer. -* **A speedup on every page**: Rust's time over Vorticity's, so that above 1.00× Vorticity took - less, and in a kernel's table its baseline's time over its own. The gates hold the inverse, - Vorticity's time over Rust's, under a ceiling (§3); their consoles print that ratio. -* **The targets**: within 2× of Rust's time on a full scan and 1.5× on a decoder, per instruction - set, and files no larger than 105 % of Rust's. The benchmark page says where each time stands; - `WrittenSizeTests` holds the bytes the writer gives the whole conformance corpus under a ceiling - below the size target. -* **One machine per figure.** Each section of the benchmark page names the machine it was measured - on: an Apple M4 Pro, arm64 and the `Vector128` paths, for [the benchmark page](../guide/benchmarks.md), - and an AMD Ryzen 9 7950X, x64 with AVX-512, for [its twin](../guide/benchmarks-x64.md). CI builds - and tests on x64 and arm64 but runs no benchmark. +* The reference is Rust. Vortex's own implementation is the one the format was designed around, and + a .NET Parquet library would measure a difference of format, not of implementation. +* Both sides do the same work. They read the same bytes and must return the same rows, or the run + fails. Every call opens its file and maps it anew, on both sides, and reads it where it lies: Rust + through `open_buffer` over a `memmap2` mapping, ours through a session that keeps no mapping once a + file is closed. Both decode every value they return to its plain form, except a constant column, + which both keep as one value and a length (our reader's form, and upstream's `Columnar`). A write + hands its bytes to a sink that counts them and keeps none, on both sides. A count is a count on + both sides, and a read the reference has no index for is asked of it as the rows it would have to + find. Neither side keeps a segment cache. +* Rust's figure is its faster setting. By default Rust cuts a chunk of more than 100 000 rows into + splits of 100 000, and its reader decodes the whole chunk again for every split. On the + per-encoding corpus, files of one chunk of a million rows, that means ten decodes of each, and the + string-heavy files took ten times as long. Asked for one split per chunk, it decodes each chunk + once but builds a chunk made of smaller arrays in one piece, three to six times slower on those + files. Neither setting wins everywhere, so on one core every Rust figure is the faster of the two, + measured on the spot, file by file and axis by axis. On all cores Rust keeps its default, whose + splits spread its work. +* Each side is built for speed. Vorticity runs as a Native AOT binary for the machine's instruction + set. Vortex 0.86.1 is built the way upstream builds its own benchmarks: mimalloc, + `-C target-cpu=native`, one codegen unit, no LTO, upstream's `release_debug` profile and the + `RUSTFLAGS` of its benchmark workflows. +* Both sides use the same threads. Every scenario runs on one core (Rust's single-threaded runtime, + our scans at one lane) and on all of them (a Tokio worker per processor for Rust, a lane per + processor for our scans and as many threads for our writer). +* Every page shows a speedup: Rust's time over Vorticity's, so above 1.00× Vorticity took less, and + in a kernel's table, its baseline's time over its own. The gates hold the inverse, Vorticity's time + over Rust's, under a ceiling ([in one process](#3-in-one-process-after-warm-up)), and their consoles + print that ratio. +* The targets are within 2× of Rust's time on a full scan and within 1.5× on a decoder, per + instruction set, and files no larger than 105 % of Rust's. The benchmark page shows where each time + stands, and `WrittenSizeTests` holds the bytes the writer produces for the whole conformance corpus + under a ceiling below the size target. +* Each figure comes from one machine. Each section of the benchmark page names the machine it was + measured on: an Apple M4 Pro, arm64 with the `Vector128` paths, for [the benchmark + page](../guide/benchmarks.md), and an AMD Ryzen 9 7950X, x64 with AVX-512, for [its + twin](../guide/benchmarks-x64.md). CI builds and tests on x64 and arm64 but runs no benchmark. ## 2. What a caller sees -Seven actions a program performs on a file, each timed in a fresh process from opening the file to -its last row: the median of five runs, the page cache warm. The data is one table of 2^20 rows and -one of ten times that, four columns: `monotone`, an increasing `i64`; `value`, an `f64`; `label`, -one of five short strings; `flag`, a `bool` with nulls. In their plain form, Arrow's layout, that is -25.5 bytes a row. - -**Two files of the same rows.** A Vortex file can store a column in many encodings, each writer -picks its own, and what a read costs depends on the pick. So the rows are written twice, once by -each implementation's writer, and both readers read both files; the benchmark page lists what each -writer chose at each size. Our writer stores `value` as zstd frames where Rust's chooses ALP-RD: a -much smaller file, slower to scan and to take from. Which of the two `Auto` should pick is the -writer's decision ([11-write-strategy.md](11-write-strategy.md) §3.4); -[the encodings, column by column](../guide/benchmarks.md#encodings-column-by-column) measure the -trade. - -The actions: `open` the file and read no rows; `scan` every column of every row; `project` one -column of four; `filter-narrow`, the rows of a band of `monotone` holding about one in a hundred; -`filter-wide`, a band holding about half; `take` a thousand rows spread across the file; `write`, -read the file and write it back out. `append`, a tenth of the rows appended to a copy of the file, -is ours alone: Rust's harness has no entry point for it. - -**GB/s** is the plain size of the rows returned over the time. **Allocated** is what Vorticity +Seven actions a program performs on a file are each timed in a fresh process, from opening the file +to its last row: the median of five runs, with the page cache warm. The data is one table of 2^20 +rows and one of ten times that, with four columns: `monotone`, an increasing `i64`, `value`, an +`f64`, `label`, one of five short strings, and `flag`, a `bool` with nulls. In their plain form +(Arrow's layout) that is 25.5 bytes per row. + +The same rows are written as two files. A Vortex file can store a column in many encodings, each +writer picks its own, and what a read costs depends on the pick. So the rows are written twice, once +by each implementation's writer, and both readers read both files. The benchmark page lists what each +writer chose at each size. Our writer stores `value` as zstd frames where Rust's chooses ALP-RD, +which gives a much smaller file that is slower to scan and to take from. Which of the two `Auto` +should pick is the writer's decision ([choosing +encodings](11-write-strategy.md#34-choose-exact-verdicts-then-bounded-trials)), and [the encodings, +column by column](../guide/benchmarks.md#encodings-column-by-column) measure the trade-off. + +The actions are: `open` the file and read no rows, `scan` every column of every row, `project` one +column of four, `filter-narrow` (the rows of a band of `monotone` holding about one in a hundred), +`filter-wide` (a band holding about half), `take` a thousand rows spread across the file, and +`write`, which reads the file and writes it back out. `append`, a tenth of the rows appended to a +copy of the file, is ours alone, because Rust's harness has no entry point for it. + +GB/s is the plain size of the rows returned divided by the time. Allocated is what Vorticity allocates on its managed heap for one call, once warm: the median of six calls in a process that has -made two before them. **Peak** is each process's peak resident memory. +made two before them. Peak is each process's peak resident memory. [Reading and writing a table](../guide/benchmarks.md#reading-and-writing-a-table-process-against-process) -has the figures, on one core and on all of them, and what a first call costs on the JIT. Reading -them: - -* **A take from our file inflates a zstd frame per row.** Every row taken inflates the frame that - holds its `value`, 64 KiB for one value, and the inflating is nearly all of the take. -* **On all cores** a take keeps three splits a lane decoded ahead of its consumer, so the lanes - that finish first do not wait for the slowest; our writer summarizes its columns and compresses - its zstd frames on every core, and chooses and writes the encodings on one. -* **Allocations on all cores**: buffers rented from the shared array pool on one thread and returned - on another miss its per-thread caches, so a call now and then allocates several times its median. - It is a known defect, not a cost of the work. -* **Native AOT, and the JIT.** The tables are the native build's. On the JIT, a fresh process - compiles the code the first time it runs it, which the page's last table measures; a process that - stays up pays that once, and then reads the speedups of §3 and §4. Starting a process is not in the - figures, and the page gives it apart. +has the figures, on one core and on all of them, and what a first call costs on the JIT. A few +points help reading them: + +* A take from our file inflates a zstd frame per row. Every row taken inflates the frame that holds + its `value`, 64 KiB for one value, and the inflating is nearly all of the take. +* On all cores, a take keeps three splits per lane decoded ahead of its consumer, so the lanes that + finish first do not wait for the slowest. Our writer summarizes its columns and compresses its + zstd frames on every core, and chooses and writes the encodings on one. +* On all cores, buffers rented from the shared array pool on one thread and returned on another miss + the pool's per-thread caches, so a call now and then allocates several times its median. This is a + known defect, not a cost of the work. +* The tables come from the native build. On the JIT, a fresh process compiles code the first time it + runs it, which the page's last table measures. A process that stays up pays that once, and then + sees the speedups of [in one process](#3-in-one-process-after-warm-up) and + [decoders](#4-decoders-per-encoding). Process start-up is not in the figures, and the page gives it + separately. ## 3. In one process, after warm-up `--ratio-check` calls both implementations in one process, Rust's through a C ABI, in turn and -against one clock: at least 21 rounds, until the 95 % interval of the per-round ratios is within -5 % of their median. Our side runs on the JIT, warmed. Each axis is a gate: its ratio, our time over -Rust's, must stay under a ceiling. The page reads the same rounds the other way up, the median of +against one clock: at least 21 rounds, until the 95 % interval of the per-round ratios is within 5 % +of their median. Our side runs on the JIT, warmed up. Each axis is a gate, whose ratio (our time over +Rust's) must stay under a ceiling. The page reads the same rounds the other way up, as the median of the per-round speedups and its interval. The first eight axes read a corpus file Rust's writer made, `containers/zoned_many_zones_nulls`: 65 536 rows, five columns, 64 zones. The key-order, string and -band axes read files of 65 536 rows our writer makes for them; the last two, generated tables of a -million rows and six columns, and of fifty columns. +band axes read files of 65 536 rows that our writer makes for them, and the last two read generated +tables of a million rows with six columns, and of fifty columns. [In one process](../guide/benchmarks.md#in-one-process-after-warm-up) has the figures. -* **`open to first batch` is a latency.** Both sides decode their first batch. Rust's first batch - costs most of what its whole scan does on this file, 1.4 ms of 1.6, where ours costs - a fifth of ours. The speedup is on the time to the first row, and nothing more. -* **Where the reference has no counterpart, it answers the same question its own way.** A - key-ordered read of a band is held against Rust's filtered scan of that band, which returns the - same rows in file order; an exact count from the index, against Rust's count of the band with no - column decoded. The speedup then prices the index, which is what a caller would weigh. -* The gate has seven more axes, which the page leaves out: two time Rust's scan without decoding, - which asks another question; one holds the uncorrelated key-order read to a take of 64 rows whose - positions the reference is given, the floor that read cannot go under; and four read one file as +* `open to first batch` is a latency. Both sides decode their first batch. Rust's first batch costs + most of what its whole scan does on this file, 1.4 ms of 1.6, where ours costs a fifth of our full + scan. The speedup is about the time to the first row, and nothing more. +* Where the reference has no counterpart, it answers the same question its own way. A key-ordered + read of a band is compared with Rust's filtered scan of that band, which returns the same rows in + file order, and an exact count from the index with Rust's count of the band with no column decoded. + The speedup then prices the index, which is what a caller would weigh. +* The gate has seven more axes that the page leaves out. Two time Rust's scan without decoding, which + asks a different question. One holds the uncorrelated key-order read against a take of 64 rows + whose positions the reference is given, the floor that read cannot go under. Four read one file as each writer wrote it. [bench/README.md](../../bench/README.md) describes them. ## 4. Decoders, per encoding -One file per encoding, and per notable shape of one: 57 files of a million rows -(`bench/gen-throughput.sh`), written by Rust's writer. Each side scans each file 12 times in a -process of its own, on one core, every value decoded to its plain form; a figure is the median of the -last 10 scans and of 3 processes, Native AOT against Rust's `vxbench` binary: what a decoder costs -once warm. [Decoding, per encoding](../guide/benchmarks.md#decoding-per-encoding) has all 57. - -* **Past what memory can move is not decoding.** A column stored in its plain form is handed out as - a view of the mapped file on both sides, nothing copied: the figure measures the walk of the - layout, and the speedup compares two walks. -* **The widest gaps are UTF-8 validation.** In `varbin`, `varbinview`, `struct` and the tables, the - stored arrays are already in their plain form, and the time is the check that every string is - UTF-8, which both sides make: a sampled profile of Rust's scan of `varbinview` spends it in - `from_utf8` and `validate_view`, under `VarBinViewData::try_new`. -* **Each scan opens the file again and maps it anew**, on both sides. - -Take and write per encoding are measured in one process, on the JIT after warm-up, each file held to -a ceiling (`--throughput --take`, `--throughput --write`): -[taking rows](../guide/benchmarks.md#taking-rows-per-encoding) and +There is one file per encoding, and per notable shape of one: 61 files of a million rows +(`bench/gen-throughput.sh`), written by Rust's writer. The arm64 page was measured on the earlier set +of 57, and the four `pco` shapes added since (`pco_f32`, `pco_i16`, `pco_u32`, `pco_nullable`) are on +the x64 page only. Each side scans each file 12 times in a process of its own, on one core, every +value decoded to its plain form. A figure is the median of the last 10 scans and of 3 processes, +Native AOT against Rust's `vxbench` binary, which is what a decoder costs once warm. [Decoding, per +encoding](../guide/benchmarks.md#decoding-per-encoding) has every file. + +* A column stored in plain form is not decoding at all. It is handed out as a view of the mapped file + on both sides, with nothing copied, so the figure measures the walk of the layout and the speedup + compares two walks. +* The widest gaps are UTF-8 validation. In `varbin`, `varbinview`, `struct` and the tables, the + stored arrays are already in plain form, and the time goes to checking that every string is UTF-8, + which both sides do. A sampled profile of Rust's scan of `varbinview` spends it in `from_utf8` and + `validate_view`, under `VarBinViewData::try_new`. +* Each scan opens the file again and maps it anew, on both sides. + +Take and write per encoding are measured in one process, on the JIT after warm-up, with each file +held to a ceiling (`--throughput --take`, `--throughput --write`). See [taking +rows](../guide/benchmarks.md#taking-rows-per-encoding) and [writing](../guide/benchmarks.md#writing-per-encoding), per encoding. Rust's writer declines `parquet_variant`. ### How the hot paths are built -* **FastLanes: the lane is the vector lane, and each packed word is loaded once.** A bit-packed - block is 1 024 values transposed so that a row of the block, and each packed word of all its - lanes, is 128 contiguous bytes whatever the element type: whole-vector loads and stores, no - gather, no scatter. The kernel walks a block a packed word at a time, keeping the word in - registers for every row inside it; what a step does depends on the width alone and is laid out - once per width, so no branch looks at the data. The scalar loop stays as fallback and oracle +* In FastLanes, the lane is the vector lane, and each packed word is loaded once. A bit-packed block + is 1 024 values transposed so that a row of the block, and each packed word across all its lanes, + is 128 contiguous bytes whatever the element type: whole-vector loads and stores, no gather, no + scatter. The kernel walks a block one packed word at a time, keeping the word in registers for + every row inside it. What a step does depends only on the bit width and is laid out once per width, + so no branch looks at the data. The scalar loop stays as fallback and oracle (`FastLanesKernelBenchmarks` on the benchmark page). -* **FSST and OnPair: one wide store per symbol.** A symbol of one to eight bytes is written as one - unaligned 8-byte store, and the output advances by its real length; an OnPair token of up to 16 - bytes, as one `Vector128` store. An exact copy only where the slack runs out at the end of a - buffer. Checking that every code names a symbol and stays inside the destination costs next to - nothing against the bare loop. -* **Zstd: a frame per block, and only the frames a take needs.** Our writer compresses a zstd column - a block of 8 192 rows to a frame, and our reader inflates only the frames that hold the rows it is - asked for. -* **A take reads by the run, and decodes ahead.** Consecutive zones holding a take's rows are read - as one batch, up to sixteen rows a batch on several lanes, and a column whose encoding decodes - ranges without selecting decodes the clusters of rows it is asked for rather than the span between - them. On several lanes a take keeps three splits a lane in flight, and its degree bounds how many - decode at once. -* **Zone maps prune before any decode.** The file keeps the minimum, maximum and null count of every - zone of 8 192 rows. A filter skips the zones it proves empty and hands out the zones it proves full - without testing their rows: the narrow filter of §2 decodes about a hundredth of the rows. -* **Views, not copies.** On a mapped file, a column stored in its plain form is a view of the mapped - bytes; a test checks it by comparing pointers. -* **Nothing allocated per batch.** A scan's arenas and pools are its own, its `RecordBatch` is bound +* FSST and OnPair write one wide store per symbol. A symbol of one to eight bytes is written as one + unaligned 8-byte store and the output advances by its real length, and an OnPair token of up to 16 + bytes as one `Vector128` store. An exact copy is only needed where the slack runs out at the end of + a buffer. Checking that every code names a symbol and stays inside the destination costs next to + nothing compared with the bare loop. +* Zstd uses a frame per block, and a take only inflates the frames it needs. Our writer compresses a + zstd column one block of 8 192 rows per frame, and our reader inflates only the frames that hold + the rows it is asked for. +* A take reads by runs and decodes ahead. Consecutive zones holding a take's rows are read as one + batch, up to sixteen rows per batch on several lanes, and a column whose encoding can decode ranges + without a selection decodes the clusters of rows asked for rather than the span between them. On + several lanes, a take keeps three splits per lane in flight, and its degree bounds how many decode + at once. +* Zone maps prune before any decode. The file keeps the minimum, maximum and null count of every zone + of 8 192 rows. A filter skips the zones it proves empty and hands out the zones it proves full + without testing their rows, so the narrow filter of [what a caller sees](#2-what-a-caller-sees) + decodes about a hundredth of the rows. +* On a mapped file, a column stored in plain form is a view of the mapped bytes rather than a copy, + and a test checks it by comparing pointers. +* Nothing is allocated per batch. A scan's arenas and pools are its own, its `RecordBatch` is bound again rather than rebuilt, and its column views are `ref struct`s the compiler keeps inside the - loop body (§5). -* **Parallel when asked, and thread-safe where shared.** Parallelism is given, never taken: a - session's `MaxDegreeOfParallelism`, 1 unless set, or a scan's or a writer's own + loop body ([allocations](#5-allocations)). +* Parallelism is opt-in, and shared objects are thread-safe. Parallelism is given, never taken: a + session's `MaxDegreeOfParallelism` (1 unless set), or a scan's or a writer's own `DegreeOfParallelism`. A scan decodes its splits side by side, each on a context and arenas of its - own, nothing shared, and delivers its batches in row order. A writer summarizes its columns and + own with nothing shared, and delivers its batches in row order. A writer summarizes its columns and compresses a column's zstd frames on its threads, and writes the same bytes whatever the degree. A - `VortexSession` and a `VortexFile` are thread-safe and meant to be shared, several scans of one - file at once included; a scan, a writer and a batch belong to one consumer at a time - ([09-contracts.md](09-contracts.md) §1–2, [threads.md](../guide/threads.md)). + `VortexSession` and a `VortexFile` are thread-safe and meant to be shared, including several scans + of one file at once, while a scan, a writer and a batch belong to one consumer at a time (see + [concurrency](09-contracts.md#1-concurrency-and-thread-safety), + [parallelism](09-contracts.md#2-parallelism) and [threads.md](../guide/threads.md)). ## 5. Allocations -Held by ratchets in the test suite, counted by the runtime in a process doing nothing else, so -exact: a test fails above its ceiling, and a ceiling only comes down. Each test prints what it -measured beside its ceiling. - -* **Per batch**: nothing on the calling thread, in steady state, on every scan - (`ScanAllocationTests`). Counted on every thread of a scan at degree 2, a few hundred bytes a batch - for its tasks, under a ceiling of 1 000. -* **Per operation**, on the 65 536-row corpus file of §3, the open included (`PathAllocationTests`): - the footer alone, the first batch, a full scan, a projection, a take of 64 rows from 64 splits, a - filter with and without its zone maps, and a full scan of one encoding's 4 096-row file, each - under a ceiling in bytes. A full scan costs what its first batch costs. -* **Per write** (`WriteAllocationTests`): each corpus file written back out, a writer on four threads +Allocations are held by ratchets in the test suite, counted by the runtime in a process doing nothing +else, so the counts are exact. A test fails above its ceiling, and a ceiling only ever comes down. +Each test prints what it measured next to its ceiling. + +* Per batch: nothing on the calling thread, in steady state, on every scan (`ScanAllocationTests`). + Counted on every thread of a scan at degree 2, a few hundred bytes per batch for its tasks, under a + ceiling of 1 000. +* Per operation, on the 65 536-row corpus file used [in one + process](#3-in-one-process-after-warm-up), the open included (`PathAllocationTests`): the footer + alone, the first batch, a full scan, a projection, a take of 64 rows from 64 splits, a filter with + and without its zone maps, and a full scan of one encoding's 4 096-row file, each under a ceiling in + bytes. A full scan costs what its first batch costs. +* Per write (`WriteAllocationTests`): each corpus file written back out, a writer on four threads against one, and a column of a wide schema, each under a ceiling. -The benchmark page gives what each of §2's actions allocates once warm, beside its time. +The benchmark page gives what each of the actions in [what a caller sees](#2-what-a-caller-sees) +allocates once warm, next to its time. ## 6. How it is measured -Every instrument but the ratchets runs as `dotnet run -c Release --project -bench/Vorticity.Benchmarks -- `, and, given `--out docs/guide/benchmarks.md`, writes -its sections of the benchmark page and leaves the others: +Every instrument except the ratchets runs as `dotnet run -c Release --project +bench/Vorticity.Benchmarks -- `, and, given `--out docs/guide/benchmarks.md`, rewrites its +own sections of the benchmark page and leaves the others alone: | instrument | what it gives | arguments | |---|---|---| -| the report | §2, and §4's decode, each side in its own process | `--report --markdown` | -| the ratio gate | §3, both sides in one process | `--ratio-check` | -| the per-encoding gates | §4's take and write, in one process | `--throughput`, with `--take` or `--write`; `--check` holds each file to its ceiling | +| the report | [what a caller sees](#2-what-a-caller-sees) and the per-encoding decode, each side in its own process | `--report --markdown` | +| the ratio gate | [in one process](#3-in-one-process-after-warm-up), both sides in one process | `--ratio-check` | +| the per-encoding gates | the per-encoding take and write, in one process | `--throughput`, with `--take` or `--write`, and `--check` to hold each file to its ceiling | | the trade-offs | every profile and hint on twenty column shapes, and what the advice picks on them | `--tradeoffs`, and `--advise` | -| BenchmarkDotNet | one kernel against its baseline, on one clock | a class name: `fastlanes` | -| the ratchets | §5 | `dotnet test Vorticity.slnx -c Release` | +| BenchmarkDotNet | one kernel against its baseline, on one clock | a class name, such as `fastlanes` | +| the ratchets | [allocations](#5-allocations) | `dotnet test Vorticity.slnx -c Release` | Rust is driven through [tools/vxbench-rs](../../tools/vxbench-rs), built against the same -`vortex = "=0.86.1"` pin the corpus was written with: a C ABI for the in-process gates, a binary for -the report. `--ffi-check` checks that both readers return the same rows and the same checksum of +`vortex = "=0.86.1"` pin the corpus was written with: a C ABI for the in-process gates, and a binary +for the report. `--ffi-check` checks that both readers return the same rows and the same checksum of every decoded value before a ratio is trusted, and the gates refuse to run under a Rust build other -than the one their ceilings were set under. [bench/README.md](../../bench/README.md) has every +than the one their ceilings were set with. [bench/README.md](../../bench/README.md) has every command, what it costs, and how a ceiling moves. -Not measured: - -* **A cold page cache.** Every run reads a warm one; upstream's benchmarks flush it. -* **Pinned cores.** macOS pins no process, and an Apple M-series processor has cores of two kinds. -* **A remote store's latency**: every file is local. -* **Other data.** TPC-H and ClickBench are gigabytes that do not belong in a repository; what makes - the comparison honest is both sides reading the same bytes, whatever they are. -* **The row encoding against Rust's**: the shim cannot hand `vortex-row` a batch, so - `RowEncodingBenchmarks` compares our builds only. -* **CI** runs no benchmark: the gates are commands run before a push, `bench/gate.sh` among them. +What is not measured: + +* A cold page cache. Every run reads a warm one, while upstream's benchmarks flush it. +* Pinned cores. macOS pins no process, and an Apple M-series processor has two kinds of cores. +* A remote store's latency, since every file is local. +* Other datasets. TPC-H and ClickBench are gigabytes that do not belong in a repository, and what + makes the comparison honest is both sides reading the same bytes, whatever they are. +* The row encoding against Rust's. The shim cannot hand `vortex-row` a batch, so + `RowEncodingBenchmarks` only compares our own builds. +* Anything in CI. CI runs no benchmark, and the gates are commands run before a push, `bench/gate.sh` + among them. diff --git a/docs/design/06-row-encoding.md b/docs/design/06-row-encoding.md index a83abe40..40fd1564 100644 --- a/docs/design/06-row-encoding.md +++ b/docs/design/06-row-encoding.md @@ -1,40 +1,39 @@ # Byte-sortable row encoding -A separate specification from the file format, implemented upstream by the `vortex-row` crate. It -converts one or more columnar arrays into a `ListView`: one byte string per row, such that -**lexicographic byte comparison of the encoded rows equals logical tuple comparison of the input -values**, under per-column sort direction and null placement. +The row encoding is a separate specification from the file format, implemented upstream by the +`vortex-row` crate. It converts one or more columnar arrays into a `ListView`, one byte string +per row, such that lexicographic byte comparison of the encoded rows equals logical tuple comparison +of the input values, under per-column sort direction and null placement. -Use cases: sort keys, row keys, range partitioning, merge joins, index entries — anywhere a -comparator must become a `memcmp`. +It serves sort keys, row keys, range partitioning, merge joins and index entries: anywhere a +comparator has to become a `memcmp`. -> **Stability.** Upstream marks this format *experimental*: the byte layout, the supported types and -> the edge cases may change between Vortex releases, and its spec says not to persist these bytes as -> an interchange format. So it ships in its own package, `Vorticity.RowEncoding`, marked +> Upstream marks this format experimental. The byte layout, the supported types and the edge cases +> may change between Vortex releases, and its spec says not to persist these bytes as an interchange +> format. So it ships in its own package, `Vorticity.RowEncoding`, marked > `[Experimental("VX0002")]`, and the encoder names the release its bytes follow (`vortex-row -> 0.86.1`). Encoded bytes are safe to compare within one process, or one cluster running one -> version; a durable index that stores them records that name -> ([10-indexes.md](10-indexes.md) §5.4). +> 0.86.1`). Encoded bytes are safe to compare within one process, or within one cluster running one +> version, and a durable index that stores them records that name (see [composite +> keys](10-indexes.md#54-composite-keys-through-the-row-encoding)). ## 1. Contract -For columns `c0…cn` with per-column `RowSortField { descending: bool, nulls_first: bool }`: +For columns `c0…cn` with a per-column `RowSortField { descending: bool, nulls_first: bool }`: ``` encode(row_a) < encode(row_b) ⟺ (a.c0, …, a.cn) < (b.c0, …, b.cn) ``` -The bytes carry **no type tags, no field names, no sort options**. Two encoded rows are comparable -only if produced from the same schema with the same field settings. `descending` and `nulls_first` -are independent: nulls can sort first or last in either direction. +The bytes carry no type tags, no field names and no sort options. Two encoded rows are only +comparable if they were produced from the same schema with the same field settings. `descending` and +`nulls_first` are independent, so nulls can sort first or last in either direction. -Notation: `||` concatenation, `BE(x)` fixed-width big-endian bytes (never length-prefixed, never -trimmed), `!b` = `b XOR 0xFF`, `zero(n)` / `ff(n)` = n bytes of `0x00` / `0xFF`. +Notation: `||` is concatenation, `BE(x)` fixed-width big-endian bytes (never length-prefixed, never +trimmed), `!b` is `b XOR 0xFF`, and `zero(n)` and `ff(n)` are n bytes of `0x00` and `0xFF`. ## 2. Sentinels -Every value starts with a sentinel byte that classifies nullness before any value byte is -compared. +Every value starts with a sentinel byte that classifies nullness before any value byte is compared. | Family | Case | Asc, nulls first | Desc, nulls first | Asc, nulls last | Desc, nulls last | |---|---|---|---|---|---| @@ -47,9 +46,9 @@ compared. Fixed-width sentinels serve Null, Bool, Primitive, Decimal, Struct and FixedSizeList. Variable-width sentinels serve Utf8 and Binary. -**The null sentinel is never inverted by `descending`** — only non-null value bytes (and, for -variable-width, the empty/non-empty sentinels) are. This is what keeps null placement independent -of sort direction, and it is the single easiest thing to get wrong. +`descending` never inverts the null sentinel. Only non-null value bytes are inverted, plus, for +variable-width types, the empty and non-empty sentinels. That is what keeps null placement +independent of sort direction, and it is the single easiest thing to get wrong. A fixed-width null is followed by zero-filled value bytes, so fixed types keep a constant encoded width on every row. @@ -59,28 +58,29 @@ width on every row. | Type | Ascending | Descending | |---|---|---| | `Null` | `sentinel` (no body) | same | -| `Bool` | `0x01 \|\| 0x01` (false) / `0x01 \|\| 0x02` (true) | `0x01 \|\| 0xFE` (false) / `0x01 \|\| 0xFD` (true) | +| `Bool` | `0x01 \|\| 0x01` (false), `0x01 \|\| 0x02` (true) | `0x01 \|\| 0xFE` (false), `0x01 \|\| 0xFD` (true) | | `u8/u16/u32/u64` | `0x01 \|\| BE(v)` | `0x01 \|\| !BE(v)` | | `i8/i16/i32/i64` (and `i128`) | `0x01 \|\| ordered`, where `ordered = BE(v)` with `ordered[0] ^= 0x80` | `0x01 \|\| !ordered` | | `f16/f32/f64` | `0x01 \|\| BE(ordered)` | `0x01 \|\| !BE(ordered)` | -| `Decimal` | signed-integer encoding of the scaled storage value | idem | -| `Utf8` / `Binary` | variable-width sentinel + block body (§4) | inverted (§4) | -| `Struct` | `sentinel \|\| field_0 \|\| … \|\| field_n`, fields recursive in schema order | idem | -| `FixedSizeList` | `sentinel \|\| elem_0 \|\| … \|\| elem_{N-1}` | idem | +| `Decimal` | signed-integer encoding of the scaled storage value | same | +| `Utf8` / `Binary` | variable-width sentinel + block body ([below](#4-variable-length-body)) | inverted ([below](#4-variable-length-body)) | +| `Struct` | `sentinel \|\| field_0 \|\| … \|\| field_n`, fields recursive in schema order | same | +| `FixedSizeList` | `sentinel \|\| elem_0 \|\| … \|\| elem_{N-1}` | same | -Null bodies for fixed-width types: `null_sentinel || zero(width(T))`. +Null bodies for fixed-width types are `null_sentinel || zero(width(T))`. -**Float transform**, from raw IEEE bits: +Floats are transformed from their raw IEEE bits: ``` ordered = sign_bit(bits) == 0 ? bits XOR sign_bit_mask : bits XOR all_ones ``` -This yields a total order: negatives before positives, `-0.0` before `+0.0`. NaNs order by raw bit -pattern and are **not** canonicalized: two NaNs with different payloads are two keys. +This yields a total order, negatives before positives and `-0.0` before `+0.0`. NaNs order by their +raw bit pattern and are not canonicalized, so two NaNs with different payloads are two different +keys. -**Decimal storage width**, selected from precision: +A decimal's storage width is selected from its precision: | Precision | Storage | |---|---| @@ -92,40 +92,40 @@ pattern and are **not** canonicalized: two NaNs with different payloads are two `Decimal256` is not supported. -**Option inheritance is identity, and inversion happens once.** A nested field uses its parent's -`RowSortField` unchanged — verified in `vortex-row/src/codec.rs`, where `encode_struct` passes the +Option inheritance is the identity, and inversion happens exactly once. A nested field uses its +parent's `RowSortField` unchanged, as `vortex-row/src/codec.rs` shows: `encode_struct` passes the same `field` to `field_encode` for every child. Inversion for `descending` is applied by the leaf -encoder to its own value bytes only; the parent never re-inverts the concatenated body. The classic -double-inversion trap is therefore avoided by *not* doing anything at the parent level. - -**The child null sentinel is chosen by the child's dtype, using the inherited field.** -`child_canonical_null_byte(child_dtype, field)` returns the variable-width null sentinel for -`Utf8`/`Binary` children and the fixed-width null sentinel for everything else (including a nested -struct or FSL acting as a variable-width child). The `nulls_first` that selects between `0x00` and -`0x02`/`0xFF` is the inherited one, i.e. the root column's. For a fixed-width child under a null -parent, the body is that sentinel byte followed by **zero** fill to the child's fixed width — zero +encoder to its own value bytes only, and the parent never inverts the concatenated body again. The +classic double-inversion trap is avoided by doing nothing at the parent level. + +The child null sentinel is chosen by the child's dtype, using the inherited field. +`child_canonical_null_byte(child_dtype, field)` returns the variable-width null sentinel for `Utf8` +and `Binary` children and the fixed-width null sentinel for everything else, including a nested +struct or FSL acting as a variable-width child. The `nulls_first` that selects between `0x00` and +`0x02` or `0xFF` is the inherited one, which is the root column's. For a fixed-width child under a +null parent, the body is that sentinel byte followed by zero fill up to the child's fixed width, zero even under `descending`. -**Nested nulls are canonicalized.** A null struct or null fixed-size list must still emit a body, -so that two null parents compare byte-equal regardless of what their physical child arrays -contain: fixed-width children contribute their fixed-width null encoding; variable-width children -contribute exactly one child null sentinel byte. Skipping this is a correctness bug that only -shows up on arrays with garbage under a null — exactly what a differential test against Rust will -catch and a naive round-trip test will not. +Nested nulls are canonicalized. A null struct or null fixed-size list must still emit a body, so that +two null parents compare byte-equal whatever their physical child arrays contain. Fixed-width +children contribute their fixed-width null encoding, and variable-width children contribute exactly +one child null sentinel byte. Skipping this is a correctness bug that only shows up on arrays with +garbage under a null, which is exactly what a differential test against Rust catches and a naive +round-trip test does not. ## 4. Variable-length body -Non-empty Utf8/Binary bodies are written in blocks of **32 data bytes + 1 marker byte**: +Non-empty Utf8 and Binary bodies are written in blocks of 32 data bytes plus 1 marker byte: -* Ascending: every non-final full block uses marker `0xFF`; the final block is zero-padded to 32 - data bytes and its marker is the real data length in `1..=32`. -* Descending: every data byte inverted, continuation marker `0x00`, final block padded with - `0xFF`, final marker `final_len XOR 0xFF`. +* Ascending: every non-final full block uses marker `0xFF`, and the final block is zero-padded to 32 + data bytes with the real data length, in `1..=32`, as its marker. +* Descending: every data byte is inverted, the continuation marker is `0x00`, the final block is + padded with `0xFF`, and the final marker is `final_len XOR 0xFF`. * If the length is an exact multiple of 32, the final block carries marker `32` and earlier blocks use the continuation marker. -This is what preserves prefix ordering: a shorter value reaches its final marker (`≤ 32`) while a -longer one is still emitting continuation markers (`0xFF`), so the prefix sorts first. +This preserves prefix ordering: a shorter value reaches its final marker (`≤ 32`) while a longer one +is still emitting continuation markers (`0xFF`), so the prefix sorts first. ## 5. Output layout @@ -137,29 +137,34 @@ sizes: per-row byte length Rows are not self-delimiting without `sizes`. The encoder computes the sizes first (a fixed-width column contributes a constant, a variable-width one what its values need), then reuses the `sizes` -array as the per-row write cursor: **one sizing pass, one write pass**, into three pooled buffers. +array as the per-row write cursor. That makes one sizing pass and one write pass, into three pooled +buffers. ## 6. Unsupported types -Variable-size `List`, `Variant`, `Union`, `Extension` and `Decimal256`. The absence is deliberate -upstream: adding one means defining both its order and its bytes. This encoder refuses them as the -reference does, and never invents an ordering. +Variable-size `List`, `Variant`, `Union` and `Decimal256` are not supported. The absence is +deliberate upstream, since adding one means defining both its order and its bytes. This encoder +refuses them as the reference does, and never invents an ordering. -`Extension` being unsupported means a timestamp or a date is not row-encoded directly: a caller -encodes its storage column. Unwrapping extensions silently would make the bytes stop matching -Rust's. +Upstream does not support `Extension` either. Here, an extension column at the top level of the key +(a date, a time, a timestamp, a uuid) is encoded as its storage column, which gives exactly the bytes +Rust produces when a caller encodes that storage column itself. An extension nested inside a struct +or a fixed-size list is refused with `VortexUnsupportedException`, as the reference refuses it, +because the encoder would otherwise be silently choosing an ordering the format has not defined. ## 7. Implementation -- **Column at a time.** The encoder writes one field for every row, then the next field: better - for branch prediction and vector width than a row at a time, and what the two-pass sizing assumes. - Big-endian conversion is a byte swap, descending a bitwise NOT over a span. -- **No allocation per row.** The output size is known before a byte is written. The kernels call - `TryWriteBigEndian`, never `WriteBigEndian`: the latter is a default interface method the - primitive types do not override, so a generic call to it boxes its receiver on every row. -- **Synchronous.** It is CPU work over arrays already in memory; the async-only rule is for I/O. - -The surface is `RowEncoder` (a borrowed batch in, a pooled `RowKeys` out; one key's bytes with -`EncodeKey`), `RowSortField`, and `RowKeyEncoder`, which a writer uses to key a composite index -([10-indexes.md](10-indexes.md) §5.4). The code is `src/Vorticity.RowEncoding`, and -[row-keys.md](../guide/row-keys.md) is the guide's page. +- The encoder works a column at a time. It writes one field for every row, then the next field, + which suits branch prediction and vector width better than a row at a time, and is what the + two-pass sizing assumes. Big-endian conversion is a byte swap, and descending is a bitwise NOT over + a span. +- There is no allocation per row, since the output size is known before a byte is written. The + kernels call `TryWriteBigEndian`, never `WriteBigEndian`: the latter is a default interface method + the primitive types do not override, so a generic call to it boxes its receiver on every row. +- It is synchronous. It is CPU work over arrays already in memory, and the async-only rule is for + I/O. + +The public API is `RowEncoder` (a borrowed batch in, a pooled `RowKeys` out, and one key's bytes +with `EncodeKey`), `RowSortField`, and `RowKeyEncoder`, which a writer uses to key a composite index +(see [composite keys](10-indexes.md#54-composite-keys-through-the-row-encoding)). The code is in +`src/Vorticity.RowEncoding`, and [row-keys.md](../guide/row-keys.md) is the guide's page. diff --git a/docs/design/07-dotnet-mapping.md b/docs/design/07-dotnet-mapping.md index fa7085bd..91801859 100644 --- a/docs/design/07-dotnet-mapping.md +++ b/docs/design/07-dotnet-mapping.md @@ -1,7 +1,8 @@ # DType → .NET mapping -The type contract: what a caller receives for each Vortex dtype, and where a naive mapping would -lose data. The dtypes themselves are [02-format.md](02-format.md) §4. +The type contract: what a caller receives for each Vortex dtype, and where a naive mapping would lose +data. The dtypes themselves are described in [the format's DType +section](02-format.md#4-dtype-dtypefbs--dtypeproto). ## 1. The mapping @@ -9,112 +10,113 @@ What a record member and a `Column` may be, and what the column exposes: | dtype | `T` | what `Column` exposes | notes | |---|---|---|---| -| `Null` | none | — | the schema shows it; no `T` reads it | +| `Null` | none | nothing | the schema shows it, but no `T` reads it | | `Bool` | `bool` | `Bits` as `ReadOnlySpan`, an indexer | zero-copy | | `Primitive` i8…i64, u8…u64 | `sbyte`…`long`, `byte`…`ulong` | `Values` as `ReadOnlySpan`, an indexer | zero-copy | | `Primitive` u16 | `char` too | the same | the UTF-16 code unit, lone surrogates included | | `Primitive` i64, u64 | `nint`, `nuint` too, on a 64-bit host | the same | refused on a 32-bit host, whose native integer is narrower | -| `Primitive` i64 | `TimeSpan` too | an indexer, `CopyTo`, `Storage()` | 100 ns ticks: exact over the whole range | -| `Primitive` f16, f32, f64 | `Half`, `float`, `double` | the same | exact: `Half` is IEEE binary16 | -| `Decimal(p, s)` | `decimal` within §2's bounds, `VortexDecimal` always | an indexer, `Storage()`, `Scale` | never lossy: a column too wide for `decimal` is refused when it binds | -| `Decimal(p, 0)` | `Int128`, `UInt128` for `p ≤ 39`; `BigInteger` always | an indexer, `CopyTo`; `Values` zero-copy over 128-bit storage | §2.1 | -| `Utf8` | `string` | a UTF-8 span per value; `GetString(i)` on request | §4 | +| `Primitive` i64 | `TimeSpan` too | an indexer, `CopyTo`, `Storage()` | 100 ns ticks, exact over the whole range | +| `Primitive` f16, f32, f64 | `Half`, `float`, `double` | the same | exact, since `Half` is IEEE binary16 | +| `Decimal(p, s)` | `decimal` within [the bounds below](#2-decimal-why-systemdecimal-is-not-the-answer), `VortexDecimal` always | an indexer, `Storage()`, `Scale` | never lossy: a column too wide for `decimal` is refused when it binds | +| `Decimal(p, 0)` | `Int128`, `UInt128` for `p ≤ 39`, `BigInteger` always | an indexer, `CopyTo`, and `Values` zero-copy over 128-bit storage | see [wider integers](#21-integers-wider-than-64-bits) | +| `Utf8` | `string` | a UTF-8 span per value, `GetString(i)` on request | see [strings](#4-strings-spans-first-string-on-request) | | `Binary` | `ReadOnlyMemory`, `Memory`, `byte[]` | a span per value | | | `Struct` | a nested `[VortexRecord]` type | `Columns` | | | `List`, `FixedSizeList` | `ReadOnlyMemory`, `Memory`, `T[]`, `List`, `ImmutableArray`, an interface an array implements | a `Range` per row, `Elements` as `Column` | reading rows allocates one array, or one `List`, per list | -| `List` of `Struct` | any of the above over a `[VortexRecord]` type | `ListOf()`: a `Range` per row, `Elements` as `Columns` | the elements read in one pass, then cut per row | -| `Map` | `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | `MapOf()`: a `Range` per row, `Keys` and `Values` | scalar keys and values; a key is never null | +| `List` of `Struct` | any of the above over a `[VortexRecord]` type | `ListOf()`: a `Range` per row, `Elements` as `Columns` | the elements are read in one pass, then cut per row | +| `Map` | `Dictionary`, `IDictionary`, `IReadOnlyDictionary` | `MapOf()`: a `Range` per row, `Keys` and `Values` | scalar keys and values, and a key is never null | | `vortex.date` | `DateOnly` | an indexer, `Storage()` or `Storage()` | | | `vortex.time` | `TimeOnly` | an indexer, `Storage()`, `Unit` | | -| `vortex.timestamp` | `DateTime` when naive or UTC, `DateTimeOffset` with a zone | an indexer, `Storage()`, `Unit`, `TimeZone` | §3 | +| `vortex.timestamp` | `DateTime` when naive or UTC, `DateTimeOffset` with a zone | an indexer, `Storage()`, `Unit`, `TimeZone` | see [temporal extensions](#3-temporal-extensions-one-resolution-at-binding) | | `vortex.uuid` | `Guid` | an indexer | the byte order is converted explicitly, never implied | -| an extension registered on the session | the registered type | an indexer, `Storage()` | [14-public-api.md](14-public-api.md) §2 | +| an extension registered on the session | the registered type | an indexer, `Storage()` | see [the session](14-public-api.md#2-the-session) | `Union` and `Variant` columns appear in a schema but no `T` maps to them. An `enum` member maps to its underlying integer. -**Nullability is carried by `T`**: `Column` is a nullable column and `Column` is -not, and a non-nullable value type over a nullable column is refused when it binds +Nullability is carried by `T`. `Column` is a nullable column and `Column` is not, +and a non-nullable value type over a nullable column is refused when it binds (`VortexSchemaException`). The `T?` names the column, not its storage: `Values` is still the -`ReadOnlySpan` of the slots, a null slot holds an unspecified value, and `ValidityWords` -gives the validity as 64-bit words, empty when every row is valid, so a caller skips per-row checks -on the common case. Only the indexer and row materialization produce a `double?` per value. +`ReadOnlySpan` of the slots, where a null slot holds an unspecified value, and +`ValidityWords` gives the validity as 64-bit words, empty when every row is valid, so a caller can +skip per-row checks in the common case. Only the indexer and row materialization produce a `double?` +per value. ## 2. Decimal: why `System.Decimal` is not the answer -Vortex allows a precision up to **76**, backed by `i256`; `System.Decimal` holds 28 to 29 -significant digits. A default mapping to `decimal` would be **silently lossy over a legal range of -the format**, which is the one failure this library must not have. +Vortex allows a precision of up to 76, backed by `i256`, while `System.Decimal` holds 28 to 29 +significant digits. A default mapping to `decimal` would be silently lossy over a legal range of the +format, which is the one failure this library must not have. So a `decimal` is accepted where it cannot lose anything, and refused everywhere else. A `decimal` -record member, `Column`, `Sym` or `ColumnBuilder` binds to a column -whose precision is at most 28 and whose scale lies in `[0, 28]`, which `System.Decimal` holds -exactly. The check is made once, when the record or the column binds, and a wider column is refused -there with `VortexSchemaException` naming `VortexDecimal`, never rounded value by value. -`VortexDecimal` (`Types/Numerics/VortexDecimal.cs`) binds to every decimal column and is the only -mapping of one beyond those bounds; it converts to `decimal` or `Int128` when the value fits and -formats exactly always. Written from a record, a `decimal` member declares `Decimal(28, 10)` unless -`[VortexColumn]` gives its precision and scale. - -The exact bounds, from the reference and transcribed in -[spec/METADATA.md](../../spec/METADATA.md): +record member, `Column`, `Sym` or `ColumnBuilder` binds to a column whose +precision is at most 28 and whose scale lies in `[0, 28]`, which `System.Decimal` holds exactly. The +check is made once, when the record or the column binds, and a wider column is refused there with a +`VortexSchemaException` naming `VortexDecimal`, never rounded value by value. `VortexDecimal` +(`Types/Numerics/VortexDecimal.cs`) binds to every decimal column and is the only mapping for one +beyond those bounds. It converts to `decimal` or `Int128` when the value fits, and always formats +exactly. Written from a record, a `decimal` member declares `Decimal(28, 10)` unless `[VortexColumn]` +gives its precision and scale. + +The exact bounds, from the reference and transcribed in [spec/METADATA.md](../../spec/METADATA.md): * `1 ≤ precision ≤ 76` * `scale ≤ 76` -* `scale ≤ precision` **only when `scale > 0`**: a negative scale is legal, bounded only by `i8` +* `scale ≤ precision` only when `scale > 0`. A negative scale is legal, bounded only by `i8` A reader that refuses a precision above 38, or requires `scale ≥ −precision`, refuses legal files. -**Storage width** is carried in `vortex.decimal`'s metadata as `values_type`: `i8` (0), `i16` (1), -`i32` (2), `i64` (3), `i128` (4), `i256` (5). The precision selects the *smallest* legal width — -1–2 → `i8`, 3–4 → `i16`, 5–9 → `i32`, 10–18 → `i64`, 19–38 → `i128`, 39–76 → `i256` — but a file may -declare any **wider** one, and often does: upstream allows precision-2 values in an `i256` buffer, -and a decimal column from Arrow or Parquet arrives as `i128` or `i256` whatever its precision. The -width is therefore read from `values_type`, and only a width *narrower* than the precision requires -is refused; `Storage()` exposes what the file declared, for callers who do their own -arithmetic without widening. +The storage width is carried in `vortex.decimal`'s metadata as `values_type`: `i8` (0), `i16` (1), +`i32` (2), `i64` (3), `i128` (4), `i256` (5). The precision selects the smallest legal width (1–2 → +`i8`, 3–4 → `i16`, 5–9 → `i32`, 10–18 → `i64`, 19–38 → `i128`, 39–76 → `i256`), but a file may +declare any wider one, and often does. Upstream allows precision-2 values in an `i256` buffer, and a +decimal column from Arrow or Parquet arrives as `i128` or `i256` whatever its precision. The width is +therefore read from `values_type`, and only a width narrower than the precision requires is refused. +`Storage()` exposes what the file declared, for callers who do their own arithmetic without +widening. -.NET has `Int128` and no `Int256`, so this library carries an internal `Int256` with what the -decimal path needs: comparison, negation, two's-complement bytes in both endiannesses and exact -text. It is correctness insurance, not a hot path: precision ≤ 38 is what files hold in practice. +.NET has `Int128` but no `Int256`, so this library carries an internal `Int256` with what the decimal +path needs: comparison, negation, two's-complement bytes in both endiannesses and exact text. It is +correctness insurance, not a hot path, since precision ≤ 38 is what files hold in practice. -The row encoder's narrower table (up to `i128`, no `Decimal256`) is `vortex-row`'s own limit, not -the format's ([06-row-encoding.md](06-row-encoding.md) §6). +The row encoder's narrower table (up to `i128`, no `Decimal256`) is `vortex-row`'s own limit, not the +format's (see [unsupported types](06-row-encoding.md#6-unsupported-types)). ### 2.1 Integers wider than 64 bits -No primitive dtype is wider than 64 bits, so an `Int128`, a `UInt128` or a `BigInteger` is a -decimal of scale 0: the unscaled value is the integer. The default width is the choice that matters: +No primitive dtype is wider than 64 bits, so an `Int128`, a `UInt128` or a `BigInteger` is stored as +a decimal of scale 0, whose unscaled value is the integer. The default width is the choice that +matters: -* **`Int128` and `UInt128` declare `Decimal(38, 0)`**, stored in 128 bits. Every reader of a - 128-bit decimal reads it, Arrow's `Decimal128` among them, and `Values` is the stored buffer with - no copy. Thirty-eight digits are not every value of either type, whose extremes have 39, so a - value past them is refused when it is written -- never wrapped, never rounded -- with the way - out in the message: `[VortexColumn(Precision = 39)]`, stored in 256 bits. -* **`BigInteger` declares `Decimal(76, 0)`**, as far as a decimal goes; a value past it is refused. +* `Int128` and `UInt128` declare `Decimal(38, 0)`, stored in 128 bits. Every reader of a 128-bit + decimal reads it, Arrow's `Decimal128` among them, and `Values` is the stored buffer with no copy. + Thirty-eight digits do not cover every value of either type, whose extremes have 39, so a value + beyond them is refused when it is written, never wrapped or rounded, and the message gives the way + out: `[VortexColumn(Precision = 39)]`, stored in 256 bits. +* `BigInteger` declares `Decimal(76, 0)`, as far as a decimal goes, and a value beyond it is refused. -Reading, `Int128` and `UInt128` bind to a decimal of scale 0 and at most 39 digits, whatever width -stores it. Thirty-eight digits always fit; a 39th can hold a value past the type, which is refused -as it is read, as a timestamp past `DateTime` is (§3), and a negative value is refused by a -`UInt128`. `BigInteger` binds to any decimal of scale 0 and reads every value. +On the read side, `Int128` and `UInt128` bind to a decimal of scale 0 and at most 39 digits, whatever +width stores it. Thirty-eight digits always fit. A 39th digit can hold a value beyond the type, which +is refused as it is read (the same way a timestamp beyond `DateTime` is), and a `UInt128` refuses a +negative value. `BigInteger` binds to any decimal of scale 0 and reads every value. -The aggregates follow: a sum over such a column is exact in 320 bits whatever its row count, and -converts to the type asked for, `OverflowException` when it does not fit; a sum as `BigInteger` -never throws. +The aggregates follow suit. A sum over such a column is exact in 320 bits whatever its row count, and +converts to the type asked for, throwing `OverflowException` when it does not fit. A sum as +`BigInteger` never throws. ## 3. Temporal extensions: one resolution, at binding -`vortex.timestamp` carries a unit and a time zone in its metadata. Resolving an IANA zone takes the -operating system's time-zone database through `TimeZoneInfo`, which makes the result -**machine-dependent**. +`vortex.timestamp` carries a unit and a time zone in its metadata. Resolving an IANA zone goes +through the operating system's time-zone database via `TimeZoneInfo`, which makes the result +machine-dependent. So the zone is resolved once, when a `DateTimeOffset` member or column binds to the file, by `TimeZoneInfo.FindSystemTimeZoneById`, and never per value. A naive or UTC timestamp binds as -`DateTime`, with `Kind` `Unspecified` or `Utc`, and needs no database. A zone the host cannot -resolve refuses the `DateTimeOffset` binding with `VortexSchemaException`; the column still binds -as `long`, its storage, and `Storage()`, `Unit` and `TimeZone` let a caller convert with a zone of -its own choosing. The machine dependence is confined to one call, at one moment, with a refusal -rather than a guess. +`DateTime`, with `Kind` `Unspecified` or `Utc`, and needs no database. A zone the host cannot resolve +makes the `DateTimeOffset` binding fail with `VortexSchemaException`. The column still binds as +`long`, its storage, and `Storage()`, `Unit` and `TimeZone` let a caller convert with a zone of its +own choosing. The machine dependence is confined to one call, at one moment, with a refusal rather +than a guess. ## 4. Strings: spans first, `string` on request @@ -122,17 +124,17 @@ A `vortex.varbinview` value is a 16-byte view into the file's buffers, so the na UTF-8 span, and a `string` is an allocation the caller asks for (`GetString`). [text-columns.md](../guide/text-columns.md) measures the difference. -The lifetime rule is the sharp edge: **a span borrowed from a batch is invalid once the enumeration -moves on.** `Columns` and `Column` are `ref struct`s, so the compiler keeps them inside -the loop body and out of an `await`, and analyzer VX1001 flags a span stored outside it. A caller -who needs the values later copies them, or takes an owned `RecordBatch`. +The lifetime rule is the sharp edge: a span borrowed from a batch becomes invalid once the +enumeration moves on. `Columns` and `Column` are `ref struct`s, so the compiler keeps +them inside the loop body and out of an `await`, and analyzer VX1001 flags a span stored outside it. +A caller who needs the values later copies them, or takes an owned `RecordBatch`. ## 5. A root that is not a struct -A file's root dtype may be a bare `Float64`, a `Bool`, or any type, and the reader does not assume a -table. The schema of such a file is one field with an empty name, whose type is the root. A record -binds to the fields of a struct root only, and `Scan()` on any other root throws +A file's root dtype may be a bare `Float64`, a `Bool` or any other type, and the reader does not +assume a table. The schema of such a file is one field with an empty name, whose type is the root. A +record only binds to the fields of a struct root, and `Scan()` on any other root throws a `VortexSchemaException` pointing at the tool scan, which reads the root as its single column -([untyped-files.md](../guide/untyped-files.md)). Inventing a synthetic column name instead would -make a round trip through this writer produce another schema than its input, which is worse than an +([untyped-files.md](../guide/untyped-files.md)). Inventing a synthetic column name instead would make +a round trip through this writer produce a different schema from its input, which is worse than an awkward API. diff --git a/docs/design/08-semantics.md b/docs/design/08-semantics.md index a846f2ec..b6e6dcb2 100644 --- a/docs/design/08-semantics.md +++ b/docs/design/08-semantics.md @@ -1,197 +1,206 @@ # Query semantics: pruning, predicates, unknown components -The rules that decide whether a scan returns *correct* rows. None of them is visible in the byte -format, and each produces silently wrong results if guessed. Where a rule was settled by reading the -reference implementation rather than its prose, the source is cited. +The rules that decide whether a scan returns correct rows. None of them is visible in the byte +format, and each one produces silently wrong results if guessed. Where a rule was settled by reading +the reference implementation rather than its prose, the source is cited. ## 1. Statistic precision: what `Inexact` licenses -`ArrayStats` carries `min_precision` / `max_precision` with values `Exact` / `Inexact` -([02-format.md](02-format.md) §5.2). The reference defines the semantics unambiguously -(`vortex-array/src/expr/stats/precision.rs`): +`ArrayStats` carries `min_precision` and `max_precision`, with the values `Exact` and `Inexact` (see +[the array schema](02-format.md#52-schema-arrayfbs)). The reference defines their meaning without +ambiguity (`vortex-array/src/expr/stats/precision.rs`): > Inexact statistics form a range of possible values that the statistic could be. This is statistic > specific, for max this will be an upper bound. Meaning that the actual max in an array is > guaranteed to be less than or equal to the inexact value, but equal to the exact value. -So **`Inexact` is a conservative bound, not an unqualified approximation**: `min` is a lower bound -on the true minimum, `max` an upper bound on the true maximum. The consequences are exact: +So `Inexact` is a conservative bound, not a loose approximation: `min` is a lower bound on the true +minimum and `max` an upper bound on the true maximum. The consequences are precise: | Operation | `Exact` | `Inexact` | |---|---|---| -| Range pruning (`x > k` vs zone `max`) | legal | **legal** — the bound points the safe way | -| Equality shortcut (`min == max` ⇒ constant zone) | legal | forbidden | -| Reporting the value to the caller as a true min/max | legal | forbidden | +| Range pruning (`x > k` against the zone's `max`) | legal | legal, since the bound points the safe way | +| Equality shortcut (`min == max` means a constant zone) | legal | forbidden | +| Reporting the value to the caller as a true min or max | legal | forbidden | | Skipping decode because `min == max` | legal | forbidden | -A third state, `Absent`, exists in the reference and maps to `Inexact` with no value on the wire. -A statistic with no value licenses nothing. +A third state, `Absent`, exists in the reference and maps to `Inexact` with no value on the wire. A +statistic with no value licenses nothing. -**The invariant that governs every pruning decision**, and the one the property test asserts: +One invariant governs every pruning decision, and it is the one the property test asserts: > Pruning may never eliminate a row that full materialization would have returned. The converse, pruning too little, only costs time. That asymmetry is what makes the invariant -testable: every filter test runs with pruning and indexes on and off, and the row sets must be -equal, including over deliberately loose but legal `Inexact` statistics. +testable: every filter test runs with pruning and indexes on and off, and the row sets must be equal, +including over deliberately loose but legal `Inexact` statistics. ## 2. NaN -The reference computes `min`/`max` with `NumericalAggregateOpts::skip_nans()` +The reference computes `min` and `max` with `NumericalAggregateOpts::skip_nans()` (`vortex-array/src/stats/expr.rs`), and tracks NaN separately in the `nan_count` statistic. So: -* **Zone `min`/`max` exclude NaN.** A zone containing NaN still reports the min/max of its +* Zone `min` and `max` exclude NaN. A zone containing NaN still reports the min and max of its non-NaN values. -* Filter expressions follow **IEEE 754**, which is more precise than "NaN never matches". The five - ordering-and-equality predicates — `==`, `<`, `<=`, `>`, `>=` — are all **false** when either - operand is NaN, `NaN == NaN` included. **`!=` is the exception**: it is defined as the negation of - `==`, so `NaN != x` is **true**, for every `x`, NaN included. C, C#, Rust and SQL all agree on - this; a reader that makes `!=` false for NaN "for consistency" drops rows every other - implementation returns. -* This is all distinct from the row encoding, which deliberately defines a *total* order over floats - where NaN is ordered by raw bit pattern ([06-row-encoding.md](06-row-encoding.md) §3). The two - orderings serve different purposes and must not be unified — an implementer who reuses the - row-encoding comparator for filter evaluation introduces a correctness bug. -* Therefore a zone with `max <= 10` may be pruned for `x > 10` **even when `nan_count > 0`**, - because the NaN rows would not have matched anyway. This is only sound given the two facts - above; it is written down here so it is not rediscovered by guesswork. +* Filter expressions follow IEEE 754, which is more precise than "NaN never matches". The five + ordering and equality predicates (`==`, `<`, `<=`, `>`, `>=`) are all false when either operand is + NaN, `NaN == NaN` included. `!=` is the exception: it is defined as the negation of `==`, so + `NaN != x` is true for every `x`, NaN included. C, C#, Rust and SQL all agree on this, and a reader + that makes `!=` false for NaN "for consistency" drops rows every other implementation returns. +* All of this is distinct from the row encoding, which deliberately defines a total order over + floats where NaN is ordered by its raw bit pattern (see [per-type + encodings](06-row-encoding.md#3-per-type-encodings)). The two orderings serve different purposes + and must not be unified. An implementer who reuses the row-encoding comparator to evaluate filters + introduces a correctness bug. +* Therefore a zone with `max <= 10` may be pruned for `x > 10` even when `nan_count > 0`, because the + NaN rows would not have matched anyway. This is only sound given the two facts above, and it is + written down here so it is not rediscovered by guesswork. ## 3. Three-valued logic Predicates evaluate over `{true, false, unknown}`, SQL-style: * A comparison with a null operand yields `unknown`. -* `unknown AND false = false`; `unknown OR true = true`; `NOT unknown = unknown`. -* A row is returned only when the filter evaluates to `true` — `unknown` does not match. -* `IS NULL` / `IS NOT NULL` never yield `unknown`. -* `ListContains(list, v)` is `unknown` on a null list and under a null `v`; a null element matches - nothing and leaves the row `false` ([12-index-reads.md](12-index-reads.md) §6). - -A literal of a type the column cannot compare to is **not a silent non-match**: it is refused -before a row is read. On the typed path it does not compile, because `Sym` compares only with a -`T` or with another column of the same `T`. On the tool path `Where` throws +* `unknown AND false = false`, `unknown OR true = true`, and `NOT unknown = unknown`. +* A row is returned only when the filter evaluates to `true`, so `unknown` does not match. +* `IS NULL` and `IS NOT NULL` never yield `unknown`. +* `ListContains(list, v)` is `unknown` on a null list and under a null `v`. A null element matches + nothing and leaves the row `false` (see [predicates on text and + lists](12-index-reads.md#6-predicates-on-text-and-lists)). + +A literal of a type the column cannot be compared with is not a silent non-match. It is refused +before a row is read. On the typed path it does not compile, because `Sym` only compares with a +`T` or with another column of the same `T`. On the tool path, `Where` throws a `VortexSchemaException` naming the column, its type and the literal, and a hole of an interpolated -filter throws as it is appended; two columns compared with each other must be of the same type, or -`Where` throws the same exception. What looks like a mismatch and is a conversion: text compared -with a date, time, timestamp, uuid or decimal column is parsed in the column's type, and refused -when it does not parse; a number compared with a decimal column is scaled exactly from its digits, -and a value the column's scale cannot hold makes an equality false rather than rounding into a -match. On the typed path a comparison with `null` is nullity, never `unknown`: `== null` is -`IS NULL`, `!= null` is `IS NOT NULL`, an ordering against `null` matches nothing, and a `null` among -the values of `In` is ignored. - -Pruning consequence: a zone where `null_count == row_count` can be skipped for any predicate that -is not satisfiable by nulls — that is, anything except `IS NULL` and expressions reducible to it. -This is one of the highest-value pruning rules in practice and it depends entirely on the 3VL -choice above being fixed. +filter throws as it is appended. Two columns compared with each other must have the same type, or +`Where` throws the same exception. + +Some apparent mismatches are conversions. Text compared with a date, time, timestamp, uuid or +decimal column is parsed in the column's type, and refused when it does not parse. A number compared +with a decimal column is scaled exactly from its digits, and a value the column's scale cannot hold +makes an equality false rather than rounding into a match. On the typed path a comparison with +`null` tests nullity and is never `unknown`: `== null` is `IS NULL`, `!= null` is `IS NOT NULL`, an +ordering against `null` matches nothing, and a `null` among the values of `In` is ignored. + +For pruning, a zone where `null_count == row_count` can be skipped for any predicate that nulls +cannot satisfy, which means anything except `IS NULL` and expressions reducible to it. This is one of +the most valuable pruning rules in practice, and it depends entirely on the three-valued logic above +being fixed. ## 4. Unknown components: resolve lazily, fail on use -[02-format.md](02-format.md) §5.3 covers the *contract* rule (a frozen ID means a frozen payload -shape). This section covers the *resolution* rule, which decides how long a reader survives format -evolution. +[Per-encoding metadata](02-format.md#53-per-encoding-metadata) covers the contract rule: a frozen id +means a frozen payload shape. This section covers the resolution rule, which decides how long a +reader survives the format's evolution. -**Unknown component IDs are resolved at open time into a known/unknown-inert classification, but -only fail when the component is actually required.** +Unknown component ids are classified at open time as known or unknown and inert, but only fail when +the component is actually needed. | Situation | Behavior | |---|---| -| Unknown array encoding in a column that is not projected | scan succeeds | +| Unknown array encoding in a column that is not projected | the scan succeeds | | Unknown array encoding in a projected column | `VortexUnsupportedException(id, ComponentKind.Array)` | -| Unknown layout encoding on a subtree never visited | scan succeeds | +| Unknown layout encoding on a subtree never visited | the scan succeeds | | Unknown layout encoding on the path to projected data | throws | -| Unknown extension dtype in the schema | schema exposes it as opaque; throws only if that field is read | -| Unknown zone-map aggregate | pruning for that aggregate is disabled; scan succeeds | +| Unknown extension dtype in the schema | the schema exposes it as opaque, and only reading that field throws | +| Unknown zone-map aggregate | pruning for that aggregate is disabled, and the scan succeeds | The aggregate row is already mandated by the format spec. Extending the same principle to arrays, layouts and dtypes is what lets a reader keep working when upstream freezes a new core edition and -the default writer starts emitting a new ID in *one* column of a fifty-column file. An -open-time hard failure would make that file entirely unreadable for no reason. +the default writer starts emitting a new id in one column of a fifty-column file. A hard failure at +open would make that file entirely unreadable for no reason. -Lazy resolution is unconditional, and there is no option to fail at open instead: the inspection -such an option would give is always there. `VortexFile.ArrayEncodings` and `LayoutEncodings` list -what the footer declares, each with whether this build supports it, and `vxdump` prints them. +Lazy resolution is unconditional, and there is no option to fail at open instead, because the +inspection such an option would give is always available. `VortexFile.ArrayEncodings` and +`LayoutEncodings` list what the footer declares, each with whether this build supports it, and +`vxdump` prints them. -Every such exception must name the component ID and kind, because that is exactly the input the -upstream troubleshooting procedure requires ("which edition, which minimum library version"). +Every such exception must name the component id and its kind, because that is exactly the input the +upstream troubleshooting procedure asks for ("which edition, which minimum library version"). ### Binary preservation on rewrite Preserving unknown nodes byte for byte through a read-modify-write cycle is a separate feature with invariants of its own: which segments are copied verbatim, what happens when the surrounding layout -changes, how edition enforcement applies to a component nobody can name. It is not offered; a +changes, how edition enforcement applies to a component nobody can name. It is not offered, and a rewrite fails on the component it cannot decode. ## 5. Untrusted hints: a three-class policy -Offsets and lengths are validated ([03-architecture.md](03-architecture.md) §6). The subtler class -is *semantic hints* the file asserts and a decoder might believe. Every field falls into exactly -one class: - -**Class I — memory safety depends on it. Validated unconditionally, whatever the cost.** -`DictMetadata.values_len` vs the actual values length and every code in `[0, values_len)`; -`BitPackedMetadata.bit_width` ≤ the ptype width; `BitPackedMetadata.offset` < 1024; -`RLEMetadata.offset` < 1024; `PatchesMetadata` indices in range; ALP `exp_e`/`exp_f` within the -target float's domain; `BoolMetadata.offset` < 8; a `vortex.listview` row's `offset + size` ≤ -`elements_len`, added **unsigned** so two legal i64 operands cannot wrap past the bound; a -`vortex.varbinview` reference view's 4-byte prefix equal to the value it points at; every buffer -index in `ArrayNode.buffers`; every segment index in `Layout.segments`; `alignment_exponent` ≤ 6 -(see §6). - -**Class II — only result correctness depends on it.** Monotonicity of `RunEndMetadata` ends, -`is_sorted`, `is_strict_sorted`, zone-map min/max. By default they are **not** validated, and the -threat model says so ([09-contracts.md](09-contracts.md) §4): a well-formed file that lies produces -wrong results, as in every format with embedded statistics, Parquet included. -`VortexOpenOptions.VerifyStatistics` validates them in O(n) at first decode, for files from +Offsets and lengths are always validated (see [parser safety](03-architecture.md#6-parser-safety)). +The subtler class is the semantic hints a file asserts and a decoder might believe. Every such field +falls into exactly one of three classes. + +Class I holds the fields memory safety depends on, and they are validated unconditionally, whatever +the cost: + +- `DictMetadata.values_len` against the actual values length, and every code in `[0, values_len)` +- `BitPackedMetadata.bit_width` ≤ the ptype width, and `BitPackedMetadata.offset` < 1024 +- `RLEMetadata.offset` < 1024 +- `PatchesMetadata` indices in range +- ALP `exp_e` and `exp_f` within the target float's domain +- `BoolMetadata.offset` < 8 +- a `vortex.listview` row's `offset + size` ≤ `elements_len`, added unsigned so two legal i64 + operands cannot wrap past the bound +- a `vortex.varbinview` reference view's 4-byte prefix, which must equal the value it points at +- every buffer index in `ArrayNode.buffers` and every segment index in `Layout.segments` +- `alignment_exponent` ≤ 6 (see [resource caps](#6-resource-caps)) + +Class II holds the fields only result correctness depends on: the monotonicity of `RunEndMetadata` +ends, `is_sorted`, `is_strict_sorted`, and zone-map min and max. By default they are not validated, +and the threat model says so (see [the threat model](09-contracts.md#4-threat-model)). A well-formed +file that lies produces wrong results, as in every format with embedded statistics, Parquet +included. `VortexOpenOptions.VerifyStatistics` validates them in O(n) at first decode, for files from producers a caller does not trust. -The distinction matters because a lying `is_sorted` cannot corrupt memory but a lying +The distinction matters because a lying `is_sorted` cannot corrupt memory, while a lying `values_len` can. Conflating the two either costs throughput everywhere or leaves a hole. -**Class III — pure performance hints.** `is_constant`, `uncompressed_size_in_bytes`, -`null_count` used only for fast paths. Ignored unless free to verify; never a correctness input. +Class III holds pure performance hints: `is_constant`, `uncompressed_size_in_bytes`, and +`null_count` when used only for fast paths. They are ignored unless free to verify, and are never a +correctness input. -A dedicated fuzzing mutation targets Class I fields specifically — a generic bit-flip fuzzer -almost never produces a *structurally valid* file with an out-of-range `values_len`, which is -precisely the input that finds these bugs. +A dedicated fuzzing mutation targets class I fields specifically. A generic bit-flip fuzzer almost +never produces a structurally valid file with an out-of-range `values_len`, which is precisely the +input that finds these bugs. ## 6. Resource caps -`VortexFormatException` on violation. Every cap is a constant, documented here so it is not +A violation throws `VortexFormatException`. Every cap is a constant, documented here so it is not invented per call site: | Cap | Value | Why | |---|---|---| -| `alignment_exponent` | ≤ 6 (64 bytes) | the field is a `u8`; `2^255` is expressible. 64 covers every legitimate alignment including 16-byte views | +| `alignment_exponent` | ≤ 6 (64 bytes) | the field is a `u8`, so `2^255` is expressible. 64 covers every legitimate alignment, including 16-byte views | | DType nesting depth | 64 | a 10 000-deep nested struct blows the stack during schema parsing, before any data is touched | | Layout tree depth | 64 | same | | Array tree depth | 64 | same | | Postscript length | 65527 (format-mandated) | enforced on write too, with an early, clear error | -| Decompressed segment size | configurable, default 256 MiB | a 1 KiB Zstd segment can claim to expand to 100 GiB | -| Metadata segment count | 16 (format-mandated) | — | -| FlatBuffers tables per traversal | 1 000 000 | depth cannot bound work: forward-only uoffsets exclude cycles but not *sharing*, so a few hundred bytes of shared children describe a DAG with 2^depth paths. Matches the reference verifiers' `max_tables` | +| Decompressed segment size | configurable, 256 MiB by default | a 1 KiB Zstd segment can claim to expand to 100 GiB | +| Metadata segment count | 16 (format-mandated) | | +| FlatBuffers tables per traversal | 1 000 000 | depth cannot bound work. Forward-only uoffsets exclude cycles but not sharing, so a few hundred bytes of shared children describe a DAG with 2^depth paths. Matches the reference verifiers' `max_tables` | -The decompression cap shapes the *design*, not just a constant: `ZstdDecompressor.Decompress` -decodes a whole frame into a pre-sized destination, so the declared size is validated against the -cap **before** allocating. A `vortex.zstd` frame must declare its content size, which has to match -its metadata; a `vortex.zstd_buffers` buffer is sized by the node's metadata. +The decompression cap shapes the design, not just a constant. `ZstdDecompressor.Decompress` decodes +a whole frame into a pre-sized destination, so the declared size is validated against the cap before +anything is allocated. A `vortex.zstd` frame must declare its content size, which has to match its +metadata, and a `vortex.zstd_buffers` buffer is sized by the node's metadata. ## 7. Buffer-level LZ4 -`Buffer.compression` admits `LZ4` ([02-format.md](02-format.md) §5.2), and the segment table -reserves a compression byte too. **A compressed buffer is refused**, with -`VortexUnsupportedException` naming the id `lz4` and the kind `compression`, and nothing is written -compressed. - -There is nothing to be conformant with. In the whole of Vortex 0.86.1, `lz4` appears in the two -schemas that declare the enum and the generated code that mirrors them, and nowhere else: the -reference writes `Compression::None`, never reads the field, and depends on no LZ4 implementation. -The schema names an algorithm and stops: it does not say whether the bytes would be a raw LZ4 block -or a frame, and it records no decompressed length anywhere, so a decoder could not even size its -output without inventing a rule. Writing one would mean choosing a framing and calling it the -format, the same mistake as inventing an order for `List` in the row encoder -([06-row-encoding.md](06-row-encoding.md) §6). - -Refusing is also the safer behaviour: the reference, which never inspects the field, would read -compressed bytes as data and return wrong values without a word. The decision changes the day -upstream implements it and a real file shows the framing. +`Buffer.compression` admits `LZ4` (see [the array schema](02-format.md#52-schema-arrayfbs)), and +the segment table reserves a compression byte too. A compressed buffer is refused with a +`VortexUnsupportedException` naming the id `lz4` and the kind `compression`, and nothing is ever +written compressed. + +There is nothing to conform to. In the whole of Vortex 0.86.1, `lz4` appears in the two schemas that +declare the enum and in the generated code that mirrors them, and nowhere else. The reference writes +`Compression::None`, never reads the field, and depends on no LZ4 implementation. The schema names an +algorithm and stops there: it does not say whether the bytes would be a raw LZ4 block or a frame, and +it records no decompressed length anywhere, so a decoder could not even size its output without +inventing a rule. Writing one would mean choosing a framing and calling it the format, the same +mistake as inventing an order for `List` in the row encoder (see [unsupported +types](06-row-encoding.md#6-unsupported-types)). + +Refusing is also the safer behaviour. The reference, which never inspects the field, would read +compressed bytes as data and silently return wrong values. The decision will change the day upstream +implements it and a real file shows the framing. diff --git a/docs/design/09-contracts.md b/docs/design/09-contracts.md index 59fa7684..8de44459 100644 --- a/docs/design/09-contracts.md +++ b/docs/design/09-contracts.md @@ -1,156 +1,157 @@ # Library contracts: concurrency, parallelism, versioning, threat model, observability, licensing -The promises made to a caller, as opposed to the promises made about bytes, written once so that -each is not deduced differently at each call site. +The promises made to a caller, as opposed to the promises made about bytes, written down once so +that each one is not deduced differently at each call site. ## 1. Concurrency and thread safety | type | contract | |---|---| -| `VortexSession` | **Thread-safe**, meant to be shared by every request of a process; its options are frozen when `Create` returns, and `VortexSession.Default` is immutable. Disposing it while a file it opened is still open throws `InvalidOperationException` naming the file | -| `VortexFile` | **Thread-safe.** Concurrent scans of one open file are expected; its footer, layout tree, index directory and the runs it caches are immutable or published once | -| `ISegmentSource` | implementations **must** be thread-safe; the built-in ones are | -| `Scan`, `Scan` | **Single use, one thread.** Composition returns the same builder; build it on one thread, then run one sink. A second sink throws `InvalidOperationException`; `ExplainAsync` may be asked before the sink | -| `GroupedScan`, `Aggregation`, `Aggregation`, `Projection`, `Projection`, the `Scan` of a result | **Single use, one thread**, like the scan they are built on: enumerating one runs its query once, and a second sink throws `InvalidOperationException` | +| `VortexSession` | thread-safe, and meant to be shared by every request of a process. Its options are frozen when `Create` returns, and `VortexSession.Default` is immutable. Disposing it while a file it opened is still open throws `InvalidOperationException` naming the file | +| `VortexFile` | thread-safe. Concurrent scans of one open file are expected, and its footer, layout tree, index directory and the runs it caches are immutable or published once | +| `ISegmentSource` | implementations must be thread-safe, and the built-in ones are | +| `Scan`, `Scan` | single use, one thread. Composition returns the same builder, so build it on one thread, then run one sink. A second sink throws `InvalidOperationException`. `ExplainAsync` may be called before the sink | +| `GroupedScan`, `Aggregation`, `Aggregation`, `Projection`, `Projection`, the `Scan` of a result | single use, one thread, like the scan they are built on. Enumerating one runs its query once, and a second sink throws `InvalidOperationException` | | the scan's enumerator | one consumer, as the language requires | -| `Columns`, `Column`, `BatchView` | **Bound to the loop body**, enforced by the compiler: `ref struct`s valid until the next `MoveNextAsync` or `DisposeAsync`, which cannot be stored, captured or carried across an `await`. The one escape the compiler does not see, a span copied into a variable declared outside the loop, is analyzer VX1001's | -| `RecordBatch` | **Owned by its consumer**, not thread-safe: it may be handed to another thread, through a channel for instance, and disposed there; its spans die with it | +| `Columns`, `Column`, `BatchView` | bound to the loop body, enforced by the compiler. They are `ref struct`s valid until the next `MoveNextAsync` or `DisposeAsync`, and cannot be stored, captured or carried across an `await`. The one escape the compiler does not see, a span copied into a variable declared outside the loop, is caught by analyzer VX1001 | +| `RecordBatch` | owned by its consumer, and not thread-safe. It may be handed to another thread (through a channel, for instance) and disposed there, and its spans become invalid with it | | `KeyCursorBuilder`, `KeyCursor` | not thread-safe, like the scan | -| `ColumnsBuilder`, `ColumnBuilder` | **One thread.** The builder a writer hands out is its own, reused after every `WriteAsync` | -| `VortexFileWriter` (an append included), `VortexFileIndexer` | one writer per file; an append is not atomic, and `VortexFileRepair` truncates a torn one | +| `ColumnsBuilder`, `ColumnBuilder` | one thread. The builder a writer hands out belongs to the writer and is reused after every `WriteAsync` | +| `VortexFileWriter` (an append included), `VortexFileIndexer` | one writer per file. An append is not atomic, and `VortexFileRepair` truncates a torn one | | `ScanPlan`, `CountPlan`, `OrderPlan`, `KeyPlan`, `ScanStatistics`, `WriteReport` | immutable records | -| decoders and kernels | pure functions over borrowed memory, no shared mutable state | +| decoders and kernels | pure functions over borrowed memory, with no shared mutable state | ## 2. Parallelism -A ratio against Rust means something only at equal threading, and a library must not take a host's -thread pool without consent: a server running 200 concurrent requests does not want each scan +A ratio against Rust only means something at equal threading, and a library must not take over a +host's thread pool without consent: a server running 200 concurrent requests does not want each scan fanning out. So: -- **Sequential by default.** `VortexSessionOptions.MaxDegreeOfParallelism` is 1 unless set, and +- The default is sequential. `VortexSessionOptions.MaxDegreeOfParallelism` is 1 unless set, and `VortexSession.Default` is immutable, so a caller who configures nothing gets one thread. -- **Consent is given on the session, once**: its degree is where every scan and every writer of the - session starts, for a host that decides its threading at start-up rather than at a hundred call - sites. Two sessions in one process keep two degrees. -- **The call may say otherwise**: `ScanOptions.DegreeOfParallelism` for one scan and - `VortexWriteOptions.DegreeOfParallelism` for one file, 0 taking the session's. -- **A scan** decodes its splits side by side, each on a context and arenas of its own, nothing - shared, and still delivers its batches in row order. A take, whose splits are many and small, - keeps up to three a lane decoded ahead of its consumer, so the lanes that finish first do not - wait for the slowest; its degree still bounds how many decode at once. **A query** that streams - runs on that ordered stream of batches, its decode parallel; a blocking aggregation keeps a state - per group on each lane and merges the lanes' states part by part of the key space, as tasks it - awaits, never a thread blocked on another. The target, [16-queries.md](16-queries.md) §9.4 and - §9.5, folds each lane's rows into a cache of bounded capacity and holds every group once, in the - sub-tables of 256 parts. Its answers are the same bits at every degree, float sums included, and - stay so at every capacity of the caches and every order the states merge in. **A dataset** opens its objects ahead of the one being read and reads them side by - side. **A writer** summarizes its columns and compresses a - column's zstd frames on its threads, and chooses and writes the encodings on the calling one; the - file is the same bytes at any degree. -- **I/O concurrency is separate and always on**: the batch read of the seam issues overlapping reads - whatever the degree, because latency hiding is the point on an object store. It is bounded per - session by `MaxConcurrentReads`, 16 by default, across every scan of every file. A mapped file and - bytes in memory have no reads to bound. - -[threads.md](../guide/threads.md) shows what each costs and when parallelism pays. +- Consent is given once, on the session. Its degree is where every scan and every writer of the + session starts, which suits a host that decides its threading at start-up rather than at a hundred + call sites. Two sessions in one process keep two separate degrees. +- A single call may override it: `ScanOptions.DegreeOfParallelism` for one scan and + `VortexWriteOptions.DegreeOfParallelism` for one file, where 0 takes the session's. +- A scan decodes its splits side by side, each on a context and arenas of its own with nothing + shared, and still delivers its batches in row order. A take, whose splits are many and small, keeps + up to three per lane decoded ahead of its consumer, so the lanes that finish first do not wait for + the slowest, and its degree still bounds how many decode at once. +- A streaming query runs on that ordered stream of batches, with its decoding in parallel. A blocking + aggregation keeps a state per group on each lane and merges the lanes' states one part of the key + space at a time, as tasks it awaits, never a thread blocked on another. The design (see + [parallelism and the merge](16-queries.md#94-parallelism-and-the-merge) and + [memory](16-queries.md#95-memory)) folds each lane's rows into a cache of bounded capacity and holds + every group once, in sub-tables of 256 parts. Its answers are the same bits at every degree, float + sums included, and stay so at every cache capacity and every merge order. +- A dataset opens its objects ahead of the one being read, and reads them side by side. +- A writer summarizes its columns and compresses a column's zstd frames on its threads, and chooses + and writes the encodings on the calling one. The file is the same bytes at any degree. +- I/O concurrency is separate and always on. The batch read of the I/O interface issues overlapping + reads whatever the degree, because hiding latency is the point on an object store. It is bounded + per session by `MaxConcurrentReads`, 16 by default, across every scan of every file. A mapped file + and bytes in memory have no reads to bound. + +[threads.md](../guide/threads.md) shows what each one costs and when parallelism pays off. ## 3. Versioning -Semantic versioning, with each level spelled out, since a format library's version has two axes: -its API and the format it understands. +The library follows semantic versioning, with each level spelled out, since a format library's +version has two axes: its API and the format it understands. | change | level | |---|---| | reading a newly frozen upstream edition (more ids) | minor | | an encoder for an encoding already decoded | minor | | a case that threw `VortexFormatException` now throwing `VortexUnsupportedException` | minor | -| dropping or adding a target framework | **major** | -| changing the lifetime of borrowed spans | **major** | -| changing the default write edition | **major**: it changes who can read the output | -| the row encoding's bytes following upstream | not applicable, below | - -The default write edition is `VortexEditions.Default`, and it is not the newest by definition: it is -the edition the most deployed Rust reader accepts, chosen per release and named in the release -notes beside the minimum Rust version that reads it (`VortexEditions.MinimumRustVersion`). It is -`core2026.08.3`, read from Vortex Rust 0.85.0: the first edition with `vortex.uuid`, without -which a `Guid` column cannot be written. - -**The row encoding ships apart**, in `Vorticity.RowEncoding`, versioned `0.x` and marked -experimental: upstream reserves the right to change its bytes between releases, and binding it into +| dropping or adding a target framework | major | +| changing the lifetime of borrowed spans | major | +| changing the default write edition | major, since it changes who can read the output | +| the row encoding's bytes following upstream | not applicable, see below | + +The default write edition is `VortexEditions.Default`, and it is not the newest by definition. It is +the edition the most widely deployed Rust reader accepts, chosen per release and named in the release +notes next to the minimum Rust version that reads it (`VortexEditions.MinimumRustVersion`). Today it +is `core2026.08.3`, read by Vortex Rust 0.85.0 and later, the first edition with `vortex.uuid`, +without which a `Guid` column cannot be written. + +The row encoding ships separately, in `Vorticity.RowEncoding`, versioned `0.x` and marked +experimental. Upstream reserves the right to change its bytes between releases, and binding it into a stable core would force a choice between breaking this library's semver and diverging from Rust, -which would lose the byte-exactness that is the feature's point. +which would lose the byte-exactness that is the whole point of the feature. Each release records the Vortex version its corpus was generated with and the highest edition it -reads: that pair, more than the version number, is what a user needs. +reads. That pair, more than the version number, is what a user needs. ## 4. Threat model -Two sentences, because the alternative is an unbounded promise: +It fits in two sentences, because the alternative is an unbounded promise: -1. **Malformed input** — bad offsets, truncation, out-of-range fields memory depends on, exceeded - caps — produces a clean `VortexFormatException` or `VortexUnsupportedException`: no out-of-bounds - access, no unbounded allocation, no hang. Fuzzing enforces it. -2. **Well-formed but lying input** — valid structure over false statistics — may produce wrong - results, as in every format with embedded statistics, Parquet included. +1. Malformed input (bad offsets, truncation, out-of-range fields that memory depends on, exceeded + caps) produces a clean `VortexFormatException` or `VortexUnsupportedException`, with no + out-of-bounds access, no unbounded allocation and no hang. Fuzzing enforces it. +2. Well-formed but lying input (a valid structure over false statistics) may produce wrong results, + as in every format with embedded statistics, Parquet included. `VortexOpenOptions.VerifyStatistics` trades throughput for validation when the producer is not - trusted ([08-semantics.md](08-semantics.md) §5). An index that lies costs pruning, never rows: - its regions carry checksums, and a failed one claims nothing - ([10-indexes.md](10-indexes.md) §3). + trusted (see [untrusted hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). An index + that lies costs pruning, never rows: its regions carry checksums, and a region that fails claims + nothing (see [the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)). Not claimed: protection against resource exhaustion below the configured caps, or any confidentiality property (the format's encryption slot is an empty reserved table). ## 5. Observability -Three channels, each free when nobody listens. +There are three channels, each free when nobody listens. -**A meter and an activity source**, both named `Vorticity` (`VortexDiagnostics`), which +The first is a meter and an activity source, both named `Vorticity` (`VortexDiagnostics`), which OpenTelemetry and `dotnet-counters` read without an adapter. Each instrument is created on its own, -so a listener enabling one pays for that one: +so a listener that enables one only pays for that one: | instrument | diagnoses | |---|---| | `vortex.scan.rows` | rows delivered | -| `vortex.scan.requests`, `vortex.scan.bytes_requested` | what the scans asked their sources for: read amplification, against the plan's `BytesToRead` | +| `vortex.scan.requests`, `vortex.scan.bytes_requested` | what the scans asked their sources for, which shows read amplification against the plan's `BytesToRead` | | `vortex.scan.blocks_decoded`, `vortex.scan.blocks_pruned` | whether pruning works at all | | `vortex.cache.hits`, `vortex.cache.misses` | whether the session's segment cache does anything | -| `vortex.write.bytes` | bytes the writers handed their sinks | +| `vortex.write.bytes` | bytes the writers handed to their sinks | -A scan is one activity, `vortex.scan.typed` or `vortex.scan.tool`, tagged when it ends with its -rows, batches, requests, bytes requested and blocks decoded and pruned; a write is one activity, -`vortex.write`, tagged with its rows, bytes and whether it completed, and marked as an error when it -is abandoned. +A scan is one activity, `vortex.scan.typed` or `vortex.scan.tool`, tagged when it ends with its rows, +batches, requests, bytes requested, and blocks decoded and pruned. A write is one activity, +`vortex.write`, tagged with its rows, its bytes and whether it completed, and marked as an error when +it is abandoned. -**An event source**, `Vorticity`, whose polling counters say whether an index earns its bytes: -`segments-requested`, `bytes-requested`, `zones-pruned`, `zones-total`, `index-runs-read`, +The second is an event source, `Vorticity`, whose polling counters say whether an index earns its +bytes: `segments-requested`, `bytes-requested`, `zones-pruned`, `zones-total`, `index-runs-read`, `cursor-seeks`, `cursor-steps`, `count-blocks-proven`, `count-blocks-decoded`, `key-order-windows` and `key-order-window-splits`. -**Per query**: a scan's `ExplainAsync` is its plan before it runs, and its `Statistics` the same -quantities measured after ([observability.md](../guide/observability.md)). +The third works per query: a scan's `ExplainAsync` is its plan before it runs, and its `Statistics` +the same quantities measured afterwards ([observability.md](../guide/observability.md)). ## 6. Licensing and attribution -Vorticity is licensed under Apache-2.0 (`LICENSE`, the canonical text). +Vorticity is licensed under Apache-2.0 (`LICENSE` holds the canonical text). -Upstream material arrives under **two** licenses, which Vortex's `REUSE.toml` splits: +Upstream material arrives under two licenses, which Vortex's `REUSE.toml` splits: | upstream material | license | what this repository does with it | |---|---|---| -| source tree, including the `.fbs` and `.proto` schemas and the format constants | Apache-2.0 | schemas vendored verbatim under `spec/`; constants transcribed into `docs/` | -| documentation at docs.vortex.dev and upstream `docs/` | **CC BY 4.0** | condensed and adapted throughout `docs/` | +| source tree, including the `.fbs` and `.proto` schemas and the format constants | Apache-2.0 | schemas vendored verbatim under `spec/`, constants transcribed into `docs/` | +| documentation at docs.vortex.dev and upstream `docs/` | CC BY 4.0 | condensed and adapted throughout `docs/` | -Two obligations bind, and both are met in `NOTICE`: +Two obligations apply, and both are met in `NOTICE`: -1. **Apache-2.0 §4(c)**: the vendored schemas keep their SPDX copyright and license headers, and +1. Apache-2.0, section 4(c): the vendored schemas keep their SPDX copyright and license headers, and `NOTICE` lists each vendored file against its upstream path. -2. **CC BY 4.0**: adapting the upstream documentation requires attribution, a link to the license, - and an **indication that changes were made**, which citing the source alone does not give. +2. CC BY 4.0: adapting the upstream documentation requires attribution, a link to the license, and + an indication that changes were made, which citing the source alone does not provide. -`NOTICE` also records, per component, whether it was **reimplemented from the specification** or -**translated from reference code**; only the second inherits the upstream license's attribution -requirements. Reimplementation from the specification is the rule, with the reference consulted -for semantics. +`NOTICE` also records, per component, whether it was reimplemented from the specification or +translated from reference code. Only the second inherits the upstream license's attribution +requirements. Reimplementation from the specification is the rule, with the reference consulted for +semantics. "Vortex" is a trademark of LF Projects, LLC. Vorticity is an independent implementation, not affiliated with or endorsed by the Vortex project, and says so at the top of `NOTICE` and of the diff --git a/docs/design/10-indexes.md b/docs/design/10-indexes.md index 26e4bb5a..4aef39fb 100644 --- a/docs/design/10-indexes.md +++ b/docs/design/10-indexes.md @@ -1,326 +1,334 @@ # Skipping and locating indexes What an index is in a Vorticity file: its kinds, its bytes, where it lives, how it survives an -append, and how it stays invisible to every other reader. What a caller can *ask* of an index — a -cursor, a count, key-ordered rows — is [12-index-reads.md](12-index-reads.md); how the writer builds -one in its single pass is [11-write-strategy.md](11-write-strategy.md). - -An index is an **optional accelerator**. A reader that ignores it returns the same rows as a reader -that uses it, and [08-semantics.md](08-semantics.md) §1's invariant — pruning never removes a row a -full scan returns — holds for every kind. It is also **ahead of upstream**: Vortex 0.86.1 ships a -Bloom filter aggregate its authors call unstable and no edition declares, and its index designs are -proposals. This design takes their structure, and adds one rule they have not fixed: **a file with -an index stays readable by a strict Rust 0.86.1 reader, with the index simply invisible to it.** -That rule decides where an index lives (§2), and the rest follows. +append, and how it stays invisible to every other reader. What a caller can ask of an index (a +cursor, a count, key-ordered rows) is covered in [12-index-reads.md](12-index-reads.md), and how the +writer builds one in its single pass in [11-write-strategy.md](11-write-strategy.md). + +An index is an optional accelerator. A reader that ignores it returns the same rows as a reader that +uses it, and the pruning invariant (pruning never removes a row a full scan returns, see [statistic +precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)) holds for every kind. It is +also ahead of upstream. Vortex 0.86.1 ships a Bloom filter aggregate its authors call unstable and no +edition declares, and its index designs are still proposals. This design takes their structure and +adds one rule they have not fixed: a file with an index stays readable by a strict Rust 0.86.1 +reader, with the index simply invisible to it. That rule decides [where an index +lives](#2-where-an-index-lives), and the rest follows. ## 1. Terms | term | meaning | |---|---| -| **skipping index** | a conservative summary per region, "no row here can match": it clears bits of a block mask | -| **locating index** | a value-keyed structure, "the rows or blocks where this value occurs": it may answer a predicate outright | -| **exactness** | `Exact`: the rows it names are exactly the matching ones. `Superset`: what it clears is proven, what it keeps is "maybe" | -| **granularity** | a row, a block (the zone length, 8 192 rows by default, counted from row 0), a node of 16 blocks, the file | -| **run** | an immutable payload over a contiguous range of blocks; an index is a sequence of runs | +| skipping index | a conservative summary per region, "no row here can match", which clears bits of a block mask | +| locating index | a value-keyed structure, "the rows or blocks where this value occurs", which may answer a predicate outright | +| exactness | `Exact` means the rows it names are exactly the matching ones. `Superset` means what it clears is proven and what it keeps is "maybe" | +| granularity | a row, a block (the zone length, 8 192 rows by default, counted from row 0), a node of 16 blocks, or the file | +| run | an immutable payload over a contiguous range of blocks. An index is a sequence of runs | -An index is a **kind** (a versioned string), a **column path**, a **block length** and its -**runs**, which are ordinary Vortex arrays. Everything a reader needs to decide whether it can use -an index is in its directory entry; everything it needs to probe it is in the runs. +An index is a kind (a versioned string), a column path, a block length and its runs, which are +ordinary Vortex arrays. Everything a reader needs to decide whether it can use an index is in its +directory entry, and everything it needs to probe it is in the runs. ## 2. Where an index lives -**The rule.** A file with an index opens and scans, with no configuration, in Rust 0.86.1 with a -strict session (the default, and what DuckDB, DataFusion, Spark and the Python bindings use), in -any earlier Vorticity, and in this one, which then uses the index. +The rule is that a file with an index opens and scans, with no configuration, in Rust 0.86.1 with a +strict session (the default, and what DuckDB, DataFusion, Spark and the Python bindings use), in any +earlier Vorticity, and in this one, which then uses the index. -**The carrier is a postscript metadata entry.** The postscript already carries up to sixteen -`(key, segment)` pairs that every reader loads only on request ([02-format.md](02-format.md) §2). -One entry, `vorticity.index`, holds the **index directory** (§3). The directory points at **run -segments**: regions written after the data and before the footer, 64-byte aligned, referenced by -neither the layout tree nor the footer's segment table. A reader that does not ask for the key never -touches them; Rust validates that the directory's own segment lies inside the file, which it does, -and does not know the runs exist. A second entry, `vorticity.identity`, is the file's identity -([13-dataset.md](13-dataset.md) §7), which leaves fourteen for the caller. +The carrier is a postscript metadata entry. The postscript already carries up to sixteen +`(key, segment)` pairs that every reader only loads on request (see [the +postscript](02-format.md#2-postscript-footerfbs)). One entry, `vorticity.index`, holds the index +directory ([below](#3-the-directory-and-the-runs)). The directory points at run segments: regions +written after the data and before the footer, 64-byte aligned, referenced by neither the layout tree +nor the footer's segment table. A reader that does not ask for the key never touches them. Rust +checks that the directory's own segment lies inside the file, which it does, and does not know the +runs exist. A second entry, `vorticity.identity`, is the file's identity (see [identity and +integrity](13-dataset.md#7-identity-and-integrity)), which leaves fourteen entries for the caller. -**The carriers not used:** +The carriers not used: | carrier | in a strict Rust 0.86.1 reader | verdict | |---|---|---| -| a new aggregate in `vortex.zoned` | the open fails: unknown aggregate | only once an edition declares one, with upstream's id and bytes (§8) | -| a wrapper layout, upstream's `vortex.indexed` | the open fails, and the data beneath is unreachable | the migration target when it freezes (§8) | -| a new field in the postscript table | ignored | refused: it would take a field id upstream may assign | -| a second object beside the file | unaffected | used, for stores that cannot append, as a fragment (§7) | +| a new aggregate in `vortex.zoned` | the open fails on an unknown aggregate | only once an edition declares one, with upstream's id and bytes ([the road upstream](#8-compatibility-and-the-road-upstream)) | +| a wrapper layout, upstream's `vortex.indexed` | the open fails, and the data beneath is unreachable | the migration target once it freezes ([the road upstream](#8-compatibility-and-the-road-upstream)) | +| a new field in the postscript table | ignored | refused, since it would take a field id upstream may assign | +| a second object next to the file | unaffected | used, for stores that cannot append, as a fragment ([indexing after the fact](#7-building-appending-and-indexing-after-the-fact)) | ## 3. The directory and the runs -**The directory** is a Protobuf message of this library's, preceded by a version byte as +The directory is a Protobuf message of this library's own, preceded by a version byte the way `vortex.zoned`'s metadata is, and followed by an XXH3-64 of both (`Indexes/IndexDirectory.cs`). It holds the file's row count when it was written, the write policy (so an append needs no options), -the index budget, and one entry per index: its kind, column path, block length, kind-defined -options, and runs. A run names its first block and block count, its payload regions — each with its -offset, length, alignment, **XXH3-64 checksum** and array dtype — its entry count for a locating -kind, and kind-defined options. +the index budget, and one entry per index: its kind, column path, block length, kind-defined options +and runs. A run names its first block and block count, its payload regions (each with its offset, +length, alignment, XXH3-64 checksum and array dtype), its entry count for a locating kind, and +kind-defined options. -**What a reader checks** before it believes an entry, under the hint policy of -[08-semantics.md](08-semantics.md) §5: +A reader checks the following before it believes an entry, under the policy for [untrusted +hints](08-semantics.md#5-untrusted-hints-a-three-class-policy): -- the directory's checksum, and that its row count is the file's: otherwise the whole directory is - ignored, and the file reads without it; +- the directory's checksum, and that its row count is the file's. Otherwise the whole directory is + ignored, and the file reads without it. - that the column path resolves to a column whose dtype the kind supports, that the kind and its - options are known, and that the runs are disjoint, in order and inside the file: otherwise the - **entry** is ignored, never the file; -- every region it reads, against its checksum: a region that fails claims nothing in a pruner and + options are known, and that the runs are disjoint, in order and inside the file. Otherwise the + entry is ignored, never the file. +- every region it reads, against its checksum. A region that fails claims nothing in a pruner, and makes a key source refuse. -**An index can only ever cost pruning, never correctness.** A block no run covers is live; a zeroed -filter is caught by its checksum rather than dropping rows. +An index can therefore only ever cost pruning, never correctness. A block no run covers stays live, +and a zeroed filter is caught by its checksum rather than dropping rows. -**Runs are ordinary arrays.** Each payload is an array blob, the form of a `vortex.flat` segment, +Runs are ordinary arrays. Each payload is an array blob, the same form as a `vortex.flat` segment, written with the file's encoding table and compressed like data under the same target edition. An index therefore inherits every encoding: a sorted key column bit-packs or FSST-compresses like any -column, and no index defines a binary format of its own, which is upstream's own design decision -for locating indexes. The footer may list an encoding used only by an index run, which is harmless, +column, and no index defines a binary format of its own, which matches upstream's own design decision +for locating indexes. The footer may list an encoding used only by an index run, which is harmless since every id is in the target edition. -**A run is immutable and covers a contiguous block range.** A write produces **one run per entry**: -chunk runs are spilled as they close and merged when the file completes, plus a run of its own for -the last chunk when the rows are not whole blocks, since an append re-opens that chunk. An append -adds runs, and merges an entry's runs once it holds more than **K = 4**, so a lookup reads at most K -runs. A rewrite writes one. +A run is immutable and covers a contiguous range of blocks. A write produces one run per entry. Chunk +runs are spilled as they close and merged when the file completes, plus a separate run for the last +chunk when the rows do not fill whole blocks, since an append reopens that chunk. An append adds runs, +and merges an entry's runs once it holds more than four, so a lookup reads at most four runs. A +rewrite writes a single run. -**Long runs are paged.** A locating run of at most 64 segments of 65 536 entries lists its segments -inline. A longer one writes its segment table as **fence pages** of at most 64 KiB, a tree whose root -is inline in the run's options and whose pages carry their children's checksums; two levels cover -more than 10¹² entries, so a lookup reads at most two fence pages before its payload +Long runs are paged. A locating run of at most 64 segments of 65 536 entries lists its segments +inline. A longer one writes its segment table as fence pages of at most 64 KiB, forming a tree whose +root is inline in the run's options and whose pages carry their children's checksums. Two levels +cover more than 10¹² entries, so a lookup reads at most two fence pages before its payload (`Indexes/FenceTable.cs`). -**Bloom filters form a tree.** A run's filters are laid out with a fan-out of 16: the leaves are -the per-block filters, an internal node filters the union of its 16 children and lists their sizes, -and the children of a node are contiguous behind it, so a probe reads the root and then one region -per node that still says "maybe", level by level, in one coalesced read per level -(`Indexes/BloomTree.cs`). A node whose union exceeds what `max_blocks` holds at the target rate is +Bloom filters form a tree. A run's filters are laid out with a fan-out of 16: the leaves are the +per-block filters, an internal node filters the union of its 16 children and lists their sizes, and +the children of a node sit contiguously behind it. A probe reads the root and then one region per +node that still says "maybe", level by level, in one coalesced read per level +(`Indexes/BloomTree.cs`). A node whose union exceeds what `max_blocks` can hold at the target rate is built without a filter, and the descent starts below it. ## 4. Skipping indexes -### 4.1 `vorticity.bloom.sbbf.v1` — a split-block Bloom filter per block +### 4.1 `vorticity.bloom.sbbf.v1`: a split-block Bloom filter per block -**Structure.** `n` blocks of 256 bits (eight 32-bit words). Insert: `h = XXH3-64(bytes, seed 0)`; -block `((h >> 32) × n) >> 32`; in it, set one bit per word with the eight Parquet salts applied to -`(uint)h`. Probe: the same eight bits all set. The layout and salts are Parquet's and the hash is -upstream's, so a filter is **bit-identical to Rust's `BloomPartial` of 0.86.1**, tested against -vectors it built, and becomes upstream's by a change of id the day they freeze it. A variant with -Parquet's xxHash64 is **bit-identical to a Parquet SBBF**, tested against the `parquet` crate's own -filters, for a caller who exchanges filters with Parquet tooling. +A filter is `n` blocks of 256 bits (eight 32-bit words). To insert, compute `h = XXH3-64(bytes, seed +0)`, pick block `((h >> 32) × n) >> 32`, and in it set one bit per word using the eight Parquet salts +applied to `(uint)h`. A probe checks that the same eight bits are all set. The layout and salts are +Parquet's and the hash is upstream's, so a filter is bit-identical to Rust's `BloomPartial` of +0.86.1, tested against vectors it built, and becomes upstream's by a change of id the day they freeze +it. A variant using Parquet's xxHash64 is bit-identical to a Parquet SBBF, tested against the +`parquet` crate's own filters, for a caller who exchanges filters with Parquet tooling. -**What is hashed**, so that equal values hash equal whatever encoding they came from: +What is hashed is chosen so that equal values hash equal whatever encoding they came from: | dtype | bytes hashed | |---|---| -| integers, floats, decimals | the value's little-endian bytes at the column's width; floats by bit pattern, so `-0.0` and `+0.0` are two values | +| integers, floats, decimals | the value's little-endian bytes at the column's width. Floats hash by bit pattern, so `-0.0` and `+0.0` are two values | | text, binary | the bytes | | extension | its storage value's bytes | -| the elements of a list | each element, into the **row's** block: "does any element of a row in this block equal v", the question `ListContains` asks | -| bool | not indexed: a two-value domain has nothing to skip | -| null | not inserted: `x = v` and `x IN (…)` never match a null | +| the elements of a list | each element, into the row's block, which answers "does any element of a row in this block equal v", the question `ListContains` asks | +| bool | not indexed, since a two-value domain has nothing to skip | +| null | not inserted, since `x = v` and `x IN (…)` never match a null | -**Size.** Built when a block closes, from its **exact** distinct count: the smallest power of two +A filter is built when a block closes, sized from its exact distinct count: the smallest power of two of blocks that gives the target false-positive rate (1 % by default) by Parquet's sizing formula, -clamped to `max_blocks` (4 096 blocks, 128 KiB). A block under `min_distinct` (8) distinct values -gets no filter: the zone map or the dictionary answers equality there. A tree node is sized from the -exact union of its blocks' values, never clamped into a saturated filter. The file-level filter is -the root of the tree when the policy asks for three resolutions (§4.4). +clamped to `max_blocks` (4 096 blocks, 128 KiB). A block with fewer than `min_distinct` (8) distinct +values gets no filter, since the zone map or the dictionary answers equality there. A tree node is +sized from the exact union of its blocks' values, never clamped into a saturated filter. The +file-level filter is the root of the tree when the policy asks for three resolutions (see [the +file-level filter](#44-the-file-level-filter)). -**Probe rules**, as a pruner after the zone maps in the scan's block mask -([11-write-strategy.md](11-write-strategy.md) §6): +The filters act as a pruner after the zone maps in the scan's block mask (see [reading what the +writer produced](11-write-strategy.md#6-reading-what-the-writer-produced)), with these rules: | predicate | proof | |---|---| | `x = v` | a block whose filter does not contain `v` is pruned | -| `x IN (v1 … vn)` | pruned if it contains none; the literals are hashed once per query | -| `AND`, `OR` | a conjunction of proofs; pruned under `OR` only if every arm prunes | +| `x IN (v1 … vn)` | pruned if the filter contains none of them. The literals are hashed once per query | +| `AND`, `OR` | a conjunction of proofs, and under `OR` a block is pruned only if every arm prunes it | | `ListContains(x, v)` | as `x = v`, over the element filters | -| `!=`, `<`, `LIKE` | no claim (§4.2 for `LIKE`) | +| `!=`, `<`, `LIKE` | no claim (see [trigrams](#42-vorticitybloomngram3v1-trigrams-for-text-predicates) for `LIKE`) | -A literal is hashed only when it converts exactly into the column's type — an integer past 2⁵³ from -a double, a non-integral float or an out-of-range integer claims nothing — a zero asks for both -zeros, and a NaN asks for nothing. +A literal is only hashed when it converts exactly into the column's type. An integer beyond 2⁵³ from +a double, a non-integral float or an out-of-range integer claims nothing, a zero asks for both zeros, +and a NaN asks for nothing. -### 4.2 `vorticity.bloom.ngram3.v1` — trigrams for text predicates +### 4.2 `vorticity.bloom.ngram3.v1`: trigrams for text predicates -The same structure over every byte **trigram** of every value of a text or binary column. +This is the same structure over every byte trigram of every value of a text or binary column. `StartsWith`, `Contains` and `LIKE` require the trigrams of each literal run of three bytes or more between their wildcards, and one absent trigram prunes the block. An option folds ASCII case on both -sides; a folded index still answers a case-sensitive predicate, since a value containing `Foo` -contains `foo` once folded. `Superset`: the predicate is always evaluated on the survivors. +sides. A folded index still answers a case-sensitive predicate, since a value containing `Foo` +contains `foo` once folded. It is a `Superset` index, so the predicate is always evaluated on the +survivors. -### 4.3 `vorticity.dict.probe.v1` — the dictionary is already an index +### 4.3 `vorticity.dict.probe.v1`: the dictionary is already an index -When the writer dictionary-encoded a chunk, the chunk's values child *is* the set of its distinct -values, exact. This kind has **no payload**: its entry says which chunks of the column are +When the writer dictionary-encoded a chunk, the chunk's values child already is the exact set of its +distinct values. This kind therefore has no payload. Its entry says which chunks of the column are dictionaries, and the pruner answers `x = v` and `x IN (…)` from those values without decoding the -codes, last in the chain since it reads the column's own bytes. A chunk claims only the blocks it -covers whole. It costs nothing to write. What it costs to read is the chunk's segment, since a flat -chunk is one segment: it pays when the probed column is one of several — on a 1M-row table of six -columns, an absent value reads 0.073× the bytes of the scan — and breaks even on a file of one +codes, last in the chain since it reads the column's own bytes. A chunk only claims the blocks it +covers whole. It costs nothing to write. Reading it costs the chunk's segment, since a flat chunk is +one segment, so it pays off when the probed column is one of several (on a 1M-row table of six +columns, an absent value reads 0.073× the bytes of the scan) and breaks even on a file with a single column. ### 4.4 The file-level filter -The root of each run's tree filters the whole run. A scan's plan reads the roots with the file -statistics, and `ScanPlan.MayMatch` is false when either proves the scan empty; `VortexFile.MayMatch` +The root of each run's tree filters the whole run. A scan's plan reads the roots along with the file +statistics, and `ScanPlan.MayMatch` is false when either proves the scan empty. `VortexFile.MayMatch` answers from the statistics alone, without a read, for an engine that prunes many files before -opening any. An append does not extend a tree, since filters of different sizes do not merge: it -adds a tree for its own blocks. +opening any. An append does not extend a tree, since filters of different sizes do not merge. +Instead it adds a tree for its own blocks. ## 5. Locating indexes -### 5.1 `vorticity.postings.blocks.v1` — value to blocks +### 5.1 `vorticity.postings.blocks.v1`: value to blocks -A run holds the distinct values of its range, sorted, and for each the blocks it occurs in: three -arrays, `keys`, `offsets` and `blocks`, compressed like data. A lookup locates `v` in the keys — -the run's segment table skips to the one segment that can hold it — and keeps exactly the blocks -listed. `Exact` at block granularity: a kept block holds the value, though not in every row. The -keys alone serve a walk of distinct values ([12-index-reads.md](12-index-reads.md) §2). +A run holds the distinct values of its range, sorted, and for each one the blocks it occurs in, as +three arrays (`keys`, `offsets` and `blocks`) compressed like data. A lookup locates `v` in the keys, +where the run's segment table skips straight to the one segment that can hold it, and keeps exactly +the blocks listed. It is `Exact` at block granularity: a kept block holds the value, though not in +every row. The keys alone serve a walk of distinct values (see [sources of +order](12-index-reads.md#2-sources-of-order)). -### 5.2 `vorticity.sorted.runs.v1` — value to rows, exact +### 5.2 `vorticity.sorted.runs.v1`: value to rows, exact -A run holds its range's keys sorted, with a row position per entry (`u32` relative to the run's -first row, `u64` when a run spans 2³² rows or more). It answers `x = v`, `x IN (…)` and ranges -**exactly**: a scan takes its rows as a selection and does not evaluate the predicate again. It is +A run holds its range's keys sorted, with a row position per entry (a `u32` relative to the run's +first row, or a `u64` when a run spans 2³² rows or more). It answers `x = v`, `x IN (…)` and ranges +exactly, so a scan takes its rows as a selection and does not evaluate the predicate again. It is also the key source of a cursor and of key-ordered delivery ([12-index-reads.md](12-index-reads.md)). -Keys are in the total order of [12-index-reads.md](12-index-reads.md) §3.4, and the run is built by a -sort of each chunk's distinct keys, a counting sort of its rows by rank, and a k-way merge of the -chunk runs: memory is one chunk and the merge's buffers, never the column. +Keys follow the total order described in [key order per +dtype](12-index-reads.md#34-key-order-per-dtype). The run is built by sorting each chunk's distinct +keys, counting-sorting its rows by rank, and k-way merging the chunk runs, so memory is one chunk plus +the merge's buffers, never the whole column. -### 5.3 `vorticity.postings.ngram3.v1` — trigrams to blocks +### 5.3 `vorticity.postings.ngram3.v1`: trigrams to blocks -§5.1 with the trigrams of a text column as keys, typed `binary` since a trigram may cut a UTF-8 -character: the locating form of §4.2, larger than the filter and exact at block granularity for the -conjunction of a pattern's trigrams. +This is the [postings index](#51-vorticitypostingsblocksv1-value-to-blocks) with the trigrams of a +text column as keys, typed `binary` since a trigram may cut a UTF-8 character. It is the locating +form of the [trigram filter](#42-vorticitybloomngram3v1-trigrams-for-text-predicates), larger than +the filter and exact at block granularity for the conjunction of a pattern's trigrams. ### 5.4 Composite keys, through the row encoding A sorted run over `(a, b, …)` keys each row by the row encoding of the tuple ([06-row-encoding.md](06-row-encoding.md)): bytes whose `memcmp` order is the tuple order, so a -prefix on the leading columns is a range. The entry keeps the kind, leaves its column path empty, -and lists its key columns in its options with the name of the encoding its bytes follow -(`vortex-row 0.86.1 …`), since that format is experimental upstream. The core ships no encoder: +prefix on the leading columns is a range. The entry keeps the kind, leaves its column path empty, and +lists its key columns in its options together with the name of the encoding its bytes follow +(`vortex-row 0.86.1 …`), since that format is experimental upstream. The core ships no encoder. `IndexPolicy.ForKey(columns, kind, encoder)` takes an `IKeyEncoder`, and the row-encoding package's -`RowKeyEncoder` is one. A tuple holding a null is no entry. +`RowKeyEncoder` is one. A tuple holding a null produces no entry. ### 5.5 Considered and not built -- **Hash to rows.** Sorted runs whose keys are 64-bit hashes: smaller than sorted keys only on - incompressible keys, where a Bloom filter is three times smaller again for the equality it would - answer; where keys have structure, the sorted keys compress below the hashes and stay exact. - Measured at four key shapes, it was never the best answer. A workload of row-level lookups on long - incompressible keys would reopen it. -- **A Bloom filter over column pairs.** A tuple's cardinality is the product of its columns', which - is exactly the regime where filters cost more than they save, and a composite sorted run answers - the same question exactly. +- Hash to rows: sorted runs whose keys are 64-bit hashes. They are smaller than sorted keys only on + incompressible keys, where a Bloom filter is three times smaller again for the equality they would + answer. Where keys have structure, the sorted keys compress below the hashes and stay exact. + Measured on four key shapes, it was never the best answer. A workload of row-level lookups on long + incompressible keys would reopen the question. +- A Bloom filter over column pairs. A tuple's cardinality is the product of its columns', which is + exactly the regime where filters cost more than they save, and a composite sorted run answers the + same question exactly. ## 6. The policy -`VortexWriteOptions.Indexes` is an `IndexPolicy`, **`None` by default**: a file written with the -defaults carries no index. A caller names indexes by column path, nested paths included — -`IndexPolicy.None.Bloom("id").SortedRuns("ts")` — or starts from `IndexPolicy.Auto` +`VortexWriteOptions.Indexes` is an `IndexPolicy`, `None` by default, so a file written with the +defaults carries no index. A caller names indexes by column path, nested paths included +(`IndexPolicy.None.Bloom("id").SortedRuns("ts")`), or starts from `IndexPolicy.Auto` (`Indexes/IndexPolicy.cs`, [indexes.md](../guide/indexes.md)). -**`Auto`** records the dictionary probe wherever it applies, since it is free, and starts a Bloom -filter on every top-level column, which it gives up — whole, with its reason, and always before its -payload is written, so an abandoned index leaves no byte — at the earliest fact that condemns it: +`Auto` records the dictionary probe wherever it applies, since it is free, and starts a Bloom filter +on every top-level column. It gives a filter up, whole and with its reason, at the earliest fact that +condemns it, and always before its payload is written, so an abandoned index leaves no byte behind: -- the column's first block climbs: the zone map already prunes it; +- the column's first block climbs, so the zone map already prunes it - its filters would pass 2 % of the column's raw bytes, checked at every block, or 2 % of its written - bytes, checked at every chunk; -- a full first node holds fewer distinct values than the floor, or its first four blocks hold one - and the same set, so that every filter says "maybe" for every value; -- its first node holds more values than a filter of `max_blocks` can, where a sorted run is the - structure. - -The hash per row is the whole cost of a live filter, which is why the verdicts come early: with them, -`Auto` adds 0.6 % to the write of a million-row table of six columns. Postings and sorted runs cost -a key table per row and are never `Auto`'s. - -**The file budget**, `IndexPolicy.WithBudgetPerMille`, is 100 by default: indexes may add a tenth of -the data's bytes. It is judged before payloads are written, once the data passes 1 MiB, and drops -optional indexes first; an index whose payloads are already written is kept rather than left as -dead bytes. - -**A required index** (`required: true`) is never dropped by the budget to make room. If it cannot -be built — its builder gave it up, or the required indexes alone exceed the budget — + bytes, checked at every chunk +- a full first node holds fewer distinct values than the floor, or its first four blocks hold one and + the same set, so every filter would say "maybe" for every value +- its first node holds more values than a filter of `max_blocks` can, which is where a sorted run is + the right structure + +The hash per row is the whole cost of a live filter, which is why the verdicts come early. With them, +`Auto` adds 0.6 % to the write of a million-row table of six columns. Postings and sorted runs cost a +key table per row and are never chosen by `Auto`. + +The file budget, `IndexPolicy.WithBudgetPerMille`, is 100 by default, meaning indexes may add a tenth +of the data's bytes. It is judged before payloads are written, once the data passes 1 MiB, and drops +optional indexes first. An index whose payloads are already written is kept rather than left as dead +bytes. + +A required index (`required: true`) is never dropped by the budget to make room. If it cannot be +built, because its builder gave it up or the required indexes alone exceed the budget, `CompleteAsync` throws a `VortexException` naming it and the reason before the file's tail is written. The flag is stored in the directory, so an append that reuses the policy keeps it. A -dataset's clustering key is such an index ([13-dataset.md](13-dataset.md) §5.3), which the dataset -writes spared by the budget: kept when it alone is over it, and reported with what it cost, since a -run on a narrow table is as large as the column it orders and no object is big enough to bring it -under a share of the data. +dataset's clustering key is such an index (see [what a compaction +does](13-dataset.md#53-what-a-compaction-does)), and the dataset writes it exempt from the budget: it +is kept even when it alone is over the budget, and reported with what it cost, since a run on a +narrow table is as large as the column it orders and no object is big enough to bring it under a +share of the data. The `WriteReport` names every index built, with its bytes, and every index abandoned, with its reason. ## 7. Building, appending, and indexing after the fact -**Building.** The builders are fed by the writer's pass over each block: a Bloom builder hashes its -rows into an exact set per block and per open node, a key builder keeps its own table of the chunk's -keys and their positions. Filter runs are written between chunks as their nodes close; the merged -key runs, fence pages and upper tree levels after the last chunk, before the zone maps; the -directory last. Nothing is buffered across chunks but the open nodes and the spilled chunk runs, -which go to pooled memory and then to a scratch file. +The builders are fed by the writer's pass over each block. A Bloom builder hashes its rows into an +exact set per block and per open node, and a key builder keeps its own table of the chunk's keys and +their positions. Filter runs are written between chunks as their nodes close, then the merged key +runs, fence pages and upper tree levels after the last chunk, before the zone maps, and the directory +last. Nothing is buffered across chunks except the open nodes and the spilled chunk runs, which go to +pooled memory and then to a scratch file. -**Appending** ([11-write-strategy.md](11-write-strategy.md) §3.8). Runs that end before the -append's first block are kept and listed again beside the new ones; a run reaching into a re-opened -chunk is dropped and rebuilt; past K runs an entry's runs are merged. +When appending (see [appending to a file](11-write-strategy.md#38-appending-to-a-file)), runs that end +before the append's first block are kept and listed again next to the new ones, a run reaching into a +reopened chunk is dropped and rebuilt, and once an entry has more than four runs they are merged. -**Indexing a file that exists**, without touching a data byte: +A file that already exists can be indexed without touching a data byte: -- **In place**, on a file system: `VortexFileIndexer.AppendIndexesAsync(path, policy)` reads the - file, builds the runs, and appends them with a new directory, footer and postscript over the same - data. The old file is a byte prefix of the new one. -- **Outside the file**, on a store that cannot append: `VortexFileIndexer.BuildFragmentAsync(file, +- In place, on a file system: `VortexFileIndexer.AppendIndexesAsync(path, policy)` reads the file, + builds the runs, and appends them with a new directory, footer and postscript over the same data. + The old file is a byte prefix of the new one. +- Outside the file, on a store that cannot append: `VortexFileIndexer.BuildFragmentAsync(file, policy, rows?)` returns an `IndexFragment`, a container of runs and a directory over a range of - blocks, which a caller keeps where it likes and hands back at open through - `VortexOpenOptions.IndexFragments`. A fragment is **bound to the file's identity**, or to the - store's token for a file written without one, from bytes the open reads anyway; a content hash is - recorded, and checked only offline by `VortexFile.VerifyIndexesAsync` (`vxdump --fragment P - --verify`). Fragments add entries beside the file's own; an entry of the same identity joins its - runs when their blocks are disjoint. In a dataset, fragments live in commit objects - ([13-dataset.md](13-dataset.md) §6.4). - -**A key source needs complete coverage.** A pruner uses whatever blocks the runs cover and leaves -the others live; a cursor over partial runs would miss keys, so a key source refuses an entry whose + blocks, which a caller keeps where it likes and passes back at open through + `VortexOpenOptions.IndexFragments`. A fragment is bound to the file's identity, or to the store's + token for a file written without one, using bytes the open reads anyway. A content hash is also + recorded, and only checked offline by `VortexFile.VerifyIndexesAsync` (`vxdump --fragment P + --verify`). Fragments add entries next to the file's own, and an entry with the same identity joins + its runs when their blocks are disjoint. In a dataset, fragments live in commit objects (see + [fragments](13-dataset.md#64-fragments-in-commit-objects-transient)). + +A key source needs complete coverage. A pruner uses whatever blocks the runs cover and leaves the +others live, but a cursor over partial runs would miss keys, so a key source refuses an entry whose runs do not cover every block, and names the blocks it has. ## 8. Compatibility, and the road upstream | reader | a file with an index directory | |---|---| -| Rust 0.86.1, strict or permissive session | opens, scans, prunes with the zone maps; never fetches the entry | -| an earlier Vorticity | the same: the key is listed, never fetched | -| this Vorticity | uses the index; ignores an entry that fails a check (§3) | - -The file stays a pure core-edition file: every array in it, index runs included, carries an id of -the target edition, and the kinds are versioned strings inside the directory, which no edition -needs to know. When upstream freezes a Bloom aggregate in an edition, a file targeting it may carry -filters in the zone map as well, under upstream's id and bytes; when `vortex.indexed` freezes, a -locating index becomes its auxiliary child, the runs the same arrays and the directory entry its +| Rust 0.86.1, strict or permissive session | opens, scans and prunes with the zone maps, and never fetches the entry | +| an earlier Vorticity | the same: the key is listed but never fetched | +| this Vorticity | uses the index, and ignores an entry that fails a check ([the directory and the runs](#3-the-directory-and-the-runs)) | + +The file stays a pure core-edition file. Every array in it, index runs included, carries an id of the +target edition, and the kinds are versioned strings inside the directory, which no edition needs to +know. When upstream freezes a Bloom aggregate in an edition, a file targeting it may also carry +filters in the zone map, under upstream's id and bytes. When `vortex.indexed` freezes, a locating +index becomes its auxiliary child, with the runs as the same arrays and the directory entry as its metadata. Neither migration changes a payload byte. ## 9. How it is tested -- **Rust reads it**: the cross-check writes the whole corpus under rotating policies — every kind, - filters at three resolutions, runs cut into small segments, appends, indexing after the fact, a - dataset's compacted objects — and a strict Rust reader compares every value. -- **Equivalence**: every supported predicate with indexes on and off, on a file written once and on - the same rows appended in two, three and eleven pieces, returns the same rows. -- **Lying indexes**: a zeroed filter with its framing kept is caught by its checksum and drops no - row; torn fence pages, tree nodes and regions claim nothing (`LyingIndexTests`, `BloomTreeTests`, +- Rust reads it. The cross-check writes the whole corpus under rotating policies (every kind, filters + at three resolutions, runs cut into small segments, appends, indexing after the fact, a dataset's + compacted objects), and a strict Rust reader compares every value. +- Equivalence: every supported predicate, with indexes on and off, on a file written once and on the + same rows appended in two, three and eleven pieces, returns the same rows. +- Lying indexes: a zeroed filter with its framing kept is caught by its checksum and drops no row, + and torn fence pages, tree nodes and regions claim nothing (`LyingIndexTests`, `BloomTreeTests`, `FenceTreeTests`). -- **Bit-exactness**: 66 filters against Rust's `BloomPartial`, 26 against Parquet's `Sbbf`. -- **Allocation**: a ceiling per kind on the write (`IndexAllocationTests`). +- Bit-exactness: 66 filters against Rust's `BloomPartial`, and 26 against Parquet's `Sbbf`. +- Allocation: a ceiling per kind on the write (`IndexAllocationTests`). -The upstream designs this follows: the Bloom aggregate (vortex PR #9398, marked unstable in #9753), -the skip-index interface (#9413), the skipping and locating index epics (#8900, #8948, #9024), and -multi-resolution zone maps (#7939); Parquet's `BloomFilter.md` for the filter; O'Neil et al., *The -log-structured merge-tree*, 1996, for the runs. +The upstream designs this follows are the Bloom aggregate (vortex PR #9398, marked unstable in +#9753), the skip-index interface (#9413), the skipping and locating index epics (#8900, #8948, +#9024), and multi-resolution zone maps (#7939). Parquet's `BloomFilter.md` describes the filter, and +O'Neil et al., *The log-structured merge-tree*, 1996, the runs. diff --git a/docs/design/11-write-strategy.md b/docs/design/11-write-strategy.md index 4f2472e5..10d51ba3 100644 --- a/docs/design/11-write-strategy.md +++ b/docs/design/11-write-strategy.md @@ -1,284 +1,301 @@ # The writer: one pass per block -How a file is written: rows arrive in batches of any size, and each value is summarized once, -encoded once, and written once, with every pruning structure exact and every index built by the same -pass. Four constraints shape it, in this order: **throughput on one thread**, with the rest of the -session's threads used where the work splits cleanly; **pruning structures that are exact**, never -sampled; **indexes as outputs of the same pass**, across appends too -([10-indexes.md](10-indexes.md)); **no allocation per row, and no chunk-sized scratch per column**. +How a file is written. Rows arrive in batches of any size, and each value is summarized once, encoded +once and written once, with every pruning structure exact and every index built by the same pass. +Four constraints shape the writer, in this order: + +1. throughput on one thread, with the rest of the session's threads used where the work splits + cleanly +2. pruning structures that are exact, never sampled +3. indexes produced by the same pass, across appends too ([10-indexes.md](10-indexes.md)) +4. no allocation per row, and no chunk-sized scratch buffer per column ## 1. The principle -A value is **read twice and copied never** between its arrival and its emission. The statistics -pass reads it on the batch where the caller put it; the encoder of the scheme that won reads the -same rows again, from cache, while the chunk is still warm. One pass for everything is not possible -without guessing: every wire form but the dictionary carries a parameter known only once the chunk -is whole — the bit width, the frame of reference, ALP's exponents — and a guess from the previous -chunk would have to be checked and, when wrong, redone. The dictionary is the exception: it assigns -its codes in the first pass, and its encoder never reads the values again. - -Everything that must be exact — minimum, maximum, null count, the Bloom insertions, the keys behind -an index — is computed in the first pass over every row. Everything the chooser needs — runs, -order, progressions, bit-width histograms, distinct counts, byte lengths — is computed in the same -pass. The chooser then decides by **exact formulas** for every scheme whose cost is a function of -those statistics, and runs a **trial** only for the four whose cost is not (ALP, ALP-RD, FSST and -zstd), with the best cost in hand as its abort bound. The winner encodes into outputs sized exactly, -which become the segment's buffers. +Between its arrival and its emission, a value is read twice and never copied. The statistics pass +reads it in the batch where the caller put it, and the encoder of the winning scheme reads the same +rows again, from the CPU cache, while the chunk is still warm. A single pass for everything is not +possible without guessing: every wire form except the dictionary carries a parameter that is only +known once the chunk is complete (the bit width, the frame of reference, ALP's exponents), and a +guess from the previous chunk would have to be checked and, when wrong, redone. The dictionary is +the exception, because it assigns its codes in the first pass and its encoder never reads the values +again. + +Everything that must be exact (minimum, maximum, null count, the Bloom insertions, the keys behind an +index) is computed in the first pass over every row. Everything the chooser needs (runs, order, +progressions, bit-width histograms, distinct counts, byte lengths) is computed in the same pass. The +chooser then decides by exact formulas for every scheme whose cost is a function of those +statistics, and only runs a trial for the schemes whose cost is not: ALP, ALP-RD, FSST, OnPair and +zstd, plus pco under the `Smallest` profile, each with the best cost so far as its abort bound. The +winner encodes into outputs sized exactly, which become the segment's buffers. ## 2. Blocks and chunks -A **block** is 8 192 rows counted from row 0 of the file, whatever the caller's batching +A block is 8 192 rows counted from row 0 of the file, whatever the caller's batching (`VortexWriteOptions.BlockRows`). It is the zone of the zone map, upstream's row block, eight -FastLanes blocks of 1 024, and the unit of an index. A batch that straddles blocks produces -**partials** per column and block that merge exactly, so a block's statistics are final when its -last row arrives, whichever batch brings it. +FastLanes blocks of 1 024, and the unit of an index. A batch that straddles blocks produces partial +summaries per column and block that merge exactly, so a block's statistics are final when its last +row arrives, whichever batch brings it. -A **chunk** is a run of whole blocks, written as one segment per column. It stays a multiple of the +A chunk is a run of whole blocks, written as one segment per column. It stays a multiple of the block length, because a chunk edge inside a block would put a zone edge inside a segment, where a -pruned zone saves no I/O. Its size weighs four costs: a read pays a fixed cost per chunk, which stops -mattering between 65 536 and 131 072 rows of an eight-byte column; a dictionary pays only when a -chunk holds many more rows than distinct values; a filtered or indexed read fetches the whole chunk -of every column it keeps; and an encoding that rides on local ranges loses when a chunk spans -several. So by default a chunk holds as many whole blocks as keep its **widest column within a -megabyte** of values, between one block and 128, with all its columns within 64 MiB. -`ChunkTargetBytes` fixes a byte target instead, and `ColumnChunkTargetBytes` gives one column -chunks of its own, spanning whole chunks of the file: a dictionary paid once for more rows, while -the other columns keep what a selective read fetches (§8). - -A batch that is a multiple of the block length and fills a chunk is encoded where it lies; any other +pruned zone saves no I/O. Its size balances four costs. A read pays a fixed cost per chunk, which +stops mattering somewhere between 65 536 and 131 072 rows of an eight-byte column. A dictionary only +pays off when a chunk holds many more rows than distinct values. A filtered or indexed read fetches +the whole chunk of every column it keeps. And an encoding that relies on local ranges loses when a +chunk spans several. So by default a chunk holds as many whole blocks as keep its widest column +within a megabyte of values, between one block and 128, with all its columns within 64 MiB. +`ChunkTargetBytes` sets a byte target instead, and `ColumnChunkTargetBytes` gives one column chunks +of its own, spanning whole chunks of the file, which pays for a dictionary once over more rows while +the other columns keep what a selective read fetches (see [the advisor](#8-choosing-encodings-ahead-the-advisor)). + +A batch that is a multiple of the block length and fills a chunk is encoded where it lies. Any other batch waits in the builder's buffers for the rows that complete its blocks. `FlushAsync` seals the -whole blocks into a chunk; `CompleteAsync` writes the rest as the tail. A caller who writes in -multiples of `BlockRows` gets a file whose every chunk ends on a block, which appends without -rewriting anything (§3.8). +whole blocks into a chunk, and `CompleteAsync` writes the rest as the tail. A caller who writes in +multiples of `BlockRows` gets a file in which every chunk ends on a block, which can be appended to +without rewriting anything (see [appending to a file](#38-appending-to-a-file)). ## 3. The pipeline, stage by stage ### 3.1 One column writer per leaf -The writer keeps the segment list, a small shared scratch, and a tree of column writers mirroring -the schema, one per physical leaf: a struct field, a list's elements, an extension's storage. A -column writer holds the open block's partials, one 56-byte summary per closed block until the zone -map is written, its plan memory (§3.4), its index builders, and while it runs the chunk's distinct -table. A nested column's blocks are defined by its parent's rows: block `i` of a list's elements -covers the elements of the parent's rows of block `i`, so a filter over the elements answers "does -any element of a row in this block match". +The writer keeps the segment list, a small shared scratch area, and a tree of column writers that +mirrors the schema, one per physical leaf: a struct field, a list's elements, an extension's storage. +A column writer holds the open block's partial summaries, one 56-byte summary per closed block until +the zone map is written, its plan memory (see [choosing](#34-choose-exact-verdicts-then-bounded-trials)), +its index builders, and, while it runs, the chunk's distinct table. A nested column's blocks are +defined by its parent's rows. Block `i` of a list's elements covers the elements of the parent's rows +in block `i`, so a filter over the elements answers "does any element of a row in this block match". ### 3.2 The statistics pass -Per column and block, one loop, resolved on the physical type before it starts and walking the -validity as bitmap words, never a row at a time. It keeps: +Per column and block there is one loop, resolved on the physical type before it starts, that walks +the validity as bitmap words, never a row at a time. It keeps: | what | exact | for | |---|---|---| -| minimum and maximum; NaN skipped, `-0.0` and `+0.0` told apart by their bits | yes | the zone map, frame of reference, progressions, constants | +| minimum and maximum, NaN skipped, `-0.0` and `+0.0` told apart by their bits | yes | the zone map, frame of reference, progressions, constants | | null count | yes | the zone map, all-null chunks, every cost formula | | whether the values ascend, strictly or not | yes | the file statistics, progressions, run-end | | a progression's step, and whether it broke | yes | `vortex.sequence` | | the count of runs | yes | run-end | | bit-width histograms, raw and zigzag | yes | bit-packing with patches, kept only where such a packing is held | | distinct values, through the distinct table while it runs | yes | the dictionary | -| text and binary byte totals; bounded minimum and maximum | yes, the bounds as bounds | varbin against views, the trials' ceilings, the zones of a text column | +| text and binary byte totals, bounded minimum and maximum | yes, the bounds as bounds | varbin against views, the trials' ceilings, the zones of a text column | -It runs on the batch's own memory, while the decode that produced a rewrite's batch is still in -cache, with no copy before it. +It runs on the batch's own memory, with no copy before it, and when the batch comes from decoding a +file being rewritten, while that decode is still in the CPU cache. ### 3.3 The distinct table -One open-addressing table per column, **running over a chunk**, maps a value to its code, assigned -in first-seen order, and writes each row's code as it arrives. Equality is on bytes, never on the -hash alone, and the table **owns** its keys: a block straddles batches, and the previous batch's -memory is recycled the moment the next one is decoded, so a string is copied once, the first time it -is seen, into a heap that is, in code order, the dictionary's values child. +There is one open-addressing table per column, running over a chunk, that maps a value to its code, +assigned in first-seen order, and writes each row's code as it arrives. Equality is checked on bytes, +never on the hash alone, and the table owns its keys. A block straddles batches, and the previous +batch's memory is recycled the moment the next one is decoded, so a string is copied once, the first +time it is seen, into a heap that is, in code order, the dictionary's values child. -Probing every row of a column whose values are all distinct costs a few nanoseconds a row for -nothing, so the table runs **only on a chunk whose remembered plan is a dictionary that held**. A -column's first chunk, and a chunk whose memory sends it back to full pricing, count their distinct -values in the chooser's own walk, which gives up early on a column where a dictionary loses; the -table then runs for the next chunk if the dictionary won. +Probing every row of a column whose values are all distinct costs a few nanoseconds per row for +nothing, so the table only runs on a chunk whose remembered plan is a dictionary that held. A +column's first chunk, and a chunk whose plan memory sends it back to full pricing, count their +distinct values in the chooser's own walk, which gives up early on a column where a dictionary loses. +The table then runs for the next chunk if the dictionary won. ### 3.4 Choose: exact verdicts, then bounded trials -In order, per column chunk: - -1. **Degenerate cases**, from the statistics alone: a chunk of one value, or of nulls only, of any - dtype -- a struct's and a list's included, whose validity would otherwise be a bitmap of zeros - -- as `vortex.constant`, a scalar and the length; an integer progression as `vortex.sequence`, - about thirty bytes whatever the rows. The constant comes first, integers included: a sequence - of step zero decodes to its rows, where a constant stays one value through a scan. -2. **Nulls on nine rows in ten or more**, as `vortex.sparse` with a null fill: the valid rows' - positions and values, and no validity, priced as a frame of reference packs them against the - runs the nulls make. It comes before run-end's rule below, which would take every such chunk as - runs, and before a decimal's parts. A **decimal** whose values fit 64 bits is then written as - `vortex.decimal_byte_parts` over the integers, which the integer schemes price in turn. -3. **Exact candidates**, each priced by a formula that is what its encoder will write: the plain +Per column chunk, in order: + +1. Degenerate cases are decided from the statistics alone. A chunk of a single value, or of nulls + only, of any dtype (a struct's and a list's included, whose validity would otherwise be a bitmap + of zeros) is written as `vortex.constant`, a scalar and a length. An integer progression is + written as `vortex.sequence`, about thirty bytes whatever the row count. The constant comes first, + integers included, because a sequence of step zero decodes to all its rows, whereas a constant + stays a single value through a scan. +2. A chunk with nulls on nine rows in ten or more is written as `vortex.sparse` with a null fill: the + valid rows' positions and values, and no validity, priced the way a frame of reference packs them + against the runs the nulls make. It comes before run-end's rule below, which would otherwise take + every such chunk as runs, and before a decimal's parts. A decimal whose values fit in 64 bits is + then written as `vortex.decimal_byte_parts` over the integers, which the integer schemes price in + turn. +3. Exact candidates are each priced by a formula equal to what their encoder will write: the plain form (for text, `vortex.varbin` or views, whichever is smaller), run-end (which wins outright - inside a quarter as many runs as rows), bit-packing in the raw, zigzag or frame-of-reference - domain with the cheapest width and its patches, a **timestamp's** instants split into days, - seconds and subseconds (`vortex.datetimeparts`), priced by each part's packed width and taken - when it comes a tenth under the best so far, and the dictionary, priced by its own layer at the - codes' width plus its values child. A 256-row sample bounds the timestamp split from below, so - instants at full resolution are refused without the pass that splits them. -4. **Trials**, only when no exact verdict won, each carrying its encoded result so that a winner is - never encoded twice: for floats, ALP, then ALP-RD against a zstd frame; for text, a zstd frame - priced first, then FSST, which must save a tenth on the plain form and not lose to the frame by - more than a tenth, then OnPair, which must also come a tenth under FSST: a dictionary of up to - 4 096 tokens trained on a random sixth of the chunk's bytes, whose own codes on that sample - decide before the pass over every row. Zstd is tried only on chunks of at least 16 KiB. - -`CompressionProfile` changes the arithmetic, not the pass: `Auto` weighs bytes and decode speed as -above; `Smallest` prices every candidate by its bytes alone and tries every trial under the bytes -the best exact plan writes, pco among them for numbers of 16 bits and more, on the column and on -every array a scheme makes of it, as the reference's compact compressor cascades; `Fastest` writes -the cheapest encodings and no index; `None` writes every column plain. - -**pco under `Smallest`.** A numeric chunk is compressed as pco does it at its default level: the -latents in chunks of up to 2^18 and pages of 8 192 as Vortex's compact compressor pages them, an -integer's common divisor found by pco's own test on pco's own sample (`IntMult`), the consecutive -delta order a sample prices best, then equal-count bins merged by pco's bit-cost dynamic program -and entropy-coded by its four-way tANS. What upstream's automatic choice also tries and this one -does not is the lookback delta and the float modes, whose values ALP and ALP-RD already take. The -bytes match upstream's own within the framing on every shape measured -- event times, a random -walk, multiples of a hundred, skewed counts, doubles in full -- and the plan is priced with the -framing its pages bring, a buffer and a count each, since a column that compresses to little has -as much there as in the pages. Before the pass that bins and writes a chunk, the sample that chose -its delta prices it: a chunk a quarter over the bytes it must beat is refused unwritten. The trial -costs what pco costs, a partition of the latents into bins rather than a sort, and a count in their -place where their range is narrower than their number, which is what deltas leave. - -**Plan memory.** A column keeps its last plan, the bytes the formulas predicted for it and the bytes -its encoder produced. The next chunk re-prices only the remembered plan, from its own statistics, -as long as the last chunk's bytes stayed within 5 % of the prediction; otherwise it prices -everything again. A progression and a run count are checked before memory, since they are -arithmetic on the summary: a plan that held never costs a progression its 32 bytes. What memory + below a quarter as many runs as rows), bit-packing in the raw, zigzag or frame-of-reference domain + at the cheapest width with its patches, a timestamp's instants split into days, seconds and + subseconds (`vortex.datetimeparts`), priced by each part's packed width and taken when it comes a + tenth under the best so far, and the dictionary, priced by its own layer at the codes' width plus + its values child. A 256-row sample bounds the timestamp split from below, so instants at full + resolution are refused without the pass that splits them. +4. Trials only run when no exact verdict won, and each carries its encoded result so a winner is + never encoded twice. For floats, ALP is tried, then ALP-RD against a zstd frame. For text, a zstd + frame is priced first, then FSST, which must save a tenth on the plain form and not lose to the + frame by more than a tenth, then OnPair, which must also come a tenth under FSST. OnPair trains a + dictionary of up to 4 096 tokens on a random sixth of the chunk's bytes, and its codes on that + sample decide before the pass over every row. Zstd is only tried on chunks of at least 16 KiB. + +`CompressionProfile` changes the arithmetic, not the pass. `Auto` weighs bytes and decode speed as +above. `Smallest` prices every candidate by its bytes alone and tries every trial under the bytes the +best exact plan writes, pco among them for numbers of 16 bits and more, on the column and on every +array a scheme makes of it, the way the reference's compact compressor cascades. `Fastest` writes the +cheapest encodings and no index, and `None` writes every column plain. + +Under `Smallest`, a numeric chunk is compressed the way pco does it at its default level. The latents +are split into chunks of up to 2^18 and pages of 8 192, as Vortex's compact compressor pages them, an +integer's common divisor is found by pco's own test on pco's own sample (`IntMult`), the consecutive +delta order is the one a sample prices best, and equal-count bins are merged by pco's bit-cost +dynamic program and entropy-coded by its four-way tANS. What upstream's automatic choice also tries +and this one does not is the lookback delta and the float modes, whose values ALP and ALP-RD already +handle. The bytes match upstream's own, framing aside, on every shape measured (event times, a random +walk, multiples of a hundred, skewed counts, full doubles), and the plan is priced with the framing +its pages bring, a buffer and a count each, since a column that compresses to little has as much +there as in the pages. Before the pass that bins and writes a chunk, the sample that chose its delta +prices it, and a chunk a quarter over the bytes it must beat is refused unwritten. The trial costs +what pco costs: a partition of the latents into bins rather than a sort, and a count in their place +where their range is narrower than their number, which is what deltas leave. + +A column also keeps a plan memory: its last plan, the bytes the formulas predicted for it and the +bytes its encoder produced. As long as the last chunk's bytes stayed within 5 % of the prediction, +the next chunk only prices the remembered plan again, from its own statistics. Otherwise it prices +everything again. A progression and a run count are checked before the memory, since they are +arithmetic on the summary, so a plan that held never costs a progression its 32 bytes. What memory saves is the walks and the trials, never a candidate the statistics already decide. An FSST symbol -table is still trained per chunk (sharing one costs 4.3 % of the output); what is remembered is -the decision. - -**Hints.** `VortexWriteOptions.Hints` pins a column's scheme by path (`EncodingHint`: `Canonical`, -`RunEnd`, `Dictionary`, `BitPacked`, `Fsst`, `Alp`, `AlpRd`, `Sequence`, `Zstd`, `DecimalByteParts`, -`Constant`, `DateTimeParts`, `Sparse`, `OnPair`, `Pco`): plan memory with an infinite tolerance, priced on every -chunk and written when it applies, under every profile; only a constant and a progression, which -cost nothing per row, are written before it. `Sequence` and `Constant` pin nothing the writer would -not do anyway; they name what a report says a chunk was written as. A chunk the scheme cannot describe -is priced in full, and the next chunk is offered the hint again. A hint on a path the schema does -not have throws at `CreateWriter`. - -**The cascade has context.** A child never enters the chooser blind: a dictionary's codes are dense -in `[0, entries)` and packed at their width; run-end ends are strictly increasing; offsets and sizes -are monotone or bounded, and a list of fixed-width rows has offsets that are an exact progression -and sizes that are constant, so both cost nothing. A dictionary's or a run's values are a column -like any other, chosen in full on a small input. A timestamp's storage knows its unit, which is -what lets it price its split; a part of one value -- the subseconds of instants to the second -- -is handed on as a constant rather than discovered by a walk. +table is still trained per chunk (sharing one costs 4.3 % more output), and only the decision is +remembered. + +`VortexWriteOptions.Hints` pins a column's scheme by path (`EncodingHint`: `Canonical`, `RunEnd`, +`Dictionary`, `BitPacked`, `Fsst`, `Alp`, `AlpRd`, `Sequence`, `Zstd`, `DecimalByteParts`, +`Constant`, `DateTimeParts`, `Sparse`, `OnPair`, `Pco`). A hint acts as plan memory with an infinite +tolerance, priced on every chunk and written whenever it applies, under every profile. Only a +constant and a progression, which cost nothing per row, are written before it. `Sequence` and +`Constant` pin nothing the writer would not do anyway, and simply name what a report says a chunk was +written as. A chunk the scheme cannot describe is priced in full, and the next chunk is offered the +hint again. A hint on a path the schema does not have throws at `CreateWriter`. + +The cascade has context. A child never enters the chooser blind: a dictionary's codes are dense in +`[0, entries)` and packed at their width, run-end ends are strictly increasing, offsets and sizes are +monotone or bounded, and a list of fixed-width rows has offsets that form an exact progression and +sizes that are constant, so both cost nothing. A dictionary's or a run's values are a column like any +other, chosen in full on a small input. A timestamp's storage knows its unit, which is what lets it +price its split, and a part that is the same for every value (the subseconds of instants rounded to +the second) is handed on as a constant rather than discovered by a walk. ### 3.5 Encode once -The winner reads the chunk's rows block by block, from the batch or the retained batches, and -writes outputs sized exactly: patches found inside the pack, frame of reference and zigzag applied -on load, a dictionary's codes packed from the codes the first pass wrote, ALP's exceptions as -patches. A value is ALP-encoded only when decoding the integer gives back its **exact bit pattern**, -compared by bits and never with `==`, since `-0.0 == 0.0` and NaN payloads are invisible to `==`. -An FSST symbol table is ordered by symbol length, two to eight bytes then one, as the reference -requires. +The winner reads the chunk's rows block by block, from the batch or the retained batches, and writes +outputs sized exactly: patches found inside the pack, frame of reference and zigzag applied on load, +a dictionary's codes packed from the codes the first pass wrote, and ALP's exceptions as patches. A +value is only ALP-encoded when decoding the integer gives back its exact bit pattern, compared by +bits and never with `==`, since `-0.0 == 0.0` and NaN payloads are invisible to `==`. An FSST symbol +table is ordered by symbol length, two to eight bytes then one, as the reference requires. -A zstd column is compressed as **one frame per block**, so a take inflates only the frames that hold -the rows it wants, and the frames of a large column are compressed on the writer's threads (§3.7): -each into a room of its own, closed up in order, so the bytes are the same on any number of -threads. +A zstd column is compressed as one frame per block, so a take only inflates the frames that hold the +rows it wants. The frames of a large column are compressed on the writer's threads (see +[threads](#37-memory-allocation-and-threads)), each into its own buffer and then closed up in order, +so the bytes are the same on any number of threads. ### 3.6 Emit -A chunk goes out when enough whole blocks are pending, and a Bloom filter's runs as their tree nodes -close, between chunks. When the file completes: the key runs, fence pages and upper filter levels; -then the zone maps, from the closed blocks' summaries; the file statistics; the index directory; -the file's identity ([13-dataset.md](13-dataset.md) §7); the footer and the postscript. +A chunk goes out when enough whole blocks are pending, and a Bloom filter's runs go out as their tree +nodes close, between chunks. When the file completes, the writer emits the key runs, fence pages and +upper filter levels, then the zone maps from the closed blocks' summaries, the file statistics, the +index directory, the file's identity (see [identity and +integrity](13-dataset.md#7-identity-and-integrity)), and finally the footer and the postscript. -**The file statistics** are one `ArrayStats` per top-level column: the exact minimum and maximum of -a numeric column, the null count, and whether the column ascends, in the reference's semantics — a -null below every value, equal neighbours allowed by `is_sorted` and refused by `is_strict_sorted`, a -NaN claiming nothing. The Rust cross-check holds every one against its own recomputation. A text -column's zones carry **bounded** minimum and maximum, 16 bytes by default (`StringBoundBytes`), so -that a text filter prunes. +The file statistics are one `ArrayStats` per top-level column: the exact minimum and maximum of a +numeric column, the null count, and whether the column ascends, in the reference's semantics (a null +below every value, equal neighbours allowed by `is_sorted` and refused by `is_strict_sorted`, a NaN +claiming nothing). They carry no sum. The Rust cross-check compares every one with its own +recomputation. A text column's zones carry bounded minimum and maximum values, 16 bytes by default +(`StringBoundBytes`), so that a text filter can prune. -`CompleteAsync` returns a **`WriteReport`**: the file's bytes by kind, the chunk sizes, per column -the scheme of every chunk and how often plan memory held, and per index what was built, its bytes, -or why it was abandoned. +`CompleteAsync` returns a `WriteReport`: the file's bytes by kind, the chunk sizes, per column the +scheme of every chunk and how often plan memory held, and per index what was built and its bytes, or +why it was abandoned. ### 3.7 Memory, allocation and threads | what | owner | size | lifetime | |---|---|---|---| -| the open block's partials and 56 bytes per closed block | the column | small | the file | -| the distinct table, its key heap and its codes | the column | at most a small multiple of the column's share of the pending chunk | the chunk; only while it runs | +| the open block's partial summaries, and 56 bytes per closed block | the column | small | the file | +| the distinct table, its key heap and its codes | the column | at most a small multiple of the column's share of the pending chunk | the chunk, and only while it runs | | index builders and spilled chunk runs | the column | pooled pages, then a scratch file | the file | | encoder outputs | the writer's arena | the segment's exact size | until the segment is written | -| per row | — | **nothing** | — | +| per row | none | nothing | none | -A schema of a thousand columns costs a thousand small states, 1 892 bytes each, not a thousand -scratches ([05-benchmarks.md](05-benchmarks.md) §5). +A schema of a thousand columns costs a thousand small states of 1 892 bytes each, not a thousand +scratch buffers ([allocations](05-benchmarks.md#5-allocations)). -**Threads.** At a degree above one (`VortexWriteOptions.DegreeOfParallelism`, or the session's), a -batch of a block or more is summarized column by column side by side: each column's statistics, and -apart from them a text column's bounds and the distinct values a dictionary is priced from, each -into state of its own. A column's zstd frames are compressed side by side too. Choosing and writing -the encodings stays on the calling thread, and the file is the same bytes at any degree. A write -that builds indexes summarizes on one thread. +At a degree above one (`VortexWriteOptions.DegreeOfParallelism`, or the session's), a batch of a +block or more is summarized column by column, side by side: each column's statistics, and separately +a text column's bounds and the distinct values a dictionary is priced from, each into its own state. +A column's zstd frames are compressed side by side too. Choosing and writing the encodings stays on +the calling thread, and the file is the same bytes at any degree. A write that builds indexes +summarizes on one thread. ### 3.8 Appending to a file -`VortexSession.AppendAsync(path)` returns a writer that continues a file this writer's shape can -extend — a struct of columns, each a chunked layout of flat segments, chunked alike — and refuses any -other with the advice to rewrite it. The old segments stay where they are; new blocks are numbered from the old -row count, so their boundaries are the ones a single write would have made; new segments, runs, -zone maps, statistics, directory, footer and postscript are appended, and the old postscript -becomes dead bytes. Three rules make it exact: - -- **A last chunk that is not whole blocks is re-opened.** A zone map allows only its last zone to be - short, and the last block's filter and runs were built on a short block. So the last chunk is - read, dropped from the layout, and written again as the append's first rows, with the runs that - covered it: one chunk, whatever the file's size. Every block before it becomes a summary taken - from the old zone map, so the zone map is written again over the whole file without reading them. -- **Plan memory is seeded from the last chunk's encodings**, one segment read per column, so the - appended rows keep the file's encodings unless their statistics say otherwise. -- **The index policy and budget are read from the directory**, so an append needs no options; the - old runs are kept and merged past K ([10-indexes.md](10-indexes.md) §7). +`VortexSession.AppendAsync(path)` returns a writer that continues a file whose shape this writer can +extend (a struct of columns, each a chunked layout of flat segments, all chunked alike), and refuses +any other file with the advice to rewrite it. The old segments stay where they are. New blocks are +numbered from the old row count, so their boundaries are the ones a single write would have made. +New segments, runs, zone maps, statistics, directory, footer and postscript are appended, and the old +postscript becomes dead bytes. Three rules make it exact: + +- A last chunk that is not made of whole blocks is reopened. A zone map only allows its last zone to + be short, and the last block's filter and runs were built on a short block. So the last chunk is + read, dropped from the layout, and written again as the append's first rows, together with the runs + that covered it. That is one chunk, whatever the file's size. Every block before it becomes a + summary taken from the old zone map, so the zone map is written again over the whole file without + reading those blocks. +- Plan memory is seeded from the last chunk's encodings, at the cost of one segment read per column, + so the appended rows keep the file's encodings unless their statistics say otherwise. +- The index policy and budget are read from the directory, so an append needs no options. The old + runs are kept, and merged once there are more than four (see [indexing after the + fact](10-indexes.md#7-building-appending-and-indexing-after-the-fact)). An append is not atomic. It never truncates while it writes, so a torn append leaves the previous -version intact before garbage: the open finds it by walking back to the last end-of-file record, -reports the tear in `VortexFile.TornTail`, and reads the previous version, unless -`VortexOpenOptions.TornTail` says `Refuse`; `VortexFileRepair.RepairAsync` truncates the file to -it. An append, and an indexing pass, refuse a torn file until it is repaired. `Abandon()` on an +version intact, followed by garbage. The open finds that version by walking back to the last +end-of-file record, reports the tear in `VortexFile.TornTail`, and reads the previous version, unless +`VortexOpenOptions.TornTail` says `Refuse`. `VortexFileRepair.RepairAsync` truncates the file back +to it. An append, and an indexing pass, refuse a torn file until it is repaired. `Abandon()` on an append truncates the file back to the length it had. ## 4. Kernels -`Vector128` everywhere, wider vectors where the hardware has them, and a scalar twin whose tests run -again with intrinsics disabled ([03-architecture.md](03-architecture.md) §1). In the statistics pass: -minimum and maximum per lane, with validity folded in as a mask; null counts by population count -per 64-bit word; run boundaries by comparing a vector with itself shifted by one; order and -progressions by lane-wise subtraction against a broadcast step, exact because the progression's -endpoint is checked to fit the type first; bit widths by lane leading-zero counts where the -instruction set has them; string runs by one 16-byte comparison per pair of views. In the encoders: -frame of reference and zigzag applied on load, patch detection by a comparison mask per 1 024-block, -the FastLanes pack, and ALP's multiply, round and convert with the exception mask. - -Some kernels were priced before being written and were not, because the saving was not there: an -eight-lane Bloom probe saves 0.2 µs of a 7 ms filtered scan, since a scan probes about 140 times per -literal; a vectorized zone-map pass has no register's worth of work in front of the decode that -feeds it; and a Swiss-table group probe would save at most five per cent of a dictionary's write. -The one that was written is the merge-join of an `IN` of many keys against a sorted run's keys, -which took a thousand-key `IN` from 117 to 51 ms. +`Vector128` is used everywhere, wider vectors where the hardware has them, plus a scalar twin whose +tests run again with intrinsics disabled (see [the founding +constraints](03-architecture.md#1-founding-constraints)). + +The statistics pass vectorizes: + +- minimum and maximum per lane, with the validity folded in as a mask +- null counts, by population count per 64-bit word +- run boundaries, by comparing a vector with itself shifted by one +- order and progressions, by lane-wise subtraction against a broadcast step, exact because the + progression's endpoint is first checked to fit the type +- bit widths, by lane leading-zero counts where the instruction set has them +- string runs, by one 16-byte comparison per pair of views + +The encoders apply frame of reference and zigzag on load, detect patches with a comparison mask per +1 024-value block, run the FastLanes pack, and do ALP's multiply, round and convert with the +exception mask. + +Some kernels were priced before being written and then not written, because the saving was not +there. An eight-lane Bloom probe would save 0.2 µs of a 7 ms filtered scan, since a scan probes +about 140 times per literal. A vectorized zone-map pass has less than a register's worth of work +ahead of the decode that feeds it. A Swiss-table group probe would save at most five per cent of a +dictionary's write. The one that was written is the merge-join of an `IN` of many keys against a +sorted run's keys, which took a thousand-key `IN` from 117 to 51 ms. ## 5. What it guarantees, and what holds it -1. **Nothing a contract rests on is sampled.** Minimum, maximum, null count, Bloom insertions and - the keys behind an index cover every row. ALP's exponent search and FSST's symbol training - sample, and neither is a statistic anything prunes by. -2. **Equality is on bytes.** A table hit is confirmed on the key's bytes; hashes only route. -3. **An index is whole or absent**, and the report says which ([10-indexes.md](10-indexes.md) §6). -4. **Merges are exact or conservative.** Statistics merge exactly; a filter node is sized from the - exact union of its blocks; a bounded string minimum or maximum is marked as a bound. -5. **Pruning never removes a row a full scan returns** ([08-semantics.md](08-semantics.md) §1). +1. Nothing a contract rests on is sampled. Minimum, maximum, null count, Bloom insertions and the + keys behind an index cover every row. ALP's exponent search and FSST's symbol training do sample, + but neither is a statistic anything prunes by. +2. Equality is on bytes. A table hit is confirmed on the key's bytes, and hashes only route. +3. An index is whole or absent, and the report says which (see [the policy](10-indexes.md#6-the-policy)). +4. Merges are exact or conservative. Statistics merge exactly, a filter node is sized from the exact + union of its blocks, and a bounded string minimum or maximum is marked as a bound. +5. Pruning never removes a row a full scan returns (see [statistic + precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)). | what | how it is held | |---|---| @@ -286,115 +303,116 @@ which took a thousand-key `IN` from 117 to 51 ms. | Rust reads it | the cross-check, appended files and chunked-apart columns among them | | statistics | property tests against a naive per-row computation, at block and batch edges, for every dtype and validity shape | | formulas | the formula's cost equals the encoder's bytes, for every scheme on the corpus | -| plan memory | a column whose data changes shape re-prices within a chunk; a stable column never | -| append | a file written once and the same rows appended in two, three and eleven pieces answer the same, zone maps and indexes included | +| plan memory | a column whose data changes shape is priced again within a chunk, and a stable column never is | +| append | a file written once and the same rows appended in two, three and eleven pieces give the same answers, zone maps and indexes included | | threads | the file is byte for byte the same at every degree | -| allocations and dispatch | ratchets per written file and per column; no per-row dispatch in `Writing/` beyond the calls named in `PerRowDispatchTests` | +| allocations and dispatch | ratchets per written file and per column, and no per-row dispatch in `Writing/` beyond the calls named in `PerRowDispatchTests` | ## 6. Reading what the writer produced The writer's structures are only worth what the scan does with them. Every pruning structure -implements one operation: **refine a mask of live blocks** for the scan's predicate, a bit per block. -The scan builds the mask once per query and runs the structures **cheapest first**: the file -statistics, the zone maps (minimum and maximum, null counts, the bounded string bounds), the Bloom -filters from the root down, the postings and sorted runs, and last the dictionary probe, the one -structure that reads the column's own bytes. It stops as soon as the mask is empty. Then, per split: - -- a split whose blocks are all dead is skipped before its read: the I/O saving; -- a split with dead blocks becomes a **row selection** over its live blocks, which the decoders - serve through `DecodeSelected`, so only live blocks are decoded: the decode saving; +implements one operation: refine a mask of live blocks, one bit per block, for the scan's predicate. +The scan builds the mask once per query and runs the structures cheapest first: the file statistics, +the zone maps (minimum and maximum, null counts, the bounded string bounds), the Bloom filters from +the root down, the postings and sorted runs, and last the dictionary probe, the one structure that +reads the column's own bytes. It stops as soon as the mask is empty. Then, per split: + +- a split whose blocks are all dead is skipped before it is read, which saves the I/O +- a split with some dead blocks becomes a row selection over its live blocks, which the decoders + serve through `DecodeSelected`, so only live blocks are decoded, which saves the decode - an `Exact` index covering the whole predicate turns its rows into that selection, and the - predicate is not evaluated again; under `OR`, `NOT` or a partial cover it only narrows, and the - filter runs on the survivors. + predicate is not evaluated again. Under `OR`, `NOT` or a partial cover, it only narrows, and the + filter runs on the survivors -Two more promises close the contract: +Two more promises complete the contract: -- **One lease per segment per scan.** The live splits register their segments in one request, each - distinct segment is read once in as few requests as coalescing allows, and every block of it +- A scan takes one lease per segment. The live splits register their segments in one request, each + distinct segment is read once in as few requests as coalescing allows, and every block in it decodes from that lease. `ScanStatistics.Requests` is the plan's distinct segments, and - `BytesRequested` its bytes, within the coalescing gap. -- **A selection instead of a compaction.** A filtered batch is compacted to its survivors by - default; with `ScanOptions.Compact = false` each live block is delivered whole with the - `Selection` of the rows that passed, and nothing is copied. Either way `StartRow` is the block's + `BytesRequested` their bytes, within the coalescing gap. +- A selection can replace a compaction. A filtered batch is compacted to its surviving rows by + default. With `ScanOptions.Compact = false`, each live block is delivered whole with the + `Selection` of the rows that passed, and nothing is copied. Either way, `StartRow` is the block's first row in the file. -**Granularity, honestly.** I/O is per segment, and a segment is a chunk of several blocks: a block -pruned inside a live chunk saves decode, not bytes read. That is why chunks stay whole blocks, and -why a file meant for selective reads wants smaller chunks. +The granularity has to be stated honestly. I/O is per segment, and a segment is a chunk of several +blocks, so a block pruned inside a live chunk saves decode, not bytes read. That is why chunks stay +whole blocks, and why a file meant for selective reads wants smaller chunks. The index directory is read lazily, on the first filter that could use it, and usually lies inside -the tail the open already read; the runs are read on the first query that needs them and cached on -the file, shared by every scan. `ExplainAsync` returns the plan without executing it — blocks, blocks -pruned by each structure with what consulting it cost, segments and bytes to read — and the scan's +the tail the open already read. The runs are read on the first query that needs them and cached on +the file, shared by every scan. `ExplainAsync` returns the plan without executing it (blocks, blocks +pruned by each structure with what consulting it cost, segments and bytes to read), and the scan's `Statistics` report the same quantities once it has run, counted by the same sink. ## 7. Determinism, and its limit -The file is a pure function of the **sequence of batches**: no seed, no sample, plan memory a -function of the chunks before. The one exception is the file's sixteen-byte identity, random unless -`VortexWriteOptions.Identity` pins it. +The file is a pure function of the sequence of batches: no seed, no sample, and plan memory is a +function of the chunks before. The one exception is the file's sixteen-byte identity, which is random +unless `VortexWriteOptions.Identity` pins it. -It is **not** a function of the rows alone. Feed the same rows in another batching and the file is -valid, reads back value for value, and is a different size: rewriting an 8 193-row binary column at -forced batch counts costs 65 980 bytes in 2 chunks, 92 212 in 3, 146 076 in 5 and 169 860 in 9, -about 26 kB per extra chunk, because each chunk builds and writes its own dictionary of the same +The file is not a function of the rows alone. Feed the same rows with a different batching and the +file is valid and reads back value for value, but its size changes. Rewriting an 8 193-row binary +column at forced batch counts costs 65 980 bytes in 2 chunks, 92 212 in 3, 146 076 in 5 and 169 860 +in 9, about 26 kB per extra chunk, because each chunk builds and writes its own dictionary of the same distinct values. The reference shares one values child across a column's chunks through the -`vortex.dict` layout, which this writer does not emit. `BatchingSizeTests` pins those numbers, as a -ratchet on a cost known to be bad. A caller who wants the smallest file writes the largest batches -it can, or gives the column chunks of its own. +`vortex.dict` layout, which this writer does not emit. `BatchingSizeTests` pins those numbers as a +ratchet on a cost known to be bad. A caller who wants the smallest file writes the largest batches it +can, or gives the column chunks of its own. ## 8. Choosing encodings ahead: the advisor The writer optimizes one thing per chunk, bytes, under rules that keep decoding fast. The -measurements of [the encodings, column by column](../guide/benchmarks.md#encodings-column-by-column) -show where that is not what a caller wants: text that does not repeat goes to zstd, which a scan -decodes several times slower than FSST and a take far slower; a column whose values repeat across the -file more than within a chunk cannot take a dictionary, which larger chunks would give it; and which -is right turns on the storage under the file — below a few hundred MB/s the smaller file reads -faster end to end — and on whether the file is scanned or read by row. None of that is the writer's -to know; all of it is the caller's, or can be measured on the caller's data -([choose-encodings.md](../guide/choose-encodings.md)). - -`VortexSession.AdviseAsync` measures it (`Advice/`): it takes a sample of the data, writes it under +measurements in [the encodings, column by +column](../guide/benchmarks.md#encodings-column-by-column) show where that is not what a caller +wants. Text that does not repeat goes to zstd, which a scan decodes several times slower than FSST, +and a take far slower. A column whose values repeat across the file more than within a chunk cannot +take a dictionary, which larger chunks would give it. And the right choice depends on the storage +under the file (below a few hundred MB/s the smaller file reads faster end to end) and on whether the +file is scanned or read by row. None of that is the writer's to know. All of it is the caller's, or +can be measured on the caller's data ([choose-encodings.md](../guide/choose-encodings.md)). + +`VortexSession.AdviseAsync` measures it (`Advice/`). It takes a sample of the data, writes it under each choice that applies, reads it back, and ranks the choices by the cost of the reads the caller -declares in an `EncodingGoal` — a storage throughput, lookups per scan, or size. It returns per column -the numbers, the recommendation and why, and `ToWriteOptions` turns them into hints and chunk -targets that the writer then applies exactly, as any options. - -- **The cost of a candidate**, per value: to minimize read time, `scan + bytes ÷ throughput`, plus - `lookupsPerScan × (lookup + chunkBytes ÷ throughput) ÷ rows`; to minimize size, its bytes, the scan - breaking a tie. `scan` and `lookup` are timed in memory, on the machine the advisor runs on, so one - sample prices any throughput; each time is the least of several passes, and under the JIT the - advisor first waits until the compiler has settled. -- **The sample**: eight windows of contiguous rows spread over the data, 1 048 576 rows in all by - default. Contiguous, because a chunk's encoding depends on the rows that are together; spread, +declares in an `EncodingGoal`: a storage throughput, lookups per scan, or size. It returns, per +column, the numbers, the recommendation and the reason, and `ToWriteOptions` turns them into hints and +chunk targets that the writer then applies exactly, like any other options. + +- The cost of a candidate is computed per value. To minimize read time, it is + `scan + bytes ÷ throughput`, plus `lookupsPerScan × (lookup + chunkBytes ÷ throughput) ÷ rows`. To + minimize size, it is the candidate's bytes, with the scan breaking a tie. `scan` and `lookup` are + timed in memory, on the machine the advisor runs on, so one sample prices any throughput. Each time + is the best of several passes, and under the JIT the advisor first waits until the compiler has + settled. +- The sample is eight windows of contiguous rows spread over the data, 1 048 576 rows in all by + default. Contiguous, because a chunk's encoding depends on the rows that sit together, and spread, because data drifts. -- **The candidates**: per kind, the plain form, `Auto`, and the hints that apply, at the writer's - chunk size, and at 4 MiB and 16 MiB chunks when the sample says the chunk decides. -- **The choice**: the cheapest candidate, unless `Auto` is within 5 % of it, since advice should not - move with a measurement's noise; a larger chunk replaces the writer's own only when it costs the - column 5 % less. - -The advice is a measurement, run once for a kind of data rather than for every file: a -million-row sample costs a fraction of a second for a column of numbers and a few seconds for text. -Held outside CI to the twenty column shapes of `--tradeoffs`, it names the choice the guide's -tables measured as best on every one. Taking the advice by default would change what a write -produces and its determinism, and is not done. +- The candidates, per kind, are the plain form, `Auto` and the hints that apply, at the writer's + chunk size, and also at 4 MiB and 16 MiB chunks when the sample says the chunk size decides. +- The choice is the cheapest candidate, unless `Auto` is within 5 % of it, since advice should not + move with measurement noise. A larger chunk only replaces the writer's own when it costs the column + 5 % less. + +The advice is a measurement, run once for a kind of data rather than for every file. A million-row +sample costs a fraction of a second for a column of numbers and a few seconds for text. Checked +outside CI against the twenty column shapes of `--tradeoffs`, it names the choice the guide's tables +measured as best on every one. Taking the advice by default would change what a write produces and +its determinism, so it is not done. ## 9. Deliberately not done | approach | why not | |---|---| -| pricing from a sample, as BtrBlocks does | it prices what exact formulas price for free, and costs determinism; plan memory takes most of what it would save on drifting text | -| `fastlanes.rle` | its per-row indices need `fastlanes.delta`, which no edition carries, to come under run-end's ends; without it run-end is smaller at every run length | -| a sparse fill other than null | a dominant value that is the chunk's minimum is already a bit-packing of width zero with patches, and one that is not makes runs; finding it would take a frequency count on every chunk | -| emitting a chunk at every batch | a chunk edge inside a block saves no I/O on a pruned zone; a caller who wants no transit writes in multiples of `BlockRows` | -| choosing and writing encodings on several threads | the summaries and zstd frames are parallel; the choice and emission stay ordered on one thread, which keeps the bytes independent of the degree | -| rewriting without decoding, passing encoded arrays through | a separate feature: an order of magnitude on rewrites and compactions, nothing on first writes | -| buffering a column for global decisions, or a global sorted index | it breaks streaming and bounds memory by the file; sorted runs per chunk, merged, give the same answers ([10-indexes.md](10-indexes.md) §5.2) | +| pricing from a sample, as BtrBlocks does | it prices what exact formulas price for free, and costs determinism. Plan memory takes most of what it would save on drifting text | +| `fastlanes.rle` | its per-row indices need `fastlanes.delta`, which no edition carries, to come under run-end's ends. Without it, run-end is smaller at every run length | +| a sparse fill other than null | a dominant value that is the chunk's minimum is already a bit-packing of width zero with patches, and one that is not makes runs. Finding it would take a frequency count on every chunk | +| emitting a chunk at every batch | a chunk edge inside a block saves no I/O on a pruned zone. A caller who wants no buffering writes in multiples of `BlockRows` | +| choosing and writing encodings on several threads | the summaries and zstd frames are parallel, and the choice and emission stay ordered on one thread, which keeps the bytes independent of the degree | +| rewriting without decoding, passing encoded arrays through | a separate feature, worth an order of magnitude on rewrites and compactions and nothing on first writes | +| buffering a column for global decisions, or a global sorted index | it breaks streaming and bounds memory by the file. Sorted runs per chunk, merged, give the same answers (see [sorted runs](10-indexes.md#52-vorticitysortedrunsv1-value-to-rows-exact)) | | one FSST symbol table shared across chunks | measured at 4.3 % more bytes | -| the `vortex.dict` layout, one values child per column | what §7's limit would need; it moves the bytes of every dictionary column, and is weighed on the size ratchet and the cross-check together | +| the `vortex.dict` layout, one values child per column | what the [determinism limit](#7-determinism-and-its-limit) would need. It changes the bytes of every dictionary column, and has to be weighed on the size ratchet and the cross-check together | The reference writer this is measured against is `vortex-file`'s strategy and `vortex-btrblocks`' -schemes at 0.86.1; the kernels follow FastLanes (Afroozeh and Boncz, VLDB 2023) and ALP (Afroozeh, +schemes at 0.86.1. The kernels follow FastLanes (Afroozeh and Boncz, VLDB 2023) and ALP (Afroozeh, Kuiper and Boncz, SIGMOD 2024). diff --git a/docs/design/12-index-reads.md b/docs/design/12-index-reads.md index 816b8360..c1ada0c3 100644 --- a/docs/design/12-index-reads.md +++ b/docs/design/12-index-reads.md @@ -2,268 +2,277 @@ What a caller can ask of the read path beyond a scan: a cursor that seeks and steps in key order, counts and extremes computed without materializing rows, and rows delivered in key order. Every -structure these read is one [10-indexes.md](10-indexes.md) defines or one the file statistics -already carry: **nothing on disk changes** for any of it. +structure these read is one that [10-indexes.md](10-indexes.md) defines, or one the file statistics +already carry, so nothing on disk changes for any of it. -Three more constraints, in order: +Three more constraints apply, in order: -1. **Memory is bounded by a batch, never by the file.** An operation that would need a whole column - in memory — a sort, a distinct table over an unindexed column — is **refused**, naming the index - kind that would serve it, never emulated. -2. **Every answer equals a full materialization's**: a cursor walk equals a sort of the column, a - count equals the length of the scan's output, with indexes on or off. -3. **No allocation per step or per batch** in steady state - ([03-architecture.md](03-architecture.md) §4). +1. Memory is bounded by a batch, never by the file. An operation that would need a whole column in + memory (a sort, a distinct table over an unindexed column) is refused, naming the index kind that + would serve it, and never emulated. +2. Every answer equals what a full materialization would give. A cursor walk equals a sort of the + column, and a count equals the length of the scan's output, with indexes on or off. +3. No allocation per step or per batch in steady state (see [performance + invariants](03-architecture.md#4-performance-invariants)). ## 1. Two models over one payload -**The set model** is the scan: a predicate becomes a block mask, the mask a row selection, the -selection batches in file order ([11-write-strategy.md](11-write-strategy.md) §6). It answers *which -rows*, and it is the right engine for a filter: each touched split is read once and only live blocks -are decoded. +The set model is the scan: a predicate becomes a block mask, the mask a row selection, and the +selection batches in file order (see [reading what the writer +produced](11-write-strategy.md#6-reading-what-the-writer-produced)). It answers which rows match, and +it is the right engine for a filter, since each touched split is read once and only live blocks are +decoded. -**The order model** is a cursor: position at a key, step to the neighbour. It answers *what is -next*, which no set can. A streaming writer produces sorted runs, never a globally sorted column, so -key order exists inside each run; a cursor over the union is a **k-way merge** over the runs the -key range reaches, the merging iterator every log-structured store has. It is a reader, not a -format. +The order model is a cursor: position at a key, then step to the neighbour. It answers what comes +next, which no set can. A streaming writer produces sorted runs, never a globally sorted column, so +key order exists inside each run, and a cursor over their union is a k-way merge over the runs the +key range reaches, the merging iterator every log-structured store has. It is a reader, not a format. | model | entry point | answers | |---|---|---| -| set | `file.Scan()` | which rows match; how many; whether any; the smallest and largest | -| order | `Scan().Keys(r => r.Column)` | what is at, after or before a key; the first, the last, the i-th; the next distinct key; how many share a key | +| set | `file.Scan()` | which rows match, how many, whether any, the smallest and largest | +| order | `Scan().Keys(r => r.Column)` | what is at, after or before a key, the first, the last, the i-th, the next distinct key, how many share a key | | set driven by order | `Scan().OrderBy(r => r.Column)` | the matching rows, in key order | -Neither model mutates. Insertion into a key range, deletion and update belong to a storage engine -that owns a memtable and tombstones; the format carries no tombstone, and this library provides the -immutable half: bulk load, append, and every read below. +Neither model changes a file. Inserting into a key range, deleting and updating belong to a storage +layer that owns a memtable and tombstones, and the file format carries no tombstone. This library +provides the immutable half (bulk load, append, and every read below), and its dataset builds row +deletes and updates on top of whole objects ([deleting and updating +rows](13-dataset.md#12-deleting-and-updating-rows)). ## 2. Sources of order -A cursor needs a **key source**: the entries of a column in key order, with a row per entry, able to -locate a key in `O(log n)`. The cheapest one the file offers is used, and the builder's -`ExplainAsync` returns a `KeyPlan` naming it and why each other was rejected. +A cursor needs a key source: the entries of a column in key order, with a row per entry, able to +locate a key in `O(log n)`. The cheapest source the file offers is used, and the builder's +`ExplainAsync` returns a `KeyPlan` naming it and saying why each other one was rejected. | source | exists when | entries | seek | |---|---|---|---| -| **sorted column** | the file statistics say the column ascends, and it has a zone map | `(value, row)` per non-null row, in row order, which *is* key order | the zone bounds in memory, then one zone decoded and bisected | -| **sorted runs** | the column carries `vorticity.sorted.runs.v1` | `(key, row)` per non-null row, per run | the runs' bounds exclude runs, then a binary search in each survivor, through its fences | -| **postings** | `vorticity.postings.blocks.v1` | the distinct keys of each run; no rows | the same, on keys only | -| **dictionary** | `vorticity.dict.probe.v1` | the values of each dictionary chunk, sorted at open; no rows | in memory | -| none | an unsorted, unindexed column | — | **refused**: `VortexUnsupportedException` naming `IndexPolicy.SortedRuns` | +| sorted column | the file statistics say the column ascends, and it has a zone map | `(value, row)` per non-null row, in row order, which is key order | the zone bounds in memory, then one zone decoded and bisected | +| sorted runs | the column carries `vorticity.sorted.runs.v1` | `(key, row)` per non-null row, per run | the runs' bounds exclude runs, then a binary search in each survivor, through its fences | +| postings | `vorticity.postings.blocks.v1` | the distinct keys of each run, without rows | the same, on keys only | +| dictionary | `vorticity.dict.probe.v1` | the values of each dictionary chunk, sorted at open, without rows | in memory | +| none | an unsorted, unindexed column | none | refused with a `VortexUnsupportedException` naming `IndexPolicy.SortedRuns` | -**Cheapest first**: a sorted column costs one zone-map read and one zone decode, so it beats sorted -runs whenever both exist. **Rows when rows are wanted**: a cursor opened without `Distinct()` takes -only the first two, because a postings or dictionary source cannot say where a key's rows are, and -answering with a block's first row would make `Row` a lie. +The cheapest source wins: a sorted column costs one zone-map read and one zone decode, so it beats +sorted runs whenever both exist. A cursor opened without `Distinct()` only takes the first two +sources, because a postings or dictionary source cannot say where a key's rows are, and answering +with a block's first row would make `Row` a lie. The sorted-column source serves the files this writer produces, whose statistics carry the order -flags in the reference's semantics; the reference writer drops those flags from its file statistics. +flags in the reference's semantics. The reference writer drops those flags from its file statistics. An `Inexact` zone bound still brackets correctly, since the true minimum is at or above a stated one -and the true maximum at or below ([08-semantics.md](08-semantics.md) §1). A lying `is_sorted` gives a -wrong order, never a memory fault ([08-semantics.md](08-semantics.md) §5). +and the true maximum at or below (see [statistic +precision](08-semantics.md#1-statistic-precision-what-inexact-licenses)). A lying `is_sorted` gives a +wrong order, never a memory fault (see [untrusted +hints](08-semantics.md#5-untrusted-hints-a-three-class-policy)). -**Nulls are in no source**: no index inserts a null, and a sorted column's null rows carry no key. A -cursor never visits one; `IS NULL` is the scan's question, answered from the zone maps' null counts. +No source holds nulls. No index inserts a null, and a sorted column's null rows carry no key, so a +cursor never visits one. `IS NULL` is the scan's question, answered from the zone maps' null counts. ## 3. The cursor -`KeyCursor` (`Keys/KeyCursor.Typed.cs`), `TKey` inferred from the record member, so a key of the +`KeyCursor` (`Keys/KeyCursor.Typed.cs`) infers `TKey` from the record member, so a key of the wrong type does not compile. Positioning is asynchronous because it may read a run segment or a zone -not yet loaded; inside a loaded run a step completes synchronously. A cursor is not thread-safe; the -runs it reads are cached on the file and shared by every cursor and scan of it. +not loaded yet, while a step inside a loaded run completes synchronously. A cursor is not +thread-safe, but the runs it reads are cached on the file and shared by every cursor and scan of it. ### 3.1 Positioning The entries of a source are totally ordered by `(key, row)`, and `SeekAsync(k, op)` positions the -cursor on that order: +cursor in that order: | `op` | positions on | is | |---|---|---| -| `Exact` | the first entry whose key is `k` | a point lookup; invalid when `k` is absent | -| `AtOrAfter` | the first entry with key `≥ k` | a lower bound: the start of a range | -| `After` | the first entry with key `> k` | an upper bound: the successor | +| `Exact` | the first entry whose key is `k` | a point lookup, invalid when `k` is absent | +| `AtOrAfter` | the first entry with key `≥ k` | a lower bound, the start of a range | +| `After` | the first entry with key `> k` | an upper bound, the successor | | `AtOrBefore` | the last entry with key `≤ k` | the end of a closed range, walking backwards | | `Before` | the last entry with key `< k` | the predecessor | `SeekFirstAsync` and `SeekLastAsync` are the minimum and the maximum. `Exact` on a key with duplicates -lands on its lowest row, `AtOrBefore` on its highest. A range `[a, b)` is `AtOrAfter(a)`, then +lands on its lowest row, and `AtOrBefore` on its highest. A range `[a, b)` is `AtOrAfter(a)`, then `NextAsync` while `Key < b`. A null seek key throws, since no entry has one. ### 3.2 Stepping and direction `NextAsync` and `PrevAsync` move one entry in `(key, row)` order and return false past the ends. -`NextKeyAsync` moves to the first entry of the next **distinct** key by seeking `After(Key)` rather -than stepping through the duplicates: a loose index scan, one seek per group, so a `GROUP BY` over a -key with a million rows costs one seek per value. `PrevKeyAsync` is `Before(Key)`. +`NextKeyAsync` moves to the first entry of the next distinct key by seeking `After(Key)` rather than +stepping through the duplicates. That is a loose index scan, one seek per group, so a `GROUP BY` over +a key with a million rows costs one seek per value. `PrevKeyAsync` is `Before(Key)`. -A cursor has a direction. Stepping the other way re-seeks at the current entry, `O(r log n)` for `r` -runs, because a heap of run positions does not run backwards; that is what every merging iterator -charges, and it is documented rather than optimized. On a sorted column, one contiguous run, a flip -costs nothing. +A cursor has a direction. Stepping the other way seeks again at the current entry, which costs +`O(r log n)` for `r` runs, because a heap of run positions does not run backwards. Every merging +iterator charges that, and it is documented rather than optimized. On a sorted column, which is one +contiguous run, a change of direction costs nothing. ### 3.3 Keys and rows `Key` is the entry's key as the member's .NET type ([07-dotnet-mapping.md](07-dotnet-mapping.md)), so it goes straight back into a seek, a filter or a comparison. `Row` is the entry's row in the file, the -coordinate a scan's `Rows` accepts: that is how a cursor's answer becomes rows, and what §5 does for -the consumer. `HasRows` is false for a `Distinct()` cursor over a keys-only source, where `Row` throws. -A step that reads no key allocates nothing. +coordinate a scan's `Rows` accepts. That is how a cursor's answer becomes rows, and what [rows in key +order](#5-rows-in-key-order) does for the consumer. `HasRows` is false for a `Distinct()` cursor over +a keys-only source, where `Row` throws. A step that reads no key allocates nothing. ### 3.4 Key order per dtype -A cursor is an ordinal structure and needs a **total** order; a filter follows IEEE 754, where NaN is -unordered ([08-semantics.md](08-semantics.md) §2). The two are kept apart by stating the total order -once, for the writer of sorted runs and for this reader: +A cursor is an ordinal structure and needs a total order, whereas a filter follows IEEE 754, where +NaN is unordered (see [NaN](08-semantics.md#2-nan)). The two are kept apart by stating the total +order once, for the writer of sorted runs and for this reader: | dtype | key order | notes | |---|---|---| -| integers | numeric | the run's key column is the column's own type | -| floats | negative NaN, −∞, negatives, −0.0, +0.0, positives, +∞, positive NaN: the row encoding's total order ([06-row-encoding.md](06-row-encoding.md) §3) | −0.0 and +0.0 are two keys. A range from a filter maps onto the slice with its IEEE meaning, `x > v` never reaching a NaN: the two orders meet by construction of the bounds, never by a comparison | +| integers | numeric | the run's key column has the column's own type | +| floats | negative NaN, −∞, negatives, −0.0, +0.0, positives, +∞, positive NaN, the row encoding's total order (see [per-type encodings](06-row-encoding.md#3-per-type-encodings)) | −0.0 and +0.0 are two keys. A range from a filter maps onto the slice with its IEEE meaning, so `x > v` never reaches a NaN. The two orders meet by the way the bounds are built, never by a comparison | | text, binary | bytewise, which is code-point order for UTF-8 | | | extensions | their storage's order | a timestamp orders as its integer | | bool | not a key source | the dictionary or the postings answer a two-value domain | -| composite `(a, b, …)` | bytewise over the row encoding of the tuple | §3.6 | +| composite `(a, b, …)` | bytewise over the row encoding of the tuple | see [composite keys](#36-composite-keys) | -Within one key, entries are in **row order**, which makes a walk deterministic, a `Distinct()` -cursor's row the key's first occurrence, and the reverse walk the exact reverse. The source owns its -comparator: runs are walked in the total order, a sorted column in the IEEE order its flag was -computed in, where −0.0 and +0.0 are one key and a column holding a NaN is not sorted at all. +Within one key, entries are in row order, which makes a walk deterministic, makes a `Distinct()` +cursor's row the key's first occurrence, and makes the reverse walk the exact reverse. The source +owns its comparator. Runs are walked in the total order, and a sorted column in the IEEE order its +flag was computed in, where −0.0 and +0.0 are one key and a column holding a NaN is not sorted at +all. ### 3.5 Rank, select, and the count of a key -`RankAsync(k)` is the number of entries with a key `< k`; `SeekRankAsync(i)` positions on the entry of -rank `i`; `KeyCountAsync()` is the number of entries sharing the current key. All three are exact on -the two sources with rows, and refused on the others. A sorted array is its own counter: rank is the -sum over runs of each run's lower bound, `O(r log n)`; select is a binary search over positions with a -rank computed at each probe, `O(r log² n)`. +`RankAsync(k)` is the number of entries with a key `< k`, `SeekRankAsync(i)` positions on the entry +of rank `i`, and `KeyCountAsync()` is the number of entries that share the current key. All three +are exact on the two sources with rows, and refused on the others. A sorted array is its own +counter: rank is the sum over runs of each run's lower bound, in `O(r log n)`, and select is a binary +search over positions with a rank computed at each probe, in `O(r log² n)`. ### 3.6 Composite keys -A sorted run over the row encoding of `(a, b, …)` ([10-indexes.md](10-indexes.md) §5.4) orders -tuples bytewise, and the encoding of `(a)` is a byte prefix of the encoding of `(a, b)`, so a prefix -on the leading columns is a range. The public cursor walks one column: a composite key's runs serve -the engine's own key-ordered reads, which a dataset's clustering key drives -([13-dataset.md](13-dataset.md) §6.6), and the row-encoding package's `RowEncoder.EncodeKey` gives -one tuple's bytes. +A sorted run over the row encoding of `(a, b, …)` (see [composite +keys](10-indexes.md#54-composite-keys-through-the-row-encoding)) orders tuples bytewise, and the +encoding of `(a)` is a byte prefix of the encoding of `(a, b)`, so a prefix on the leading columns is +a range. The public cursor walks one column. A composite key's runs serve the engine's own key-ordered +reads, which a dataset's clustering key drives (see [order beyond the +lookup](13-dataset.md#66-order-beyond-the-lookup)), and the row-encoding package's +`RowEncoder.EncodeKey` gives one tuple's bytes. ## 4. Answers without rows -Four terminals on a scan, each computed without a batch and bounded by one split of memory, whatever -the file's size. Each honours the scan's filter, row range, and pruning and index switches exactly as -its enumeration would: a terminal is the scan with a different output, never a different scan. +Four terminal operations on a scan are each computed without a batch and bounded by one split of +memory, whatever the file's size. Each honours the scan's filter, row range, and pruning and index +switches exactly as its enumeration would. A terminal is the same scan with a different output, +never a different scan. ### 4.1 `AnyAsync` -Whether the scan would return a row. Cheapest proof first, stopping at the first that decides: an -empty block mask (no data read); an exact source covering the predicate, with an entry in its slice; -a block the zone maps prove full; otherwise the live splits in order, stopping at the first row that -passes. It is also the public form of a point probe: `Where(x == v).AnyAsync()` has the whole pruning -chain in front of it. +`AnyAsync` says whether the scan would return a row, trying the cheapest proof first and stopping at +the first one that decides: an empty block mask (no data read), an exact source covering the +predicate with an entry in its slice, a block the zone maps prove full, and otherwise the live splits +in order, stopping at the first row that passes. It is also the public form of a point probe: +`Where(x == v).AnyAsync()` has the whole pruning chain in front of it. ### 4.2 `CountAsync` -The number of rows the scan would return, exactly, taking per block the cheapest of three tiers: +`CountAsync` returns the exact number of rows the scan would return, taking the cheapest of three +tiers per block: | tier | proof | cost | |---|---|---| -| **exact cover** | a comparison, `IN`, `StartsWith`, or a conjunction of those on one column with a sorted source: the sum of the slices' lengths | `O(r log n)`, no data read | -| **full-block proof** | the zone maps prove every non-null row of the block matches, and its null count is known | the zone maps, already read | -| **count-only decode** | the block decoded, the filter evaluated, its true rows counted | one decode, no gather, no batch | +| exact cover | a comparison, `IN`, `StartsWith`, or a conjunction of those on one column with a sorted source: the sum of the slices' lengths | `O(r log n)`, no data read | +| full-block proof | the zone maps prove every non-null row of the block matches, and its null count is known | the zone maps, already read | +| count-only decode | the block is decoded, the filter evaluated, and its true rows counted | one decode, no gather, no batch | -The full-block proof is the dual of pruning, with the asymmetry reversed: it proves only when the -statistics point the safe way. For `x ≥ v` a block matches whole when its stated minimum is `≥ v`, -and an `Inexact` minimum still proves it; `x = v` only when the minimum and maximum are both `v` and -both `Exact`. A zone's bounds exclude NaN, so a float block's NaN count is subtracted or the block -decoded. Predicates combine as three counts per range — true, false and unknown, any two deciding -the third — so that `NOT` stays exact and two predicates on one nullable column share their unknowns, -its nulls. What that algebra cannot see falls to the third tier. +The full-block proof is the dual of pruning, with the asymmetry reversed: it only proves when the +statistics point the safe way. For `x ≥ v`, a block matches whole when its stated minimum is `≥ v`, +and an `Inexact` minimum still proves it. For `x = v`, only when the minimum and maximum are both `v` +and both `Exact`. A zone's bounds exclude NaN, so a float block's NaN count is subtracted or the block +is decoded. Predicates combine as three counts per range (true, false and unknown, any two deciding +the third), so that `NOT` stays exact and two predicates on one nullable column share their unknowns, +which are its nulls. Whatever that algebra cannot see falls to the third tier. -**A wrong proof is a wrong count**, which is what makes it testable: every count is compared with -the materialized scan with each tier forced off in turn. +A wrong proof gives a wrong count, which is what makes it testable: every count is compared with the +materialized scan with each tier forced off in turn. ### 4.3 `MinAsync` and `MaxAsync` -The smallest and largest non-null value of a column among the scan's rows, in the filter's IEEE -order, so a NaN is never an extreme, as in the statistics. In order: the file statistic, when the -scan is the whole file and the statistic `Exact`; the zone maps, where an `Exact` bound answers a -split and an `Inexact` one is a candidate, decoded best bound first, usually once; otherwise a decode -of the live blocks with a running extreme, one split at a time. +These return the smallest and largest non-null value of a column among the scan's rows, in the +filter's IEEE order, so a NaN is never an extreme, as in the statistics. They try, in order, the file +statistic when the scan covers the whole file and the statistic is `Exact`, then the zone maps, where +an `Exact` bound answers a split and an `Inexact` one is a candidate decoded best bound first (usually +just once), and otherwise a decode of the live blocks with a running extreme, one split at a time. ### 4.4 Distinct keys -`Keys(r => r.Column).Distinct()` opens a cursor that steps by `NextKey`: every distinct non-null value, -in key order, once. It admits the keys-only sources, so a `GROUP BY status` over a billion rows reads -the postings keys of each run and never a data segment. A column with no source is refused naming -`IndexPolicy.Postings`, the cheapest structure that would serve, rather than filled into a hash set -bounded by the column. Distinct is a property of the cursor, not a terminal returning a list, because -it is an ordered walk with an early exit, and a list would buffer it. +`Keys(r => r.Column).Distinct()` opens a cursor that steps by `NextKey`, visiting every distinct +non-null value once, in key order. It accepts the keys-only sources, so a `GROUP BY status` over a +billion rows reads the postings keys of each run and never a data segment. A column with no source is +refused, naming `IndexPolicy.Postings`, the cheapest structure that would serve it, rather than +filled into a hash set bounded by the column. Distinct is a property of the cursor, not a terminal +returning a list, because it is an ordered walk with an early exit, and a list would buffer it. ## 5. Rows in key order -`OrderBy(r => r.Column, descending)` makes the scan deliver its rows in the key order of a column -instead of file order. It composes with the projection and the filter, and excludes `Rows`. +`OrderBy(r => r.Column)` and `OrderByDescending(r => r.Column)` make the scan deliver its rows in the +key order of a column instead of file order. They compose with the projection and the filter, and +exclude `Rows`. -**How.** The scan is driven by a cursor instead of the split planner. The key range is the -intersection of the slices of the filter's top-level conjuncts on the key, so an `OR` or an `IN` on -the key narrows to its exact slices. The cursor walks a **window** of entries, a batch's worth; the -entries whose block the mask pruned are skipped, since the filter's other conjuncts prune as in any -scan; the window's rows are taken as a selection, the filter evaluated on them, and the batch -permuted into the window's key order. Then the next window. Memory is one window and one split's +The scan is then driven by a cursor instead of the split planner. The key range is the intersection +of the slices of the filter's top-level conjuncts on the key, so an `OR` or an `IN` on the key +narrows to its exact slices. The cursor walks a window of entries, a batch's worth at a time. Entries +whose block the mask pruned are skipped, since the filter's other conjuncts prune as in any scan, the +window's rows are taken as a selection, the filter is evaluated on them, and the batch is permuted +into the window's key order. Then comes the next window. Memory is one window plus one split's decode, never the result. -**What it delivers.** Batches in key order, ties in row order, the descending order the exact -reverse. Rows whose key is null come **last in both directions**, read after the walk as a filtered -scan the zone maps' null counts prune. A consumer that stops early is a `LIMIT`: the windows past it -are never read. A degree of parallelism applies within a window, whose splits are independent; -windows stay in order. - -**What it costs**, which a caller should read before using it. A window of `W` entries touches as many -splits as its rows are scattered over. On a sorted column the rows of consecutive keys are -consecutive, so the walk is a plain scan. On sorted runs over a column **uncorrelated with file -order**, a window can touch a split per row, and a scattered take costs what it costs: 64 rows of an -uncorrelated column read in key order take longer than Rust's reader takes for 64 rows scattered the -same way whose positions it is given, a take over 64 splits (the `against a take` axis of -`--ratio-check`, a gate the benchmark page leaves out), and less than Rust's filtered scan of the -same band, which is what a reader without the index has to do (the key-order axes of -[05-benchmarks.md](05-benchmarks.md) §3, measured on -[the benchmark page](../guide/benchmarks.md#in-one-process-after-warm-up)). So it is the tool for a -selective range, a top-k or a merge, and not for a whole uncorrelated column, where sorting the -scan's output costs less. The plan's `OrderPlan` states the source, the runs the range reaches and -the entries it admits before the read; the scan's statistics count the windows and the splits they -touched after it. - -The dataset's key-ordered reads and its compaction merge are built on this, one call per object -([13-dataset.md](13-dataset.md) §5.3, §6.6). +What it delivers is batches in key order, ties in row order, with the descending order the exact +reverse. Rows whose key is null come last in both directions, read after the walk as a filtered scan +that the zone maps' null counts prune. A consumer that stops early acts as a `LIMIT`, and the windows +past it are never read. A degree of parallelism applies within a window, whose splits are +independent, while the windows stay in order. + +What it costs should be read before using it. A window of `W` entries touches as many splits as its +rows are scattered over. On a sorted column, the rows of consecutive keys are consecutive, so the +walk is a plain scan. On sorted runs over a column uncorrelated with file order, a window can touch +one split per row, and a scattered take costs what it costs. Reading 64 rows of an uncorrelated +column in key order takes longer than Rust's reader takes for 64 rows scattered the same way whose +positions it is given, a take over 64 splits (the `against a take` axis of `--ratio-check`, a gate +the benchmark page leaves out). It takes less than Rust's filtered scan of the same band, which is +what a reader without the index has to do (the key-order axes [in one +process](05-benchmarks.md#3-in-one-process-after-warm-up), measured on [the benchmark +page](../guide/benchmarks.md#in-one-process-after-warm-up)). So it is the tool for a selective range, +a top-k or a merge, not for a whole uncorrelated column, where sorting the scan's output costs less. +The plan's `OrderPlan` states the source, the runs the range reaches and the entries it admits +before the read, and the scan's statistics count the windows and the splits they touched afterwards. + +The dataset's key-ordered reads and its compaction merge are built on this, one call per object (see +[what a compaction does](13-dataset.md#53-what-a-compaction-does) and [order beyond the +lookup](13-dataset.md#66-order-beyond-the-lookup)). ## 6. Predicates on text and lists -Four predicates beyond comparisons, `IN` and nullity, each with its pruning: +Four predicates go beyond comparisons, `IN` and nullity, each with its own pruning: | predicate | true when | null | pruning | |---|---|---|---| -| `StartsWith(p)` | the value's bytes begin with `p` | unknown | as a range `[p, succ(p))`, where `succ` increments the last byte after dropping trailing `0xFF`s: the zone bounds and a sorted run answer it, a count by exact cover | -| `Contains(p)` | `p` occurs in the value | unknown | the trigram filter and postings when `p` has three bytes or more ([10-indexes.md](10-indexes.md) §4.2, §5.3) | -| `Like(pattern)` | SQL `LIKE`: `%` any run, `_` any one character of a utf8 column and one byte of a binary one, an escape quoting either | unknown | a leading literal as `StartsWith`, every literal run of three bytes or more through the trigram structures; always re-evaluated on survivors | -| `ListContains(list, v)` | an element of the row's list equals `v`, under the comparison kernels' equality | unknown on a null list or under a null `v`; a null element matches nothing | the list's null counts; the Bloom filter over the elements ([10-indexes.md](10-indexes.md) §4.1) | +| `StartsWith(p)` | the value's bytes begin with `p` | unknown | as a range `[p, succ(p))`, where `succ` increments the last byte after dropping trailing `0xFF`s. The zone bounds and a sorted run answer it, and a count by exact cover | +| `Contains(p)` | `p` occurs in the value | unknown | the trigram filter and postings when `p` has three bytes or more (see [trigram filters](10-indexes.md#42-vorticitybloomngram3v1-trigrams-for-text-predicates) and [trigram postings](10-indexes.md#53-vorticitypostingsngram3v1-trigrams-to-blocks)) | +| `Like(pattern)` | SQL `LIKE`: `%` matches any run, `_` any one character of a utf8 column or one byte of a binary one, and an escape quotes either | unknown | a leading literal as `StartsWith`, and every literal run of three bytes or more through the trigram structures. Always evaluated again on the survivors | +| `ListContains(list, v)` | an element of the row's list equals `v`, under the comparison kernels' equality | unknown on a null list or under a null `v`. A null element matches nothing | the list's null counts, and the Bloom filter over the elements (see [the Bloom filter](10-indexes.md#41-vorticitybloomsbbfv1-a-split-block-bloom-filter-per-block)) | -Matching is bytewise, but for `_` over utf8, and case-sensitive; `NOT StartsWith` claims nothing +Matching is bytewise, except for `_` over utf8, and case-sensitive. `NOT StartsWith` claims nothing from a zone map, which is the safe answer. On the typed path these are members of `Sym` and -of a list member's symbol; on the tool path, expressions of `VortexExpr`. +of a list member's symbol, and on the tool path, expressions of `VortexExpr`. -## 7. What is not on the surface, on purpose +## 7. What is not in the public API, on purpose -A key-range type (the filter says it), a `Probe(column, value)` (`AnyAsync` is it), a distinct -terminal returning a list (the cursor walks it), a key-ordered take (the window inside `OrderBy` is -the only reorder), a sort (constraint 1), and a comparator on the cursor (keys arrive typed and in -order). Each would be a second way to say something one member already says, or a way to buffer a -file. +A key-range type (the filter already says it), a `Probe(column, value)` (`AnyAsync` is it), a +distinct terminal returning a list (the cursor walks it), a key-ordered take (the window inside +`OrderBy` is the only reordering), a sort (ruled out by the first constraint), and a comparator on the +cursor (keys arrive typed and in order). Each would be a second way to say something one member +already says, or a way to buffer a whole file. ## 8. How it is tested - A cursor's walk, forwards and backwards, with and without `Distinct()`, equals a sort of the materialized column, on every source, with windows and lanes. -- `rank(select(i)) ≤ i < rank(select(i)) + KeyCount(select(i))` over every entry. +- `rank(select(i)) ≤ i < rank(select(i)) + KeyCount(select(i))` holds over every entry. - Every count and extreme equals the materialized scan's, with pruning, indexes and each count tier forced off in turn. - A key-ordered scan equals a sort of the filtered scan's rows, nulls last, in both directions. diff --git a/docs/design/13-dataset.md b/docs/design/13-dataset.md index c9a662ac..7fc78979 100644 --- a/docs/design/13-dataset.md +++ b/docs/design/13-dataset.md @@ -1,55 +1,56 @@ # Dataset: a log-structured tree of Vortex files -`Vorticity.Dataset`, experimental (`VX0001`), keeps a versioned table over an object store: rows +`Vorticity.Dataset`, experimental under `VX0001`, keeps a versioned table over an object store: rows appended over time into plain Vortex files, compacted in the background, indexed progressively, and read as one table at any version. [datasets.md](../guide/datasets.md) and [dataset-maintenance.md](../guide/dataset-maintenance.md) show it in use. -**The requirement: no read-path cost proportional to any N.** Not the bytes of data, not the number -of data objects, commits, appends since the last compaction, blocks of an object or index fragments. -Every bound is a constant of the design — a fan-out, a page size, a level count — times a logarithm, +The requirement is that no read-path cost grows with any N: not the bytes of data, not the number of +data objects, commits, appends since the last compaction, blocks of an object or index fragments. +Every bound is a constant of the design (a fan-out, a page size, a level count) times a logarithm, plus the size of the answer. The write path pays instead, once and amortized, in compaction, and the -price is stated (§5.4). +price is stated (see [the price](#54-the-price)). -**What does not bend:** every data object is a plain Vortex file a strict Rust 0.86.1 reader opens, -with this library's structures invisible to it; the core takes no third-party dependency; an index is -a hint unless it is proven complete ([10-indexes.md](10-indexes.md) §7); no interleaving of readers, -writers, indexers, compactions and vacuums produces a wrong answer; and **the S3 client is not this -library's**: another library implements the store seam (§11), and this one never names S3 in code. +Some things do not bend. Every data object is a plain Vortex file that a strict Rust 0.86.1 reader +opens, with this library's structures invisible to it. The core takes no third-party dependency. An +index is a hint unless it is proven complete (see [indexing after the +fact](10-indexes.md#7-building-appending-and-indexing-after-the-fact)). No interleaving of readers, +writers, indexers, compactions and vacuums produces a wrong answer. And the S3 client is not this +library's: another library implements the store interface ([the store +abstraction](#11-the-store-abstraction)), and this one never names S3 in code. ## 1. Where the N's hide, and the bound that replaces each | N | the bound | mechanism | |---|---|---| -| bytes of data | nothing on the read path | identity from bytes the open already reads; the content hash is the writer's, checked only offline (§7) | -| data objects | O(log N) pages read, at most 8 + L objects touched by a key lookup | a copy-on-write tree of objects with summaries per node (§4), leveled compaction on a clustering key (§5) | -| commits | one `List` finds the latest; a commit writes O(depth) pages in one object | newest-first keys (§8.3); path copying into one commit object (§4.3) | -| appends since compaction | at most K runs per index entry per object, at most 8 objects in level 0 | one rule inside a file and across files (§5.2, §6.1) | -| blocks of an object | `1 + 16 · log₁₆ blocks` filter reads, output-sensitive | the filter tree ([10-indexes.md](10-indexes.md) §3) | -| entries of a run | two fence pages to 10¹² entries | fence pages ([10-indexes.md](10-indexes.md) §3) | -| fragments | one ranged read each, inside the commit object that wrote it | §6.4 | -| distinct keys, on a Bloom filter | the structure is chosen by cardinality | §6.5 | -| rows deleted since compaction | a binary search over at most 1 KiB of runs per batch or step, and at most an eighth of an object's rows stepped over | a deletion vector per leaf entry, folded by compaction (§12) | +| bytes of data | nothing on the read path | identity from bytes the open already reads. The content hash is the writer's, checked only offline ([identity](#7-identity-and-integrity)) | +| data objects | O(log N) pages read, at most 8 + L objects touched by a key lookup | a copy-on-write tree of objects with summaries per node ([the tree](#4-the-tree)), leveled compaction on a clustering key ([levels](#5-levels-and-compaction)) | +| commits | one `List` finds the latest, and a commit writes O(depth) pages in one object | newest-first keys ([the latest version](#83-the-latest-version-in-one-request)), path copying into one commit object ([commits](#43-commits-are-paths-written-as-one-object)) | +| appends since compaction | at most four runs per index entry per object, at most 8 objects in level 0 | one rule inside a file and across files ([the invariant](#52-the-invariant), [runs](#61-one-run-per-entry-per-object)) | +| blocks of an object | `1 + 16 · log₁₆ blocks` filter reads, output-sensitive | the filter tree ([the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)) | +| entries of a run | two fence pages up to 10¹² entries | fence pages ([the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)) | +| fragments | one ranged read each, inside the commit object that wrote it | [fragments](#64-fragments-in-commit-objects-transient) | +| distinct keys, on a Bloom filter | the structure is chosen by cardinality | [which structure](#65-which-structure-for-which-column) | +| rows deleted since compaction | a binary search over at most 1 KiB of runs per batch or step, and at most an eighth of an object's rows stepped over | a deletion vector per leaf entry, folded by compaction ([deleting rows](#12-deleting-and-updating-rows)) | ## 2. Principles -1. **Immutable objects, one commit object per version.** Nothing committed is rewritten; a reader - holding a version's header holds a snapshot. -2. **Metadata is a tree, not a list.** The set of data objects is a copy-on-write tree whose nodes - carry summaries: reading descends, committing copies a path, and neither is O(objects). -3. **Bounded fan-in everywhere**: K runs per index entry per object, 8 objects in level 0, L levels, - one object per level for a clustering-key lookup. Compaction holds the bounds; a plan reports - when it lags. -4. **Cost is output-sensitive or logarithmic**, never proportional to the dataset: a predicate that +1. Objects are immutable, with one commit object per version. Nothing committed is rewritten, and a + reader holding a version's header holds a snapshot. +2. Metadata is a tree, not a list. The set of data objects is a copy-on-write tree whose nodes carry + summaries. Reading descends and committing copies a path, and neither is O(objects). +3. Fan-in is bounded everywhere: four runs per index entry per object, 8 objects in level 0, L levels, + and one object per level for a clustering-key lookup. Compaction holds the bounds, and a plan + reports when it lags. +4. Cost is output-sensitive or logarithmic, never proportional to the dataset. A predicate that matches everything reads everything, which is the answer's own size. -5. **The write path pays**: merges when a file completes, filter trees, fences, compaction, each +5. The write path pays: merges when a file completes, filter trees, fences and compaction, each stated as a write amplification. -6. **Identity from bytes already read; integrity by checksums of what is read.** Pages are +6. Identity comes from bytes already read, and integrity from checksums of what is read. Pages are referenced by their XXH3-128, commit headers and index regions carry an XXH3-64, and nothing on the read path hashes data. -7. **Derived data folds back into the data**: a compaction that rewrites an object embeds its - indexes, so external fragments are a transient layer between "written without an index" and - "next rewritten". +7. Derived data folds back into the data. A compaction that rewrites an object embeds its indexes, so + external fragments are a transient layer between "written without an index" and "next rewritten". ## 3. Objects and keys @@ -57,264 +58,284 @@ A dataset is a store prefix holding two kinds of immutable object: | object | key | content | |---|---|---| -| data object | `data/.vortex` | a plain Vortex file, its indexes embedded, its identity in its postscript (§7) | -| commit object | `commit/.vxc` | everything else one commit produced: a header, the tree pages it wrote, the index fragments it wrote, a table of both, a trailer | - -**The commit object** opens by its head, where a Vortex file opens by its tail: magic `VXCOMMIT`, a -format version, the header's length, the header (Protobuf), then pages and fragments in the order -they were added, the table, and a 32-byte trailer — the table's offset and length, **the object's own -length**, an XXH3-64 over the header and the table, and magic `VXCT`. Recording the length turns any -truncation into a stated error rather than a shorter object. The **header** holds the version, its -parent, the schema and the names its columns gave up (§13), the clustering key, the write policy, -the tree's chunking parameters and seed, -the compaction and retention settings, and per level where its last round-robin compaction stopped -and its top page **inlined**, with the pages below it while the header stays under 256 KiB. One ranged read of the first 256 KiB opens a commit. The -pages region follows the header, so a page the commit wrote itself that ends inside those 256 KiB is -not inlined: the read that opens the commit brings it back once, where inlined it would be written -twice. Nor is a leaf that holds an object with marked rows (§12), unless this commit wrote it: its -bytes grow with every mark, and carried by every header they would cost each commit what only those -that change its level have to pay. For every page it names without carrying it — a level's top, or -a child of a page it carries — the header records where that page's version keeps its pages, as far -as the writer knows it, so a reader reads such a page in one request rather than two. - -**A page reference** is a fixed 36 bytes: the version that wrote the page, its offset relative to that +| data object | `data/.vortex` | a plain Vortex file, its indexes embedded, its identity in its postscript ([identity](#7-identity-and-integrity)) | +| commit object | `commit/.vxc` | everything else one commit produced: a header, the tree pages it wrote, the index fragments it wrote, a table of both, and a trailer | + +The commit object opens by its head, where a Vortex file opens by its tail: magic `VXCOMMIT`, a format +version, the header's length, the header (Protobuf), then pages and fragments in the order they were +added, the table, and a 32-byte trailer. The trailer holds the table's offset and length, the +object's own length, an XXH3-64 over the header and the table, and magic `VXCT`. Recording the length +turns any truncation into a stated error rather than a shorter object. + +The header holds the version, its parent, the schema and the names its columns gave up (see [schema +evolution](#13-schema-evolution)), the clustering key, the write policy, the tree's chunking +parameters and seed, the compaction and retention settings, and, per level, where its last +round-robin compaction stopped and its top page inlined, with the pages below it as long as the +header stays within 192 KiB. One ranged read of the first 256 KiB opens a commit. The pages region +follows the header, so a page the commit wrote itself that ends inside those 256 KiB is not inlined, +since the read that opens the commit brings it back once, where inlined it would be written twice. +Nor is a leaf holding an object with marked rows (see [deleting rows](#12-deleting-and-updating-rows)) +inlined, unless this commit wrote it: its bytes grow with every mark, and carried by every header they +would cost each commit what only the commits that change its level should pay. For every page it +names without carrying it (a level's top, or a child of a page it carries), the header records where +that page's version keeps its pages, as far as the writer knows, so a reader fetches such a page in +one request rather than two. + +A page reference is a fixed 36 bytes: the version that wrote the page, its offset relative to that object's pages region, its length, and its XXH3-128. A commit references every page it did not change -where it already lies, in an older commit object, and checks the hash on read; fixed-width, -relative references are what let a header name offsets that depend on its own length, and what make -two identical trees identical page for page. There is no separate page object, no orphan page after -a crash, and one conditional creation per commit. A reference names its page in every version that -shares it, so a handle keeps what it has from one version to the next: the pages it read from the -store, least recently used first out within `DatasetOptions.PageCacheBytes` (32 MiB by default); what -the reads opening its last sixteen versions brought back of their pages regions; and where each -version's pages start. A refresh or a commit asks the store again only for the pages that changed. -Pages are kept page by page only when read from the store, and by version when a read brought them -back with their commit: most of what a version wrote, the next one rewrites, and a budget in bytes -would fill with those. - -**``** is `10²⁰ − 1 − version` in twenty digits, so the **newest commit sorts -first** and the first key of a listing of `commit/` is the latest version (§8.3). There is no mutable -object anywhere. - -An existing Vortex file becomes a leaf by import, **without a copy**: the store grows by the commit -object alone. A process that lists `data/` and reads every file sees uncommitted and superseded -objects, as in every table format: the latest commit object is the only definition of the content. +where it already lies, in an older commit object, and checks the hash on read. Fixed-width, relative +references are what let a header name offsets that depend on its own length, and what make two +identical trees identical page for page. There is no separate page object, no orphan page after a +crash, and one conditional creation per commit. + +A reference names its page in every version that shares it, so a handle keeps what it has from one +version to the next: the pages it read from the store, evicted least recently used first within +`DatasetOptions.PageCacheBytes` (32 MiB by default), what the reads that opened its last sixteen +versions brought back of their pages regions, and where each version's pages start. A refresh or a +commit only asks the store again for the pages that changed. Pages are kept page by page only when +read from the store, and kept by version when a read brought them back with their commit, since most +of what a version wrote the next one rewrites, and a budget in bytes would fill up with those. + +`` is `10²⁰ − 1 − version` in twenty digits, so the newest commit sorts first and +the first key of a listing of `commit/` is the latest version (see [the latest +version](#83-the-latest-version-in-one-request)). There is no mutable object anywhere. + +An existing Vortex file becomes a leaf by import, without a copy: the store only grows by the commit +object. A process that lists `data/` and reads every file sees uncommitted and superseded objects, as +in every table format. The latest commit object is the only definition of the content. ## 4. The tree ### 4.1 A prolly tree per level -Each level (§5) is a **copy-on-write search tree over data objects**, ordered by the clustering key -when the dataset declares one and by first row position otherwise. It is a **prolly tree**, the -probabilistic B-tree of Noms and Dolt. Once a commit is a sorted batch of changes per level, a prolly -tree and a B+tree with path copying are one algorithm — load the touched leaves, merge the changes, -re-emit them, rebuild the nodes above up to the top — and only the rule that places page boundaries -differs. From the prolly rule alone follow history independence, deduplication across writers and -the strongest test oracle there is (§14). - -**The boundary rule.** A boundary is decided per entry from the XXH3-64 of the entry's key alone, -seeded with sixteen random bytes the dataset draws once, so values never move a boundary and an -indexer that updates an object's descriptor rewrites exactly `depth` pages. It is normalized by -size: no boundary before 64 KiB of entries, a probability proportional to the room left after that, -`k · entryBytes / (max − bytes)` with `k = 2`, which puts the mean page at 128 KiB by construction, -and a forced boundary at 256 KiB, the page cap. After an edit, chunking continues until a new -boundary meets an old one, so **the tree is a pure function of the key set and the parameters**: the -same objects committed in any order give byte-identical pages and one root hash. The rule is behind a -seam, whose other implementation, a B+tree's fill factor, is measured on the same bench. +Each level ([levels](#5-levels-and-compaction)) is a copy-on-write search tree over data objects, +ordered by the clustering key when the dataset declares one and by first row position otherwise. It +is a prolly tree, the probabilistic B-tree of Noms and Dolt. Once a commit is a sorted batch of +changes per level, a prolly tree and a B+tree with path copying are the same algorithm (load the +touched leaves, merge the changes, emit them again, rebuild the nodes above up to the top), and only +the rule that places page boundaries differs. The prolly rule alone gives history independence, +deduplication across writers and the strongest test oracle available ([how it is +tested](#15-how-it-is-tested)). + +A boundary is decided per entry from the XXH3-64 of the entry's key alone, seeded with sixteen random +bytes the dataset draws once, so values never move a boundary and an indexer that updates an object's +descriptor rewrites exactly `depth` pages. It is normalized by size: no boundary before 64 KiB of +entries, then a probability proportional to the room left, `k · entryBytes / (max − bytes)` with +`k = 2`, which puts the mean page at 128 KiB by construction, and a forced boundary at 256 KiB, the +page cap. After an edit, chunking continues until a new boundary meets an old one, so the tree is a +pure function of the key set and the parameters: the same objects committed in any order give +byte-identical pages and one root hash. The rule sits behind an interface whose other +implementation, a B+tree's fill factor, is measured on the same bench. With entries of about 180 bytes the fan-out is about 680. The header inlines a level's top page and, -while it fits, the pages below, so a lookup reads **0 dependent pages** up to about 650 objects, -**1** up to about 400 000, **2** up to about 280 million; a leaf with marked rows that another commit -wrote is one more, once (§3). +while they fit, the pages below it, so a lookup reads 0 dependent pages up to about 650 objects, 1 up +to about 400 000, and 2 up to about 280 million. A leaf with marked rows that another commit wrote +costs one more read, once (see [objects and keys](#3-objects-and-keys)). ### 4.2 What a node carries -A **leaf entry** describes one data object: +A leaf entry describes one data object: -- **its key**: the row encoding of the object's smallest clustering key followed by its first row +- its key: the row encoding of the object's smallest clustering key followed by its first row position, or the position alone without a clustering key, then the object's uid, which makes the key - a function of the object rather than of the tree it was committed into (§8.2); + a function of the object rather than of the tree it was committed into (see + [rebase](#82-rebase-by-re-applying-operations)) - its size, row count and identity: the uid in its postscript, the store's token when the store - gives one, and the XXH3-128 the writer computed (§7); -- **summaries**: per summarized column its minimum, maximum and null count, taken from the object's - own file statistics before its put, so they cost the append no request. The first 32 top-level - columns are summarized by default (`DatasetOptions.SummaryColumns`), and the clustering key's - always; -- its index descriptor: embedded in the object, or per entry the fragments that cover it (§6.4); -- **its deletion vector**, when a delete marked rows in it: their positions as sorted runs, after - everything else the entry holds, so that an entry without one is the bytes it always was. Its row - count is then the live rows, and its summaries bounds only (§12). A vector takes at most 1 KiB, - which the fan-out of the page holding it pays until the object is next rewritten: a page of - entries all at the cap holds about a hundred. - -An **internal entry** holds a page reference to a child, the sum of its rows, the **union** of its -children's summaries, and the **tally** of the objects under it: their bytes, the largest of them, the -rows marked deleted in them, the most index fragments one of them carries, and the most any one of -them has marked, as a share of its rows and as the bytes of its vector, which a compaction plans by -(§5.3). The union is intersection-shaped, which is the one thing about it a reader must -get right: a column travels up only when every child carries it, and a bound only when every child -has one, since keeping a bound because another child was silent would prune a subtree that holds the -answer. One inexact child makes the union inexact. - -**What the walk skips.** A subtree whose union refutes the predicate is skipped unread; an object whose -own summaries refute it is not opened; an object outside a row range is neither read nor counted, by -the row sums at every level. The question is asked by the core's own pruner, the one a file's -statistics use, so there is one implementation of [08-semantics.md](08-semantics.md) §1 and two -callers. Measured: a filter inside one of eight objects opens one object with the summaries and eight -without, for the same rows. + provides one, and the XXH3-128 the writer computed ([identity](#7-identity-and-integrity)) +- summaries: per summarized column, its minimum, maximum and null count, taken from the object's own + file statistics before its put, so they cost the append no request. The first 32 top-level columns + are summarized by default (`DatasetOptions.SummaryColumns`), and the clustering key's always are +- its index descriptor: indexes embedded in the object, or per entry the fragments that cover it (see + [fragments](#64-fragments-in-commit-objects-transient)) +- its deletion vector, when a delete marked rows in it: their positions as sorted runs, placed after + everything else the entry holds, so an entry without one keeps exactly the bytes it always had. Its + row count is then the live rows, and its summaries become bounds only ([deleting + rows](#12-deleting-and-updating-rows)). A vector takes at most 1 KiB, which reduces the fan-out of + the page holding it until the object is next rewritten: a page of entries all at the cap holds + about a hundred + +An internal entry holds a page reference to a child, the sum of its rows, the union of its children's +summaries, and the tally of the objects under it: their bytes, the largest of them, the rows marked +deleted in them, the most index fragments any one of them carries, and the most any one of them has +marked, both as a share of its rows and as the bytes of its vector, which a compaction plans by (see +[what a compaction does](#53-what-a-compaction-does)). The union behaves like an intersection, which +is the one thing about it a reader must get right. A column only travels up when every child carries +it, and a bound only when every child has one, since keeping a bound because another child was silent +would prune a subtree that holds the answer. One inexact child makes the union inexact. + +The walk skips a subtree whose union refutes the predicate without reading it. An object whose own +summaries refute it is not opened, and an object outside a row range is neither read nor counted, +thanks to the row sums at every level. The question is asked by the core's own pruner, the same one +a file's statistics use, so there is one implementation of [statistic +precision](08-semantics.md#1-statistic-precision-what-inexact-licenses) and two callers. Measured: a +filter inside one of eight objects opens one object with the summaries, and eight without, for the +same rows. ### 4.3 Commits are paths, written as one object -A commit is a sorted batch of changes per level: entries added, removed, or updated. Adding an object -re-chunks its leaf and the pages above it, O(depth) pages, laid out inside the commit object with the -header and the fragments; every page the commit did not change is referenced where it lies, and a page -whose new content equals one the writer has already read is referenced rather than rewritten, which -the prolly rule makes common after a rebase. One conditional creation commits it (§8), its cost does -not depend on the number of objects, and a crash leaves nothing to clean up but data objects. +A commit is a sorted batch of changes per level: entries added, removed or updated. Adding an object +chunks its leaf again along with the pages above it, O(depth) pages, laid out inside the commit object +with the header and the fragments. Every page the commit did not change is referenced where it lies, +and a page whose new content equals one the writer has already read is referenced rather than +rewritten, which the prolly rule makes common after a rebase. One conditional creation commits it +([the commit protocol](#8-the-commit-protocol)), its cost does not depend on the number of objects, +and a crash leaves nothing to clean up except data objects. ## 5. Levels and compaction ### 5.1 Why levels are part of the read bound -Without a merge policy, the objects a lookup touches are the appends. The bound of §2 holds only if -compaction holds an invariant, so the invariant is specified, and a plan (`PlanCompactionAsync`) -reports every violation of it as a **lag** without opening a data object. A lag is reported, never -refused: compaction is the caller's background job, and **a library never stalls a writer**. +Without a merge policy, the objects a lookup touches are the appends themselves. The bound of the +[principles](#2-principles) only holds if compaction maintains an invariant, so the invariant is +specified, and a plan (`PlanCompactionAsync`) reports every violation of it as a lag without opening +a data object. A lag is reported, never refused: compaction is the caller's background job, and a +library never stalls a writer. ### 5.2 The invariant -- **Level 0** holds appended objects as they came, overlapping in key: at most **8**. -- **Levels 1 to L**, when the dataset declares a clustering key, hold objects with **disjoint key - ranges** within a level, so a lookup by key touches at most one object per level: **8 + L** in all. - This is leveled compaction as RocksDB does it, with Vortex files as the sorted tables. Level `i` - holds `128 KiB × F^i`, `F` = 10, and an object written into it targets `128 KiB × F^(i−1)`, **capped - at 4 MiB**: the cap is what a delete rewrites (§12), and the capacities keep growing past it, so - that a level holds `F` times the one below whatever the size of its objects and a dataset of `N` - bytes spans about `log_F(N / 128 KiB)` levels. A level holding a single object is never over its - size. -- Without a clustering key, levels are **size tiers**: a lookup touches every object the summaries do - not refute, which is output-sensitive rather than bounded, and the plan says so. -- **Inside every object, at most K = 4 runs per index entry**, the same rule as a single file - appended in place ([10-indexes.md](10-indexes.md) §3). - -`CompactionOptions` carries the fan-out, the level-0 ceiling, the level-1 target, the object cap and -a cap on levels: under a cap, the top level has no size bound and only grows, since merging it into -itself would rewrite key-disjoint objects into the same objects. +- Level 0 holds appended objects as they came, overlapping in key, and at most 8 of them. +- Levels 1 to L, when the dataset declares a clustering key, hold objects with disjoint key ranges + within a level, so a lookup by key touches at most one object per level, 8 + L in all. This is + leveled compaction as RocksDB does it, with Vortex files as the sorted tables. Level `i` holds + `128 KiB × F^i`, with `F` = 10, and an object written into it targets `128 KiB × F^(i−1)`, capped + at 4 MiB. The cap is what a delete rewrites ([deleting rows](#12-deleting-and-updating-rows)), and + the capacities keep growing past it, so a level holds `F` times the one below whatever the size of + its objects, and a dataset of `N` bytes spans about `log_F(N / 128 KiB)` levels. A level holding a + single object is never over its size. +- Without a clustering key, levels are size tiers. A lookup touches every object the summaries do not + refute, which is output-sensitive rather than bounded, and the plan says so. +- Inside every object, an index entry has at most four runs, the same rule as for a single file + appended in place (see [the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)). + +`CompactionOptions` carries the fan-out, the level 0 ceiling, the level 1 target, the object cap and a +cap on the number of levels. Under a level cap, the top level has no size bound and only grows, since +merging it into itself would rewrite key-disjoint objects into the same objects. ### 5.3 What a compaction does -It reads `k` objects and writes one or more at the destination level's target: a k-way merge of the -inputs on the clustering key when there is one, a concatenation otherwise, and **the writer builds the -outputs' embedded indexes once**, dropping the inputs' fragments. Compaction is therefore also the -index compaction of everything it touches. - -**Level 0 is not sorted, and nobody sorts it.** An append arrives in any order and the writer streams -with bounded memory ([11-write-strategy.md](11-write-strategy.md)). So the write policy **requires a -sorted run on the clustering key** in every object whose rows may be out of order -([10-indexes.md](10-indexes.md) §6), and the compaction reads each input **in key order through that -run** ([12-index-reads.md](12-index-reads.md) §5): the merge's inputs are ordered, its outputs are -sorted by construction, and the writer never learns to sort. The run is the object's structure rather -than a hint, so the index budget spares it: on a narrow table it is as large as the column it orders, -and a budget of a tenth of the data would refuse it on every object past a mebibyte, the merge's -outputs included. - -**An object whose rows come in key order carries no run** when the key is one integer column that -holds no null: a merge's output, or the rewrite of one (§12). Its statistics say the column is -sorted, and a sorted column is the source a key cursor and a key-ordered read take before any run, -its zone map as the index, so a run beside it is bytes nobody reads. The seal checks that a key cursor -opens on the column alone, so that an object whose order was lost is refused rather than read as one -without the key. - -**Level 0 empties into the first level whose capacity holds it**, not always into level 1. A few -small commits, a few kilobytes, merge into level 1, rewriting a level a few times their size rather -than the levels that hold the dataset; a load of gigabytes goes past the levels it would only -overflow, and is written once rather than once per level on its way up. Nothing orders the levels -among themselves but their sizes: a row in any level is a row of the dataset, and each level above 0 +A compaction reads `k` objects and writes one or more at the destination level's target: a k-way +merge of the inputs on the clustering key when there is one, and a concatenation otherwise. The +writer builds the outputs' embedded indexes once, dropping the inputs' fragments, so compaction is +also the index compaction of everything it touches. + +Level 0 is not sorted, and nothing ever sorts it. An append arrives in any order, and the writer +streams with bounded memory ([11-write-strategy.md](11-write-strategy.md)). So the write policy +requires a sorted run on the clustering key in every object whose rows may be out of order (see [the +policy](10-indexes.md#6-the-policy)), and the compaction reads each input in key order through that +run (see [rows in key order](12-index-reads.md#5-rows-in-key-order)). The merge's inputs are ordered, +its outputs are sorted by construction, and the writer never has to learn to sort. The run is the +object's structure rather than a hint, so the index budget spares it. On a narrow table it is as large +as the column it orders, and a budget of a tenth of the data would refuse it on every object past a +mebibyte, the merge's outputs included. + +An object whose rows come in key order carries no run when the key is a single integer column with +no nulls, as is the case for a merge's output or its rewrite ([deleting +rows](#12-deleting-and-updating-rows)). Its statistics say the column is sorted, and a sorted column +is the source a key cursor and a key-ordered read take before any run, with its zone map as the index, +so a run next to it would be bytes nobody reads. Sealing the object checks that a key cursor opens on +the column alone, so an object whose order was lost is refused rather than read as one without the +key. + +Level 0 empties into the first level whose capacity can hold it, not always into level 1. A few +small commits of a few kilobytes merge into level 1, rewriting a level a few times their size rather +than the levels that hold the dataset. A load of gigabytes skips the levels it would only overflow, +and is written once rather than once per level on its way up. Nothing orders the levels among +themselves except their sizes: a row in any level is a row of the dataset, and each level above 0 only has to stay key-disjoint. -The merge emits **windows, not rows**: at each step the input holding the smallest key emits every -row at or below the smallest key the others hold, a contiguous window of its current batch, so the +The merge emits windows, not rows. At each step the input holding the smallest key emits every row at +or below the smallest key the others hold, which is a contiguous window of its current batch, so the cost is `k` comparisons per window. Rows are compared by their row encoding, the order the tree, the -runs and the seeks already use. An output is sealed at the first key change past its target, **never -inside a key**, since two outputs sharing a boundary value would overlap; the destination inputs are -chosen against the union of the source range. Rows with a null key come last. A composite key merges -on its tuple; a composite key whose column holds nulls is refused, since its run holds no such tuple. -The rows written are checked against the inputs' count: a rewrite that loses rows is the one failure -it must not have. Measured on four interleaved level-0 objects: 42 808 bytes in, one sorted object of -16 596 out. - -Triggers: level 0 above its ceiling, a level above its size, an object whose marks are due (§12), -then an entry above K fragments, a compaction of index bytes only (§6.4). - -**A plan descends the trees rather than reading them.** A level's bytes are the sum of the -tallies its top page carries, and the header carries that page, or the handle keeps it when it is a -leaf with marks (§3). The object a job pushes down from a level over its size is at the end of one -descent: the largest, a leveled dataset's default, following the largest tally at each page, or -under a round robin (`CompactionOptions.Pick`) the first past the key the level's last job stopped -at, which the header records with the level, following the keys. The objects of the level below -that the job's key range meets, the objects over their fragments and the one whose marks are the -most due are found by walks that skip every subtree whose summary or tally rules it out. A cold plan -over 125 000 objects asks the store for two pages, where reading every leaf asks for 124. A tiered -job concatenates a run of one level's objects that nothing else sits between. Under a round robin, -a tiered dataset's default, it takes the run that starts past the level's pointer, which ends -before the first object another level holds past that start, one descent per level: a plan over -100 000 objects by arrival asks for one page, where reading every leaf asks for 71. The longest run -(`CompactionPick.Largest`) is a fact of every level's order in full, and a plan that takes it still -reads every leaf. So does a plan over a level whose pages were written before tallies, until a -commit rewrites them. Both choose the job the other would. When many processes append, one coordinator batching their -rows into one commit answers the contention; without it the protocol of §8 stays correct, only -slower. +runs and the seeks already use. An output is sealed at the first key change past its target, never +inside a key, since two outputs sharing a boundary value would overlap, and the destination inputs +are chosen against the union of the source range. Rows with a null key come last. A composite key +merges on its tuple, and a composite key whose columns hold nulls is refused, since its run holds no +such tuple. The rows written are checked against the inputs' count, because a rewrite that loses rows +is the one failure it must not have. Measured on four interleaved level 0 objects: 42 808 bytes in, +and one sorted object of 16 596 out. + +The triggers, in order, are level 0 above its ceiling, a level above its size, an object whose marks +are due ([deleting rows](#12-deleting-and-updating-rows)), and then an entry with more than four +fragments, which is a compaction of index bytes only (see +[fragments](#64-fragments-in-commit-objects-transient)). + +A plan descends the trees rather than reading them. A level's bytes are the sum of the tallies its top +page carries, and the header carries that page, or the handle keeps it when it is a leaf with marks. +The object a job pushes down from a level over its size sits at the end of one descent: the largest, +the default for a leveled dataset, found by following the largest tally at each page, or, under a +round robin (`CompactionOptions.Pick`), the first one past the key where the level's previous job +stopped, which the header records with the level, found by following the keys. The objects of the +level below that the job's key range meets, the objects with too many fragments and the one whose +marks are most due are found by walks that skip every subtree whose summary or tally rules it out. A +cold plan over 125 000 objects asks the store for two pages, where reading every leaf would ask for +124. + +A tiered job concatenates a run of one level's objects with nothing else between them. Under a round +robin, which is a tiered dataset's default, it takes the run that starts past the level's pointer and +ends before the first object another level holds past that start, one descent per level: a plan over +100 000 objects by arrival asks for one page, where reading every leaf would ask for 71. The longest +run (`CompactionPick.Largest`) depends on every level's order in full, so a plan that takes it still +reads every leaf, as does a plan over a level whose pages were written before tallies existed, until a +commit rewrites them. Both choose the job the other would. When many processes append, one +coordinator batching their rows into one commit solves the contention. Without it, the [commit +protocol](#8-the-commit-protocol) stays correct, only slower. ### 5.4 The price -Leveled compaction with `F` = 10 rewrites each row about `F/2` times per level it crosses: about -**20 to 25×** the appended bytes over four levels. A small commit crosses every level, a load only -those above the one it lands in. Tiered rewrites each row about once per level, about **L×**. The -default is leveled when a clustering key is declared and tiered otherwise. A dataset that appends -1 TiB a day, leveled, writes about 25 TiB a day in compaction: the honest cost of the -one-object-per-level read bound, which every LSM store pays. +Leveled compaction with `F` = 10 rewrites each row about `F/2` times per level it crosses, about 20 to +25× the appended bytes over four levels. A small commit crosses every level, and a bulk load only +those above the one it lands in. Tiered compaction rewrites each row about once per level, about L×. +The default is leveled when a clustering key is declared and tiered otherwise. A dataset that appends +1 TiB a day, leveled, writes about 25 TiB a day in compaction. That is the honest cost of the +one-object-per-level read bound, and every LSM store pays it. -Who pays it, and when, is the caller's: compaction runs when `CompactAsync` is called, in a loop -any host runs, `RunCompactionAsync`, which plans, runs the job due for it at a rate it is given, and -sleeps when nothing is due, or, for level 0 under a byte budget, in the writer's own commit -(`DatasetOptions.InlineCompactionBytes`). [15-compaction.md](15-compaction.md) weighs the drivers — inline in a -commit, on demand, in the background — planning in the depth of the tree, and what a store that -cannot delete does to the price. +Who pays it, and when, is the caller's decision. Compaction runs when `CompactAsync` is called, in a +loop any host runs (`RunCompactionAsync`, which plans, runs the job that is due at the rate it is +given, and sleeps when nothing is due), or, for level 0 under a byte budget, in the writer's own +commit (`DatasetOptions.InlineCompactionBytes`). [15-compaction.md](15-compaction.md) weighs those +drivers (inline in a commit, on demand, in the background), planning from the top of the tree, and +what a store that cannot delete does to the price. ## 6. Indexes at scale ### 6.1 One run per entry per object An index entry has one run per object, merged when the object completes, and an append in place keeps -it at K = 4 runs by merging past that ([10-indexes.md](10-indexes.md) §3). A lookup on a key -uncorrelated with row order therefore reads at most K fence pages per object, never one per chunk. +it at no more than four runs by merging past that (see [the directory and the +runs](10-indexes.md#3-the-directory-and-the-runs)). A lookup on a key uncorrelated with row order +therefore reads at most four fence pages per object, never one per chunk. ### 6.2 The filter tree and the fences -Bloom filters form a tree of fan-out 16 and long runs page their segment tables into fences -([10-indexes.md](10-indexes.md) §3): a point probe reads `1 + 16 · log₁₆ blocks` filters at most, a -lookup two fence pages before its payload, whatever the object's size. +Bloom filters form a tree with a fan-out of 16, and long runs page their segment tables into fences +(see [the directory and the runs](10-indexes.md#3-the-directory-and-the-runs)). A point probe reads at +most `1 + 16 · log₁₆ blocks` filters, and a lookup reads two fence pages before its payload, whatever +the object's size. ### 6.3 Checksums -Every page, fragment and index region carries its checksum (§7), and a region that fails claims -nothing: a reader answers exactly without it, and `VerifyAsync` names it. +Every page, fragment and index region carries its checksum ([identity](#7-identity-and-integrity)), +and a region that fails claims nothing. A reader still answers exactly without it, and `VerifyAsync` +names it. ### 6.4 Fragments, in commit objects, transient -A **fragment** is one entry's runs over a block range of one object, in the container the core -builds for a file it cannot append to ([10-indexes.md](10-indexes.md) §7), with its own encoding table +A fragment is one entry's runs over a block range of one object, in the container the core builds for +a file it cannot append to (see [indexing after the +fact](10-indexes.md#7-building-appending-and-indexing-after-the-fact)), with its own encoding table and dtypes so that it decodes without the object's footer. An indexer works by object and block -range, and each batch is one commit whose **commit object carries the fragment's bytes**; the leaf -entry references it by `(version, offset, length, XXH3-64)`, one ranged read. Coverage may be partial: -a pruner uses the covered blocks, and a key source waits for complete coverage, which the entry -states, so an object can be indexed progressively without ever answering wrong. - -A fragment is its content: two indexers of one range write one fragment, and a drop removes the -fragment of those bytes wherever it lies. Past K fragments an entry's fragments are **bundled** — the -containers copied as they are, one after another, behind a table — rather than merged, since a Bloom -tree's upper filters cannot be rebuilt from its leaves and a run's payloads hold offsets into their -own container. A data compaction that rewrites the object embeds the index and drops every fragment, -so at steady state the external layer holds only the indexes of objects nobody has rewritten. +range, and each batch is one commit whose commit object carries the fragment's bytes. The leaf entry +references it by `(version, offset, length, XXH3-64)`, one ranged read. Coverage may be partial: a +pruner uses the covered blocks, and a key source waits for complete coverage, which the entry states, +so an object can be indexed progressively without ever answering wrong. + +A fragment is identified by its content. Two indexers of the same range write one fragment, and +dropping a fragment removes the fragment with those bytes wherever it lies. Past four fragments, an +entry's fragments are bundled (the containers copied as they are, one after another, behind a table) +rather than merged, since a Bloom tree's upper filters cannot be rebuilt from its leaves and a run's +payloads hold offsets into their own container. A data compaction that rewrites the object embeds the +index and drops every fragment, so at steady state the external layer only holds the indexes of +objects nobody has rewritten yet. ### 6.5 Which structure for which column @@ -323,198 +344,208 @@ so at steady state the external layer holds only the indexes of objects nobody h | few, or dictionary-encoded | zone map and dictionary probe | one values child per chunk the zone map keeps | | up to about 10⁵ per 16 blocks | Bloom filter tree | `1 + 16 · log₁₆ blocks` filters, output-sensitive | | many, or a range or an order | one sorted run per object, with fences | two fence pages and two segments per object touched | -| the dataset's lookup key | the **clustering key** | one object per level, then its run: the only structure that bounds the objects | +| the dataset's lookup key | the clustering key | one object per level, then its run: the only structure that bounds the objects | -`Auto` gives up a Bloom filter on a column whose first node passes a filter's capacity, where a sorted -run is the structure ([10-indexes.md](10-indexes.md) §6). +`Auto` gives up a Bloom filter on a column whose first node exceeds a filter's capacity, where a +sorted run is the right structure (see [the policy](10-indexes.md#6-the-policy)). ### 6.6 Order beyond the lookup -The trees are ordered by the clustering key, so the order of **that one key** is bounded everywhere: +The trees are ordered by the clustering key, so the order of that one key is bounded everywhere: | need | mechanism | cost | |---|---|---| -| walk the objects by key | an in-order walk of a level's tree, the next pages prefetched eight ahead | O(log N + objects touched), about one dependent round trip per window of eight pages | -| rows in key order, `OrderBy` | a merge of each object's key-ordered read; inside a level above 0 the objects are key-disjoint | at most 8 + L objects open at once | +| walk the objects by key | an in-order walk of a level's tree, with the next pages prefetched eight ahead | O(log N + objects touched), about one dependent round trip per window of eight pages | +| rows in key order, `OrderBy` | a merge of each object's key-ordered read. Inside a level above 0 the objects are key-disjoint | at most 8 + L objects open at once | | rows by position, `Rows(a, b)` | the row sums of every node | O(log N) plus the objects the range touches | -| count and rank on the clustering key | objects whose summaries lie wholly inside the range counted unopened; the core's exact cover in the boundary objects | at most level 0's objects plus two per level opened | +| count and rank on the clustering key | objects whose summaries lie wholly inside the range are counted unopened, and the core's exact cover handles the boundary objects | at most level 0's objects plus two per level opened | | an order on another column | no tree helps | one cursor per object the summaries cannot refute | -**An object opens when it could hold the next row, not before.** Objects are offered in the order of a -lower bound on their keys — on the clustering key, upward, a leaf key is the object's exact minimum — -and one is opened only once that bound is at or below the smallest key an open input holds. So a -consumer that stops after `k` rows opens no object whose minimum lies past the k-th key: `LIMIT 10` -opens one object of four. Measured on the same rows: four cursors over four interleaved level-0 -objects, one over the key-disjoint level a compaction made of them. **A float has no such bound**, -since a zone's minimum and maximum exclude NaN while the key order places NaNs at the ends; float -objects open before the first row, and an object whose first row lies below the bound it was ordered -by raises `VortexFormatException` rather than deliver a wrong order. Ties between objects follow the -dataset's order, so a descending read is the exact reverse of an ascending one. - -**A key cursor walks both ways.** Walking down is the same merge with a max-heap. The objects of a -level above 0 are key-disjoint, so the minimum of the next object of the level is a strict upper -bound on an object's keys, and the object opens only once the walk passes below it; level 0's -objects, which nothing but their summaries bounds from above, open at the seek, at most eight and the -lag. A step against the walk's direction seeks every object again just past the current entry, whose -place in the order is its key and then its row in the dataset: an object before it in the dataset's -order lands at or before the key, one after it strictly before, and its own object steps. That is the -flip every merging iterator pays ([12-index-reads.md](12-index-reads.md) §3.2), and the walk down is -the walk up reversed, ties included. - -**A key cursor reads the trees only where it walks.** In each level the walk stands between two -objects, a place found by one path down the level's tree and moved an object at a time, a page read -only when the walk crosses into it; the pages read, and the objects' entries in them, are parsed once -per version and shared by its cursors. An object's first row is counted when the walk first takes one -of its entries: the rows of every level's objects before its tree key, along one path of each. A seek -on a handle that holds nothing yet so reads the depth of each level, whatever the number of objects: -over 20 000 objects in one level, one page, where a listing of the objects read every leaf, 23. - -Any other order costs, at best, one cursor per object the summaries cannot refute: `ORDER BY x LIMIT -k` prunes by the summaries, which is good in practice and output-sensitive, never bounded. A second -bounded order is a second copy of the data clustered by that key, a dataset of its own; a global -secondary index is excluded, since every compaction would rewrite it. +An object opens when it could hold the next row, and not before. Objects are offered in the order of +a lower bound on their keys (on the clustering key, ascending, a leaf key is the object's exact +minimum), and one is only opened once that bound is at or below the smallest key an open input holds. +So a consumer that stops after `k` rows opens no object whose minimum lies past the k-th key, and +`LIMIT 10` opens one object of four. Measured on the same rows: four cursors over four interleaved +level 0 objects, and one over the key-disjoint level a compaction made of them. A float has no such +bound, since a zone's minimum and maximum exclude NaN while the key order places NaNs at the ends, so +float objects open before the first row, and an object whose first row lies below the bound it was +ordered by raises `VortexFormatException` rather than deliver a wrong order. Ties between objects +follow the dataset's order, so a descending read is the exact reverse of an ascending one. + +A key cursor walks both ways. Walking down is the same merge with a max-heap. The objects of a level +above 0 are key-disjoint, so the minimum of the next object in the level is a strict upper bound on an +object's keys, and the object only opens once the walk passes below it. Level 0's objects, which +nothing but their summaries bounds from above, open at the seek, at most eight plus the lag. A step +against the walk's direction seeks every object again just past the current entry, whose place in the +order is its key and then its row in the dataset: an object before it in the dataset's order lands at +or before the key, one after it strictly before, and its own object steps. That is the change of +direction every merging iterator pays for (see [stepping and +direction](12-index-reads.md#32-stepping-and-direction)), and the walk down is the walk up reversed, +ties included. + +A key cursor only reads the trees where it walks. In each level the walk stands between two objects, +a place found by one path down the level's tree and moved one object at a time, with a page read only +when the walk crosses into it. The pages read, and the objects' entries in them, are parsed once per +version and shared by its cursors. An object's first row is counted when the walk first takes one of +its entries, from the rows of every level's objects before its tree key, along one path of each. A +seek on a handle that holds nothing yet therefore reads the depth of each level, whatever the number +of objects: over 20 000 objects in one level it reads one page, where a listing of the objects would +read every leaf, 23 pages. + +Any other order costs, at best, one cursor per object the summaries cannot refute. `ORDER BY x LIMIT +k` prunes by the summaries, which is good in practice and output-sensitive, but never bounded. A +second bounded order requires a second copy of the data clustered by that key, a dataset of its own. +A global secondary index is excluded, since every compaction would rewrite it. ## 7. Identity and integrity -- **Every postscript this library writes mints a uid**: sixteen random bytes in the metadata entry - `vorticity.identity`, written right after the footer, so the open's tail read covers it and - reading it costs no request. It identifies a **version of the bytes**, not a file: an append in - place mints a new one, and a fragment bound to the old uid is refused. It costs 72 to 96 bytes a - file; `VortexWriteOptions.Identity` pins it for a reproducible write. -- **The store's token** — a version id or an entity tag, or on a file system the length and the - modification time — is recorded when the store gives one and compared as an opaque string. It is the - only binding for a foreign file without a uid; on a file system it is a heuristic, and the entry - says so. `VortexOpenOptions.IndexFragmentsNeedIdentity` refuses a fragment bound so. -- **The content hash is the writer's.** The writer sees every byte it emits, so an XXH3-128 of the - object costs nothing then and is recorded in the leaf entry. **No reader computes it**: `verify` - does, offline. -- **Pages are addressed by their XXH3-128**, and a commit object carries an XXH3-64 over its header - and table: integrity, deduplication and idempotent retries in one mechanism. A torn page or header - makes its version unreadable, with the reason; a corrupt fragment or index region claims nothing; a - corrupt data segment is a `VortexFormatException`. XXH3 guards against accident, not forgery. +- Every postscript this library writes mints a uid: sixteen random bytes in the metadata entry + `vorticity.identity`, written right after the footer, so the open's tail read covers it and reading + it costs no request. It identifies a version of the bytes, not a file: an append in place mints a + new one, and a fragment bound to the old uid is refused. It costs 72 to 96 bytes per file, and + `VortexWriteOptions.Identity` pins it for a reproducible write. +- The store's token (a version id or an entity tag, or, on a file system, the length and the + modification time) is recorded when the store provides one and compared as an opaque string. It is + the only binding for a foreign file without a uid. On a file system it is a heuristic, and the entry + says so. `VortexOpenOptions.IndexFragmentsNeedIdentity` refuses a fragment bound that way. +- The content hash is the writer's. The writer sees every byte it emits, so an XXH3-128 of the object + costs nothing at that point and is recorded in the leaf entry. No reader computes it, and only + verification does, offline. +- Pages are addressed by their XXH3-128, and a commit object carries an XXH3-64 over its header and + table, which gives integrity, deduplication and idempotent retries in one mechanism. A torn page or + header makes its version unreadable, with the reason. A corrupt fragment or index region claims + nothing. A corrupt data segment raises `VortexFormatException`. XXH3 guards against accidents, not + forgery. ## 8. The commit protocol ### 8.1 A commit is one conditional creation -A writer reads the latest commit `N`, uploads its data objects under fresh uid keys, in parallel, -then creates `commit/` — header, pages and fragments in one object — with the store's -**put-if-absent**, streamed from a pipe whose length is given up front, so the store chooses between -one request and a multipart upload. Put-if-absent linearizes commits: no lease, no lock, no external -service. Uncontended, a commit is **three dependent requests**: the listing, the read of `N`'s header, -the creation; the data objects cost the commit nothing more. A handle that knows its version was -the latest less than half a retention window ago (§8.3) builds on it as it holds it, so its commit is -**one request**, the creation: a refusal is what tells it another writer went first, and it then asks -for the versions after the number it lost and rebases. Past that half window it lists first. Either -way it reads the version it created as it wrote it, with nothing read back. A handle whose commit -found nothing to change reads the version that commit was decided on, which is asked for first when -the batch was decided on the version held, since a no-op names the latest version. +A writer reads the latest commit `N`, uploads its data objects under fresh uid keys, in parallel, then +creates `commit/` (header, pages and fragments in one object) with the store's +put-if-absent, streamed from a pipe whose length is given up front, so the store can choose between +one request and a multipart upload. Put-if-absent linearizes commits with no lease, no lock and no +external service. + +Uncontended, a commit is three dependent requests: the listing, the read of `N`'s header, and the +creation. The data objects add nothing to the commit itself. A handle that knows its version was the +latest less than half a retention window ago (see [the latest +version](#83-the-latest-version-in-one-request)) builds on that version directly, so its commit is a +single request, the creation. A refusal tells it another writer went first, and it then asks for the +versions after the number it lost and rebases. Past that half window it lists first. Either way, it +reads the version it created as it wrote it, with nothing read back. A handle whose commit found +nothing to change reads the version that decision was made on, which is fetched first when the batch +was decided on the version it held, since a no-op names the latest version. ### 8.2 Rebase by re-applying operations -If `N+1` exists, the writer reads it and **re-applies its logical operations** to that version's trees -— add objects, add fragments, drop fragments, replace objects by a compaction's outputs — and never -merges two trees. Each operation carries its own answer to a moved ground: +If `N+1` already exists, the writer reads it and applies its logical operations again on that +version's trees (add objects, add fragments, drop fragments, replace objects by a compaction's +outputs), and never merges two trees. Each operation carries its own answer to a moved ground: | concurrent operations | result | |---|---| | append, append | both objects land in level 0 | -| append, compaction of level 0 | the compaction took a snapshot; the new object is not among its inputs and stays | +| append, compaction of level 0 | the compaction took a snapshot, so the new object is not among its inputs and stays | | indexer, indexer on the same range | the second finds the same fragment already there and writes nothing | -| indexer, compaction of that object | the object is gone; the fragment is dropped and never written | -| compaction, compaction on overlapping inputs | the second finds an input missing and abandons; its outputs are garbage | -| vacuum, anything | vacuum deletes only what no retained version references and what is older than the window (§10) | -| delete or update, append | the appended object is not among those the change read: it stays, and its rows are not touched | -| delete or update, anything that rewrites or marks an object it read | an input is gone or its entry changed: the replacement is abandoned, and the change is worked out again on the version that won (§12) | -| compaction, delete or update that marks one of its inputs | the entry the compaction read changed: it abandons, since its outputs would bring the marked rows back, and they are garbage | -| schema change, schema change | the second finds another schema than the one it was checked against, and is dropped (§13) | -| append, schema change | the object lands as it was written, and reads as the new schema like any earlier one (§13) | -| reader, anything | the reader holds a version; everything it references is immutable | - -An object's key ends with its uid, which makes it unique and a function of the object: two appenders -that read the same version would otherwise compute one key, and the second would replace the first's -entry, which an interleaving fuzzer caught. Because a commit header carries the pages it wrote and the -pages above them it read, a rebase iteration is also three dependent requests while the tree fits the -header. A writer that loses too many times (`DatasetOptions.MaxAttempts`) is told to use a -coordinator, in the exception's own words. +| indexer, compaction of that object | the object is gone, so the fragment is dropped and never written | +| compaction, compaction on overlapping inputs | the second finds an input missing and abandons, and its outputs are garbage | +| vacuum, anything | vacuum only deletes what no retained version references and what is older than the window ([lifecycle](#10-lifecycle)) | +| delete or update, append | the appended object is not among those the change read, so it stays and its rows are not touched | +| delete or update, anything that rewrites or marks an object it read | an input is gone or its entry changed, so the replacement is abandoned and the change is worked out again on the version that won ([deleting rows](#12-deleting-and-updating-rows)) | +| compaction, delete or update that marks one of its inputs | the entry the compaction read changed, so it abandons, since its outputs would bring the marked rows back, and they are garbage | +| schema change, schema change | the second finds a different schema from the one it was checked against, and is dropped ([schema evolution](#13-schema-evolution)) | +| append, schema change | the object lands as it was written, and reads as the new schema like any earlier one ([schema evolution](#13-schema-evolution)) | +| reader, anything | the reader holds a version, and everything it references is immutable | + +An object's key ends with its uid, which makes it unique and a function of the object. Without it, +two appenders that read the same version would compute the same key, and the second would replace the +first's entry, a case the interleaving fuzzer found. Because a commit header carries the pages it +wrote and the pages above them it read, one rebase iteration is also three dependent requests while +the tree fits in the header. A writer that loses too many times (`DatasetOptions.MaxAttempts`) is told +to use a coordinator, in the exception's own words. ### 8.3 The latest version in one request Commit keys sort newest first, so the first key a listing of `commit/` yields is the whole discovery, -and the listing stops after one page. That needs a strongly consistent listing, which S3 has had since -2020 and which §11 requires of any store. A reader that wants to read its own writes is handed the -version by its writer. There is no hint object. +and the listing stops after one page. That needs a strongly consistent listing, which S3 has offered +since 2020 and which [the store abstraction](#11-the-store-abstraction) requires of any store. A +reader that wants to read its own writes is handed the version by its writer. There is no hint +object. A handle that knows its version was the latest less than half a retention window ago asks for the -versions after it instead, a head request each: every one of them was created after that moment, and -vacuum takes a commit only a window after the next one superseded it, so they are all still there -and the first missing one ends the run. Past that half window the handle lists. Below the window the -commits do not run without a gap: vacuum keeps an old commit object whose pages a retained version -still names and takes the unnamed ones after it, so a handle that took the gap above its own version -for the end would read it as the latest, and commit on it. The handle's clock must therefore agree -with the store's to within the other half window, as a vacuum's already must. +versions after it instead, with a head request each. Every one of them was created after that moment, +and vacuum only takes a commit a full window after the next one superseded it, so they are all still +there and the first missing one ends the run. Past that half window, the handle lists. Below the +window the commits are not contiguous: vacuum keeps an old commit object whose pages a retained +version still names and removes the unnamed ones after it, so a handle that took the gap above its +own version for the end would read it as the latest and commit on it. The handle's clock therefore +has to agree with the store's to within the other half window, as a vacuum's already must. ## 9. The read-path budget ### 9.1 Dependent round trips, not bytes On an object store a request costs 10 to 100 ms whatever its size below a megabyte, so the budget is -**dependent round trips**. Cold, a point lookup on the clustering key through a sorted run: +counted in dependent round trips. Cold, a point lookup on the clustering key through a sorted run +costs: | step | dependent requests | cached by | |---|---|---| | list the latest commit, read its header | 2 | version | -| descend a level's tree | 0 up to ~650 objects, 1 up to ~400 000, 2 up to ~280 million | page, across versions: pages are immutable | +| descend a level's tree | 0 up to ~650 objects, 1 up to ~400 000, 2 up to ~280 million | page, across versions, since pages are immutable | | open the object's tail | 1, in parallel across the ≤ 8 + L objects touched | object | | its fences, then the two payload segments | 1 + 1 | run | -| the data segments | 1, coalesced | — | +| the data segments | 1, coalesced | none | -About **6 to 8 cold and 2 to 3 warm**, for any number of rows and objects. Latency does not choose the -structure; it sets the page cap, the inlining, the one-object commit, the in-process cache of -immutable headers, pages, tails and fences, and the parallel requests. Hedged requests belong to the -store library. +That is about 6 to 8 requests cold and 2 to 3 warm, for any number of rows and objects. Latency does +not choose the structure. It sets the page cap, the inlining, the one-object commit, the in-process +cache of immutable headers, pages, tails and fences, and the parallel requests. Hedged requests belong +to the store library. A scan in the tree's order opens the next object ahead once its consumer reads past the first batch -([16-queries.md](16-queries.md) §2.6): an object's tail and its first data request wait on each -other, not on the object before it, and a scan that stops at its first batch opens one object. +(see [the first batch of each source](16-queries.md#26-the-first-batch-of-each-source)), so an +object's tail and its first data request wait on each other rather than on the object before it, and +a scan that stops at its first batch opens a single object. ### 9.2 Held by a counting store -- **One file, any size**: opening a file and answering a count through a sorted run in fence pages, a - scan through a filter tree, a key seek and the roots costs the same **13 requests and 415 811 bytes** - on a sparse 1.31 GiB file and a sparse 13.1 GiB one (`ReadBudgetTests`). A content hash on the read +- For one file of any size, opening it and answering a count through a sorted run in fence pages, a + scan through a filter tree, a key seek and the roots costs the same 13 requests and 415 811 bytes on + a sparse 1.31 GiB file and on a sparse 13.1 GiB one (`ReadBudgetTests`). A content hash on the read path would turn that into 1 157 against 11 525 requests, and the test would fail. -- **Any number of objects**: a cold clustering-key lookup costs **2, 2 and 3** requests over datasets - of 1, 1 000 and 100 000 objects, and one object added costs **3, 3 and 4** dependent steps +- For any number of objects, a cold clustering-key lookup costs 2, 2 and 3 requests over datasets of + 1, 1 000 and 100 000 objects, and adding one object costs 3, 3 and 4 dependent steps (`DatasetBudgetTests`). -- **The critical path**: with 20 ms of injected latency, the lookup waits for its round trips one after +- On the critical path, with 20 ms of injected latency, the lookup waits for its round trips one after another, which no count of requests can show. ## 10. Lifecycle -- **Vacuum** (`VacuumAsync`, explicit, never automatic) marks everything the retained versions - reference — a subtree already seen is not walked again, so the versions cost their distinct pages — - then deletes the unmarked commit objects and data objects older than the **retention window**, - seven days by default (`DatasetOptions.RetentionWindow`), keeping the latest version always. What - ages is the moment a version was superseded, by the store's own clock. An unreferenced object - younger than the window is a writer in flight and is kept: **a writer must commit within the - window**. Only `commit/`, `data/` and the compaction loops' `leases/` are swept; an imported file - is its owner's. On a store under a retention lock, what the lock or a legal hold still keeps is left - and reported with the date the first of it may go (`DatasetOptions.LockedStore`, - [15-compaction.md](15-compaction.md) §5). A reader that - outlives the window gets an `ObjectNotFoundException` naming the version it was reading, never a - partial answer. A dry run reports without deleting. -- **Repack**: a commit object kept alive by a few live pages among dead ones is reported as sparse, and - a metadata-only commit copies those pages and fragments into itself, so the old object can go at the - next vacuum. -- **Verify** (`VerifyAsync`, offline) checks every object's length, content hash, rows and identity; - every fragment against its reference and with its object; every commit object whole; and it reads - the pages from the store, not from a header's inlined copies, which is the only way to see a torn - stored page a reader never reads. It reports every problem by name and throws none. Given an older - version, it walks the two trees' difference only, O(changed pages). -- **Rebuilding an index** drops an object's fragments and attaches one over the whole object in the - same commit, so no version holds the object with neither; rewriting the object, which a compaction - does, embeds it instead. +- Vacuum (`VacuumAsync`, explicit, never automatic) marks everything the retained versions reference + (a subtree already seen is not walked again, so the versions cost their distinct pages), then + deletes the unmarked commit objects and data objects older than the retention window, seven days by + default (`DatasetOptions.RetentionWindow`), always keeping the latest version. What ages is the + moment a version was superseded, by the store's own clock. An unreferenced object younger than the + window may belong to a writer in flight and is kept, which means a writer must commit within the + window. Only `commit/`, `data/` and the compaction loops' `leases/` are swept, and an imported file + belongs to its owner. On a store with retention locks, whatever the lock or a legal hold still keeps + is left and reported with the date the first of it may go (`DatasetOptions.LockedStore`, see + [stores that cannot delete](15-compaction.md#5-stores-that-cannot-delete)). A reader that outlives + the window gets an `ObjectNotFoundException` naming the version it was reading, never a partial + answer. A dry run reports without deleting. +- Repacking is internal for now. A commit object kept alive by a few live pages among dead ones is + detected as sparse during vacuum, and a metadata-only commit can copy those pages and fragments + into itself so the old object can go at the next vacuum, but neither the detection nor the repack + is exposed in the public API yet. +- Verification (`VerifyAsync`, offline) checks every object's length, content hash, rows and + identity, every fragment against its reference and its object, and every commit object whole. It + reads the pages from the store, not from a header's inlined copies, which is the only way to see a + torn stored page that a reader never reads. It reports every problem by name and throws none. Given + an older version, it only walks the difference between the two trees, O(changed pages). +- Rebuilding an index drops an object's fragments and attaches one over the whole object in the same + commit, so no version holds the object with neither. Rewriting the object, as a compaction does, + embeds the index instead. ## 11. The store abstraction @@ -522,195 +553,214 @@ other, not on the object before it, and a scan that stops at its first batch ope | operation | used by | |---|---| -| `GetRangeAsync(key, offset, length)` → bytes and the object's token, in one answer | every read; a data object's segment source plans its reads with the core's coalescer and issues them together | -| `HeadAsync(key)` → size, token, the store's creation time, and the retention date and legal hold of a store under a lock | opening; vacuum's ages, and what a lock still keeps | -| `PutIfAbsentAsync(key, pipe, length)` → created or exists | every write, streamed | -| `DeleteAsync(keys)` | vacuum, a batch at a time; an absent key is not an error | -| `ListAsync(prefix, startAfter)` | the latest version, one page (§8.3); vacuum | +| `GetRangeAsync(key, offset, length)` returns the bytes and the object's token in one answer | every read. A data object's segment source plans its reads with the core's coalescer and issues them together | +| `HeadAsync(key)` returns the size, the token, the store's creation time, and the retention date and legal hold on a locked store | opening, vacuum's ages, and what a lock still keeps | +| `PutIfAbsentAsync(key, pipe, length)` returns created or exists | every write, streamed | +| `DeleteAsync(keys)` | vacuum, a batch at a time, where an absent key is not an error | +| `ListAsync(prefix, startAfter)` | the latest version, in one page ([the latest version](#83-the-latest-version-in-one-request)), and vacuum | A store must document an atomic put-if-absent, a strongly consistent listing, a token that changes -whenever the bytes under a key change, and ranged reads on objects of any size. Three shapes belong to -the seam: a put **streams** and creates nothing when its content ends early or runs long; a delete -takes a batch and reports nothing; a listing pages by itself as it is enumerated. Retries, hedging, -connection pools and credentials are the store library's and never this one's. +whenever the bytes under a key change, and ranged reads on objects of any size. Three behaviours are +part of the interface: a put streams and creates nothing when its content ends early or runs long, a +delete takes a batch and reports nothing, and a listing pages by itself as it is enumerated. Retries, +hedging, connection pools and credentials belong to the store library, never to this one. -Three stores ship: `FileObjectStore`, whose put-if-absent is the kernel's create-new, and whose -content a crash may leave short, which the format detects; `MemoryObjectStore`, with injectable -latency, failures and crashes after a put; and `CountingObjectStore`, a decorator counting requests, -bytes and the **dependent steps** of the critical path. The contract is a test suite, -`ObjectStoreContractTests`, which an S3 library runs against its own store. +Three stores ship. `FileObjectStore` uses the kernel's create-new for put-if-absent, and a crash may +leave its content short, which the format detects. `MemoryObjectStore` has injectable latency, +failures and crashes after a put. `CountingObjectStore` is a decorator that counts requests, bytes and +the dependent steps of the critical path. The contract is a test suite, `ObjectStoreContractTests`, +which an S3 library runs against its own store. ## 12. Deleting and updating rows -**A delete marks rows, and rewrites no object it can mark.** A delete takes the rows its filter is -true for — a row it is false or unknown for stays, as a scan would not return it — and records them in -the leaf entry of each object that holds one: a **deletion vector**, the sorted runs of the object's -own row positions it takes, two varints a run, after everything else the entry holds (§4.2). The -object's bytes do not change, so the commit writes the pages on the path to the entries it changed and -no data object. An object every row of which goes is removed from the tree without being read, and one -whose summaries refute the filter is not opened. An update is a delete and an insert applied together: -the rows it takes are marked where they were, read as records, passed through the caller's function -and written into new objects of level 0, where appended rows go, because a changed row may carry -another key. One `ReplaceObjects` commits the whole change, so a reader sees all of it or none. - -**When a delete rewrites instead.** An object is rewritten without the rows, rather than marked, when -the marks would cost its reads more than they save the write: under `MarkedObjectBytes`, a mebibyte, -where a rewrite is a few requests and leaves the object exact; when the rows marked in it would pass an -eighth of its rows (`MarkedShare`), since every read of it steps over them; and when its vector would -pass 1 KiB (`MarkedVectorBytes`), a few hundred runs. The vector is counted as the runs it will form: -at a run a row when that already fits, and otherwise from the rows' places, read before anything is -written, since the rows a range takes are one run whatever their number. The vector lives in a leaf page: a commit that -marks a row in any object of a page writes the page again, so the bound weighs the rewrites the marks -save against the bytes they add to each of those commits; a commit that does not change the page -leaves it out of its header (§3). A vector past 256 bytes lies out of line instead: written once into -the commit object that made it, as a fragment is (§6.4), and named by its entry by a reference, so -the page carries 37 bytes for it whatever the marks. A rent of the object reads it beside the -object's open, where the version's pages lie or the handle kept it; vacuum keeps the commit object it -lies in, and a repack moves it. The rows a rewritten object keeps are a subset of -its rows, so its keys lie inside its old range: a level above 0 stays key-disjoint and the tree's order -holds; without a clustering key the rewritten object keeps the old one's position. Either way the rows -are judged by the scan's own evaluator, so a delete and a scan with the same filter agree on every row, -nulls included, and a rewrite checks that it kept exactly the rows the count left. - -**What a read does with a vector.** An entry's row count is its live rows, so a count without a filter -and every position stay arithmetic: a position maps to the object's own through the runs before it. -The rest is the object's own read with its marked rows left out, for a binary search over the runs per -batch or per step: +A delete marks rows, and does not rewrite any object it can mark. It takes the rows its filter is true +for (a row it is false or unknown for stays, as a scan would not return it) and records them in the +leaf entry of each object that holds one, as a deletion vector: the sorted runs of the object's own +row positions it takes, two varints per run, placed after everything else the entry holds (see [what a +node carries](#42-what-a-node-carries)). The object's bytes do not change, so the commit writes the +pages on the path to the entries it changed and no data object. An object whose rows all go is removed +from the tree without being read, and one whose summaries refute the filter is not opened. + +An update is a delete and an insert applied together. The rows it takes are marked where they were, +read as records, passed through the caller's function, and written into new level 0 objects, where +appended rows go, because a changed row may carry a different key. One `ReplaceObjects` commits the +whole change, so a reader sees all of it or none of it. + +### When a delete rewrites instead + +An object is rewritten without the rows, rather than marked, when the marks would cost its reads more +than they save the write. That happens under `MarkedObjectBytes` (a mebibyte), where a rewrite is a +few requests and leaves the object exact, when the rows marked in it would pass an eighth of its rows +(`MarkedShare`), since every read of it steps over them, and when its vector would pass 1 KiB +(`MarkedVectorBytes`), a few hundred runs. The vector is counted as the runs it will form: one run per +row when that already fits, and otherwise from the rows' positions, read before anything is written, +since the rows a range takes form one run whatever their number. + +The vector lives in a leaf page, so a commit that marks a row in any object of a page writes that page +again, and the bounds weigh the rewrites the marks save against the bytes they add to each of those +commits. A commit that does not change the page leaves it out of its header (see [objects and +keys](#3-objects-and-keys)). A vector past 256 bytes lies out of line instead: it is written once into +the commit object that made it, like a fragment ([fragments](#64-fragments-in-commit-objects-transient)), +and its entry names it by reference, so the page carries 37 bytes for it whatever the marks. A read of +the object fetches it alongside the object's open, from where the version's pages lie or from what the +handle kept. Vacuum keeps the commit object it lies in. + +The rows a rewritten object keeps are a subset of its rows, so its keys stay inside its old range: a +level above 0 stays key-disjoint and the tree's order holds, and without a clustering key the +rewritten object keeps the old one's position. Either way, rows are judged by the scan's own +evaluator, so a delete and a scan with the same filter agree on every row, nulls included, and a +rewrite checks that it kept exactly the rows the count left. + +### What a read does with a vector + +An entry's row count is its live rows, so a count without a filter and every position stay +arithmetic: a position maps to the object's own through the runs before it. The rest is the object's +own read with its marked rows left out, at the cost of a binary search over the runs per batch or per +step: | read | with marked rows | |---|---| -| a filtered count, an any | the object's count with the rows left out, tier by tier, in one pass: on a sorted column's slices, arithmetic; on an index's rows, a test of each; on a zone the maps decide whole, its rows less the marked ones; on a decode, the marked rows cleared before the count | -| rows in file order | a batch without a marked row as it came, renumbered; one with some, gathered without them; an aggregation's batches whole, the marked rows deselected, their columns still encoded | -| rows in key order, a key cursor | the key source steps over the marked rows' entries as it walks, and a rank takes away the entries marked below it: arithmetic when the ranks are rows, a sorted column's or the clustering run of a level above 0, and otherwise against the marked rows' keys, read by one take when the cursor opens | -| a minimum, a maximum | on a column the object's statistics say is sorted and holds no null, its first or last live row: the statistic while that row is the column's end, one row read once a delete took it; otherwise the object's extreme when a count finds a live row holding it, and its live rows read when none does | -| the summaries | an entry with marks keeps its bounds, which its live rows still lie inside, and drops what they no longer prove: exactness, a null count other than zero, a column left with no bound. They prune; they no longer answer | - -**Compaction folds the marks.** It reads each input's live rows and writes objects without marks, so -a vector lasts until its object is next compacted. A level no merge reaches, the top one above all, is -purged instead. An object whose marks reach half a delete's bounds, a sixteenth of its rows or half a -kilobyte of vector, is rewritten alone, in its level and inside its range, by a job of its own -(`CompactionTrigger.Marks`), the most marked first; the plan finds it by descending the shares and -vectors the tallies carry (§5.3). A delete that would pass the bounds rewrites the object itself only -when compaction lags. A compaction that read an object before a delete marked -rows in it would bring those rows back: its `ReplaceObjects` names the entries it read — uid, key and -vector — and is abandoned when one of them changed (§8.2). - -**What it costs.** Measured by the churn (§15) on ten million rows and thirty thousand commits of ten -rows at random keys: an update takes 1.9 to 2.9 ms and a delete 0.9 to 1.6 ms, where rewriting the -4 MiB objects that hold the rows takes 19 and 14 ms. A commit writes 12 KiB at first and 16 to 55 KiB -once the vectors have filled, where a rewrite writes 3.3 MiB: the rows' own objects, and a commit -object of 8 to 10 KiB throughout, whose leaf page carries a reference for each vector past 256 bytes. -Compaction, merges and purges together, writes 61 to 111 KiB a commit. After a thousand commits the store holds 137 MiB rather than 449 MiB, since the retention -window keeps every object a rewrite replaced (§10). The reads cost what they cost on the rewritten -dataset: a filter over every object makes the same 220 requests, and a point lookup, a seek, a count -under a key range and the largest key stay where they were. What the marks keep is space: up to an -eighth of an object's rows, until its compaction. - -**Concurrent writers.** The change is worked out on the version the handle holds. An append that lands -first adds objects the change never read, whose rows it does not touch; a commit that removes, rewrites -or marks an object the change read — a compaction, another delete — abandons the replacement (§8.2), -and the change is worked out again on the version that won, up to `MaxAttempts` times. What a lost -attempt wrote is left for vacuum. +| a filtered count, an any | the object's count with the rows left out, tier by tier, in one pass: arithmetic on a sorted column's slices, a test of each row on an index's rows, the zone's rows minus the marked ones on a zone the maps decide whole, and the marked rows cleared before counting on a decode | +| rows in file order | a batch without a marked row is delivered as it came, renumbered, and one with some is gathered without them. An aggregation's batches stay whole, with the marked rows deselected and their columns still encoded | +| rows in key order, a key cursor | the key source steps over the marked rows' entries as it walks, and a rank subtracts the entries marked below it. That is arithmetic when the ranks are rows (a sorted column's, or the clustering run of a level above 0), and otherwise uses the marked rows' keys, read by one take when the cursor opens | +| a minimum, a maximum | on a column the object's statistics say is sorted and that holds no null, its first or last live row: the statistic while that row is the column's end, and one row read once a delete took it. Otherwise the object's extreme when a count finds a live row holding it, and its live rows read when none does | +| the summaries | an entry with marks keeps its bounds, which its live rows still lie inside, and drops what they no longer prove: exactness, a null count other than zero, a column left with no bound. They prune, but no longer answer | + +### Compaction folds the marks + +Compaction reads each input's live rows and writes objects without marks, so a vector only lasts until +its object is next compacted. A level no merge reaches, the top one above all, is purged instead. An +object whose marks reach half a delete's limits (a sixteenth of its rows or half a kilobyte of vector) +is rewritten alone, in its level and inside its range, by a job of its own +(`CompactionTrigger.Marks`), the most marked first, and the plan finds it by descending the shares and +vectors the tallies carry (see [what a compaction does](#53-what-a-compaction-does)). A delete that +would pass the limits only rewrites the object itself when compaction lags. A compaction that read an +object before a delete marked rows in it would bring those rows back, so its `ReplaceObjects` names +the entries it read (uid, key and vector) and is abandoned when one of them changed (see +[rebase](#82-rebase-by-re-applying-operations)). + +### What it costs + +The churn bench ([how it is tested](#15-how-it-is-tested)) measures it on ten million rows and thirty +thousand commits of ten rows at random keys. An update takes 1.9 to 2.9 ms and a delete 0.9 to 1.6 ms, +where rewriting the 4 MiB objects that hold the rows takes 19 and 14 ms. A commit writes 12 KiB at +first and 16 to 55 KiB once the vectors have filled, where a rewrite writes 3.3 MiB (the rows' own +objects), with a commit object of 8 to 10 KiB throughout, whose leaf page carries a reference for each +vector past 256 bytes. Compaction, merges and purges together, writes 61 to 111 KiB per commit. After +a thousand commits the store holds 137 MiB rather than 449 MiB, since the retention window keeps every +object a rewrite replaced. Reads cost what they cost on the rewritten dataset: a filter over every +object makes the same 220 requests, and a point lookup, a seek, a count under a key range and the +largest key stay where they were. What the marks keep is space, up to an eighth of an object's rows, +until its next compaction. + +### Concurrent writers + +The change is worked out on the version the handle holds. An append that lands first adds objects the +change never read, and their rows are not touched. A commit that removes, rewrites or marks an object +the change read (a compaction, another delete) makes the replacement abandon (see +[rebase](#82-rebase-by-re-applying-operations)), and the change is worked out again on the version +that won, up to `MaxAttempts` times. Whatever a lost attempt wrote is left for vacuum. ## 13. Schema evolution A change of schema is a commit that changes the header's schema and rewrites no object. Per top-level -column, a change is one of: added, nullable; dropped, unless the clustering key holds it; renamed; -widened within its kind, a signed integer to a wider signed one, an unsigned to a wider unsigned, a -float to a wider float; made nullable. Each keeps every value already written readable, and anything -else is refused before the commit. The clustering key's columns keep their names and types in every -object: they order the objects and the trees. - -**Names are identities.** An object's field belongs to the column of its name, or to the column a -rename moved that name to. The header records every name a column gave up, with the column carrying it -now, or none when it was dropped, and no column takes one of them again. That is what makes names -enough: an object written while a name was in use never lends its values to another column, and the -summaries its entry carries, keyed by the names it had, never describe another column. A summary under -a former name prunes nothing for the new one, which is conservative and ends at the next rewrite. - -**Reading an object of an earlier schema.** When an object's dtype is the version's — every object -written since the last change — its read is its own scan, after one comparison of dtypes -whose hashes are cached. Otherwise its columns are mapped when it opens, and each batch its own scan -delivers is reshaped in the batch's arena: a struct over its fields in the dataset's order, a missing -column filled with nulls, a column written non-nullable declared nullable over the same buffers, -narrower numbers widened into a buffer of the arena. The filter goes down to the object in its own -terms: renamed, and each predicate over a column the object lacks folded to what a null makes it, a -comparison unknown and `IS NULL` true, so that a filter over an added column refutes every earlier -object without a read. Under a negation an unknown is not false; a predicate that cannot be stated -over the object's columns there is evaluated on the reshaped batches instead. Key cursors, key-ordered -reads, compaction, deletes and updates read through the same mapping, and an object that lacks a -cursor's column holds no entry. - -**Compaction folds the change into the data**, as it folds indexes (§2): every object it writes is in -the current schema. Until then, reading an earlier object costs the reshaping — records per batch, and -a copy for a widened column — and reading any other costs nothing more. - -**Concurrent changes.** The operation carries the schema it was checked against. A rebase onto a -version whose schema is another drops it, since what it checked no longer holds, and the caller is told -to refresh; one that finds its own schema already there is done. An append prepared under the earlier -schema lands after the change, its object read as the new schema like any other. +column, a change is one of the following: a nullable column added, a column dropped (unless the +clustering key uses it), a column renamed, a number widened within its kind (a signed integer to a +wider signed one, an unsigned to a wider unsigned, a float to a wider float), or a column made +nullable. Each keeps every value already written readable, and anything else is refused before the +commit. The clustering key's columns keep their names and types in every object, since they order the +objects and the trees. + +Names are identities. An object's field belongs to the column with its name, or to the column a +rename moved that name to. The header records every name a column gave up, along with the column +that carries it now, or none when it was dropped, and no column ever takes one of those names again. +That is what makes names enough: an object written while a name was in use never lends its values to +another column, and the summaries its entry carries, keyed by the names it had, never describe +another column. A summary under a former name prunes nothing for the new one, which is conservative +and ends at the next rewrite. + +When an object's dtype is the version's, as for every object written since the last change, its read +is its own scan, after one comparison of dtypes whose hashes are cached. Otherwise its columns are +mapped when it opens, and each batch its own scan delivers is reshaped in the batch's arena: a struct +over its fields in the dataset's order, a missing column filled with nulls, a column written +non-nullable declared nullable over the same buffers, and narrower numbers widened into a buffer of +the arena. The filter reaches the object in its own terms. It is renamed, and each predicate over a +column the object lacks is folded to what a null makes it (a comparison unknown, `IS NULL` true), so a +filter over an added column refutes every earlier object without a read. Under a negation an unknown +is not false, so a predicate that cannot be stated over the object's columns there is evaluated on the +reshaped batches instead. Key cursors, key-ordered reads, compaction, deletes and updates read through +the same mapping, and an object that lacks a cursor's column holds no entry. + +Compaction folds the change into the data, as it folds indexes ([principles](#2-principles)): every +object it writes is in the current schema. Until then, reading an earlier object costs the reshaping +(records per batch, and a copy for a widened column), and reading any other object costs nothing +more. + +The operation carries the schema it was checked against. A rebase onto a version with a different +schema drops it, since what it checked no longer holds, and the caller is told to refresh, while one +that finds its own schema already there is done. An append prepared under the earlier schema lands +after the change, and its object reads as the new schema like any other. ## 14. Alternatives weighed | question | chosen | rejected, and why | |---|---|---| -| metadata structure | a prolly tree per level | a flat manifest per version (O(N) per open and per commit); checkpoint and deltas (the checkpoint is still the list); a manifest list of manifests, as Iceberg (fine to 10⁶ files, not a bound); an external catalog (a service dependency) | -| prolly or B+tree | prolly | same lookup and commit costs; the prolly adds history independence, deduplication across writers, a one-hash equality of datasets, an incremental diff across lineages, and a rebuild-from-scratch oracle, for a chunker of about a hundred lines | -| the latest version | newest-first keys and one listing | a hint object and probes (O(lag), a mutable object); a pointer updated by compare-and-swap (one more mutable thing) | -| concurrency | put-if-absent, rebase by operations | leases and lock files; three-way merges of trees | -| data across objects | leveled on a clustering key, tiered without | no compaction (O(appends)); a global secondary index (rewritten at every compaction) | -| index placement | in the file, external fragments transient | in-file only (adding an index rewrites data); external only (the index no longer travels with the file) | -| deleting rows | a deletion vector in the leaf entry, bounded, folded by compaction | copy on write of every object a row is taken from (an object per delete, 3 MiB a commit of ten rows, each kept through the retention window); positional delete files, as Iceberg (a read joins them, and they pile up until a rewrite); equality deletes (a read evaluates each of them against every row) | -| binding | uid and store token; the writer's content hash for verify | a content hash on the read path (reads the data) | +| metadata structure | a prolly tree per level | a flat manifest per version (O(N) per open and per commit), a checkpoint plus deltas (the checkpoint is still the list), a manifest list of manifests as in Iceberg (fine up to 10⁶ files, but not a bound), an external catalog (a service dependency) | +| prolly or B+tree | prolly | the same lookup and commit costs, but the prolly tree adds history independence, deduplication across writers, a one-hash equality of datasets, an incremental diff across lineages, and a rebuild-from-scratch oracle, for a chunker of about a hundred lines | +| the latest version | newest-first keys and one listing | a hint object and probes (O(lag), and a mutable object), a pointer updated by compare-and-swap (one more mutable thing) | +| concurrency | put-if-absent, rebase by operations | leases and lock files, three-way merges of trees | +| data across objects | leveled on a clustering key, tiered without | no compaction (O(appends)), a global secondary index (rewritten at every compaction) | +| index placement | in the file, with transient external fragments | in-file only (adding an index rewrites data), external only (the index no longer travels with the file) | +| deleting rows | a bounded deletion vector in the leaf entry, folded by compaction | copy-on-write of every object a row is taken from (an object per delete, 3 MiB per commit of ten rows, each kept through the retention window), positional delete files as in Iceberg (a read joins them, and they pile up until a rewrite), equality deletes (a read evaluates each of them against every row) | +| binding | uid and store token, with the writer's content hash for verification | a content hash on the read path (it reads the data) | | metadata objects | one commit object per version | an object per page or per pack (sequential reads, orphans after a crash, two creations per commit) | -| Iceberg or Delta as the table layer | — | their data formats are Parquet, ORC and Avro; a Vortex data file would make the table unreadable to every engine that reads them, which is the only reason to adopt them | +| Iceberg or Delta as the table layer | neither | their data formats are Parquet, ORC and Avro, and a Vortex data file would make the table unreadable to every engine that reads them, which is the only reason to adopt them | -**Not in this design:** a clustering key picked automatically, since it decides the write amplification -and is the user's to state; evolving the fields of a nested struct, or a column's type otherwise than -by widening a number or making it nullable (§13). +Not in this design: a clustering key picked automatically, since it decides the write amplification +and is the user's to state, and evolving the fields of a nested struct, or changing a column's type +other than by widening a number or making it nullable ([schema evolution](#13-schema-evolution)). ## 15. How it is tested -- **The counting matrix** of §9.2. -- **An interleaving fuzzer** over the in-memory store: readers, writers, indexers, compactions and - vacuums in seeded schedules, commits prepared on a stale version and crashes after a put; every read +- The counting matrix of [the read-path budget](#92-held-by-a-counting-store). +- An interleaving fuzzer over the in-memory store: readers, writers, indexers, compactions and vacuums + in seeded schedules, with commits prepared on a stale version and crashes after a put. Every read must equal a scan with indexes and summaries off on the same version, and every version a vacuum retains must verify whole. -- **The rebase matrix**, every row of §8.2. -- **The invariant** of §5.2 after every step of a randomized append stream drained at random moments. -- **Progressive indexing**: at every intermediate commit, the answers equal those without indexes. -- **The tree's oracles**: incremental edits equal a rebuild from scratch, byte for byte, and the same +- The rebase matrix, covering every row of the [rebase table](#82-rebase-by-re-applying-operations). +- [The invariant](#52-the-invariant), checked after every step of a randomized append stream drained + at random moments. +- Progressive indexing: at every intermediate commit, the answers equal those without indexes. +- The tree's oracles: incremental edits equal a rebuild from scratch, byte for byte, and the same operations in two orders give one root hash. -- **Order**: key-ordered reads across levels equal a sort of the rows, in both directions, with and - without summaries; a key cursor's walk down is its walk up reversed, a step against its direction +- Order: key-ordered reads across levels equal a sort of the rows, in both directions, with and + without summaries. A key cursor's walk down is its walk up reversed, a step against its direction lands next to the entry before it, and its seeks below a key land where one sorted file's would. -- **Deletes and updates**: the rows left equal the rows written filtered in C# under three-valued - logic, across levels and without a clustering key, and a delete that loses its objects to a - compaction is worked out again. -- **The plan**: a descent chooses the job a read of every leaf chooses, on random shapes and options - that meet every trigger; every page's tally is the one its leaves add up to; a cold plan asks for +- Deletes and updates: the rows left equal the rows written, filtered in C# under three-valued logic, + across levels and without a clustering key, and a delete that loses its objects to a compaction is + worked out again. +- The plan: a descent chooses the job a read of every leaf chooses, on random shapes and options that + reach every trigger, every page's tally is the one its leaves add up to, and a cold plan asks for the same one or two pages over 3 000, 31 000 and 125 000 objects. -- **Marks**: a dataset that marks reads as a second one taking the same changes by rewrites, through - every read — rows by position, rows in key order both ways with null keys around the marks, key - cursors, their ranks and seeks, counts, extremes, aggregates — under a seeded fuzzer of deletes and - updates; a compaction that read an object before rows were marked in it is abandoned; a count told - to leave rows out counts none of them on any tier, each switched off in turn; an object whose marks - are due by share or by vector is purged alone in its level; a range too long for a vector's worst - case is marked as the run it is, an update's rows written once. -- **Schema evolution**: objects of each earlier schema read as the current one through scans, counts, +- Marks: a dataset that marks reads like a second one taking the same changes by rewrites, through + every read (rows by position, rows in key order both ways with null keys around the marks, key + cursors with their ranks and seeks, counts, extremes, aggregates), under a seeded fuzzer of deletes + and updates. A compaction that read an object before rows were marked in it is abandoned, a count + told to leave rows out counts none of them on any tier, each switched off in turn, an object whose + marks are due by share or by vector is purged alone in its level, and a range too long for a + vector's worst case is marked as the single run it is, with an update's rows written once. +- Schema evolution: objects of each earlier schema read as the current one through scans, counts, extremes, key order, key cursors, deletes, updates and compaction, filters over added columns - included; every change that would lose a value is refused. -- **Tampering**: an object replaced at equal size, a page, a root and a fragment torn at every byte, a - fragment of another object: readers answer exactly, and verify names each. -- **Rust**: compaction outputs are among the files the cross-check hands to Rust 0.86.1. -- **The churn** (`bench/Vorticity.Benchmarks.Churn`): ten million rows, then thirty thousand commits of - ten rows — appends, updates and deletes at random keys — with compaction drained and vacuum run - between them. Every cost stays flat, or levels off once the marks have filled: an append 0.5 to - 0.7 ms, an update 1.9 to 2.9 ms, a delete 0.9 to 1.6 ms, five to six requests an operation with - its commit's one among them, compaction about a millisecond a commit, a commit's bytes 12 KiB and - then 16 to 55 KiB, of which its commit object is 8 to 10 KiB; a point lookup 0.7 to 1.0 ms in three + included, and every change that would lose a value is refused. +- Tampering: an object replaced at equal size, and a page, a root and a fragment torn at every byte, or + a fragment of another object. Readers answer exactly, and verification names each one. +- Rust: compaction outputs are among the files the cross-check hands to Rust 0.86.1. +- The churn bench (`bench/Vorticity.Benchmarks.Churn`): ten million rows, then thirty thousand commits + of ten rows (appends, updates and deletes at random keys), with compaction drained and vacuum run + between them. Every cost stays flat, or levels off once the marks have filled: an append takes 0.5 + to 0.7 ms, an update 1.9 to 2.9 ms and a delete 0.9 to 1.6 ms, each making five to six requests, one + of them its commit. Compaction costs about a millisecond per commit, and a commit writes 12 KiB, then + 16 to 55 KiB, of which its commit object is 8 to 10 KiB. A point lookup takes 0.7 to 1.0 ms in three to four requests, a seek and ten steps either way 0.3 to 0.7 ms, a hundred rows by position 0.2 to - 0.4 ms, a scan of every row 30 ms; the heap between 60 and 92 MiB. Rewriting instead of marking, an update takes 19 ms and - a commit writes 3.3 MiB. Before the level-0 destination, the cap on an object and the chunked buffer, - an update cost 820 ms, compaction 100 ms a commit and the heap grew to 2.2 GiB over the first 150 - commits. + 0.4 ms, and a scan of every row 30 ms, with the heap between 60 and 92 MiB. Rewriting instead of + marking, an update takes 19 ms and a commit writes 3.3 MiB. diff --git a/docs/design/14-public-api.md b/docs/design/14-public-api.md index c5ec5cc6..ee6cabaf 100644 --- a/docs/design/14-public-api.md +++ b/docs/design/14-public-api.md @@ -1,107 +1,110 @@ -# The public surface +# The public API -Why the surface a caller programs against has the shape it has, and where each part lives. The -complete listing of every public type and member is +Why the API a caller programs against has the shape it has, and where each part lives. The complete +listing of every public type and member is [tests/Vorticity.Tests/Api/PublicSurface.txt](../../tests/Vorticity.Tests/Api/PublicSurface.txt), -which `PublicSurfaceTests` holds equal to the assemblies; the XML documentation is in the source; the -[guide](../guide/README.md) shows each use in a program that runs. This document does not repeat -them: it gives the rules they follow and the decisions behind them. The engine under the surface is -[03-architecture.md](03-architecture.md), the type mapping [07-dotnet-mapping.md](07-dotnet-mapping.md), -the meaning of a predicate [08-semantics.md](08-semantics.md). +which `PublicSurfaceTests` keeps equal to the assemblies. The XML documentation is in the source, and +the [guide](../guide/README.md) shows each use in a program that runs. This document does not repeat +them. It gives the rules they follow and the decisions behind them. The engine under the API is +described in [03-architecture.md](03-architecture.md), the type mapping in +[07-dotnet-mapping.md](07-dotnet-mapping.md), and the meaning of a predicate in +[08-semantics.md](08-semantics.md). Seven rules decide every signature: -1. **A type is a schema, not a row.** A record declares which columns exist and how they are typed; - reading it yields columns. Rows are a sink a caller asks for, and pays for. A query's result of - several values is a record's schema too: the record a scan reads from, the record a result is +1. A type is a schema, not a row. A record declares which columns exist and how they are typed, and + reading it yields columns. Rows are a sink a caller asks for, and pays for. A query result made of + several values is a record's schema too: the record a scan reads from, and the record a result is read as. -2. **The compiler holds the lifetimes.** Everything borrowed from a scan is a `ref struct`: no - `Dispose` on the hot path, nothing to forget. -3. **The plan lives under the API.** A query is described by ordinary lambdas over a symbolic probe, - run once, in the shape of LINQ: its method names, and its query syntax. What cannot be pushed - down does not compile. No expression tree, no string column name and no untyped value on the +2. The compiler enforces the lifetimes. Everything borrowed from a scan is a `ref struct`, so there is + no `Dispose` on the hot path and nothing to forget. +3. The plan lives under the API. A query is described by ordinary lambdas over a symbolic probe, run + once, in the shape of LINQ (its method names and its query syntax). What cannot be pushed down + does not compile. There is no expression tree, no string column name and no untyped value on the typed path. -4. **Generic, specialized, BCL first**: `INumber`, static abstract members, intrinsics, - `System.IO.Pipelines`, `System.Diagnostics.Metrics`; nothing virtual per value. -5. **A session, not a static.** Pools, caches, I/O concurrency, parallelism and extension registries - live on a `VortexSession`. -6. **`Async` names what runs.** A method that starts the work — an awaitable, or a stream that ends - the query — ends in `Async`: `CountAsync`, `AverageAsync`, `ToRecordsAsync`. A method that only +4. Generic, specialized, BCL first: `INumber`, static abstract members, intrinsics, + `System.IO.Pipelines`, `System.Diagnostics.Metrics`, and nothing virtual per value. +5. A session, not a static. Pools, caches, I/O concurrency, parallelism and extension registries live + on a `VortexSession`. +6. `Async` names what runs. A method that starts the work (an awaitable, or a stream that ends the + query) ends in `Async`: `CountAsync`, `AverageAsync`, `ToRecordsAsync`. A method that only describes the query does not: `Where`, `GroupBy`, `Select`, `Take`, whose result is run by - `await foreach`. A builder never reads: what it checks is in the schema the open read. -7. **Batches flow.** Every source, operator and sink exchanges batches, pulled by its consumer and - decoded ahead within a bounded window. A query answers as soon as its first batch is final, holds - what is still open rather than what it read, stops reading when its consumer stops, and - allocates nothing per batch; a result is a stream of batches like a file's, and a scan of its - own. Time to first batch, peak memory and allocations are promises with gates (§9). - -What the surface is not: a DataFrame, a SQL engine, a join, a bridge to another columnar library. A -query over one table has the shape of LINQ ([16-queries.md](16-queries.md)); what is not one, a -`join` or a `let`, does not compile. An owned `RecordBatch` is where a bridge would attach, and the -plan a scan builds is what a query provider would target. + `await foreach`. A builder never reads, and what it checks is in the schema the open already read. +7. Batches flow. Every source, operator and sink exchanges batches, pulled by its consumer and decoded + ahead within a bounded window. A query answers as soon as its first batch is final, holds what is + still open rather than what it read, stops reading when its consumer stops, and allocates nothing + per batch. A result is a stream of batches like a file's, and a scan in its own right. Time to + first batch, peak memory and allocations are promises backed by gates (see [the performance + contract](#9-the-performance-contract-and-the-gates-that-hold-it)). + +The API is not a DataFrame, a SQL engine, a join, or a bridge to another columnar library. A query +over one table has the shape of LINQ ([16-queries.md](16-queries.md)), and what is not part of that +shape, a `join` or a `let`, does not compile. An owned `RecordBatch` is where a bridge would attach, +and the plan a scan builds is what a query provider would target. ## 1. Packages and namespaces | package | what it holds | depends on | |---|---|---| -| `Vorticity` | the session, files, the writer, schema, the typed scan and its columns, the tool scan, the I/O seam, options, plans, exceptions | `System.IO.Hashing`, `Vorticity.Zstd` | +| `Vorticity` | the session, files, the writer, schema, the typed scan and its columns, the tool scan, the I/O interface, options, plans, exceptions | `System.IO.Hashing`, `Vorticity.Zstd` | | `Vorticity.Generators` | the `[VortexRecord]` generator and the analyzers | build time only | | `Vorticity.Dataset` | the dataset over an object store, `[Experimental("VX0001")]` | `Vorticity` | | `Vorticity.RowEncoding` | the row encoding, `[Experimental("VX0002")]` | `Vorticity` | | `Vorticity.Zstd` | Zstandard compression and decompression in managed C#: `ZstdCompressor`, `ZstdDecompressor` | `System.IO.Hashing` | -The core has two public namespaces: `Vorticity`, for everything a caller uses, and -`Vorticity.IO`, for the segment seam. The attributes the generator reads live in the core, so a -record can implement its interface by hand without referencing the generator. +The core has two public namespaces: `Vorticity`, for everything a caller uses, and `Vorticity.IO`, +for the segment interface. The attributes the generator reads live in the core, so a record can +implement its interface by hand without referencing the generator. ## 2. The session -`VortexSession` (`Session/`) owns what must not be global: +`VortexSession` (`Session/`) owns everything that must not be global: | owned | why | |---|---| | the memory pool, 64-byte aligned by default (`AlignedMemoryPool`) | every batch, segment and builder buffer comes from it, and disposing the session returns everything | | the segment cache | one budget for every file of the session, for sources that do I/O | | the bound on reads in flight, `MaxConcurrentReads` | an object store sees at most that many requests from the process | -| the degree of parallelism, `MaxDegreeOfParallelism`, 1 by default | where every scan and writer of the session starts ([09-contracts.md](09-contracts.md) §2) | -| the index cache, `IndexCacheBytes` | what each open file keeps of decoded index runs | +| the degree of parallelism, `MaxDegreeOfParallelism`, 1 by default | where every scan and writer of the session starts (see [parallelism](09-contracts.md#2-parallelism)) | +| the query memory and scratch budgets, `MemoryBudget` and `ScratchBudget` | what the session's group bys may hold in memory and spill to disk, shared with every session given the same budget | +| the index cache, `IndexCacheBytes` | how much of the decoded index runs each open file keeps | | the files kept mapped once closed, `MappedFileCacheCount` | opening a file again takes over its mapping and every page already mapped in it | -| whether a path is mapped at all, `MapFiles`, true by default | a service that reads files others may truncate reads them positionally, so that a cut file fails a read rather than the process | -| the extension registry | dtype ids this process knows beyond the editions, so a registered type binds to a record member or a `Column`; binding checks the storage type the registration declares | +| whether a path is mapped at all, `MapFiles`, true by default | a service that reads files others may truncate reads them positionally, so a truncated file fails a read rather than the process | +| the extension registry | dtype ids this process knows beyond the editions, so a registered type binds to a record member or a `Column`. Binding checks the storage type the registration declares | `VortexSession.Default` is immutable and is what `VortexFile.OpenAsync(path)` uses. A session is thread-safe and meant to be shared by every request of a process, and its options are frozen when it is created. It opens files from a path or from any `ISegmentSource`, creates writers over a path or a -`PipeWriter`, and appends; how a path is read is [03-architecture.md](03-architecture.md) §3.5. +`PipeWriter`, and appends. How a path is read is described in [I/O](03-architecture.md#35-io). ## 3. Schema and types The schema is a value: `VortexSchema`, a list of `VortexField(Name, VortexType)` that a collection expression builds, and `VortexType`, immutable and interned by value, with a factory per dtype -(`Schema/`). `Map`, `Union` and `Variant` have no factory: a file may declare them, so the schema -shows them, but no .NET type maps to them. The arena the reader decodes into is built once at open -and never appears. +(`Schema/`), `Map(key, value)` included. `Union` and `Variant` have no factory: a file may declare +them, so the schema shows them, but no .NET type maps to them. The arena the reader decodes into is +built once at open and never appears in the API. ## 4. Records: the generated contract -A `[VortexRecord]` type — a `partial` record struct, record class, struct or class — declares -columns by its public members, in declaration order and under their names; `[VortexColumn]` renames -one and carries what a .NET type does not say (a decimal's precision and scale, a time unit, a time -zone), and `[VortexIgnore]` skips one. The generator (`src/Vorticity.Generators/RecordGenerator.cs`) -implements `IVortexRecord` — the schema, and a column-at-a-time read and write of rows — and +A `[VortexRecord]` type (a `partial` record struct, record class, struct or class) declares columns +through its public members, in declaration order and under their names. `[VortexColumn]` renames one +and carries what a .NET type does not say (a decimal's precision and scale, a time unit, a time zone), +and `[VortexIgnore]` skips one. The generator (`src/Vorticity.Generators/RecordGenerator.cs`) +implements `IVortexRecord` (the schema, and a column-at-a-time read and write of rows) and emits extension members on the probe, the columns and the builder, so that `r.Day`, `batch.Day` and -`builder.Day` are typed properties. Everything it emits can be written by hand +`builder.Day` are typed properties. Everything it emits can also be written by hand ([records.md](../guide/records.md)). -A record binds to a file once, at the first sink of a scan or at `CreateWriter`: each member +A record binds to a file once, at the first sink of a scan or at `CreateWriter`. Each member matches a column by exact name, then by a unique case-insensitive match, and a missing column, an -ambiguous match or a type that does not fit throws `VortexSchemaException` naming both. A file with -more columns than the record is fine: **a scan reads the columns its query names** — the record's -members for batches and records, the elements of a `Select`, the keys and inputs of an aggregate — -so the record is the projection of a scan that delivers it. Where the mapping leaves room, -the generator decides: an `enum` is its underlying integer on the columns and an enum symbol on the -probe; a `DateTimeOffset` member without a declared zone is UTC; a list of records and a nullable -registered extension are refused with VX1005. +ambiguous match or a type that does not fit throws a `VortexSchemaException` naming both. A file with +more columns than the record is fine, because a scan only reads the columns its query names (the +record's members for batches and records, the elements of a `Select`, the keys and inputs of an +aggregate), so the record is the projection of a scan that delivers it. Where the mapping leaves +room, the generator decides: an `enum` is its underlying integer in the columns and an enum symbol on +the probe, a `DateTimeOffset` member without a declared zone is UTC, a list of records is a list of +structs, and a nullable registered extension is refused with VX1005. ## 5. Reading @@ -110,68 +113,69 @@ registered extension are refused with VX1005. `file.Scan()` reads the columns the record names and nothing else. `Where` adds a predicate, `Rows` a range or a set of row indices, `OrderBy` or `OrderByDescending` a key order, and `With(ScanOptions)` the scan's batch cap, prefetch, degree, compaction and switches -(`Scan/Scan.Typed.cs`, `Scan/ScanOptions.cs`); `Select` and `GroupBy` turn it into a query of values -or of groups (§5.5). A -scan is consumed with `await foreach`, whatever the source: there is no synchronous enumeration. The -batch is **borrowed** — valid until the next `MoveNextAsync`, the contract of a `PipeReader`'s buffer — -and the loop body runs synchronously between two batches, so a `ref struct` in it is legal. Under -the loop, the decode of the next batch overlaps the caller's work on the current one, by `Prefetch` -batches, with buffers that alternate rather than accumulate. +(`Scan/Scan.Typed.cs`, `Scan/ScanOptions.cs`). `Select` and `GroupBy` turn it into a query of values +or of groups (see [queries](#55-queries-select-group-aggregate)). A scan is consumed with +`await foreach`, whatever the source, and there is no synchronous enumeration. The batch is borrowed, +valid until the next `MoveNextAsync`, which is the contract of a `PipeReader`'s buffer, and the loop +body runs synchronously between two batches, so a `ref struct` inside it is legal. Under the loop, +decoding the next batch overlaps the caller's work on the current one, by `Prefetch` batches, with +buffers that alternate rather than accumulate. ### 5.2 The symbolic algebra The lambda given to `Where` has the syntax of LINQ and none of its machinery (`Symbolic/`). `Probe` has a `Sym` per member, whose operators record a predicate instead of evaluating -one; the lambda runs once, when the scan is built, and its result is the plan. **What compiles is -exactly what is pushed down**: +one. The lambda runs once, when the scan is built, and its result is the plan. What compiles is +exactly what is pushed down: | written | meaning | |---|---| | `r.Day >= 900`, `r.Day.In(1, 2, 3)` | comparison and membership, against a literal of the column's type | -| `r.Celsius > 45.0 && r.City == "Paris"` | conjunction; `\|\|` and `!` likewise | -| `r.Celsius == null`, `r.Celsius.IsNull` | nullity, never unknown ([08-semantics.md](08-semantics.md) §3) | -| `r.City.StartsWith("Par")`, `r.City.Like("P_r%")`, `r.Pages.Contains(42)` | text and list predicates ([12-index-reads.md](12-index-reads.md) §6) | -| `r.High > r.Low` | column against column of one type | -| `r.At.Truncate(CalendarUnit.Day) == day`, `r.Price.Bucket(10m) >= 100m` | a function of a column; non-decreasing, so the zone maps still prune ([16-queries.md](16-queries.md) §3) | -| `r.Day >= "900"`, `r.Day % 2 == 0`, `Math.Abs(r.Celsius) > 1` | **does not compile**: a literal of the wrong type, or no pushdown | - -The same symbols are every lambda's: the keys of a `GroupBy`, the inputs of an aggregate, the -elements of a `Select`; and the results of a group compare in the `Where` that follows a `GroupBy` -as columns do in a filter ([16-queries.md](16-queries.md)). - -A captured local is read once, when the lambda runs, and an optional filter is two `Where` calls, -since the second joins with `AND`. A literal is compared **exactly**, never rounded into the column's -unit: a `DateTime` finer than a timestamp column's unit lies strictly between two stored values, so -`==` holds for no row and an ordering keeps or drops the boundary row correctly; a decimal literal -follows the same rule against the column's scale. A breakpoint inside the lambda sees symbols, not -values, and hits once. +| `r.Celsius > 45.0 && r.City == "Paris"` | conjunction, and `\|\|` and `!` likewise | +| `r.Celsius == null`, `r.Celsius.IsNull` | nullity, never unknown (see [three-valued logic](08-semantics.md#3-three-valued-logic)) | +| `r.City.StartsWith("Par")`, `r.City.Like("P_r%")`, `r.Pages.Contains(42)` | text and list predicates (see [predicates on text and lists](12-index-reads.md#6-predicates-on-text-and-lists)) | +| `r.High > r.Low` | one column against another of the same type | +| `r.At.Truncate(CalendarUnit.Day) == day`, `r.Celsius.Bucket(5.0) >= 40.0` | a function of a column, non-decreasing, so the zone maps still prune (see [value expressions](16-queries.md#3-value-expressions)) | +| `r.Day >= "900"`, `r.Day % 2 == 0`, `Math.Abs(r.Celsius) > 1` | does not compile: a literal of the wrong type, or no pushdown | + +The same symbols serve every lambda: the keys of a `GroupBy`, the inputs of an aggregate, the +elements of a `Select`. The results of a group compare in the `Where` that follows a `GroupBy` the +way columns do in a filter ([16-queries.md](16-queries.md)). + +A captured local is read once, when the lambda runs, and an optional filter is just two `Where` +calls, since the second joins with `AND`. A literal is compared exactly, never rounded into the +column's unit. A `DateTime` finer than a timestamp column's unit lies strictly between two stored +values, so `==` holds for no row, and an ordering keeps or drops the boundary row correctly. A decimal +literal follows the same rule against the column's scale. A breakpoint inside the lambda sees +symbols, not values, and is hit once. ### 5.3 `Columns` and `Column` A batch is a `Columns`, with a typed property per member and a deconstruction, and each -column a `Column` (`Columns/`): `Values` as a span for fixed-width types, a span per value for text -and binary, a range per row for a list, and the accessors of [07-dotnet-mapping.md](07-dotnet-mapping.md) -§1. `Values` is **64-byte aligned** whatever the file's segment alignment: a buffer laid out on a -smaller boundary is copied once into an aligned block at its first read. On a nullable column `Values` -is the raw buffer, meaningless where the validity bit is 0, and `ValidityWords` gives the validity as -64-bit words; the indexer returns `T?` at the cost of a branch per value, and says so. `StartRow` is -the batch's first row in the file, and `Selection` the rows a filtered or taken block keeps when it is -delivered whole (§5.4). +column is a `Column` (`Columns/`): `Values` as a span for fixed-width types, a span per value for +text and binary, a range per row for a list, and the accessors listed in [the +mapping](07-dotnet-mapping.md#1-the-mapping). `Values` is 64-byte aligned whatever the file's segment +alignment, because a buffer laid out on a smaller boundary is copied once into an aligned block at +its first read. On a nullable column, `Values` is the raw buffer, meaningless where the validity bit +is 0, and `ValidityWords` gives the validity as 64-bit words. The indexer returns `T?` at the cost of +a branch per value, and says so. `StartRow` is the batch's first row in the file, and `Selection` the +rows a filtered or taken block keeps when it is delivered whole (see [encoded views and the +selection](#54-encoded-views-and-the-selection)). ### 5.4 Encoded views and the selection A column says what its block holds before any decode (`Encoding`). A dictionary block has not been -decoded: `Values` forces the decode, while `AsDictionary()` gives its codes and values, so a consumer -that groups by it groups by code and never touches its strings; `AsRunEnd()` gives runs and their +decoded. `Values` forces the decode, while `AsDictionary()` gives its codes and values, so a consumer +that groups by it groups by code and never touches its strings. `AsRunEnd()` gives runs and their lengths ([encoded-forms.md](../guide/encoded-forms.md)). With `ScanOptions.Compact = false`, a filtered scan delivers each live block whole with its `Selection` instead of copying the surviving -rows: a filter that keeps half a block copies nothing ([selection.md](../guide/selection.md)). +rows, so a filter that keeps half a block copies nothing ([selection.md](../guide/selection.md)). ### 5.5 Queries: select, group, aggregate A projection, a group by and an aggregate are operators of the scan, not loops the caller writes -(`Aggregation/`, `Scan/Scan.Agg*.cs`, `Scan/Scan.Grouping.cs`), and they read as LINQ does, in -method or query syntax: +(`Aggregation/`, `Scan/Scan.Agg*.cs`, `Scan/Scan.Grouping.cs`), and they read as LINQ does, in method +or query syntax: ```csharp Scan hourly = @@ -184,33 +188,33 @@ Scan hourly = .As(); ``` -`GroupBy` takes one symbol or a tuple of them, whose names become the key's: `g.Key.Hour`. The -group's aggregates take a lambda over the rows, as LINQ's do, and the rows' probe exists only inside -it: a column that is neither a key nor aggregated does not compile. `Where`, `OrderBy`, `ThenBy`, -`Skip` and `Take` on the groups are a filter, an order and a top-k the engine applies. `Select` of -one value delivers an `Aggregation`, one `T` per group; `Select` of several delivers an -`Aggregation` that a record reads, as many values as the record has members, through -`As()`. `Select` on a scan delivers a `Projection` or a `Projection` of computed values -the same way. `As()` turns any of them into a `Scan` over the result, which the -result's columns fill, and on which every operator and sink of a scan runs again: a filter, a group -by of the groups, a write. `AggAsync` answers a whole scan into a record the same way. - -A query runs block by block on the encoded form — a dictionary key by code, a run-end key by run, a -sorted key by run detection, a constant one whole — and never hands a batch to the caller's code. It -**flows**: its result is a stream of batches, delivered as soon as each is final. A group by on a -key the statistics prove ordered, or a function of one that keeps the order, closes its groups as -the key moves on and delivers them then, holding the open ones only; on any other key it holds one -state per group, at any degree, and delivers at the end of its input. The whole semantics, from the -flow and the catalog of aggregates to the order of groups and the engine's strategies, is +`GroupBy` takes one symbol or a tuple of them, whose names become the key's: `g.Key.Hour`. The group's +aggregates take a lambda over the rows, as LINQ's do, and the rows' probe only exists inside it, so a +column that is neither a key nor aggregated does not compile. `Where`, `OrderBy`, `ThenBy`, `Skip` and +`Take` on the groups are a filter, an order and a top-k the engine applies. `Select` of one value +delivers an `Aggregation`, one `T` per group. `Select` of several delivers an `Aggregation` that a +record reads, with as many values as the record has members, through `As()`. `Select` on a +scan delivers a `Projection` or a `Projection` of computed values the same way. `As()` +turns any of them into a `Scan` over the result, filled by the result's columns, on which +every operator and sink of a scan runs again: a filter, a group by of the groups, a write. +`AggAsync` answers a whole scan into a record the same way. + +A query runs block by block on the encoded form (a dictionary key by code, a run-end key by run, a +sorted key by run detection, a constant one whole) and never hands a batch to the caller's code. It +flows: its result is a stream of batches, each delivered as soon as it is final. A group by on a key +the statistics prove ordered, or on a function of one that keeps the order, closes its groups as the +key moves on and delivers them then, holding only the open ones. On any other key it holds one state +per group, at any degree, and delivers at the end of its input. The full semantics, from the flow and +the catalog of aggregates to the order of groups and the engine's strategies, is in [16-queries.md](16-queries.md). -A caller's aggregator implements `IAggregator` and receives the plain form, or -`IEncodedAggregator` to receive runs and dictionaries too. A sum is exact whatever the order of rows -or the cut of a parallel aggregation: an integer sum is delivered as a 64-bit integer, or as its own -type when it is one already, throwing only when the total does not fit it; a float sum is the same -bits at every degree and under every cut of the data into chunks, files or objects. Floats skip -NaN, as the statistics do; a null key is a group of its own. Groups come in the order a query asks -for, and in no promised order without one. +A caller's aggregator implements `IAggregator` and receives the plain form, or `IEncodedAggregator` +to receive runs and dictionaries too. A sum is exact whatever the order of rows or the split of a +parallel aggregation. An integer sum is delivered as a 64-bit integer, or as its own type when it is +one already, and only throws when the total does not fit. A float sum is the same bits at every +degree and under every split of the data into chunks, files or objects. Floats skip NaN, as the +statistics do, and a null key is a group of its own. Groups come in the order a query asks for, and +in no promised order without one. ### 5.6 The sinks @@ -218,106 +222,109 @@ for, and in no promised order without one. |---|---|---|---| | `await foreach` | borrowed `Columns` | nothing per batch | the first split | | `ToBatchesAsync()` | owned `RecordBatch`es | one copy per batch into pooled buffers | the first split | -| `ToRecordsAsync()` | `IAsyncEnumerable` | a copy per row; an allocation per row for text, lists and nested classes | the first split | +| `ToRecordsAsync()` | `IAsyncEnumerable` | a copy per row, and an allocation per row for text, lists and nested classes | the first split | | `Select(r => e)` | `Projection`, a value computed per row | the same as `ToRecordsAsync`, for the columns its element names | the first split | | `Select(r => (…)).As()` | a `Scan` of values computed per row | nothing per batch, for the columns its elements name | the first split | -| `CountAsync`, `AnyAsync`, `MinAsync`, `MaxAsync`, `SumAsync`, `AverageAsync`, `CountDistinctAsync`, `AggregateAsync`, `AggAsync(a => e)` | one value | the statistics when they settle it, blocks otherwise ([12-index-reads.md](12-index-reads.md) §4) | at once when the statistics settle it, the end of the pass otherwise | -| `AggAsync(a => (…))` | several answers, one pass, into the record `TResult` | the same | the same | -| `GroupBy(…).Select(g => r)` | `Aggregation`, one value per group | one state per group, the open groups only on a key that streams; `k` groups for a top-k | the first group closed on a key that streams, the end of the pass otherwise | -| `GroupBy(…).Select(g => (…)).As()` | a `Scan` over the groups, its batches borrowed | the same, and nothing per batch nor per group | the same | -| `writer.WriteAsync(scan)` | the batches of any scan, written as they come | the writer's own work; a rollup builds no record | — | -| `ExplainAsync()` | the `ScanPlan` | statistics and zone maps, never data | — | - -**The plan ends at the sink.** A query has LINQ's names and syntax up to its last operator, and -everything up to there is pushed. Past it are values: `ToRecordsAsync()` and `ToValuesAsync()` hand -them out as an `IAsyncEnumerable`, to which `System.Linq.AsyncEnumerable` applies, on -materialized values. A query is enumerated, like a scan, by the pattern of `await foreach` rather -than as an `IAsyncEnumerable`, since what returns a stream to await carries `Async`; its own -`Skip`, `Take`, `ToListAsync`, `ToArrayAsync`, `As` and `Distinct` stay in the plan. A -query of several values is not enumerable: `As` hands it, like any result, to a scan, where -the plan goes on ([16-queries.md](16-queries.md) §1, [read-rows.md](../guide/read-rows.md)). +| `CountAsync`, `AnyAsync`, `MinAsync`, `MaxAsync`, `SumAsync`, `AverageAsync`, `CountDistinctAsync`, `AggregateAsync`, `AggAsync(a => e)` | one value | the statistics when they settle it, blocks otherwise (see [answers without rows](12-index-reads.md#4-answers-without-rows)) | at once when the statistics settle it, at the end of the pass otherwise | +| `AggAsync(a => (…))` | several answers in one pass, into the record `TResult` | the same | the same | +| `GroupBy(…).Select(g => r)` | `Aggregation`, one value per group | one state per group, only the open groups on a key that streams, and `k` groups for a top-k | the first group closed on a key that streams, the end of the pass otherwise | +| `GroupBy(…).Select(g => (…)).As()` | a `Scan` over the groups, its batches borrowed | the same, and nothing per batch or per group | the same | +| `writer.WriteAsync(scan)` | the batches of any scan, written as they come | the writer's own work, and a rollup builds no record | | +| `ExplainAsync()` | the `ScanPlan` | statistics and zone maps, never data | | + +The plan ends at the sink. A query has LINQ's names and syntax up to its last operator, and +everything up to there is pushed down. Past it are values: `ToRecordsAsync()` and `ToValuesAsync()` +hand them out as an `IAsyncEnumerable`, to which `System.Linq.AsyncEnumerable` applies, on +materialized values. A query is enumerated, like a scan, through the `await foreach` pattern rather +than as an `IAsyncEnumerable`, since whatever returns a stream to await carries `Async`. Its own +`Skip`, `Take`, `ToListAsync`, `ToArrayAsync`, `As` and `Distinct` stay in the plan. A query +of several values is not enumerable: `As` hands it, like any result, to a scan, where the plan +continues (see [the shape](16-queries.md#1-the-shape) and [read-rows.md](../guide/read-rows.md)). ### 5.7 The key cursor -`Scan().Keys(r => r.Column)` builds a `KeyCursorBuilder`, `TKey` inferred from the -member, and `OrderBy(r => r.Column)` delivers rows in key order: [12-index-reads.md](12-index-reads.md). +`Scan().Keys(r => r.Column)` builds a `KeyCursorBuilder`, with `TKey` inferred from +the member, and `OrderBy(r => r.Column)` delivers rows in key order. Both are described in +[12-index-reads.md](12-index-reads.md). ### 5.8 The tool path -For a file whose schema is not known when the program is compiled — `vxdump`, an ad hoc query, a -test — `file.Scan(columns)` reads columns by name, and `Where` takes a filter as text with typed holes -(`Scan/Scan.Tool.cs`, `Expressions/VortexExpr.Surface.cs`). It is the one place in the surface where -names are strings, because there is no other name. A batch is a `BatchView` holding the named columns -in the file's order; a number in a filter is the number the text wrote, scaled exactly against a -decimal column, and text compared with a date, time, timestamp, uuid or decimal column is parsed in +For a file whose schema is not known when the program is compiled (`vxdump`, an ad hoc query, a +test), `file.Scan(columns)` reads columns by name, and `Where` takes a filter as text with typed holes +(`Scan/Scan.Tool.cs`, `Expressions/VortexExpr.Surface.cs`). It is the one place in the API where names +are strings, because there is no other way to name them. A batch is a `BatchView` holding the named +columns in the file's order. A number in a filter is the number the text wrote, scaled exactly against +a decimal column, and text compared with a date, time, timestamp, uuid or decimal column is parsed in that type and refused when it does not parse ([untyped-files.md](../guide/untyped-files.md)). ## 6. Writing -The shape of a `PipeWriter`: the caller fills builders synchronously, and the writer encodes and hands -the bytes to its sink asynchronously (`Writing/`). A writer comes from the session, over a path or a -pipe, typed by a record or by a schema; `Builder()` hands out its `ColumnsBuilder`, whose -`ColumnBuilder` gives spans to fill in place (`GetSpan`, `Advance`, as `IBufferWriter`), +Writing has the shape of a `PipeWriter`: the caller fills builders synchronously, and the writer +encodes and hands the bytes to its sink asynchronously (`Writing/`). A writer comes from the session, +over a path or a pipe, typed by a record or by a schema. `Builder()` hands out its `ColumnsBuilder`, +whose `ColumnBuilder` gives spans to fill in place (`GetSpan`, `Advance`, as `IBufferWriter`), appends values, nulls, text and lists, and is reused after every `WriteAsync`. A primitive column is -encoded from the caller's bytes without a copy. A list appended in one call that fails part way -leaves nothing behind. A nullable column costs nothing until its first null. +encoded from the caller's bytes without a copy. A list appended in one call that fails part way leaves +nothing behind. A nullable column costs nothing until its first null. `WriteAsync` takes a builder, a span or a stream of records, or a batch read from another file. `FlushAsync` seals the whole blocks into a chunk and is the only I/O before `CompleteAsync`, which writes the tail and returns the `WriteReport`. `Abandon()` gives the file up: a new file is deleted, -an append truncated back to its length, and a caller's pipe completed with an error. What the writer -does with the rows is [11-write-strategy.md](11-write-strategy.md); the guide's writing pages show each -form. +an append is truncated back to its length, and a caller's pipe is completed with an error. What the +writer does with the rows is described in [11-write-strategy.md](11-write-strategy.md), and the +guide's writing pages show each form. ## 7. Options, plans, reports, diagnostics, exceptions | type | what it says | |---|---| | `VortexOpenOptions` | the initial tail read, a known length or schema, the decompression cap, statistics verification, a torn tail's policy, index fragments to attach | -| `ScanOptions` | batch cap, prefetch, degree, `Compact`, and the `Pruning` and `UseIndexes` switches, which exist to check the structures, never to change a result | +| `ScanOptions` | batch cap, prefetch, degree, `Compact`, and the `Pruning` and `UseIndexes` switches, which exist to check the structures and never change a result | | `VortexWriteOptions` | blocks and chunk targets, the compression profile and per-column hints, the target edition, statistics, string bounds, the index policy, the identity, user metadata, the degree | -| `IndexPolicy`, `EncodingHint`, `CompressionProfile` | [10-indexes.md](10-indexes.md) §6, [11-write-strategy.md](11-write-strategy.md) §3.4 | -| `ScanPlan`, `PruningStep`, `CountPlan`, `OrderPlan`, `GroupPlan`, `GroupKeyPlan`, `GroupOrdering` | the plan before a read: blocks, live blocks, segments and bytes to read, what each structure pruned and what consulting it cost, how a count, an order and a group by will be answered ([16-queries.md](16-queries.md) §10) | -| `ScanStatistics`, `GroupStatistics` | the same quantities, measured after the scan; for a group by, its groups, the most held at once and the bytes, what its core did, how each key block was grouped, and the time to its first batch | +| `IndexPolicy`, `EncodingHint`, `CompressionProfile` | see [the index policy](10-indexes.md#6-the-policy) and [choosing encodings](11-write-strategy.md#34-choose-exact-verdicts-then-bounded-trials) | +| `ScanPlan`, `PruningStep`, `CountPlan`, `OrderPlan`, `GroupPlan`, `GroupKeyPlan`, `GroupOrdering` | the plan before a read: blocks, live blocks, segments and bytes to read, what each structure pruned and what consulting it cost, and how a count, an order and a group by will be answered (see [plan and statistics](16-queries.md#10-plan-and-statistics)) | +| `ScanStatistics`, `GroupStatistics` | the same quantities, measured after the scan, and for a group by its groups, the most held at once and their bytes, what its core did, how each key block was grouped, and the time to its first batch | | `WriteReport` | what the writer chose, per column and per chunk, and every index built or abandoned | -| `VortexDiagnostics` | the names of the meter and the activity source ([09-contracts.md](09-contracts.md) §5) | -| `VortexException` and its three kinds | a malformed file, an unsupported component with its id and `ComponentKind`, a schema that does not fit ([03-architecture.md](03-architecture.md) §5) | +| `VortexDiagnostics` | the names of the meter and the activity source (see [observability](09-contracts.md#5-observability)) | +| `VortexException` and its four kinds | a malformed file, an unsupported component with its id and `ComponentKind`, a schema that does not fit, and a query that outgrows its memory or scratch budget (see [error handling](03-architecture.md#5-error-handling)) | | `VortexEditions` | the default, newest and floor editions, and the Rust version that reads each | -**The plan and the statistics count in the same units**: a plan's `Segments` and `BytesToRead` are what +The plan and the statistics count in the same units. A plan's `Segments` and `BytesToRead` are what the source will be asked for, each segment once, and `Requests` and `BytesRequested` are what it was asked for, so on a file with statistics the two agree. `BlocksDecoded` counts the blocks that reached -the plain form: a block answered from a dictionary's codes or a run's lengths is not decoded. +the plain form, so a block answered from a dictionary's codes or a run's lengths is not counted as +decoded. -## 8. The I/O seam +## 8. The I/O interface `Vorticity.IO` holds `ISegmentSource` and its three built-in implementations, whose contract is -[03-architecture.md](03-architecture.md) §3.5, and a writer's sink is any `PipeWriter`. What a scan -promises its source: **a segment is requested at most once per scan**, and the session's cache adds -reuse across scans. [object-store.md](../guide/object-store.md) implements the seam for a remote store. +described in [I/O](03-architecture.md#35-io), and a writer's sink is any `PipeWriter`. A scan promises +its source that a segment is requested at most once per scan, and the session's cache adds reuse +across scans. [object-store.md](../guide/object-store.md) implements the interface for a remote +store. ## 9. The performance contract, and the gates that hold it -Each promise is a test (`tests/Vorticity.Tests/Api/*ContractTests.cs` and their neighbours), and -the build fails when one stops holding: +Each promise is a test (`tests/Vorticity.Tests/Api/*ContractTests.cs` and their neighbours), and the +build fails when one stops holding: | promise | gate | |---|---| -| no allocation per batch on every sink but rows | allocation counted across a full scan after one warm-up batch | -| `Values` contiguous and 64-byte aligned | the address of every `Values` span of the conformance corpus | -| a segment requested at most once per scan, and the plan is the execution | `Requests` equals the plan's distinct segments, `BytesRequested` equals `BytesToRead`, on every query of the corpus | +| no allocation per batch on every sink except rows | allocations counted across a full scan after one warm-up batch | +| `Values` contiguous and 64-byte aligned | the address of every `Values` span over the conformance corpus | +| a segment requested at most once per scan, and the plan is the execution | `Requests` equals the plan's distinct segments, and `BytesRequested` equals `BytesToRead`, on every query of the corpus | | a pruned block is not decoded | `BlocksDecoded` equals `LiveBlocks` | | a question the statistics answer reads nothing | no request after the open for a count, a minimum or a maximum on a file with statistics | -| a dictionary or run-end aggregate does not decode | `BlocksDecoded` is 0 for a group by on a dictionary column and a sum over runs | +| a dictionary or run-end aggregate does not decode | `BlocksDecoded` is 0 for a group by on a dictionary column and for a sum over runs | | a chunk is decoded once per scan, whatever runs it | the values decoded equal rows × columns at every prefetch and degree, and for a parallel aggregation | -| a query's answers and order do not depend on its syntax, its degree or the cut of its data | method and query syntax build one plan; every answer, float sums included, the same bits at every degree, over two row orders, and over a dataset cut into several objects, before and after compaction ([16-queries.md](16-queries.md) §13) | -| the first batch waits for the first split | the time to first batch of a scan, a projection, a `Distinct` and a group by that streams, flat over two files sixteen times apart, locally and over the HTTP source with latency | -| memory is the window and the open state | `LiveMemoryTests`: a streaming group by's peak flat as its groups grow a hundredfold, a top-k's proportional to `k`, a high-cardinality group by's at most its groups' | +| a query's answers and order do not depend on its syntax, its degree or how its data is split | method and query syntax build one plan, and every answer, float sums included, is the same bits at every degree, over two row orders, and over a dataset split into several objects, before and after compaction (see [how queries are tested](16-queries.md#13-how-it-is-tested)) | +| the first batch only waits for the first split | the time to first batch of a scan, a projection, a `Distinct` and a group by that streams stays flat over two files sixteen times apart, locally and over the HTTP source with latency | +| memory is the window plus the open state | `LiveMemoryTests`: a streaming group by's peak stays flat as its groups grow a hundredfold, a top-k's is proportional to `k`, and a high-cardinality group by's is at most its groups' | | a `Take` reads what it uses | `Requests` and `BlocksDecoded` bounded by the splits that hold its rows and the window | -| a group by is no slower than the loop by hand | a throughput axis per key form, the hand-written loops of the guide as floor | -| what cannot be pushed down does not compile | `tests/MustNotCompile` builds with exactly the expected diagnostics, a column outside an aggregate, `let` and `join` included | -| the surface does not drift | `PublicSurfaceTests` against `PublicSurface.txt`, and rule 6: a member that returns an awaitable or a terminal stream ends in `Async`, a builder does not | -| `vxdump` uses the public surface only | it compiles without access to internals | +| a group by is no slower than the hand-written loop | a throughput axis per key form, with the guide's hand-written loops as the floor | +| what cannot be pushed down does not compile | `tests/MustNotCompile` builds with exactly the expected diagnostics, including a column outside an aggregate, `let` and `join` | +| the API does not drift | `PublicSurfaceTests` against `PublicSurface.txt`, and rule 6: a member that returns an awaitable or a terminal stream ends in `Async`, and a builder does not | +| `vxdump` uses the public API only | it compiles without access to internals | The analyzers ship in `Vorticity.Generators` ([diagnostics.md](../guide/diagnostics.md)): @@ -331,29 +338,30 @@ The analyzers ship in `Vorticity.Generators` ([diagnostics.md](../guide/diagnost | VX1006 | a record type, or a type containing it, that is not `partial` | | VX1007 | a type that cannot be a record: generic, `ref struct`, static, abstract, file-local, or without a usable constructor | | VX1008 | a member the reader cannot fill | -| VX1009 | a component of a `GroupBy` key that is not a symbol of the scan: a literal, a captured value | +| VX1009 | a component of a `GroupBy` key that is not a symbol of the scan, such as a literal or a captured value | | VX1010 | an `As()` or an `AggAsync` whose record members do not take the selected values, position by position | -| VX1011 | several selected values read without a record: enumerated, or awaited without `TResult`; it names `As()` and the record to declare | +| VX1011 | several selected values read without a record (enumerated, or awaited without `TResult`). It names `As()` and the record to declare | -VX1001 to VX1004 are warnings; VX1005 to VX1008 are errors, and the generator emits nothing for a type -that has one; VX1009 and VX1010 are errors, since what they flag throws when the query is built, and -VX1011 goes with the compiler's own error, to name the fix. +VX1001 to VX1004 are warnings. VX1005 to VX1008 are errors, and the generator emits nothing for a type +that has one. VX1009 and VX1010 are errors, since what they flag throws when the query is built, and +VX1011 accompanies the compiler's own error, to name the fix. ## 10. The satellites -- **`Vorticity.Dataset`**: `VortexDataset.Scan()` returns the same `Scan` as a - file, over every object of the version the handle read, which a later commit does not move; a - batch's `StartRow` and a cursor's `Row` are positions in the dataset, and a cursor walks both ways - as a file's does. Its objects are opened ahead of the one being read and read side by side at a - degree above one, so the first batch waits for the first object and no object boundary stalls the - stream; every query of [16-queries.md](16-queries.md) runs over it, an object a range of a parallel +- `Vorticity.Dataset`: `VortexDataset.Scan()` returns the same `Scan` as a file, + over every object of the version the handle read, which a later commit does not move. A batch's + `StartRow` and a cursor's `Row` are positions in the dataset, and a cursor walks both ways like a + file's. Its objects are opened ahead of the one being read, and read side by side at a degree above + one, so the first batch only waits for the first object and no object boundary stalls the stream. + Every query of [16-queries.md](16-queries.md) runs over it, with an object as a range of a parallel aggregation. `DeleteAsync` and `UpdateAsync` take the same lambda as `Where`, and - `EvolveSchemaAsync` a `VortexSchema` and the renamed columns. The store seam, the maintenance calls, - what each costs and how a change of rows or of schema commits are [13-dataset.md](13-dataset.md). -- **`Vorticity.RowEncoding`**: `RowEncoder`, `RowSortField`, `RowKeys` and `RowKeyEncoder` + `EvolveSchemaAsync` takes a `VortexSchema` and the renamed columns. The store interface, the + maintenance calls, what each costs and how a change of rows or of schema commits are described in + [13-dataset.md](13-dataset.md). +- `Vorticity.RowEncoding`: `RowEncoder`, `RowSortField`, `RowKeys` and `RowKeyEncoder` ([06-row-encoding.md](06-row-encoding.md)). `[Experimental]` is set on the assembly of both experimental packages, so every type reports its diagnostic to a caller, and the packages' own code needs no suppression. -- **`vxdump`** is written against the tool path, the inspection members of `VortexFile` and the plans - only, and published ahead of time: a section it cannot print from the public surface is a gap in the - surface, not in the tool ([vxdump.md](../guide/vxdump.md)). +- `vxdump` is written against the tool path, the inspection members of `VortexFile` and the plans + only, and published ahead of time. A section it could not print from the public API would be a gap + in the API, not in the tool ([vxdump.md](../guide/vxdump.md)). diff --git a/docs/design/15-compaction.md b/docs/design/15-compaction.md index 27ec35ef..86782a9e 100644 --- a/docs/design/15-compaction.md +++ b/docs/design/15-compaction.md @@ -1,322 +1,341 @@ # Compaction: when it runs, and what a store that cannot delete does to it -Compaction is [13-dataset.md](13-dataset.md) §5 together with what this document describes: one -planner, whose jobs run one per `CompactAsync` call, per turn of a background loop, -or in the commit that takes level 0 past its ceiling (§2); a plan that descends the trees rather -than reading them (§3); purges of marks, and long vectors out of line (§4); and a profile for a -store under a retention lock (§5). This document weighs the ways to drive the same work: inline in a -writer's commit, on demand, or in the background. It also weighs what a store under a retention lock -changes, when it refuses to delete or overwrite an object before a date: S3 Object Lock, Azure -immutable blob storage, a GCS bucket lock. Nothing here changes the commit protocol. The format -carries the tallies in a page's summary and a pointer in each level of a header (§3), a vector's -reference in an entry (§4) and the locked store in a header (§5). A page without tallies still -reads, and a reader that knows no vector out of line refuses such an entry rather than reading it -wrong. +Compaction is the [levels and compaction](13-dataset.md#5-levels-and-compaction) section of the +dataset design, plus what this document describes: + +- one planner, whose jobs run one per `CompactAsync` call, one per turn of a background loop, or in + the commit that takes level 0 past its ceiling ([three drivers](#2-three-drivers-for-one-job)) +- a plan that descends the trees rather than reading them + ([planning](#3-planning-in-the-depth-of-the-tree)) +- purges of marks, and long vectors stored out of line + ([folding marks](#4-folding-marks-where-no-merge-goes)) +- a profile for a store under a retention lock ([stores that cannot + delete](#5-stores-that-cannot-delete)) + +The document weighs the ways to drive the same work (inline in a writer's commit, on demand, or in +the background), and what changes on a store under a retention lock, one that refuses to delete or +overwrite an object before a date: S3 Object Lock, Azure immutable blob storage, a GCS bucket lock. +Nothing here changes the commit protocol. The format carries the tallies in a page's summary and a +pointer in each level of a header, a vector's reference in an entry, and the locked-store flag in a +header. A page without tallies still reads, and a reader that does not know about vectors out of line +refuses such an entry rather than reading it wrong. ## 1. What compaction buys, and what it costs -Compaction holds the read bounds of [13-dataset.md](13-dataset.md) §1: +Compaction holds the read bounds of the dataset (see [where the N's +hide](13-dataset.md#1-where-the-ns-hide-and-the-bound-that-replaces-each)): -- **Level 0 holds at most eight objects.** A lookup touches at most eight objects there, rather than +- Level 0 holds at most eight objects, so a lookup touches at most eight objects there, rather than one per append since the last merge. -- **The levels above 0 are key-disjoint.** A lookup touches one object in each of them. -- **The marks a delete leaves are folded.** Their rows are no longer stepped over, and their bytes are - freed (§12 of the same document). +- The levels above 0 are key-disjoint, so a lookup touches one object in each of them. +- The marks a delete leaves are folded, so their rows are no longer stepped over and their bytes are + freed (see [deleting rows](13-dataset.md#12-deleting-and-updating-rows)). It pays for these in bytes written. A row appended in a small commit moves up through every level, and each merge rewrites the part of the next level it overlaps. That is about `F / 2` rewrites per -level at a fan-out `F`: about 5 at the default of 10. A load of gigabytes goes straight to the level -that holds it and is written once (§5.2). The levels grow by `F`. With level 1 at 1.25 MiB, a -tebibyte of data needs seven levels, so a byte that came in with a small commit is written about 35 -times before it settles. +level at a fan-out `F`, about 5 at the default of 10. A load of gigabytes goes straight to the level +that holds it and is written once (see [the invariant](13-dataset.md#52-the-invariant)). The levels +grow by `F`. With level 1 at 1.25 MiB, a tebibyte of data needs seven levels, so a byte that came in +with a small commit is written about 35 times before it settles. -Three costs are left over, and this document is about them: +Three costs remain, and this document is about them: -1. **Who pays, and when.** On demand, whoever calls `CompactAsync`, when they do; until then a +1. Who pays, and when. On demand, whoever calls `CompactAsync` pays, when they call it. Until then a dataset reports its lag, and lookups touch the extra objects. -2. **Planning.** With a level's bytes recorded nowhere but in its leaves, a plan would walk every - level's tree: nothing at the 40 objects of the churn, but at a tebibyte of 4 MiB objects 262 144 - entries of a few hundred bytes each, about a thousand pages and a hundred megabytes, on every - plan. A plan descends instead (§3). -3. **Marks in levels no merge reaches.** Level 4 of a ten-million-row dataset is rewritten only when +2. Planning. If a level's bytes were only recorded in its leaves, a plan would walk every level's + tree. That is nothing at the 40 objects of the churn bench, but at a tebibyte of 4 MiB objects it + is 262 144 entries of a few hundred bytes each, about a thousand pages and a hundred megabytes, on + every plan. A plan descends instead ([planning](#3-planning-in-the-depth-of-the-tree)). +3. Marks in levels no merge reaches. Level 4 of a ten-million-row dataset is only rewritten when level 3 overflows into it, and the top level never is. The marks there last until a delete would pass an object's share or its vector's cap, and that delete then rewrites the object itself. Meanwhile each commit that marks a row writes again the leaf page holding the object's entry, and - that page grows with the vectors it holds (§4). + that page grows with the vectors it holds ([folding marks](#4-folding-marks-where-no-merge-goes)). ## 2. Three drivers for one job -A **job** stays what it is: one plan's inputs read in key order, the outputs written, and one -`ReplaceObjects` committed. That commit names the entries it read, and is abandoned when one of -them changed (13-dataset.md §8.2). Correctness never depends on who runs the job, so every driver -below is safe against every other one and against every writer. What they differ in is who waits -for the work, and how much of it is wasted. +A job is always the same thing: one plan's inputs read in key order, the outputs written, and one +`ReplaceObjects` committed. That commit names the entries it read, and is abandoned when one of them +changed (see [rebase](13-dataset.md#82-rebase-by-re-applying-operations)). Correctness never depends +on who runs the job, so every driver below is safe against every other one and against every writer. +What they differ in is who waits for the work, and how much of it is wasted. | driver | who pays | the bounds | what goes wrong | |---|---|---|---| -| inline, in the commit | the writer whose commit crosses a bound | held after every commit | the writer's latency jumps by a merge; a writer does work for everyone; a crash leaves the outputs for vacuum | -| on demand, `CompactAsync` | the application's scheduler | held when it runs; `Lag` says by how much they are not | a scheduler that stops leaves lookups touching every object since | +| inline, in the commit | the writer whose commit crosses a bound | held after every commit | the writer's latency jumps by a merge, a writer does work for everyone, and a crash leaves the outputs for vacuum | +| on demand, `CompactAsync` | the application's scheduler | held when it runs, and `Lag` says by how much they are not | a scheduler that stops leaves lookups touching every object since | | in the background | a loop in some process | held within the loop's reaction time | two loops on one dataset race for the same inputs, and the loser's outputs are garbage | -**Inline.** A commit that takes level 0 past its ceiling runs one job before it returns, so that no -version the writer publishes leaves level 0 over its bound for long. The price is the merge's own -latency, added to that one commit: level 0 into level 1 costs 0.3 to 1 ms a commit in the churn, -spread over the commits between merges. A merge into a deep level is seconds. So inline mode is -bounded to level 0, and to inputs under a byte budget: past it, the job is left to another driver -and the writer returns at once. Inline mode suits a dataset with one writer and no process to spare. -The writer is the one party that knows level 0 just grew. - -**Implementation** (`DatasetOptions.InlineCompactionBytes`, 0 by default): an append, an import, a -replacement or a change that adds to level 0 runs the level-0 job before it returns, when the job -reads no more than the budget. The commit stands whatever befalls the job: one that fails, or that -the caller cancels, leaves level 0 for the next commit or another driver, and the caller is told the -commit's outcome. On the churn the same bytes are written either way; inline, the merges move into -the writers' commits, and an append takes 0.9 to 1.1 ms on average rather than 0.5 to 0.7, its 99th -percentile 6 to 9 ms rather than 3. - -**On demand.** The default. `PlanCompactionAsync` says what is due and what it costs, and -`CompactAsync` runs one job. The application loops, from a scheduled task or after a batch of -commits. It is the right default for a library, because the library never starts a thread of its own. -It stays the primitive the other two drivers are made of. - -**In the background.** A long-running call, `RunCompactionAsync(schedule, cancellationToken)`, that -any host runs: a hosted service, a worker, a console. It asks for the latest version, one head -request for a handle that knows its own is recent (13-dataset.md §8.3). It plans, runs the job, and -sleeps when nothing is due. Two limits shape it: a budget of bytes written per second, so that -compaction never takes the store's bandwidth from the writers, and one job at a time per loop. A -hosting integration, such as `IHostedService`, belongs in a separate package, since the dataset -package takes no dependency. - -**Two loops on one dataset** are safe and wasteful: both plan the same job, and one commit loses. -The waste is avoided without any object that coordinates: - -- **Deterministic job choice.** The planner ranks the due jobs, and a loop that knows it is one of - `n` takes the job whose rank is its index modulo `n`, which spreads `n` loops over `n` jobs. -- **Or a lease.** Before a job, a loop creates `leases//