From 6bcd25fa6a84fc55b941e9714c45c981730c2b8d Mon Sep 17 00:00:00 2001 From: Robin Kooyman Date: Wed, 9 Sep 2026 16:03:08 +0200 Subject: [PATCH] release: bump the version to 2.0.0-rc6 Set the workspace version to 2.0.0-rc.6. All 29 crates inherit it. Set @beacon/client to 2.0.0-rc.6. Set beacondb and beacon-datalake-cli to 2.0.0rc6, which is the PEP 440 spelling. Rename docs/docs/2.0.0-rc5 to docs/docs/2.0.0-rc6. Point every link at the new directory. Keep LATEST_VERSION at 1.8.0, because /docs/latest names the newest stable release, not the newest directory. --- CHANGELOG.md | 26 +-- Cargo.lock | 58 +++--- Cargo.toml | 2 +- QUICKSTART.md | 4 +- README.md | 18 +- .../beacon-datalake-cli/pyproject.toml | 2 +- beacon-clients/beacon-ts/package-lock.json | 4 +- beacon-clients/beacon-ts/package.json | 2 +- beacon-clients/package-lock.json | 2 +- beacon-db/beacon-db-py/docs/api-reference.md | 16 +- beacon-db/beacon-db-py/docs/data-in.md | 2 +- .../beacon-db-py/docs/getting-started.md | 10 +- beacon-db/beacon-db-py/docs/index.md | 22 +-- beacon-db/beacon-db-py/docs/overview.md | 16 +- beacon-db/beacon-db-py/docs/querying.md | 2 +- .../beacon-db-py/docs/remote-catalogs.md | 6 +- beacon-db/beacon-db-py/docs/secrets.md | 4 +- beacon-db/beacon-db-py/pyproject.toml | 2 +- beacon-server/beacon-server/src/axum/home.rs | 8 +- docs/.vitepress/config.mts | 180 +++++++++--------- .../theme/components/PreReleaseNotice.vue | 2 +- docs/.vitepress/theme/version.js | 2 +- docs/available-nodes/available-nodes.md | 4 +- .../api/exploring-data.md | 10 +- .../{2.0.0-rc5 => 2.0.0-rc6}/api/index.md | 12 +- .../api/querying/examples.md | 0 .../api/querying/index.md | 8 +- .../api/querying/json.md | 4 +- .../api/querying/sql.md | 6 +- .../arrays-to-tables.md | 8 +- .../{2.0.0-rc5 => 2.0.0-rc6}/cf-decoding.md | 8 +- .../coming-from-xarray.md | 12 +- .../docs/{2.0.0-rc5 => 2.0.0-rc6}/concepts.md | 20 +- .../{2.0.0-rc5 => 2.0.0-rc6}/connect/cli.md | 0 .../connect/datagrip.md | 0 .../connect/python-adbc.md | 0 .../connect/python.md | 0 .../connect/typescript.md | 8 +- .../connect/web-admin-ui.md | 10 +- .../data-sources/attach.md | 14 +- .../data-sources/external-tables.md | 36 ++-- .../data-sources/index.md | 32 ++-- .../data-sources/object-storage.md | 22 +-- .../data-sources/remote-tables.md | 6 +- .../data-sources/sql-databases.md | 8 +- docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/faq.md | 60 +++--- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/arrow.md | 6 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/atlas.md | 18 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/bbf.md | 6 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/csv.md | 6 +- .../formats/delta-lake.md | 16 +- .../formats/geoparquet.md | 16 +- .../formats/geotiff.md | 6 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/hdf5.md | 26 +-- .../formats/iceberg.md | 16 +- .../formats/icechunk.md | 10 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/index.md | 58 +++--- .../formats/inspect-a-schema.md | 8 +- .../formats/netcdf.md | 10 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/odv.md | 8 +- .../formats/parquet.md | 8 +- .../{2.0.0-rc5 => 2.0.0-rc6}/formats/zarr.md | 10 +- .../getting-started.md | 46 ++--- .../guides/export-results.md | 12 +- .../{2.0.0-rc5 => 2.0.0-rc6}/guides/index.md | 12 +- .../guides/query-a-collection.md | 12 +- .../guides/query-s3.md | 14 +- .../guides/speed-up-queries.md | 14 +- .../{2.0.0-rc5 => 2.0.0-rc6}/how-it-works.md | 20 +- .../internals/file-statistics.md | 10 +- .../internals/storage.md | 8 +- .../{2.0.0-rc5 => 2.0.0-rc6}/introduction.md | 44 ++--- docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/mcp.md | 4 +- .../{2.0.0-rc5 => 2.0.0-rc6}/quickstart.md | 22 +-- .../security/access-control.md | 2 +- .../server/configuration.md | 20 +- .../server/crawlers.md | 14 +- .../server/datasets.md | 44 ++--- .../server/extensions.md | 12 +- .../{2.0.0-rc5 => 2.0.0-rc6}/server/index.md | 38 ++-- .../server/performance-tuning.md | 8 +- .../{2.0.0-rc5 => 2.0.0-rc6}/server/view.md | 0 .../sql/create-external-table.md | 72 +++---- .../sql/create-materialized-view.md | 6 +- .../sql/create-view.md | 4 +- .../sql/function-reference.md | 40 ++-- .../{2.0.0-rc5 => 2.0.0-rc6}/sql/group-by.md | 2 +- .../sql/identifiers.md | 2 +- .../{2.0.0-rc5 => 2.0.0-rc6}/sql/index.md | 16 +- .../docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/join.md | 0 .../sql/managed-tables.md | 4 +- .../sql/remote-tables.md | 10 +- .../{2.0.0-rc5 => 2.0.0-rc6}/sql/secrets.md | 12 +- .../{2.0.0-rc5 => 2.0.0-rc6}/sql/select.md | 0 .../sql/spatial-functions.md | 4 +- .../{2.0.0-rc5 => 2.0.0-rc6}/sql/summarize.md | 2 +- .../sql/table-functions-utility.md | 0 .../sql/table-functions.md | 18 +- .../sql/union-by-name.md | 0 .../{2.0.0-rc5 => 2.0.0-rc6}/sql/where.md | 0 .../troubleshooting.md | 38 ++-- docs/index.md | 16 +- docs/why-beacon.md | 24 +-- 103 files changed, 766 insertions(+), 766 deletions(-) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/api/exploring-data.md (96%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/api/index.md (83%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/api/querying/examples.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/api/querying/index.md (94%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/api/querying/json.md (98%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/api/querying/sql.md (93%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/arrays-to-tables.md (96%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/cf-decoding.md (95%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/coming-from-xarray.md (93%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/concepts.md (83%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/connect/cli.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/connect/datagrip.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/connect/python-adbc.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/connect/python.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/connect/typescript.md (94%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/connect/web-admin-ui.md (94%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/data-sources/attach.md (81%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/data-sources/external-tables.md (85%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/data-sources/index.md (78%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/data-sources/object-storage.md (87%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/data-sources/remote-tables.md (96%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/data-sources/sql-databases.md (95%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/faq.md (82%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/arrow.md (79%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/atlas.md (93%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/bbf.md (86%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/csv.md (88%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/delta-lake.md (92%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/geoparquet.md (88%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/geotiff.md (86%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/hdf5.md (91%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/iceberg.md (92%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/icechunk.md (94%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/index.md (75%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/inspect-a-schema.md (92%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/netcdf.md (91%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/odv.md (82%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/parquet.md (81%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/formats/zarr.md (91%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/getting-started.md (80%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/guides/export-results.md (87%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/guides/index.md (62%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/guides/query-a-collection.md (91%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/guides/query-s3.md (84%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/guides/speed-up-queries.md (88%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/how-it-works.md (86%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/internals/file-statistics.md (98%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/internals/storage.md (92%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/introduction.md (78%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/mcp.md (98%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/quickstart.md (87%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/security/access-control.md (99%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/configuration.md (97%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/crawlers.md (95%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/datasets.md (60%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/extensions.md (93%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/index.md (75%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/performance-tuning.md (97%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/server/view.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/create-external-table.md (75%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/create-materialized-view.md (93%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/create-view.md (91%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/function-reference.md (93%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/group-by.md (95%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/identifiers.md (97%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/index.md (52%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/join.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/managed-tables.md (98%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/remote-tables.md (91%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/secrets.md (87%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/select.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/spatial-functions.md (99%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/summarize.md (96%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/table-functions-utility.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/table-functions.md (95%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/union-by-name.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/sql/where.md (100%) rename docs/docs/{2.0.0-rc5 => 2.0.0-rc6}/troubleshooting.md (88%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 86063a25..757795ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -75,8 +75,8 @@ tag. Releases before 2.0.0 are recorded in the as `default`. The strategy is an `ArrowTypeWideningStrategy` like the default one, so an embedded build passes `NumpyArrowTypeWidening` to `RuntimeBuilder::with_type_widening` or `OpenOptions::with_type_widening`. See - [Configuration](docs/docs/2.0.0-rc5/server/configuration.md#query-engine) and - [Troubleshooting](docs/docs/2.0.0-rc5/troubleshooting.md#a-column-has-two-types-across-the-files). + [Configuration](docs/docs/2.0.0-rc6/server/configuration.md#query-engine) and + [Troubleshooting](docs/docs/2.0.0-rc6/troubleshooting.md#a-column-has-two-types-across-the-files). - **`BEACON_TYPE_WIDENING_ON_CONFLICT` settles a column that no type holds.** A collection can type one column as a number in one file and as a string in another. No type holds both, so the schema merge refused the whole table and the table answered no query: `Incompatible types for @@ -91,8 +91,8 @@ tag. Releases before 2.0.0 are recorded in the `Float64`, widens as before under either setting. An unknown value logs a warning and reads as `fail`, because a server that cannot start over a typo is worse than one that names the column. An embedded build sets the same rule with `RuntimeBuilder::with_type_conflict`. See - [Configuration](docs/docs/2.0.0-rc5/server/configuration.md#query-engine) and - [Troubleshooting](docs/docs/2.0.0-rc5/troubleshooting.md#a-column-has-two-types-across-the-files). + [Configuration](docs/docs/2.0.0-rc6/server/configuration.md#query-engine) and + [Troubleshooting](docs/docs/2.0.0-rc6/troubleshooting.md#a-column-has-two-types-across-the-files). - **A gateway can sign you in to the admin UI.** A deployment that puts nginx or oauth2-proxy in front of Beacon holds the super-user credentials in the gateway, and the operator had to type them a second time in the login screen. The UI now calls `GET /admin/api/admin/check` with no @@ -103,7 +103,7 @@ tag. Releases before 2.0.0 are recorded in the the detection for that browser tab, and a new tab starts a proxy session again. An injected header makes every caller behind the gateway a super-user, so put your own authentication in front of it. See - [the admin web UI](docs/docs/2.0.0-rc5/connect/web-admin-ui.md#a-gateway-can-sign-you-in). + [the admin web UI](docs/docs/2.0.0-rc6/connect/web-admin-ui.md#a-gateway-can-sign-you-in). - **`COMPACT TABLE` reclaims what a managed table's writes leave behind.** A Lance table never shrinks on its own: every `INSERT` commits its own fragments, a `DELETE` writes a deletion file and keeps the rows, an `UPDATE` rewrites fragments, and each superseded version still holds its @@ -171,7 +171,7 @@ tag. Releases before 2.0.0 are recorded in the argument types and so gets no row there ([datafusion-spatial#1](https://github.com/robinskil/datafusion-spatial/issues/1)). Every function runs, listed or not. See - [the function reference](docs/docs/2.0.0-rc5/sql/function-reference.md#geospatial-functions), + [the function reference](docs/docs/2.0.0-rc6/sql/function-reference.md#geospatial-functions), which is the full list. - **`ST_Transform` reprojects a geometry**, and a standard build ships it. That makes 123 spatial functions. It links [PROJ](https://proj.org), so **a build from source now needs PROJ 9.6.2 or @@ -200,7 +200,7 @@ tag. Releases before 2.0.0 are recorded in the column another writer commits shows on the next query without a restart. A `WHERE` clause is pushed into the Iceberg scan, which drops data files from the manifests' statistics. Reads only: no `INSERT`, `MERGE` or snapshot expiry, and no REST or Glue catalog yet. See - [Apache Iceberg](docs/docs/2.0.0-rc5/formats/iceberg.md). + [Apache Iceberg](docs/docs/2.0.0-rc6/formats/iceberg.md). - **Icechunk repositories read through the Zarr reader.** An Icechunk repository is a Zarr v3 store with commits, branches and snapshots. `read_icechunk('sst/repo')` and `CREATE EXTERNAL TABLE … STORED AS ICECHUNK` read one version of it: the tip of a branch by default, or a fixed `tag` / @@ -215,11 +215,11 @@ tag. Releases before 2.0.0 are recorded in the handler as `POST /api/query`, and `/admin/api/admin/crawlers` the same as `/api/admin/crawlers`. The whole alias sits behind the admin Basic auth gate, the client endpoints included: `/api/info` answers any caller, `/admin/api/info` only the super-user. The - [admin web UI](docs/docs/2.0.0-rc5/connect/web-admin-ui.md) now calls the alias, so a deployment + [admin web UI](docs/docs/2.0.0-rc6/connect/web-admin-ui.md) now calls the alias, so a deployment that puts its own security in front of `/api/*` keeps a working admin panel. `@beacon/client` reaches the alias with the new `apiPrefix: ADMIN_API_PREFIX` client option. The alias stays out of `/openapi.json`: publishing it would list every operation twice and repeat each operation id. See - [the REST API reference](docs/docs/2.0.0-rc5/api/index.md#admin-path-alias). + [the REST API reference](docs/docs/2.0.0-rc6/api/index.md#admin-path-alias). ### Changed @@ -238,7 +238,7 @@ tag. Releases before 2.0.0 are recorded in the over a large archive runs for minutes. The timer also starts its interval again when a pass ends, so a pass that outruns the interval no longer fires every missed tick back to back, each re-listing the store for a queue the pass just emptied. See - [File statistics](docs/docs/2.0.0-rc5/internals/file-statistics.md). + [File statistics](docs/docs/2.0.0-rc6/internals/file-statistics.md). - **The GeoJSON filter of the JSON query plans `ST_Within`.** A request carries `longitude_column`, `latitude_column` and `geometry`. It used to build `st_within_point(st_geojson_as_wkt(''), lon, lat)`. That path turned the geometry into @@ -356,7 +356,7 @@ tag. Releases before 2.0.0 are recorded in the fills that name only when the name is free, so your own table under it survives a restart, and a `CREATE` on a name a table holds still fails. That last error now names the stand-in and tells you to run `DROP TABLE` first, because a table nobody made is a confusing thing to collide with. - See [Configuration](docs/docs/2.0.0-rc5/server/configuration.md#the-default-table). + See [Configuration](docs/docs/2.0.0-rc6/server/configuration.md#the-default-table). - **`CREATE EXTERNAL TABLE` and `CREATE VIEW` no longer discard the table under the name.** Both registered straight over whatever held the name. `CREATE EXTERNAL TABLE obs STORED AS CSV LOCATION 'other/'` therefore repointed an existing `obs` with no warning, and a second @@ -441,7 +441,7 @@ tag. Releases before 2.0.0 are recorded in the `CREATE MATERIALIZED VIEW MyView` was worse: it registered `myview` but persisted `MyView`, so a restart renamed the view and broke every query that used the old spelling. A new `table_name` module builds the reference and keeps the case, and every path uses it. A new - [identifiers page](docs/docs/2.0.0-rc5/sql/identifiers.md) states the rule and its limits. + [identifiers page](docs/docs/2.0.0-rc6/sql/identifiers.md) states the rule and its limits. - **`OPTIONS` on a NetCDF, HDF5, Zarr or BBF external table had no effect** ([#421](https://github.com/maris-development/beacon/issues/421)). DataFusion's SQL planner renames an `OPTIONS` key without a `.` to `format.`. Those four factories read the bare key @@ -453,7 +453,7 @@ tag. Releases before 2.0.0 are recorded in the - **`CREATE EXTERNAL TABLE` did not document `OPTIONS`** ([#421](https://github.com/maris-development/beacon/issues/421)). The syntax block omitted the clause, and no page listed the keys of a format, so a reader had to open the Rust source. The - [create external table page](docs/docs/2.0.0-rc5/sql/create-external-table.md#options) now holds the clause, the + [create external table page](docs/docs/2.0.0-rc6/sql/create-external-table.md#options) now holds the clause, the rules that apply to every key, and an index of the keys of each `STORED AS` value. Each format page holds a table of its own keys, with a type, a default and a description. The page for a format that reads no key says so. Two examples also spelled an option `OPTIONS`, which does not parse: an `OPTIONS` list takes a key and a value, with no `=`. diff --git a/Cargo.lock b/Cargo.lock index 9cbdd5c4..76c84774 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1442,7 +1442,7 @@ dependencies = [ [[package]] name = "beacon-arrow-atlas" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1467,7 +1467,7 @@ dependencies = [ [[package]] name = "beacon-arrow-bbf" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "async-trait", @@ -1486,7 +1486,7 @@ dependencies = [ [[package]] name = "beacon-arrow-csv" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "async-trait", @@ -1503,7 +1503,7 @@ dependencies = [ [[package]] name = "beacon-arrow-geoparquet" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "async-trait", @@ -1531,7 +1531,7 @@ dependencies = [ [[package]] name = "beacon-arrow-hdf5" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1555,7 +1555,7 @@ dependencies = [ [[package]] name = "beacon-arrow-ipc" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "async-trait", @@ -1572,7 +1572,7 @@ dependencies = [ [[package]] name = "beacon-arrow-netcdf" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1602,7 +1602,7 @@ dependencies = [ [[package]] name = "beacon-arrow-odv" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1631,7 +1631,7 @@ dependencies = [ [[package]] name = "beacon-arrow-parquet" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "async-trait", @@ -1646,7 +1646,7 @@ dependencies = [ [[package]] name = "beacon-arrow-tiff" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1668,7 +1668,7 @@ dependencies = [ [[package]] name = "beacon-arrow-zarr" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1694,7 +1694,7 @@ dependencies = [ [[package]] name = "beacon-auth" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "argon2", @@ -1769,7 +1769,7 @@ dependencies = [ [[package]] name = "beacon-common" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "async-trait", @@ -1789,7 +1789,7 @@ dependencies = [ [[package]] name = "beacon-core" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1850,7 +1850,7 @@ dependencies = [ [[package]] name = "beacon-datafusion-ext" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1886,7 +1886,7 @@ dependencies = [ [[package]] name = "beacon-db-py" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1905,7 +1905,7 @@ dependencies = [ [[package]] name = "beacon-delta" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1926,7 +1926,7 @@ dependencies = [ [[package]] name = "beacon-file-stats" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -1952,7 +1952,7 @@ dependencies = [ [[package]] name = "beacon-functions" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "arrow 58.4.0", "beacon-arrow-atlas", @@ -1990,7 +1990,7 @@ dependencies = [ [[package]] name = "beacon-iceberg" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2013,7 +2013,7 @@ dependencies = [ [[package]] name = "beacon-icechunk" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2035,7 +2035,7 @@ dependencies = [ [[package]] name = "beacon-lance" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2062,7 +2062,7 @@ dependencies = [ [[package]] name = "beacon-mcp" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2079,7 +2079,7 @@ dependencies = [ [[package]] name = "beacon-nd-array" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2107,7 +2107,7 @@ dependencies = [ [[package]] name = "beacon-nd-arrow" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2128,7 +2128,7 @@ dependencies = [ [[package]] name = "beacon-redb-store" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "async-trait", "bincode", @@ -2148,7 +2148,7 @@ dependencies = [ [[package]] name = "beacon-server" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", @@ -2192,7 +2192,7 @@ dependencies = [ [[package]] name = "beacon-server-config" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "base64", "beacon-arrow-atlas", @@ -2210,7 +2210,7 @@ dependencies = [ [[package]] name = "beacon-sql-databases" -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" dependencies = [ "anyhow", "arrow 58.4.0", diff --git a/Cargo.toml b/Cargo.toml index 307e4a7f..16da6bec 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,7 +4,7 @@ members = ["beacon-server/beacon-server", "beacon-server/beacon-server-config", [workspace.package] -version = "2.0.0-rc.5" +version = "2.0.0-rc.6" rust-version = "1.94" diff --git a/QUICKSTART.md b/QUICKSTART.md index fe3ec7cd..44b6d317 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -62,7 +62,7 @@ Interactive API docs are at . ## Next steps - [README](README.md) — start a server, point it at data, query it from Python. -- [Quick Start](https://maris-development.github.io/beacon/docs/2.0.0-rc5/quickstart) — the same steps, with a public server to try. +- [Quick Start](https://maris-development.github.io/beacon/docs/2.0.0-rc6/quickstart) — the same steps, with a public server to try. - [Documentation](https://maris-development.github.io/beacon/) — full data model and API reference. -- [Configuration reference](https://maris-development.github.io/beacon/docs/2.0.0-rc5/server/configuration.html) — all `BEACON_*` settings. +- [Configuration reference](https://maris-development.github.io/beacon/docs/2.0.0-rc6/server/configuration.html) — all `BEACON_*` settings. - Community [Slack](https://beacontechnic-wwa5548.slack.com/join/shared_invite/zt-2dp1vv56r-tj_KFac0sAKNuAgUKPPDRg). diff --git a/README.md b/README.md index 033eedea..f207b903 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ A path in a query is relative to the datasets root. A file at `./datasets/obs/a. `obs/a.parquet`. Do not repeat `datasets/`. Do not write an `s3://` scheme. To use a bucket instead of a directory, set `BEACON_S3_DATASETS=true` and `BEACON_S3_BUCKET`. See -[Object Storage](https://maris-development.github.io/beacon/docs/2.0.0-rc5/data-sources/object-storage). +[Object Storage](https://maris-development.github.io/beacon/docs/2.0.0-rc6/data-sources/object-storage). ## 3. Query it from Python @@ -114,24 +114,24 @@ curl -X POST http://localhost:5001/api/query \ ``` Both interfaces compile to the same plan. See the -[JSON Query DSL](https://maris-development.github.io/beacon/docs/2.0.0-rc5/api/querying/json). +[JSON Query DSL](https://maris-development.github.io/beacon/docs/2.0.0-rc6/api/querying/json). ### Other clients There is a -[TypeScript SDK](https://maris-development.github.io/beacon/docs/2.0.0-rc5/connect/typescript), a -[terminal client](https://maris-development.github.io/beacon/docs/2.0.0-rc5/connect/cli), and an +[TypeScript SDK](https://maris-development.github.io/beacon/docs/2.0.0-rc6/connect/typescript), a +[terminal client](https://maris-development.github.io/beacon/docs/2.0.0-rc6/connect/cli), and an Arrow Flight SQL endpoint for JDBC and ADBC tools. ## Next steps | | | | --- | --- | -| Name your files as tables and views | [Server Setup](https://maris-development.github.io/beacon/docs/2.0.0-rc5/server/) | -| Set ports, storage and limits | [Configuration](https://maris-development.github.io/beacon/docs/2.0.0-rc5/server/configuration) | -| Decide who reads what | [Access Control](https://maris-development.github.io/beacon/docs/2.0.0-rc5/security/access-control) | -| Write queries | [SQL Reference](https://maris-development.github.io/beacon/docs/2.0.0-rc5/sql/) · [REST API](https://maris-development.github.io/beacon/docs/2.0.0-rc5/api/querying/) | -| Move from an xarray loop | [Coming from xarray](https://maris-development.github.io/beacon/docs/2.0.0-rc5/coming-from-xarray) | +| Name your files as tables and views | [Server Setup](https://maris-development.github.io/beacon/docs/2.0.0-rc6/server/) | +| Set ports, storage and limits | [Configuration](https://maris-development.github.io/beacon/docs/2.0.0-rc6/server/configuration) | +| Decide who reads what | [Access Control](https://maris-development.github.io/beacon/docs/2.0.0-rc6/security/access-control) | +| Write queries | [SQL Reference](https://maris-development.github.io/beacon/docs/2.0.0-rc6/sql/) · [REST API](https://maris-development.github.io/beacon/docs/2.0.0-rc6/api/querying/) | +| Move from an xarray loop | [Coming from xarray](https://maris-development.github.io/beacon/docs/2.0.0-rc6/coming-from-xarray) | Documentation home: diff --git a/beacon-clients/beacon-datalake-cli/pyproject.toml b/beacon-clients/beacon-datalake-cli/pyproject.toml index 24ef0c4d..5d9f48f8 100644 --- a/beacon-clients/beacon-datalake-cli/pyproject.toml +++ b/beacon-clients/beacon-datalake-cli/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "beacon-datalake-cli" -version = "2.0.0rc5" +version = "2.0.0rc6" description = "Terminal client for a Beacon server: run SQL, explore tables/datasets, render results, and export to CSV/Parquet/Arrow/NetCDF." readme = "README.md" requires-python = ">=3.10" diff --git a/beacon-clients/beacon-ts/package-lock.json b/beacon-clients/beacon-ts/package-lock.json index aff2f138..f2465785 100644 --- a/beacon-clients/beacon-ts/package-lock.json +++ b/beacon-clients/beacon-ts/package-lock.json @@ -1,12 +1,12 @@ { "name": "@beacon/client", - "version": "2.0.0-rc.5", + "version": "2.0.0-rc.6", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@beacon/client", - "version": "2.0.0-rc.5", + "version": "2.0.0-rc.6", "license": "Apache-2.0", "dependencies": { "apache-arrow": "21.1.0", diff --git a/beacon-clients/beacon-ts/package.json b/beacon-clients/beacon-ts/package.json index 2b144877..557447fe 100644 --- a/beacon-clients/beacon-ts/package.json +++ b/beacon-clients/beacon-ts/package.json @@ -1,6 +1,6 @@ { "name": "@beacon/client", - "version": "2.0.0-rc.5", + "version": "2.0.0-rc.6", "description": "Isomorphic TypeScript SDK for managing and querying a Beacon instance.", "license": "Apache-2.0", "repository": { diff --git a/beacon-clients/package-lock.json b/beacon-clients/package-lock.json index 4e4d07c8..8186ac92 100644 --- a/beacon-clients/package-lock.json +++ b/beacon-clients/package-lock.json @@ -12,7 +12,7 @@ }, "beacon-ts": { "name": "@beacon/client", - "version": "2.0.0-rc.5", + "version": "2.0.0-rc.6", "license": "Apache-2.0", "dependencies": { "apache-arrow": "21.1.0", diff --git a/beacon-db/beacon-db-py/docs/api-reference.md b/beacon-db/beacon-db-py/docs/api-reference.md index e22169d5..c45fc29b 100644 --- a/beacon-db/beacon-db-py/docs/api-reference.md +++ b/beacon-db/beacon-db-py/docs/api-reference.md @@ -44,7 +44,7 @@ beacondb.connect( | --- | --- | --- | | `database` | `":memory:"` | The path to a `beacon.db` file. Use `":memory:"` for a throwaway database in the process. Beacon creates a missing file. | | `read_only` | `False` | Refuse every write. This covers DDL, DML and statements with a side effect, such as `ATTACH`, `CREATE SECRET` and `INSERT`. `SELECT` and `SHOW` still work. | -| `auth` | `False` | Switch the RBAC of Beacon on. With auth off, the file gives full control. See [Getting started](/docs/2.0.0-rc5/beacondb/python/getting-started#authentication). | +| `auth` | `False` | Switch the RBAC of Beacon on. With auth off, the file gives full control. See [Getting started](/docs/2.0.0-rc6/beacondb/python/getting-started#authentication). | | `username`, `password`, `token` | `None` | The credentials of your identity. They need `auth=True`. With auth off they give an error. Beacon does not ignore them. | | `admin_username`, `admin_password` | `None` | Create the super-user on a database without an auth store. | | `anonymous` | `True` | `True` lets the anonymous, read-only principal connect. | @@ -53,7 +53,7 @@ beacondb.connect( | `memory_limit` | `None` | The soft memory budget of a query, in bytes. | | `cpu_limit` | `None` | The maximum number of worker threads. The default is the core count of the machine. | | `crawlers` | `False` | Switch the background dataset crawlers on. | -| `secrets_key` | `None` | A base64 key of 32 bytes. It encrypts the persistent secrets. `$BEACON_SECRETS_KEY` also works. A persistent secret needs it, because Beacon never stores a plaintext credential. See [Secrets](/docs/2.0.0-rc5/beacondb/python/secrets). | +| `secrets_key` | `None` | A base64 key of 32 bytes. It encrypts the persistent secrets. `$BEACON_SECRETS_KEY` also works. A persistent secret needs it, because Beacon never stores a plaintext credential. See [Secrets](/docs/2.0.0-rc6/beacondb/python/secrets). | ## `Connection` @@ -228,9 +228,9 @@ The package also exports `Warning`, as PEP 249 requires. ## See also -- [Getting started](/docs/2.0.0-rc5/beacondb/python/getting-started): connect, auth modes, read-only. -- [Querying](/docs/2.0.0-rc5/beacondb/python/querying): relations, readers, sinks and streams, with examples. -- [Bring data in](/docs/2.0.0-rc5/beacondb/python/data-in): `register()` and `append()`. -- [Remote catalogs](/docs/2.0.0-rc5/beacondb/python/remote-catalogs): `ATTACH` and pushdown. -- [Secrets](/docs/2.0.0-rc5/beacondb/python/secrets): object store credentials. -- [SQLAlchemy](/docs/2.0.0-rc5/beacondb/python/sqlalchemy): the `beacondb://` dialect. +- [Getting started](/docs/2.0.0-rc6/beacondb/python/getting-started): connect, auth modes, read-only. +- [Querying](/docs/2.0.0-rc6/beacondb/python/querying): relations, readers, sinks and streams, with examples. +- [Bring data in](/docs/2.0.0-rc6/beacondb/python/data-in): `register()` and `append()`. +- [Remote catalogs](/docs/2.0.0-rc6/beacondb/python/remote-catalogs): `ATTACH` and pushdown. +- [Secrets](/docs/2.0.0-rc6/beacondb/python/secrets): object store credentials. +- [SQLAlchemy](/docs/2.0.0-rc6/beacondb/python/sqlalchemy): the `beacondb://` dialect. diff --git a/beacon-db/beacon-db-py/docs/data-in.md b/beacon-db/beacon-db-py/docs/data-in.md index 6144860b..e6874a22 100644 --- a/beacon-db/beacon-db-py/docs/data-in.md +++ b/beacon-db/beacon-db-py/docs/data-in.md @@ -44,7 +44,7 @@ columns do not match. ## Managed tables in SQL You can create and change the same managed tables with standard SQL. See -[Managed tables](/docs/2.0.0-rc5/beacondb/sql/managed-tables): +[Managed tables](/docs/2.0.0-rc6/beacondb/sql/managed-tables): ```python con.execute("CREATE TABLE obs AS SELECT * FROM read_parquet('obs/*.parquet')") diff --git a/beacon-db/beacon-db-py/docs/getting-started.md b/beacon-db/beacon-db-py/docs/getting-started.md index db60efe2..fe4d1352 100644 --- a/beacon-db/beacon-db-py/docs/getting-started.md +++ b/beacon-db/beacon-db-py/docs/getting-started.md @@ -16,7 +16,7 @@ The wheel is **abi3** (`cp310-abi3`). One wheel per platform therefore covers CP Beacon publishes wheels for Linux glibc (`x86_64`, `aarch64`), macOS (`arm64`, `x86_64`) and Windows (`x64`). There is **no wheel for Alpine or musl**. There, pip uses the source distribution and compiles the engine. See -[platform support](/docs/2.0.0-rc5/beacondb/python/#platform-support). +[platform support](/docs/2.0.0-rc6/beacondb/python/#platform-support). ## Open a database and query @@ -67,7 +67,7 @@ con.whoami() # {'username': 'local', 'roles': [], 'is_super_user': True, 'auth': False} ``` -Pass `auth=True` to switch on the [RBAC](/docs/2.0.0-rc5/security/access-control) of Beacon. A +Pass `auth=True` to switch on the [RBAC](/docs/2.0.0-rc6/security/access-control) of Beacon. A session then starts as the anonymous, read-only principal. It stays that way until you give credentials: @@ -95,6 +95,6 @@ and `CREATE SECRET`. `SELECT` and `SHOW …` still work. ## Next -- [Querying](/docs/2.0.0-rc5/beacondb/python/querying): the lazy relation, the readers and the file sinks. -- [Bring data in](/docs/2.0.0-rc5/beacondb/python/data-in): `register()` and `append()`. -- The full [SQL reference](/docs/2.0.0-rc5/beacondb/sql/) applies without a change. +- [Querying](/docs/2.0.0-rc6/beacondb/python/querying): the lazy relation, the readers and the file sinks. +- [Bring data in](/docs/2.0.0-rc6/beacondb/python/data-in): `register()` and `append()`. +- The full [SQL reference](/docs/2.0.0-rc6/beacondb/sql/) applies without a change. diff --git a/beacon-db/beacon-db-py/docs/index.md b/beacon-db/beacon-db-py/docs/index.md index a7acbbbd..157396b7 100644 --- a/beacon-db/beacon-db-py/docs/index.md +++ b/beacon-db/beacon-db-py/docs/index.md @@ -21,14 +21,14 @@ con.sql("SELECT * FROM read_parquet('obs/*.parquet')").df() - **In-process, no setup.** You run no server. The engine links into your Python process. - **Reads your formats in place.** NetCDF, Zarr, Parquet, GeoParquet, CSV, HDF5, ODV, GeoTIFF, Delta and more. There is no import step. It uses the same - [readers](/docs/2.0.0-rc5/beacondb/python/querying#reading-files) and the same - [SQL](/docs/2.0.0-rc5/beacondb/sql/) as Beacon Data Lake. + [readers](/docs/2.0.0-rc6/beacondb/python/querying#reading-files) and the same + [SQL](/docs/2.0.0-rc6/beacondb/sql/) as Beacon Data Lake. - **One portable file.** One `beacon.db` holds everything that Beacon *owns*: its catalog and its managed data. It references everything else. Copy the file and the managed lake goes with it. - **Arrow-native results.** Results cross into Python over the Arrow PyCapsule protocol. Any Arrow consumer reads them without an extra dependency. Use `.df()`, `.pl()` or `.arrow()` for pandas, polars or pyarrow. -- **Federates outward.** [`ATTACH`](/docs/2.0.0-rc5/beacondb/python/remote-catalogs) a remote Beacon +- **Federates outward.** [`ATTACH`](/docs/2.0.0-rc6/beacondb/python/remote-catalogs) a remote Beacon Data Lake and query it. You can then join remote tables against local files. ## Install @@ -75,7 +75,7 @@ RUN pip install beacondb ``` Must you stay on Alpine? Then see -[build from source](/docs/2.0.0-rc5/beacondb/python/building#building-on-alpine-musl) for the `apk` +[build from source](/docs/2.0.0-rc6/beacondb/python/building#building-on-alpine-musl) for the `apk` prerequisites. Beacon plans to publish musllinux wheels again. This gap is temporary. ::: @@ -84,10 +84,10 @@ You can force a source build on a platform that *does* have a wheel. Use ## Next steps -- [Getting started](/docs/2.0.0-rc5/beacondb/python/getting-started): connect, first query, auth modes. -- [Querying](/docs/2.0.0-rc5/beacondb/python/querying): lazy relations, readers, file sinks, streams. -- [Bring data in](/docs/2.0.0-rc5/beacondb/python/data-in): `register()` and `append()`. -- [Remote catalogs](/docs/2.0.0-rc5/beacondb/python/remote-catalogs): `ATTACH` a remote Beacon. -- [Secrets](/docs/2.0.0-rc5/beacondb/python/secrets): credentials for S3, GCS, Azure and a remote Beacon. -- [SQLAlchemy](/docs/2.0.0-rc5/beacondb/python/sqlalchemy): the `beacondb://` dialect. -- [API reference](/docs/2.0.0-rc5/beacondb/python/api-reference) · [Building from source](/docs/2.0.0-rc5/beacondb/python/building) +- [Getting started](/docs/2.0.0-rc6/beacondb/python/getting-started): connect, first query, auth modes. +- [Querying](/docs/2.0.0-rc6/beacondb/python/querying): lazy relations, readers, file sinks, streams. +- [Bring data in](/docs/2.0.0-rc6/beacondb/python/data-in): `register()` and `append()`. +- [Remote catalogs](/docs/2.0.0-rc6/beacondb/python/remote-catalogs): `ATTACH` a remote Beacon. +- [Secrets](/docs/2.0.0-rc6/beacondb/python/secrets): credentials for S3, GCS, Azure and a remote Beacon. +- [SQLAlchemy](/docs/2.0.0-rc6/beacondb/python/sqlalchemy): the `beacondb://` dialect. +- [API reference](/docs/2.0.0-rc6/beacondb/python/api-reference) · [Building from source](/docs/2.0.0-rc6/beacondb/python/building) diff --git a/beacon-db/beacon-db-py/docs/overview.md b/beacon-db/beacon-db-py/docs/overview.md index 67cf5433..1adcdb06 100644 --- a/beacon-db/beacon-db-py/docs/overview.md +++ b/beacon-db/beacon-db-py/docs/overview.md @@ -43,18 +43,18 @@ engines cannot open those formats. | Prepare extracts for other tools | Export the results to Parquet, CSV, NetCDF or Arrow. | Do you need to serve datasets to many people over the network, with access control and a web UI? -Then use the same engine as [Beacon Data Lake](/docs/2.0.0-rc5/getting-started). +Then use the same engine as [Beacon Data Lake](/docs/2.0.0-rc6/getting-started). ## How this chapter is organized -1. **[How It Works](/docs/2.0.0-rc5/beacondb/how-it-works)**: the engine, the query pipeline and the +1. **[How It Works](/docs/2.0.0-rc6/beacondb/how-it-works)**: the engine, the query pipeline and the storage model. -2. **[Data Sources](/docs/2.0.0-rc5/beacondb/data-sources/)**: external files, object storage, other +2. **[Data Sources](/docs/2.0.0-rc6/beacondb/data-sources/)**: external files, object storage, other databases and the internal format of BeaconDB. -3. **[SQL Reference](/docs/2.0.0-rc5/beacondb/sql/)**: statements, functions and syntax. -4. **[Python Binding](/docs/2.0.0-rc5/beacondb/python/)**: the client API, from `connect()` to +3. **[SQL Reference](/docs/2.0.0-rc6/beacondb/sql/)**: statements, functions and syntax. +4. **[Python Binding](/docs/2.0.0-rc6/beacondb/python/)**: the client API, from `connect()` to dataframes. -5. **[Guides](/docs/2.0.0-rc5/beacondb/guides/)**: step-by-step instructions for common tasks. +5. **[Guides](/docs/2.0.0-rc6/beacondb/guides/)**: step-by-step instructions for common tasks. ## Language bindings @@ -63,11 +63,11 @@ stay the same. | Binding | Status | Docs | |---|---|---| -| **Python** (`pip install beacondb`) | Available | [Python binding](/docs/2.0.0-rc5/beacondb/python/) | +| **Python** (`pip install beacondb`) | Available | [Python binding](/docs/2.0.0-rc6/beacondb/python/) | | **Rust** | Planned | | | **C ABI** (for .NET, Go, Node, Java) | Planned | | The Python binding is the reference front-end today. It gives a PEP 249 connection, a lazy relation API, Arrow-native results, `ATTACH`, secrets, streams and file sinks. -**[Get started with the Python binding](/docs/2.0.0-rc5/beacondb/python/getting-started)** +**[Get started with the Python binding](/docs/2.0.0-rc6/beacondb/python/getting-started)** diff --git a/beacon-db/beacon-db-py/docs/querying.md b/beacon-db/beacon-db-py/docs/querying.md index c1bfd404..357c12f0 100644 --- a/beacon-db/beacon-db-py/docs/querying.md +++ b/beacon-db/beacon-db-py/docs/querying.md @@ -100,7 +100,7 @@ A sink writes to a local path only. A `scheme://` destination raises `NotSupport con.sql("SUMMARIZE read_parquet('obs/*.parquet')").df() ``` -See [SUMMARIZE](/docs/2.0.0-rc5/beacondb/sql/summarize). +See [SUMMARIZE](/docs/2.0.0-rc6/beacondb/sql/summarize). ## Beyond SQL diff --git a/beacon-db/beacon-db-py/docs/remote-catalogs.md b/beacon-db/beacon-db-py/docs/remote-catalogs.md index d5f64627..98abf162 100644 --- a/beacon-db/beacon-db-py/docs/remote-catalogs.md +++ b/beacon-db/beacon-db-py/docs/remote-catalogs.md @@ -38,10 +38,10 @@ schema of each table at the first use. Authenticate with a `username` and `password` pair for HTTP Basic, or with a bearer `token`. Do not give both. Omit both only if the remote server allows anonymous access. The remote server applies its -own [RBAC](/docs/2.0.0-rc5/security/access-control) to your identity. Local file access has no such +own [RBAC](/docs/2.0.0-rc6/security/access-control) to your identity. Local file access has no such boundary. This connection *does* have one. -You can also store the credentials as a [secret](/docs/2.0.0-rc5/beacondb/python/secrets). Then +You can also store the credentials as a [secret](/docs/2.0.0-rc6/beacondb/python/secrets). Then reference the secret: ```python @@ -60,4 +60,4 @@ con.execute("ATTACH 'beacon://datalake:50051' AS lake " con.execute("DETACH lake") ``` -See the shared [`ATTACH` reference](/docs/2.0.0-rc5/beacondb/sql/remote-tables). +See the shared [`ATTACH` reference](/docs/2.0.0-rc6/beacondb/sql/remote-tables). diff --git a/beacon-db/beacon-db-py/docs/secrets.md b/beacon-db/beacon-db-py/docs/secrets.md index 44de9c61..45809ef6 100644 --- a/beacon-db/beacon-db-py/docs/secrets.md +++ b/beacon-db/beacon-db-py/docs/secrets.md @@ -6,7 +6,7 @@ description: Give BeaconDB credentials for object stores and remote Beacons as n Give Beacon the credentials of a cloud object store or a remote Beacon as a named, scoped `SECRET`. A secret replaces an environment variable. This page uses the shared -[`CREATE SECRET`](/docs/2.0.0-rc5/beacondb/sql/secrets) statement. The notes below cover beacondb +[`CREATE SECRET`](/docs/2.0.0-rc6/beacondb/sql/secrets) statement. The notes below cover beacondb only. ## Object store secrets @@ -43,7 +43,7 @@ A persistent secret also needs a file-backed database, not `:memory:`. ## Remote Beacon secrets A `TYPE BEACON` secret holds the credentials for -[`ATTACH`](/docs/2.0.0-rc5/beacondb/python/remote-catalogs): +[`ATTACH`](/docs/2.0.0-rc6/beacondb/python/remote-catalogs): ```python con.execute("CREATE SECRET lake (TYPE BEACON, USERNAME 'analyst', PASSWORD '…')") # or TOKEN '…' diff --git a/beacon-db/beacon-db-py/pyproject.toml b/beacon-db/beacon-db-py/pyproject.toml index c70cec35..f763d1fb 100644 --- a/beacon-db/beacon-db-py/pyproject.toml +++ b/beacon-db/beacon-db-py/pyproject.toml @@ -12,7 +12,7 @@ build-backend = "maturin" [project] name = "beacondb" -version = "2.0.0rc5" +version = "2.0.0rc6" description = "An embeddable, in-process analytical database for scientific data." readme = "README.md" requires-python = ">=3.10" diff --git a/beacon-server/beacon-server/src/axum/home.rs b/beacon-server/beacon-server/src/axum/home.rs index 3868670f..92fe33af 100644 --- a/beacon-server/beacon-server/src/axum/home.rs +++ b/beacon-server/beacon-server/src/axum/home.rs @@ -52,7 +52,7 @@ fn docs_url(version: &str) -> String { /// Names the documentation directory of `version`. /// /// The directory of a pre-release drops the dot of its pre-release part: version -/// `2.0.0-rc.5` publishes as `docs/2.0.0-rc5`. A release bump renames that +/// `2.0.0-rc.6` publishes as `docs/2.0.0-rc6`. A release bump renames that /// directory, so the running binary and its documentation stay in step. fn docs_directory(version: &str) -> String { // Build metadata is not part of a directory name. @@ -138,9 +138,9 @@ mod tests { #[test] fn the_docs_link_points_at_the_running_version() { - let page = render("Beacon", "2.0.0-rc.5", "", false, false); + let page = render("Beacon", "2.0.0-rc.6", "", false, false); assert!(page.contains( - "href=\"https://maris-development.github.io/beacon/docs/2.0.0-rc5/introduction\"" + "href=\"https://maris-development.github.io/beacon/docs/2.0.0-rc6/introduction\"" )); // A stable version publishes under its own number, dots intact. @@ -149,7 +149,7 @@ mod tests { "https://maris-development.github.io/beacon/docs/1.8.0/introduction" ); // Build metadata names no directory. - assert_eq!(docs_directory("2.0.0-rc.5+abc123"), "2.0.0-rc5"); + assert_eq!(docs_directory("2.0.0-rc.6+abc123"), "2.0.0-rc6"); } /// A release bump renames the docs directory. This test holds the two diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index b92edfc8..85a3ae94 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -12,7 +12,7 @@ const DESCRIPTION = // from this list, drop its `search: false` frontmatter, and uncomment the // sidebar entries marked "MCP is unreleased" below. const UNRELEASED_PAGES = [ - 'docs/2.0.0-rc5/mcp.md' + 'docs/2.0.0-rc6/mcp.md' ] const isUnreleased = (relativePath: string) => @@ -126,9 +126,9 @@ export default defineConfig({ { text: 'Docs', items: [ { - text: '2.0.0-rc5 (pre-release)', - link: '/docs/2.0.0-rc5/introduction', - activeMatch: '/docs/2.0.0-rc5/' + text: '2.0.0-rc6 (pre-release)', + link: '/docs/2.0.0-rc6/introduction', + activeMatch: '/docs/2.0.0-rc6/' }, { text: '1.8.0 (latest)', @@ -155,22 +155,22 @@ export default defineConfig({ ], sidebar: { - '/docs/2.0.0-rc5/': [ + '/docs/2.0.0-rc6/': [ { text: 'Overview', items: [ - { text: 'Introduction', link: '/docs/2.0.0-rc5/introduction' }, + { text: 'Introduction', link: '/docs/2.0.0-rc6/introduction' }, { text: 'Quick Start', - link: '/docs/2.0.0-rc5/quickstart', + link: '/docs/2.0.0-rc6/quickstart', collapsed: true, items: [ - { text: 'Deploy a server', link: '/docs/2.0.0-rc5/quickstart#deploy-a-server' }, - { text: 'Query a server', link: '/docs/2.0.0-rc5/quickstart#query-a-server' }, + { text: 'Deploy a server', link: '/docs/2.0.0-rc6/quickstart#deploy-a-server' }, + { text: 'Query a server', link: '/docs/2.0.0-rc6/quickstart#query-a-server' }, ] }, - { text: 'Concepts', link: '/docs/2.0.0-rc5/concepts' }, - { text: 'FAQ', link: '/docs/2.0.0-rc5/faq' }, + { text: 'Concepts', link: '/docs/2.0.0-rc6/concepts' }, + { text: 'FAQ', link: '/docs/2.0.0-rc6/faq' }, ] }, { @@ -179,72 +179,72 @@ export default defineConfig({ items: [ { text: 'Getting Started', - link: '/docs/2.0.0-rc5/getting-started', + link: '/docs/2.0.0-rc6/getting-started', collapsed: true, items: [ - { text: 'Quick Start', link: '/docs/2.0.0-rc5/getting-started#quick-start' }, - { text: 'Local', link: '/docs/2.0.0-rc5/getting-started#local' }, - { text: 'S3 / Object Storage', link: '/docs/2.0.0-rc5/getting-started#s3-compatible-object-storage' }, + { text: 'Quick Start', link: '/docs/2.0.0-rc6/getting-started#quick-start' }, + { text: 'Local', link: '/docs/2.0.0-rc6/getting-started#local' }, + { text: 'S3 / Object Storage', link: '/docs/2.0.0-rc6/getting-started#s3-compatible-object-storage' }, ] }, - { text: 'Configuration', link: '/docs/2.0.0-rc5/server/configuration' }, - { text: 'Access Control', link: '/docs/2.0.0-rc5/security/access-control' }, - { text: 'Performance Tuning', link: '/docs/2.0.0-rc5/server/performance-tuning' }, - { text: 'Storage internals', link: '/docs/2.0.0-rc5/internals/storage' }, - { text: 'File statistics', link: '/docs/2.0.0-rc5/internals/file-statistics' }, - { text: 'Troubleshooting', link: '/docs/2.0.0-rc5/troubleshooting' }, + { text: 'Configuration', link: '/docs/2.0.0-rc6/server/configuration' }, + { text: 'Access Control', link: '/docs/2.0.0-rc6/security/access-control' }, + { text: 'Performance Tuning', link: '/docs/2.0.0-rc6/server/performance-tuning' }, + { text: 'Storage internals', link: '/docs/2.0.0-rc6/internals/storage' }, + { text: 'File statistics', link: '/docs/2.0.0-rc6/internals/file-statistics' }, + { text: 'Troubleshooting', link: '/docs/2.0.0-rc6/troubleshooting' }, ] }, { text: 'Server Setup', collapsed: false, items: [ - { text: 'Overview', link: '/docs/2.0.0-rc5/server/' }, + { text: 'Overview', link: '/docs/2.0.0-rc6/server/' }, { text: 'Datasets & Formats', - link: '/docs/2.0.0-rc5/server/datasets', + link: '/docs/2.0.0-rc6/server/datasets', collapsed: true, items: [ - { text: 'All formats', link: '/docs/2.0.0-rc5/formats/' }, - { text: 'Inspect a schema', link: '/docs/2.0.0-rc5/formats/inspect-a-schema' }, - { text: 'Parquet', link: '/docs/2.0.0-rc5/formats/parquet' }, - { text: 'GeoParquet', link: '/docs/2.0.0-rc5/formats/geoparquet' }, - { text: 'CSV / TSV', link: '/docs/2.0.0-rc5/formats/csv' }, - { text: 'Arrow IPC', link: '/docs/2.0.0-rc5/formats/arrow' }, - { text: 'NetCDF', link: '/docs/2.0.0-rc5/formats/netcdf' }, - { text: 'HDF5', link: '/docs/2.0.0-rc5/formats/hdf5' }, - { text: 'Zarr', link: '/docs/2.0.0-rc5/formats/zarr' }, - { text: 'Atlas', link: '/docs/2.0.0-rc5/formats/atlas' }, - { text: 'GeoTIFF / COG', link: '/docs/2.0.0-rc5/formats/geotiff' }, - { text: 'BBF', link: '/docs/2.0.0-rc5/formats/bbf' }, - { text: 'Delta Lake', link: '/docs/2.0.0-rc5/formats/delta-lake' }, - { text: 'Apache Iceberg', link: '/docs/2.0.0-rc5/formats/iceberg' }, - { text: 'Icechunk', link: '/docs/2.0.0-rc5/formats/icechunk' }, - { text: 'ODV ASCII', link: '/docs/2.0.0-rc5/formats/odv' }, + { text: 'All formats', link: '/docs/2.0.0-rc6/formats/' }, + { text: 'Inspect a schema', link: '/docs/2.0.0-rc6/formats/inspect-a-schema' }, + { text: 'Parquet', link: '/docs/2.0.0-rc6/formats/parquet' }, + { text: 'GeoParquet', link: '/docs/2.0.0-rc6/formats/geoparquet' }, + { text: 'CSV / TSV', link: '/docs/2.0.0-rc6/formats/csv' }, + { text: 'Arrow IPC', link: '/docs/2.0.0-rc6/formats/arrow' }, + { text: 'NetCDF', link: '/docs/2.0.0-rc6/formats/netcdf' }, + { text: 'HDF5', link: '/docs/2.0.0-rc6/formats/hdf5' }, + { text: 'Zarr', link: '/docs/2.0.0-rc6/formats/zarr' }, + { text: 'Atlas', link: '/docs/2.0.0-rc6/formats/atlas' }, + { text: 'GeoTIFF / COG', link: '/docs/2.0.0-rc6/formats/geotiff' }, + { text: 'BBF', link: '/docs/2.0.0-rc6/formats/bbf' }, + { text: 'Delta Lake', link: '/docs/2.0.0-rc6/formats/delta-lake' }, + { text: 'Apache Iceberg', link: '/docs/2.0.0-rc6/formats/iceberg' }, + { text: 'Icechunk', link: '/docs/2.0.0-rc6/formats/icechunk' }, + { text: 'ODV ASCII', link: '/docs/2.0.0-rc6/formats/odv' }, ] }, { text: 'Tables & Views', - link: '/docs/2.0.0-rc5/data-sources/', + link: '/docs/2.0.0-rc6/data-sources/', collapsed: true, items: [ - { text: 'External Tables', link: '/docs/2.0.0-rc5/data-sources/external-tables' }, - { text: 'Managed Tables', link: '/docs/2.0.0-rc5/sql/managed-tables' }, - { text: 'Views', link: '/docs/2.0.0-rc5/server/view' }, - { text: 'Materialized Views', link: '/docs/2.0.0-rc5/sql/create-materialized-view' }, - { text: 'Crawlers', link: '/docs/2.0.0-rc5/server/crawlers' }, - { text: 'Extensions', link: '/docs/2.0.0-rc5/server/extensions' }, + { text: 'External Tables', link: '/docs/2.0.0-rc6/data-sources/external-tables' }, + { text: 'Managed Tables', link: '/docs/2.0.0-rc6/sql/managed-tables' }, + { text: 'Views', link: '/docs/2.0.0-rc6/server/view' }, + { text: 'Materialized Views', link: '/docs/2.0.0-rc6/sql/create-materialized-view' }, + { text: 'Crawlers', link: '/docs/2.0.0-rc6/server/crawlers' }, + { text: 'Extensions', link: '/docs/2.0.0-rc6/server/extensions' }, ] }, { text: 'Other Sources', collapsed: true, items: [ - { text: 'Object Storage (S3)', link: '/docs/2.0.0-rc5/data-sources/object-storage' }, - { text: 'SQL Databases', link: '/docs/2.0.0-rc5/data-sources/sql-databases' }, - { text: 'Remote Tables', link: '/docs/2.0.0-rc5/data-sources/remote-tables' }, - { text: 'ATTACH another server', link: '/docs/2.0.0-rc5/data-sources/attach' }, - { text: 'Secrets', link: '/docs/2.0.0-rc5/sql/secrets' }, + { text: 'Object Storage (S3)', link: '/docs/2.0.0-rc6/data-sources/object-storage' }, + { text: 'SQL Databases', link: '/docs/2.0.0-rc6/data-sources/sql-databases' }, + { text: 'Remote Tables', link: '/docs/2.0.0-rc6/data-sources/remote-tables' }, + { text: 'ATTACH another server', link: '/docs/2.0.0-rc6/data-sources/attach' }, + { text: 'Secrets', link: '/docs/2.0.0-rc6/sql/secrets' }, ] }, ] @@ -253,67 +253,67 @@ export default defineConfig({ text: 'SQL Reference', collapsed: true, items: [ - { text: 'Overview', link: '/docs/2.0.0-rc5/sql/' }, - { text: 'Identifiers & Case', link: '/docs/2.0.0-rc5/sql/identifiers' }, - { text: 'SELECT', link: '/docs/2.0.0-rc5/sql/select' }, - { text: 'WHERE', link: '/docs/2.0.0-rc5/sql/where' }, - { text: 'GROUP BY', link: '/docs/2.0.0-rc5/sql/group-by' }, - { text: 'JOIN', link: '/docs/2.0.0-rc5/sql/join' }, - { text: 'UNION BY NAME', link: '/docs/2.0.0-rc5/sql/union-by-name' }, - { text: 'CREATE TABLE', link: '/docs/2.0.0-rc5/sql/managed-tables' }, - { text: 'CREATE EXTERNAL TABLE', link: '/docs/2.0.0-rc5/sql/create-external-table' }, - { text: 'CREATE VIEW', link: '/docs/2.0.0-rc5/sql/create-view' }, - { text: 'CREATE MATERIALIZED VIEW', link: '/docs/2.0.0-rc5/sql/create-materialized-view' }, - { text: 'Remote Tables & ATTACH', link: '/docs/2.0.0-rc5/sql/remote-tables' }, - { text: 'CREATE SECRET', link: '/docs/2.0.0-rc5/sql/secrets' }, - { text: 'SUMMARIZE', link: '/docs/2.0.0-rc5/sql/summarize' }, - { text: 'Table Functions', link: '/docs/2.0.0-rc5/sql/table-functions' }, - { text: 'Introspection', link: '/docs/2.0.0-rc5/sql/table-functions-utility' }, - { text: 'Function Reference', link: '/docs/2.0.0-rc5/sql/function-reference' }, - { text: 'Spatial Functions', link: '/docs/2.0.0-rc5/sql/spatial-functions' }, + { text: 'Overview', link: '/docs/2.0.0-rc6/sql/' }, + { text: 'Identifiers & Case', link: '/docs/2.0.0-rc6/sql/identifiers' }, + { text: 'SELECT', link: '/docs/2.0.0-rc6/sql/select' }, + { text: 'WHERE', link: '/docs/2.0.0-rc6/sql/where' }, + { text: 'GROUP BY', link: '/docs/2.0.0-rc6/sql/group-by' }, + { text: 'JOIN', link: '/docs/2.0.0-rc6/sql/join' }, + { text: 'UNION BY NAME', link: '/docs/2.0.0-rc6/sql/union-by-name' }, + { text: 'CREATE TABLE', link: '/docs/2.0.0-rc6/sql/managed-tables' }, + { text: 'CREATE EXTERNAL TABLE', link: '/docs/2.0.0-rc6/sql/create-external-table' }, + { text: 'CREATE VIEW', link: '/docs/2.0.0-rc6/sql/create-view' }, + { text: 'CREATE MATERIALIZED VIEW', link: '/docs/2.0.0-rc6/sql/create-materialized-view' }, + { text: 'Remote Tables & ATTACH', link: '/docs/2.0.0-rc6/sql/remote-tables' }, + { text: 'CREATE SECRET', link: '/docs/2.0.0-rc6/sql/secrets' }, + { text: 'SUMMARIZE', link: '/docs/2.0.0-rc6/sql/summarize' }, + { text: 'Table Functions', link: '/docs/2.0.0-rc6/sql/table-functions' }, + { text: 'Introspection', link: '/docs/2.0.0-rc6/sql/table-functions-utility' }, + { text: 'Function Reference', link: '/docs/2.0.0-rc6/sql/function-reference' }, + { text: 'Spatial Functions', link: '/docs/2.0.0-rc6/sql/spatial-functions' }, ] }, { text: 'REST API', collapsed: true, items: [ - { text: 'Overview', link: '/docs/2.0.0-rc5/api/' }, - { text: 'Querying', link: '/docs/2.0.0-rc5/api/querying/' }, - { text: 'SQL Queries', link: '/docs/2.0.0-rc5/api/querying/sql' }, - { text: 'JSON Queries', link: '/docs/2.0.0-rc5/api/querying/json' }, - { text: 'Examples', link: '/docs/2.0.0-rc5/api/querying/examples' }, - { text: 'Exploring the catalog', link: '/docs/2.0.0-rc5/api/exploring-data' }, + { text: 'Overview', link: '/docs/2.0.0-rc6/api/' }, + { text: 'Querying', link: '/docs/2.0.0-rc6/api/querying/' }, + { text: 'SQL Queries', link: '/docs/2.0.0-rc6/api/querying/sql' }, + { text: 'JSON Queries', link: '/docs/2.0.0-rc6/api/querying/json' }, + { text: 'Examples', link: '/docs/2.0.0-rc6/api/querying/examples' }, + { text: 'Exploring the catalog', link: '/docs/2.0.0-rc6/api/exploring-data' }, ] }, { text: 'Connect', collapsed: true, items: [ - { text: 'Python SDK', link: '/docs/2.0.0-rc5/connect/python' }, - { text: 'TypeScript SDK', link: '/docs/2.0.0-rc5/connect/typescript' }, - { text: 'CLI', link: '/docs/2.0.0-rc5/connect/cli' }, - { text: 'Web Admin UI', link: '/docs/2.0.0-rc5/connect/web-admin-ui' }, - { text: 'DataGrip / JDBC', link: '/docs/2.0.0-rc5/connect/datagrip' }, - { text: 'Python ADBC', link: '/docs/2.0.0-rc5/connect/python-adbc' }, + { text: 'Python SDK', link: '/docs/2.0.0-rc6/connect/python' }, + { text: 'TypeScript SDK', link: '/docs/2.0.0-rc6/connect/typescript' }, + { text: 'CLI', link: '/docs/2.0.0-rc6/connect/cli' }, + { text: 'Web Admin UI', link: '/docs/2.0.0-rc6/connect/web-admin-ui' }, + { text: 'DataGrip / JDBC', link: '/docs/2.0.0-rc6/connect/datagrip' }, + { text: 'Python ADBC', link: '/docs/2.0.0-rc6/connect/python-adbc' }, ] }, { text: 'Scientific Data', collapsed: true, items: [ - { text: 'How It Works', link: '/docs/2.0.0-rc5/how-it-works' }, - { text: 'Arrays to tables', link: '/docs/2.0.0-rc5/arrays-to-tables' }, - { text: 'CF decoding', link: '/docs/2.0.0-rc5/cf-decoding' }, - { text: 'Coming from xarray', link: '/docs/2.0.0-rc5/coming-from-xarray' }, + { text: 'How It Works', link: '/docs/2.0.0-rc6/how-it-works' }, + { text: 'Arrays to tables', link: '/docs/2.0.0-rc6/arrays-to-tables' }, + { text: 'CF decoding', link: '/docs/2.0.0-rc6/cf-decoding' }, + { text: 'Coming from xarray', link: '/docs/2.0.0-rc6/coming-from-xarray' }, { text: 'Guides', - link: '/docs/2.0.0-rc5/guides/', + link: '/docs/2.0.0-rc6/guides/', collapsed: true, items: [ - { text: 'Query a File Collection', link: '/docs/2.0.0-rc5/guides/query-a-collection' }, - { text: 'Query Data on S3', link: '/docs/2.0.0-rc5/guides/query-s3' }, - { text: 'Export Query Results', link: '/docs/2.0.0-rc5/guides/export-results' }, - { text: 'Speed Up Slow Queries', link: '/docs/2.0.0-rc5/guides/speed-up-queries' }, + { text: 'Query a File Collection', link: '/docs/2.0.0-rc6/guides/query-a-collection' }, + { text: 'Query Data on S3', link: '/docs/2.0.0-rc6/guides/query-s3' }, + { text: 'Export Query Results', link: '/docs/2.0.0-rc6/guides/export-results' }, + { text: 'Speed Up Slow Queries', link: '/docs/2.0.0-rc6/guides/speed-up-queries' }, ] }, ] diff --git a/docs/.vitepress/theme/components/PreReleaseNotice.vue b/docs/.vitepress/theme/components/PreReleaseNotice.vue index 7faa8cf4..57bae581 100644 --- a/docs/.vitepress/theme/components/PreReleaseNotice.vue +++ b/docs/.vitepress/theme/components/PreReleaseNotice.vue @@ -6,7 +6,7 @@ import { LATEST_VERSION, LATEST_ENTRY } from '../version.js' // Versions whose docs are published but not released. Keep in sync with the // version dropdown label in config.mts. `route.path` carries the site base, so // match on the version segment rather than a full path. -const PRE_RELEASE = ['2.0.0-rc5'] +const PRE_RELEASE = ['2.0.0-rc6'] const route = useRoute() diff --git a/docs/.vitepress/theme/version.js b/docs/.vitepress/theme/version.js index dcf1c4be..7d8b72c8 100644 --- a/docs/.vitepress/theme/version.js +++ b/docs/.vitepress/theme/version.js @@ -5,7 +5,7 @@ // (config.mts imports this for the no-JS , and the theme imports // it for the client-side redirects). // -// This is the newest *stable* version, not the newest folder: 2.0.0-rc5 is a +// This is the newest *stable* version, not the newest folder: 2.0.0-rc6 is a // pre-release, so `/docs/latest` and the 404 fallback deliberately resolve to // 1.8.0. Bump to 2.0.0 when the RC goes GA. export const LATEST_VERSION = '1.8.0' diff --git a/docs/available-nodes/available-nodes.md b/docs/available-nodes/available-nodes.md index 3d3f5925..e8c546d1 100644 --- a/docs/available-nodes/available-nodes.md +++ b/docs/available-nodes/available-nodes.md @@ -2,7 +2,7 @@ ## Try it now: the World Ocean Database node - + One node is **open to everyone**. It needs no account and no token: @@ -24,7 +24,7 @@ curl -X POST https://beacon-wod.maris.nl/api/query \ | `temperature`, `salinity`, `oxygen` | float | Reads are anonymous and rate-limited. See the -[Quick Start](/docs/2.0.0-rc5/quickstart#query-a-public-node) for the Python client. +[Quick Start](/docs/2.0.0-rc6/quickstart#query-a-public-node) for the Python client. ## The Blue-Cloud and FAIR-EASE nodes diff --git a/docs/docs/2.0.0-rc5/api/exploring-data.md b/docs/docs/2.0.0-rc6/api/exploring-data.md similarity index 96% rename from docs/docs/2.0.0-rc5/api/exploring-data.md rename to docs/docs/2.0.0-rc6/api/exploring-data.md index 1544152f..3c7b83ba 100644 --- a/docs/docs/2.0.0-rc5/api/exploring-data.md +++ b/docs/docs/2.0.0-rc6/api/exploring-data.md @@ -127,7 +127,7 @@ A table outside `default_catalog`.`default_schema` needs its full name in SQL, f This endpoint and `/api/tables` answer per caller. The metadata schemas belong to the super-user only. Those schemas are `information_schema` and `beacon.system`. Every other caller sees only the tables with a `SELECT` grant from their roles. A listing therefore shows exactly what that caller -can read. See [Access control](/docs/2.0.0-rc5/security/access-control). +can read. See [Access control](/docs/2.0.0-rc6/security/access-control). ### Table schema @@ -200,9 +200,9 @@ function. endpoint therefore always returns an empty list. ::: -The [table function reference](/docs/2.0.0-rc5/sql/table-functions) holds every table +The [table function reference](/docs/2.0.0-rc6/sql/table-functions) holds every table function and its signature. The -[Function Reference](/docs/2.0.0-rc5/sql/function-reference) holds the other functions. +[Function Reference](/docs/2.0.0-rc6/sql/function-reference) holds the other functions. ## Table lifecycle @@ -255,7 +255,7 @@ GET /api/admin/check Authorization: Basic ``` -Create a crawler. This endpoint matches [`CREATE CRAWLER`](/docs/2.0.0-rc5/server/crawlers): +Create a crawler. This endpoint matches [`CREATE CRAWLER`](/docs/2.0.0-rc6/server/crawlers): ```http POST /api/admin/crawlers @@ -267,7 +267,7 @@ Content-Type: application/json ``` Create an external table. This endpoint matches -[`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table): +[`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table): ```http POST /api/admin/external-tables diff --git a/docs/docs/2.0.0-rc5/api/index.md b/docs/docs/2.0.0-rc6/api/index.md similarity index 83% rename from docs/docs/2.0.0-rc5/api/index.md rename to docs/docs/2.0.0-rc6/api/index.md index bf1967d7..9a86e212 100644 --- a/docs/docs/2.0.0-rc5/api/index.md +++ b/docs/docs/2.0.0-rc6/api/index.md @@ -41,7 +41,7 @@ to the endpoints that answer any caller on `/api/*`. `GET /api/info` is open. `GET /admin/api/info` is not. Use the alias when a proxy in front of Beacon protects `/api/*`. Your proxy keeps -control of `/api/*`. The [admin web UI](/docs/2.0.0-rc5/connect/web-admin-ui) calls +control of `/api/*`. The [admin web UI](/docs/2.0.0-rc6/connect/web-admin-ui) calls the alias only, so the UI stays in service. ```bash @@ -63,8 +63,8 @@ Returns `200 OK` when Beacon runs and is ready. | Section | Description | | ------- | ----------- | -| [Exploring the catalog](/docs/2.0.0-rc5/api/exploring-data) | Find datasets, tables and schemas | -| [Querying](/docs/2.0.0-rc5/api/querying/) | Run a query with the JSON DSL or with SQL, and get the results | -| [JSON Query DSL](/docs/2.0.0-rc5/api/querying/json) | A structured query format for a client program | -| [SQL](/docs/2.0.0-rc5/api/querying/sql) | Full SQL through DataFusion | -| [Examples](/docs/2.0.0-rc5/api/querying/examples) | Query patterns that you can copy | +| [Exploring the catalog](/docs/2.0.0-rc6/api/exploring-data) | Find datasets, tables and schemas | +| [Querying](/docs/2.0.0-rc6/api/querying/) | Run a query with the JSON DSL or with SQL, and get the results | +| [JSON Query DSL](/docs/2.0.0-rc6/api/querying/json) | A structured query format for a client program | +| [SQL](/docs/2.0.0-rc6/api/querying/sql) | Full SQL through DataFusion | +| [Examples](/docs/2.0.0-rc6/api/querying/examples) | Query patterns that you can copy | diff --git a/docs/docs/2.0.0-rc5/api/querying/examples.md b/docs/docs/2.0.0-rc6/api/querying/examples.md similarity index 100% rename from docs/docs/2.0.0-rc5/api/querying/examples.md rename to docs/docs/2.0.0-rc6/api/querying/examples.md diff --git a/docs/docs/2.0.0-rc5/api/querying/index.md b/docs/docs/2.0.0-rc6/api/querying/index.md similarity index 94% rename from docs/docs/2.0.0-rc5/api/querying/index.md rename to docs/docs/2.0.0-rc6/api/querying/index.md index f9e565df..7aac4e5e 100644 --- a/docs/docs/2.0.0-rc5/api/querying/index.md +++ b/docs/docs/2.0.0-rc6/api/querying/index.md @@ -11,8 +11,8 @@ The request body chooses one of two query styles: | Style | When to use | Body key | | ----- | ----------- | -------- | -| [JSON DSL](/docs/2.0.0-rc5/api/querying/json) | A client program or a query builder | `select`, `from`, `filters`, … | -| [SQL](/docs/2.0.0-rc5/api/querying/sql) | An expert user or an ad-hoc analysis | `sql` | +| [JSON DSL](/docs/2.0.0-rc6/api/querying/json) | A client program or a query builder | `select`, `from`, `filters`, … | +| [SQL](/docs/2.0.0-rc6/api/querying/sql) | An expert user or an ad-hoc analysis | `sql` | Both styles use the same `output` field. They also use the same support endpoints. @@ -180,5 +180,5 @@ Most queries use the name of a **registered table**: ``` Both styles also query files directly, without a registered table. See the -[JSON DSL `from` reference](/docs/2.0.0-rc5/api/querying/json#choosing-the-data-source-from) and the -[SQL table functions](/docs/2.0.0-rc5/api/querying/sql#query-files-directly). +[JSON DSL `from` reference](/docs/2.0.0-rc6/api/querying/json#choosing-the-data-source-from) and the +[SQL table functions](/docs/2.0.0-rc6/api/querying/sql#query-files-directly). diff --git a/docs/docs/2.0.0-rc5/api/querying/json.md b/docs/docs/2.0.0-rc6/api/querying/json.md similarity index 98% rename from docs/docs/2.0.0-rc5/api/querying/json.md rename to docs/docs/2.0.0-rc6/api/querying/json.md index 165032ea..8d956641 100644 --- a/docs/docs/2.0.0-rc5/api/querying/json.md +++ b/docs/docs/2.0.0-rc6/api/querying/json.md @@ -242,7 +242,7 @@ ST_Within( ``` The three functions come from the [spatial -functions](/docs/2.0.0-rc5/sql/spatial-functions) of the SQL path. A JSON filter and a SQL `WHERE` +functions](/docs/2.0.0-rc6/sql/spatial-functions) of the SQL path. A JSON filter and a SQL `WHERE` clause therefore state the same test under the same names. Any GeoJSON geometry type works, such as `Polygon`, `MultiPolygon` and `Point`. @@ -315,5 +315,5 @@ Content-Type: application/json ## Output formats -See [Querying, Output formats](/docs/2.0.0-rc5/api/querying/#output-formats) for the full list. The +See [Querying, Output formats](/docs/2.0.0-rc6/api/querying/#output-formats) for the full list. The `output` field is the same for a JSON DSL query and a SQL query. diff --git a/docs/docs/2.0.0-rc5/api/querying/sql.md b/docs/docs/2.0.0-rc6/api/querying/sql.md similarity index 93% rename from docs/docs/2.0.0-rc5/api/querying/sql.md rename to docs/docs/2.0.0-rc6/api/querying/sql.md index 4905f698..1aa14eed 100644 --- a/docs/docs/2.0.0-rc5/api/querying/sql.md +++ b/docs/docs/2.0.0-rc6/api/querying/sql.md @@ -45,7 +45,7 @@ GET /api/table-schema?table_name=default A table function queries files. You register no table first. Beacon resolves a path against its dataset root. A path also takes a glob pattern. -The [table function reference](/docs/2.0.0-rc5/sql/table-functions) holds every table +The [table function reference](/docs/2.0.0-rc6/sql/table-functions) holds every table function and its signature. `GET /api/table-functions` is deprecated. It returns an empty list. ### NetCDF @@ -98,12 +98,12 @@ Content-Type: application/json ``` The other table functions are `read_arrow`, `read_csv`, `read_odv_ascii`, `read_bbf` and -`read_tiff`. See [Read Files](/docs/2.0.0-rc5/sql/table-functions) for the full +`read_tiff`. See [Read Files](/docs/2.0.0-rc6/sql/table-functions) for the full signatures. ## Output formats -See [Querying, Output formats](/docs/2.0.0-rc5/api/querying/#output-formats) for the full list. Put +See [Querying, Output formats](/docs/2.0.0-rc6/api/querying/#output-formats) for the full list. Put `output` next to `sql` in the same request body: ```http diff --git a/docs/docs/2.0.0-rc5/arrays-to-tables.md b/docs/docs/2.0.0-rc6/arrays-to-tables.md similarity index 96% rename from docs/docs/2.0.0-rc5/arrays-to-tables.md rename to docs/docs/2.0.0-rc6/arrays-to-tables.md index 3f8e3aaa..aa198070 100644 --- a/docs/docs/2.0.0-rc5/arrays-to-tables.md +++ b/docs/docs/2.0.0-rc6/arrays-to-tables.md @@ -83,7 +83,7 @@ The row count grows as the product, so it grows fast: Beacon streams the rows and never materializes the whole grid. A filter still costs a scan of the selected chunks, so narrow the grid before you widen the query. See -[Speed up slow queries](/docs/2.0.0-rc5/guides/speed-up-queries). +[Speed up slow queries](/docs/2.0.0-rc6/guides/speed-up-queries). ::: tip This is not a cartesian product across variables The row count comes from **one** grid, not from every variable multiplied together. Two variables on @@ -198,8 +198,8 @@ not guess. ## Next -- [CF decoding](/docs/2.0.0-rc5/cf-decoding): what Beacon does with `scale_factor`, `_FillValue` +- [CF decoding](/docs/2.0.0-rc6/cf-decoding): what Beacon does with `scale_factor`, `_FillValue` and time units -- [Coming from xarray](/docs/2.0.0-rc5/coming-from-xarray): the same operations, side by side -- [NetCDF](/docs/2.0.0-rc5/formats/netcdf) and [Zarr](/docs/2.0.0-rc5/formats/zarr) for +- [Coming from xarray](/docs/2.0.0-rc6/coming-from-xarray): the same operations, side by side +- [NetCDF](/docs/2.0.0-rc6/formats/netcdf) and [Zarr](/docs/2.0.0-rc6/formats/zarr) for format-specific behaviour diff --git a/docs/docs/2.0.0-rc5/cf-decoding.md b/docs/docs/2.0.0-rc6/cf-decoding.md similarity index 95% rename from docs/docs/2.0.0-rc5/cf-decoding.md rename to docs/docs/2.0.0-rc6/cf-decoding.md index 2ec4fc48..7b6de1f2 100644 --- a/docs/docs/2.0.0-rc5/cf-decoding.md +++ b/docs/docs/2.0.0-rc6/cf-decoding.md @@ -25,7 +25,7 @@ acts on, and which it ignores. | `standard_name`, `long_name` | Beacon keeps each as a column. It applies neither | Every attribute also appears as its own column, decoded or not. See -[Attribute columns](/docs/2.0.0-rc5/arrays-to-tables#attribute-columns). +[Attribute columns](/docs/2.0.0-rc6/arrays-to-tables#attribute-columns). ## Time @@ -188,6 +188,6 @@ a column like any other attribute — `"temperature.units"` — and it changes n ## Next -- [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables): the row count and the grid rule -- [NetCDF](/docs/2.0.0-rc5/formats/netcdf) for format-specific behaviour -- [Troubleshooting](/docs/2.0.0-rc5/troubleshooting) for the errors these attributes cause +- [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables): the row count and the grid rule +- [NetCDF](/docs/2.0.0-rc6/formats/netcdf) for format-specific behaviour +- [Troubleshooting](/docs/2.0.0-rc6/troubleshooting) for the errors these attributes cause diff --git a/docs/docs/2.0.0-rc5/coming-from-xarray.md b/docs/docs/2.0.0-rc6/coming-from-xarray.md similarity index 93% rename from docs/docs/2.0.0-rc5/coming-from-xarray.md rename to docs/docs/2.0.0-rc6/coming-from-xarray.md index 46ea0158..ca205695 100644 --- a/docs/docs/2.0.0-rc5/coming-from-xarray.md +++ b/docs/docs/2.0.0-rc6/coming-from-xarray.md @@ -134,7 +134,7 @@ result = client.sql_query(""" ``` Beacon joins the files by column name. A file without `TEMP` gives `NULL` in that column. The read -does not fail. See [UNION BY NAME](/docs/2.0.0-rc5/sql/union-by-name). +does not fail. See [UNION BY NAME](/docs/2.0.0-rc6/sql/union-by-name). ## Translation table @@ -156,14 +156,14 @@ does not fail. See [UNION BY NAME](/docs/2.0.0-rc5/sql/union-by-name). ## Things that do not carry over - **A Beacon result is a table, not a `Dataset`.** The array shape is flattened into rows. See - [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables) for the row count and the broadcast rule. + [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables) for the row count and the broadcast rule. - **No `.to_xarray()` on the client yet.** Round-trip through `to_nd_netcdf()` and `xr.open_dataset()`, or build a `Dataset` from the dataframe with `df.set_index([...]).to_xarray()`. - **No lazy dask arrays.** The server streams the result instead. Aggregate or filter on the server. Then read back what fits in memory. - **Non-standard calendars are unsupported.** `noleap` and `360_day` files are rejected. See - [CF decoding](/docs/2.0.0-rc5/cf-decoding#supported-calendars). + [CF decoding](/docs/2.0.0-rc6/cf-decoding#supported-calendars). ## Keep both @@ -190,6 +190,6 @@ ds = xr.open_dataset("subset.nc") # back in xarray, on a subset that fits ## Next -- [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables): what a dimension does to your row count -- [CF decoding](/docs/2.0.0-rc5/cf-decoding): units, packing and fill values -- [Query a file collection](/docs/2.0.0-rc5/guides/query-a-collection): the full walkthrough +- [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables): what a dimension does to your row count +- [CF decoding](/docs/2.0.0-rc6/cf-decoding): units, packing and fill values +- [Query a file collection](/docs/2.0.0-rc6/guides/query-a-collection): the full walkthrough diff --git a/docs/docs/2.0.0-rc5/concepts.md b/docs/docs/2.0.0-rc6/concepts.md similarity index 83% rename from docs/docs/2.0.0-rc5/concepts.md rename to docs/docs/2.0.0-rc6/concepts.md index 2da4ca06..c2c55922 100644 --- a/docs/docs/2.0.0-rc5/concepts.md +++ b/docs/docs/2.0.0-rc6/concepts.md @@ -18,10 +18,10 @@ filters and projections down into the files that it scans. Beacon needs no load step. The **reader table functions** open your existing files and expose them as tables at query time. The files can be local or on S3. The functions include -[`read_parquet`](/docs/2.0.0-rc5/sql/table-functions), `read_netcdf`, `read_zarr`, +[`read_parquet`](/docs/2.0.0-rc6/sql/table-functions), `read_netcdf`, `read_zarr`, `read_csv` and `read_hdf5`. Supported formats include NetCDF, Zarr, Parquet, GeoParquet, CSV, ODV ASCII, GeoTIFF, Atlas, Arrow IPC and Delta Lake. See -[Supported formats](/docs/2.0.0-rc5/server/datasets). +[Supported formats](/docs/2.0.0-rc6/server/datasets). ```sql SELECT time, latitude, longitude, temperature @@ -43,7 +43,7 @@ them in the same way: - **Views** and **materialized views**: saved queries. - **Remote tables and catalogs**: tables on *another* Beacon. Beacon reaches them over Flight SQL and pushes filters, joins and aggregates down. See - [`ATTACH`](/docs/2.0.0-rc5/sql/remote-tables). + [`ATTACH`](/docs/2.0.0-rc6/sql/remote-tables). ## The `beacon.db` file @@ -51,26 +51,26 @@ Beacon holds its state in one file, `beacon.db`. The file holds everything that catalog and its managed data. The file *references* everything else: files on disk or S3, remote databases and remote Beacon servers. The file is one portable container, a redb object store. Copy the file and the owned state goes with it. See -[Storage internals](/docs/2.0.0-rc5/internals/storage). +[Storage internals](/docs/2.0.0-rc6/internals/storage). ## Secrets Beacon holds the credentials for another Beacon server as a named **secret**. Use `CREATE SECRET`, -then name it in [`ATTACH`](/docs/2.0.0-rc5/data-sources/attach). Beacon encrypts a persistent secret -into the `beacon.db` file. See [Secrets](/docs/2.0.0-rc5/sql/secrets). +then name it in [`ATTACH`](/docs/2.0.0-rc6/data-sources/attach). Beacon encrypts a persistent secret +into the `beacon.db` file. See [Secrets](/docs/2.0.0-rc6/sql/secrets). Storage credentials are not secrets. A server reads one store, local or one bucket, chosen at -startup from [configuration](/docs/2.0.0-rc5/server/configuration). +startup from [configuration](/docs/2.0.0-rc6/server/configuration). ## SQL and JSON You can write every query in **SQL**. The server also accepts an equal **JSON** query form over its HTTP API. The web UI and the SDKs send this payload. Both forms compile to the same plan. See the -[SQL reference](/docs/2.0.0-rc5/sql/) and the -[querying guide](/docs/2.0.0-rc5/api/querying/). +[SQL reference](/docs/2.0.0-rc6/sql/) and the +[querying guide](/docs/2.0.0-rc6/api/querying/). ## Auth Beacon has role-based access control with users, roles and grants. It controls who reads which table and which path. Grants deny first: a deny always wins over a grant. See -[Access Control](/docs/2.0.0-rc5/security/access-control). +[Access Control](/docs/2.0.0-rc6/security/access-control). diff --git a/docs/docs/2.0.0-rc5/connect/cli.md b/docs/docs/2.0.0-rc6/connect/cli.md similarity index 100% rename from docs/docs/2.0.0-rc5/connect/cli.md rename to docs/docs/2.0.0-rc6/connect/cli.md diff --git a/docs/docs/2.0.0-rc5/connect/datagrip.md b/docs/docs/2.0.0-rc6/connect/datagrip.md similarity index 100% rename from docs/docs/2.0.0-rc5/connect/datagrip.md rename to docs/docs/2.0.0-rc6/connect/datagrip.md diff --git a/docs/docs/2.0.0-rc5/connect/python-adbc.md b/docs/docs/2.0.0-rc6/connect/python-adbc.md similarity index 100% rename from docs/docs/2.0.0-rc5/connect/python-adbc.md rename to docs/docs/2.0.0-rc6/connect/python-adbc.md diff --git a/docs/docs/2.0.0-rc5/connect/python.md b/docs/docs/2.0.0-rc6/connect/python.md similarity index 100% rename from docs/docs/2.0.0-rc5/connect/python.md rename to docs/docs/2.0.0-rc6/connect/python.md diff --git a/docs/docs/2.0.0-rc5/connect/typescript.md b/docs/docs/2.0.0-rc6/connect/typescript.md similarity index 94% rename from docs/docs/2.0.0-rc5/connect/typescript.md rename to docs/docs/2.0.0-rc6/connect/typescript.md index cfe68c12..b133713c 100644 --- a/docs/docs/2.0.0-rc5/connect/typescript.md +++ b/docs/docs/2.0.0-rc6/connect/typescript.md @@ -6,7 +6,7 @@ description: "@beacon/client is a TypeScript SDK for Beacon. Run SQL or the JSON `@beacon/client` is a TypeScript and JavaScript SDK for a Beacon server. It runs in **Node.js 18 and later** and in the **browser**. It uses the global `fetch`. -It runs SQL and the [JSON query DSL](/docs/2.0.0-rc5/api/querying/json). It +It runs SQL and the [JSON query DSL](/docs/2.0.0-rc6/api/querying/json). It decodes the zstd-compressed Arrow IPC results of Beacon into plain JS row objects. It also gives a fluent query builder in the style of EF Core and LINQ. @@ -99,7 +99,7 @@ const beacon = new BeaconClient({ ``` The client sends the credentials on every request. The server checks them with -`GET /api/admin/check`. The [REST API reference](/docs/2.0.0-rc5/api/) lists every +`GET /api/admin/check`. The [REST API reference](/docs/2.0.0-rc6/api/) lists every admin endpoint. ## Call the admin path alias @@ -122,7 +122,7 @@ await beacon.tables(); // GET /admin/api/tables Use this when a proxy in front of Beacon protects `/api/*`. The alias needs the admin credentials for each endpoint. Read the -[admin path alias](/docs/2.0.0-rc5/api/#admin-path-alias) for the rules. +[admin path alias](/docs/2.0.0-rc6/api/#admin-path-alias) for the rules. `basePath` and `apiPrefix` are separate. `basePath` gives the prefix that the server runs under (`BEACON_BASE_PATH`). `apiPrefix` comes after it. @@ -132,5 +132,5 @@ server runs under (`BEACON_BASE_PATH`). `apiPrefix` comes after it. The SDK works with a bundler. It runs in the browser without a change. The default CORS policy of Beacon is `*`, and it allows the `Authorization` header. A browser application can therefore call the server directly. The -[Admin Web UI](/docs/2.0.0-rc5/connect/web-admin-ui) uses this SDK only. It is a +[Admin Web UI](/docs/2.0.0-rc6/connect/web-admin-ui) uses this SDK only. It is a good example. diff --git a/docs/docs/2.0.0-rc5/connect/web-admin-ui.md b/docs/docs/2.0.0-rc6/connect/web-admin-ui.md similarity index 94% rename from docs/docs/2.0.0-rc5/connect/web-admin-ui.md rename to docs/docs/2.0.0-rc6/connect/web-admin-ui.md index 6c6c57ef..111d9361 100644 --- a/docs/docs/2.0.0-rc5/connect/web-admin-ui.md +++ b/docs/docs/2.0.0-rc6/connect/web-admin-ui.md @@ -14,7 +14,7 @@ http://localhost:5001/admin The UI is a React single-page application. It uses Vite, Tailwind CSS and shadcn/ui. It reaches Beacon through the -[`@beacon/client`](/docs/2.0.0-rc5/connect/typescript) TypeScript SDK +[`@beacon/client`](/docs/2.0.0-rc6/connect/typescript) TypeScript SDK only. ## Log in @@ -68,7 +68,7 @@ your own authentication in front of it. ## Every call sits below `/admin` -The UI calls the [admin path alias](/docs/2.0.0-rc5/api/#admin-path-alias) of the +The UI calls the [admin path alias](/docs/2.0.0-rc6/api/#admin-path-alias) of the API. `/admin/api/query` runs the same handler as `/api/query`. Each request of the UI therefore uses the same prefix that serves the UI. @@ -95,14 +95,14 @@ controls who uses the UI. It keeps no secret. Serve Beacon over HTTPS. Give tree. You can also **save** a query in the browser and load it again. - **Tables**: browse the registered tables, their Arrow schemas and their configuration. Register an - [external table](/docs/2.0.0-rc5/data-sources/external-tables) over the + [external table](/docs/2.0.0-rc6/data-sources/external-tables) over the files in the datasets store. Drop a table with `DROP TABLE`. Beacon keeps the files. - **Datasets**: explore the dataset files that Beacon finds. Inspect the schema of each file. -- **Crawlers**: list, [create, run and delete crawlers](/docs/2.0.0-rc5/server/crawlers). +- **Crawlers**: list, [create, run and delete crawlers](/docs/2.0.0-rc6/server/crawlers). - **Users and roles**: manage the - [role-based access control](/docs/2.0.0-rc5/security/access-control). This covers + [role-based access control](/docs/2.0.0-rc6/security/access-control). This covers users, roles and privileges. - **Server**: the runtime information, the health and the available scalar and table functions. diff --git a/docs/docs/2.0.0-rc5/data-sources/attach.md b/docs/docs/2.0.0-rc6/data-sources/attach.md similarity index 81% rename from docs/docs/2.0.0-rc5/data-sources/attach.md rename to docs/docs/2.0.0-rc6/data-sources/attach.md index 4308750c..0c632c3a 100644 --- a/docs/docs/2.0.0-rc5/data-sources/attach.md +++ b/docs/docs/2.0.0-rc6/data-sources/attach.md @@ -7,11 +7,11 @@ description: ATTACH mirrors a whole remote Beacon catalog under a local name. Qu `ATTACH` mirrors the **whole** catalog of a remote Beacon under a local name. You can then query every remote schema and table as `name.schema.table`. You register no table one at a time. `ATTACH` is the catalog version of a single -[remote table](/docs/2.0.0-rc5/data-sources/remote-tables). Both use Arrow Flight SQL +[remote table](/docs/2.0.0-rc6/data-sources/remote-tables). Both use Arrow Flight SQL federation. Both push down filters, aggregates and joins between tables on that remote. Use `ATTACH` to query many tables from another Beacon, or to explore its catalog. Use a -[remote table](/docs/2.0.0-rc5/data-sources/remote-tables) when you need one table under a +[remote table](/docs/2.0.0-rc6/data-sources/remote-tables) when you need one table under a fixed local name. ## Attach a catalog @@ -32,8 +32,8 @@ DETACH lake; with `'tls' 'true'` or with an `https://` URL. - **Credentials**: give exactly one of three options. Use `'username'` and `'password'` for HTTP Basic. Use `'token'` for a bearer token. Use `'secret'` with the name of a - [`TYPE BEACON` secret](/docs/2.0.0-rc5/sql/secrets). Omit all three for anonymous access. - The remote applies its own [RBAC](/docs/2.0.0-rc5/security/access-control) to your identity. + [`TYPE BEACON` secret](/docs/2.0.0-rc6/sql/secrets). Omit all three for anonymous access. + The remote applies its own [RBAC](/docs/2.0.0-rc6/security/access-control) to your identity. - Beacon contacts the remote at `ATTACH` time. It lists the schemas and tables as a snapshot. It resolves the schema of each table at the first use. @@ -43,7 +43,7 @@ A query over attached tables pushes down like a remote table. Beacon runs the fi columns, the `LIMIT` and whole aggregates on the remote. It also runs a join between two tables of the **same** attached catalog on the remote. Only the reduced result travels over the network. Use `EXPLAIN` to check what Beacon pushes down. See -[Remote Tables](/docs/2.0.0-rc5/data-sources/remote-tables#how-pushdown-works) for the +[Remote Tables](/docs/2.0.0-rc6/data-sources/remote-tables#how-pushdown-works) for the details and the limitations. They apply here in the same way. ## Detach a catalog @@ -58,5 +58,5 @@ server. ## From a client `ATTACH` runs on the server, so any client can issue it. Send it as an ordinary statement from the -[Python client](/docs/2.0.0-rc5/connect/python), the [CLI](/docs/2.0.0-rc5/connect/cli) or the HTTP -API. See [Remote Tables](/docs/2.0.0-rc5/sql/remote-tables) for the table-level equivalent. +[Python client](/docs/2.0.0-rc6/connect/python), the [CLI](/docs/2.0.0-rc6/connect/cli) or the HTTP +API. See [Remote Tables](/docs/2.0.0-rc6/sql/remote-tables) for the table-level equivalent. diff --git a/docs/docs/2.0.0-rc5/data-sources/external-tables.md b/docs/docs/2.0.0-rc6/data-sources/external-tables.md similarity index 85% rename from docs/docs/2.0.0-rc5/data-sources/external-tables.md rename to docs/docs/2.0.0-rc6/data-sources/external-tables.md index a7b9a2a8..483a6b98 100644 --- a/docs/docs/2.0.0-rc5/data-sources/external-tables.md +++ b/docs/docs/2.0.0-rc6/data-sources/external-tables.md @@ -14,19 +14,19 @@ survives a restart. :::tip External vs managed tables An **external table** points at existing files. Beacon reads them in place. Beacon never writes to them. Do you need a table that Beacon **owns** and can change with `INSERT`, `UPDATE` and `DELETE`? -Then use a [managed table](/docs/2.0.0-rc5/sql/managed-tables). +Then use a [managed table](/docs/2.0.0-rc6/sql/managed-tables). ::: :::tip Register many datasets at once -A [crawler](/docs/2.0.0-rc5/server/crawlers) finds and registers many datasets under a prefix. It +A [crawler](/docs/2.0.0-rc6/server/crawlers) finds and registers many datasets under a prefix. It also handles partitioned layouts. You write no DDL for each dataset. ::: This page is a **setup guide** with an example for each format. The -[`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table) reference gives the full +[`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table) reference gives the full statement grammar and every clause: `OR REPLACE`, `IF NOT EXISTS`, `PARTITIONED BY`, `OPTIONS` and `DROP TABLE`. -[Keys of each format](/docs/2.0.0-rc5/sql/create-external-table#keys-of-each-format) lists the +[Keys of each format](/docs/2.0.0-rc6/sql/create-external-table#keys-of-each-format) lists the `OPTIONS` keys that each `STORED AS` value reads. You can send the DDL through any SQL interface of Beacon: @@ -75,7 +75,7 @@ LOCATION 'spatial/stations/*.geoparquet' ``` Beacon decodes the geometry columns to native [GeoArrow](https://geoarrow.org/) on read. See -[GeoParquet](/docs/2.0.0-rc5/formats/geoparquet) for the details. +[GeoParquet](/docs/2.0.0-rc6/formats/geoparquet) for the details. ### NetCDF @@ -121,7 +121,7 @@ STORED AS ATLAS LOCATION 'collections/*/data.atlas' ``` -See [Atlas](/docs/2.0.0-rc5/formats/atlas) for the format details, its `OPTIONS` and its column +See [Atlas](/docs/2.0.0-rc6/formats/atlas) for the format details, its `OPTIONS` and its column naming. That page also explains how Atlas speeds up NetCDF and Zarr work. ### CSV @@ -143,7 +143,7 @@ LOCATION 'cruises/' ### ODV ASCII ODV ASCII is **not** an external table format. There is no `STORED AS ODV`. Read an ODV file with -the [`read_odv_ascii()`](/docs/2.0.0-rc5/sql/table-functions#read-odv-ascii) table function. +the [`read_odv_ascii()`](/docs/2.0.0-rc6/sql/table-functions#read-odv-ascii) table function. The `odv` source in the JSON query API also works: ```sql @@ -161,7 +161,7 @@ LOCATION 'rasters/elevation.tif' ### Delta Lake `STORED AS DELTA` registers an existing -[Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) table. The `LOCATION` points +[Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) table. The `LOCATION` points at the Delta **table directory**. That directory holds `_delta_log/`. Do not give a glob of files: ```sql @@ -172,12 +172,12 @@ LOCATION 'delta/ocean_profiles' A Delta table supports consistent snapshots. It supports **time travel** through `OPTIONS ('version' '12')` or `('timestamp' '…')`. It also supports `INSERT INTO`, which commits a -new Delta version. See [Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) for the +new Delta version. See [Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) for the full reference. ### Apache Iceberg -`STORED AS ICEBERG` registers an [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) +`STORED AS ICEBERG` registers an [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) table that already exists. `LOCATION` gives the Iceberg **table directory**. That directory holds `metadata/`. Give no glob of files: @@ -190,11 +190,11 @@ LOCATION 'iceberg/ocean_profiles' Beacon **reads** an Iceberg table. Another system writes the table. Beacon reads each new snapshot and each new column. You restart nothing. `OPTIONS ('snapshot_id' '…')` pins the table to one snapshot. Beacon writes no Iceberg table, so the table accepts no `INSERT INTO`. See -[Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) for the full reference. +[Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) for the full reference. ### Icechunk -`STORED AS ICECHUNK` adds an [Icechunk](/docs/2.0.0-rc5/formats/icechunk) repository to the +`STORED AS ICECHUNK` adds an [Icechunk](/docs/2.0.0-rc6/formats/icechunk) repository to the catalog. A repository is a Zarr store with commits, branches, tags and snapshots. The repository must exist. @@ -210,33 +210,33 @@ OPTIONS ('branch' 'main') `OPTIONS` selects the version. Use `branch`, `tag` or `snapshot`. Set one of the three. A tag and a snapshot do not move. The table reads the same rows after a later commit. -The table is read only. See [Icechunk](/docs/2.0.0-rc5/formats/icechunk) for the full reference. +The table is read only. See [Icechunk](/docs/2.0.0-rc6/formats/icechunk) for the full reference. ## Partitioned data Your files can use Hive-style partition directories such as `year=2024/month=01/...`. Declare the partition columns. Beacon can then prune them at query time. The directory names hold the values. The columns become normal queryable columns. See -[`PARTITIONED BY`](/docs/2.0.0-rc5/sql/create-external-table#partitioned-by) for the syntax. +[`PARTITIONED BY`](/docs/2.0.0-rc6/sql/create-external-table#partitioned-by) for the syntax. ## Remote tables `STORED AS REMOTE` registers a table on **another Beacon server** instead of local files. A query pushes the filters, the projection, the limits and whole joins and aggregates down to the remote server over Arrow Flight SQL. See -[Remote Tables (Federation)](/docs/2.0.0-rc5/data-sources/remote-tables) for the full +[Remote Tables (Federation)](/docs/2.0.0-rc6/data-sources/remote-tables) for the full reference. ## Views A view holds a persistent SQL query over any external table or table function. See the -[Views](/docs/2.0.0-rc5/server/view) page for the full reference. It also covers +[Views](/docs/2.0.0-rc6/server/view) page for the full reference. It also covers `UNION ALL BY NAME` for datasets with different schemas. ## Remove a table `DROP TABLE` removes an external table from the catalog. Beacon does **not** delete the files. See -[`DROP TABLE`](/docs/2.0.0-rc5/sql/create-external-table#drop-table). +[`DROP TABLE`](/docs/2.0.0-rc6/sql/create-external-table#drop-table). ## List and inspect tables @@ -252,5 +252,5 @@ Inspect the columns and data types of a table: GET /api/table-schema?table_name=ocean_profiles ``` -The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table#querying-and-inspecting) +The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table#querying-and-inspecting) reference gives the SQL equivalents, `SHOW TABLES` and `DESCRIBE`. diff --git a/docs/docs/2.0.0-rc5/data-sources/index.md b/docs/docs/2.0.0-rc6/data-sources/index.md similarity index 78% rename from docs/docs/2.0.0-rc5/data-sources/index.md rename to docs/docs/2.0.0-rc6/data-sources/index.md index 8858aeab..087082c2 100644 --- a/docs/docs/2.0.0-rc5/data-sources/index.md +++ b/docs/docs/2.0.0-rc6/data-sources/index.md @@ -6,17 +6,17 @@ description: Everything that Beacon can query. Files on disk or object storage, Beacon queries data where it lives. It reads most sources **in place**, with no import step. Point at a path or a connection. Then query it with SQL. Beacon can also own the data. It then uses -[its own internal format](/docs/2.0.0-rc5/internals/storage). +[its own internal format](/docs/2.0.0-rc6/internals/storage). | Source | Read with | Chapter | | --- | --- | --- | -| Files on local disk | `read_*()` functions or external tables | [File Formats](/docs/2.0.0-rc5/formats/) | -| Datasets on an S3 bucket | the same relative paths; the store is set at startup | [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage) | -| A named set of files | `CREATE EXTERNAL TABLE` | [External Tables](/docs/2.0.0-rc5/data-sources/external-tables) | -| Postgres, MySQL, ODBC | federated external tables | [SQL Databases](/docs/2.0.0-rc5/data-sources/sql-databases) | -| One table on another Beacon | `STORED AS REMOTE` | [Remote Tables](/docs/2.0.0-rc5/data-sources/remote-tables) | -| A whole remote Beacon catalog | `ATTACH` | [ATTACH](/docs/2.0.0-rc5/data-sources/attach) | -| Data that Beacon owns and can change | managed tables in `beacon.db` | [Storage internals](/docs/2.0.0-rc5/internals/storage) | +| Files on local disk | `read_*()` functions or external tables | [File Formats](/docs/2.0.0-rc6/formats/) | +| Datasets on an S3 bucket | the same relative paths; the store is set at startup | [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage) | +| A named set of files | `CREATE EXTERNAL TABLE` | [External Tables](/docs/2.0.0-rc6/data-sources/external-tables) | +| Postgres, MySQL, ODBC | federated external tables | [SQL Databases](/docs/2.0.0-rc6/data-sources/sql-databases) | +| One table on another Beacon | `STORED AS REMOTE` | [Remote Tables](/docs/2.0.0-rc6/data-sources/remote-tables) | +| A whole remote Beacon catalog | `ATTACH` | [ATTACH](/docs/2.0.0-rc6/data-sources/attach) | +| Data that Beacon owns and can change | managed tables in `beacon.db` | [Storage internals](/docs/2.0.0-rc6/internals/storage) | ## Read files @@ -38,13 +38,13 @@ SELECT * FROM read_csv(['a.csv', 'b.csv']); A glob (`*`, `**`) expands across directories. One query can therefore cover thousands of files. Beacon merges their schemas. It also prunes the files that cannot match your filters. Array formats -such as [Zarr](/docs/2.0.0-rc5/formats/zarr) and -[Atlas](/docs/2.0.0-rc5/formats/atlas) are not plain files. Point at Zarr's `zarr.json` marker, +such as [Zarr](/docs/2.0.0-rc6/formats/zarr) and +[Atlas](/docs/2.0.0-rc6/formats/atlas) are not plain files. Point at Zarr's `zarr.json` marker, and at the `data.atlas` file of an Atlas collection, not at the chunks. Each format has its own chapter. The chapter covers the read behaviour, the attribute columns and -the limitations. See [File Formats](/docs/2.0.0-rc5/formats/) for the full -list. The [table functions reference](/docs/2.0.0-rc5/sql/table-functions) holds every +the limitations. See [File Formats](/docs/2.0.0-rc6/formats/) for the full +list. The [table functions reference](/docs/2.0.0-rc6/sql/table-functions) holds every signature in one place. ## Functions or tables? @@ -52,7 +52,7 @@ signature in one place. Both read the same files in the same way. Only the name makes the difference: - A **`read_*()` call** fits ad-hoc queries, notebooks and exploration. -- An **[external table](/docs/2.0.0-rc5/data-sources/external-tables)** puts a stable name +- An **[external table](/docs/2.0.0-rc6/data-sources/external-tables)** puts a stable name in the catalog. Use it when many queries share one source. It also lets you give a colleague a table name instead of a glob. @@ -79,7 +79,7 @@ GROUP BY o.platform, s.station_name; ``` Some files share a concept but have different columns. Combine those files with -[`UNION BY NAME`](/docs/2.0.0-rc5/sql/union-by-name). +[`UNION BY NAME`](/docs/2.0.0-rc6/sql/union-by-name). ## Where paths resolve @@ -87,8 +87,8 @@ A path points at local disk or at object storage: - **Local disk**, for example `argo/**/*.nc`. Beacon resolves the path against its storage root. - **The datasets store**, whether that is a local directory or an S3 bucket. Paths are relative to its root either way, for example `obs/*.parquet`. See - [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). + [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). Beacon holds the access keys for object stores and remote Beacons as named, scoped -[secrets](/docs/2.0.0-rc5/sql/secrets). Secrets replace scattered environment variables. +[secrets](/docs/2.0.0-rc6/sql/secrets). Secrets replace scattered environment variables. Beacon can store a secret encrypted inside the `beacon.db` file. diff --git a/docs/docs/2.0.0-rc5/data-sources/object-storage.md b/docs/docs/2.0.0-rc6/data-sources/object-storage.md similarity index 87% rename from docs/docs/2.0.0-rc5/data-sources/object-storage.md rename to docs/docs/2.0.0-rc6/data-sources/object-storage.md index 54c015b5..74fbe5f0 100644 --- a/docs/docs/2.0.0-rc5/data-sources/object-storage.md +++ b/docs/docs/2.0.0-rc6/data-sources/object-storage.md @@ -32,7 +32,7 @@ matches nothing. Write relative paths. Always. ::: -The same holds for [external tables](/docs/2.0.0-rc5/data-sources/external-tables): +The same holds for [external tables](/docs/2.0.0-rc6/data-sources/external-tables): ```sql CREATE EXTERNAL TABLE ocean_profiles @@ -74,14 +74,14 @@ Beacon then finds every file in that bucket automatically, exactly as it does in Beacon exits at startup when `BEACON_S3_DATASETS` has no bucket. It does not wait for the first query to fail. -See [Configuration](/docs/2.0.0-rc5/server/configuration#s3-object-storage) for every option. It +See [Configuration](/docs/2.0.0-rc6/server/configuration#s3-object-storage) for every option. It also covers virtual-hosted addressing and plain HTTP endpoints for a local MinIO. ::: info There is no SQL statement for storage credentials Credentials belong to the store. You choose the store at startup. The credentials therefore come -from the environment. [`CREATE SECRET`](/docs/2.0.0-rc5/sql/secrets) covers one case only. It holds +from the environment. [`CREATE SECRET`](/docs/2.0.0-rc6/sql/secrets) covers one case only. It holds the credentials for **another Beacon server**, which you reach with -[`ATTACH`](/docs/2.0.0-rc5/data-sources/attach). +[`ATTACH`](/docs/2.0.0-rc6/data-sources/attach). ::: ## Data that a server does not own @@ -95,21 +95,21 @@ SELECT * FROM wod."easy-wod" LIMIT 10; ``` The remote server runs its own scan. It then streams the result back. See -[ATTACH](/docs/2.0.0-rc5/data-sources/attach) and -[Remote Tables](/docs/2.0.0-rc5/sql/remote-tables). +[ATTACH](/docs/2.0.0-rc6/data-sources/attach) and +[Remote Tables](/docs/2.0.0-rc6/sql/remote-tables). ## Performance notes - **Cloud-optimized formats are faster.** Parquet, - [Zarr](/docs/2.0.0-rc5/formats/zarr), - [Atlas](/docs/2.0.0-rc5/formats/atlas) and Cloud-Optimized GeoTIFF support + [Zarr](/docs/2.0.0-rc6/formats/zarr), + [Atlas](/docs/2.0.0-rc6/formats/atlas) and Cloud-Optimized GeoTIFF support range requests. Beacon fetches only the chunks that a query needs. - **NetCDF and HDF5 need an anonymous bucket on the netCDF-c reader.** netCDF-c opens a file by URL and does not use the credential chain. Keep the default `BEACON_NETCDF_USE_RUST_READER=true` and `BEACON_HDF5_USE_RUST_READER=true`, or set `AWS_SKIP_SIGNATURE=true`. The pure-Rust reader fetches byte ranges through the object store, so a private bucket works and no local copy is made. See - [NetCDF](/docs/2.0.0-rc5/formats/netcdf) and [HDF5](/docs/2.0.0-rc5/formats/hdf5). -- **A large bucket is slow to list.** Register a [crawler](/docs/2.0.0-rc5/server/crawlers). It + [NetCDF](/docs/2.0.0-rc6/formats/netcdf) and [HDF5](/docs/2.0.0-rc6/formats/hdf5). +- **A large bucket is slow to list.** Register a [crawler](/docs/2.0.0-rc6/server/crawlers). It builds the file list before a query needs it. - **Use narrow filters.** Predicate and projection pushdown reduce the bytes Beacon fetches. See - [Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning). + [Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning). diff --git a/docs/docs/2.0.0-rc5/data-sources/remote-tables.md b/docs/docs/2.0.0-rc6/data-sources/remote-tables.md similarity index 96% rename from docs/docs/2.0.0-rc5/data-sources/remote-tables.md rename to docs/docs/2.0.0-rc6/data-sources/remote-tables.md index 18a38899..7add58ff 100644 --- a/docs/docs/2.0.0-rc5/data-sources/remote-tables.md +++ b/docs/docs/2.0.0-rc6/data-sources/remote-tables.md @@ -24,9 +24,9 @@ authentication error. Anonymous Flight SQL access is read-only. Federation needs :::tip External vs managed vs remote -- An [**external table**](/docs/2.0.0-rc5/data-sources/external-tables) reads files in the +- An [**external table**](/docs/2.0.0-rc6/data-sources/external-tables) reads files in the storage of Beacon, in place. -- A [**managed table**](/docs/2.0.0-rc5/sql/managed-tables) belongs to Beacon. You can +- A [**managed table**](/docs/2.0.0-rc6/sql/managed-tables) belongs to Beacon. You can change the rows with `INSERT`, `UPDATE` and `DELETE`. - A **remote table** holds no local data. It is a federated pointer at a table on another Beacon. Beacon queries it on demand. @@ -128,7 +128,7 @@ GET /api/tables GET /api/table-schema?table_name=remote_profiles ``` -The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table#querying-and-inspecting) +The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table#querying-and-inspecting) reference gives the SQL equivalents, `SHOW TABLES` and `DESCRIBE`. ## Remove a remote table diff --git a/docs/docs/2.0.0-rc5/data-sources/sql-databases.md b/docs/docs/2.0.0-rc6/data-sources/sql-databases.md similarity index 95% rename from docs/docs/2.0.0-rc5/data-sources/sql-databases.md rename to docs/docs/2.0.0-rc6/data-sources/sql-databases.md index 7d2f786b..80a3c96b 100644 --- a/docs/docs/2.0.0-rc5/data-sources/sql-databases.md +++ b/docs/docs/2.0.0-rc6/data-sources/sql-databases.md @@ -23,16 +23,16 @@ result travels over the network. This feature uses [`datafusion-table-providers`](https://github.com/datafusion-contrib/datafusion-table-providers) and the federation layer of DataFusion. -[Remote tables](/docs/2.0.0-rc5/data-sources/remote-tables) use the same pushdown +[Remote tables](/docs/2.0.0-rc6/data-sources/remote-tables) use the same pushdown mechanism. :::tip External vs managed vs remote vs database -- An [**external table**](/docs/2.0.0-rc5/data-sources/external-tables) reads files in the +- An [**external table**](/docs/2.0.0-rc6/data-sources/external-tables) reads files in the storage of Beacon, in place. -- A [**managed table**](/docs/2.0.0-rc5/sql/managed-tables) belongs to Beacon. You can +- A [**managed table**](/docs/2.0.0-rc6/sql/managed-tables) belongs to Beacon. You can change the rows with `INSERT`, `UPDATE` and `DELETE`. -- A [**remote table**](/docs/2.0.0-rc5/data-sources/remote-tables) is a federated pointer at +- A [**remote table**](/docs/2.0.0-rc6/data-sources/remote-tables) is a federated pointer at a table on another Beacon server. - A **SQL database table** is a federated pointer at a table in an external PostgreSQL or MySQL database. Beacon queries it on demand. It is read-only. diff --git a/docs/docs/2.0.0-rc5/faq.md b/docs/docs/2.0.0-rc6/faq.md similarity index 82% rename from docs/docs/2.0.0-rc5/faq.md rename to docs/docs/2.0.0-rc6/faq.md index c7b70ac6..bcf6d7bc 100644 --- a/docs/docs/2.0.0-rc5/faq.md +++ b/docs/docs/2.0.0-rc6/faq.md @@ -12,28 +12,28 @@ follow your task. ### Do I have to run a server to try Beacon? No. Query a public node first. Install the Python client, point it at the node and run SQL. See -[Query the public node](/docs/2.0.0-rc5/quickstart#query-the-public-node). +[Query the public node](/docs/2.0.0-rc6/quickstart#query-the-public-node). Run your own server when the data is yours, or when you need access control, a web UI or crawlers. ### Do I need to import or convert my files first? No. Beacon reads files in place. Point a -[`read_*()` function](/docs/2.0.0-rc5/formats/) or an -[external table](/docs/2.0.0-rc5/data-sources/external-tables) at a path. Then query it. +[`read_*()` function](/docs/2.0.0-rc6/formats/) or an +[external table](/docs/2.0.0-rc6/data-sources/external-tables) at a path. Then query it. Beacon copies nothing. ### Which formats are supported? Parquet, GeoParquet, CSV, TSV, Arrow IPC, NetCDF, Zarr, Atlas, GeoTIFF, COG, BBF, Delta Lake and ODV ASCII. Each format has its own chapter in -[File Formats](/docs/2.0.0-rc5/formats/). +[File Formats](/docs/2.0.0-rc6/formats/). ## Read files ### How do I see what columns and types a file has? -[`read__schema()`](/docs/2.0.0-rc5/sql/table-functions-utility#read-format-schema) returns the +[`read__schema()`](/docs/2.0.0-rc6/sql/table-functions-utility#read-format-schema) returns the schema. It reads no data: ```sql @@ -41,13 +41,13 @@ SELECT * FROM read_netcdf_schema('argo/**/*.nc'); ``` It covers `parquet`, `netcdf`, `zarr`, `atlas`, `arrow`, `csv`, `bbf` and `tiff`. For GeoParquet, -Delta Lake and ODV, use a `LIMIT 0` query. [`SUMMARIZE`](/docs/2.0.0-rc5/sql/summarize) also +Delta Lake and ODV, use a `LIMIT 0` query. [`SUMMARIZE`](/docs/2.0.0-rc6/sql/summarize) also works. ### My files have different columns The query fails or returns nulls. Use -[`UNION BY NAME`](/docs/2.0.0-rc5/sql/union-by-name) to combine the files by column name, +[`UNION BY NAME`](/docs/2.0.0-rc6/sql/union-by-name) to combine the files by column name, not by position. Beacon sets a missing column to null. The columns stay aligned. Some NetCDF collections mix variables with different dimensions. Give an explicit dimension list. @@ -67,15 +67,15 @@ marker, not at the store directory or the chunks. These three formats are the exception. Beacon finds every other format in the dataset store automatically. Beacon does not find ODV ASCII, Delta Lake and Apache Iceberg. Point -[`read_odv_ascii()`](/docs/2.0.0-rc5/formats/odv) or -[`read_delta()`](/docs/2.0.0-rc5/formats/delta-lake) or -[`read_iceberg()`](/docs/2.0.0-rc5/formats/iceberg) at the data. For Delta and Iceberg, +[`read_odv_ascii()`](/docs/2.0.0-rc6/formats/odv) or +[`read_delta()`](/docs/2.0.0-rc6/formats/delta-lake) or +[`read_iceberg()`](/docs/2.0.0-rc6/formats/iceberg) at the data. For Delta and Iceberg, you can also create an external table with `CREATE EXTERNAL TABLE … STORED AS DELTA|ICEBERG`. ### Is there a `STORED AS ODV`? No. ODV ASCII has no external table form. Read it with `read_odv_ascii()`. Wrap the call in a -[view](/docs/2.0.0-rc5/server/view) to get a stable name. +[view](/docs/2.0.0-rc6/server/view) to get a stable name. ## Object storage and credentials @@ -83,13 +83,13 @@ No. ODV ASCII has no external table form. Read it with `read_odv_ascii()`. Wrap Use an `s3://` path. For a public bucket, set `AWS_SKIP_SIGNATURE=true`. You need nothing else. For a private bucket, store the credentials as a named -[secret](/docs/2.0.0-rc5/sql/secrets): +[secret](/docs/2.0.0-rc6/sql/secrets): ```sql CREATE SECRET my_s3 (TYPE S3, KEY_ID '…', SECRET '…', REGION 'eu-west-1', SCOPE 's3://my-bucket'); ``` -See [Query Data on S3](/docs/2.0.0-rc5/guides/query-s3). +See [Query Data on S3](/docs/2.0.0-rc6/guides/query-s3). ### Beacon ignores my region setting @@ -99,8 +99,8 @@ Beacon reads `AWS_REGION`. It does **not** use `AWS_DEFAULT_REGION`. NetCDF on object storage supports **anonymous access only**. Beacon cannot yet do an authenticated S3 read for NetCDF. Make the objects public, or copy them to local disk. You can also convert the -collection to [Zarr](/docs/2.0.0-rc5/formats/zarr) or -[Atlas](/docs/2.0.0-rc5/formats/atlas). Both formats have full object storage +collection to [Zarr](/docs/2.0.0-rc6/formats/zarr) or +[Atlas](/docs/2.0.0-rc6/formats/atlas). Both formats have full object storage support. ### `CREATE PERSISTENT SECRET` fails @@ -122,8 +122,8 @@ scan prunes nothing. The usual fixes come in this order of impact. Select fewer columns. Filter on coordinate columns. Do not put a predicate inside a function that the reader cannot interpret. Merge large NetCDF or Zarr -collections into [Atlas](/docs/2.0.0-rc5/formats/atlas). See -[Speed Up Slow Queries](/docs/2.0.0-rc5/guides/speed-up-queries) for the full detail. +collections into [Atlas](/docs/2.0.0-rc6/formats/atlas). See +[Speed Up Slow Queries](/docs/2.0.0-rc6/guides/speed-up-queries) for the full detail. ### My spatial filter on GeoParquet prunes nothing @@ -133,7 +133,7 @@ Five predicates state a bounding box, and only those skip row groups: `ST_Inters Rewrite the filter as one of the five. `EXPLAIN ANALYZE` reports `geoparquet_row_groups_pruned`, which tells you if the rewrite worked. See -[what the scan skips](/docs/2.0.0-rc5/formats/geoparquet). +[what the scan skips](/docs/2.0.0-rc6/formats/geoparquet). ### My query runs out of memory @@ -152,7 +152,7 @@ while (batch := cursor.fetch_record_batch().read_next_batch()) is not None: On the server, a query can spill to disk. The spill goes to the OS temp area. Put that area on fast storage with free space. See -[Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning). +[Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning). ## The `beacon.db` file @@ -160,20 +160,20 @@ storage with free space. See No. Beacon holds the file under an **exclusive lock**. One server opens one `beacon.db`, so two servers cannot share a data directory. For access from several processes or machines, run one -[server](/docs/2.0.0-rc5/getting-started) and connect every client to it. +[server](/docs/2.0.0-rc6/getting-started) and connect every client to it. ### Does a copy of `beacon.db` include my data? A copy includes everything that Beacon **owns**: the catalog and the managed table data. A copy does not include the external files. Beacon must still reach those files from the new location. See -[Storage internals](/docs/2.0.0-rc5/internals/storage). +[Storage internals](/docs/2.0.0-rc6/internals/storage). ### When should I use a managed table instead of an external table? Use an external table to read data that you already have. Use a managed table when Beacon owns the rows. A managed table also gives you `INSERT`, `UPDATE` and `DELETE`. For a cached result with a periodic refresh, use a -[materialized view](/docs/2.0.0-rc5/sql/create-materialized-view). +[materialized view](/docs/2.0.0-rc6/sql/create-materialized-view). ### Can I write query results to S3? @@ -192,13 +192,13 @@ SQL. The `BEACON_ADMIN_*` credentials protect the admin UI and the write operations. They do not protect reads. To control who reads data, set `BEACON_AUTH_ENFORCE=true`. See -[Access Control](/docs/2.0.0-rc5/security/access-control). +[Access Control](/docs/2.0.0-rc6/security/access-control). ### My remote table fails with an authentication error A remote table connects **anonymously** and stores no credentials. The remote server must allow anonymous Flight SQL access. Set `BEACON_FLIGHT_SQL_ALLOW_ANONYMOUS=true` on the remote server. For -an authenticated connection, use [`ATTACH`](/docs/2.0.0-rc5/data-sources/attach). It accepts +an authenticated connection, use [`ATTACH`](/docs/2.0.0-rc6/data-sources/attach). It accepts a user name and password, a token or a secret. ### My remote table still shows the old columns @@ -215,7 +215,7 @@ comparisons in the predicates that must push down. ### Beacon does not find new files automatically Beacon does not watch storage for changes. It reads no local file system events and no S3 events. -Use a [crawler](/docs/2.0.0-rc5/server/crawlers) with a schedule to find new files. +Use a [crawler](/docs/2.0.0-rc6/server/crawlers) with a schedule to find new files. ## Python @@ -227,17 +227,17 @@ Use a [crawler](/docs/2.0.0-rc5/server/crawlers) with a schedule to find new fil pip install beacon-api ``` -See [Python client](/docs/2.0.0-rc5/connect/python). For a terminal instead of a notebook, install -[`beacon-datalake-cli`](/docs/2.0.0-rc5/connect/cli). +See [Python client](/docs/2.0.0-rc6/connect/python). For a terminal instead of a notebook, install +[`beacon-datalake-cli`](/docs/2.0.0-rc6/connect/cli). ### Can I connect a SQL tool instead? Yes. The server speaks Arrow Flight SQL and JDBC. See -[DataGrip / JDBC](/docs/2.0.0-rc5/connect/datagrip) and -[Python ADBC](/docs/2.0.0-rc5/connect/python-adbc). +[DataGrip / JDBC](/docs/2.0.0-rc6/connect/datagrip) and +[Python ADBC](/docs/2.0.0-rc6/connect/python-adbc). --- Do you still have a problem? Open an issue on [GitHub](https://github.com/maris-development/beacon). Or read -[Concepts](/docs/2.0.0-rc5/concepts) for the full picture. +[Concepts](/docs/2.0.0-rc6/concepts) for the full picture. diff --git a/docs/docs/2.0.0-rc5/formats/arrow.md b/docs/docs/2.0.0-rc6/formats/arrow.md similarity index 79% rename from docs/docs/2.0.0-rc5/formats/arrow.md rename to docs/docs/2.0.0-rc6/formats/arrow.md index 57e1c711..e7878cb0 100644 --- a/docs/docs/2.0.0-rc5/formats/arrow.md +++ b/docs/docs/2.0.0-rc6/formats/arrow.md @@ -24,7 +24,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_arrow('cruises/*.arrow') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -40,10 +40,10 @@ STORED AS ARROW LOCATION 'cruises/' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` `STORED AS ARROW` reads no key. Beacon ignores an `OPTIONS` clause on this format. See -[`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the formats that do read one. +[`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the formats that do read one. diff --git a/docs/docs/2.0.0-rc5/formats/atlas.md b/docs/docs/2.0.0-rc6/formats/atlas.md similarity index 93% rename from docs/docs/2.0.0-rc5/formats/atlas.md rename to docs/docs/2.0.0-rc6/formats/atlas.md index 20d54d6a..436d8f18 100644 --- a/docs/docs/2.0.0-rc5/formats/atlas.md +++ b/docs/docs/2.0.0-rc6/formats/atlas.md @@ -35,7 +35,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_atlas('collections/sensor/data.atlas') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -102,8 +102,8 @@ Atlas stores its own types, including a native nanosecond timestamp, so **Beacon decoding to a collection**. The ingest path does it instead: `atlas create` reads each NetCDF file with xarray, which applies `scale_factor`, `add_offset` and the CF time units before the write. An array therefore reads back exactly as it is stored. This is the one place Atlas differs from -[NetCDF](/docs/2.0.0-rc5/formats/netcdf) and [Zarr](/docs/2.0.0-rc5/formats/zarr) — see -[CF decoding](/docs/2.0.0-rc5/cf-decoding). +[NetCDF](/docs/2.0.0-rc6/formats/netcdf) and [Zarr](/docs/2.0.0-rc6/formats/zarr) — see +[CF decoding](/docs/2.0.0-rc6/cf-decoding). A cell nobody wrote reads as the array's fill value, and the fill reads as null. Two consequences are worth knowing: @@ -127,7 +127,7 @@ Incompatible types for field 'value': Utf8 in 'obs/data.atlas#a' vs Int64 in 'ob ``` Set `BEACON_TYPE_WIDENING_ON_CONFLICT=keep_first` to take the first dataset's type instead. See -[Configuration](/docs/2.0.0-rc5/server/configuration#query-engine). +[Configuration](/docs/2.0.0-rc6/server/configuration#query-engine). ### Building a collection @@ -157,7 +157,7 @@ scan of the original files. :::tip Cache a large, repeated aggregation with a -[materialized view](/docs/2.0.0-rc5/sql/create-materialized-view). This works over an Atlas +[materialized view](/docs/2.0.0-rc6/sql/create-materialized-view). This works over an Atlas collection and over any other table. Run `REFRESH` when the source data changes. ::: @@ -179,8 +179,8 @@ STORED AS ATLAS LOCATION 'collections/*/data.atlas' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See -[Data Sources](/docs/2.0.0-rc5/data-sources/) for the full read model. +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See +[Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` @@ -197,6 +197,6 @@ LOCATION 'collections/*/data.atlas' OPTIONS ('read_dimensions' 'time,lat,lon') ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every -key. See [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables#the-dimensions-argument) for the grid +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every +key. See [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables#the-dimensions-argument) for the grid rule. diff --git a/docs/docs/2.0.0-rc5/formats/bbf.md b/docs/docs/2.0.0-rc6/formats/bbf.md similarity index 86% rename from docs/docs/2.0.0-rc5/formats/bbf.md rename to docs/docs/2.0.0-rc6/formats/bbf.md index 989e9b36..56e4bffd 100644 --- a/docs/docs/2.0.0-rc5/formats/bbf.md +++ b/docs/docs/2.0.0-rc6/formats/bbf.md @@ -24,7 +24,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_bbf('data/*.bbf') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -48,7 +48,7 @@ STORED AS BBF LOCATION 'path/to/files'; ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` @@ -66,4 +66,4 @@ LOCATION 'path/to/files' OPTIONS ('split_streams_slice' 'true') ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. diff --git a/docs/docs/2.0.0-rc5/formats/csv.md b/docs/docs/2.0.0-rc6/formats/csv.md similarity index 88% rename from docs/docs/2.0.0-rc5/formats/csv.md rename to docs/docs/2.0.0-rc6/formats/csv.md index dd6cf617..07625632 100644 --- a/docs/docs/2.0.0-rc5/formats/csv.md +++ b/docs/docs/2.0.0-rc6/formats/csv.md @@ -32,7 +32,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_csv('stations/*.csv') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -49,7 +49,7 @@ STORED AS CSV LOCATION 'metadata/stations/' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` @@ -69,4 +69,4 @@ LOCATION 'metadata/stations/' OPTIONS ('delimiter' '\t', 'infer_records' '5000') ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. diff --git a/docs/docs/2.0.0-rc5/formats/delta-lake.md b/docs/docs/2.0.0-rc6/formats/delta-lake.md similarity index 92% rename from docs/docs/2.0.0-rc5/formats/delta-lake.md rename to docs/docs/2.0.0-rc6/formats/delta-lake.md index 6e5b3755..69e5a506 100644 --- a/docs/docs/2.0.0-rc5/formats/delta-lake.md +++ b/docs/docs/2.0.0-rc6/formats/delta-lake.md @@ -17,9 +17,9 @@ dataset storage root, like every other source. :::tip External vs managed vs Delta -- An [**external table**](/docs/2.0.0-rc5/data-sources/external-tables) (`STORED AS +- An [**external table**](/docs/2.0.0-rc6/data-sources/external-tables) (`STORED AS PARQUET`, `NETCDF`, …) reads a folder or a glob of files in place. It is read-only. -- A [**managed table**](/docs/2.0.0-rc5/sql/managed-tables) belongs to Beacon. Lance holds +- A [**managed table**](/docs/2.0.0-rc6/sql/managed-tables) belongs to Beacon. Lance holds the data. You can change the rows with `INSERT`, `UPDATE` and `DELETE`. - A **Delta table** points at an existing Delta Lake table directory. Beacon reads it in place. It supports consistent snapshots and time travel. It accepts `INSERT INTO`, which commits a new Delta @@ -38,7 +38,7 @@ with any Delta writer, such as delta-rs or Spark. Then register it in Beacon. ### Ad-hoc with `read_delta` Query a Delta table directly in a `FROM` clause. You register nothing first. This helps during -exploration. See the [`read_delta`](/docs/2.0.0-rc5/sql/table-functions#read-delta) table +exploration. See the [`read_delta`](/docs/2.0.0-rc6/sql/table-functions#read-delta) table function. ```sql @@ -94,7 +94,7 @@ Pin the table to an older snapshot. Use **one** of these options: | `timestamp` | RFC-3339 timestamp | The latest committed version | Beacon takes the last version at or before it, for example `'2026-01-01T00:00:00Z'`. | `version` wins if you set both. See -[`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. +[`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. ```sql -- Register the table as it looked at version 12 @@ -140,7 +140,7 @@ on both backends: - **Object storage**: under the configured bucket and prefix. You need no Delta configuration. Beacon reads the table location like any other dataset path. See -[Configuration](/docs/2.0.0-rc5/server/configuration) for the storage setup. +[Configuration](/docs/2.0.0-rc6/server/configuration) for the storage setup. ## Query and inspect @@ -151,7 +151,7 @@ GET /api/tables GET /api/table-schema?table_name=ocean_profiles ``` -The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table#querying-and-inspecting) +The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table#querying-and-inspecting) reference gives the SQL equivalents, `SHOW TABLES` and `DESCRIBE`. ## Remove a Delta table @@ -169,7 +169,7 @@ DROP TABLE ocean_profiles; list. Beacon does not support `CREATE TABLE AS … STORED AS DELTA`. Use a managed table or an external Delta writer. - **`INSERT` appends only.** Beacon does not expose `UPDATE`, `DELETE` or `MERGE` on a Delta table. - Use a [managed table](/docs/2.0.0-rc5/sql/managed-tables) to change rows. + Use a [managed table](/docs/2.0.0-rc6/sql/managed-tables) to change rows. - **One directory per table.** A Delta table maps to one table directory with `_delta_log/`. It is not a glob over many tables. @@ -181,5 +181,5 @@ Check the columns and the types before you write a query: SELECT * FROM read_delta('delta/ocean_profiles') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. diff --git a/docs/docs/2.0.0-rc5/formats/geoparquet.md b/docs/docs/2.0.0-rc6/formats/geoparquet.md similarity index 88% rename from docs/docs/2.0.0-rc5/formats/geoparquet.md rename to docs/docs/2.0.0-rc6/formats/geoparquet.md index d61baf39..c254a224 100644 --- a/docs/docs/2.0.0-rc5/formats/geoparquet.md +++ b/docs/docs/2.0.0-rc6/formats/geoparquet.md @@ -26,7 +26,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_geoparquet('spatial/**/*.geoparquet') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -41,14 +41,14 @@ and a `geo` metadata key. Beacon reads and writes this format. - The reader works on local disk and on S3-compatible object stores. Query a GeoParquet file with the -[`read_geoparquet()`](/docs/2.0.0-rc5/sql/table-functions#read-geoparquet) table function: +[`read_geoparquet()`](/docs/2.0.0-rc6/sql/table-functions#read-geoparquet) table function: ```sql SELECT * FROM read_geoparquet(['spatial/**/*.geoparquet']) LIMIT 100 ``` You can also register a stable table name with an -[external table](/docs/2.0.0-rc5/data-sources/external-tables): +[external table](/docs/2.0.0-rc6/data-sources/external-tables): ```sql CREATE EXTERNAL TABLE stations @@ -70,7 +70,7 @@ FROM stations ### Spatial filters -Use the [spatial functions](/docs/2.0.0-rc5/sql/spatial-functions). Those +Use the [spatial functions](/docs/2.0.0-rc6/sql/spatial-functions). Those functions carry PostGIS names. Build the geometry from the two coordinate columns: ```sql @@ -102,7 +102,7 @@ WHERE ST_Intersects(geometry, ST_GeomFromText('POLYGON((3 51, 5 51, 5 53, 3 53, :::tip Beacon also *writes* GeoParquet. It maps the longitude and latitude columns of a query result into a -geometry column on output. See [output formats](/docs/2.0.0-rc5/api/querying/). +geometry column on output. See [output formats](/docs/2.0.0-rc6/api/querying/). ::: ## What the scan skips @@ -133,13 +133,13 @@ LOCATION 'spatial/stations/*.geoparquet' ``` Beacon decodes the geometry columns to native [GeoArrow](https://geoarrow.org/) on read. See -[GeoParquet in Supported Formats](/docs/2.0.0-rc5/server/datasets#supported-formats) for the +[GeoParquet in Supported Formats](/docs/2.0.0-rc6/server/datasets#supported-formats) for the details. -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` `STORED AS GEOPARQUET` reads no key. Beacon ignores an `OPTIONS` clause on this format. See -[`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the formats that do read one. +[`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the formats that do read one. diff --git a/docs/docs/2.0.0-rc5/formats/geotiff.md b/docs/docs/2.0.0-rc6/formats/geotiff.md similarity index 86% rename from docs/docs/2.0.0-rc5/formats/geotiff.md rename to docs/docs/2.0.0-rc6/formats/geotiff.md index f067cea1..341f6e0a 100644 --- a/docs/docs/2.0.0-rc5/formats/geotiff.md +++ b/docs/docs/2.0.0-rc6/formats/geotiff.md @@ -24,7 +24,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_tiff('rasters/*.tif') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -57,10 +57,10 @@ STORED AS TIFF LOCATION 'rasters/elevation.tif' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` `STORED AS TIFF` reads no key. Beacon ignores an `OPTIONS` clause on this format. See -[`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the formats that do read one. +[`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the formats that do read one. diff --git a/docs/docs/2.0.0-rc5/formats/hdf5.md b/docs/docs/2.0.0-rc6/formats/hdf5.md similarity index 91% rename from docs/docs/2.0.0-rc5/formats/hdf5.md rename to docs/docs/2.0.0-rc6/formats/hdf5.md index d679cf95..492f5f1c 100644 --- a/docs/docs/2.0.0-rc5/formats/hdf5.md +++ b/docs/docs/2.0.0-rc6/formats/hdf5.md @@ -15,9 +15,9 @@ Beacon recognizes `.h5` and `.hdf5`. It finds them in the dataset store automati A netCDF-4 file **is** an HDF5 file, and the netCDF-c library also opens plain HDF5. Beacon reads HDF5 through the pure-Rust reader by default. Beacon reads it through that library when you set the Rust reader to false. HDF5 behaves exactly like -[NetCDF](/docs/2.0.0-rc5/formats/netcdf): the same data model, the same -[array to table mapping](/docs/2.0.0-rc5/arrays-to-tables), the same -[CF decoding](/docs/2.0.0-rc5/cf-decoding) and the same attribute columns. +[NetCDF](/docs/2.0.0-rc6/formats/netcdf): the same data model, the same +[array to table mapping](/docs/2.0.0-rc6/arrays-to-tables), the same +[CF decoding](/docs/2.0.0-rc6/cf-decoding) and the same attribute columns. The **pure-Rust HDF5 reader** is the default. It reads the same files and gives the same answer for a netCDF-4 file, and it adds five things over netCDF-c: @@ -27,7 +27,7 @@ a netCDF-4 file, and it adds five things over netCDF-c: | Nested groups | Root group only | Every group | | Compound datasets | Not read | One column for each member | | Object storage | Anonymous access only | Full credential chain, no local copy | -| [File statistics](/docs/2.0.0-rc5/internals/file-statistics) | None | Per-file column ranges | +| [File statistics](/docs/2.0.0-rc6/internals/file-statistics) | None | Per-file column ranges | | Concurrent scans | One file at a time | In parallel | | Writes | netCDF-c | netCDF-c | @@ -130,9 +130,9 @@ the server, to keep one dimension per length per group instead. A file that names no dimension also picks its `SELECT *` grid by volume rather than by variable count, so a query lands on the payload rather than on the metadata around it. See -[Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables#a-file-that-names-no-dimension). +[Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables#a-file-that-names-no-dimension). -See [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables#the-dimensions-argument). +See [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables#the-dimensions-argument). ## Inspect the schema @@ -142,7 +142,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_hdf5_schema('experiments/**/*.h5'); ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## As an external table @@ -177,7 +177,7 @@ LOCATION 'acquisition/*.hdf5' OPTIONS ('unify_phony_dimensions' 'false') ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. ## On object storage @@ -194,7 +194,7 @@ STORED AS HDF5 LOCATION 's3://bucket/experiments/'; ``` -See [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +See [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). ## As a query output @@ -216,8 +216,8 @@ header padding as part of the value. The datasets are unaffected. This is ## See also -- [NetCDF](/docs/2.0.0-rc5/formats/netcdf): the same readers, with the full detail -- [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables): the row count and the grid rule -- [CF decoding](/docs/2.0.0-rc5/cf-decoding): units, packing and fill values -- [Performance tuning](/docs/2.0.0-rc5/server/performance-tuning#hdf5-pure-rust-reader): when to +- [NetCDF](/docs/2.0.0-rc6/formats/netcdf): the same readers, with the full detail +- [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables): the row count and the grid rule +- [CF decoding](/docs/2.0.0-rc6/cf-decoding): units, packing and fill values +- [Performance tuning](/docs/2.0.0-rc6/server/performance-tuning#hdf5-pure-rust-reader): when to change the reader diff --git a/docs/docs/2.0.0-rc5/formats/iceberg.md b/docs/docs/2.0.0-rc6/formats/iceberg.md similarity index 92% rename from docs/docs/2.0.0-rc5/formats/iceberg.md rename to docs/docs/2.0.0-rc6/formats/iceberg.md index 9993a961..80c701bf 100644 --- a/docs/docs/2.0.0-rc5/formats/iceberg.md +++ b/docs/docs/2.0.0-rc6/formats/iceberg.md @@ -21,9 +21,9 @@ the datasets root. Every other source works the same way. :::tip External, managed and Iceberg tables -- An [**external table**](/docs/2.0.0-rc5/data-sources/external-tables) reads a directory or a glob +- An [**external table**](/docs/2.0.0-rc6/data-sources/external-tables) reads a directory or a glob of files in place. It is read-only. -- A [**managed table**](/docs/2.0.0-rc5/sql/managed-tables) belongs to Beacon. Lance holds the data. +- A [**managed table**](/docs/2.0.0-rc6/sql/managed-tables) belongs to Beacon. Lance holds the data. Change the rows with `INSERT`, `UPDATE` and `DELETE`. - An **Iceberg table** points to an Iceberg table directory. Beacon reads the table in place. Beacon reads each new snapshot. @@ -40,7 +40,7 @@ table in Beacon. ### Direct read with `read_iceberg` Read an Iceberg table in a `FROM` clause. You register nothing first. Use this form for one query. -See the [`read_iceberg`](/docs/2.0.0-rc5/sql/table-functions#read-iceberg) table function. +See the [`read_iceberg`](/docs/2.0.0-rc6/sql/table-functions#read-iceberg) table function. ```sql SELECT count(*) FROM read_iceberg('iceberg/ocean_profiles'); @@ -105,7 +105,7 @@ Pin the table to one snapshot: | --- | --- | --- | --- | | `snapshot_id` | Whole number | The current snapshot | An Iceberg snapshot id, for example `'3821550127947089060'`. | -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. ```sql -- Read the table at one snapshot @@ -157,7 +157,7 @@ backends: - **Object storage**: under the configured bucket and prefix. Beacon makes no local copy. Beacon needs no Iceberg configuration. Beacon needs no second set of credentials. See -[Configuration](/docs/2.0.0-rc5/server/configuration) for the storage setup. +[Configuration](/docs/2.0.0-rc6/server/configuration) for the storage setup. ## Table information @@ -168,7 +168,7 @@ GET /api/tables GET /api/table-schema?table_name=ocean_profiles ``` -The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table#querying-and-inspecting) +The [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table#querying-and-inspecting) reference gives the SQL forms, `SHOW TABLES` and `DESCRIBE`. ## Remove an Iceberg table @@ -182,7 +182,7 @@ DROP TABLE ocean_profiles; ## Limitations - **Read-only.** Beacon writes no Iceberg table. `INSERT`, `UPDATE`, `DELETE`, `MERGE` and snapshot - expiry fail. Use a [managed table](/docs/2.0.0-rc5/sql/managed-tables) to change rows. You also + expiry fail. Use a [managed table](/docs/2.0.0-rc6/sql/managed-tables) to change rows. You also write with Spark or PyIceberg. - **No catalog.** A location names a table. Beacon supports no REST catalog and no Glue catalog. Give the directory of the table. @@ -199,5 +199,5 @@ Read the columns and the types before you write a query: SELECT * FROM read_iceberg_schema('iceberg/ocean_profiles'); ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`. It also gives the cost of each one. diff --git a/docs/docs/2.0.0-rc5/formats/icechunk.md b/docs/docs/2.0.0-rc6/formats/icechunk.md similarity index 94% rename from docs/docs/2.0.0-rc5/formats/icechunk.md rename to docs/docs/2.0.0-rc6/formats/icechunk.md index 019b0a23..56ffbdc6 100644 --- a/docs/docs/2.0.0-rc5/formats/icechunk.md +++ b/docs/docs/2.0.0-rc6/formats/icechunk.md @@ -9,7 +9,7 @@ SELECT * FROM read_icechunk('sst/repo'); ``` An **Icechunk repository** is a Zarr v3 store with versions. It holds the same arrays, groups and -attributes as a plain [Zarr](/docs/2.0.0-rc5/formats/zarr) store. It adds commits, branches, tags +attributes as a plain [Zarr](/docs/2.0.0-rc6/formats/zarr) store. It adds commits, branches, tags and snapshots. Beacon reads a repository through the Zarr reader. The schema, the arrays and the chunk selection @@ -89,7 +89,7 @@ LOCATION 'sst/repo' OPTIONS ('tag' 'v1', 'read_dimensions' 'time,lat,lon'); ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. A table on a branch reads new data for each query. Beacon reads the branch tip again for each scan. A table on a tag or on a snapshot reads the same rows each time. @@ -106,7 +106,7 @@ arguments as `read_icechunk`. It gives one row for each column. It reads no data SELECT * FROM read_icechunk_schema('sst/repo'); ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`. It gives the cost of each one. ## Predicate pushdown @@ -120,7 +120,7 @@ FROM read_icechunk('sst/repo') WHERE lat BETWEEN 40 AND 45 AND time >= '2024-01-01'; ``` -[Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables) shows how Beacon changes an array into rows. +[Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables) shows how Beacon changes an array into rows. ## Storage @@ -134,7 +134,7 @@ A `location` without a scheme resolves against the datasets store: - A local datasets store reads from disk. - An S3 datasets store reads over HTTP without a signature. The bucket must allow anonymous reads. - [NetCDF](/docs/2.0.0-rc5/formats/netcdf) works the same way. + [NetCDF](/docs/2.0.0-rc6/formats/netcdf) works the same way. - Give an `s3://` location to read a private bucket. ## Virtual chunk references diff --git a/docs/docs/2.0.0-rc5/formats/index.md b/docs/docs/2.0.0-rc6/formats/index.md similarity index 75% rename from docs/docs/2.0.0-rc5/formats/index.md rename to docs/docs/2.0.0-rc6/formats/index.md index dc15d1af..9a0aa8c9 100644 --- a/docs/docs/2.0.0-rc5/formats/index.md +++ b/docs/docs/2.0.0-rc6/formats/index.md @@ -20,26 +20,26 @@ SELECT * FROM read_csv(['a.csv', 'b.csv']); Beacon resolves a path against its storage root. That root is either a local directory or one S3-compatible bucket, chosen at startup. See -[Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +[Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). ## Formats | Format | Function | `STORED AS` | Recognized files | | --- | --- | --- | --- | -| [Parquet](/docs/2.0.0-rc5/formats/parquet) | `read_parquet` | `PARQUET` | `.parquet` | -| [GeoParquet](/docs/2.0.0-rc5/formats/geoparquet) | `read_geoparquet` | `GEOPARQUET` | `.geoparquet` | -| [CSV / TSV](/docs/2.0.0-rc5/formats/csv) | `read_csv` | `CSV` | `.csv`, `.tsv` | -| [Arrow IPC](/docs/2.0.0-rc5/formats/arrow) | `read_arrow` | `ARROW` | `.arrow`, `.feather` | -| [NetCDF](/docs/2.0.0-rc5/formats/netcdf) | `read_netcdf` | `NC` | `.nc` | -| [HDF5](/docs/2.0.0-rc5/formats/hdf5) | `read_hdf5` | `HDF5`, `H5` | `.h5`, `.hdf5` | -| [Zarr](/docs/2.0.0-rc5/formats/zarr) | `read_zarr` | `ZARR` | `zarr.json` marker | -| [Atlas](/docs/2.0.0-rc5/formats/atlas) | `read_atlas` | `ATLAS` | `data.atlas` file | -| [GeoTIFF / COG](/docs/2.0.0-rc5/formats/geotiff) | `read_tiff` | `TIFF` | `.tif`, `.tiff` | -| [BBF](/docs/2.0.0-rc5/formats/bbf) | `read_bbf` | `BBF` | `.bbf` | -| [Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) | `read_delta` | `DELTA` | `_delta_log/` directory | -| [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) | `read_iceberg` | `ICEBERG` | `metadata/` directory | -| [Icechunk](/docs/2.0.0-rc5/formats/icechunk) | `read_icechunk` | `ICECHUNK` | repository directory | -| [ODV ASCII](/docs/2.0.0-rc5/formats/odv) | `read_odv_ascii` | not supported | `.txt` | +| [Parquet](/docs/2.0.0-rc6/formats/parquet) | `read_parquet` | `PARQUET` | `.parquet` | +| [GeoParquet](/docs/2.0.0-rc6/formats/geoparquet) | `read_geoparquet` | `GEOPARQUET` | `.geoparquet` | +| [CSV / TSV](/docs/2.0.0-rc6/formats/csv) | `read_csv` | `CSV` | `.csv`, `.tsv` | +| [Arrow IPC](/docs/2.0.0-rc6/formats/arrow) | `read_arrow` | `ARROW` | `.arrow`, `.feather` | +| [NetCDF](/docs/2.0.0-rc6/formats/netcdf) | `read_netcdf` | `NC` | `.nc` | +| [HDF5](/docs/2.0.0-rc6/formats/hdf5) | `read_hdf5` | `HDF5`, `H5` | `.h5`, `.hdf5` | +| [Zarr](/docs/2.0.0-rc6/formats/zarr) | `read_zarr` | `ZARR` | `zarr.json` marker | +| [Atlas](/docs/2.0.0-rc6/formats/atlas) | `read_atlas` | `ATLAS` | `data.atlas` file | +| [GeoTIFF / COG](/docs/2.0.0-rc6/formats/geotiff) | `read_tiff` | `TIFF` | `.tif`, `.tiff` | +| [BBF](/docs/2.0.0-rc6/formats/bbf) | `read_bbf` | `BBF` | `.bbf` | +| [Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) | `read_delta` | `DELTA` | `_delta_log/` directory | +| [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) | `read_iceberg` | `ICEBERG` | `metadata/` directory | +| [Icechunk](/docs/2.0.0-rc6/formats/icechunk) | `read_icechunk` | `ICECHUNK` | repository directory | +| [ODV ASCII](/docs/2.0.0-rc6/formats/odv) | `read_odv_ascii` | not supported | `.txt` | Beacon finds every format in the dataset store automatically. **Delta Lake**, **Apache Iceberg**, **Icechunk** and **ODV ASCII** are the exception. Point a function at them. For Delta, Iceberg and @@ -63,7 +63,7 @@ The table above says how to read each format. This one says what you get. | BBF | Full | Predicate + projection | No | `read_bbf_schema` | No | | Delta Lake | Full | Predicate + projection, file skipping | No, but see below | `read_delta_schema` | No | | Apache Iceberg | Full | Predicate + projection, file skipping | No | `read_iceberg_schema` | No | -| Icechunk | Full, see [Icechunk](/docs/2.0.0-rc5/formats/icechunk) | Projection + dimension selection, chunk pruning | No | `read_icechunk_schema` | Yes | +| Icechunk | Full, see [Icechunk](/docs/2.0.0-rc6/formats/icechunk) | Projection + dimension selection, chunk pruning | No | `read_icechunk_schema` | Yes | | ODV ASCII | Full | Projection only | Yes | `read_odv_ascii_schema` | No | Reading the columns: @@ -77,17 +77,17 @@ Reading the columns: query — paths in SQL are relative either way. - **Pushdown** — how much of a query reaches storage instead of running after the read. *Predicate* means a `WHERE` clause prunes data. *Projection* means a narrow `SELECT` reads fewer columns. - [Atlas](/docs/2.0.0-rc5/formats/atlas) is the strongest: the statistics in a collection's footer + [Atlas](/docs/2.0.0-rc6/formats/atlas) is the strongest: the statistics in a collection's footer drop whole datasets before any array is opened. - **Query output** — whether a query result can be written back in that format, with `COPY TO` or an `output.format` on the API. Writing rows into an existing table is a different capability: **Delta Lake** external tables accept `INSERT INTO`, and - [managed tables](/docs/2.0.0-rc5/sql/managed-tables) accept the full `INSERT` / `UPDATE` / + [managed tables](/docs/2.0.0-rc6/sql/managed-tables) accept the full `INSERT` / `UPDATE` / `DELETE` set, but neither is a query output format. - **Schema function** — the `_schema` counterpart that returns columns and types without a scan. - See [Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema). + See [Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema). - **Array format** — whether the file holds N-dimensional arrays, which Beacon flattens into rows. - See [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables). + See [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables). ## See inside a file @@ -98,14 +98,14 @@ counterpart that does this **without a read of any data**: SELECT * FROM read_netcdf_schema('argo/**/*.nc'); ``` -[`SUMMARIZE`](/docs/2.0.0-rc5/sql/summarize) also gives value ranges, distinct counts and +[`SUMMARIZE`](/docs/2.0.0-rc6/sql/summarize) also gives value ranges, distinct counts and null shares. It profiles every column in one pass: ```sql SUMMARIZE (SELECT * FROM read_netcdf('argo/**/*.nc')); ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares all four ways and says what +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares all four ways and says what each one costs. ## Read many files at once @@ -120,17 +120,17 @@ WHERE depth < 100 GROUP BY platform; ``` -[Zarr](/docs/2.0.0-rc5/formats/zarr) is a directory behind a marker file: point at `zarr.json`, -not at the chunk files. An [Atlas](/docs/2.0.0-rc5/formats/atlas) collection is a single file: +[Zarr](/docs/2.0.0-rc6/formats/zarr) is a directory behind a marker file: point at `zarr.json`, +not at the chunk files. An [Atlas](/docs/2.0.0-rc6/formats/atlas) collection is a single file: point at `data.atlas`. Some files share a schema but have different columns. Combine those files with -[`UNION BY NAME`](/docs/2.0.0-rc5/sql/union-by-name). +[`UNION BY NAME`](/docs/2.0.0-rc6/sql/union-by-name). ## Give files a table name A `read_*` call fits an ad-hoc query. When many queries share one source, register it once as an -[external table](/docs/2.0.0-rc5/data-sources/external-tables). Then query it by name: +[external table](/docs/2.0.0-rc6/data-sources/external-tables). Then query it by name: ```sql CREATE EXTERNAL TABLE ocean_profiles @@ -142,6 +142,6 @@ SELECT * FROM ocean_profiles LIMIT 10; ## See also -- [Table Functions](/docs/2.0.0-rc5/sql/table-functions): every reader signature in one place. -- [Data Sources](/docs/2.0.0-rc5/data-sources/): the full read model. -- [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage): running against a bucket. +- [Table Functions](/docs/2.0.0-rc6/sql/table-functions): every reader signature in one place. +- [Data Sources](/docs/2.0.0-rc6/data-sources/): the full read model. +- [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage): running against a bucket. diff --git a/docs/docs/2.0.0-rc5/formats/inspect-a-schema.md b/docs/docs/2.0.0-rc6/formats/inspect-a-schema.md similarity index 92% rename from docs/docs/2.0.0-rc5/formats/inspect-a-schema.md rename to docs/docs/2.0.0-rc6/formats/inspect-a-schema.md index 82914da4..f98b6ac8 100644 --- a/docs/docs/2.0.0-rc5/formats/inspect-a-schema.md +++ b/docs/docs/2.0.0-rc6/formats/inspect-a-schema.md @@ -53,7 +53,7 @@ WHERE data_type LIKE 'Timestamp%'; ## `SUMMARIZE`: profile every column -[`SUMMARIZE`](/docs/2.0.0-rc5/sql/summarize) gives more than names and types. It profiles every +[`SUMMARIZE`](/docs/2.0.0-rc6/sql/summarize) gives more than names and types. It profiles every column in one pass. It adds the minimum, the maximum, the distinct count and the share of nulls: ```sql @@ -98,6 +98,6 @@ schema = client.sql_query( ## Next -- [File formats](/docs/2.0.0-rc5/formats/) and the capability matrix -- [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables), for what a dimension does to the column list -- [SUMMARIZE](/docs/2.0.0-rc5/sql/summarize) for the full statement grammar +- [File formats](/docs/2.0.0-rc6/formats/) and the capability matrix +- [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables), for what a dimension does to the column list +- [SUMMARIZE](/docs/2.0.0-rc6/sql/summarize) for the full statement grammar diff --git a/docs/docs/2.0.0-rc5/formats/netcdf.md b/docs/docs/2.0.0-rc6/formats/netcdf.md similarity index 91% rename from docs/docs/2.0.0-rc5/formats/netcdf.md rename to docs/docs/2.0.0-rc6/formats/netcdf.md index 29d3980f..20f2ec16 100644 --- a/docs/docs/2.0.0-rc5/formats/netcdf.md +++ b/docs/docs/2.0.0-rc6/formats/netcdf.md @@ -34,7 +34,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_netcdf('argo/**/*.nc') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -74,7 +74,7 @@ Limitations: :::tip For a large NetCDF collection, convert the files into one -[Atlas](/docs/2.0.0-rc5/formats/atlas) collection. Atlas merges many NetCDF +[Atlas](/docs/2.0.0-rc6/formats/atlas) collection. Atlas merges many NetCDF files into one array store with statistics. Beacon can then drop whole datasets and read only the arrays that you select. This is much faster than a scan of the original NetCDF files. ::: @@ -87,7 +87,7 @@ STORED AS NC LOCATION 'argo/**/*.nc' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` @@ -108,5 +108,5 @@ LOCATION 'argo/**/*.nc' OPTIONS ('read_dimensions' 'time,latitude,longitude', 'use_rust_reader' 'false') ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. See -[Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables#the-dimensions-argument) for the grid rule. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. See +[Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables#the-dimensions-argument) for the grid rule. diff --git a/docs/docs/2.0.0-rc5/formats/odv.md b/docs/docs/2.0.0-rc6/formats/odv.md similarity index 82% rename from docs/docs/2.0.0-rc5/formats/odv.md rename to docs/docs/2.0.0-rc6/formats/odv.md index 55917d32..5741fbb0 100644 --- a/docs/docs/2.0.0-rc5/formats/odv.md +++ b/docs/docs/2.0.0-rc6/formats/odv.md @@ -24,15 +24,15 @@ Check the columns and the types before you write a query: SELECT * FROM read_odv_ascii('odv/**/*.txt') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details Beacon does **not** find ODV in the dataset store automatically. -[Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) is the other such format. ODV +[Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) is the other such format. ODV also has no `CREATE EXTERNAL TABLE ... STORED AS ODV` form. Point `read_odv_ascii()` at the files. -Wrap the call in a [view](/docs/2.0.0-rc5/server/view) to get a stable name: +Wrap the call in a [view](/docs/2.0.0-rc6/server/view) to get a stable name: ```sql CREATE VIEW odv_profiles AS @@ -48,4 +48,4 @@ zstd -9 < input.txt > output.txt.zst Beacon detects the compression automatically. It decompresses the file during the read. A zstd-compressed ODV file also works on object storage. -See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the full read model. +See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. diff --git a/docs/docs/2.0.0-rc5/formats/parquet.md b/docs/docs/2.0.0-rc6/formats/parquet.md similarity index 81% rename from docs/docs/2.0.0-rc5/formats/parquet.md rename to docs/docs/2.0.0-rc6/formats/parquet.md index e01d9240..d58b52ee 100644 --- a/docs/docs/2.0.0-rc5/formats/parquet.md +++ b/docs/docs/2.0.0-rc6/formats/parquet.md @@ -22,7 +22,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_parquet('obs/*.parquet') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -31,7 +31,7 @@ DataFusion reads Parquet directly. Parquet suits analytical work, because it sto and supports predicate pushdown. - Beacon supports column pruning and predicate pushdown in full. -- Beacon supports Hive-style directory partitions. Use `PARTITIONED BY` on an [external table](/docs/2.0.0-rc5/data-sources/external-tables). +- Beacon supports Hive-style directory partitions. Use `PARTITIONED BY` on an [external table](/docs/2.0.0-rc6/data-sources/external-tables). - Beacon reads files from DuckDB, Spark, pandas and similar tools. ## As an external table @@ -50,10 +50,10 @@ STORED AS PARQUET LOCATION 'profiles/**/*.parquet' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` `STORED AS PARQUET` reads no key. Beacon ignores an `OPTIONS` clause on this format. See -[`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the formats that do read one. +[`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the formats that do read one. diff --git a/docs/docs/2.0.0-rc5/formats/zarr.md b/docs/docs/2.0.0-rc6/formats/zarr.md similarity index 91% rename from docs/docs/2.0.0-rc5/formats/zarr.md rename to docs/docs/2.0.0-rc6/formats/zarr.md index 04a60e7e..a87f4c8d 100644 --- a/docs/docs/2.0.0-rc5/formats/zarr.md +++ b/docs/docs/2.0.0-rc6/formats/zarr.md @@ -38,7 +38,7 @@ Check the columns and the types before you write a query: SELECT * FROM read_zarr('sst/*/zarr.json') LIMIT 0; ``` -[Inspect a schema](/docs/2.0.0-rc5/formats/inspect-a-schema) compares the `_schema` functions, +[Inspect a schema](/docs/2.0.0-rc6/formats/inspect-a-schema) compares the `_schema` functions, `SUMMARIZE`, `DESCRIBE` and `LIMIT 0`, and says what each one costs. ## Format details @@ -78,7 +78,7 @@ Predicate pushdown is automatic. Beacon prunes chunks and slices the coordinate `time`, `latitude` and `longitude`. It uses the filters of your query. You configure nothing. Do you query a collection often? Then convert the Zarr stores into one -[Atlas](/docs/2.0.0-rc5/formats/atlas) collection. Atlas is an array store +[Atlas](/docs/2.0.0-rc6/formats/atlas) collection. Atlas is an array store with statistics. Beacon can drop whole datasets before it reads a chunk. ::: @@ -100,7 +100,7 @@ STORED AS ZARR LOCATION 'sst/*/zarr.json' ``` -See [Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc5/data-sources/) for the +See [Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables) for the full DDL. See [Data Sources](/docs/2.0.0-rc6/data-sources/) for the full read model. ### `OPTIONS` @@ -119,5 +119,5 @@ LOCATION 'sst/*/zarr.json' OPTIONS ('read_dimensions' 'time,lat,lon') ``` -See [`OPTIONS`](/docs/2.0.0-rc5/sql/create-external-table#options) for the rules that hold for every key. See -[Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables#the-dimensions-argument) for the grid rule. +See [`OPTIONS`](/docs/2.0.0-rc6/sql/create-external-table#options) for the rules that hold for every key. See +[Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables#the-dimensions-argument) for the grid rule. diff --git a/docs/docs/2.0.0-rc5/getting-started.md b/docs/docs/2.0.0-rc6/getting-started.md similarity index 80% rename from docs/docs/2.0.0-rc5/getting-started.md rename to docs/docs/2.0.0-rc6/getting-started.md index 0e5026f3..ff3dfdc4 100644 --- a/docs/docs/2.0.0-rc5/getting-started.md +++ b/docs/docs/2.0.0-rc6/getting-started.md @@ -13,11 +13,11 @@ examples with MinIO and sample datasets. Running a node is four jobs, in this order: 1. **Deploy** it, on this page. -2. **[Configure](/docs/2.0.0-rc5/server/configuration)** the ports, the datasets store and the +2. **[Configure](/docs/2.0.0-rc6/server/configuration)** the ports, the datasets store and the resource limits. -3. **[Register your data](/docs/2.0.0-rc5/server/)** as tables and views, so clients query names +3. **[Register your data](/docs/2.0.0-rc6/server/)** as tables and views, so clients query names instead of paths. -4. **[Secure and expose](/docs/2.0.0-rc5/security/access-control)** it, then point clients at it. +4. **[Secure and expose](/docs/2.0.0-rc6/security/access-control)** it, then point clients at it. ## Prerequisites @@ -54,7 +54,7 @@ Copy supported files into the `./datasets` folder. Supported files include `.par Open . Sign in with the admin user name and password from step 1 (`admin` / `securepassword`). The server and the Docker image include the -[admin web UI](/docs/2.0.0-rc5/connect/web-admin-ui). You deploy nothing extra. The UI gives you: +[admin web UI](/docs/2.0.0-rc6/connect/web-admin-ui). You deploy nothing extra. The UI gives you: - **Query editor**: write SQL, run it (⌘/Ctrl + Enter), read the results and download CSV or Parquet. - **Datasets**: browse the files that Beacon found and inspect their schemas. @@ -119,30 +119,30 @@ services: ::: For Compose, run `docker compose up -d`. Beacon now runs. Open the -[admin UI](/docs/2.0.0-rc5/connect/web-admin-ui) at `http://localhost:5001/admin` to explore and +[admin UI](/docs/2.0.0-rc6/connect/web-admin-ui) at `http://localhost:5001/admin` to explore and query. Open `http://localhost:5001/swagger` for the API docs. You can query any file in `./datasets` at once. ::: tip Log files The `./logs` volume writes the log files to your machine. Beacon starts one file each day, for example `beacon.log.2026-08-19`. Without the volume the files stay in the container. See -[Log files](/docs/2.0.0-rc5/server/configuration#log-files). +[Log files](/docs/2.0.0-rc6/server/configuration#log-files). ::: ::: tip Two ways to connect Beacon exposes two endpoints. The **HTTP API** on port `5001` serves SQL and JSON queries, the admin UI and the OpenAPI docs. The **Arrow Flight SQL** server on port `32011` uses a columnar protocol with high throughput. Clients such as -[JetBrains DataGrip](/docs/2.0.0-rc5/connect/datagrip) and the -[Python ADBC driver](/docs/2.0.0-rc5/connect/python-adbc) use it. Flight SQL authenticates with a +[JetBrains DataGrip](/docs/2.0.0-rc6/connect/datagrip) and the +[Python ADBC driver](/docs/2.0.0-rc6/connect/python-adbc) use it. Flight SQL authenticates with a bearer token. Tune it or switch it off with the `BEACON_FLIGHT_SQL_*` -[settings](/docs/2.0.0-rc5/server/configuration#arrow-flight-sql). +[settings](/docs/2.0.0-rc6/server/configuration#arrow-flight-sql). ::: ::: warning Secure your instance The `BEACON_ADMIN_*` credentials protect the admin UI and all write operations. **Change them from the defaults** before you expose Beacon. To control who reads data, switch on -[access control](/docs/2.0.0-rc5/security/access-control) with `BEACON_AUTH_ENFORCE=true`. +[access control](/docs/2.0.0-rc6/security/access-control) with `BEACON_AUTH_ENFORCE=true`. ::: ## S3-Compatible Object Storage @@ -207,29 +207,29 @@ For Compose, run `docker compose up -d`. You can query the files in the S3 bucke | | | | - | - | -| **Every setting** | [Configuration](/docs/2.0.0-rc5/server/configuration) | -| **Put the datasets on a bucket** | [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage) | -| **Memory, concurrency, caches** | [Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning) | +| **Every setting** | [Configuration](/docs/2.0.0-rc6/server/configuration) | +| **Put the datasets on a bucket** | [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage) | +| **Memory, concurrency, caches** | [Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning) | **Register the data** | | | | - | - | -| **Name a set of files** | [External Tables](/docs/2.0.0-rc5/data-sources/external-tables) | -| **Save a query** | [Views](/docs/2.0.0-rc5/server/view) · [Materialized Views](/docs/2.0.0-rc5/sql/create-materialized-view) | -| **Register a large tree on a schedule** | [Crawlers](/docs/2.0.0-rc5/server/crawlers) | -| **Own the rows yourself** | [Managed Tables](/docs/2.0.0-rc5/sql/managed-tables) | -| **Reach another node or a database** | [ATTACH](/docs/2.0.0-rc5/data-sources/attach) · [SQL Databases](/docs/2.0.0-rc5/data-sources/sql-databases) | +| **Name a set of files** | [External Tables](/docs/2.0.0-rc6/data-sources/external-tables) | +| **Save a query** | [Views](/docs/2.0.0-rc6/server/view) · [Materialized Views](/docs/2.0.0-rc6/sql/create-materialized-view) | +| **Register a large tree on a schedule** | [Crawlers](/docs/2.0.0-rc6/server/crawlers) | +| **Own the rows yourself** | [Managed Tables](/docs/2.0.0-rc6/sql/managed-tables) | +| **Reach another node or a database** | [ATTACH](/docs/2.0.0-rc6/data-sources/attach) · [SQL Databases](/docs/2.0.0-rc6/data-sources/sql-databases) | **Expose it** | | | | - | - | -| **Decide who reads what** | [Access Control](/docs/2.0.0-rc5/security/access-control) | -| **Explore in the browser** | [Admin Web UI](/docs/2.0.0-rc5/connect/web-admin-ui) | -| **Point clients at it** | [Python](/docs/2.0.0-rc5/connect/python) · [TypeScript](/docs/2.0.0-rc5/connect/typescript) · [CLI](/docs/2.0.0-rc5/connect/cli) · [DataGrip](/docs/2.0.0-rc5/connect/datagrip) · [Python ADBC](/docs/2.0.0-rc5/connect/python-adbc) | -| **Document the query API** | [REST API](/docs/2.0.0-rc5/api/) | +| **Decide who reads what** | [Access Control](/docs/2.0.0-rc6/security/access-control) | +| **Explore in the browser** | [Admin Web UI](/docs/2.0.0-rc6/connect/web-admin-ui) | +| **Point clients at it** | [Python](/docs/2.0.0-rc6/connect/python) · [TypeScript](/docs/2.0.0-rc6/connect/typescript) · [CLI](/docs/2.0.0-rc6/connect/cli) · [DataGrip](/docs/2.0.0-rc6/connect/datagrip) · [Python ADBC](/docs/2.0.0-rc6/connect/python-adbc) | +| **Document the query API** | [REST API](/docs/2.0.0-rc6/api/) | **When something is wrong** -[Troubleshooting](/docs/2.0.0-rc5/troubleshooting) · [FAQ](/docs/2.0.0-rc5/faq) +[Troubleshooting](/docs/2.0.0-rc6/troubleshooting) · [FAQ](/docs/2.0.0-rc6/faq) diff --git a/docs/docs/2.0.0-rc5/guides/export-results.md b/docs/docs/2.0.0-rc6/guides/export-results.md similarity index 87% rename from docs/docs/2.0.0-rc5/guides/export-results.md rename to docs/docs/2.0.0-rc6/guides/export-results.md index 048b5090..1c69339d 100644 --- a/docs/docs/2.0.0-rc5/guides/export-results.md +++ b/docs/docs/2.0.0-rc6/guides/export-results.md @@ -24,7 +24,7 @@ gdf = query.to_geo_pandas_dataframe("lon", "lat") ``` Beacon runs nothing until you call a `to_*` method. Build the query first. Then check it with -[`EXPLAIN`](/docs/2.0.0-rc5/sql/select). +[`EXPLAIN`](/docs/2.0.0-rc6/sql/select). ## To a file @@ -60,10 +60,10 @@ curl -X POST https://beacon.example.com/api/query \ ``` The same works for `csv`, `netcdf` and `ipc`. See -[output formats](/docs/2.0.0-rc5/api/querying/#output-formats). +[output formats](/docs/2.0.0-rc6/api/querying/#output-formats). For a streaming read in Python instead of a file, connect over Arrow Flight SQL and pull record -batches. See [Python ADBC](/docs/2.0.0-rc5/connect/python-adbc). +batches. See [Python ADBC](/docs/2.0.0-rc6/connect/python-adbc). ## Keep results on the server @@ -75,9 +75,9 @@ CREATE TABLE hot_profiles AS SELECT * FROM read_netcdf('argo/**/*.nc') WHERE temperature > 25; ``` -This creates a [managed table](/docs/2.0.0-rc5/sql/managed-tables) inside `beacon.db`. Does the +This creates a [managed table](/docs/2.0.0-rc6/sql/managed-tables) inside `beacon.db`. Does the source data change? Do you want a refresh on demand? Then use a -[materialized view](/docs/2.0.0-rc5/sql/create-materialized-view). +[materialized view](/docs/2.0.0-rc6/sql/create-materialized-view). ## Output formats @@ -91,5 +91,5 @@ source data change? Do you want a refresh on demand? Then use a | Arrow IPC | `ipc` | none | | ODV ASCII | `odv` | none | -See [output formats](/docs/2.0.0-rc5/api/querying/#output-formats) for the full option set of each +See [output formats](/docs/2.0.0-rc6/api/querying/#output-formats) for the full option set of each one. diff --git a/docs/docs/2.0.0-rc5/guides/index.md b/docs/docs/2.0.0-rc6/guides/index.md similarity index 62% rename from docs/docs/2.0.0-rc5/guides/index.md rename to docs/docs/2.0.0-rc6/guides/index.md index a7e4b7b2..f1752f17 100644 --- a/docs/docs/2.0.0-rc5/guides/index.md +++ b/docs/docs/2.0.0-rc6/guides/index.md @@ -9,10 +9,10 @@ the reference chapters. | Guide | What it covers | | --- | --- | -| [Query a File Collection](/docs/2.0.0-rc5/guides/query-a-collection) | Make one table from thousands of NetCDF or Parquet files | -| [Query Data on S3](/docs/2.0.0-rc5/guides/query-s3) | Credentials, public buckets, and fast object storage | -| [Export Query Results](/docs/2.0.0-rc5/guides/export-results) | Write results to Parquet, CSV, NetCDF, GeoParquet or ODV | -| [Speed Up Slow Queries](/docs/2.0.0-rc5/guides/speed-up-queries) | Diagnose with `EXPLAIN`. Then fix with pushdown, Atlas or a materialized view | +| [Query a File Collection](/docs/2.0.0-rc6/guides/query-a-collection) | Make one table from thousands of NetCDF or Parquet files | +| [Query Data on S3](/docs/2.0.0-rc6/guides/query-s3) | Credentials, public buckets, and fast object storage | +| [Export Query Results](/docs/2.0.0-rc6/guides/export-results) | Write results to Parquet, CSV, NetCDF, GeoParquet or ODV | +| [Speed Up Slow Queries](/docs/2.0.0-rc6/guides/speed-up-queries) | Diagnose with `EXPLAIN`. Then fix with pushdown, Atlas or a materialized view | -New to Beacon? Start with the [Quick Start](/docs/2.0.0-rc5/quickstart). Then read -[How It Works](/docs/2.0.0-rc5/how-it-works). +New to Beacon? Start with the [Quick Start](/docs/2.0.0-rc6/quickstart). Then read +[How It Works](/docs/2.0.0-rc6/how-it-works). diff --git a/docs/docs/2.0.0-rc5/guides/query-a-collection.md b/docs/docs/2.0.0-rc6/guides/query-a-collection.md similarity index 91% rename from docs/docs/2.0.0-rc5/guides/query-a-collection.md rename to docs/docs/2.0.0-rc6/guides/query-a-collection.md index ce71c69f..3e2eb14a 100644 --- a/docs/docs/2.0.0-rc5/guides/query-a-collection.md +++ b/docs/docs/2.0.0-rc6/guides/query-a-collection.md @@ -57,7 +57,7 @@ LIMIT 5; | 6901234 | 2024-01-03 06:12:00 | 43.117 | 20.0 | 14.702 | `PLATFORM_NUMBER`, `JULD` and `LATITUDE` repeat, because they are per-profile values broadcast -across the 400 levels. See [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables). +across the 400 levels. See [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables). Check the size: @@ -71,7 +71,7 @@ SELECT count(*) AS rows FROM read_netcdf('argo/2024/01/*.nc'); Three files, 400 levels each. -[`SUMMARIZE`](/docs/2.0.0-rc5/sql/summarize) profiles every column in one pass. It gives the type, +[`SUMMARIZE`](/docs/2.0.0-rc6/sql/summarize) profiles every column in one pass. It gives the type, the null count and the range: ```sql @@ -121,7 +121,7 @@ UNION ALL BY NAME SELECT time, temperature FROM read_parquet('gliders/*.parquet'); ``` -See [UNION BY NAME](/docs/2.0.0-rc5/sql/union-by-name). +See [UNION BY NAME](/docs/2.0.0-rc6/sql/union-by-name). Some NetCDF collections mix variables with different dimensions. Give an explicit dimension list. Beacon then returns only the compatible variables: @@ -153,11 +153,11 @@ LOCATION 'argo/**/*.nc'; SELECT platform, avg(temperature) FROM argo GROUP BY platform; ``` -See [External Tables](/docs/2.0.0-rc5/data-sources/external-tables). +See [External Tables](/docs/2.0.0-rc6/data-sources/external-tables). ## 6. If it is slow Do you scan thousands of NetCDF or Zarr files often? Then merge them into one -[Atlas](/docs/2.0.0-rc5/formats/atlas) collection. Atlas drops whole datasets +[Atlas](/docs/2.0.0-rc6/formats/atlas) collection. Atlas drops whole datasets with its stored statistics. See -[Speed Up Slow Queries](/docs/2.0.0-rc5/guides/speed-up-queries). +[Speed Up Slow Queries](/docs/2.0.0-rc6/guides/speed-up-queries). diff --git a/docs/docs/2.0.0-rc5/guides/query-s3.md b/docs/docs/2.0.0-rc6/guides/query-s3.md similarity index 84% rename from docs/docs/2.0.0-rc5/guides/query-s3.md rename to docs/docs/2.0.0-rc6/guides/query-s3.md index 626fb72e..59b85464 100644 --- a/docs/docs/2.0.0-rc5/guides/query-s3.md +++ b/docs/docs/2.0.0-rc6/guides/query-s3.md @@ -37,12 +37,12 @@ docker run -d --name beacon -p 5001:5001 \ For a public bucket, drop the keys and set `AWS_SKIP_SIGNATURE=true`. -See [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage) for every setting, and -[Configuration](/docs/2.0.0-rc5/server/configuration) for the full list. +See [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage) for every setting, and +[Configuration](/docs/2.0.0-rc6/server/configuration) for the full list. ::: info There is no SQL statement for storage credentials `CREATE SECRET` covers one case. It holds the credentials for **another Beacon server**, which you -reach with [`ATTACH`](/docs/2.0.0-rc5/data-sources/attach). Storage credentials come from the +reach with [`ATTACH`](/docs/2.0.0-rc6/data-sources/attach). Storage credentials come from the configuration. A server has one store. It selects that store at startup. ::: @@ -51,14 +51,14 @@ configuration. A server has one store. It selects that store at startup. Object storage has a high latency. Fetch as few bytes as possible: - **Use cloud-optimized formats.** Parquet, - [Zarr](/docs/2.0.0-rc5/formats/zarr), - [Atlas](/docs/2.0.0-rc5/formats/atlas) and Cloud-Optimized GeoTIFF support + [Zarr](/docs/2.0.0-rc6/formats/zarr), + [Atlas](/docs/2.0.0-rc6/formats/atlas) and Cloud-Optimized GeoTIFF support range requests. Beacon then fetches only the chunks that it needs. - **Select only the columns that you need.** Projection pushdown turns a narrow `SELECT` into fewer bytes. - **Filter early.** A predicate prunes row groups and chunks before any transfer. - **Watch NetCDF.** On object storage, NetCDF supports anonymous access only. See - [NetCDF](/docs/2.0.0-rc5/formats/netcdf). + [NetCDF](/docs/2.0.0-rc6/formats/netcdf). ## Register a prefix as a table @@ -71,4 +71,4 @@ STORED AS PARQUET LOCATION 'obs/'; ``` -See [External Tables](/docs/2.0.0-rc5/data-sources/external-tables). +See [External Tables](/docs/2.0.0-rc6/data-sources/external-tables). diff --git a/docs/docs/2.0.0-rc5/guides/speed-up-queries.md b/docs/docs/2.0.0-rc6/guides/speed-up-queries.md similarity index 88% rename from docs/docs/2.0.0-rc5/guides/speed-up-queries.md rename to docs/docs/2.0.0-rc6/guides/speed-up-queries.md index 364eea60..e63b9a80 100644 --- a/docs/docs/2.0.0-rc5/guides/speed-up-queries.md +++ b/docs/docs/2.0.0-rc6/guides/speed-up-queries.md @@ -49,16 +49,16 @@ interpret runs after the read. It therefore prunes nothing. - **Partition** the files Hive-style, for example `year=2024/month=01/`. Declare the partition columns with `PARTITIONED BY`. Beacon then skips whole directories. See - [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table). + [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table). - **Do not use very many small files.** The cost of each file dominates. A merge into fewer files helps more than any change to a query. - **Use a cloud-optimized format** for data on object storage. See - [Query Data on S3](/docs/2.0.0-rc5/guides/query-s3). + [Query Data on S3](/docs/2.0.0-rc6/guides/query-s3). ## 5. Merge large array collections with Atlas Do you query a large NetCDF or Zarr collection often? Then convert it to -[Atlas](/docs/2.0.0-rc5/formats/atlas). This gives the largest gain. Atlas +[Atlas](/docs/2.0.0-rc6/formats/atlas). This gives the largest gain. Atlas keeps statistics for each dataset. Beacon drops the datasets that cannot match a predicate. It drops them *before it reads any array data*. It then reads only the arrays that you select. @@ -72,7 +72,7 @@ LOCATION 'collections/*/data.atlas'; Does the same expensive aggregate run often? Then compute it once. -A [materialized view](/docs/2.0.0-rc5/sql/create-materialized-view) caches the result. You +A [materialized view](/docs/2.0.0-rc6/sql/create-materialized-view) caches the result. You refresh it on demand: ```sql @@ -84,7 +84,7 @@ GROUP BY 1; REFRESH MATERIALIZED VIEW monthly_means; ``` -Use a [managed table](/docs/2.0.0-rc5/internals/storage) for a working subset +Use a [managed table](/docs/2.0.0-rc6/internals/storage) for a working subset that you also change: ```sql @@ -108,10 +108,10 @@ while (batch := cursor.fetch_record_batch().read_next_batch()) is not None: process(batch) ``` -See [Export Query Results](/docs/2.0.0-rc5/guides/export-results). +See [Export Query Results](/docs/2.0.0-rc6/guides/export-results). ## Server-side tuning The server adds more settings. They cover concurrency, memory, disk spill, reader caches and object store listings. -[Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning) documents them. +[Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning) documents them. diff --git a/docs/docs/2.0.0-rc5/how-it-works.md b/docs/docs/2.0.0-rc6/how-it-works.md similarity index 86% rename from docs/docs/2.0.0-rc5/how-it-works.md rename to docs/docs/2.0.0-rc6/how-it-works.md index e38bd386..7a736755 100644 --- a/docs/docs/2.0.0-rc5/how-it-works.md +++ b/docs/docs/2.0.0-rc6/how-it-works.md @@ -34,7 +34,7 @@ WHERE temperature > 20; A glob expands across directories. One statement can therefore cover thousands of files. Beacon merges their schemas and reads them in parallel. You can also register the same files once as an -[external table](/docs/2.0.0-rc5/data-sources/external-tables) and query them by name. +[external table](/docs/2.0.0-rc6/data-sources/external-tables) and query them by name. ## Pushdown @@ -44,13 +44,13 @@ moves filters and column selections as close to the data as possible: - **Projection pushdown**: Beacon decodes only the columns that a query names. A query over 3 of 200 variables reads about 3 columns of bytes. - **Predicate pushdown**: Beacon turns filters into file, row group and chunk pruning. A time range - filter on a [Zarr](/docs/2.0.0-rc5/formats/zarr) store fetches only the - chunks in that range. On [Atlas](/docs/2.0.0-rc5/formats/atlas), every array records its own + filter on a [Zarr](/docs/2.0.0-rc6/formats/zarr) store fetches only the + chunks in that range. On [Atlas](/docs/2.0.0-rc6/formats/atlas), every array records its own range in the collection's footer, so Beacon judges every dataset in one pass and never opens the ones that cannot match. - **Federated pushdown**: Beacon sends filters, projections, limits and whole aggregates to - [SQL databases](/docs/2.0.0-rc5/data-sources/sql-databases) and to - [remote Beacons](/docs/2.0.0-rc5/data-sources/remote-tables). Only the reduced result + [SQL databases](/docs/2.0.0-rc6/data-sources/sql-databases) and to + [remote Beacons](/docs/2.0.0-rc6/data-sources/remote-tables). Only the reduced result travels back. Use `EXPLAIN` to see what the planner pushes down: @@ -66,14 +66,14 @@ Ordinary SQL then works on it. Beacon also exposes the metadata as columns. The variable appear as `variable.attribute`, for example `temperature.units`. File attributes appear as `.attribute`. -[Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables) specifies the mapping: the row count, the -broadcast rule and the `dimensions` argument. [File Formats](/docs/2.0.0-rc5/formats/) documents the +[Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables) specifies the mapping: the row count, the +broadcast rule and the `dimensions` argument. [File Formats](/docs/2.0.0-rc6/formats/) documents the behaviour of each format. ## Storage: what Beacon owns Most data stays where it is. Beacon *owns* the content of one -[`beacon.db` file](/docs/2.0.0-rc5/internals/storage). That file holds the +[`beacon.db` file](/docs/2.0.0-rc6/internals/storage). That file holds the catalog with the table, view and secret definitions. It also holds the managed table data. Copy that one file and the database goes with it. The copy still references the external files and the remote systems. @@ -96,5 +96,5 @@ API, a managed dataset store, crawlers, role-based access control and a web admi change who reaches the engine, not what it does. The planner, the readers, the formats and the SQL dialect are the same on every transport. -See [Access Control](/docs/2.0.0-rc5/security/access-control) for the grant model, and -[Getting Started](/docs/2.0.0-rc5/getting-started) to run a server. +See [Access Control](/docs/2.0.0-rc6/security/access-control) for the grant model, and +[Getting Started](/docs/2.0.0-rc6/getting-started) to run a server. diff --git a/docs/docs/2.0.0-rc5/internals/file-statistics.md b/docs/docs/2.0.0-rc6/internals/file-statistics.md similarity index 98% rename from docs/docs/2.0.0-rc5/internals/file-statistics.md rename to docs/docs/2.0.0-rc6/internals/file-statistics.md index 0406badc..68eb0cd4 100644 --- a/docs/docs/2.0.0-rc5/internals/file-statistics.md +++ b/docs/docs/2.0.0-rc6/internals/file-statistics.md @@ -59,7 +59,7 @@ does not stop part-way, so one pass covers a fresh archive. Only one pass runs at a time. A tick that finds a pass still running is skipped. `ANALYZE FILES` reports an error rather than waiting, because a pass over a large archive runs for minutes. -[Configuration](/docs/2.0.0-rc5/server/configuration#file-statistics) lists each variable. +[Configuration](/docs/2.0.0-rc6/server/configuration#file-statistics) lists each variable. The **timer runs its first pass one interval after startup**, not at startup. Beacon starts the interval again on each boot, and records no due time. A server that restarts more often than the @@ -640,7 +640,7 @@ Beacon stores only the cells with data. That is about 20 million cells and 780 M | Manifest | The columns that each segment holds | Beacon holds all three parts in `beacon.db`. Copy that file and the statistics go with it. See -[Storage internals](/docs/2.0.0-rc5/internals/storage). +[Storage internals](/docs/2.0.0-rc6/internals/storage). Beacon gives each file a number. A path is long and a number is short. 20 million records with a 200-byte path cost more than the ranges. A file keeps its number for ever. A deleted file keeps its @@ -695,6 +695,6 @@ change. Beacon has no compaction step. ## Related pages -- [Configuration](/docs/2.0.0-rc5/server/configuration#file-statistics) lists each variable. -- [Storage internals](/docs/2.0.0-rc5/internals/storage) shows the other content of `beacon.db`. -- [Performance tuning](/docs/2.0.0-rc5/server/performance-tuning) shows the other controls. +- [Configuration](/docs/2.0.0-rc6/server/configuration#file-statistics) lists each variable. +- [Storage internals](/docs/2.0.0-rc6/internals/storage) shows the other content of `beacon.db`. +- [Performance tuning](/docs/2.0.0-rc6/server/performance-tuning) shows the other controls. diff --git a/docs/docs/2.0.0-rc5/internals/storage.md b/docs/docs/2.0.0-rc6/internals/storage.md similarity index 92% rename from docs/docs/2.0.0-rc5/internals/storage.md rename to docs/docs/2.0.0-rc6/internals/storage.md index 439ecbbe..c237db74 100644 --- a/docs/docs/2.0.0-rc5/internals/storage.md +++ b/docs/docs/2.0.0-rc6/internals/storage.md @@ -38,7 +38,7 @@ This is the main decision when you bring data into Beacon: | Changes | Read-only | `INSERT`, `UPDATE`, `DELETE` | | Best for | Existing archives that you query | Results, curated subsets, working sets | -Use an [external table](/docs/2.0.0-rc5/data-sources/external-tables) to query data that you already +Use an [external table](/docs/2.0.0-rc6/data-sources/external-tables) to query data that you already have. Use a managed table when Beacon owns the rows and changes them. ## Managed tables @@ -60,7 +60,7 @@ DELETE FROM observations WHERE temperature < -5; ``` A managed table supports `ALTER TABLE`, indexes and the usual DDL. See -[Managed Tables](/docs/2.0.0-rc5/sql/managed-tables) in the SQL reference. It gives the full +[Managed Tables](/docs/2.0.0-rc6/sql/managed-tables) in the SQL reference. It gives the full statement grammar, the index types and the limitations. A common pattern runs an expensive scan once, then queries the result many times: @@ -71,12 +71,12 @@ SELECT * FROM read_netcdf('argo/**/*.nc') WHERE temperature > 25; ``` Does the result need a periodic refresh instead of one copy? Then use a -[materialized view](/docs/2.0.0-rc5/sql/create-materialized-view). +[materialized view](/docs/2.0.0-rc6/sql/create-materialized-view). ## Secrets Beacon holds the credentials for another Beacon server as named -[secrets](/docs/2.0.0-rc5/sql/secrets). A session secret lives in memory. A +[secrets](/docs/2.0.0-rc6/sql/secrets). A session secret lives in memory. A `CREATE PERSISTENT SECRET` goes into `beacon.db` **encrypted** with XChaCha20-Poly1305. Beacon reloads it when it opens the file. A persistent secret needs a master key. Beacon never writes a plaintext credential to disk. diff --git a/docs/docs/2.0.0-rc5/introduction.md b/docs/docs/2.0.0-rc6/introduction.md similarity index 78% rename from docs/docs/2.0.0-rc5/introduction.md rename to docs/docs/2.0.0-rc6/introduction.md index dde1ee00..89b26ed2 100644 --- a/docs/docs/2.0.0-rc5/introduction.md +++ b/docs/docs/2.0.0-rc6/introduction.md @@ -13,13 +13,13 @@ conversion into a proprietary format. Beacon reads the files *in place*. It uses Many clients reach one server at the same time. A client is a notebook, a portal, a dashboard, a BI tool or a terminal. All clients share one catalog, one set of paths and one set of -[grants](/docs/2.0.0-rc5/security/access-control). You therefore decide in one place what each user +[grants](/docs/2.0.0-rc6/security/access-control). You therefore decide in one place what each user may read. File copies no longer decide it. Beacon solves one problem. An institution holds an archive. Many people need parts of it. Nobody needs to download the whole archive. -New here? Go to the **[Quick Start](/docs/2.0.0-rc5/quickstart)**. It takes a few minutes. +New here? Go to the **[Quick Start](/docs/2.0.0-rc6/quickstart)**. It takes a few minutes. ## How it fits together @@ -46,7 +46,7 @@ SELECT * FROM read_parquet('obs/*.parquet') LIMIT 10; ``` Where the bytes live is an operator's decision, made once in configuration. See -[Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +[Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). ## Four ways to name data @@ -55,10 +55,10 @@ four kinds, and you query all of them the same way: | Kind | What it is | Beacon stores | |---|---|---| -| [External table](/docs/2.0.0-rc5/data-sources/external-tables) | A name over files in the datasets store | The definition | -| [View](/docs/2.0.0-rc5/sql/create-view) | A saved query | The definition | -| [Materialized view](/docs/2.0.0-rc5/sql/create-materialized-view) | A saved query, with its result kept and refreshed | The definition and the rows | -| [Managed table](/docs/2.0.0-rc5/sql/managed-tables) | A table Beacon owns and writes | The rows | +| [External table](/docs/2.0.0-rc6/data-sources/external-tables) | A name over files in the datasets store | The definition | +| [View](/docs/2.0.0-rc6/sql/create-view) | A saved query | The definition | +| [Materialized view](/docs/2.0.0-rc6/sql/create-materialized-view) | A saved query, with its result kept and refreshed | The definition and the rows | +| [Managed table](/docs/2.0.0-rc6/sql/managed-tables) | A table Beacon owns and writes | The rows | ```sql CREATE EXTERNAL TABLE obs STORED AS PARQUET LOCATION 'obs/'; @@ -77,12 +77,12 @@ file: - **The catalog.** Every external table, view and materialized view definition above. - **Managed table rows.** The only data Beacon holds itself. -- **Users, roles and grants.** See [Access Control](/docs/2.0.0-rc5/security/access-control). +- **Users, roles and grants.** See [Access Control](/docs/2.0.0-rc6/security/access-control). - **Secrets.** Credentials for another Beacon server, encrypted at rest. See - [CREATE SECRET](/docs/2.0.0-rc5/sql/secrets). + [CREATE SECRET](/docs/2.0.0-rc6/sql/secrets). Everything else stays where it is. Beacon never copies your source files. See -[Storage internals](/docs/2.0.0-rc5/internals/storage). +[Storage internals](/docs/2.0.0-rc6/internals/storage). ## One SQL, every source @@ -96,7 +96,7 @@ JOIN remote_wod.stations AS b ON a.platform = b.platform WHERE a.temperature > 20; ``` -Read the [SQL reference](/docs/2.0.0-rc5/sql/) for the full dialect. +Read the [SQL reference](/docs/2.0.0-rc6/sql/) for the full dialect. @@ -106,20 +106,20 @@ Read the [SQL reference](/docs/2.0.0-rc5/sql/) for the full dialect. | You want to… | Read | |---|---| -| Deploy one with Docker | [Getting Started](/docs/2.0.0-rc5/getting-started) | -| Set ports, storage and limits | [Configuration](/docs/2.0.0-rc5/server/configuration) | -| Register your data as tables | [Server Setup](/docs/2.0.0-rc5/server/) | -| Decide who may read what | [Access Control](/docs/2.0.0-rc5/security/access-control) | -| Make a slow query fast | [Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning) | +| Deploy one with Docker | [Getting Started](/docs/2.0.0-rc6/getting-started) | +| Set ports, storage and limits | [Configuration](/docs/2.0.0-rc6/server/configuration) | +| Register your data as tables | [Server Setup](/docs/2.0.0-rc6/server/) | +| Decide who may read what | [Access Control](/docs/2.0.0-rc6/security/access-control) | +| Make a slow query fast | [Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning) | **Querying one** | You want to… | Read | |---|---| -| Run your first query against a live node | [Quick Start](/docs/2.0.0-rc5/quickstart) | -| Understand how a file becomes rows | [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables) | -| Replace an xarray loop with SQL | [Coming from xarray](/docs/2.0.0-rc5/coming-from-xarray) | -| See which formats support what | [File formats](/docs/2.0.0-rc5/formats/) | -| Query another institution's node | [ATTACH](/docs/2.0.0-rc5/data-sources/attach) | +| Run your first query against a live node | [Quick Start](/docs/2.0.0-rc6/quickstart) | +| Understand how a file becomes rows | [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables) | +| Replace an xarray loop with SQL | [Coming from xarray](/docs/2.0.0-rc6/coming-from-xarray) | +| See which formats support what | [File formats](/docs/2.0.0-rc6/formats/) | +| Query another institution's node | [ATTACH](/docs/2.0.0-rc6/data-sources/attach) | -See [Concepts](/docs/2.0.0-rc5/concepts) for the engine, the `beacon.db` file, catalogs and tables. +See [Concepts](/docs/2.0.0-rc6/concepts) for the engine, the `beacon.db` file, catalogs and tables. diff --git a/docs/docs/2.0.0-rc5/mcp.md b/docs/docs/2.0.0-rc6/mcp.md similarity index 98% rename from docs/docs/2.0.0-rc5/mcp.md rename to docs/docs/2.0.0-rc6/mcp.md index 32a7c1b0..aba3ad88 100644 --- a/docs/docs/2.0.0-rc5/mcp.md +++ b/docs/docs/2.0.0-rc6/mcp.md @@ -53,7 +53,7 @@ statement. This holds for every caller. Each tool carries `annotations.readOnlyH ## Expose a table to MCP A table becomes an MCP tool when you enable its -`mcp` [extension](/docs/2.0.0-rc5/server/extensions). Set the extension with SQL +`mcp` [extension](/docs/2.0.0-rc6/server/extensions). Set the extension with SQL (`SET EXTENSION`) or with the admin REST API. The optional `preset` extension adds named filter sets. @@ -143,7 +143,7 @@ script sends. ## Authenticate an agent `/mcp` authenticates with the HTTP `Authorization` header. It resolves the identity in the same way -as the [client API](/docs/2.0.0-rc5/security/access-control): +as the [client API](/docs/2.0.0-rc6/security/access-control): - **Basic**: `Authorization: Basic base64(user:pass)` gives the roles of a Beacon user. - **Bearer**: `Authorization: Bearer ` takes an OIDC or OAuth2 JWT. diff --git a/docs/docs/2.0.0-rc5/quickstart.md b/docs/docs/2.0.0-rc6/quickstart.md similarity index 87% rename from docs/docs/2.0.0-rc5/quickstart.md rename to docs/docs/2.0.0-rc6/quickstart.md index 8929f48e..3932ee10 100644 --- a/docs/docs/2.0.0-rc5/quickstart.md +++ b/docs/docs/2.0.0-rc6/quickstart.md @@ -32,11 +32,11 @@ the credentials above. It lists the datasets it found and gives you a query edit That mounted `./datasets` directory **is** the root, so a file at `./datasets/obs/a.parquet` is `obs/a.parquet` in a query. Do not repeat `datasets/` in the path, and do not write an `s3://` scheme: the server resolves everything against its own store. See -[Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +[Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). ::: That command starts one server. Put it on a workstation to try it. The same image runs in production -behind a real archive. Continue to **[Getting Started](/docs/2.0.0-rc5/getting-started)** for Docker +behind a real archive. Continue to **[Getting Started](/docs/2.0.0-rc6/getting-started)** for Docker Compose, an S3 store, access control and performance. ## Query a server @@ -70,9 +70,9 @@ df = client.sql_query( ).to_pandas_dataframe() ``` -There are clients for [TypeScript](/docs/2.0.0-rc5/connect/typescript), the -[terminal](/docs/2.0.0-rc5/connect/cli), and any JDBC or ADBC tool over -[Arrow Flight SQL](/docs/2.0.0-rc5/connect/datagrip). +There are clients for [TypeScript](/docs/2.0.0-rc6/connect/typescript), the +[terminal](/docs/2.0.0-rc6/connect/cli), and any JDBC or ADBC tool over +[Arrow Flight SQL](/docs/2.0.0-rc6/connect/datagrip). ### Query the public node @@ -145,15 +145,15 @@ JOIN wod."easy-wod" ON local_files.time = wod."easy-wod".time; ``` `ATTACH` uses Arrow Flight SQL. The remote server must open its Flight SQL port. It must also permit -the connection. See [ATTACH](/docs/2.0.0-rc5/data-sources/attach) for authentication. See -[Remote Tables](/docs/2.0.0-rc5/sql/remote-tables) to attach one table instead of a whole +the connection. See [ATTACH](/docs/2.0.0-rc6/data-sources/attach) for authentication. See +[Remote Tables](/docs/2.0.0-rc6/sql/remote-tables) to attach one table instead of a whole catalog. ## Next | | | |---|---| -| **Run it properly** | [Getting Started](/docs/2.0.0-rc5/getting-started) · [Configuration](/docs/2.0.0-rc5/server/configuration) | -| **Name your data** | [Server Setup](/docs/2.0.0-rc5/server/) | -| **Write queries** | [SQL Reference](/docs/2.0.0-rc5/sql/) | -| **Coming from Python** | [Coming from xarray](/docs/2.0.0-rc5/coming-from-xarray) | +| **Run it properly** | [Getting Started](/docs/2.0.0-rc6/getting-started) · [Configuration](/docs/2.0.0-rc6/server/configuration) | +| **Name your data** | [Server Setup](/docs/2.0.0-rc6/server/) | +| **Write queries** | [SQL Reference](/docs/2.0.0-rc6/sql/) | +| **Coming from Python** | [Coming from xarray](/docs/2.0.0-rc6/coming-from-xarray) | diff --git a/docs/docs/2.0.0-rc5/security/access-control.md b/docs/docs/2.0.0-rc6/security/access-control.md similarity index 99% rename from docs/docs/2.0.0-rc5/security/access-control.md rename to docs/docs/2.0.0-rc6/security/access-control.md index 2761ff14..7f341e46 100644 --- a/docs/docs/2.0.0-rc5/security/access-control.md +++ b/docs/docs/2.0.0-rc6/security/access-control.md @@ -263,5 +263,5 @@ directly. That check runs before both providers. Over HTTP, a client sends its credentials with **Basic auth**. With OIDC it sends a `Bearer` token. Arrow Flight SQL authenticates in its handshake. It then issues a bearer token. See the -[Flight SQL settings](/docs/2.0.0-rc5/server/configuration#arrow-flight-sql). +[Flight SQL settings](/docs/2.0.0-rc6/server/configuration#arrow-flight-sql). ::: diff --git a/docs/docs/2.0.0-rc5/server/configuration.md b/docs/docs/2.0.0-rc6/server/configuration.md similarity index 97% rename from docs/docs/2.0.0-rc5/server/configuration.md rename to docs/docs/2.0.0-rc6/server/configuration.md index 4184c387..44b178f9 100644 --- a/docs/docs/2.0.0-rc5/server/configuration.md +++ b/docs/docs/2.0.0-rc6/server/configuration.md @@ -42,7 +42,7 @@ HTTP, and the admin endpoints. Beacon adds role-based access control on top of the super-user above. It gives read-only users and roles in SQL. It gives grants and denies on a table or a path. It also gives anonymous access and optional OIDC. The -[Access Control guide](/docs/2.0.0-rc5/security/access-control) holds the full +[Access Control guide](/docs/2.0.0-rc6/security/access-control) holds the full model and the SQL reference. These variables control it: | Variable | Default | Description | @@ -61,7 +61,7 @@ model and the SQL reference. These variables control it: The master key encrypts the stored credentials at rest. Today it covers the `password` of an external -[SQL database table](/docs/2.0.0-rc5/data-sources/sql-databases). You need +[SQL database table](/docs/2.0.0-rc6/data-sources/sql-databases). You need the key to create a database table with a password. Without the key, Beacon rejects that `CREATE`. Beacon never writes plaintext. @@ -80,8 +80,8 @@ rejects that `CREATE`. Beacon never writes plaintext. | `BEACON_ENABLE_ND_PIPELINE` | `false` | Enable the N-dimensional pipeline optimizer for zarr/netcdf reads: sink element-wise projections below the grid broadcast so `lat * 2` and similar run on the coordinate axis instead of the full cross-product. The base nd pipeline always runs; this only enables the node-rewriting optimization. | | `BEACON_BATCH_SIZE` | `64000` | Batch size, in rows, for NetCDF reads (local and MPIO). | | `BEACON_STATS_CACHE_CAPACITY` | `10000` | Maximum number of per-file statistics entries cached for query pruning. Read once at startup. | -| `BEACON_TYPE_WIDENING_STRATEGY` | `default` | The rule a schema merge applies to a column that two files type in two ways. `default` widens inside one family: a wider integer, a finer timestamp, a longer string. It refuses a boolean beside a number and a number beside a string, and it reads every integer beside a `Float32` as `Float64`. `numpy` promotes as `numpy.result_type` does: a boolean joins the numbers, `Float16` joins the floats, a narrow integer beside a `Float32` stays a `Float32`, a number beside a string reads as text, and a date beside a timestamp is a timestamp. numpy resolves the set of types of a column at once, so the listing order does not change the result. `numpy` reads every schema in one pass, as `keep_first` does. An unknown value logs a warning and reads as `default`. See [a column has two types](/docs/2.0.0-rc5/troubleshooting#a-column-has-two-types-across-the-files). | -| `BEACON_TYPE_WIDENING_ON_CONFLICT` | `fail` | What a schema merge does with a column that two files type in two families, such as a number and a timestamp. `fail` refuses the collection and names the column, both types and both files. `keep_first` keeps the type of the first file, casts every other file to it, and reads a value that type cannot hold as null. A pair the strategy widens, such as `Int32` beside `Float64`, widens either way. An unknown value logs a warning and reads as `fail`. See [a column has two types](/docs/2.0.0-rc5/troubleshooting#a-column-has-two-types-across-the-files). | +| `BEACON_TYPE_WIDENING_STRATEGY` | `default` | The rule a schema merge applies to a column that two files type in two ways. `default` widens inside one family: a wider integer, a finer timestamp, a longer string. It refuses a boolean beside a number and a number beside a string, and it reads every integer beside a `Float32` as `Float64`. `numpy` promotes as `numpy.result_type` does: a boolean joins the numbers, `Float16` joins the floats, a narrow integer beside a `Float32` stays a `Float32`, a number beside a string reads as text, and a date beside a timestamp is a timestamp. numpy resolves the set of types of a column at once, so the listing order does not change the result. `numpy` reads every schema in one pass, as `keep_first` does. An unknown value logs a warning and reads as `default`. See [a column has two types](/docs/2.0.0-rc6/troubleshooting#a-column-has-two-types-across-the-files). | +| `BEACON_TYPE_WIDENING_ON_CONFLICT` | `fail` | What a schema merge does with a column that two files type in two families, such as a number and a timestamp. `fail` refuses the collection and names the column, both types and both files. `keep_first` keeps the type of the first file, casts every other file to it, and reads a value that type cannot hold as null. A pair the strategy widens, such as `Int32` beside `Float64`, widens either way. An unknown value logs a warning and reads as `fail`. See [a column has two types](/docs/2.0.0-rc6/troubleshooting#a-column-has-two-types-across-the-files). | ### The default table @@ -125,7 +125,7 @@ many small batches. Beacon also gives an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) endpoint on its own port. Clients such as JetBrains DataGrip and the Python ADBC -driver use it. See [Connect](/docs/2.0.0-rc5/connect/datagrip). Flight +driver use it. See [Connect](/docs/2.0.0-rc6/connect/datagrip). Flight SQL authenticates with a bearer token. The HTTP API works differently. | Variable | Default | Description | @@ -230,7 +230,7 @@ environment chain. These cover the **datasets store itself**, which is what every query reads through. Paths in SQL stay relative to that store's root, so a client never names the bucket and never supplies a -credential. See [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +credential. See [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). | Variable | Default | Description | | --- | --- | --- | @@ -242,7 +242,7 @@ credential. See [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). ## Crawler -A [crawler](/docs/2.0.0-rc5/server/crawlers) finds the files under a prefix. It +A [crawler](/docs/2.0.0-rc6/server/crawlers) finds the files under a prefix. It then registers them as external tables. | Variable | Default | Description | @@ -253,7 +253,7 @@ then registers them as external tables. ## File statistics Beacon records the value range of each column in each file. A query then prunes the files that -cannot match. See [File statistics](/docs/2.0.0-rc5/internals/file-statistics). +cannot match. See [File statistics](/docs/2.0.0-rc6/internals/file-statistics). Beacon enables this feature by default. The pure-Rust readers are the default for netCDF and HDF5 (see [File formats](#file-formats)). Beacon records a real range for those formats. A server that @@ -290,7 +290,7 @@ store now. Set `BEACON_FILE_STATS_ON_STARTUP=true` to collect at each boot. ## File formats These settings tune one format each. See -[Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning) to know when to +[Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning) to know when to change them. ### NetCDF @@ -315,7 +315,7 @@ can move one format at a time. The pure-Rust reader also reads two layouts the netCDF data model cannot express: a nested group, and a compound dataset. See -[Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning#hdf5-pure-rust-reader). +[Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning#hdf5-pure-rust-reader). ### Zarr diff --git a/docs/docs/2.0.0-rc5/server/crawlers.md b/docs/docs/2.0.0-rc6/server/crawlers.md similarity index 95% rename from docs/docs/2.0.0-rc5/server/crawlers.md rename to docs/docs/2.0.0-rc6/server/crawlers.md index fe226dc9..c84d5e66 100644 --- a/docs/docs/2.0.0-rc5/server/crawlers.md +++ b/docs/docs/2.0.0-rc6/server/crawlers.md @@ -9,7 +9,7 @@ RUN CRAWLER argo; ``` A **crawler** finds the datasets in the storage of Beacon. It registers them as -[external tables](/docs/2.0.0-rc5/data-sources/external-tables). You therefore write no +[external tables](/docs/2.0.0-rc6/data-sources/external-tables). You therefore write no `CREATE EXTERNAL TABLE` for each dataset. A crawler scans a prefix. It groups the files by format. It detects Hive-style partitions. It infers the schema of each table. It then registers a table for each dataset. A crawler also keeps the catalog current when new files arrive. It runs on a schedule @@ -21,7 +21,7 @@ survive a restart. You run `SELECT`, `JOIN` and `DROP` on them, like any other t :::tip When to use a crawler Use a crawler when you have **many** datasets under a prefix. Such datasets often use a partition by date or by region. The crawler registers them and keeps them current. You write no DDL by hand. For -one table, a plain [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table) is simpler. +one table, a plain [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table) is simpler. ::: You can send the crawler DDL through any SQL interface of Beacon: @@ -196,7 +196,7 @@ readers open those files directly. Register such a file with a table function or The crawler **skips** a store with a directory and a marker file. **Zarr** (`*.zarr/zarr.json`) is such a store. The listing path does not register it as an external table. Read a Zarr store with -[`read_zarr`](/docs/2.0.0-rc5/sql/table-functions#read-zarr). A crawl ignores these stores +[`read_zarr`](/docs/2.0.0-rc6/sql/table-functions#read-zarr). A crawl ignores these stores and continues with the other datasets. Register them with a table function or with `CREATE EXTERNAL TABLE`. @@ -213,10 +213,10 @@ table, also with `geo` metadata. To get geometry decoding for such a file, give The crawler also **skips** a **Delta Lake** table. Such a table is a directory with a `_delta_log/` folder. Register it with -[`CREATE EXTERNAL TABLE ... STORED AS DELTA`](/docs/2.0.0-rc5/formats/delta-lake). +[`CREATE EXTERNAL TABLE ... STORED AS DELTA`](/docs/2.0.0-rc6/formats/delta-lake). ## See also -- [External Tables](/docs/2.0.0-rc5/data-sources/external-tables): the tables that a crawler produces -- [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc5/sql/create-external-table): the manual form, with `PARTITIONED BY` -- [Configuration](/docs/2.0.0-rc5/server/configuration): every Beacon setting +- [External Tables](/docs/2.0.0-rc6/data-sources/external-tables): the tables that a crawler produces +- [`CREATE EXTERNAL TABLE`](/docs/2.0.0-rc6/sql/create-external-table): the manual form, with `PARTITIONED BY` +- [Configuration](/docs/2.0.0-rc6/server/configuration): every Beacon setting diff --git a/docs/docs/2.0.0-rc5/server/datasets.md b/docs/docs/2.0.0-rc6/server/datasets.md similarity index 60% rename from docs/docs/2.0.0-rc5/server/datasets.md rename to docs/docs/2.0.0-rc6/server/datasets.md index 44c09d36..442de302 100644 --- a/docs/docs/2.0.0-rc5/server/datasets.md +++ b/docs/docs/2.0.0-rc6/server/datasets.md @@ -6,8 +6,8 @@ description: The file formats that Beacon finds in its datasets store. This page Beacon finds the datasets in its storage root automatically. You register nothing. Copy your files into the datasets folder or the S3 prefix. You can then query them at once. Use a -[reader function](/docs/2.0.0-rc5/formats/) or an -[external table](/docs/2.0.0-rc5/data-sources/external-tables). +[reader function](/docs/2.0.0-rc6/formats/) or an +[external table](/docs/2.0.0-rc6/data-sources/external-tables). The default local path in the Docker container is `/beacon/data/datasets/`. @@ -15,28 +15,28 @@ The default local path in the Docker container is `/beacon/data/datasets/`. | Format | Recognized files | `STORED AS` | `read_*` function | Output format | | --- | --- | --- | --- | --- | -| [Parquet](/docs/2.0.0-rc5/formats/parquet) | `.parquet` | `PARQUET` | `read_parquet` | yes | -| [GeoParquet](/docs/2.0.0-rc5/formats/geoparquet) | `.geoparquet` | `GEOPARQUET` | `read_geoparquet` | yes | -| [CSV / TSV](/docs/2.0.0-rc5/formats/csv) | `.csv`, `.tsv` | `CSV` | `read_csv` | yes | -| [Arrow IPC](/docs/2.0.0-rc5/formats/arrow) | `.arrow`, `.feather` | `ARROW` | `read_arrow` | yes (`ipc`) | -| [NetCDF](/docs/2.0.0-rc5/formats/netcdf) | `.nc` | `NC` | `read_netcdf` | yes (+ ND-NetCDF) | -| [Zarr](/docs/2.0.0-rc5/formats/zarr) | `zarr.json` marker | `ZARR` | `read_zarr` | no | -| [Atlas](/docs/2.0.0-rc5/formats/atlas) | `data.atlas` file | `ATLAS` | `read_atlas` | yes | -| [GeoTIFF / COG](/docs/2.0.0-rc5/formats/geotiff) | `.tif`, `.tiff` | `TIFF` | `read_tiff` | no | -| [BBF](/docs/2.0.0-rc5/formats/bbf) | `.bbf` | `BBF` | `read_bbf` | no | -| [Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) | `_delta_log/` directory | `DELTA` | `read_delta` | no | -| [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) | `metadata/` directory | `ICEBERG` | `read_iceberg` | no | -| [ODV ASCII](/docs/2.0.0-rc5/formats/odv) | `.txt` | not supported | `read_odv_ascii` | yes | +| [Parquet](/docs/2.0.0-rc6/formats/parquet) | `.parquet` | `PARQUET` | `read_parquet` | yes | +| [GeoParquet](/docs/2.0.0-rc6/formats/geoparquet) | `.geoparquet` | `GEOPARQUET` | `read_geoparquet` | yes | +| [CSV / TSV](/docs/2.0.0-rc6/formats/csv) | `.csv`, `.tsv` | `CSV` | `read_csv` | yes | +| [Arrow IPC](/docs/2.0.0-rc6/formats/arrow) | `.arrow`, `.feather` | `ARROW` | `read_arrow` | yes (`ipc`) | +| [NetCDF](/docs/2.0.0-rc6/formats/netcdf) | `.nc` | `NC` | `read_netcdf` | yes (+ ND-NetCDF) | +| [Zarr](/docs/2.0.0-rc6/formats/zarr) | `zarr.json` marker | `ZARR` | `read_zarr` | no | +| [Atlas](/docs/2.0.0-rc6/formats/atlas) | `data.atlas` file | `ATLAS` | `read_atlas` | yes | +| [GeoTIFF / COG](/docs/2.0.0-rc6/formats/geotiff) | `.tif`, `.tiff` | `TIFF` | `read_tiff` | no | +| [BBF](/docs/2.0.0-rc6/formats/bbf) | `.bbf` | `BBF` | `read_bbf` | no | +| [Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) | `_delta_log/` directory | `DELTA` | `read_delta` | no | +| [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) | `metadata/` directory | `ICEBERG` | `read_iceberg` | no | +| [ODV ASCII](/docs/2.0.0-rc6/formats/odv) | `.txt` | not supported | `read_odv_ascii` | yes | Beacon finds every format above in the datasets store. **Delta Lake**, **Apache Iceberg** and **ODV -ASCII** are the exception. Point a [`read_*` function](/docs/2.0.0-rc5/formats/) at them. For +ASCII** are the exception. Point a [`read_*` function](/docs/2.0.0-rc6/formats/) at them. For Delta and Iceberg, `CREATE EXTERNAL TABLE … STORED AS DELTA|ICEBERG LOCATION …` also works. The "Output format" column marks the formats that -[`output.format`](/docs/2.0.0-rc5/api/querying/#output-formats) can export a query result to. +[`output.format`](/docs/2.0.0-rc6/api/querying/#output-formats) can export a query result to. :::tip Per-format reference Each format has its own chapter. The chapter covers the read behaviour, the attribute columns, the -limitations and the tuning. See [External Files](/docs/2.0.0-rc5/formats/). +limitations and the tuning. See [External Files](/docs/2.0.0-rc6/formats/). ::: ## Where files live @@ -45,15 +45,15 @@ The datasets store is a local directory or an S3-compatible bucket: - **Local disk**: the files under the datasets folder. Mount that folder into the container. - **Object storage**: an S3, GCS or Azure prefix. See - [Object Storage](/docs/2.0.0-rc5/data-sources/object-storage) for the credentials and the + [Object Storage](/docs/2.0.0-rc6/data-sources/object-storage) for the credentials and the setup. Beacon reads the files in place. Beacon never copies or converts them. A new file is queryable at -once. Use a [crawler](/docs/2.0.0-rc5/server/crawlers) to register many datasets under a prefix +once. Use a [crawler](/docs/2.0.0-rc6/server/crawlers) to register many datasets under a prefix as named tables in one step. A crawler also handles a partitioned layout. ## Next -- **[External Files](/docs/2.0.0-rc5/formats/)**: the read reference for each format. -- **[Create External Tables](/docs/2.0.0-rc5/data-sources/external-tables)**: give a set of files a stable table name. -- **[Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning)**: the layout and format choices that make a scan faster. +- **[External Files](/docs/2.0.0-rc6/formats/)**: the read reference for each format. +- **[Create External Tables](/docs/2.0.0-rc6/data-sources/external-tables)**: give a set of files a stable table name. +- **[Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning)**: the layout and format choices that make a scan faster. diff --git a/docs/docs/2.0.0-rc5/server/extensions.md b/docs/docs/2.0.0-rc6/server/extensions.md similarity index 93% rename from docs/docs/2.0.0-rc5/server/extensions.md rename to docs/docs/2.0.0-rc6/server/extensions.md index 3fddfaea..87496cb9 100644 --- a/docs/docs/2.0.0-rc5/server/extensions.md +++ b/docs/docs/2.0.0-rc6/server/extensions.md @@ -14,7 +14,7 @@ table. It survives a restart. Beacon has two extensions today: -- **`mcp`**: give the table to AI agents as a read-only [MCP](/docs/2.0.0-rc5/mcp) tool. +- **`mcp`**: give the table to AI agents as a read-only [MCP](/docs/2.0.0-rc6/mcp) tool. - **`preset`**: define named filter sets. Other features, such as the `mcp` tool, use them by name. --> @@ -47,7 +47,7 @@ silence. ## The `mcp` extension Enable the `mcp` extension to give the table to AI agents. Beacon then generates a tool for that -table on its [MCP Server](/docs/2.0.0-rc5/mcp). The tool is **read-only**. It covers the columns +table on its [MCP Server](/docs/2.0.0-rc6/mcp). The tool is **read-only**. It covers the columns that you expose. ```sql @@ -89,7 +89,7 @@ SET EXTENSION 'mcp' FOR obs TO '{ ``` These values are hints only. The built-in `run_sql` preview limit controls the result size. The -[MCP Server guide](/docs/2.0.0-rc5/mcp) gives the full tool set. It also shows how to connect a +[MCP Server guide](/docs/2.0.0-rc6/mcp) gives the full tool set. It also shows how to connect a client and how to authenticate an agent. --> @@ -133,7 +133,7 @@ with `AND`. ## See also -- [External Tables](/docs/2.0.0-rc5/data-sources/external-tables): register the tables that take an extension. -- [Access control](/docs/2.0.0-rc5/security/access-control): the identity and role grants that control what an agent reads. +- [External Tables](/docs/2.0.0-rc6/data-sources/external-tables): register the tables that take an extension. +- [Access control](/docs/2.0.0-rc6/security/access-control): the identity and role grants that control what an agent reads. diff --git a/docs/docs/2.0.0-rc5/server/index.md b/docs/docs/2.0.0-rc6/server/index.md similarity index 75% rename from docs/docs/2.0.0-rc5/server/index.md rename to docs/docs/2.0.0-rc6/server/index.md index 1c0a7d48..8c3ecfde 100644 --- a/docs/docs/2.0.0-rc5/server/index.md +++ b/docs/docs/2.0.0-rc6/server/index.md @@ -14,14 +14,14 @@ at query time. A managed table is the one exception. Beacon owns those rows. | | What it covers | |---|---| -| **[Datasets & formats](/docs/2.0.0-rc5/server/datasets)** | Which files Beacon recognizes, and what each format supports | -| **[Tables & views](/docs/2.0.0-rc5/data-sources/)** | Giving files names: external tables, views, materialized views, crawlers, managed tables | -| **[Other sources](/docs/2.0.0-rc5/data-sources/object-storage)** | Data the node does not hold: a bucket, a Postgres database, another Beacon node | +| **[Datasets & formats](/docs/2.0.0-rc6/server/datasets)** | Which files Beacon recognizes, and what each format supports | +| **[Tables & views](/docs/2.0.0-rc6/data-sources/)** | Giving files names: external tables, views, materialized views, crawlers, managed tables | +| **[Other sources](/docs/2.0.0-rc6/data-sources/object-storage)** | Data the node does not hold: a bucket, a Postgres database, another Beacon node | -Before this, [deploy the node](/docs/2.0.0-rc5/getting-started) and -[configure it](/docs/2.0.0-rc5/server/configuration). After it, -[decide who may read what](/docs/2.0.0-rc5/security/access-control) and -[point clients at it](/docs/2.0.0-rc5/connect/python). +Before this, [deploy the node](/docs/2.0.0-rc6/getting-started) and +[configure it](/docs/2.0.0-rc6/server/configuration). After it, +[decide who may read what](/docs/2.0.0-rc6/security/access-control) and +[point clients at it](/docs/2.0.0-rc6/connect/python). ## Core concepts @@ -29,12 +29,12 @@ Before this, [deploy the node](/docs/2.0.0-rc5/getting-started) and dataset directly. It is the smallest unit in Beacon. - **External tables**: a registered name over one or more files. Give a folder or a glob pattern. Beacon merges the schemas. You query the files as one table. See - [External Tables](/docs/2.0.0-rc5/data-sources/external-tables). + [External Tables](/docs/2.0.0-rc6/data-sources/external-tables). - **Managed tables**: tables that Beacon owns. You change them with `INSERT`, `UPDATE` and `DELETE`. The Lance engine holds them by default. Iceberg is the other option. See - [Managed Tables](/docs/2.0.0-rc5/sql/managed-tables). + [Managed Tables](/docs/2.0.0-rc6/sql/managed-tables). - **Views**: a saved query that behaves like a table. See - [Views](/docs/2.0.0-rc5/server/view). + [Views](/docs/2.0.0-rc6/server/view). - **Metadata and schema**: Beacon reads the dataset metadata and builds the schemas. You can therefore see the available columns before you write a query. - **Pushdown and partitions**: Beacon pushes filters and projections down. This reduces the I/O and @@ -42,7 +42,7 @@ Before this, [deploy the node](/docs/2.0.0-rc5/getting-started) and ## How it works at a glance @@ -52,23 +52,23 @@ Before this, [deploy the node](/docs/2.0.0-rc5/getting-started) and 3. **Query a dataset or a table** with SQL or with the JSON query DSL. -For the full detail, see the [SQL query docs](/docs/2.0.0-rc5/api/querying/sql) and the -[JSON query docs](/docs/2.0.0-rc5/api/querying/json). +For the full detail, see the [SQL query docs](/docs/2.0.0-rc6/api/querying/sql) and the +[JSON query docs](/docs/2.0.0-rc6/api/querying/json). diff --git a/docs/docs/2.0.0-rc5/server/performance-tuning.md b/docs/docs/2.0.0-rc6/server/performance-tuning.md similarity index 97% rename from docs/docs/2.0.0-rc5/server/performance-tuning.md rename to docs/docs/2.0.0-rc6/server/performance-tuning.md index 0e775bce..a4796784 100644 --- a/docs/docs/2.0.0-rc5/server/performance-tuning.md +++ b/docs/docs/2.0.0-rc6/server/performance-tuning.md @@ -78,7 +78,7 @@ These settings decide if the datasets store lives on an S3-compatible bucket. Th address form. Every listing and every read on object storage costs network latency. - Put Beacon near the object store in the network. This gives better performance. -- Are the listings slow? Then use a [crawler](/docs/2.0.0-rc5/server/crawlers). It keeps the +- Are the listings slow? Then use a [crawler](/docs/2.0.0-rc6/server/crawlers). It keeps the catalog current in the background. Beacon then does not scan at query time. ## NetCDF Tuning @@ -105,7 +105,7 @@ the work blocks queries. Your core count does not change this. With `BEACON_NETCDF_USE_RUST_READER=false`, Beacon reports no statistics for netCDF. It prunes no file. This variable does not change that result. See -[File statistics](/docs/2.0.0-rc5/internals/file-statistics). +[File statistics](/docs/2.0.0-rc6/internals/file-statistics). ### Pure-Rust reader (parallel reads and object storage) @@ -221,13 +221,13 @@ Content-Type: application/json ::: tip Do you query a collection often? Then convert the Zarr stores into one -[Atlas](/docs/2.0.0-rc5/formats/atlas) collection. Atlas adds dataset pruning +[Atlas](/docs/2.0.0-rc6/formats/atlas) collection. Atlas adds dataset pruning with statistics, next to the chunk pruning. It drops whole datasets before it reads a chunk. ::: ## Atlas Tuning -Beacon opens an [Atlas](/docs/2.0.0-rc5/formats/atlas) collection by reading the footer of its +Beacon opens an [Atlas](/docs/2.0.0-rc6/formats/atlas) collection by reading the footer of its `data.atlas` file. Each table keeps its open collections in a cache of 512 entries, so a query does not read that footer again. Each cached collection holds its own block cache, 256 MiB of decompressed blocks and 64 MiB of raw slabs, so the cache is a memory bound as much as a handle diff --git a/docs/docs/2.0.0-rc5/server/view.md b/docs/docs/2.0.0-rc6/server/view.md similarity index 100% rename from docs/docs/2.0.0-rc5/server/view.md rename to docs/docs/2.0.0-rc6/server/view.md diff --git a/docs/docs/2.0.0-rc5/sql/create-external-table.md b/docs/docs/2.0.0-rc6/sql/create-external-table.md similarity index 75% rename from docs/docs/2.0.0-rc5/sql/create-external-table.md rename to docs/docs/2.0.0-rc6/sql/create-external-table.md index bcc8099a..79f5fa9e 100644 --- a/docs/docs/2.0.0-rc5/sql/create-external-table.md +++ b/docs/docs/2.0.0-rc6/sql/create-external-table.md @@ -10,7 +10,7 @@ An external table is a SQL table over files in the storage of Beacon. After you query it with `SELECT` and `JOIN`. A `VIEW` can also reference it. Beacon reads the files on demand. It does not copy them. A table definition survives a restart. -For a table that Beacon owns and writes, see [`CREATE TABLE`](/docs/2.0.0-rc5/sql/managed-tables). +For a table that Beacon owns and writes, see [`CREATE TABLE`](/docs/2.0.0-rc6/sql/managed-tables). ## Syntax @@ -24,7 +24,7 @@ LOCATION '' ::: tip Names keep their case `` means exactly what you write. `MyTable` and `mytable` are two different tables. -See [Identifiers and case](/docs/2.0.0-rc5/sql/identifiers). +See [Identifiers and case](/docs/2.0.0-rc6/sql/identifiers). ::: Beacon resolves `LOCATION` against its storage root. Give a folder or a glob pattern: @@ -41,32 +41,32 @@ CREATE EXTERNAL TABLE argo STORED AS NC LOCATION 'argo/**/*.nc' | `STORED AS` | File types | Format page | | ----------- | ---------- | ----------- | -| `PARQUET` | `.parquet` | [Parquet](/docs/2.0.0-rc5/formats/parquet) | -| `GEOPARQUET` | `.geoparquet` | [GeoParquet](/docs/2.0.0-rc5/formats/geoparquet) | -| `NC` | `.nc` | [NetCDF](/docs/2.0.0-rc5/formats/netcdf) | -| `HDF5`, `H5` | `.h5`, `.hdf5` | [HDF5](/docs/2.0.0-rc5/formats/hdf5) | -| `ZARR` | Zarr v3 (`zarr.json`) | [Zarr](/docs/2.0.0-rc5/formats/zarr) | -| `ATLAS` | Atlas collection (`data.atlas`) | [Atlas](/docs/2.0.0-rc5/formats/atlas) | -| `CSV` | `.csv`, `.tsv` | [CSV](/docs/2.0.0-rc5/formats/csv) | -| `ARROW` | Arrow IPC (`.arrow`, `.feather`) | [Arrow IPC](/docs/2.0.0-rc5/formats/arrow) | -| `TIFF` | GeoTIFF / Cloud-Optimized GeoTIFF | [GeoTIFF](/docs/2.0.0-rc5/formats/geotiff) | -| `BBF` | Beacon Binary Format | [BBF](/docs/2.0.0-rc5/formats/bbf) | -| `DELTA` | Delta Lake table directory (`_delta_log/`) | [Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) | -| `ICEBERG` | Apache Iceberg table directory (`metadata/`) | [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) | -| `ICECHUNK` | Icechunk repository directory | [Icechunk](/docs/2.0.0-rc5/formats/icechunk) | -| `POSTGRES` | External PostgreSQL table (federated) | [SQL Databases](/docs/2.0.0-rc5/data-sources/sql-databases) | -| `MYSQL` | External MySQL table (federated) | [SQL Databases](/docs/2.0.0-rc5/data-sources/sql-databases) | -| `REMOTE` | A table on another Beacon server | [Remote Tables](/docs/2.0.0-rc5/data-sources/remote-tables) | +| `PARQUET` | `.parquet` | [Parquet](/docs/2.0.0-rc6/formats/parquet) | +| `GEOPARQUET` | `.geoparquet` | [GeoParquet](/docs/2.0.0-rc6/formats/geoparquet) | +| `NC` | `.nc` | [NetCDF](/docs/2.0.0-rc6/formats/netcdf) | +| `HDF5`, `H5` | `.h5`, `.hdf5` | [HDF5](/docs/2.0.0-rc6/formats/hdf5) | +| `ZARR` | Zarr v3 (`zarr.json`) | [Zarr](/docs/2.0.0-rc6/formats/zarr) | +| `ATLAS` | Atlas collection (`data.atlas`) | [Atlas](/docs/2.0.0-rc6/formats/atlas) | +| `CSV` | `.csv`, `.tsv` | [CSV](/docs/2.0.0-rc6/formats/csv) | +| `ARROW` | Arrow IPC (`.arrow`, `.feather`) | [Arrow IPC](/docs/2.0.0-rc6/formats/arrow) | +| `TIFF` | GeoTIFF / Cloud-Optimized GeoTIFF | [GeoTIFF](/docs/2.0.0-rc6/formats/geotiff) | +| `BBF` | Beacon Binary Format | [BBF](/docs/2.0.0-rc6/formats/bbf) | +| `DELTA` | Delta Lake table directory (`_delta_log/`) | [Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) | +| `ICEBERG` | Apache Iceberg table directory (`metadata/`) | [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) | +| `ICECHUNK` | Icechunk repository directory | [Icechunk](/docs/2.0.0-rc6/formats/icechunk) | +| `POSTGRES` | External PostgreSQL table (federated) | [SQL Databases](/docs/2.0.0-rc6/data-sources/sql-databases) | +| `MYSQL` | External MySQL table (federated) | [SQL Databases](/docs/2.0.0-rc6/data-sources/sql-databases) | +| `REMOTE` | A table on another Beacon server | [Remote Tables](/docs/2.0.0-rc6/data-sources/remote-tables) | `DELTA` points at an existing -[Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) table directory. It also -supports time travel and `INSERT INTO`. `ICEBERG` points to an [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) table +[Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) table directory. It also +supports time travel and `INSERT INTO`. `ICEBERG` points to an [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) table directory that already exists. It supports time travel. It reads each new snapshot. It is read-only. `REMOTE` federates a table on another Beacon server. See -[Remote Tables](/docs/2.0.0-rc5/data-sources/remote-tables). `POSTGRES` and `MYSQL` +[Remote Tables](/docs/2.0.0-rc6/data-sources/remote-tables). `POSTGRES` and `MYSQL` federate a table in an external SQL database. See -[SQL Databases](/docs/2.0.0-rc5/data-sources/sql-databases). Their `LOCATION` is the remote +[SQL Databases](/docs/2.0.0-rc6/data-sources/sql-databases). Their `LOCATION` is the remote table name. The connection details go in `OPTIONS`, with an encrypted `password`. A Zarr table must point at a `zarr.json` entry file. An Atlas collection is a single file, so an @@ -79,7 +79,7 @@ CREATE EXTERNAL TABLE sensor STORED AS ATLAS LOCATION 'sensor/data.atlas' ``` `GEOPARQUET` reads Parquet files. Beacon decodes their geometry columns to native GeoArrow. See -[GeoParquet in File Formats](/docs/2.0.0-rc5/formats/geoparquet) for the read +[GeoParquet in File Formats](/docs/2.0.0-rc6/formats/geoparquet) for the read behaviour and for geometry queries. ## `IF NOT EXISTS` @@ -154,7 +154,7 @@ The rules are the same for every format: - A key with a list value takes the members separated by a comma, in one string. Most keys have a server-wide default. The -[File formats](/docs/2.0.0-rc5/server/configuration#file-formats) settings hold those defaults. A +[File formats](/docs/2.0.0-rc6/server/configuration#file-formats) settings hold those defaults. A table option wins over the default of the server. The one exception is `enable_statistics`: Beacon validates it and then reads the server setting alone. See the format page of each key. @@ -162,17 +162,17 @@ validates it and then reads the server setting alone. See the format page of eac | `STORED AS` | Keys | Details | | ----------- | ---- | ------- | -| `NC` | `read_dimensions`, `use_rust_reader`, `enable_statistics` | [NetCDF](/docs/2.0.0-rc5/formats/netcdf#options) | -| `HDF5`, `H5` | `read_dimensions`, `use_rust_reader`, `enable_statistics`, `unify_phony_dimensions`, `convention` | [HDF5](/docs/2.0.0-rc5/formats/hdf5#options) | -| `ZARR` | `read_dimensions`, `enable_statistics` | [Zarr](/docs/2.0.0-rc5/formats/zarr#options) | -| `ATLAS` | `read_dimensions` | [Atlas](/docs/2.0.0-rc5/formats/atlas#options) | -| `CSV` | `delimiter`, `infer_records` | [CSV](/docs/2.0.0-rc5/formats/csv#options) | -| `BBF` | `split_streams_slice` | [BBF](/docs/2.0.0-rc5/formats/bbf#options) | -| `DELTA` | `version`, `timestamp` | [Delta Lake](/docs/2.0.0-rc5/formats/delta-lake#options) | -| `ICEBERG` | `snapshot_id` | [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg#options) | -| `ICECHUNK` | `branch`, `tag`, `snapshot`, `read_dimensions` | [Icechunk](/docs/2.0.0-rc5/formats/icechunk#options) | -| `POSTGRES`, `MYSQL` | `host`, `port`, `user`, `password`, `database`, `sslmode` | [SQL Databases](/docs/2.0.0-rc5/data-sources/sql-databases#options) | -| `REMOTE` | `tls` | [Remote Tables](/docs/2.0.0-rc5/data-sources/remote-tables#options) | +| `NC` | `read_dimensions`, `use_rust_reader`, `enable_statistics` | [NetCDF](/docs/2.0.0-rc6/formats/netcdf#options) | +| `HDF5`, `H5` | `read_dimensions`, `use_rust_reader`, `enable_statistics`, `unify_phony_dimensions`, `convention` | [HDF5](/docs/2.0.0-rc6/formats/hdf5#options) | +| `ZARR` | `read_dimensions`, `enable_statistics` | [Zarr](/docs/2.0.0-rc6/formats/zarr#options) | +| `ATLAS` | `read_dimensions` | [Atlas](/docs/2.0.0-rc6/formats/atlas#options) | +| `CSV` | `delimiter`, `infer_records` | [CSV](/docs/2.0.0-rc6/formats/csv#options) | +| `BBF` | `split_streams_slice` | [BBF](/docs/2.0.0-rc6/formats/bbf#options) | +| `DELTA` | `version`, `timestamp` | [Delta Lake](/docs/2.0.0-rc6/formats/delta-lake#options) | +| `ICEBERG` | `snapshot_id` | [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg#options) | +| `ICECHUNK` | `branch`, `tag`, `snapshot`, `read_dimensions` | [Icechunk](/docs/2.0.0-rc6/formats/icechunk#options) | +| `POSTGRES`, `MYSQL` | `host`, `port`, `user`, `password`, `database`, `sslmode` | [SQL Databases](/docs/2.0.0-rc6/data-sources/sql-databases#options) | +| `REMOTE` | `tls` | [Remote Tables](/docs/2.0.0-rc6/data-sources/remote-tables#options) | | `PARQUET`, `GEOPARQUET`, `ARROW`, `TIFF` | None | | ## `DROP TABLE` @@ -193,5 +193,5 @@ SHOW TABLES; DESCRIBE ocean_profiles; ``` -The [External Tables](/docs/2.0.0-rc5/data-sources/external-tables) setup guide gives an +The [External Tables](/docs/2.0.0-rc6/data-sources/external-tables) setup guide gives an example for each format. It also shows the HTTP API that lists the tables. diff --git a/docs/docs/2.0.0-rc5/sql/create-materialized-view.md b/docs/docs/2.0.0-rc6/sql/create-materialized-view.md similarity index 93% rename from docs/docs/2.0.0-rc5/sql/create-materialized-view.md rename to docs/docs/2.0.0-rc6/sql/create-materialized-view.md index 39c1f176..f3e234ec 100644 --- a/docs/docs/2.0.0-rc5/sql/create-materialized-view.md +++ b/docs/docs/2.0.0-rc6/sql/create-materialized-view.md @@ -15,11 +15,11 @@ file. A query on the view reads that Parquet file. Beacon does not run the origi a materialized view for an expensive, repeated or aggregate query. The query can read any source that Beacon knows. Sources include registered tables, -[external tables](/docs/2.0.0-rc5/data-sources/external-tables), -[views](/docs/2.0.0-rc5/sql/create-view) and table functions such as `read_netcdf()`, +[external tables](/docs/2.0.0-rc6/data-sources/external-tables), +[views](/docs/2.0.0-rc6/sql/create-view) and table functions such as `read_netcdf()`, `read_zarr()` and `read_atlas()`. -A regular [view](/docs/2.0.0-rc5/sql/create-view) runs its query on every reference. A +A regular [view](/docs/2.0.0-rc6/sql/create-view) runs its query on every reference. A materialized view changes only when you run [`REFRESH`](#refresh). ## Syntax diff --git a/docs/docs/2.0.0-rc5/sql/create-view.md b/docs/docs/2.0.0-rc6/sql/create-view.md similarity index 91% rename from docs/docs/2.0.0-rc5/sql/create-view.md rename to docs/docs/2.0.0-rc6/sql/create-view.md index e92ad7d4..27ac90d2 100644 --- a/docs/docs/2.0.0-rc5/sql/create-view.md +++ b/docs/docs/2.0.0-rc6/sql/create-view.md @@ -29,7 +29,7 @@ CREATE OR REPLACE VIEW north_atlantic AS ## Query over a table function -A view works over a [table function](/docs/2.0.0-rc5/sql/table-functions) and over an +A view works over a [table function](/docs/2.0.0-rc6/sql/table-functions) and over an external table. Use it to give a set of files a table name: ```sql @@ -42,7 +42,7 @@ CREATE VIEW argo_2024 AS ## Combine datasets with `UNION ALL BY NAME` A view can show several datasets with different schemas as one table. See -[UNION ALL BY NAME](/docs/2.0.0-rc5/sql/union-by-name) for the column matching and the type +[UNION ALL BY NAME](/docs/2.0.0-rc6/sql/union-by-name) for the column matching and the type widening. ```sql diff --git a/docs/docs/2.0.0-rc5/sql/function-reference.md b/docs/docs/2.0.0-rc6/sql/function-reference.md similarity index 93% rename from docs/docs/2.0.0-rc5/sql/function-reference.md rename to docs/docs/2.0.0-rc6/sql/function-reference.md index 6adda362..34e96359 100644 --- a/docs/docs/2.0.0-rc5/sql/function-reference.md +++ b/docs/docs/2.0.0-rc6/sql/function-reference.md @@ -11,7 +11,7 @@ The SQL runtime of Beacon gives three families of functions: 2. **[Beacon-specific functions](#beacon-specific-functions)**: Beacon adds these functions for type conversion and vocabulary mapping. Those sections hold the full list. -3. **[Spatial functions](/docs/2.0.0-rc5/sql/spatial-functions)**: 123 functions +3. **[Spatial functions](/docs/2.0.0-rc6/sql/spatial-functions)**: 123 functions with PostGIS names, over a geometry that `ST_Point` builds from coordinate columns. Its own chapter holds the full list. @@ -288,31 +288,31 @@ FROM ocean_profiles ## Geospatial functions Beacon holds 123 spatial functions with PostGIS names: 118 scalar functions, 3 aggregate functions -and 2 window functions. The [spatial functions chapter](/docs/2.0.0-rc5/sql/spatial-functions) +and 2 window functions. The [spatial functions chapter](/docs/2.0.0-rc6/sql/spatial-functions) lists every one of them, with its arguments and its return type. A name is case insensitive, so `ST_Distance` and `st_distance` are the same function. | Group | Functions | Examples | | ----- | --------- | -------- | -| [Accessors](/docs/2.0.0-rc5/sql/spatial-functions#accessors) | 16 | `ST_X`, `ST_Y`, `ST_SRID`, `ST_GeometryType` | -| [Components](/docs/2.0.0-rc5/sql/spatial-functions#components) | 6 | `ST_StartPoint`, `ST_PointN`, `ST_GeometryN` | -| [Constructors](/docs/2.0.0-rc5/sql/spatial-functions#constructors) | 7 | `ST_Point`, `ST_MakeLine`, `ST_MakeEnvelope` | -| [Input and output](/docs/2.0.0-rc5/sql/spatial-functions#input-and-output) | 10 | `ST_AsText`, `ST_AsGeoJSON`, `ST_GeomFromText` | -| [Predicates](/docs/2.0.0-rc5/sql/spatial-functions#predicates) | 15 | `ST_Intersects`, `ST_Within`, `ST_DWithin` | -| [Measurement](/docs/2.0.0-rc5/sql/spatial-functions#measurement) | 9 | `ST_Area`, `ST_Length`, `ST_Distance` | -| [Linear reference](/docs/2.0.0-rc5/sql/spatial-functions#linear-reference) | 4 | `ST_ClosestPoint`, `ST_LineLocatePoint` | -| [Overlay](/docs/2.0.0-rc5/sql/spatial-functions#overlay) | 4 | `ST_Union`, `ST_Intersection`, `ST_Difference` | -| [Processing](/docs/2.0.0-rc5/sql/spatial-functions#processing) | 18 | `ST_Buffer`, `ST_Centroid`, `ST_Simplify` | -| [Validity](/docs/2.0.0-rc5/sql/spatial-functions#validity) | 3 | `ST_IsValid`, `ST_MakeValid` | -| [Affine](/docs/2.0.0-rc5/sql/spatial-functions#affine) | 4 | `ST_Translate`, `ST_Rotate`, `ST_Affine` | -| [Bounding box](/docs/2.0.0-rc5/sql/spatial-functions#bounding-box) | 8 | `ST_Envelope`, `ST_XMin`, `ST_YMax` | -| [Tessellation](/docs/2.0.0-rc5/sql/spatial-functions#tessellation) | 4 | `ST_DelaunayTriangles`, `ST_VoronoiPolygons` | -| [Bearings](/docs/2.0.0-rc5/sql/spatial-functions#bearings) | 2 | `ST_Azimuth`, `ST_Project` | -| [Edits](/docs/2.0.0-rc5/sql/spatial-functions#edits) | 7 | `ST_Multi`, `ST_SnapToGrid`, `ST_Dump` | -| [Aggregate functions](/docs/2.0.0-rc5/sql/spatial-functions#aggregate-functions) | 3 | `ST_Extent`, `ST_Collect`, `ST_MemUnion` | -| [Window functions](/docs/2.0.0-rc5/sql/spatial-functions#window-functions) | 2 | `ST_ClusterKMeans`, `ST_ClusterDBSCAN` | -| [Reprojection](/docs/2.0.0-rc5/sql/spatial-functions#reprojection) | 1 | `ST_Transform` | +| [Accessors](/docs/2.0.0-rc6/sql/spatial-functions#accessors) | 16 | `ST_X`, `ST_Y`, `ST_SRID`, `ST_GeometryType` | +| [Components](/docs/2.0.0-rc6/sql/spatial-functions#components) | 6 | `ST_StartPoint`, `ST_PointN`, `ST_GeometryN` | +| [Constructors](/docs/2.0.0-rc6/sql/spatial-functions#constructors) | 7 | `ST_Point`, `ST_MakeLine`, `ST_MakeEnvelope` | +| [Input and output](/docs/2.0.0-rc6/sql/spatial-functions#input-and-output) | 10 | `ST_AsText`, `ST_AsGeoJSON`, `ST_GeomFromText` | +| [Predicates](/docs/2.0.0-rc6/sql/spatial-functions#predicates) | 15 | `ST_Intersects`, `ST_Within`, `ST_DWithin` | +| [Measurement](/docs/2.0.0-rc6/sql/spatial-functions#measurement) | 9 | `ST_Area`, `ST_Length`, `ST_Distance` | +| [Linear reference](/docs/2.0.0-rc6/sql/spatial-functions#linear-reference) | 4 | `ST_ClosestPoint`, `ST_LineLocatePoint` | +| [Overlay](/docs/2.0.0-rc6/sql/spatial-functions#overlay) | 4 | `ST_Union`, `ST_Intersection`, `ST_Difference` | +| [Processing](/docs/2.0.0-rc6/sql/spatial-functions#processing) | 18 | `ST_Buffer`, `ST_Centroid`, `ST_Simplify` | +| [Validity](/docs/2.0.0-rc6/sql/spatial-functions#validity) | 3 | `ST_IsValid`, `ST_MakeValid` | +| [Affine](/docs/2.0.0-rc6/sql/spatial-functions#affine) | 4 | `ST_Translate`, `ST_Rotate`, `ST_Affine` | +| [Bounding box](/docs/2.0.0-rc6/sql/spatial-functions#bounding-box) | 8 | `ST_Envelope`, `ST_XMin`, `ST_YMax` | +| [Tessellation](/docs/2.0.0-rc6/sql/spatial-functions#tessellation) | 4 | `ST_DelaunayTriangles`, `ST_VoronoiPolygons` | +| [Bearings](/docs/2.0.0-rc6/sql/spatial-functions#bearings) | 2 | `ST_Azimuth`, `ST_Project` | +| [Edits](/docs/2.0.0-rc6/sql/spatial-functions#edits) | 7 | `ST_Multi`, `ST_SnapToGrid`, `ST_Dump` | +| [Aggregate functions](/docs/2.0.0-rc6/sql/spatial-functions#aggregate-functions) | 3 | `ST_Extent`, `ST_Collect`, `ST_MemUnion` | +| [Window functions](/docs/2.0.0-rc6/sql/spatial-functions#window-functions) | 2 | `ST_ClusterKMeans`, `ST_ClusterDBSCAN` | +| [Reprojection](/docs/2.0.0-rc6/sql/spatial-functions#reprojection) | 1 | `ST_Transform` | A netCDF, Zarr, CSV or Parquet table holds coordinate columns, not geometry. `ST_Point` builds a geometry from them, so every function above reaches every format: diff --git a/docs/docs/2.0.0-rc5/sql/group-by.md b/docs/docs/2.0.0-rc6/sql/group-by.md similarity index 95% rename from docs/docs/2.0.0-rc5/sql/group-by.md rename to docs/docs/2.0.0-rc6/sql/group-by.md index 2c9c5241..d6e867a2 100644 --- a/docs/docs/2.0.0-rc5/sql/group-by.md +++ b/docs/docs/2.0.0-rc6/sql/group-by.md @@ -55,4 +55,4 @@ ORDER BY lat_bin | `STDDEV(col)` | The standard deviation | | `MEDIAN(col)` | The median | -See [Function Reference](/docs/2.0.0-rc5/sql/function-reference#aggregate-functions) for the full list. +See [Function Reference](/docs/2.0.0-rc6/sql/function-reference#aggregate-functions) for the full list. diff --git a/docs/docs/2.0.0-rc5/sql/identifiers.md b/docs/docs/2.0.0-rc6/sql/identifiers.md similarity index 97% rename from docs/docs/2.0.0-rc5/sql/identifiers.md rename to docs/docs/2.0.0-rc6/sql/identifiers.md index 5b994f5d..9ee2ffdb 100644 --- a/docs/docs/2.0.0-rc5/sql/identifiers.md +++ b/docs/docs/2.0.0-rc6/sql/identifiers.md @@ -88,7 +88,7 @@ GRANT SELECT ON TABLE mytable TO ROLE reader -- does not cover MyTable | Keywords | Case-insensitive | `select`, `SELECT`, `SeLeCt` | | Format after `STORED AS` | Case-insensitive | `STORED AS csv`, `STORED AS CSV` | | Built-in functions | Case-insensitive | `count(*)`, `COUNT(*)` | -| [Table functions](/docs/2.0.0-rc5/sql/table-functions) | **Case-sensitive.** Always lowercase | `read_csv(…)`, never `READ_CSV(…)` | +| [Table functions](/docs/2.0.0-rc6/sql/table-functions) | **Case-sensitive.** Always lowercase | `read_csv(…)`, never `READ_CSV(…)` | | Table, column and alias names | **Case-sensitive** | `MyTable` ≠ `mytable` | ## Find the exact name diff --git a/docs/docs/2.0.0-rc5/sql/index.md b/docs/docs/2.0.0-rc6/sql/index.md similarity index 52% rename from docs/docs/2.0.0-rc5/sql/index.md rename to docs/docs/2.0.0-rc6/sql/index.md index 67ac9bf0..12645e3a 100644 --- a/docs/docs/2.0.0-rc5/sql/index.md +++ b/docs/docs/2.0.0-rc6/sql/index.md @@ -17,14 +17,14 @@ and more. Every transport below speaks the same dialect. ## What you can query -**Registered tables**: any [external table](/docs/2.0.0-rc5/data-sources/external-tables) or -[view](/docs/2.0.0-rc5/server/view): +**Registered tables**: any [external table](/docs/2.0.0-rc6/data-sources/external-tables) or +[view](/docs/2.0.0-rc6/server/view): ```sql SELECT * FROM ocean_profiles LIMIT 100 ``` -**Files on demand**: use a [table function](/docs/2.0.0-rc5/sql/table-functions). You +**Files on demand**: use a [table function](/docs/2.0.0-rc6/sql/table-functions). You register no table: ```sql @@ -33,11 +33,11 @@ SELECT * FROM read_netcdf(['argo/**/*.nc']) LIMIT 100 ## Reference -- Rules: [Identifiers and case](/docs/2.0.0-rc5/sql/identifiers) -- Query: [SELECT](/docs/2.0.0-rc5/sql/select) · [WHERE](/docs/2.0.0-rc5/sql/where) · [GROUP BY](/docs/2.0.0-rc5/sql/group-by) · [JOIN](/docs/2.0.0-rc5/sql/join) · [UNION BY NAME](/docs/2.0.0-rc5/sql/union-by-name) -- Tables and views: [CREATE TABLE](/docs/2.0.0-rc5/sql/managed-tables) · [CREATE EXTERNAL TABLE](/docs/2.0.0-rc5/sql/create-external-table) · [CREATE VIEW](/docs/2.0.0-rc5/sql/create-view) · [CREATE MATERIALIZED VIEW](/docs/2.0.0-rc5/sql/create-materialized-view) · [Remote tables and `ATTACH`](/docs/2.0.0-rc5/sql/remote-tables) -- Functions: [Table functions](/docs/2.0.0-rc5/sql/table-functions) · [Utility table functions](/docs/2.0.0-rc5/sql/table-functions-utility) · [Function reference](/docs/2.0.0-rc5/sql/function-reference) · [Spatial functions](/docs/2.0.0-rc5/sql/spatial-functions) -- Secrets and profiles: [`CREATE SECRET`](/docs/2.0.0-rc5/sql/secrets) · [`SUMMARIZE`](/docs/2.0.0-rc5/sql/summarize) +- Rules: [Identifiers and case](/docs/2.0.0-rc6/sql/identifiers) +- Query: [SELECT](/docs/2.0.0-rc6/sql/select) · [WHERE](/docs/2.0.0-rc6/sql/where) · [GROUP BY](/docs/2.0.0-rc6/sql/group-by) · [JOIN](/docs/2.0.0-rc6/sql/join) · [UNION BY NAME](/docs/2.0.0-rc6/sql/union-by-name) +- Tables and views: [CREATE TABLE](/docs/2.0.0-rc6/sql/managed-tables) · [CREATE EXTERNAL TABLE](/docs/2.0.0-rc6/sql/create-external-table) · [CREATE VIEW](/docs/2.0.0-rc6/sql/create-view) · [CREATE MATERIALIZED VIEW](/docs/2.0.0-rc6/sql/create-materialized-view) · [Remote tables and `ATTACH`](/docs/2.0.0-rc6/sql/remote-tables) +- Functions: [Table functions](/docs/2.0.0-rc6/sql/table-functions) · [Utility table functions](/docs/2.0.0-rc6/sql/table-functions-utility) · [Function reference](/docs/2.0.0-rc6/sql/function-reference) · [Spatial functions](/docs/2.0.0-rc6/sql/spatial-functions) +- Secrets and profiles: [`CREATE SECRET`](/docs/2.0.0-rc6/sql/secrets) · [`SUMMARIZE`](/docs/2.0.0-rc6/sql/summarize) **Friendly SQL.** Beacon also supports these DataFusion extensions: `SELECT * EXCLUDE (col)`, `REPLACE (…)`, `GROUP BY ALL`, `QUALIFY`, `UNION BY NAME`, `FROM` first (`FROM t SELECT …`), diff --git a/docs/docs/2.0.0-rc5/sql/join.md b/docs/docs/2.0.0-rc6/sql/join.md similarity index 100% rename from docs/docs/2.0.0-rc5/sql/join.md rename to docs/docs/2.0.0-rc6/sql/join.md diff --git a/docs/docs/2.0.0-rc5/sql/managed-tables.md b/docs/docs/2.0.0-rc6/sql/managed-tables.md similarity index 98% rename from docs/docs/2.0.0-rc5/sql/managed-tables.md rename to docs/docs/2.0.0-rc6/sql/managed-tables.md index 8fa02404..ea9ddae8 100644 --- a/docs/docs/2.0.0-rc5/sql/managed-tables.md +++ b/docs/docs/2.0.0-rc6/sql/managed-tables.md @@ -7,7 +7,7 @@ SELECT * FROM observations; ``` A **managed table** is a SQL table. Beacon owns and stores its data. An -[external table](/docs/2.0.0-rc5/data-sources/external-tables) only points at existing +[external table](/docs/2.0.0-rc6/data-sources/external-tables) only points at existing files. A managed table starts empty, or Beacon fills it from a query. You add rows with `INSERT`. A managed table supports `UPDATE`, `DELETE`, schema changes with `ALTER TABLE` and secondary indexes. The table definition and the data survive a restart. @@ -17,7 +17,7 @@ A managed table needs write access. `CREATE`, `INSERT`, `UPDATE`, `DELETE`, `ALT ::: tip Names keep their case A table name means exactly what you write. `MyTable` and `mytable` are two different tables. -See [Identifiers and case](/docs/2.0.0-rc5/sql/identifiers). +See [Identifiers and case](/docs/2.0.0-rc6/sql/identifiers). ::: ## Storage engine diff --git a/docs/docs/2.0.0-rc5/sql/remote-tables.md b/docs/docs/2.0.0-rc6/sql/remote-tables.md similarity index 91% rename from docs/docs/2.0.0-rc5/sql/remote-tables.md rename to docs/docs/2.0.0-rc6/sql/remote-tables.md index 4bca81ea..3f0443ea 100644 --- a/docs/docs/2.0.0-rc5/sql/remote-tables.md +++ b/docs/docs/2.0.0-rc6/sql/remote-tables.md @@ -6,7 +6,7 @@ Flight SQL. After you create it, it behaves like any other table in your SQL. Yo possible down to it. Only the reduced result travels over the network. This chapter covers queries over a remote table. The -[Remote Tables (Federation) setup chapter](/docs/2.0.0-rc5/data-sources/remote-tables) +[Remote Tables (Federation) setup chapter](/docs/2.0.0-rc6/data-sources/remote-tables) covers the DDL with `LOCATION`, `OPTIONS` and TLS. It also covers the schema handling and the full list of limitations. @@ -54,7 +54,7 @@ two *different* remote servers. Beacon fetches the remote rows and joins them on Use standard SQL comparisons in the predicates that must push down. Beacon pushes a filter or projection with a UDF down only if the remote server has the same function. A geospatial `st_*` function is an example. If not, Beacon runs it locally. See the -[limitations in the setup chapter](/docs/2.0.0-rc5/data-sources/remote-tables#limitations). +[limitations in the setup chapter](/docs/2.0.0-rc6/data-sources/remote-tables#limitations). ::: ## Attach a whole remote catalog (`ATTACH`) @@ -80,10 +80,10 @@ DETACH lake; with `'tls' 'true'` or with an `https://` URL. - **Credentials**: give exactly one of three options. Use `'username'` and `'password'` for HTTP Basic. Use `'token'` for a bearer token. Use `'secret'` with the name of a - [`TYPE BEACON` secret](/docs/2.0.0-rc5/sql/secrets). Omit all three for anonymous access. - The remote applies its own [RBAC](/docs/2.0.0-rc5/security/access-control) to your identity. + [`TYPE BEACON` secret](/docs/2.0.0-rc6/sql/secrets). Omit all three for anonymous access. + The remote applies its own [RBAC](/docs/2.0.0-rc6/security/access-control) to your identity. - Beacon contacts the remote at `ATTACH` time. It lists the schemas and tables as a snapshot. It resolves the schema of each table at the first use. `ATTACH` mirrors a whole remote catalog instead of one table. See -[ATTACH](/docs/2.0.0-rc5/data-sources/attach). +[ATTACH](/docs/2.0.0-rc6/data-sources/attach). diff --git a/docs/docs/2.0.0-rc5/sql/secrets.md b/docs/docs/2.0.0-rc6/sql/secrets.md similarity index 87% rename from docs/docs/2.0.0-rc5/sql/secrets.md rename to docs/docs/2.0.0-rc6/sql/secrets.md index f2adb30a..6f9203a6 100644 --- a/docs/docs/2.0.0-rc5/sql/secrets.md +++ b/docs/docs/2.0.0-rc6/sql/secrets.md @@ -4,16 +4,16 @@ description: CREATE SECRET stores the credentials for ATTACH to a remote Beacon # Secrets -`CREATE SECRET` stores a credential under a name. [`ATTACH`](/docs/2.0.0-rc5/data-sources/attach) +`CREATE SECRET` stores a credential under a name. [`ATTACH`](/docs/2.0.0-rc6/data-sources/attach) then uses that name to reach a remote Beacon server. The credential never appears in the statement that uses it, and never appears in a log. ::: info Storage credentials are not secrets A server reads its datasets from **one** store: either a local directory or **one** S3-compatible bucket. That store is chosen at startup, by -[configuration](/docs/2.0.0-rc5/server/configuration), and it takes its credentials from the +[configuration](/docs/2.0.0-rc6/server/configuration), and it takes its credentials from the standard `AWS_*` environment chain. There is nothing to declare in SQL. See -[Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +[Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). A secret covers the one case that the configuration cannot. It holds the credentials for **another Beacon server**. You choose that server for each query, not at startup. @@ -80,6 +80,6 @@ drop each persistent secret. Create each one again under the new key. ## Next -- [ATTACH](/docs/2.0.0-rc5/data-sources/attach): mirror a remote catalog -- [Remote Tables](/docs/2.0.0-rc5/sql/remote-tables): one remote table instead of a whole catalog -- [Access Control](/docs/2.0.0-rc5/security/access-control): who may run `CREATE SECRET` +- [ATTACH](/docs/2.0.0-rc6/data-sources/attach): mirror a remote catalog +- [Remote Tables](/docs/2.0.0-rc6/sql/remote-tables): one remote table instead of a whole catalog +- [Access Control](/docs/2.0.0-rc6/security/access-control): who may run `CREATE SECRET` diff --git a/docs/docs/2.0.0-rc5/sql/select.md b/docs/docs/2.0.0-rc6/sql/select.md similarity index 100% rename from docs/docs/2.0.0-rc5/sql/select.md rename to docs/docs/2.0.0-rc6/sql/select.md diff --git a/docs/docs/2.0.0-rc5/sql/spatial-functions.md b/docs/docs/2.0.0-rc6/sql/spatial-functions.md similarity index 99% rename from docs/docs/2.0.0-rc5/sql/spatial-functions.md rename to docs/docs/2.0.0-rc6/sql/spatial-functions.md index 1d446fd1..5a267d3b 100644 --- a/docs/docs/2.0.0-rc5/sql/spatial-functions.md +++ b/docs/docs/2.0.0-rc6/sql/spatial-functions.md @@ -39,7 +39,7 @@ FROM read_parquet(['obs/*.parquet']) A column without geometry metadata also reads as a geometry. A `VARCHAR` column reads as WKT. A `VARBINARY` column reads as WKB. So a raw CSV column needs no cast. -A [GeoParquet](/docs/2.0.0-rc5/formats/geoparquet) file holds a geometry column, and Beacon decodes +A [GeoParquet](/docs/2.0.0-rc6/formats/geoparquet) file holds a geometry column, and Beacon decodes it to native GeoArrow on read. Every function here reads such a column directly: ```sql @@ -49,7 +49,7 @@ FROM read_geoparquet(['spatial/stations/*.geoparquet']) :::tip A filter over a GeoParquet geometry column also skips row groups. See -[what the scan skips](/docs/2.0.0-rc5/formats/geoparquet). +[what the scan skips](/docs/2.0.0-rc6/formats/geoparquet). ::: ## Accessors diff --git a/docs/docs/2.0.0-rc5/sql/summarize.md b/docs/docs/2.0.0-rc6/sql/summarize.md similarity index 96% rename from docs/docs/2.0.0-rc5/sql/summarize.md rename to docs/docs/2.0.0-rc6/sql/summarize.md index 2119d598..dbcada65 100644 --- a/docs/docs/2.0.0-rc5/sql/summarize.md +++ b/docs/docs/2.0.0-rc6/sql/summarize.md @@ -29,7 +29,7 @@ SUMMARIZE (SELECT * FROM read_parquet('obs/*.parquet')); ::: warning A table function needs a query `SUMMARIZE` takes a **name** or a **query**. A bare -[table function](/docs/2.0.0-rc5/sql/table-functions) is neither, so this does not parse: +[table function](/docs/2.0.0-rc6/sql/table-functions) is neither, so this does not parse: ```sql SUMMARIZE read_netcdf('argo/*.nc'); -- error diff --git a/docs/docs/2.0.0-rc5/sql/table-functions-utility.md b/docs/docs/2.0.0-rc6/sql/table-functions-utility.md similarity index 100% rename from docs/docs/2.0.0-rc5/sql/table-functions-utility.md rename to docs/docs/2.0.0-rc6/sql/table-functions-utility.md diff --git a/docs/docs/2.0.0-rc5/sql/table-functions.md b/docs/docs/2.0.0-rc6/sql/table-functions.md similarity index 95% rename from docs/docs/2.0.0-rc5/sql/table-functions.md rename to docs/docs/2.0.0-rc6/sql/table-functions.md index dd9f21be..939f624d 100644 --- a/docs/docs/2.0.0-rc5/sql/table-functions.md +++ b/docs/docs/2.0.0-rc6/sql/table-functions.md @@ -1,9 +1,9 @@ # Read Files A table function queries files directly in a `FROM` clause. You create no -[external table](/docs/2.0.0-rc5/data-sources/external-tables) first. Use a table function +[external table](/docs/2.0.0-rc6/data-sources/external-tables) first. Use a table function for ad-hoc exploration. Also use it to put the file paths inside a -[view](/docs/2.0.0-rc5/server/view). +[view](/docs/2.0.0-rc6/server/view). The first argument of every function gives the file paths. Use **one path or glob string**, or **a list of strings**. Beacon resolves a glob against its dataset storage root. @@ -115,7 +115,7 @@ read_atlas(glob_paths) read_atlas(glob_paths, dimensions) ``` -Beacon reads the [Atlas](/docs/2.0.0-rc5/formats/atlas) collections that match one or more glob +Beacon reads the [Atlas](/docs/2.0.0-rc6/formats/atlas) collections that match one or more glob patterns. A collection is one file, `data.atlas`, so a path names that file. Give an exact path or a glob such as `**/data.atlas`. @@ -248,7 +248,7 @@ read_delta(location) read_delta(location, version_or_timestamp) ``` -Beacon reads a [Delta Lake](/docs/2.0.0-rc5/formats/delta-lake) table. The +Beacon reads a [Delta Lake](/docs/2.0.0-rc6/formats/delta-lake) table. The `location` argument differs from the other functions. It is **one path to the Delta table directory**. That directory holds `_delta_log/`. It is not a glob and not a list. Beacon reads the schema from the transaction log. @@ -269,7 +269,7 @@ SELECT count(*) FROM read_delta('delta/ocean_profiles', 12) SELECT * FROM read_delta('delta/ocean_profiles', '2026-01-01T00:00:00Z') ``` -Use [`CREATE EXTERNAL TABLE … STORED AS DELTA`](/docs/2.0.0-rc5/formats/delta-lake) +Use [`CREATE EXTERNAL TABLE … STORED AS DELTA`](/docs/2.0.0-rc6/formats/delta-lake) to register a Delta table permanently. That form also supports `INSERT INTO`. ## `read_iceberg` @@ -279,7 +279,7 @@ read_iceberg(location) read_iceberg(location, snapshot_id) ``` -Beacon reads an [Apache Iceberg](/docs/2.0.0-rc5/formats/iceberg) table. The `location` +Beacon reads an [Apache Iceberg](/docs/2.0.0-rc6/formats/iceberg) table. The `location` argument differs from the other functions. It gives **one path to the Iceberg table directory**. That directory holds `metadata/`. Give no glob and no list. Beacon reads the columns from the table metadata. Beacon also finds the current metadata file. @@ -295,7 +295,7 @@ SELECT * FROM read_iceberg('iceberg/ocean_profiles') LIMIT 100 SELECT count(*) FROM read_iceberg('iceberg/ocean_profiles', 3821550127947089060) ``` -Use [`CREATE EXTERNAL TABLE … STORED AS ICEBERG`](/docs/2.0.0-rc5/formats/iceberg) +Use [`CREATE EXTERNAL TABLE … STORED AS ICEBERG`](/docs/2.0.0-rc6/formats/iceberg) to keep an Iceberg table in the catalog. Beacon reads an Iceberg table. Beacon writes none. ## `read_icechunk` @@ -307,7 +307,7 @@ read_icechunk(location, branch, snapshot) read_icechunk(location, branch, snapshot, dimensions) ``` -Beacon reads an [Icechunk](/docs/2.0.0-rc5/formats/icechunk) repository as a Zarr store. A +Beacon reads an [Icechunk](/docs/2.0.0-rc6/formats/icechunk) repository as a Zarr store. A repository is a Zarr store with commits, branches, tags and snapshots. Give one `location`. The `location` is the path to the directory of the repository. The `location` @@ -334,5 +334,5 @@ SELECT count(*) FROM read_icechunk('sst/repo', 'dev') SELECT avg(sst) FROM read_icechunk('sst/repo', NULL, 'NNNGCAX7Z99K7XTTYK8G') ``` -[`CREATE EXTERNAL TABLE … STORED AS ICECHUNK`](/docs/2.0.0-rc5/formats/icechunk) adds a repository +[`CREATE EXTERNAL TABLE … STORED AS ICECHUNK`](/docs/2.0.0-rc6/formats/icechunk) adds a repository to the catalog. Beacon reads Icechunk. Beacon does not write Icechunk. diff --git a/docs/docs/2.0.0-rc5/sql/union-by-name.md b/docs/docs/2.0.0-rc6/sql/union-by-name.md similarity index 100% rename from docs/docs/2.0.0-rc5/sql/union-by-name.md rename to docs/docs/2.0.0-rc6/sql/union-by-name.md diff --git a/docs/docs/2.0.0-rc5/sql/where.md b/docs/docs/2.0.0-rc6/sql/where.md similarity index 100% rename from docs/docs/2.0.0-rc5/sql/where.md rename to docs/docs/2.0.0-rc6/sql/where.md diff --git a/docs/docs/2.0.0-rc5/troubleshooting.md b/docs/docs/2.0.0-rc6/troubleshooting.md similarity index 88% rename from docs/docs/2.0.0-rc5/troubleshooting.md rename to docs/docs/2.0.0-rc6/troubleshooting.md index 2ec4a6af..c3149edb 100644 --- a/docs/docs/2.0.0-rc5/troubleshooting.md +++ b/docs/docs/2.0.0-rc6/troubleshooting.md @@ -9,7 +9,7 @@ Each entry below names a symptom, its usual cause and the fix. ::: tip Find the log first Many entries below tell you to read the log. Beacon writes to stdout and to a dated file in `/beacon/logs/`. Use `docker logs -f beacon`, or mount the directory to keep the files. See -[Log files](/docs/2.0.0-rc5/server/configuration#log-files). +[Log files](/docs/2.0.0-rc6/server/configuration#log-files). ::: ## Reading files @@ -34,7 +34,7 @@ Ask for the other grid by name: SELECT * FROM read_netcdf('sst/*.nc', ['lat', 'nv']); ``` -See [Excluded variables](/docs/2.0.0-rc5/arrays-to-tables#excluded-variables). +See [Excluded variables](/docs/2.0.0-rc6/arrays-to-tables#excluded-variables). ### The query returns more rows than the file has values @@ -42,7 +42,7 @@ The row count is the product of the grid dimension sizes. It is not the count of `time=100, lat=180, lon=360` file gives 6,480,000 rows. That number is correct. Is the number larger still? Then your `dimensions` list probably mixes unrelated dimensions. Make -the list smaller. See [Arrays to tables](/docs/2.0.0-rc5/arrays-to-tables). +the list smaller. See [Arrays to tables](/docs/2.0.0-rc6/arrays-to-tables). ### `SELECT *` returns hundreds of columns @@ -72,7 +72,7 @@ Beacon supports the Gregorian and Julian calendars only. A file with `noleap`, ` `360_day`, `all_leap` or `366_day` is rejected rather than decoded to a wrong instant. Convert the time axis before you query. See -[Supported calendars](/docs/2.0.0-rc5/cf-decoding#supported-calendars). +[Supported calendars](/docs/2.0.0-rc6/cf-decoding#supported-calendars). ### Values look like small integers, not physical units @@ -83,7 +83,7 @@ both when they are present. Check them: SELECT DISTINCT "sst.scale_factor", "sst.add_offset" FROM read_netcdf('sst/*.nc'); ``` -See [CF decoding](/docs/2.0.0-rc5/cf-decoding). +See [CF decoding](/docs/2.0.0-rc6/cf-decoding). ### Missing data shows as `-999`, not `NULL` @@ -103,7 +103,7 @@ name: SELECT * FROM read_geoparquet_schema('spatial/*.geoparquet'); ``` -See [`read__schema`](/docs/2.0.0-rc5/sql/table-functions-utility#read-format-schema). +See [`read__schema`](/docs/2.0.0-rc6/sql/table-functions-utility#read-format-schema). ### A NetCDF file on S3 fails to open, but Parquet works @@ -111,7 +111,7 @@ NetCDF on object storage supports **anonymous access only**. The native reader o URL and does not go through the credential chain. Either make the bucket readable anonymously, or convert the files, or serve them from a server whose -datasets store is that bucket. See [NetCDF](/docs/2.0.0-rc5/formats/netcdf). +datasets store is that bucket. See [NetCDF](/docs/2.0.0-rc6/formats/netcdf). ### Zero rows from a glob that matches files @@ -155,19 +155,19 @@ listing order does not change the result. The numpy strategy also joins a boolea `Float16` with the other floats, and a date with a timestamp. The default strategy refuses each of those pairs. A number beside a timestamp stays a conflict under both strategies, and `BEACON_TYPE_WIDENING_ON_CONFLICT` settles it. See -[Configuration](/docs/2.0.0-rc5/server/configuration#query-engine). +[Configuration](/docs/2.0.0-rc6/server/configuration#query-engine). ### New files do not appear Beacon does not watch storage. It reads no filesystem events and no S3 events. Register a -[crawler](/docs/2.0.0-rc5/server/crawlers) with a schedule. +[crawler](/docs/2.0.0-rc6/server/crawlers) with a schedule. ## Performance ### The first query is slow. Later queries are fast Beacon lists the files on first use. It also infers the schema. It caches both results. For a large -tree, use a [crawler](/docs/2.0.0-rc5/server/crawlers). The crawler does this work in advance. +tree, use a [crawler](/docs/2.0.0-rc6/server/crawlers). The crawler does this work in advance. ### A filter does not reduce the time @@ -179,15 +179,15 @@ EXPLAIN SELECT * FROM read_parquet('obs/*.parquet') WHERE depth < 50; If the predicate does not appear near the scan, it did not push down. Common causes: a function on the filtered column, a type mismatch that forces a cast, or a format that carries no statistics. -See [Speed up slow queries](/docs/2.0.0-rc5/guides/speed-up-queries). +See [Speed up slow queries](/docs/2.0.0-rc6/guides/speed-up-queries). ### The query runs out of memory Do not collect the whole result. Ask for a file format and stream it to disk, or read record batches -over Flight SQL. See [Export query results](/docs/2.0.0-rc5/guides/export-results). +over Flight SQL. See [Export query results](/docs/2.0.0-rc6/guides/export-results). A server-side query can spill to disk. The spill goes to the OS temp area. Put that on fast storage -with free space. See [Performance Tuning](/docs/2.0.0-rc5/server/performance-tuning). +with free space. See [Performance Tuning](/docs/2.0.0-rc6/server/performance-tuning). ## Connecting @@ -201,13 +201,13 @@ BEACON_S3_DATASETS is set but BEACON_S3_BUCKET is missing; the bucket is never inferred from AWS_ENDPOINT ``` -See [Configuration](/docs/2.0.0-rc5/server/configuration). +See [Configuration](/docs/2.0.0-rc6/server/configuration). ### Two servers cannot share a data directory Correct. Beacon holds `beacon.db` under an exclusive lock, so one server opens one data directory. Give each server its own `BEACON_DATA_DIR`. See -[Storage internals](/docs/2.0.0-rc5/internals/storage). +[Storage internals](/docs/2.0.0-rc6/internals/storage). ### `CREATE PERSISTENT SECRET` fails @@ -215,7 +215,7 @@ It needs a master key. Set `BEACON_SECRETS_KEY` to base64 of 32 bytes. Beacon fa key. It never writes a plaintext credential. A plain `CREATE SECRET` needs no key. That secret lives for the session only. See -[Secrets](/docs/2.0.0-rc5/sql/secrets). +[Secrets](/docs/2.0.0-rc6/sql/secrets). ### `ATTACH` fails after a key change @@ -227,7 +227,7 @@ it again under the new key. A remote table connects **anonymously** and stores no credentials. The remote server must allow anonymous Flight SQL access (`BEACON_FLIGHT_SQL_ALLOW_ANONYMOUS=true`). -For an authenticated connection, use [`ATTACH`](/docs/2.0.0-rc5/data-sources/attach) with a secret. +For an authenticated connection, use [`ATTACH`](/docs/2.0.0-rc6/data-sources/attach) with a secret. ### A remote table shows the old columns @@ -240,7 +240,7 @@ to pick up a change. `BEACON_ADMIN_*` protects the admin UI and the write operations. It does not gate reads. Set `BEACON_AUTH_ENFORCE=true` to apply read grants. See -[Access Control](/docs/2.0.0-rc5/security/access-control). +[Access Control](/docs/2.0.0-rc6/security/access-control). ### A user has a grant but still cannot read @@ -254,6 +254,6 @@ By design. There is exactly one super-user, and it comes from `BEACON_ADMIN_USER ## Still stuck -- [FAQ](/docs/2.0.0-rc5/faq) +- [FAQ](/docs/2.0.0-rc6/faq) - [GitHub issues](https://github.com/maris-development/beacon/issues) - [Community Slack](https://beacontechnic-wwa5548.slack.com/join/shared_invite/zt-2dp1vv56r-tj_KFac0sAKNuAgUKPPDRg) diff --git a/docs/index.md b/docs/index.md index c885d4c6..b9b423ea 100644 --- a/docs/index.md +++ b/docs/index.md @@ -35,9 +35,9 @@ Point a server at your archive. Connect a client. Write queries. ## Choose a version
- + - 2.0.0-rc5 + 2.0.0-rc6 Pre-release This is the upcoming 2.0 version release. @@ -64,32 +64,32 @@ Point a server at your archive. Connect a client. Write queries. ## Common tasks
- + Deploy a server Point Beacon at a directory or a bucket. Set the ports, the storage and the limits. - + Connect a client Use Python, TypeScript or the terminal client. DataGrip and DBeaver use Arrow Flight SQL. - + Write SQL Query your files with SELECT, JOIN and UNION BY NAME. Beacon adds 123 spatial functions. - + Call the API One endpoint accepts SQL or a JSON query. It answers in Arrow, Parquet, CSV, NetCDF or ODV. - + Read your formats Beacon reads NetCDF, Zarr, Parquet, GeoParquet, CSV, ODV, HDF5, Arrow, GeoTIFF, Iceberg, Delta and BBF. - + Tune and fix Set every BEACON_* variable. Find the answer for a common error. diff --git a/docs/why-beacon.md b/docs/why-beacon.md index df64ea97..f5825b38 100644 --- a/docs/why-beacon.md +++ b/docs/why-beacon.md @@ -48,23 +48,23 @@ SQL. Beacon sends back only the rows and columns of the answer. give ragged profiles. They give hundreds of thousands of small files. The schema changes from file to file. `xarray.open_mfdataset` fails on that change. A manual loop is slow. Beacon joins the files by column name. It reads the collection as one table. See -[Coming from xarray](/docs/2.0.0-rc5/coming-from-xarray). +[Coming from xarray](/docs/2.0.0-rc6/coming-from-xarray). **Beacon needs no conversion step.** It reads NetCDF, Zarr, Parquet, GeoParquet, CSV, ODV, HDF5, Arrow, GeoTIFF, Delta and BBF in place. You run no import job. You keep no second copy. **Beacon gives one set of names.** Register a glob once as a table or a view. Users then query a name. You can move the storage later. No saved query breaks. See -[Server Setup](/docs/2.0.0-rc5/server/). +[Server Setup](/docs/2.0.0-rc6/server/). **Beacon controls access per table and per path.** A grant is `ON TABLE ` or `ON PATH ''`. A deny always beats a grant. Three public collections and one private collection need a few statements. They do not need a second bucket. See -[Access Control](/docs/2.0.0-rc5/security/access-control). +[Access Control](/docs/2.0.0-rc6/security/access-control). **Beacon queries other servers.** `ATTACH` a second Beacon server. Then join its tables against your own in one statement. Neither server moves its data. Each server keeps its own access rules. See -[ATTACH](/docs/2.0.0-rc5/data-sources/attach). +[ATTACH](/docs/2.0.0-rc6/data-sources/attach). ## Why it fits data in a cloud bucket @@ -87,7 +87,7 @@ The first leg costs nothing. Only the answer leaves the cloud. An answer is much files that hold it. Three effects combine: 1. **Beacon drops whole files.** A filter on time or depth removes a file before Beacon reads its - data. Beacon uses the statistics in an [Atlas](/docs/2.0.0-rc5/formats/atlas) collection, or the + data. Beacon uses the statistics in an [Atlas](/docs/2.0.0-rc6/formats/atlas) collection, or the row group statistics in a Parquet file. 2. **Beacon reads only the columns you name.** `SELECT time, temp` reads two columns from a columnar file. It does not read the other 198. @@ -112,7 +112,7 @@ You save money only when all four conditions are true: A small archive costs more this way. A large archive costs less. - **The format supports pushdown.** Parquet, Atlas, Delta and Zarr drop data early. A CSV file cannot. Beacon reads a CSV file from start to end. See the - [capability matrix](/docs/2.0.0-rc5/formats/#capability-matrix). + [capability matrix](/docs/2.0.0-rc6/formats/#capability-matrix). Range reads add `GET` requests. Request charges are much lower than egress charges. They do not change the result. @@ -144,7 +144,7 @@ the need for the portal. A path in a query is relative to the datasets root. You set that root when the server starts. You can restructure the bucket, rename it, or move to another provider. No user sees a change. See -[Object Storage](/docs/2.0.0-rc5/data-sources/object-storage). +[Object Storage](/docs/2.0.0-rc6/data-sources/object-storage). ## Beacon and DuckDB @@ -242,13 +242,13 @@ Beacon is the wrong tool in four cases: Beacon uses Rust, [Apache Arrow](https://arrow.apache.org/) and [DataFusion](https://datafusion.apache.org/). It parses a query into a plan. It pushes filters and column choices as deep as the format permits. It streams the result back as Arrow, Parquet, NetCDF, -CSV or ODV. See [How it works](/docs/2.0.0-rc5/how-it-works). +CSV or ODV. See [How it works](/docs/2.0.0-rc6/how-it-works). ## Next | | | |---|---| -| **Try it. No setup** | [Query the public server](/docs/2.0.0-rc5/quickstart#query-the-public-node) | -| **Deploy a server** | [Quick Start](/docs/2.0.0-rc5/quickstart#deploy-a-server) · [Getting Started](/docs/2.0.0-rc5/getting-started) | -| **See the full model** | [Introduction](/docs/2.0.0-rc5/introduction) | -| **Move from Python** | [Coming from xarray](/docs/2.0.0-rc5/coming-from-xarray) | +| **Try it. No setup** | [Query the public server](/docs/2.0.0-rc6/quickstart#query-the-public-node) | +| **Deploy a server** | [Quick Start](/docs/2.0.0-rc6/quickstart#deploy-a-server) · [Getting Started](/docs/2.0.0-rc6/getting-started) | +| **See the full model** | [Introduction](/docs/2.0.0-rc6/introduction) | +| **Move from Python** | [Coming from xarray](/docs/2.0.0-rc6/coming-from-xarray) |