Rebuild the Atlas reader on the single-file format

CHANGELOG.md

@@ -12,6 +12,36 @@ tag. Releases before 2.0.0 are recorded in the
 
 ### Added
 
+- **Atlas is readable again, on the single-file format.** Atlas replaced the directory of
+  per-array files with one write-once container, `data.atlas`: one segment per variable, holding
+  that array for every dataset, and a footer that names them all. Beacon's reader was written
+  against the old layout and had been excluded from the build, so `STORED AS ATLAS` and
+  `read_atlas` failed. It is rebuilt on the new format — container version 8, which Atlas 0.17
+  writes — and registered again. A `LOCATION` now names the container rather than a marker beside
+  it: `LOCATION 'obs/data.atlas'`, or a glob such as `'obs/**/data.atlas'`. A collection written
+  before 0.17 is not read at all — rewrite it with `atlas create`. Three behaviour changes come
+  with the rebuild. A dataset-level attribute is a column under a leading dot, `".platform"`,
+  matching NetCDF and Zarr instead of the bare key. A scan reads through the shared nd pipeline,
+  so one dataset is one unit of work: every partition of a query helps drain every collection,
+  and a dataset is read once. And a column two datasets type in two families now refuses the
+  merge by name rather than silently becoming text; `BEACON_TYPE_WIDENING_ON_CONFLICT=keep_first`
+  settles it the other way. Atlas collections are also crawlable now, because a collection is
+  one file whose extension is its format.
+
+- **A predicate over an Atlas collection skips whole datasets.** Atlas records the minimum, the
+  maximum and the null count of every array, so a collection can be judged before it is read.
+  The first scan of a collection pivots those statistics into one index — one row per dataset,
+  one typed Arrow column per column the predicate names — and evaluates the predicate over all of
+  them in a single vectorised pass. Because a variable lives in one segment, gathering a column's
+  statistics is one request whatever the dataset count, so a collection of a million datasets
+  costs one request per column and one pass rather than a million decisions; a dataset ruled out
+  is never opened. A dataset-level attribute is exact, so `WHERE ".platform" = 'p3'` prunes on it
+  too, and at the same cost. Pruning only
+  ever removes datasets that hold no matching row: every path falls back to reading everything,
+  and the filter above the scan still decides each row. `EXPLAIN ANALYZE` reports it as
+  `atlas_datasets_pruned` and `atlas_datasets_scanned`, with the time spent as `atlas_open_time`
+  and `atlas_prune_time`.
+
 - **The server root is a home page instead of a jump to Swagger.** `http://localhost:5001/` sent
   every visitor straight to the Swagger UI, which hid the admin panel, the API reference and the
   documentation from anyone who did not know their paths. The root now answers with a small page

Cargo.toml

@@ -1,7 +1,6 @@
 [workspace]
 resolver = "3"
-exclude = ["beacon-db/beacon-file-formats/beacon-arrow-atlas"]
-members = ["beacon-server/beacon-server", "beacon-server/beacon-server-config", "beacon-server/beacon-mcp", "beacon-db/beacon-auth", "beacon-db/beacon-common", "beacon-db/beacon-core", "beacon-db/beacon-datafusion-ext", "beacon-db/beacon-db-py", "beacon-db/beacon-file-formats/beacon-arrow-bbf", "beacon-db/beacon-file-formats/beacon-arrow-csv", "beacon-db/beacon-file-formats/beacon-arrow-geoparquet", "beacon-db/beacon-file-formats/beacon-arrow-hdf5", "beacon-db/beacon-file-formats/beacon-arrow-ipc", "beacon-db/beacon-file-formats/beacon-arrow-netcdf", "beacon-db/beacon-file-formats/beacon-arrow-odv", "beacon-db/beacon-file-formats/beacon-arrow-parquet", "beacon-db/beacon-file-formats/beacon-arrow-tiff", "beacon-db/beacon-file-formats/beacon-arrow-zarr", "beacon-db/beacon-file-formats/beacon-binary-format", "beacon-db/beacon-file-formats/beacon-binary-format-toolbox", "beacon-db/beacon-file-formats/beacon-delta", "beacon-db/beacon-file-formats/beacon-iceberg", "beacon-db/beacon-file-formats/beacon-icechunk", "beacon-db/beacon-file-formats/beacon-lance", "beacon-db/beacon-file-formats/beacon-nd-array", "beacon-db/beacon-file-formats/beacon-nd-arrow", "beacon-db/beacon-file-stats", "beacon-db/beacon-functions", "beacon-db/beacon-redb-store", "beacon-db/beacon-sql-databases"]
+members = ["beacon-server/beacon-server", "beacon-server/beacon-server-config", "beacon-server/beacon-mcp", "beacon-db/beacon-auth", "beacon-db/beacon-common", "beacon-db/beacon-core", "beacon-db/beacon-datafusion-ext", "beacon-db/beacon-db-py", "beacon-db/beacon-file-formats/beacon-arrow-atlas", "beacon-db/beacon-file-formats/beacon-arrow-bbf", "beacon-db/beacon-file-formats/beacon-arrow-csv", "beacon-db/beacon-file-formats/beacon-arrow-geoparquet", "beacon-db/beacon-file-formats/beacon-arrow-hdf5", "beacon-db/beacon-file-formats/beacon-arrow-ipc", "beacon-db/beacon-file-formats/beacon-arrow-netcdf", "beacon-db/beacon-file-formats/beacon-arrow-odv", "beacon-db/beacon-file-formats/beacon-arrow-parquet", "beacon-db/beacon-file-formats/beacon-arrow-tiff", "beacon-db/beacon-file-formats/beacon-arrow-zarr", "beacon-db/beacon-file-formats/beacon-binary-format", "beacon-db/beacon-file-formats/beacon-binary-format-toolbox", "beacon-db/beacon-file-formats/beacon-delta", "beacon-db/beacon-file-formats/beacon-iceberg", "beacon-db/beacon-file-formats/beacon-icechunk", "beacon-db/beacon-file-formats/beacon-lance", "beacon-db/beacon-file-formats/beacon-nd-array", "beacon-db/beacon-file-formats/beacon-nd-arrow", "beacon-db/beacon-file-stats", "beacon-db/beacon-functions", "beacon-db/beacon-redb-store", "beacon-db/beacon-sql-databases"]
 
 [workspace.package]
 
@@ -82,6 +81,10 @@ datafusion-spatial = { git = "https://github.com/robinskil/datafusion-spatial.gi
 datafusion-spatial-kernels = { git = "https://github.com/robinskil/datafusion-spatial.git", rev = "92245ff8e2eaad939425993cff7fa9180d471ec1" }
 object_store = { version = "0.13.1", features = ["aws", "gcp", "azure", "http"] }
 
+# The atlas container embeds array-format files verbatim, so a change to the
+# crate's bytes is a change to the format Beacon reads. Pin it exactly.
+atlas-rust = "=0.17.2"
+
 oxcdf = { git = "https://github.com/robinskil/oxcdf.git", version = "0.4.0", features = ["async", "object-store", "ndarray"] }
 
 oxcdf-hdf5 = { git = "https://github.com/robinskil/oxcdf.git", version = "0.4.0", features = ["async", "object-store"] }

beacon-clients/beacon-web/src/components/external-table-dialog.tsx

@@ -55,7 +55,7 @@ const TYPE_HINTS: Record<string, string> = {
   NC: "Datasets-store path or glob to netCDF files, e.g. argo/**/*.nc.",
   HDF5: "Datasets-store path or glob, e.g. data/**/*.h5. NetCDF-4 files are HDF5.",
   ZARR: "Datasets-store path to a Zarr v3 store (the zarr.json marker).",
-  ATLAS: "Datasets-store path to an Atlas store (the atlas.json marker).",
+  ATLAS: "Datasets-store path to an Atlas collection (the data.atlas file).",
   TIFF: "Datasets-store path or glob to GeoTIFF/COG files.",
   BBF: "Datasets-store path or glob to Beacon Binary Format files.",
   ODV: "Datasets-store path or glob to ODV files.",

beacon-clients/beacon-web/src/pages/crawlers.tsx

@@ -34,6 +34,7 @@ const FORMATS = [
   { value: "nc", label: "NetCDF" },
   { value: "csv", label: "CSV" },
   { value: "zarr", label: "Zarr" },
+  { value: "atlas", label: "Atlas" },
   { value: "arrow", label: "Arrow" },
   { value: "odv", label: "ODV" },
   { value: "tiff", label: "GeoTIFF" },

beacon-db/beacon-core/Cargo.toml

@@ -72,6 +72,7 @@ beacon-arrow-parquet = { path = "../beacon-file-formats/beacon-arrow-parquet" }
 beacon-arrow-geoparquet = { path = "../beacon-file-formats/beacon-arrow-geoparquet" }
 beacon-arrow-bbf = { path = "../beacon-file-formats/beacon-arrow-bbf" }
 beacon-arrow-zarr = { path = "../beacon-file-formats/beacon-arrow-zarr" }
+beacon-arrow-atlas = { path = "../beacon-file-formats/beacon-arrow-atlas" }
 beacon-datafusion-ext = { path = "../beacon-datafusion-ext" }
 beacon-lance = { path = "../beacon-file-formats/beacon-lance" }
 beacon-delta = { path = "../beacon-file-formats/beacon-delta" }
@@ -82,6 +83,9 @@ beacon-redb-store = { path = "../beacon-redb-store" }
 beacon-file-stats = { path = "../beacon-file-stats", features = ["datafusion"] }
 
 [dev-dependencies]
+# Writes the Atlas fixtures the read tests read; the atlas writer takes ndarray
+# views, and Beacon itself never writes a collection.
+ndarray = { workspace = true }
 # The GeoJSON filter of the JSON query renders `ST_Within`; its test registers the set.
 datafusion-spatial = { workspace = true }
 deltalake = { workspace = true }

beacon-db/beacon-core/src/crawler/discovery.rs

@@ -310,15 +310,33 @@ mod tests {
 
     #[test]
     fn skips_marker_and_overlapping_formats() {
-        // zarr marker (.json != zarr), atlas marker, geoparquet (.parquet != geoparquet)
+        // zarr marker (.json != zarr), geoparquet (.parquet != geoparquet)
         let datasets = vec![
             ds("d/foo.zarr/zarr.json", "zarr"),
-            ds("d/atlas.json", "atlas"),
             ds("d/g.parquet", "geoparquet"),
         ];
         let (cands, skipped) = group_into_tables(&datasets, &def());
         assert!(cands.is_empty());
-        assert_eq!(skipped.len(), 3);
+        assert_eq!(skipped.len(), 2);
+    }
+
+    /// An Atlas collection is one file, `data.atlas`, so its extension is its
+    /// format and the crawler can build a table over it. That is the difference
+    /// from Zarr, whose store is a directory behind a `zarr.json`.
+    ///
+    /// Each collection lives in its own directory, and tables group by
+    /// directory, so each becomes its own table. Several collections in one
+    /// table is what an external table over a glob is for.
+    #[test]
+    fn an_atlas_collection_is_crawlable() {
+        let datasets = vec![
+            ds("d/january/data.atlas", "atlas"),
+            ds("d/february/data.atlas", "atlas"),
+        ];
+        let (cands, skipped) = group_into_tables(&datasets, &def());
+        assert!(skipped.is_empty(), "{skipped:?}");
+        assert_eq!(cands.len(), 2, "one per directory");
+        assert!(cands.iter().all(|table| table.format == "atlas"));
     }
 
     #[test]

beacon-db/beacon-core/src/runtime_builder.rs

@@ -1,11 +1,11 @@
 use std::{
-    collections::HashMap,
     path::PathBuf,
     sync::{Arc, OnceLock},
 };
 
 use crate::crawler::{new_crawler_manager_handle, CrawlerConfig, CrawlerManager};
 use crate::schema_persistence::{init_tables, PersistentSchemaProvider};
+use beacon_arrow_atlas::{AtlasFormatFactory, AtlasOptions};
 use beacon_arrow_bbf::datafusion::BBFFormatFactory;
 use beacon_arrow_csv::datafusion::CsvFormatFactory;
 use beacon_arrow_geoparquet::datafusion::GeoParquetFormatFactory;
@@ -43,7 +43,7 @@ use datafusion::{
         runtime_env::{RuntimeEnv, RuntimeEnvBuilder},
         SessionStateBuilder,
     },
-    optimizer::OptimizerRule,
+    optimizer::{optimize_projections::OptimizeProjections, OptimizerRule},
     prelude::{SessionConfig, SessionContext},
 };
 use object_store::ObjectStore;
@@ -797,6 +797,7 @@ fn register_file_formats(
         Arc::new(ArrowFormatFactory),
         Arc::new(TiffFormatFactory::new(Default::default())),
         Arc::new(ZarrFormatFactory::new(builder.zarr.clone())),
+        Arc::new(AtlasFormatFactory::new(AtlasOptions::default())),
         Arc::new(BBFFormatFactory::new(Default::default())),
         Arc::new(GeoParquetFormatFactory::default()),
         Arc::new(NetCDFFormatFactory::new(
@@ -858,7 +859,11 @@ fn build_session_state(
     runtime_env: Arc<RuntimeEnv>,
     session_cell: SessionCell,
 ) -> anyhow::Result<datafusion::execution::context::SessionState> {
-    let mut optimizer_rules: Vec<Arc<dyn OptimizerRule + Send + Sync>> = vec![];
+    // Narrow every scan to the columns the query reads before any other rule.
+    // CSE copies every input column into an intermediate projection, one linear
+    // schema lookup per column, which is quadratic on a scan of 100k+ columns.
+    let mut optimizer_rules: Vec<Arc<dyn OptimizerRule + Send + Sync>> =
+        vec![Arc::new(OptimizeProjections::new())];
     // This is DataFusion's default logical rule set with `FederationOptimizerRule`
     // inserted, so replacing the defaults with it is intentional: sub-plans rooted
     // at remote tables get pushed down. The matching `FederatedPlanner` lives in