Change a setting with SET, not only with a restart

CHANGELOG.md

@@ -12,6 +12,25 @@ tag. Releases before 2.0.0 are recorded in the
 
 ### Added
 
+- **A setting changes with `SET`, not only with a restart.** Beacon's configuration was
+  environment-only, so tuning one knob meant a redeploy, and no client could read a value back.
+  The engine and format settings now live in a `beacon.*` namespace the query engine owns:
+  `SET beacon.netcdf.use_rust_reader = true` applies to the next query, `RESET` puts it back to
+  what the server started with, and `SHOW SETTINGS` lists every setting with its value, its
+  startup value and what it does — readable by any authenticated user, which is the piece that
+  made the settings discoverable at all. `ALTER SYSTEM SET` also writes the value into
+  `beacon.db`, so it survives a restart; precedence is persisted > environment > default. A `SET`
+  applies to the whole server and is super-user-only, which is the scope `SET datafusion.*`
+  already had — Beacon runs one engine for every connection, and this does not invent a
+  per-client session. A setting that is read once at startup (the port, the data directory, the
+  credentials, each reader-cache *size*) stays an environment variable and says so by name:
+  ``SET beacon.port`` answers with `BEACON_PORT` and "restart", rather than appearing to work.
+  `beacon.` is also a complete alias for the engine's own `datafusion.` namespace, so
+  `SET beacon.execution.batch_size` and `SET datafusion.execution.batch_size` are one option
+  under two spellings. Format settings keep their three layers, narrowest last: the environment
+  default, the `SET`, then a per-table `OPTIONS (...)`. A `SET` reaches a `read_netcdf(...)`
+  immediately; a registered external table builds its reader once, so `REFRESH` rebuilds it with
+  the current settings. See [Configuration](docs/docs/2.0.0-rc2/server/configuration.md).
 - **Zarr stores supply column ranges for file pruning.** A Zarr store recorded nothing in
   `beacon.system.file_stats`, so every query opened every store. It now reports a range per
   coordinate: an array of rank 0 or rank 1 is read and measured, and an array of rank 2 or higher —
@@ -79,6 +98,12 @@ tag. Releases before 2.0.0 are recorded in the
 
 ### Fixed
 
+- **`BEACON_ATLAS_*` and `BEACON_ENABLE_BBF_SPLIT_STREAMS_SLICE` now reach the reader.** Both were
+  parsed, validated and documented, and then dropped: the runtime built the Atlas and BBF formats
+  from `Default::default()` instead of the configuration, so setting either to a non-default value
+  did nothing. The `BEACON_LANCE_*` family had the opposite problem — it bypassed the configuration
+  entirely and read the process environment at each write, so it was absent from the reference.
+  Both are wired through the runtime now, and all of them are documented and settable at runtime.
 - **File statistics pruned no netCDF or HDF5 file.** The ranges were recorded and then never used.
   Pruning rewrites the file list of a built scan, and it looked for that list on the plan's root
   node. A netCDF or HDF5 scan is not that node: its arrays reach the plan encoded, so the format

beacon-db/beacon-core/src/lib.rs

@@ -13,6 +13,7 @@ pub mod runtime_builder;
 pub mod schema_persistence;
 pub(crate) mod secret_persistence;
 pub mod settings;
+pub(crate) mod settings_persistence;
 mod statement_plan;
 pub(crate) mod system_schema;
 

beacon-db/beacon-core/src/parser/beacon_parser.rs

@@ -10,7 +10,7 @@ use datafusion::sql::{
 use beacon_auth::{Privilege, PrivilegeTarget};
 
 use super::statement::{
-    AttachStatement, AuthStatement, BeaconStatement, CreateCrawlerStatement, CreateIndexStatement,
+    AlterSystemStatement, AttachStatement, AuthStatement, BeaconStatement, CreateCrawlerStatement, CreateIndexStatement,
     CreateMaterializedViewStatement, CreateSecretStatement, DetachStatement, DropCrawlerStatement,
     DropExtensionStatement, DropIndexStatement, DropSecretStatement, RefreshStatement,
     AnalyzeFilesStatement, RunCrawlerStatement, SetExtensionStatement, ShowExtensionsStatement, ShowIndexesStatement,
@@ -111,6 +111,14 @@ impl<'a> BeaconParser<'a> {
             return self.parse_analyze_files();
         }
 
+        if self.is_alter_system() {
+            return self.parse_alter_system();
+        }
+
+        if self.is_show_settings() {
+            return self.parse_show_settings();
+        }
+
         let df_statement = Box::new(self.df_parser.parse_statement()?);
 
         Ok(BeaconStatement::DFStatement(df_statement))
@@ -313,6 +321,70 @@ impl<'a> BeaconParser<'a> {
         }))
     }
 
+    /// Whether the next two tokens are `ALTER SYSTEM`.
+    ///
+    /// Both words are required, so `ALTER TABLE` still reaches DataFusion.
+    fn is_alter_system(&self) -> bool {
+        let t1 = &self.df_parser.parser.peek_nth_token(0).token;
+        let t2 = &self.df_parser.parser.peek_nth_token(1).token;
+        matches!(t1, Token::Word(w) if w.keyword == Keyword::ALTER)
+            && matches!(t2, Token::Word(w) if w.value.to_uppercase() == "SYSTEM")
+    }
+
+    /// Parse: ALTER SYSTEM SET <key> = <value> | ALTER SYSTEM RESET <key>
+    fn parse_alter_system(&mut self) -> Result<BeaconStatement> {
+        self.df_parser.parser.next_token(); // ALTER
+        self.df_parser.parser.next_token(); // SYSTEM
+
+        let is_reset = matches!(
+            &self.df_parser.parser.peek_nth_token(0).token,
+            Token::Word(w) if w.value.to_uppercase() == "RESET"
+        );
+        if is_reset {
+            self.df_parser.parser.next_token(); // RESET
+            let key = self.parse_object_name()?;
+            return Ok(BeaconStatement::AlterSystem(AlterSystemStatement {
+                key,
+                value: None,
+            }));
+        }
+
+        self.expect_keyword(Keyword::SET)?;
+        let key = self.parse_object_name()?;
+        self.expect_token(&Token::Eq)?;
+        // Every value is carried as a string; the config field parses it into its
+        // own type, exactly as it does for a plain `SET`.
+        let value = self.parse_string_value()?;
+        Ok(BeaconStatement::AlterSystem(AlterSystemStatement {
+            key,
+            value: Some(value),
+        }))
+    }
+
+    /// Whether the next two tokens are `SHOW SETTINGS`.
+    fn is_show_settings(&self) -> bool {
+        let t1 = &self.df_parser.parser.peek_nth_token(0).token;
+        let t2 = &self.df_parser.parser.peek_nth_token(1).token;
+        matches!(t1, Token::Word(w) if w.value.to_uppercase() == "SHOW")
+            && matches!(t2, Token::Word(w) if w.value.to_uppercase() == "SETTINGS")
+    }
+
+    /// Parse: SHOW SETTINGS
+    fn parse_show_settings(&mut self) -> Result<BeaconStatement> {
+        self.df_parser.parser.next_token(); // SHOW
+        self.df_parser.parser.next_token(); // SETTINGS
+        Ok(BeaconStatement::ShowSettings)
+    }
+
+    /// Consume the expected token or error.
+    fn expect_token(&mut self, token: &Token) -> Result<()> {
+        self.df_parser
+            .parser
+            .expect_token(token)
+            .map(|_| ())
+            .map_err(|e| DataFusionError::External(Box::new(e)))
+    }
+
     /// Consume the expected keyword or error.
     fn expect_keyword(&mut self, keyword: Keyword) -> Result<()> {
         self.df_parser

beacon-db/beacon-core/src/parser/statement.rs

@@ -27,6 +27,41 @@ pub enum BeaconStatement {
     DropSecret(DropSecretStatement),
     ShowSecrets,
     Summarize(SummarizeStatement),
+    AlterSystem(AlterSystemStatement),
+    ShowSettings,
+}
+
+/// `ALTER SYSTEM SET <key> = <value>` / `ALTER SYSTEM RESET <key>`
+///
+/// The persistent half of `SET`: it applies to the running server *and* is
+/// written into the database file, so a restart replays it. A plain `SET` is
+/// live-only.
+///
+/// Parsed by beacon rather than reaching DataFusion, because sqlparser 0.61 has
+/// no `ALTER SYSTEM` at all.
+#[derive(Debug, Clone)]
+pub struct AlterSystemStatement {
+    /// The setting name as typed. It is resolved (aliases, startup-only
+    /// rejection) when the plan is built, so the parser stays free of any
+    /// knowledge of the settings themselves.
+    pub key: ObjectName,
+    /// The new value, or `None` for `RESET` — which drops the persisted value and
+    /// restores the one the runtime booted with.
+    pub value: Option<String>,
+}
+
+impl Display for AlterSystemStatement {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match &self.value {
+            Some(value) => write!(
+                f,
+                "ALTER SYSTEM SET {} = '{}'",
+                self.key,
+                escape_sql_literal(value)
+            ),
+            None => write!(f, "ALTER SYSTEM RESET {}", self.key),
+        }
+    }
 }
 
 /// SUMMARIZE <table> | SUMMARIZE <query>
@@ -404,6 +439,8 @@ impl Display for BeaconStatement {
             }
             Self::ShowSecrets => write!(f, "SHOW SECRETS"),
             Self::Summarize(s) => write!(f, "SUMMARIZE {}", s.source),
+            Self::AlterSystem(s) => write!(f, "{s}"),
+            Self::ShowSettings => write!(f, "SHOW SETTINGS"),
         }
     }
 }

beacon-db/beacon-core/src/runtime.rs

@@ -544,6 +544,10 @@ impl Runtime {
             BeaconStatement::AnalyzeFiles(statement) => {
                 Ok(crate::statement_plan::analyze_files_plan(statement))
             }
+            BeaconStatement::AlterSystem(statement) => {
+                crate::statement_plan::alter_system_plan(statement)
+            }
+            BeaconStatement::ShowSettings => Ok(crate::statement_plan::show_settings_plan()),
             BeaconStatement::SetExtension(statement) => {
                 Ok(crate::statement_plan::set_extension_plan(statement))
             }