Slim data-lake: replace TableManager/FileManager/DataLake with native SessionContext - #292
Merged
Merged
Conversation
… SessionContext Collapses beacon-data-lake's manager indirection onto DataFusion's native APIs, keeping only what DataFusion doesn't provide: persisting/restoring table definitions across restarts. - TableManager -> thin PersistentSchemaProvider that delegates the in-memory catalog to a native MemorySchemaProvider and only persists/removes tables://<name>/table.json on register/deregister (deregister-then-register preserves overwrite-on-existing for MV refresh / Iceberg replace / alter). - Startup recovery -> free init_tables(ctx, schema) reusing loading/ordering. - FileManager -> free functions (create_listing_url, create_temp_output_file, list_datasets, list_dataset_schema); file_formats vec moves into Runtime. - DataLake struct -> register_object_stores(ctx, &ObjectStores) free function, fed by the runtime-owned ObjectStores (config-owned storage, not globals). - list_table_config reconstructs the TableDefinition from the live provider via definition_from_provider (no parallel definition registry). - Temp output files are created under the configured tmp dir; Output::parse takes the tmp_dir instead of a FileManager. Rebased onto current main (runtime-owned config / ObjectStores / per-format config). Workspace builds; beacon-data-lake (10), beacon-core (28) and beacon-api (9) tests pass.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes the redundant DataLake/TableManager/FileManager indirection in beacon-data-lake and switches callers to use DataFusion’s native SessionContext APIs directly, while preserving Beacon’s restart persistence behavior via a thin persistent schema provider.
Changes:
- Replace
TableManagerwithPersistentSchemaProvider(wrappingMemorySchemaProvider) plusinit_tables(ctx, schema)for startup recovery. - Replace
FileManagerwith free helper functions for dataset discovery, schema inference, listing URL resolution, and temp output file creation. - Update runtime/query call paths in
beacon-coreto useSessionContextcatalog/provider access and the new free helpers; add a restart persistence test.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| beacon-data-lake/src/table_runtime/table_manager.rs | Deleted legacy table registry/manager wrapper. |
| beacon-data-lake/src/table_runtime/provider_factory.rs | Deleted provider factory in favor of building providers from definitions directly. |
| beacon-data-lake/src/table_runtime/persistent_schema_provider.rs | New schema provider that persists table definitions on register/deregister while delegating catalog ops to MemorySchemaProvider. |
| beacon-data-lake/src/table_runtime/schema_persistence.rs | Extracts definition_from_provider helper for reconstructing persisted table definitions from live providers. |
| beacon-data-lake/src/table_runtime/mod.rs | Adds init_tables free function for startup table recovery and re-exports the new provider. |
| beacon-data-lake/src/files/mod.rs | Replaces FileManager with free dataset/file helper functions. |
| beacon-data-lake/src/files/manager.rs | Deleted legacy file manager wrapper. |
| beacon-data-lake/src/lib.rs | Removes DataLake and exposes register_object_stores, free file helpers, and table init/provider utilities. |
| beacon-data-lake/src/table_runtime/ordering.rs | Adds an ordering test covering view dependency ordering via table scan deps. |
| beacon-core/src/runtime.rs | Runtime now owns file formats and schema provider; registers object stores, initializes tables via init_tables, updates dataset/table config listing, adds restart persistence test. |
| beacon-core/src/query/output.rs | Output::parse now takes tmp_dir and uses data-lake free temp-file helper. |
| beacon-core/src/query/from.rs | JSON FROM paths use catalog/provider access and free listing URL helper (no managers). |
| beacon-core/src/query/compiler.rs | Simplifies JSON query compiler signature to only require SessionContext. |
| beacon-core/src/statement_plan/actions.rs | Updates comment to reference PersistentSchemaProvider. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+46
to
+49
| Ok(provider) => { | ||
| let _ = schema.insert_loaded(table_name.clone(), provider); | ||
| tracing::info!("Registered table '{}'", table_name); | ||
| } |
Comment on lines
+78
to
+81
| // Keep current pagination semantics to avoid behavior regressions. | ||
| let start = offset.unwrap_or(0); | ||
| let end = limit.map(|l| start + l).unwrap_or(datasets.len()); | ||
| let datasets = datasets.into_iter().skip(start).take(end - start).collect(); |
Comment on lines
+46
to
+68
| let datasets_object_store = object_stores.datasets.clone(); | ||
| // Register the Beacon-internal store (rooted at the `__beacon__` prefix) | ||
| // used by materialized views to persist and read their data directly, | ||
| // bypassing the datasets store's user-facing hiding and metadata cache. | ||
| session_context.register_object_store( | ||
| &Url::parse(beacon_datafusion_ext::table_ext::INTERNAL_STORE_URL).unwrap(), | ||
| datasets_object_store.internal_store(), | ||
| ); | ||
| // Register datasets object store | ||
| session_context.register_object_store( | ||
| &Url::parse(DATASETS_OBJECT_STORE_URL.as_str()).unwrap(), | ||
| datasets_object_store, | ||
| ); | ||
| // Register tables object store | ||
| session_context.register_object_store( | ||
| &Url::parse(TABLES_OBJECT_STORE_URL.as_str()).unwrap(), | ||
| object_stores.tables.clone(), | ||
| ); | ||
| // Register tmp object store | ||
| session_context.register_object_store( | ||
| &Url::parse(TMP_OBJECT_STORE_URL.as_str()).unwrap(), | ||
| object_stores.tmp.clone(), | ||
| ); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
beacon-data-lakewrapped DataFusion'sSessionContextin manager structs that had grown redundant with it:TableManagerwas thebeacon.publicschema provider but kept its ownHashMaptable registry — duplicating what DataFusion's nativeMemorySchemaProvideralready does. Its only non-redundant jobs were persisting table definitions and rebuilding them at startup.FileManagerwas mostly thin wrappers over constants and free functions, plus dataset discovery.DataLakestruct bundled them, was built inRuntime::new, had its managers extracted, then dropped.This collapses that indirection so table/file access goes through native DataFusion APIs, while keeping the one thing DataFusion doesn't do: persisting/restoring state across restarts.
Changes
TableManager→ thinPersistentSchemaProvider— delegates the in-memory catalog (lookup/register/deregister/names) to a nativeMemorySchemaProvider, keeping only the side effect of persisting/removingtables://<name>/table.jsonon register/deregister.init_tables(ctx, schema)reusing the existingloading/orderingmodules. Deletedtable_manager.rsandprovider_factory.rs.list_table_configreconstructs theTableDefinitionfrom the live provider via the extracteddefinition_from_providerhelper (no parallel definition registry).FileManager→ free functions (create_listing_url,create_temp_output_file,list_datasets,list_dataset_schema); thefile_formatsvec moves intoRuntime. Deletedfiles/manager.rs.DataLakestruct →register_object_stores(ctx, &ObjectStores)free function, fed by the runtime-ownedObjectStores(config-owned storage, not process-globals).Output::parsetakes atmp_dirinstead of aFileManager.Reconciled with current
main(runtime-owned config /ObjectStores/ per-format config).Behavioral note
DataFusion's
MemorySchemaProvider::register_tableerrors on an existing name, whereas the oldTableManagersilently overwrote.PersistentSchemaProvider::register_tablederegisters-then-registers to preserve overwrite-on-existing semantics that materialized-view refresh / Iceberg replace / alter rely on.