Add typed table extensions (MCP + presets) for downstream consumers

[robinskil]

What

Adds typed, consumer-facing table extensions — metadata about how to use a table, decoupled from its storage definition and from format options:

• mcp — descriptor for how a downstream MCP server should surface the table as a tool/resource (enabled, tool_name, description, exposed_columns).
• preset — named, predefined filter sets consumers can apply downstream (each a list of {column, op, value} filters).

Both are typed and validated against the live table schema on every write.

Why

Tables previously exposed only structural metadata (Arrow schema) and storage config. There was no place to attach annotations for downstream tools. Stuffing them into options is wrong — that's reserved for format options. Extensions are orthogonal annotations.

Storage

A decoupled tables://<name>/extensions.json sidecar, separate from table.json. This means extensions:

• apply uniformly to every table type (listing/Iceberg/Delta/SQL/remote/view),
• can be edited without rebuilding the provider,
• survive provider re-registration (materialized-view refresh, Iceberg alter),
• are removed automatically on DROP TABLE (the table directory is deleted).

No change to any TableDefinition struct.

Interfaces

SQL DDL (distinct leading keywords; no collision with standard SQL):

SET EXTENSION 'preset' FOR obs TO '{"presets":[{"name":"shallow","filters":[{"column":"depth","op":"<=","value":10}]}]}';
SET EXTENSION 'mcp'    FOR obs TO '{"enabled":true,"tool_name":"query_obs","exposed_columns":["lat","depth"]}';
SHOW EXTENSIONS FOR obs;          -- one JSON row
DROP EXTENSION 'preset' FOR obs;  -- leaves the mcp extension intact

REST:

• Public read: GET /api/table-extensions?table_name=X
• Admin write: PUT / DELETE /api/admin/table-extensions/{name}

All endpoints and schemas are OpenAPI-documented.

Validation

Rejects unknown columns, unsupported operators (allowed: = != < <= > >= between in), malformed between/in values, and duplicate preset names.

Tests

• Extension validation unit tests (unknown column, bad operator, bad value shape, duplicate names, MCP columns, unknown kind).
• Parser unit tests for SET/DROP EXTENSION and SHOW EXTENSIONS (incl. Display round-trip and non-shadowing of standard SQL).
• Persistence round-trip + cleanup test (sidecar load/persist/remove, and removal on DROP TABLE).
• End-to-end runtime test: CREATE TABLE → SET/SHOW/DROP EXTENSION → validation rejection.

cargo build, cargo clippy (no new warnings), and all affected-crate tests pass.

[Copilot]

Pull request overview

This PR introduces a typed, consumer-facing “table extensions” document (MCP descriptor + query presets) stored as a tables://<name>/extensions.json sidecar, and wires it through SQL DDL, runtime APIs, and REST endpoints with schema-aware validation on writes.

Changes:

• Add object-store persistence helpers for an extensions.json sidecar alongside existing table.json persistence.
• Implement typed TableExtensions (MCP + preset) contracts with schema validation, plus SQL SET/DROP EXTENSION and SHOW EXTENSIONS planning/execution.
• Expose extensions via REST (public read + admin write/delete) and add unit/integration coverage.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
beacon-data-lake/src/table_runtime/schema_persistence.rs	Adds persist/load/remove helpers for the extensions.json sidecar + round-trip/cleanup test.
beacon-data-lake/src/lib.rs	Re-exports SchemaPersistenceService for downstream use.
beacon-core/src/statement_plan/query_planner.rs	Adds physical planning for new extension logical nodes.
beacon-core/src/statement_plan/physical.rs	Adds physical exec nodes for SET/DROP EXTENSION and SHOW EXTENSIONS.
beacon-core/src/statement_plan/mod.rs	Adds logical plan constructors for extension DDL statements.
beacon-core/src/statement_plan/logical.rs	Defines logical nodes and DF schema for SHOW EXTENSIONS.
beacon-core/src/runtime.rs	Adds runtime APIs for get/set/delete extensions + end-to-end SQL round-trip test.
beacon-core/src/parser/statement.rs	Adds statement types + Display for extension DDL and SQL literal escaping helper.
beacon-core/src/parser/beacon_parser.rs	Parses SET/DROP EXTENSION and SHOW EXTENSIONS without shadowing standard SQL.
beacon-core/src/lib.rs	Exposes new extensions module.
beacon-core/src/extensions.rs	Implements typed contracts, validation, persistence read/modify/write, and SHOW output batch.
beacon-core/src/api.rs	Re-exports extension types for OpenAPI/REST contracts.
beacon-api/src/axum/client/tables.rs	Adds public GET /api/table-extensions endpoint.
beacon-api/src/axum/client/mod.rs	Registers the new client route.
beacon-api/src/axum/admin/mod.rs	Registers new admin extensions routes.
beacon-api/src/axum/admin/extensions.rs	Adds admin PUT/DELETE endpoints for extensions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.