Structured Dataset Sources

[diannewood]

Problem

Dataset sources are currently defined as raw strings:

source: "ANALYTICS.PUBLIC.ORDERS"

Issues:

• String field must be parsed
• Format varies by platform (dots, slashes, quoted identifiers)
• Ambiguous when database/schema names contain dots
• Can't reuse connection details across datasets
• No obvious place to put platform-specific properties

Proposal

Option A: Inline structured source

datasets:
- name: orders
  source:
    database: ANALYTICS
    schema: PUBLIC
    table: ORDERS

Option B: Separate connection object

connections:
- name: analytics_dwh
  database: ANALYTICS
  schema: PUBLIC
  warehouse: COMPUTE_WH  # project or catalog
  custom_extensions:...

datasets:
- name: orders
  connection: analytics_dwh  # connection object name
  table: ORDERS

Option B provides additional benefits.

• A separate connection object that is reused across datasets
• Easily extensible for platform-specific properties
• Easy to swap out one source for another

Note

Query-based datasets also need to be addressed including how to handle cases where database and schema are defined both as connection properties and within the query.