Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ filegroup(
[
".github/**/*.yaml",
".github/**/*.yml",
"docs/**/*.md",
],
) + [
"README.md",
Expand All @@ -42,6 +43,7 @@ prettier_bin.prettier_test(
"--check",
"--config",
".prettierrc.json",
"docs",
"editors",
"playground",
"*.md",
Expand All @@ -64,6 +66,7 @@ prettier_bin.prettier_binary(
"--write",
"--config",
".prettierrc.json",
"docs",
"editors",
"playground",
"*.md",
Expand Down
57 changes: 31 additions & 26 deletions docs/experimental/lineage.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,15 @@ Sqruff includes a column lineage analysis feature that traces data flow through
## Overview

Column lineage analysis answers questions like:

- Where does this column come from?
- What source tables contribute to this column?
- How does data flow through CTEs and subqueries?

## Availability

The lineage feature is currently available through:

- **Web Playground**: Try it at [playground.quary.dev](https://playground.quary.dev) by selecting the "Lineage" tool
- **Rust API**: For programmatic use in Rust applications

Expand Down Expand Up @@ -81,7 +83,7 @@ FROM users AS u
INNER JOIN tests AS t ON u.id = t.id
```

### SELECT * Expansion
### SELECT \* Expansion

Star projections are handled and can be traced to their source tables:

Expand Down Expand Up @@ -124,13 +126,13 @@ for &child in &node_data.downstream {

The `Lineage` builder provides several configuration options:

| Method | Description |
|--------|-------------|
| `new(parser, column, sql)` | Create a new lineage analyzer for a column |
| `schema(name, columns)` | Register table schema (column name to type mapping) |
| `source(name, sql)` | Register a source table's SQL definition |
| `disable_trim_selects()` | Keep intermediate SELECT projections in output |
| `build()` | Execute analysis and return `(Tables, Node)` |
| Method | Description |
| -------------------------- | --------------------------------------------------- |
| `new(parser, column, sql)` | Create a new lineage analyzer for a column |
| `schema(name, columns)` | Register table schema (column name to type mapping) |
| `source(name, sql)` | Register a source table's SQL definition |
| `disable_trim_selects()` | Keep intermediate SELECT projections in output |
| `build()` | Execute analysis and return `(Tables, Node)` |

### Registering Source Tables

Expand All @@ -145,6 +147,7 @@ let (tables, node) = Lineage::new(parser, "a", "SELECT a FROM z")
```

This traces column `a` through:

1. The main query (`SELECT a FROM z`)
2. Source `z` (`SELECT a FROM y`)
3. Source `y` (`SELECT * FROM x`)
Expand All @@ -154,28 +157,28 @@ This traces column `a` through:

Each node in the lineage tree contains:

| Field | Description |
|-------|-------------|
| `name` | Column or node identifier |
| `source` | The source expression (full query context) |
| `expression` | The specific expression for this column |
| `downstream` | Child nodes (dependencies) |
| `source_name` | Name of the source table |
| `reference_node_name` | Referenced CTE name (if applicable) |
| Field | Description |
| --------------------- | ------------------------------------------ |
| `name` | Column or node identifier |
| `source` | The source expression (full query context) |
| `expression` | The specific expression for this column |
| `downstream` | Child nodes (dependencies) |
| `source_name` | Name of the source table |
| `reference_node_name` | Referenced CTE name (if applicable) |

## Supported SQL Features

| Feature | Status |
|---------|--------|
| Basic SELECT | Supported |
| CTEs (WITH clause) | Supported |
| Subqueries | Supported |
| UNION / UNION ALL | Supported |
| JOINs | Supported |
| SELECT * | Supported |
| VALUES clause | Supported |
| Feature | Status |
| ------------------------ | --------- |
| Basic SELECT | Supported |
| CTEs (WITH clause) | Supported |
| Subqueries | Supported |
| UNION / UNION ALL | Supported |
| JOINs | Supported |
| SELECT \* | Supported |
| VALUES clause | Supported |
| UNNEST / array functions | Supported |
| Multiple dialects | Supported |
| Multiple dialects | Supported |

## Limitations

Expand All @@ -186,12 +189,14 @@ Each node in the lineage tree contains:
## Example Output

For a query like:

```sql
WITH cte AS (SELECT a FROM source_table)
SELECT a FROM cte
```

The lineage tree shows:

```
name: a
source: with cte as (select source_table.a as a from source_table) select cte.a as a from cte
Expand Down
6 changes: 6 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,35 @@
Sqruff is a SQL linter and formatter written in Rust. It focuses on formatting valid SQL for specific dialects and provides fast linting and fixing.

## Key features

- Linting: Advanced, customizable SQL linting.
- Formatting: Automated, configurable formatting for SQL code consistency.
- Lineage: Column-level data lineage analysis for SQL queries.
- Speed: Fast and efficient with minimal overhead.
- Portability: Designed to integrate into workflows, including CI.

## Sqruff vs SQLFluff

Sqruff started as an exact replacement for SQLFluff but is diverging.

- Accurate dialect definitions: Sqruff targets valid SQL for each dialect and does not try to fix partially correct SQL.
- Configuration: The config format is currently similar to SQLFluff but may diverge over time.

## Playground

Try sqruff in your browser at https://playground.quary.dev.

## Get started

- Install the CLI in [Installation](getting-started/installation.md)
- Learn basic commands: [Lint and fix](usage/lint.md)
- Configure behavior in [Configuration](usage/configuration.md)

## Experimental features

- [dbt project support](experimental/dbt.md)
- [SQL Column Lineage](experimental/lineage.md)

## Credits

The sqruff project is heavily inspired by [sqlfluff](https://sqlfluff.com/) and [ruff](https://github.com/astral-sh/ruff).
5 changes: 2 additions & 3 deletions docs/reference/sample-configurations.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Sample Configurations

The following document outlines sample configurations that may be used to achieve certain formatting/linting outcomes.
The following document outlines sample configurations that may be used to achieve certain formatting/linting outcomes.

## Aligning AS statements

Suppose you want to align as statements in a `select` to return the following outcome.
Suppose you want to align as statements in a `select` to return the following outcome.

```sql
--before
Expand All @@ -30,4 +30,3 @@ spacing_before = align
align_within = select_clause
align_scope = bracketed
```