From 0d0b20a5cc49e960f546a8d7d8bcce4778d1a2e5 Mon Sep 17 00:00:00 2001 From: Mohit Tilala Date: Wed, 25 Mar 2026 13:27:12 +0530 Subject: [PATCH] Docs: Add Redshift Serverless SYS view support for lineage, usage & profiler docs --- .../workflows/lineage/filter-query-set.mdx | 23 ++++++++++++++++++- .../workflows/usage/filter-query-set.mdx | 23 ++++++++++++++++++- .../profiler/metrics.mdx | 6 ++++- .../workflows/lineage/filter-query-set.mdx | 23 ++++++++++++++++++- .../workflows/usage/filter-query-set.mdx | 23 ++++++++++++++++++- .../profiler/metrics.mdx | 6 ++++- .../workflows/lineage/filter-query-set.mdx | 23 ++++++++++++++++++- .../workflows/usage/filter-query-set.mdx | 23 ++++++++++++++++++- .../profiler/metrics.mdx | 6 ++++- 9 files changed, 147 insertions(+), 9 deletions(-) diff --git a/v1.11.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx b/v1.11.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx index df2cc855..c921f985 100644 --- a/v1.11.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx +++ b/v1.11.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx @@ -188,6 +188,7 @@ For example if you want to add a condition to filter out queries executed by met To fetch the query history log from redshift we execute the following query +### Redshift Provisioned Cluster ``` SELECT * FROM pg_catalog.stl_query @@ -196,6 +197,7 @@ WHERE userid > 1 AND {**your filter condition**} -- Filter out all automated & cursor queries + AND label NOT IN ('maintenance', 'metrics', 'health') AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%' AND aborted = 0 @@ -204,6 +206,25 @@ WHERE userid > 1 LIMIT {result_limit} ``` -You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table. +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table. + +### Redshift Serverless +``` +SELECT * + FROM SYS_QUERY_HISTORY +WHERE user_id > 1 + + AND {**your filter condition**} + + -- Filter out all automated & cursor queries + AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health') + AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + AND LOWER(status) = 'success' + AND start_time >= '{start_time}' + AND start_time < '{end_time}' +``` + +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view. For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`. diff --git a/v1.11.x/connectors/ingestion/workflows/usage/filter-query-set.mdx b/v1.11.x/connectors/ingestion/workflows/usage/filter-query-set.mdx index 68553413..95ddd7d9 100644 --- a/v1.11.x/connectors/ingestion/workflows/usage/filter-query-set.mdx +++ b/v1.11.x/connectors/ingestion/workflows/usage/filter-query-set.mdx @@ -189,6 +189,7 @@ For example if you want to add a condition to filter out queries executed by met To fetch the query history log from redshift we execute the following query +### Redshift Provisioned Cluster ``` SELECT * FROM pg_catalog.stl_query @@ -197,6 +198,7 @@ WHERE userid > 1 AND {**your filter condition**} -- Filter out all automated & cursor queries + AND label NOT IN ('maintenance', 'metrics', 'health') AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%' AND aborted = 0 @@ -205,6 +207,25 @@ WHERE userid > 1 LIMIT {result_limit} ``` -You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table. +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table. + +### Redshift Serverless +``` +SELECT * + FROM SYS_QUERY_HISTORY +WHERE user_id > 1 + + AND {**your filter condition**} + + -- Filter out all automated & cursor queries + AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health') + AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + AND LOWER(status) = 'success' + AND start_time >= '{start_time}' + AND start_time < '{end_time}' +``` + +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view. For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`. diff --git a/v1.11.x/how-to-guides/data-quality-observability/profiler/metrics.mdx b/v1.11.x/how-to-guides/data-quality-observability/profiler/metrics.mdx index d30cc6f3..5e0539c2 100644 --- a/v1.11.x/how-to-guides/data-quality-observability/profiler/metrics.mdx +++ b/v1.11.x/how-to-guides/data-quality-observability/profiler/metrics.mdx @@ -162,7 +162,11 @@ OpenMetadata will look at the past 24-hours to fetch the operations that were pe For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric. ### Redshift -OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables. +OpenMetadata uses below to fetch DML operations as well as the number of rows affected by these operations. +- `SVV_TABLE_INFO`, `STL_INSERT`, `STL_DELETE`, `SVV_TABLE_INFO`, and `STL_QUERYTEXT` in Provisioned Cluster +- `SYS_QUERY_DETAIL` in Serverless instance + +You need to make sure the user running the profiler workflow has access to these views and tables. OpenMetadata will look at the previous day to fetch the operations that were performed against a table. diff --git a/v1.12.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx b/v1.12.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx index df2cc855..c921f985 100644 --- a/v1.12.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx +++ b/v1.12.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx @@ -188,6 +188,7 @@ For example if you want to add a condition to filter out queries executed by met To fetch the query history log from redshift we execute the following query +### Redshift Provisioned Cluster ``` SELECT * FROM pg_catalog.stl_query @@ -196,6 +197,7 @@ WHERE userid > 1 AND {**your filter condition**} -- Filter out all automated & cursor queries + AND label NOT IN ('maintenance', 'metrics', 'health') AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%' AND aborted = 0 @@ -204,6 +206,25 @@ WHERE userid > 1 LIMIT {result_limit} ``` -You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table. +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table. + +### Redshift Serverless +``` +SELECT * + FROM SYS_QUERY_HISTORY +WHERE user_id > 1 + + AND {**your filter condition**} + + -- Filter out all automated & cursor queries + AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health') + AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + AND LOWER(status) = 'success' + AND start_time >= '{start_time}' + AND start_time < '{end_time}' +``` + +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view. For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`. diff --git a/v1.12.x/connectors/ingestion/workflows/usage/filter-query-set.mdx b/v1.12.x/connectors/ingestion/workflows/usage/filter-query-set.mdx index 68553413..95ddd7d9 100644 --- a/v1.12.x/connectors/ingestion/workflows/usage/filter-query-set.mdx +++ b/v1.12.x/connectors/ingestion/workflows/usage/filter-query-set.mdx @@ -189,6 +189,7 @@ For example if you want to add a condition to filter out queries executed by met To fetch the query history log from redshift we execute the following query +### Redshift Provisioned Cluster ``` SELECT * FROM pg_catalog.stl_query @@ -197,6 +198,7 @@ WHERE userid > 1 AND {**your filter condition**} -- Filter out all automated & cursor queries + AND label NOT IN ('maintenance', 'metrics', 'health') AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%' AND aborted = 0 @@ -205,6 +207,25 @@ WHERE userid > 1 LIMIT {result_limit} ``` -You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table. +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table. + +### Redshift Serverless +``` +SELECT * + FROM SYS_QUERY_HISTORY +WHERE user_id > 1 + + AND {**your filter condition**} + + -- Filter out all automated & cursor queries + AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health') + AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + AND LOWER(status) = 'success' + AND start_time >= '{start_time}' + AND start_time < '{end_time}' +``` + +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view. For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`. diff --git a/v1.12.x/how-to-guides/data-quality-observability/profiler/metrics.mdx b/v1.12.x/how-to-guides/data-quality-observability/profiler/metrics.mdx index d30cc6f3..5e0539c2 100644 --- a/v1.12.x/how-to-guides/data-quality-observability/profiler/metrics.mdx +++ b/v1.12.x/how-to-guides/data-quality-observability/profiler/metrics.mdx @@ -162,7 +162,11 @@ OpenMetadata will look at the past 24-hours to fetch the operations that were pe For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric. ### Redshift -OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables. +OpenMetadata uses below to fetch DML operations as well as the number of rows affected by these operations. +- `SVV_TABLE_INFO`, `STL_INSERT`, `STL_DELETE`, `SVV_TABLE_INFO`, and `STL_QUERYTEXT` in Provisioned Cluster +- `SYS_QUERY_DETAIL` in Serverless instance + +You need to make sure the user running the profiler workflow has access to these views and tables. OpenMetadata will look at the previous day to fetch the operations that were performed against a table. diff --git a/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx b/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx index df2cc855..c921f985 100644 --- a/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx +++ b/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx @@ -188,6 +188,7 @@ For example if you want to add a condition to filter out queries executed by met To fetch the query history log from redshift we execute the following query +### Redshift Provisioned Cluster ``` SELECT * FROM pg_catalog.stl_query @@ -196,6 +197,7 @@ WHERE userid > 1 AND {**your filter condition**} -- Filter out all automated & cursor queries + AND label NOT IN ('maintenance', 'metrics', 'health') AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%' AND aborted = 0 @@ -204,6 +206,25 @@ WHERE userid > 1 LIMIT {result_limit} ``` -You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table. +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table. + +### Redshift Serverless +``` +SELECT * + FROM SYS_QUERY_HISTORY +WHERE user_id > 1 + + AND {**your filter condition**} + + -- Filter out all automated & cursor queries + AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health') + AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + AND LOWER(status) = 'success' + AND start_time >= '{start_time}' + AND start_time < '{end_time}' +``` + +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view. For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`. diff --git a/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx b/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx index 68553413..95ddd7d9 100644 --- a/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx +++ b/v1.13.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx @@ -189,6 +189,7 @@ For example if you want to add a condition to filter out queries executed by met To fetch the query history log from redshift we execute the following query +### Redshift Provisioned Cluster ``` SELECT * FROM pg_catalog.stl_query @@ -197,6 +198,7 @@ WHERE userid > 1 AND {**your filter condition**} -- Filter out all automated & cursor queries + AND label NOT IN ('maintenance', 'metrics', 'health') AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%' AND aborted = 0 @@ -205,6 +207,25 @@ WHERE userid > 1 LIMIT {result_limit} ``` -You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table. +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table. + +### Redshift Serverless +``` +SELECT * + FROM SYS_QUERY_HISTORY +WHERE user_id > 1 + + AND {**your filter condition**} + + -- Filter out all automated & cursor queries + AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health') + AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%' + AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%' + AND LOWER(status) = 'success' + AND start_time >= '{start_time}' + AND start_time < '{end_time}' +``` + +You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view. For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`. diff --git a/v1.13.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/metrics.mdx b/v1.13.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/metrics.mdx index d30cc6f3..5e0539c2 100644 --- a/v1.13.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/metrics.mdx +++ b/v1.13.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/metrics.mdx @@ -162,7 +162,11 @@ OpenMetadata will look at the past 24-hours to fetch the operations that were pe For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric. ### Redshift -OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables. +OpenMetadata uses below to fetch DML operations as well as the number of rows affected by these operations. +- `SVV_TABLE_INFO`, `STL_INSERT`, `STL_DELETE`, `SVV_TABLE_INFO`, and `STL_QUERYTEXT` in Provisioned Cluster +- `SYS_QUERY_DETAIL` in Serverless instance + +You need to make sure the user running the profiler workflow has access to these views and tables. OpenMetadata will look at the previous day to fetch the operations that were performed against a table.