Redshift Serverless: SYS view support for lineage, usage & profiler docs

v1.11.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx

@@ -188,6 +188,7 @@ For example if you want to add a condition to filter out queries executed by met
 
 To fetch the query history log from redshift we execute the following query
 
+### Redshift Provisioned Cluster
 ```
 SELECT *
       FROM pg_catalog.stl_query
@@ -196,6 +197,7 @@ WHERE userid > 1
     AND {**your filter condition**}
 
     -- Filter out all automated & cursor queries
+    AND label NOT IN ('maintenance', 'metrics', 'health')
     AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
     AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%'
     AND aborted = 0
@@ -204,6 +206,25 @@ WHERE userid > 1
     LIMIT {result_limit}
 ```
 
-You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table.
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table.
+
+### Redshift Serverless
+```
+SELECT *
+      FROM SYS_QUERY_HISTORY
+WHERE user_id > 1
+
+    AND {**your filter condition**}
+
+    -- Filter out all automated & cursor queries
+    AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health')
+    AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
+    AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
+    AND LOWER(status) = 'success'
+    AND start_time >= '{start_time}'
+    AND start_time < '{end_time}'
+```
+
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view.
 
 For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`.

v1.11.x/connectors/ingestion/workflows/usage/filter-query-set.mdx

@@ -189,6 +189,7 @@ For example if you want to add a condition to filter out queries executed by met
 
 To fetch the query history log from redshift we execute the following query
 
+### Redshift Provisioned Cluster
 ```
 SELECT *
       FROM pg_catalog.stl_query
@@ -197,6 +198,7 @@ WHERE userid > 1
     AND {**your filter condition**}
 
     -- Filter out all automated & cursor queries
+    AND label NOT IN ('maintenance', 'metrics', 'health')
     AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
     AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%'
     AND aborted = 0
@@ -205,6 +207,25 @@ WHERE userid > 1
     LIMIT {result_limit}
 ```
 
-You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table.
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table.
+
+### Redshift Serverless
+```
+SELECT *
+      FROM SYS_QUERY_HISTORY
+WHERE user_id > 1
+
+    AND {**your filter condition**}
+
+    -- Filter out all automated & cursor queries
+    AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health')
+    AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
+    AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
+    AND LOWER(status) = 'success'
+    AND start_time >= '{start_time}'
+    AND start_time < '{end_time}'
+```
+
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view.
 
 For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`.

v1.11.x/how-to-guides/data-quality-observability/profiler/metrics.mdx

@@ -162,7 +162,11 @@ OpenMetadata will look at the past 24-hours to fetch the operations that were pe
 For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric.
 
 ### Redshift
-OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables.
+OpenMetadata uses below to fetch DML operations as well as the number of rows affected by these operations.
+- `SVV_TABLE_INFO`, `STL_INSERT`, `STL_DELETE`, `SVV_TABLE_INFO`, and `STL_QUERYTEXT` in Provisioned Cluster
+- `SYS_QUERY_DETAIL` in Serverless instance
+
+You need to make sure the user running the profiler workflow has access to these views and tables.
 
 OpenMetadata will look at the previous day to fetch the operations that were performed against a table.
 

v1.12.x/connectors/ingestion/workflows/lineage/filter-query-set.mdx

@@ -188,6 +188,7 @@ For example if you want to add a condition to filter out queries executed by met
 
 To fetch the query history log from redshift we execute the following query
 
+### Redshift Provisioned Cluster
 ```
 SELECT *
       FROM pg_catalog.stl_query
@@ -196,6 +197,7 @@ WHERE userid > 1
     AND {**your filter condition**}
 
     -- Filter out all automated & cursor queries
+    AND label NOT IN ('maintenance', 'metrics', 'health')
     AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
     AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%'
     AND aborted = 0
@@ -204,6 +206,25 @@ WHERE userid > 1
     LIMIT {result_limit}
 ```
 
-You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table.
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table.
+
+### Redshift Serverless
+```
+SELECT *
+      FROM SYS_QUERY_HISTORY
+WHERE user_id > 1
+
+    AND {**your filter condition**}
+
+    -- Filter out all automated & cursor queries
+    AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health')
+    AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
+    AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
+    AND LOWER(status) = 'success'
+    AND start_time >= '{start_time}'
+    AND start_time < '{end_time}'
+```
+
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view.
 
 For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`.

v1.12.x/connectors/ingestion/workflows/usage/filter-query-set.mdx

@@ -189,6 +189,7 @@ For example if you want to add a condition to filter out queries executed by met
 
 To fetch the query history log from redshift we execute the following query
 
+### Redshift Provisioned Cluster
 ```
 SELECT *
       FROM pg_catalog.stl_query
@@ -197,6 +198,7 @@ WHERE userid > 1
     AND {**your filter condition**}
 
     -- Filter out all automated & cursor queries
+    AND label NOT IN ('maintenance', 'metrics', 'health')
     AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
     AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%'
     AND aborted = 0
@@ -205,6 +207,25 @@ WHERE userid > 1
     LIMIT {result_limit}
 ```
 
-You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table.
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table.
+
+### Redshift Serverless
+```
+SELECT *
+      FROM SYS_QUERY_HISTORY
+WHERE user_id > 1
+
+    AND {**your filter condition**}
+
+    -- Filter out all automated & cursor queries
+    AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health')
+    AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
+    AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
+    AND LOWER(status) = 'success'
+    AND start_time >= '{start_time}'
+    AND start_time < '{end_time}'
+```
+
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view.
 
 For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`.

v1.12.x/how-to-guides/data-quality-observability/profiler/metrics.mdx

@@ -162,7 +162,11 @@ OpenMetadata will look at the past 24-hours to fetch the operations that were pe
 For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric.
 
 ### Redshift
-OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables.
+OpenMetadata uses below to fetch DML operations as well as the number of rows affected by these operations.
+- `SVV_TABLE_INFO`, `STL_INSERT`, `STL_DELETE`, `SVV_TABLE_INFO`, and `STL_QUERYTEXT` in Provisioned Cluster
+- `SYS_QUERY_DETAIL` in Serverless instance
+
+You need to make sure the user running the profiler workflow has access to these views and tables.
 
 OpenMetadata will look at the previous day to fetch the operations that were performed against a table.
 

v1.13.x-SNAPSHOT/connectors/ingestion/workflows/lineage/filter-query-set.mdx

@@ -188,6 +188,7 @@ For example if you want to add a condition to filter out queries executed by met
 
 To fetch the query history log from redshift we execute the following query
 
+### Redshift Provisioned Cluster
 ```
 SELECT *
       FROM pg_catalog.stl_query
@@ -196,6 +197,7 @@ WHERE userid > 1
     AND {**your filter condition**}
 
     -- Filter out all automated & cursor queries
+    AND label NOT IN ('maintenance', 'metrics', 'health')
     AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
     AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%'
     AND aborted = 0
@@ -204,6 +206,25 @@ WHERE userid > 1
     LIMIT {result_limit}
 ```
 
-You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table.
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table.
+
+### Redshift Serverless
+```
+SELECT *
+      FROM SYS_QUERY_HISTORY
+WHERE user_id > 1
+
+    AND {**your filter condition**}
+
+    -- Filter out all automated & cursor queries
+    AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health')
+    AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
+    AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
+    AND LOWER(status) = 'success'
+    AND start_time >= '{start_time}'
+    AND start_time < '{end_time}'
+```
+
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view.
 
 For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`.

v1.13.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set.mdx

@@ -189,6 +189,7 @@ For example if you want to add a condition to filter out queries executed by met
 
 To fetch the query history log from redshift we execute the following query
 
+### Redshift Provisioned Cluster
 ```
 SELECT *
       FROM pg_catalog.stl_query
@@ -197,6 +198,7 @@ WHERE userid > 1
     AND {**your filter condition**}
 
     -- Filter out all automated & cursor queries
+    AND label NOT IN ('maintenance', 'metrics', 'health')
     AND querytxt NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
     AND querytxt NOT LIKE '/* {{"app": "dbt", %%}} */%%'
     AND aborted = 0
@@ -205,6 +207,25 @@ WHERE userid > 1
     LIMIT {result_limit}
 ```
 
-You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the stl_query table.
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_STL_QUERY.html) to find out more about the `STL_QUERY` table.
+
+### Redshift Serverless
+```
+SELECT *
+      FROM SYS_QUERY_HISTORY
+WHERE user_id > 1
+
+    AND {**your filter condition**}
+
+    -- Filter out all automated & cursor queries
+    AND LOWER(query_label) NOT IN ('maintenance', 'metrics', 'health')
+    AND query_text NOT LIKE '/* {{"app": "OpenMetadata", %%}} */%%'
+    AND query_text NOT LIKE '/* {{"app": "dbt", %%}} */%%'
+    AND LOWER(status) = 'success'
+    AND start_time >= '{start_time}'
+    AND start_time < '{end_time}'
+```
+
+You can refer to [this redshift documentation](https://docs.aws.amazon.com/redshift/latest/dg/SYS_QUERY_HISTORY.html) to find out more about the `SYS_QUERY_HISTORY` view.
 
 For example if you want to add a condition to filter out queries executed by metabase client, i.e. the queries staring with `-- metabase %` then you can put the condition as `query NOT LIKE '--metabase %'`.

v1.13.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/metrics.mdx

@@ -162,7 +162,11 @@ OpenMetadata will look at the past 24-hours to fetch the operations that were pe
 For snowflake system, the system will parse the DDL query and attempt to match `database`, `schema`, and `table` name to entities in OpenMetadata. If the DDL query does not include all 3 elements we will not be able to ingest this metric.
 
 ### Redshift
-OpenMetadata uses `stl_insert`, `stl_delete`, `svv_table_info`, and `stl_querytext` to fetch DML operations as well as the number of rows affected by these operations. You need to make sure the user running the profiler workflow has access to these views and tables.
+OpenMetadata uses below to fetch DML operations as well as the number of rows affected by these operations.
+- `SVV_TABLE_INFO`, `STL_INSERT`, `STL_DELETE`, `SVV_TABLE_INFO`, and `STL_QUERYTEXT` in Provisioned Cluster
+- `SYS_QUERY_DETAIL` in Serverless instance
+
+You need to make sure the user running the profiler workflow has access to these views and tables.
 
 OpenMetadata will look at the previous day to fetch the operations that were performed against a table.