+
+
+
+ -
+ PROD
+ Production-ready, fully supported connector
+
+ -
+ BETA
+ Beta connector – may have limitations or missing features
+
+
+
## API
diff --git a/v2.0.x-SNAPSHOT/connectors/api/rest.mdx b/v2.0.x-SNAPSHOT/connectors/api/rest.mdx
index 41f6dff7e..47dcc57fb 100644
--- a/v2.0.x-SNAPSHOT/connectors/api/rest.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/api/rest.mdx
@@ -4,9 +4,8 @@ description: Learn how to integrate REST APIs with OpenMetadata connectors. Comp
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentApi from '/snippets/connectors/metadata-agent/api.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-#### Connection Options
-
-**OpenAPI Schema URL**:
-An OpenAPI schema URL typically refers to the URL where the OpenAPI Specification (OAS) document of a web service is hosted. The document defines the service's API, including available endpoints, request/response formats, authentication methods, etc. It is usually in JSON format. for e.g. `https://petstore3.swagger.io/api/v3/openapi.json`
-**Token**: An authentication token to connect to an OpenAPI schema URL. It is only required if the API schema is protected or secured.
-
-
-
+
+To ingest metadata from REST, you need to create a service connection. The service connects REST with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **API Services**, then click the **REST** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other REST services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for REST. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **OpenAPI Schema URL**: An OpenAPI schema URL typically refers to the URL where the OpenAPI Specification (OAS) document of a web service is hosted. The document defines the service's API, including available endpoints, request/response formats, authentication methods, and so on It is usually in JSON format, for example `https://petstore3.swagger.io/api/v3/openapi.json`.
+- **Token**: An authentication token to connect to an OpenAPI schema URL. It is only required if the API schema is protected or secured.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your API service. Filter patterns use regular expressions applied to collection names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all API collections and their endpoints available in the source.
+
+
+**Filter Options**
+
+The API Collection section includes the following filter options:
+
+- **API Collection**: Controls which API collections (groups of endpoints defined in an OpenAPI specification) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/domo-dashboard.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/domo-dashboard.mdx
index 357a3c1ea..5fdd695ee 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/domo-dashboard.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/domo-dashboard.mdx
@@ -4,10 +4,8 @@ description: Connect Domo dashboards to OpenMetadata with our comprehensive conn
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the DomoDashboard connector.
-Configure and schedule DomoDashboard metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule DomoDashboard metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/domo-dashboard/troubleshooting)
+
## Requirements
+
For metadata ingestion, make sure to add at least `data` scopes to the clientId provided.
-For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication).
+For questions related to scopes, see the [Domo API authentication documentation](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication).
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Domo Dashboard, you need to create a service connection. The service connects Domo Dashboard with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Domo Dashboard** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Domo Dashboard services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Domo Dashboard. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Client ID**: Client ID to connect to Domo Dashboard.
+- **Secret Token**: Secret Token to connect to Domo Dashboard.
+- **Access Token**: Access Token to connect to Domo Dashboard.
+- **API Host**: API Host to connect to the Domo Dashboard instance.
+- **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
-- **Client ID**: Client ID to Connect to DOMO Dashboard.
-- **Secret Token**: Secret Token to Connect DOMO Dashboard.
-- **Access Token**: Access to Connect to DOMO Dashboard.
-- **API Host**: API Host to Connect to DOMO Dashboard instance.
-- **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`.
-
-
-
-
-
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/grafana.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/grafana.mdx
index f68e30ddb..75b62757d 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/grafana.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/grafana.mdx
@@ -4,10 +4,8 @@ description: Learn how to configure and use the Grafana connector in OpenMetadat
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Grafana connector.
-Configure and schedule Grafana metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Grafana metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
+- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/grafana/troubleshooting)
+
## Requirements
+
To access the Grafana APIs and import dashboards and panels into OpenMetadata, you need a Service Account Token with sufficient permissions (Admin role is recommended) and API access enabled on your Grafana instance.
+
## Metadata Ingestion
-
-
-
-
-
-
-Enable debug logging to troubleshoot issues:
-```bash
-export LOG_LEVEL=DEBUG
-```
+
+To ingest metadata from Grafana, you need to create a service connection. The service connects Grafana with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Grafana** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Grafana services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Grafana. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host and Port**: URL to the Grafana instance.
+- **API Token**: Service Account Token with sufficient permissions (Admin role recommended).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/hex.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/hex.mdx
index 1e8741b0e..d1cd40e48 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/hex.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/hex.mdx
@@ -3,11 +3,9 @@ title: Hex Connector | OpenMetadata Integration Documentation
description: Connect Hex to OpenMetadata with our comprehensive dashboard connector guide. Setup instructions, configuration tips, and metadata extraction made simple.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
Hex connector supports both personal and workspace tokens for authentication.
### Hex Account Setup
-Follow the steps below to configure the account setup for Hex connector:
+Follow the steps below to configure the account setup for the Hex connector:
### Step 1: Generate API Token
To generate an API token in Hex:
-1. Log into your [Hex](https://app.hex.tech) account
-2. Navigate to Account Settings
-3. Go to the API Keys section
-4. Click on "Create API Key"
+1. Log into your [Hex](https://app.hex.tech) account.
+2. Navigate to Account Settings.
+3. Go to the API Keys section.
+4. Click **Create API Key**.
5. Choose the token type:
- - **Personal Token**: Provides access to projects you own or have access to
- - **Workspace Token**: Provides access to all projects in the workspace (requires admin privileges)
-6. Copy and securely store the generated token
+ - **Personal Token**: Provides access to projects you own or have access to.
+ - **Workspace Token**: Provides access to all projects in the workspace (requires admin privileges).
+6. Copy and securely store the generated token.
API tokens are shown only once when created. Make sure to copy and store them securely.
@@ -55,76 +55,135 @@ API tokens are shown only once when created. Make sure to copy and store them se
### Step 2: Verify API Access
Ensure that the token has appropriate permissions:
-- For personal tokens: Verify you have access to the projects you want to ingest
-- For workspace tokens: Verify admin privileges are granted
+- For personal tokens: Verify you have access to the projects you want to ingest.
+- For workspace tokens: Verify admin privileges are granted.
## Metadata Ingestion
-
+To ingest metadata from Hex, you need to create a service connection. The service connects Hex with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
-## Connection Details
+From the service type dropdown, select **Dashboard Services**, then click the **Hex** connector tile.
-
+
-
+### Step 3: Add Service Name and Description
-**hostPort**: URL to the Hex instance.
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Hex services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
-Provide the URL to your Hex instance. For Hex cloud users, use `https://app.hex.tech`.
+
-For on-premise installations, provide your custom domain URL.
+
+**Note**: The service name cannot be changed after it is set.
+
-**tokenType**: Type of authentication token.
+### Step 4: Configure Connection Options
-Select the type of token you're using:
-- **personal**: Personal access token (default)
-- **workspace**: Workspace-level token (requires admin privileges)
+Specify your source credentials and verify the connection.
-**token**: Hex API Token.
+#### Enter Connection Details
-Provide the API token generated from your Hex account. This token is used for authentication with the Hex APIs.
+Enter the connection details for Hex. The right-hand panel in the UI displays inline help for each field.
-To generate a token:
-- Log into Hex
-- Go to Account Settings > API Keys
-- Create a new API key
-- Copy the generated token
+
-**includeTags**: Import Hex Categories and Status as Tags.
+- **hostPort**: URL to the Hex instance. Provide the URL to your Hex instance. For Hex cloud users, use `https://app.hex.tech`. For on-premise installations, provide your custom domain URL.
+- **tokenType**: Type of authentication token. Select the type of token you're using:
+ - **personal**: Personal access token (default)
+ - **workspace**: Workspace-level token (requires admin privileges)
+- **token**: Hex API Token. Provide the API token generated from your Hex account. This token is used for authentication with the Hex APIs.
-Enable this option to import Hex project categories and status as OpenMetadata tags. This helps in organizing and filtering your Hex projects within OpenMetadata.
+ To generate a token:
+ - Log into Hex
+ - Go to **Account Settings** > **API Keys**
+ - Create a new API key
+ - Copy the generated token
+- **includeTags**: Import Hex Categories and Status as Tags. Enable this option to import Hex project categories and status as OpenMetadata tags. This helps in organizing and filtering your Hex projects within OpenMetadata. By default, this is set to `true`.
+- **Dashboard Filter Pattern**: Use regex patterns to include or exclude specific dashboards from ingestion. This helps you control which Hex projects are imported into OpenMetadata.
-By default, this is set to `true`.
+ Examples:
+ - Include only production projects: `^prod-.*`
+ - Exclude test projects: `^(?!test-).*`
-**Dashboard Filter Pattern**:
+#### Test Connection
-Use regex patterns to include or exclude specific dashboards from ingestion. This helps you control which Hex projects are imported into OpenMetadata.
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
-Examples:
-- Include only production projects: `^prod-.*`
-- Exclude test projects: `^(?!test-).*`
+
-
+### Step 5: Configure Ingestion Options
-
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
-
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
-
+**Filter Options**
-
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
## Lineage
Hex connector extracts lineage information by:
-- **Warehouse Query History**: Directly queries data warehouse logs (Snowflake, BigQuery, Databricks, Redshift, etc.) to find queries executed by Hex
-- **Hex Metadata Identification**: Identifies Hex-originated queries using metadata comments embedded by Hex containing project IDs
-- **SQL Parsing**: Analyzes the discovered SQL queries to extract upstream table dependencies
+- **Warehouse Query History**: Directly queries data warehouse logs (Snowflake, BigQuery, Databricks, Redshift, and others) to find queries executed by Hex.
+- **Hex Metadata Identification**: Identifies Hex-originated queries using metadata comments embedded by Hex containing project IDs.
+- **SQL Parsing**: Analyzes the discovered SQL queries to extract upstream table dependencies.
The lineage information helps you understand:
-- Which data warehouse tables are actually queried by each Hex project
-- Real data usage patterns based on executed queries
-- Impact analysis when making changes to underlying data sources
+- Which data warehouse tables are actually queried by each Hex project.
+- Real data usage patterns based on executed queries.
+- Impact analysis when making changes to underlying data sources.
**Note**: The connector does not use Hex APIs for lineage. Instead, it discovers lineage by analyzing actual queries executed in your data warehouses, providing more accurate data dependency information.
@@ -133,12 +192,12 @@ The lineage information helps you understand:
For lineage extraction to work, query logging must be enabled in your data warehouse:
#### Snowflake
-- Requires access to `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY`
-- Query history is automatically maintained by Snowflake
+- Requires access to `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY`.
+- Query history is automatically maintained by Snowflake.
#### BigQuery
-- Requires access to `INFORMATION_SCHEMA.JOBS_BY_PROJECT`
-- Query history is automatically maintained by BigQuery
+- Requires access to `INFORMATION_SCHEMA.JOBS_BY_PROJECT`.
+- Query history is automatically maintained by BigQuery.
#### MySQL
- Requires general_log to be enabled:
@@ -146,40 +205,18 @@ For lineage extraction to work, query logging must be enabled in your data wareh
SET GLOBAL general_log = 'ON';
SET GLOBAL log_output = 'TABLE';
```
-- Grant access to `mysql.general_log` table
+- Grant access to the `mysql.general_log` table.
#### Databricks
-- Requires access to `system.query.history`
-- Query history is automatically maintained
+- Requires access to `system.query.history`.
+- Query history is automatically maintained.
#### Redshift
-- Requires access to `stl_query` system table
-- Query history is automatically maintained
+- Requires access to the `stl_query` system table.
+- Query history is automatically maintained.
#### Other Warehouses
-- Athena: Access to query history via CloudTrail or S3 logs
-- ClickHouse: Access to `system.query_log` table
-
-## Troubleshooting
-
-### Common Issues
-
-#### Authentication Failures
-- Verify that your API token is valid and hasn't expired
-- Check that the token type matches your access level (personal vs workspace)
-- Ensure the API token has necessary permissions
-
-#### Missing Projects
-- For personal tokens: Verify you have access to the projects
-- For workspace tokens: Confirm admin privileges
-- Check the dashboard filter pattern configuration
-
-#### Connection Timeouts
-- Verify the hostPort URL is correct
-- Check network connectivity to Hex
-- For on-premise installations, ensure firewall rules allow access
+- Athena: Access to query history via CloudTrail or S3 logs.
+- ClickHouse: Access to the `system.query_log` table.
-#### Rate Limiting
-- Hex APIs may have rate limits
-- Consider adjusting the ingestion schedule if you encounter rate limit errors
-- Contact Hex support if persistent rate limiting issues occur
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/lightdash.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/lightdash.mdx
index 7fa75bacb..6b4406db9 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/lightdash.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/lightdash.mdx
@@ -4,11 +4,9 @@ description: Connect Lightdash to OpenMetadata with our comprehensive dashboard
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Lightdash connector.
-Configure and schedule Lightdash metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Lightdash metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/lightdash/troubleshooting)
+
## Requirements
+
To integrate Lightdash, ensure you are using OpenMetadata version 1.2.x or higher.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Lightdash, you need to create a service connection. The service connects Lightdash with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Lightdash** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Lightdash services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Lightdash. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: Specify the network location where your Lightdash instance is accessible, combining both hostname and port in a URI format: either `http://hostname:port` or `https://hostname:port`, based on your security needs.
- - **Example**: For a local setup, use `http://localhost:8080`; for a server deployment, it might be `https://lightdash.example.com:3000`.
+ - **Example**: For a local setup, use `http://localhost:8080`. For a server deployment, use something like `https://lightdash.example.com:3000`.
- Ensure the specified port is open and accessible through network firewall settings.
- **API Key**: This key authenticates requests to your Lightdash instance. Keep the API Key secure, sharing it only with authorized applications or users.
- **Project UUID**: This unique identifier links API requests or configurations to a specific project in Lightdash.
- **Space UUID**: Identifies a specific "Space" in Lightdash, used to organize dashboards, charts, and assets.
- **Proxy Authentication**: If your Lightdash instance requires authentication through a proxy server, provide proxy credentials. Proxy authentication controls access to external resources and Lightdash.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/looker.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/looker.mdx
index 2c4bda012..69fbda215 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/looker.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/looker.mdx
@@ -4,11 +4,9 @@ description: Connect Looker dashboards to OpenMetadata with our comprehensive co
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Looker connector.
-Configure and schedule Looker metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Looker metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
-- [Entity Mapping](#entity-mapping)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/looker/troubleshooting)
+
## Requirements
-There are two types of metadata we ingest from Looker:
+
+OpenMetadata ingests two types of metadata from Looker:
- Dashboards & Charts
- LookML Models
+
For the `project` metadata being ingested:
-- We get the actual LookML Project an Explore or View is developed in.
-- For Dashboards, we use the folder name from the UI, since there is no other hierarchy involved there.
-In terms of permissions, we need a user with access to the Dashboards and LookML Explores that we want to ingest. You can
-create your API credentials following these [docs](https://cloud.google.com/looker/docs/api-auth).
-However, LookML Views are not present in the Looker SDK. Instead, we need to extract that information directly from
-the GitHub repository holding the source `.lkml` files. In order to get this metadata, we will require a GitHub token
-with read only access to the repository. You can follow these steps from the GitHub [documentation](https://docs.github.com/en/enterprise-server@3.4/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
+- The actual LookML Project an Explore or View is developed in.
+- For Dashboards, the folder name from the UI, since there is no other hierarchy involved there.
+
+In terms of permissions, a user with access to the Dashboards and LookML Explores to be ingested is required.
+Create your API credentials following these [docs](https://cloud.google.com/looker/docs/api-auth).
+
+However, LookML Views are not present in the Looker SDK. Instead, that information must be extracted directly from
+the GitHub repository holding the source `.lkml` files. To get this metadata, a GitHub token
+with read only access to the repository is required. Follow these steps from the GitHub [documentation](https://docs.github.com/en/enterprise-server@3.4/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
+
-The GitHub credentials are completely optional. Just note that without them, we won't be able to ingest metadata
+The GitHub credentials are completely optional. Without them, OpenMetadata cannot ingest metadata
out of LookML Views, including their lineage to the source databases.
Moreover, Looker lineage only supports LookML views configured with `sql_table_name` and `derived_table` in plain SQL.
-We do not yet support liquid variables.
+Liquid variables are not yet supported.
+
## Entity Mapping
+
The Looker connector maps Looker assets to OpenMetadata entities as follows:
+
| Looker Asset | OpenMetadata Entity | Description |
|---|---|---|
| **Dashboards** | **Dashboard** | Looker dashboards are mapped to OpenMetadata Dashboard entities. |
| **Dashboard Elements** (tiles) | **Chart** | Visualization tiles within a Looker dashboard are mapped to OpenMetadata Chart entities. |
| **Explores** and **Views** (LookML) | **Data Models** | LookML Explores and Views are mapped to OpenMetadata Data Model entities. |
| **Folders** (Dashboards) / **LookML Projects** (Explores & Views) | **Project field** | The project field means something different depending on the entity: for a Dashboard it stores the Looker folder name, while for an Explore or View it stores the LookML project (Git repository) it belongs to. |
+
### Example Structure
+
**Looker Structure:**
```
Folder: Sales Team
@@ -61,6 +68,7 @@ LookML Project: sales_analytics
├── Explore: revenue
└── View: orders
```
+
**OpenMetadata Structure:**
```
Dashboard: "Revenue Overview" (Project: "Sales Team")
@@ -69,36 +77,122 @@ Dashboard: "Revenue Overview" (Project: "Sales Team")
Data Model: "revenue" (Project: "sales_analytics")
Data Model: "orders" (Project: "sales_analytics")
```
+
This mapping ensures that:
- Looker dashboards appear as OpenMetadata dashboards, organized by their Looker folder
- Dashboard tiles appear as charts underneath their dashboard
- LookML Explores and Views appear as data models, organized by their LookML project
- The project field is sourced differently depending on entity type: a Looker folder for Dashboards, a LookML project for Data Models
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Host and Port**: URL to the Looker instance, e.g., `https://my-company.region.looker.com`.
+
+To ingest metadata from Looker, you need to create a service connection. The service connects Looker with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Looker** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Looker services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Looker. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host and Port**: URL to the Looker instance, for example, `https://my-company.region.looker.com`.
- **Client ID**: User's Client ID to authenticate to the SDK. This user should have privileges to read all the metadata in Looker.
- **Client Secret**: User's Client Secret for the same ID provided.
-Then, if we choose to inform the GitHub credentials to ingest LookML Views:
+
+To ingest LookML Views, provide your GitHub credentials:
- **Repository Owner**: The owner (user or organization) of a GitHub repository. For example, in https://github.com/open-metadata/OpenMetadata, the owner is `open-metadata`.
- **Repository Name**: The name of a GitHub repository. For example, in https://github.com/open-metadata/OpenMetadata, the name is `OpenMetadata`.
-- **API Token**: Token to use the API. This is required for private repositories and to ensure we don't hit API limits.
-Follow these [steps](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-fine-grained-personal-access-token) in order to create a fine-grained personal access token.
-When configuring, give repository access to `Only select repositories` and choose the one containing your LookML files. Then, we only need `Repository Permissions` as `Read-only` for `Contents`.
-
-
-
-
-
+- **API Token**: Token to use the API. This is required for private repositories and to avoid hitting API rate limits.
+
+Follow these [steps](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-fine-grained-personal-access-token) to create a fine-grained personal access token.
+When configuring, give repository access to `Only select repositories` and choose the one containing your LookML files. Set `Repository Permissions` to `Read-only` for `Contents`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/metabase.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/metabase.mdx
index 8927cbf54..2d8a696ef 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/metabase.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/metabase.mdx
@@ -1,14 +1,12 @@
---
title: Metabase Connector | OpenMetadata Dashboard Integration
-description: Connect Metabase dashboards to OpenMetadata effortlessly. Complete setup guide, configuration steps, and metadata extraction for seamless integration.
+description: Connect Metabase dashboards to OpenMetadata. Complete setup guide, configuration steps, and metadata extraction.
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Supported Authentication Types:**
- **Basic Auth** — Connect using a Metabase account email and password. The connector obtains a session token automatically.
-- **API Token** — Connect using a Metabase API key. Recommended when basic authentication is disabled (e.g., SSO-enforced environments).
+- **API Token** — Connect using a Metabase API key. Recommended when basic authentication is disabled (for example, SSO-enforced environments).
-Configure and schedule Metabase metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Metabase metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/metabase/troubleshooting)
+
## Requirements
+
The Metabase account or API key used to connect must have view access to all dashboards and charts you want to ingest.
To ingest **owner information**, the account must also have visibility of Metabase user profiles. This is available to admin accounts and accounts in groups with the appropriate permissions.
-**Note:** We have tested Metabase with Versions `0.42.4` and `0.43.4`.
+**Note:** OpenMetadata has been tested with Metabase versions `0.42.4` and `0.43.4`.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Metabase, you need to create a service connection. The service connects Metabase with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Metabase** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Metabase services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Metabase. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: URL of your Metabase instance, in the format `http://hostname:port` or `https://hostname:port`. For example: `https://org.metabase.com:3000`.
Metabase supports two authentication methods. Provide credentials for **one** of the following:
@@ -58,9 +86,63 @@ Metabase supports two authentication methods. Provide credentials for **one** of
**Option 2 — API Token**
- **API Token**: API key for Metabase. Use this instead of username and password when basic authentication is disabled, for example in SSO-enforced environments. Generate the key in the Metabase Admin panel under **Settings > Authentication > API Keys**.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/microstrategy.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/microstrategy.mdx
index 3a4c5ed6c..ae677fb1d 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/microstrategy.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/microstrategy.mdx
@@ -4,10 +4,8 @@ description: Connect MicroStrategy dashboards to OpenMetadata with our comprehen
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the MicroStrategy connector.
-Configure and schedule MicroStrategy metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule MicroStrategy metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/microstrategy/troubleshooting)
+
## Requirements
+
To integrate MicroStrategy, ensure you are using OpenMetadata version 1.2.x or higher.
When a service user is created, it is already provisioned with the necessary permissions.
However, if the user still cannot access the APIs, the following should be checked as part of the troubleshooting process:
-- Required DSS Privileges for MicroStrategy REST/JSON API:
-- Web Services API: Essential for REST API usage.
-- Login to MicroStrategy: User authentication.
-- Use Project Sources: Access to project sources.
-- View Metadata: Metadata browsing and viewing.
-- Access Administration Objects: Global metadata access (connections, DB instances).
-- Browse Repository: Object navigation within projects/folders.
+
+The following DSS privileges are required for the MicroStrategy REST/JSON API:
+- **Web Services API**: Essential for REST API usage.
+- **Login to MicroStrategy**: User authentication.
+- **Use Project Sources**: Access to project sources.
+- **View Metadata**: Metadata browsing and viewing.
+- **Access Administration Objects**: Global metadata access (connections, DB instances).
+- **Browse Repository**: Object navigation within projects/folders.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Username**: Username to connect to MicroStrategy, e.g., user@organization.com. This user should have access to relevant dashboards and charts in MicroStrategy to fetch the metadata.
+
+To ingest metadata from MicroStrategy, you need to create a service connection. The service connects MicroStrategy with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **MicroStrategy** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other MicroStrategy services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for MicroStrategy. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Username**: Username to connect to MicroStrategy, for example, user@organization.com. This user should have access to relevant dashboards and charts in MicroStrategy to fetch the metadata.
- **Password**: Password of the user account to connect with MicroStrategy.
-- **Host Port**: This parameter specifies the host of the MicroStrategy instance. This should be specified as a URI string in the format `http://hostname` or `https://hostname`.
-For example, you might set it to https://demo.microstrategy.com.
+- **Host Port**: This parameter specifies the host of the MicroStrategy instance. This should be specified as a URI string in the format `http://hostname` or `https://hostname`. For example, you might set it to https://demo.microstrategy.com.
- **Project Name**: The name of the project within MicroStrategy that OpenMetadata will connect to, linking to the relevant dashboards and reports for metadata retrieval.
-- **Login Mode**: Login Mode for Microstrategy's REST API connection. You can authenticate with one of the following authentication modes: `Standard (1)`, `Anonymous (8)`. Default will be `Standard (1)`.
-If you're using demo account for Microstrategy, it will be needed to authenticate through loginMode `8`.
-
-
-
-
-
+- **Login Mode**: Login Mode for MicroStrategy's REST API connection. You can authenticate with one of the following authentication modes: `Standard (1)`, `Anonymous (8)`. Default will be `Standard (1)`.
+If you're using a demo account for MicroStrategy, authenticate using loginMode `8`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/mode.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/mode.mdx
index 7ec0f73ae..bd9e9f7e2 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/mode.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/mode.mdx
@@ -4,11 +4,9 @@ description: Connect Mode Analytics dashboards to OpenMetadata with our comprehe
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Mode connector.
-Configure and schedule Mode metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Mode metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/mode/troubleshooting)
+
## Requirements
+
OpenMetadata relies on Mode's API, which is exclusive to members of the Mode Business Workspace. This means that only resources that belong to a Mode Business Workspace can be accessed via the API.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Mode, you need to create a service connection. The service connects Mode with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Mode** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Mode services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Mode. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **hostPort**: Host and port for the Mode dashboard. The hostPort parameter specifies the host and port of the Mode server. This should be specified as a string in the format `https://app.mode.com`.
+- **accessToken**: Access Token for Mode Dashboard. Get the Access Token by following the steps below:
+ - Navigate to your Mode homepage.
+ - Click on your name in the upper left corner and click My Account.
+ - Click on API Tokens on the left side.
+ - To generate a new API token and password, enter a token name and click **Create Token**.
+ - Copy the generated access token and password.
+
+ For detailed information, see the [Mode API reference](https://mode.com/developer/api-reference/introduction/).
+- **accessTokenPassword**: Access Token Password for Mode Dashboard. Copy the access token password from the step above where a new token is generated. For detailed information, see the [Mode API reference](https://mode.com/developer/api-reference/introduction/).
+- **workspaceName**: Mode Workspace Name. Name of the Mode workspace from which OpenMetadata fetches metadata.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
-**hostPort**: Host and Port Mode Dashboard.
-The hostPort parameter specifies the host and port of the Mode server. This should be specified as a string in the format `https://app.mode.com`.
-**accessToken**: Access Token for Mode Dashboard.
-Get the Access Token by following below steps:
-- Navigate to your Mode homepage.
-- Click on your name in the upper left corner and click My Account.
-- Click on API Tokens on the left side.
-- To generate a new API token and password, enter a token name and click `Create token`.
-- Copy the generated access token and password.
-For detailed information visit [here](https://mode.com/developer/api-reference/introduction/).
-**accessTokenPassword**: Access Token Password for Mode Dashboard.
-Copy the access token password from the step above where a new token is generated.
-For detailed information visit [here](https://mode.com/developer/api-reference/introduction/).
-**workspaceName**: Mode Workspace Name.
-Name of the mode workspace from where the metadata is to be fetched.
-
-
-
-
-
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/omni.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/omni.mdx
new file mode 100644
index 000000000..3ccd803e6
--- /dev/null
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/omni.mdx
@@ -0,0 +1,156 @@
+---
+title: Omni Connector | OpenMetadata Dashboard Guide
+description: Connect Omni to OpenMetadata with our comprehensive dashboard connector guide. Setup instructions, configuration tips, and metadata extraction made simple.
+sidebarTitle: Overview
+---
+import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
+import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+
+
+In this section, we provide guides and references to use the Omni connector.
+Configure and schedule Omni metadata workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
+- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/omni/troubleshooting)
+
+## Requirements
+
+OpenMetadata ingests Omni metadata through Omni's REST API. The connector reads Omni models and topics as data models, documents (workbooks) as dashboards with their tiles as charts, and builds lineage from your warehouse tables through topics to dashboards.
+
+### Generate an API Token
+
+OpenMetadata authenticates with Omni using an API token sent as a `Bearer` token. Use one of the following:
+
+- **Organization API Key**: Created by an Organization Admin. Grants access across the organization's models and documents.
+- **Personal Access Token (PAT)**: Scoped to your own access.
+
+For steps to obtain either Omni credential, see [API authentication](https://docs.omni.co/api/authentication).
+
+
+**Tip**: The token is shown only once when created. Copy and store it securely.
+
+
+## Metadata Ingestion
+
+To ingest metadata from Omni, you need to create a service connection. The service connects Omni with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Omni** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Omni services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Omni. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Host and Port**: URL of the Omni instance, for example `https://your-org.omniapp.co`. The `/api` path is added automatically.
+
+**Authentication**
+- **API Token**: API token to authenticate with Omni. Generated in [Generate an API Token](#generate-an-api-token).
+
+**Scope & Options**
+- **Verify SSL**: Client SSL verification. Configure **SSL Config** if enabled. Defaults to `no-ssl`.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **CA Certificate**: The CA certificate used for SSL validation.
+- **SSL Certificate**: The SSL certificate used for client authentication.
+- **SSL Key**: The SSL key used for client authentication.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
+Omni resolves lineage from each topic's declared base table, so it doesn't need to parse warehouse query logs. A topic built on top of a table OpenMetadata has already indexed produces a direct edge from that source table, through the topic (data model), to every dashboard tile built on it.
+
+
\ No newline at end of file
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/omni/troubleshooting.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/omni/troubleshooting.mdx
new file mode 100644
index 000000000..6e9ea63e1
--- /dev/null
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/omni/troubleshooting.mdx
@@ -0,0 +1,69 @@
+---
+title: Omni Connector Troubleshooting Guide | OpenMetadata Support
+description: Resolve Omni dashboard ingestion issues such as authentication failures, rate limits, connection timeouts, and missing lineage.
+sidebarTitle: Troubleshooting
+---
+
+import Troubleshooting from '/snippets/connectors/troubleshooting.mdx'
+
+
+
+## Omni-Specific Issues
+
+The following issues are specific to the Omni connector's REST API integration.
+
+### Authentication Failed (401)
+
+**Issue**: The Test Connection step fails with a 401 error.
+
+**Cause**: Omni rejected the API token, it may be invalid, expired, or revoked, or Host and Port does not point at your Omni organization.
+
+**Solution**: Verify the token value and confirm it hasn't been revoked in Omni. Confirm **Host and Port** points at your Omni organization, for example `https://your-org.omniapp.co`. Regenerate the token in Omni and update the connection if authentication continues to fail.
+
+### Insufficient Permissions (403)
+
+**Issue**: The Test Connection step fails with a 403 error.
+
+**Cause**: The token is valid but not authorized to read models and documents.
+
+**Solution**: Use an Organization API Key (or a Personal Access Token with sufficient access) that can read models and documents. See [Requirements](#requirements) for how to generate one.
+
+### Rate Limit Reached (429)
+
+**Issue**: The Test Connection step or a metadata ingestion run fails or is delayed with a 429 error.
+
+**Cause**: Omni throttled the request. The default limit is 60 requests/minute.
+
+**Solution**: Wait and retry, or ask Omni to raise the rate limit for the API key. If this happens consistently during ingestion, narrow the Dashboard/Chart/Data Model/Project filter patterns to reduce the number of assets processed per run.
+
+### TLS Verification Failed
+
+**Issue**: The Test Connection step fails with an SSL/TLS certificate verification error.
+
+**Cause**: The server's certificate could not be verified with the current SSL configuration.
+
+**Solution**: Provide the CA certificate under **SSL Config** with **Verify SSL** set to `validate`, or set **Verify SSL** to `ignore` for a self-signed certificate (not recommended outside of testing).
+
+### Connection Timeouts
+
+**Issue**: The Test Connection step fails with a timeout or cannot reach the host.
+
+**Cause**: **Host and Port** is incorrect, includes a trailing `/api` (the connector appends this automatically), or the ingestion runner cannot reach your Omni instance over the network.
+
+**Solution**: Verify **Host and Port** is correct and does not include a trailing `/api`. Check network connectivity between the ingestion runner and your Omni instance. If your Omni instance sits behind an internal certificate authority, set **Verify SSL** to `validate` and provide the CA certificate under **SSL Config**.
+
+### Missing Dashboards, Charts, or Data Models
+
+**Issue**: Some dashboards, charts, data models, or projects you expect to see are missing after ingestion.
+
+**Cause**: The Dashboard, Chart, Data Model, or Project filter pattern excludes them, or the token doesn't have access to the project or folder they live in.
+
+**Solution**: Check the **Dashboard Filter Pattern**, **Chart Filter Pattern**, **Data Model Filter Pattern**, and **Project Filter Pattern** for rules that might be excluding the assets you expect to see. Confirm the token has access to the project or folder the document lives in.
+
+### Missing Lineage
+
+**Issue**: Dashboards, charts, and data models are ingested, but no table-to-table lineage edges appear.
+
+**Cause**: Lineage depends on each Omni topic's declared base table matching a table OpenMetadata has already ingested. If the source database service hasn't been ingested yet, or its schema/table names don't match what the Omni topic points to, lineage can't be resolved.
+
+**Solution**: Ingest the source database service before running the Omni service, and confirm its schema and table names match what Omni's topic points to. Set **Db Service Prefixes** in the metadata agent's **Lineage Information** section to scope table matching to the correct database service.
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/omni/yaml.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/omni/yaml.mdx
new file mode 100644
index 000000000..13fe2454a
--- /dev/null
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/omni/yaml.mdx
@@ -0,0 +1,148 @@
+---
+title: Run the Omni Connector Externally
+description: Use YAML to ingest metadata from Omni, including models, topics, workbooks, dashboards, charts, and lineage.
+sidebarTitle: Run Externally
+mode: "wide"
+---
+import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import { CodePreview, ContentPanel, ContentSection, CodePanel } from '/snippets/components/CodePreview.jsx'
+import SourceConfigDef from '/snippets/connectors/yaml/dashboard/source-config-def.mdx'
+import IngestionSinkDef from '/snippets/connectors/yaml/ingestion-sink-def.mdx'
+import WorkflowConfigDef from '/snippets/connectors/yaml/workflow-config-def.mdx'
+import SourceConfig from '/snippets/connectors/yaml/dashboard/source-config.mdx'
+import IngestionSink from '/snippets/connectors/yaml/ingestion-sink.mdx'
+import WorkflowConfig from '/snippets/connectors/yaml/workflow-config.mdx'
+import IngestionCli from '/snippets/connectors/yaml/ingestion-cli.mdx'
+import PythonRequirements from '/snippets/connectors/python-requirements.mdx'
+import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
+
+
+In this section, we provide guides and references to use the Omni connector.
+
+OpenMetadata ingests Omni metadata through Omni's REST API. The connector reads Omni models and topics as data models, documents (workbooks) as dashboards with their tiles as charts, and builds lineage from your warehouse tables through topics to dashboards.
+
+Configure and schedule Omni metadata workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
+
+## Requirements
+
+OpenMetadata authenticates with Omni using an API token sent as a `Bearer` token. Use either an **Organization API Key** (created by an Organization Admin, grants access across the organization's models and documents) or a **Personal Access Token** (scoped to your own access). For steps to obtain either credential, see [API authentication](https://docs.omni.co/api/authentication).
+
+
+**Tip**: The token is shown only once when created. Copy and store it securely.
+
+
+### Python Requirements
+
+Omni only depends on the `requests` library, which already ships with the base package. To run the Omni ingestion, you will need to install:
+```bash
+pip3 install "openmetadata-ingestion"
+```
+## Metadata Ingestion
+All connectors are defined as JSON Schemas.
+[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/dashboard/omniConnection.json)
+you can find the structure to create a connection to Omni.
+In order to create and run a Metadata Ingestion workflow, we will follow
+the steps to create a YAML configuration able to connect to the source,
+process the Entities if needed, and reach the OpenMetadata server.
+The workflow is modeled around the following
+[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
+### 1. Define the YAML Config
+This is a sample config for Omni:
+
+
+
+
+
+Configure the source type and service name for your Omni connector.
+
+
+
+
+
+**hostPort**: URL of the Omni instance, for example, `https://your-org.omniapp.co`. The `/api` path is added automatically.
+
+
+
+
+
+**token**: API token to authenticate with Omni. Generated in [Requirements](#requirements) above.
+
+
+
+
+
+**verifySSL** (Optional): Client SSL verification mode: `no-ssl` (default), `ignore`, or `validate`.
+**sslConfig** (Optional): CA certificate (and, for mutual TLS, client certificate and key) used when `verifySSL` is set to `validate`.
+
+
+
+
+
+**dashboardFilterPattern**, **chartFilterPattern**, **dataModelFilterPattern**, **projectFilterPattern** (all Optional): Regex to include or exclude dashboards, charts, data models (Omni topics), and projects that match the pattern.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+```yaml
+source:
+ type: omni
+ serviceName: omni_source
+ serviceConnection:
+ config:
+ type: Omni
+ hostPort: https://your-org.omniapp.co # REQUIRED
+ token: # REQUIRED
+ # verifySSL: no-ssl
+ # dashboardFilterPattern:
+ # includes:
+ # -
+ # excludes:
+ # -
+ # chartFilterPattern:
+ # includes:
+ # -
+ # dataModelFilterPattern:
+ # includes:
+ # -
+ # projectFilterPattern:
+ # includes:
+ # -
+```
+
+
+
+
+
+
+
+
+## Lineage
+
+Omni resolves lineage from each topic's declared base table, so it doesn't need to parse warehouse query logs. A topic built on top of a table OpenMetadata has already indexed produces a direct edge from that source table, through the topic (data model), to every dashboard tile built on it. There is no separate lineage agent or pipeline: lineage is collected as part of the same metadata ingestion workflow above.
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi.mdx
index 4f2239ca4..9d8e91477 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi.mdx
@@ -4,11 +4,9 @@ description: Connect Power BI to OpenMetadata with our comprehensive dashboard c
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-Configure and schedule PowerBI metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule PowerBI metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Entity Mapping](#entity-mapping)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi/troubleshooting)
+
## Requirements
+
+Before configuring the PowerBI connector, make sure the following requirements are in place.
+
To access the PowerBI APIs and import dashboards, charts, and datasets from PowerBI into OpenMetadata, a `PowerBI Pro` license is necessary.
@@ -41,53 +42,74 @@ OpenMetadata does not support Power BI usage ingestion because the Power BI Usag
When configuring Azure Authentication, ensure that "Allow public client flows" is enabled. This setting is required to support authentication for public client applications.
-### PowerBI Admin and Non-Admin APIs:
-While configuring the PowerBI ingestion you can choose whether to use the PowerBI Admin APIs to retrieve the metadata or use the PowerBI Non-Admin APIs. Please check below for the the difference in their functionality:
-- Enabled (Use PowerBI Admin APIs)
-Using the admin APIs will fetch the dashboard and chart metadata from all the workspaces available in the PowerBI instance.
+
+### PowerBI Admin and Non-Admin APIs
+
+While configuring the PowerBI ingestion you can choose whether to use the PowerBI Admin APIs to retrieve the metadata or use the PowerBI Non-Admin APIs. See below for the difference in their functionality:
+
+- Enabled (Use PowerBI Admin APIs): Using the admin APIs fetches the dashboard and chart metadata from all the workspaces available in the PowerBI instance.
+
When using the PowerBI Admin APIs, the table and dataset information used to generate lineage is gathered using the PowerBI [Scan Result](https://learn.microsoft.com/en-us/rest/api/power-bi/admin/workspace-info-get-scan-result) API. This API has no limitations and hence does not restrict getting the necessary data for generating lineage.
-- Disabled (Use Non-Admin PowerBI APIs)
-Using the non-admin APIs will only fetch the dashboard and chart metadata from the workspaces that have the security group of the service principal assigned to them.
+
+- Disabled (Use Non-Admin PowerBI APIs): Using the non-admin APIs only fetches the dashboard and chart metadata from the workspaces that have the security group of the service principal assigned to them.
+
When using the PowerBI Non-Admin APIs, the table and dataset information used to generate lineage is gathered using the PowerBI [Get Dataset Tables](https://learn.microsoft.com/en-us/rest/api/power-bi/push-datasets/datasets-get-tables) API. This API only retrieves the table information if the dataset is a [Push Dataset](https://learn.microsoft.com/en-us/rest/api/power-bi/push-datasets).
Hence the lineage can only be created for push datasets in this case.
For more information please visit the PowerBI official documentation [here](https://learn.microsoft.com/en-us/rest/api/power-bi/push-datasets/datasets-get-tables#limitations).
+
### PowerBI Account Setup
-Follow the steps below to configure the account setup for PowerBI connector:
-### Step 1: Enable API permissions from the PowerBI Admin console
-We extract the information from PowerBI using APIs, this is a manual step a PowerBI Admin needs to do to ensure we can get the right information.
-Login to the [Power BI](https://app.powerbi.com/) as Admin and from `Tenant` settings allow below permissions.
+
+Follow the steps below to configure the account setup for the PowerBI connector:
+
+#### Step 1: Enable API permissions from the PowerBI Admin console
+
+We extract the information from PowerBI using APIs; this is a manual step a PowerBI Admin needs to do to ensure OpenMetadata can get the right information.
+Log in to the [Power BI](https://app.powerbi.com/) as Admin and from `Tenant` settings allow the below permissions:
- Allow service principles to use Power BI APIs
- Allow service principals to use read-only Power BI admin APIs
- Enhance admin APIs responses with detailed metadata
-### Step 2: Create the App in Azure AD
-Please follow the steps mentioned [here](https://docs.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal) for setting up the Azure AD application service principle.
-### Step 3: Provide necessary API permissions to the Azure AD app
-Go to the `Azure Ad app registrations` page, select your app and add the dashboard permissions to the app for PowerBI service and grant admin consent for the same:
+
+#### Step 2: Create the App in Azure AD
+
+Please follow the steps mentioned [here](https://docs.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal) for setting up the Azure AD application service principal.
+
+#### Step 3: Provide necessary API permissions to the Azure AD app
+
+Go to the `Azure Ad app registrations` page, select your app and add the dashboard permissions to the app for the PowerBI service, and grant admin consent for the same:
+
The required permissions are:
- `Dashboard.Read.All`
-Optional Permissions: (Without granting these permissions, the dataset information cannot be retrieved and the datamodel and lineage processing will be skipped)
+
+Optional Permissions: (Without granting these permissions, the dataset information cannot be retrieved and the datamodel and lineage processing is skipped)
- `Dataset.Read.All`
+
-Make sure that in the API permissions section **Tenant** related permissions are not being given to the app
-Please refer [here](https://stackoverflow.com/questions/71001110/power-bi-rest-api-requests-not-authorizing-as-expected) for detailed explanation
+Make sure that in the API permissions section, **Tenant** related permissions are not being given to the app. Please refer [here](https://stackoverflow.com/questions/71001110/power-bi-rest-api-requests-not-authorizing-as-expected) for a detailed explanation.
-### Step 4: PowerBI Workspaces
-The service principal does not take into account the default user workspaces e.g `My Workspace`.
-Create new workspaces in PowerBI by following the document [here](https://docs.microsoft.com/en-us/power-bi/collaborate-share/service-create-the-new-workspaces)
-For reference here is a [thread](https://community.powerbi.com/t5/Service/Error-while-executing-Get-dataset-call-quot-API-is-not/m-p/912360#M85711) referring to the same
+
+#### Step 4: PowerBI Workspaces
+
+The service principal does not take into account the default user workspaces, for example, `My Workspace`.
+Create new workspaces in PowerBI by following the document [here](https://docs.microsoft.com/en-us/power-bi/collaborate-share/service-create-the-new-workspaces).
+For reference, see this [thread](https://community.powerbi.com/t5/Service/Error-while-executing-Get-dataset-call-quot-API-is-not/m-p/912360#M85711) referring to the same.
+
## Entity Mapping
+
The PowerBI connector maps PowerBI assets to OpenMetadata entities as follows:
+
| PowerBI Asset | OpenMetadata Entity | Description |
|---|---|---|
| **Dashboards** and **Reports** | **Dashboard** | Both PowerBI Dashboards (tile-based) and Reports are mapped to OpenMetadata Dashboard entities. |
| **Tiles** (within a Dashboard) | **Chart** | Tiles on a PowerBI Dashboard are mapped to OpenMetadata Chart entities. Report pages do not generate separate Chart entities. |
| **Datasets, Dataflows, Datamarts** | **Data Models** | PowerBI datasets, dataflows, and datamarts are mapped to OpenMetadata Data Model entities. |
| **Workspaces** | **Project field** | The PowerBI workspace name is stored in the project field of the Dashboard and Data Model entities. |
+
### Example Structure
+
**PowerBI Structure:**
```
Workspace: Sales Team
@@ -97,6 +119,7 @@ Workspace: Sales Team
├── Report: Sales Trends
└── Dataset: Sales Data Model
```
+
**OpenMetadata Structure:**
```
Dashboard: "Revenue Overview" (Project: "Sales Team")
@@ -105,67 +128,139 @@ Dashboard: "Revenue Overview" (Project: "Sales Team")
Dashboard: "Sales Trends" (Project: "Sales Team")
Data Model: "Sales Data Model" (Project: "Sales Team")
```
+
This mapping ensures that:
- PowerBI dashboards and reports both appear as OpenMetadata dashboards
- Tiles on a dashboard appear as charts underneath it
- Datasets, dataflows, and datamarts appear as data models
- PowerBI workspace names are preserved in the project field
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from PowerBI, you need to create a service connection. The service connects PowerBI with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **PowerBI** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other PowerBI services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for PowerBI. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **clientId**: PowerBI Client ID. To get the client ID (also known as application ID), follow these steps:
+ - Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
+ - Search for App registrations and select the App registrations link.
+ - Select the Azure AD app you're using for embedding your Power BI content.
+ - From the Overview section, copy the Application (client) ID.
+- **clientSecret**: PowerBI Client Secret. To get the client secret, follow these steps:
+ - Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
+ - Search for App registrations and select the App registrations link.
+ - Select the Azure AD app you're using for embedding your Power BI content.
+ - Under Manage, select Certificates & secrets.
+ - Under Client secrets, select New client secret.
+ - In the Add a client secret pop-up window, provide a description for your application secret, select when the application secret expires, and select Add.
+ - From the Client secrets section, copy the string in the Value column of the newly created application secret.
+- **tenantId**: PowerBI Tenant ID. To get the tenant ID, follow these steps:
+ - Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
+ - Search for App registrations and select the App registrations link.
+ - Select the Azure AD app you're using for Power BI.
+ - From the Overview section, copy the Directory (tenant) ID.
+- **scope**: Service scope. To let OpenMetadata use the Power BI APIs using your Azure AD app, you'll need to add the following scopes:
+ - `https://analysis.windows.net/powerbi/api/.default`
+
+ Add this scope to your Microsoft Entra application registration.
+- **authorityUri**: Authority URI for the service. To identify a token authority, you can provide a URL that points to the authority in question. If you don't specify a URL for the token authority, OpenMetadata uses the default value of https://login.microsoftonline.com/.
+- **hostPort**: URL to the PowerBI instance. To connect with your Power BI instance, you'll need to provide the host URL. If you're using an on-premise installation of Power BI, this will be the domain name associated with your instance. If you don't specify a host URL, OpenMetadata uses the default value of https://app.powerbi.com to connect with your Power BI instance.
+- **Pagination Entity Per Page**: The pagination limit for Power BI APIs can be set using this parameter. The limit determines the number of records to be displayed per page. By default, the pagination limit is set to 100 records, which is also the maximum value allowed.
+- **Use Admin APIs**: Option for using the PowerBI admin APIs. Refer to the [Power BI Admin and Non-Admin APIs](/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi#powerbi-admin-and-non-admin-apis) section for more information.
+ - Enabled (Use PowerBI Admin APIs)
+ - Disabled (Use Non-Admin PowerBI APIs)
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
-**clientId**: PowerBI Client ID.
-To get the client ID (also known as application ID), follow these steps:
-- Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
-- Search for App registrations and select the App registrations link.
-- Select the Azure AD app you're using for embedding your Power BI content.
-- From the Overview section, copy the Application (client) ID.
-**clientSecret**: PowerBI Client Secret.
-To get the client secret, follow these steps:
-- Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
-- Search for App registrations and select the App registrations link.
-- Select the Azure AD app you're using for embedding your Power BI content.
-- Under Manage, select Certificates & secrets.
-- Under Client secrets, select New client secret.
-- In the Add a client secret pop-up window, provide a description for your application secret, select when the application secret expires, and select Add.
-- From the Client secrets section, copy the string in the Value column of the newly created application secret.
-**tenantId**: PowerBI Tenant ID.
-To get the tenant ID, follow these steps:
-- Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
-- Search for App registrations and select the App registrations link.
-- Select the Azure AD app you're using for Power BI.
-- From the Overview section, copy the Directory (tenant) ID.
-**scope**: Service scope.
-To let OM use the Power BI APIs using your Azure AD app, you'll need to add the following scopes:
-- `https://analysis.windows.net/powerbi/api/.default`
-
-Add this scope to your Microsoft Entra application registration.
-**authorityUri**: Authority URI for the service.
-To identify a token authority, you can provide a URL that points to the authority in question.
-If you don't specify a URL for the token authority, we'll use the default value of https://login.microsoftonline.com/.
-**hostPort**: URL to the PowerBI instance.
-To connect with your Power BI instance, you'll need to provide the host URL. If you're using an on-premise installation of Power BI, this will be the domain name associated with your instance.
-If you don't specify a host URL, we'll use the default value of https://app.powerbi.com to connect with your Power BI instance.
-**Pagination Entity Per Page**:
-The pagination limit for Power BI APIs can be set using this parameter. The limit determines the number of records to be displayed per page.
-By default, the pagination limit is set to 100 records, which is also the maximum value allowed.
-**Use Admin APIs**:
-Option for using the PowerBI admin APIs:
-Refer to the [Power BI Admin and Non-Admin APIs](/v2.0.x-SNAPSHOT/connectors/dashboard/powerbi#powerbi-admin-and-non-admin-apis) section for more information.
-- Enabled (Use PowerBI Admin APIs)
-- Disabled (Use Non-Admin PowerBI APIs)
-
-
-
-
-
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/qlikcloud.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/qlikcloud.mdx
index 558c62d40..97bbaa12a 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/qlikcloud.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/qlikcloud.mdx
@@ -4,11 +4,9 @@ description: Connect QlikCloud dashboards to OpenMetadata effortlessly. Complete
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Qlik Cloud connector.
-Configure and schedule QlikCloud metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Qlik Cloud metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/qlikcloud/troubleshooting)
+
## Requirements
+
To deploy OpenMetadata, check the Deployment guides.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Qlik Cloud, you need to create a service connection. The service connects Qlik Cloud with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Qlik Cloud** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Qlik Cloud services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Qlik Cloud. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Qlik Cloud Host Port**: This field refers to the base URL of your Qlik Cloud portal. OpenMetadata uses it to generate the redirect links for dashboards and charts. Example: `https://.qlikcloud.com`
+- **Qlik Cloud API Token**: Enter the API token for Qlik Cloud API access. For details on generating this token, see the [Qlik API key documentation](https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/Admin/mc-generate-api-keys.htm). Example: `eyJhbGciOiJFU***`.
+- **Qlik Cloud Space Types**: Select the relevant Qlik Cloud space types to filter the dashboards ingested into the platform. Example: `Personal`, `Shared`, `Managed`, `Data`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
-- **Qlik Cloud Host Port**: This field refers to the base url of your Qlik Cloud Portal, will be used for generating the redirect links for dashboards and charts. Example: `https://.qlikcloud.com`
-- **Qlik Cloud API Token**: Enter the API token for Qlik Cloud APIs access. Refer to [this](https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/Admin/mc-generate-api-keys.htm) document for more details about. Example: `eyJhbGciOiJFU***`.
-- **Qlik Cloud Space Types**: Select relevant space types of Qlik Cloud to filter the dashboards ingested into the platform. Example: `Personal`, `Shared`, `Managed`, `Data`.
-
-
-
-
-
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense.mdx
index 8c50314e6..31f4cf571 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense.mdx
@@ -4,11 +4,9 @@ description: Connect QlikSense dashboards to OpenMetadata with our comprehensive
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Qlik Sense connector.
-Configure and schedule Metabase metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Qlik Sense metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Enable Security](#securing-qlik-sense-connection-with-ssl-in-openmetadata)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense/troubleshooting)
+
## Requirements
+
To deploy OpenMetadata, check the Deployment guides.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Qlik Sense Base URL**: This field refers to the base url of your Qlik Sense Portal, will be used for generating the redirect links for dashboards and charts. Example: `https://server.domain.com` or `https://server.domain.com/`
-- **Qlik Engine JSON API Websocket URL**: Enter the websocket url of Qlik Sense Engine JSON API. Refer to [this](https://help.qlik.com/en-US/sense-developer/May2023/Subsystems/EngineAPI/Content/Sense_EngineAPI/GettingStarted/connecting-to-engine-api.htm) document for more details about. Example: `wss://server.domain.com:4747` or `wss://server.domain.com[/virtual proxy]`.
-Since we use the Qlik Sense Engine APIs, we need to authenticate to those APIs using certificates generated on Qlik Management Console.
-**Qlik Certificate By Values**: In this approach we provide the content of the certificates to the relevant field.
- - **Client Certificate Value**: This field specifies the value of `client.pem` certificate required for authentication.
- - **Client Key Certificate Value**: This field specifies the value of `client_key.pem` certificate required for authentication.
- - **Root Certificate Value**: This field specifies the value of `root.pem` certificate required for authentication.
- - **Staging Directory Path**: This field specifies the path to temporary staging directory, where the certificates will be stored temporarily during the ingestion process, which will de deleted once the ingestion job is over.
-when you are using this approach make sure you are passing the key in a correct format. If your certificate looks like this:
+
+To ingest metadata from QlikSense, you need to create a service connection. The service connects QlikSense with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **QlikSense** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other QlikSense services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for QlikSense. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Qlik Sense Base URL**: This field refers to the base URL of your Qlik Sense portal. It's used to generate redirect links for dashboards and charts. Example: `https://server.domain.com` or `https://server.domain.com/`
+- **Qlik Engine JSON API Websocket URL**: Enter the WebSocket URL of the Qlik Sense Engine JSON API. For more information, see the [Qlik Engine API connection guide](https://help.qlik.com/en-US/sense-developer/May2023/Subsystems/EngineAPI/Content/Sense_EngineAPI/GettingStarted/connecting-to-engine-api.htm). Example: `wss://server.domain.com:4747` or `wss://server.domain.com[/virtual proxy]`.
+
+The Qlik Sense Engine APIs require authentication using certificates generated on the Qlik Management Console.
+
+- **Qlik Certificate By Values**: This approach provides the content of the certificates in the relevant field.
+ - **Client Certificate Value**: This field specifies the value of the `client.pem` certificate required for authentication.
+ - **Client Key Certificate Value**: This field specifies the value of the `client_key.pem` certificate required for authentication.
+ - **Root Certificate Value**: This field specifies the value of the `root.pem` certificate required for authentication.
+ - **Staging Directory Path**: This field specifies the path to the temporary staging directory, where the certificates are stored temporarily during the ingestion process. The directory is cleared once the ingestion job is over.
+
+ When using this approach, make sure you pass the key in the correct format. If your certificate looks like this:
```
-----BEGIN CERTIFICATE-----
MII..
@@ -63,23 +93,83 @@ You will have to replace new lines with `\n` and the final private key that you
```
-----BEGIN CERTIFICATE-----\nMII..\nMBQ...\nCgU..\n8Lt..\n...\nh+4=\n-----END CERTIFICATE-----\n
```
-**Qlik Certificate By Path**: In this approach we provide the path of the certificates to the certificate stored in the container or environment running the ingestion workflow.
- - **Client Certificate Path**: This field specifies the path of `client.pem` certificate required for authentication.
- - **Client Key Certificate Value**: This field specifies the path of `client_key.pem` certificate required for authentication.
- - **Root Certificate Value**: This field specifies the path of `root.pem` certificate required for authentication.
-**User Directory**: This field specifies the user directory of the user.
-**User ID**: This field specifies the user id of the user.
-**Validate Host Name**: Enable/Disable this field to validate the host name against the provided certificates.
-
-
-
-
-
+- **Qlik Certificate By Path**: This approach provides the path to the certificates stored in the container or environment running the ingestion workflow.
+ - **Client Certificate Path**: This field specifies the path of the `client.pem` certificate required for authentication.
+ - **Client Key Certificate Value**: This field specifies the path of the `client_key.pem` certificate required for authentication.
+ - **Root Certificate Value**: This field specifies the path of the `root.pem` certificate required for authentication.
+ - **User Directory**: This field specifies the user directory of the user.
+ - **User ID**: This field specifies the user ID of the user.
+ - **Validate Host Name**: Enable or disable this field to validate the host name against the provided certificates.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
## Securing Qlik Sense Connection with SSL in OpenMetadata
-To establish secure connections between OpenMetadata and Qlik Sense, there are two ways to communicate: defining the certificate file path or using the certificates value. Navigate to the `Advanced Config` section.
-When using the local certificate file path, ensure that the certificates are accessible from the Airflow Server. You can specify the path for the `client certificate`, `client key certificate`, and `root certificate`.
-Alternatively, when using the certificates value, you can provide the CA certificate used for SSL validation by specifying the `CA Certificate`. If both client and server require mutual authentication, you can upload all three: `CA Certificate`, `SSL Certificate`, and `SSL Key`.
-Refer to the guide on how to generate authentication certificates so that OpenMetadata can communicate with Qlik Sense [here](/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense/certificates).
+
+OpenMetadata supports two ways to establish secure connections with Qlik Sense: defining the certificate file path or providing the certificate values directly. Navigate to the `Advanced Config` section.
+
+When using the local certificate file path, ensure that the certificates are accessible from the Airflow Server. Specify the path for the `client certificate`, `client key certificate`, and `root certificate`.
+
+Alternatively, when using the certificate values, provide the CA certificate used for SSL validation in the `CA Certificate` field. If both client and server require mutual authentication, upload all three: `CA Certificate`, `SSL Certificate`, and `SSL Key`.
+
+For more information about generating authentication certificates so that OpenMetadata can communicate with Qlik Sense, see the [certificate generation guide](/v2.0.x-SNAPSHOT/connectors/dashboard/qliksense/certificates).
+
- 
-
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/quicksight.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/quicksight.mdx
index 5657823e3..1829fa58c 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/quicksight.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/quicksight.mdx
@@ -4,11 +4,9 @@ description: Connect Amazon QuickSight to OpenMetadata with our comprehensive da
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the QuickSight connector.
-Configure and schedule QuickSight metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule QuickSight metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
-- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/quicksight)
+- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/quicksight/troubleshooting)
+
## Requirements
-To execute metadata extraction and usage workflow successfully the IAM User should have enough access to fetch required data. Following table describes the minimum required permissions
+
+To execute metadata extraction and usage workflow successfully the IAM User should have enough access to fetch required data. The following table describes the minimum required permissions:
+
| # | AWS QuickSight Permission |
| :---------- | :---------- |
| 1 | DescribeDashboard |
@@ -34,12 +34,14 @@ To execute metadata extraction and usage workflow successfully the IAM User shou
| 6 | DescribeDataSet |
| 7 | ListDataSets |
| 8 | DescribeDataSource |
+
Here is how to add Permissions to an IAM user.
-- Navigate to the IAM console in the AWS Management Console.
-- Choose the IAM user or group to which you want to attach the policy, and click on the "Permissions" tab.
-- Click on the "Add permissions" button and select "Attach existing policies directly".
-- Search for the policy by name or by filtering the available policies, and select the one you want to attach.
-- Review the policy and click on "Add permissions" to complete the process.
+1. Navigate to the IAM console in the AWS Management Console.
+2. Choose the IAM user or group to which you want to attach the policy, and click on the "Permissions" tab.
+3. Click on the "Add permissions" button and select "Attach existing policies directly".
+4. Search for the policy by name or by filtering the available policies, and select the one you want to attach.
+5. Review the policy and click on "Add permissions" to complete the process.
+
```json
{
"Version": "2012-10-17",
@@ -61,66 +63,151 @@ Here is how to add Permissions to an IAM user.
]
}
```
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
+
+To ingest metadata from QuickSight, you need to create a service connection. The service connects QuickSight with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **QuickSight** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other QuickSight services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for QuickSight. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+For more information, see [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
-Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
-services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
-- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
- and AWS Secrets Access Key. Also, these will include an AWS Session Token.
-You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
+As AWS can have instances in multiple regions, OpenMetadata needs to know the region of the service you want to reach.
+The AWS Region is the only required parameter when configuring a connection. When connecting to
+services programmatically, OpenMetadata can extract and use the rest of the AWS configuration in different ways.
+For more information, see [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, provide the AWS Access Key ID
+ and AWS Secret Access Key. Temporary credentials also include an AWS Session Token.
+For more information, see [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
-When using Assume Role authentication, ensure you provide the following details:
+**Tip**: When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
+
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
-By default, we'll use the name `OpenMetadataSession`.
-Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
+By default, OpenMetadata uses the name `OpenMetadataSession`.
+For more information, see the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
- **identityType**: The authentication method that the user uses to sign in.
- **awsAccountId**: QuickSight account ID is required to manage QuickSight users, data sources, and reports.
-- **namespace**: The Amazon QuickSight namespace that contains the dashboard IDs in this request ( To be provided when identityType is `ANONYMOUS` )
-
-
-
-
-
+- **namespace**: The Amazon QuickSight namespace that contains the dashboard IDs in this request. Provide this when identityType is `ANONYMOUS`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/redash.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/redash.mdx
index 8fe000e73..a351d7f75 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/redash.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/redash.mdx
@@ -4,11 +4,9 @@ description: Connect Redash dashboards to OpenMetadata Easily. Complete setup gu
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Redash connector.
-Configure and schedule Redash metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Redash metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/redash/troubleshooting)
+
## Requirements
-We connect to Redash through the [API](https://redash.io/help/user-guide/integrations-and-api/api) endpoint, so the user
-we use in the configuration to ingest data must have enough permissions to view all the data. For more info about the
-permissions, please visit Redash documentation [here](https://redash.io/help/user-guide/users/permissions-groups).
+
+OpenMetadata connects to Redash through the [API](https://redash.io/help/user-guide/integrations-and-api/api) endpoint, so the user
+configured for ingestion must have enough permissions to view all the data. For more information, see the
+[Redash permissions documentation](https://redash.io/help/user-guide/users/permissions-groups).
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Redash, you need to create a service connection. The service connects Redash with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Redash** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Redash services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Redash. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host and Port**: URL to the Redash instance.
+- **Username**: Specify the user account used to connect to Redash. It should have enough privileges to read all the metadata.
+- **API Key**: API key of the Redash instance to access. It has the same permissions as the user who owns it. It appears on the user's profile page.
+- **Redash Version**: Redash version of your Redash instance. Enter the numerical value from the [Redash Releases](https://github.com/getredash/redash/releases) page. Default: `10.0.0`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
-- **Host and Port**: URL to the Redash instance.
-- **Username**: Specify the User to connect to Redash. It should have enough privileges to read all the metadata.
-- **API Key**: API key of the redash instance to access. It has the same permissions as the user who owns it. Can be found on a user profile page.
-- **Redash Version**: Redash version of your redash instance. Enter the numerical value from the [Redash Releases](https://github.com/getredash/redash/releases) page. Default: `10.0.0`.
-
-
-
-
-
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/sigma.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/sigma.mdx
index 303cf1ed1..67039abba 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/sigma.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/sigma.mdx
@@ -4,11 +4,9 @@ description: Connect Sigma Analytics to OpenMetadata seamlessly. Learn how to co
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Sigma connector.
-Configure and schedule Sigma metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Sigma metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/sigma/troubleshooting)
+
## Requirements
-OpenMetadata relies on Sigma's REST API. To know more you can read the [Sigma API Get Started docs](https://help.sigmacomputing.com/reference/get-started-sigma-api#about-the-api). To [generate API client credentials](https://help.sigmacomputing.com/reference/generate-client-credentials#user-requirements), you must be assigned the Admin account type.
+
+OpenMetadata relies on Sigma's REST API. For more information, see the [Sigma API Get Started docs](https://help.sigmacomputing.com/reference/get-started-sigma-api#about-the-api). To [generate API client credentials](https://help.sigmacomputing.com/reference/generate-client-credentials#user-requirements), you must be assigned the Admin account type.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Sigma, you need to create a service connection. The service connects Sigma with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Sigma** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Sigma services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Sigma. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **hostPort**: Host and port for the Sigma REST API.
+The hostPort parameter specifies the host and port of the Sigma API request URL. This should be specified as a string in the format `https://aws-api.sigmacomputing.com`. The Sigma API request URL varies according to the Sigma cloud. For more information, see [identifying your API request URL](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url).
+- **clientId**: Client ID for the Sigma REST API.
+Get the Client ID and Client Secret by following the steps below:
+1. Navigate to your Sigma homepage.
+2. Click on Administration in the lower left corner.
+3. Click on Developer Access on the left side.
+4. To generate a new Client Id and client Secret, on the upper left corner click `Create New`.
+5. Enter the required details asked and click `Create`.
+6. Copy the generated access token and password.
+For more information, see [generating API client credentials](https://help.sigmacomputing.com/reference/generate-client-credentials#generate-api-client-credentials).
+- **clientSecret**: Client Secret for the Sigma REST API.
+Copy the access token password from the step above where the new token is generated.
+For more information, see [generating API client credentials](https://help.sigmacomputing.com/reference/generate-client-credentials#generate-api-client-credentials).
+- **apiVersion**: Sigma REST API version. Defaults to `v2`.
+To find the Sigma REST API version, see [identifying your API request URL](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url) and look at the `Token URL` section.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
-**hostPort**: Host and Port Sigma REST API.
-The hostPort parameter specifies the host and port of the Sigma's API request URL. This should be specified as a string in the format `https://aws-api.sigmacomputing.com`. Sigma's API request URL varies according to the sigma cloud. you can determine your API url by following the docs [here](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url)
-**clientId**: Client Id for Sigma REST API.
-Get the Client Id and client Secret by following below steps:
-- Navigate to your Sigma homepage.
-- Click on Administration in the lower left corner.
-- Click on Developer Access on the left side.
-- To generate a new Client Id and client Secret, On upper left corner click `Create New`.
-- Enter the required details asked and click `Create`.
-- Copy the generated access token and password.
-For detailed information visit [here](https://help.sigmacomputing.com/reference/generate-client-credentials#generate-api-client-credentials).
-**clientSecret**: Client Secret for Sigma REST API.
-Copy the access token password from the step above where a new token is generated.
-For detailed information visit [here](https://help.sigmacomputing.com/reference/generate-client-credentials#generate-api-client-credentials).
-**apiVersion**: Sigma REST API Version.
-Version of the Sigma REST API by default `v2`.
-To get to know the Sigma REST API Version visit [here](https://help.sigmacomputing.com/reference/get-started-sigma-api#identify-your-api-request-url) and look into the `Token URL` section.
-
-
-
-
-
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/ssrs.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/ssrs.mdx
index 2e8a91f62..568fbea8a 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/ssrs.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/ssrs.mdx
@@ -5,10 +5,8 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Security** > **New Role Assignment**.
3. Add the service account and assign the **Browser** role.
## Metadata Ingestion
-
+To ingest metadata from SSRS, you need to create a service connection. The service connects SSRS with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
-# Connection Details
+### Step 1: Add New Service
-
-
+1. Navigate to **Settings** > **Services**.
-- **Host and Port**: The base URL of your SSRS Report Server, e.g., `https://your-ssrs-server/reportserver`. The connector appends `/api/v2.0` to this URL when calling the REST API.
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **SSRS** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SSRS services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SSRS. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host and Port**: The base URL of your SSRS Report Server, for example, `https://your-ssrs-server/reportserver`. The connector appends `/api/v2.0` to this URL when calling the REST API.
- **Username**: The Windows account username for NTLM authentication. Use `DOMAIN\username` format for domain accounts.
@@ -65,12 +95,64 @@ To grant the required role:
- **Project Filter Pattern** *(optional)*: Regex to include or exclude SSRS folders (which map to OpenMetadata projects) by name.
-
-
-
-
-
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
## Securing SSRS Connection with SSL in OpenMetadata
To establish secure connections between OpenMetadata and your SSRS server, set `verifySSL` to `validate` and provide the CA certificate in the SSL Configuration section. This ensures the connector validates the server's certificate during ingestion.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/superset.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/superset.mdx
index 574ac0e6b..e8d17da6d 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/superset.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/superset.mdx
@@ -4,12 +4,9 @@ description: Learn how to connect Apache Superset dashboards to OpenMetadata wit
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Superset connector.
-Configure and schedule Superset metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Superset metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Enable Security](#securing-superset-connection-with-ssl-in-openmetadata)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/superset/troubleshooting)
+
## Requirements
-The ingestion also works with Superset 2.0.0 🎉
-**API Connection**: To extract metadata from Superset via API, user must have at least `can read on Chart` & `can read on Dashboard` permissions.
-**Database Connection**: To extract metadata from Superset via MySQL or Postgres database, database user must have at least `SELECT` privilege on `dashboards` & `slices` tables within superset schema.
+
+The ingestion also works with Superset 2.0.0.
+
+**API Connection**: To extract metadata from Superset via API, the user must have at least `can read on Chart` and `can read on Dashboard` permissions.
+
+**Database Connection**: To extract metadata from Superset via MySQL or Postgres database, the database user must have at least `SELECT` privilege on the `dashboards` and `slices` tables within the Superset schema.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Host and Port**: The `Host and Post` parameter is common for all three modes of authentication which specifies the host and port of the Superset instance. This should be specified as a string in the format `http://hostname:port` or `https://hostname:port`. For example, you might set the hostPort parameter to `https://org.superset.com:8088`.
-- **Superset Connection**: Add the connection details to fetch metadata from Superset either through APIs or Database.
+
+To ingest metadata from Superset, you need to create a service connection. The service connects Superset with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Superset** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Superset services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Superset. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host and Port**: The `Host and Port` parameter is common to all three authentication modes and specifies the host and port of the Superset instance. This should be specified as a string in the format `http://hostname:port` or `https://hostname:port`. For example, you might set the hostPort parameter to `https://org.superset.com:8088`.
+- **Superset Connection**: Add the connection details to fetch metadata from Superset either through APIs or a database.
+
##### For Superset API Connection
-Superset API connection is the default mode of authentication where we fetch the metadata using [Superset APIs](https://superset.apache.org/docs/api/).
+
+Superset API connection is the default authentication mode. It fetches metadata using the [Superset APIs](https://superset.apache.org/docs/api/).
+
-Superset only supports basic or ldap authentication through APIs so if you have SSO enabled on your Superset instance then this mode of authentication will not work for you and you can opt for MySQL or Postgres Connection to fetch metadata directly from the database in the backend of Superset.
+Superset only supports basic or LDAP authentication through APIs. If you have SSO enabled on your Superset instance, this authentication mode won't work. Use a MySQL or Postgres connection instead to fetch metadata directly from the Superset database.
+
- **Username**: Username to connect to Superset, for ex. `user@organization.com`. This user should have access to relevant dashboards and charts in Superset to fetch the metadata.
- **Password**: Password of the user account to connect with Superset.
- **Provider**: Choose between `db`(default) or `ldap` mode of Authentication provider for the Superset service. This parameter is used internally to connect to Superset's REST API.
@@ -57,37 +90,108 @@ Possible values:
* `validate`: Validate the certificate using the public certificate (recommended).
* `ignore`: Ignore the certification validation (not recommended for production).
* `no-ssl`: SSL validation is not needed.
-- **SSL Config**: Client SSL configuration in case we are connection to a host with SSL enabled.
-- **Certificate Path**: CA certificate path in the instance where the ingestion run. E.g., `/path/to/public.cert`. Will be used if Verify SSL is set to `validate`.
-For MySQL Connection
-You can use Mysql Connection when you have SSO enabled and your Superset is backed by Mysql database.
-- **Username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata. Make sure the user has select privileges on `dashboards`, `tables` & `slices` tables of superset schema.
+- **SSL Config**: Client SSL configuration in case you are connecting to a host with SSL enabled.
+- **Certificate Path**: CA certificate path in the instance where the ingestion run. For example, `/path/to/public.cert`. Will be used if Verify SSL is set to `validate`.
+
+##### For MySQL Connection
+
+Use a MySQL connection when you have SSO enabled and your Superset instance is backed by a MySQL database.
+
+- **Username**: Specify the user to connect to MySQL. It should have enough privileges to read all the metadata. Make sure the user has select privileges on the `dashboards`, `tables`, and `slices` tables of the Superset schema.
- **Password**: Password to connect to MySQL.
- **Host and Port**: Enter the fully qualified hostname and port number for your MySQL deployment in the Host and Port field.
- **databaseSchema**: Enter the database schema which is associated with the Superset instance.
- **caCertificate**: Provide the path to ssl ca file.
- **sslCertificate**: Provide the path to ssl client certificate file (ssl_cert).
- **sslKey**: Provide the path to ssl client certificate file (ssl_key).
-
-
-
-You can use Postgres Connection when you have SSO enabled and your Superset is backed by Postgres database.
-- **Username**: Specify the User to connect to Postgres. Make sure the user has select privileges on `dashboards`, `tables` & `slices` tables of superset schema.
+
+##### For Postgres Connection
+
+Use a Postgres connection when you have SSO enabled and your Superset instance is backed by a Postgres database.
+
+- **Username**: Specify the user to connect to Postgres. Make sure the user has select privileges on the `dashboards`, `tables`, and `slices` tables of the Superset schema.
- **Password**: Password to connect to Postgres.
- **Host and Port**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field.
-- **Database**: Initial Postgres database to connect to. Specify the name of database associated with Superset instance.
+- **Database**: Initial Postgres database to connect to. Specify the name of the database associated with the Superset instance.
- **caCertificate**: Provide the path to ssl ca file.
-
-
-
-
-
-
-## Securing Superset Connection with SSL in OpenMetadata
-1. To establish secure connections between OpenMetadata and Superset, navigate to the `Advanced Config` section. We need to update the `Certificate Path` and ensure that the certificates are accessible from the Airflow Server.
- 
-2. To establish secure connections between OpenMetadata and Superset's MySQL database, you need to configure SSL certificates appropriately. If you only require SSL validation, specify the `caCertificate` to use the CA certificate for validating the server's certificate. For mutual authentication, where both client and server need to authenticate each other, you must provide all three parameters: `ssl_key` for the client’s private key, `ssl_cert` for the client’s SSL certificate, and `ssl_ca` for the CA certificate to validate the server’s certificate.
-
-3. To establish secure connections between OpenMetadata and Superset's PostgreSQL database, you can configure SSL using different SSL modes provided by PostgreSQL, each offering varying levels of security.Under `PostgresConnection Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that PostgreSQL requires only the CA certificate for SSL validation.
-
+
+#### Advanced Configuration
+
+Database Services include an Advanced Configuration section for passing extra arguments to the connector and, if needed, changing the connection scheme. This is only needed for advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+## Securing Superset Connection with SSL in OpenMetadata
+
+1. To establish secure connections between OpenMetadata and Superset, navigate to the `Advanced Config` section. Update the `Certificate Path` and ensure that the certificates are accessible from the Airflow Server.
+
+
+
+2. To establish secure connections between OpenMetadata and Superset's MySQL database, configure SSL certificates appropriately. If you only require SSL validation, specify the `caCertificate` to use the CA certificate for validating the server's certificate. For mutual authentication, where both client and server need to authenticate each other, provide all three parameters: `ssl_key` for the client's private key, `ssl_cert` for the client's SSL certificate, and `ssl_ca` for the CA certificate to validate the server's certificate.
+
+
+
+3. To establish secure connections between OpenMetadata and Superset's PostgreSQL database, configure SSL using different SSL modes provided by PostgreSQL, each offering varying levels of security. Under `PostgresConnection Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). PostgreSQL requires only the CA certificate for SSL validation.
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/dashboard/tableau.mdx b/v2.0.x-SNAPSHOT/connectors/dashboard/tableau.mdx
index 931da5bb2..9fcfb0d7d 100644
--- a/v2.0.x-SNAPSHOT/connectors/dashboard/tableau.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/dashboard/tableau.mdx
@@ -5,11 +5,9 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import ConfigureIngestion from '/snippets/connectors/dashboard/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDashboard from '/snippets/connectors/metadata-agent/dashboard.mdx'
import DashboardLineage from '/snippets/connectors/dashboard/dashboard-lineage.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-Configure and schedule Tableau metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Tableau metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Entity Mapping](#entity-mapping)
- [Metadata Ingestion](#metadata-ingestion)
- [Enable Security](#securing-tableau-connection-with-ssl-in-openmetadata)
-- [Lineage](#lineage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/dashboard/tableau/troubleshooting)
## Requirements
-To ingest tableau metadata, minimum `Site Role: Viewer` is required for the tableau user.
-To create lineage between tableau dashboard and any database service via the queries provided from Tableau Metadata API, please enable the Tableau Metadata API for your tableau server.
-For more information on enabling the Tableau Metadata APIs follow the link [here](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html)
+
+To ingest Tableau metadata, minimum `Site Role: Viewer` is required for the Tableau user.
+To create lineage between Tableau dashboards and any database service via the queries provided from Tableau Metadata API, enable the Tableau Metadata API for your Tableau server.
+For more information, see [Get Started with the Metadata API](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html).
+
- If using a **default site** on Tableau Server, leave the **Site URL** and **Site Name** fields **blank** in the ingestion configuration.
- Ensure that the **Metadata API** is enabled for the user performing the ingestion. If it is not enabled, ingestion may fail. Follow the official Tableau documentation to [enable the Metadata API](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html#enable-the-tableau-metadata-api-for-tableau-server).
- The minimum required role to retrieve owners is Site Admin Explorer.
+
- As of OpenMetadata versions `1.7.4` and `1.7.5`, the `siteUrl` field has been removed from the Tableau connector configuration. This change was intentional, as confirmed in the release commit.
- To connect to a non-default Tableau site, use the `siteName` field instead. The Tableau Python SDK does not require `siteUrl` for authentication.
- Ensure the `siteName` field is correctly populated (do not use `*`) to enable successful metadata ingestion for multi-site Tableau environments.
+
### Source Tables and Lineage
+Lineage depends on Tableau exposing the source tables behind each data model.
+
**Table lineage requires** the ingestion account to have the **View** capability on the external assets in Tableau Catalog, granted directly or derived. For any asset it cannot view, Tableau returns the source table with its table and database names removed, keeping only internal identifiers, and reports no error. Those tables are skipped without a message, so **no lineage is created between them and the data models that use them**. Assets the account can view are unaffected, and dashboards, charts and data models are still ingested either way.
@@ -56,14 +59,18 @@ For more information on enabling the Tableau Metadata APIs follow the link [here
Grant the ingestion account **View** on the affected external assets in Tableau Catalog, or use an account that already holds it. A Viewer or Explorer site role alone is not enough for OpenMetadata to build table lineage. For the steps, see [Manage Permissions for External Assets](https://help.tableau.com/current/server/en-us/dm_perms_assets.htm) in the Tableau documentation.
## Entity Mapping
+
The Tableau connector maps Tableau assets to OpenMetadata entities as follows:
+
| Tableau Asset | OpenMetadata Entity | Description |
|---|---|---|
| **Workbooks** | **Dashboard** | Tableau workbooks are mapped to OpenMetadata Dashboard entities. |
| **Views** (worksheets, dashboard views, stories — all sheet types) | **Chart** | All Tableau views are mapped to OpenMetadata Chart entities, regardless of their sheet type. |
| **Data Sources** (embedded and published) | **Data Models** | Tableau data sources are mapped to OpenMetadata Data Model entities. |
| **Projects** | **Project field** | The Tableau project name is stored in the project field of the Dashboard. |
+
### Example Structure
+
**Tableau Structure:**
```
Project: Sales Team
@@ -72,6 +79,7 @@ Project: Sales Team
├── Worksheet: Monthly Sales (sheetType="worksheet")
└── Story: Sales Story (sheetType="story")
```
+
**OpenMetadata Structure:**
```
Dashboard: "Sales Analysis" (Project: "Sales Team")
@@ -79,60 +87,161 @@ Dashboard: "Sales Analysis" (Project: "Sales Team")
├── Chart: "Monthly Sales"
└── Chart: "Sales Story"
```
+
This mapping ensures that:
-- Tableau workbooks appear as OpenMetadata dashboards
-- All views (worksheets, dashboard views, stories) appear as charts under that dashboard
-- Tableau project names are preserved in the project field
-- URLs point directly to the specific views
+- Tableau workbooks appear as OpenMetadata dashboards.
+- All views (worksheets, dashboard views, stories) appear as charts under that dashboard.
+- Tableau project names are preserved in the project field.
+- URLs point directly to the specific views.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Tableau, you need to create a service connection. The service connects Tableau with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Dashboard Services**, then click the **Tableau** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Tableau services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Tableau. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: URL or IP address of your installation of Tableau Server.
- **Authentication Types**:
1. Basic Authentication
- Username: The name of the user whose credentials will be used to sign in.
- Password: The password of the user.
2. Access Token Authentication
- - Personal Access Token: The personal access token name. For more information to get a Personal Access Token please visit this [link](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm).
- - Personal Access Token Secret: The personal access token value. For more information to get a Personal Access Token please visit this [link](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm).
-- **API Version**: Tableau API version. If not provided, the version will be used from the Tableau server automatically. A list of versions can be found [here](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_versions.htm).
+ - Personal Access Token: The personal access token name. For more information, see [Personal Access Tokens](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm).
+ - Personal Access Token Secret: The personal access token value. For more information, see [Personal Access Tokens](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm).
+- **API Version**: Tableau API version. If not provided, the version will be used from the Tableau server automatically. For a list of Tableau Server versions and their corresponding REST API and REST API schema versions, see [REST API Versions](https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_versions.htm).
- **Site Name**: This corresponds to the `contentUrl` attribute in the Tableau REST API. The `site_name` is the portion of the URL that follows the `/site/` in the URL.
- **Pagination Limit**: The pagination limit will be used while querying the Tableau Graphql endpoint to get the data source information.
- **Proxy URL**: Optional proxy URL for the Tableau server. If not provided, the hostPort will be used. This is used to generate the dashboard and chart URLs.
+
### Site Name
+
+The Site Name field behaves differently depending on whether you're connecting to Tableau Cloud or Tableau Server.
+
#### 1. Service Connection for Tableau Cloud
+
If you're connecting to a cloud Tableau instance, add the `Site Name` with your site name.
-#### 2. Service Connection for a default tableau site
-For a default tableau site the `Site Name` field should be kept empty.
-#### 3. Service Connection for a non-default tableau site
-For a non-default tableau site the `Site Name` field is required.
+
+#### 2. Service Connection for a Default Tableau Site
+
+For a default Tableau site, the `Site Name` field should be kept empty.
+
+#### 3. Service Connection for a Non-Default Tableau Site
+
+For a non-default Tableau site, the `Site Name` field is required.
+
+
+
-If `https://xxx.tableau.com/#/site/MarketingTeam/home` represents the homepage url for your tableau site, the `MarketingTeam` from the url should be entered in the `Site Name` field.
+If `https://xxx.tableau.com/#/site/MarketingTeam/home` represents the homepage url for your Tableau site, the `MarketingTeam` from the url should be entered in the `Site Name` field.
+
### Authentication Type
-### 1. Basic Authentication
-We need the name of the user whose credentials will be used to sign in and the password of the user.
-### 2. Access Token Authentication
-In this case, the personal access token name and the personal access token value are required.
-For more information to get a Personal Access Token please visit this [link](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm).
-
-
-
-
-
-## Securing Tableau Connection with SSL in OpenMetadata
-To establish secure connections between OpenMetadata and Tableau, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you’ll need to use all three parameters: `ssl_key`, `ssl_cert`, and `caCertificate`. In this case, `ssl_cert` is used for the client’s SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `caCertificate` for the CA certificate to validate the server’s certificate.
-
+
+Tableau supports two authentication methods for the connector.
+
+#### 1. Basic Authentication
+
+Basic authentication requires the username and password of the user signing in.
+
+
+
+#### 2. Access Token Authentication
+
+Access token authentication requires the personal access token name and the personal access token value.
+For more information, see [Personal Access Tokens](https://help.tableau.com/current/server/en-us/security_personal_access_tokens.htm).
+
+
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dashboard service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all dashboards, charts, and data models available in the source.
+
+
+**Filter Options**
+
+The Dashboard, Chart, and Data Model sections each include the following filter options:
+
+- **Dashboard**: Controls which dashboards OpenMetadata ingests from the source.
+- **Chart**: Controls which charts within the ingested dashboards are included.
+- **Data Model**: Controls which data models are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
@@ -142,3 +251,10 @@ Make sure the **Tableau Metadata API** is enabled on your Tableau server. OpenMe
- To enable the Tableau Metadata API, see [Enable the Tableau Metadata API for Tableau Server](https://help.tableau.com/current/api/metadata_api/en-us/docs/meta_api_start.html#enable-the-tableau-metadata-api-for-tableau-server).
+## Securing Tableau Connection with SSL in OpenMetadata
+
+To establish secure connections between OpenMetadata and Tableau, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `caCertificate`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `caCertificate` for the CA certificate to validate the server's certificate.
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/adls-datalake.mdx b/v2.0.x-SNAPSHOT/connectors/database/adls-datalake.mdx
index 2465f89b0..449e02553 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/adls-datalake.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/adls-datalake.mdx
@@ -4,11 +4,7 @@ description: Connect your Azure Data Lake Storage to OpenMetadata with our ADLS
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-The ADLS Datalake connector supports extracting metadata from file types `JSON`, `CSV`, `TSV` & `Parquet`.
+The ADLS Datalake connector supports extracting metadata from file types `JSON`, `CSV`, `TSV`, and `Parquet`.
+
### ADLS Permissions
-To extract metadata from Azure ADLS (Storage Account - StorageV2), you will need an **App Registration** with the following
-permissions on the Storage Account:
+
+To extract metadata from Azure ADLS (a StorageV2 storage account), you will need an **App Registration** with the following permissions on the Storage Account:
- Storage Blob Data Reader
- Storage Queue Data Reader
+
## Metadata Ingestion
-
-## Connection Details
-
-
+
+To ingest metadata from ADLS Datalake, you need to create a service connection. The service connects ADLS Datalake with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **ADLS Datalake** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other ADLS Datalake services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for ADLS Datalake. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Azure Credentials**
- - **Client ID** : Client ID of the data storage account
- - **Client Secret** : Client Secret of the account
- - **Tenant ID** : Tenant ID under which the data storage account falls
- - **Account Name** : Account Name of the data Storage
+ - **Client ID**: Client ID of the data storage account
+ - **Client Secret**: Client Secret of the account
+ - **Tenant ID**: Tenant ID under which the data storage account falls
+ - **Account Name**: Account Name of the data storage account
- **Required Roles**
- Please make sure the following roles associated with the data storage account.
+ Make sure the following roles are associated with the data storage account.
- `Storage Blob Data Reader`
- `Storage Queue Data Reader`
-- The current approach for authentication is based on `app registration`, reach out to us on [slack](https://slack.open-metadata.org/) if you find the need for another auth system
-
-
-
-
-
-
+- The current approach for authentication is based on `app registration`. Reach out on [Slack](https://slack.open-metadata.org/) if you find the need for another authentication method.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Options (Optional)**: Enter any additional connection options that can be sent to the database during the connection. These details must be added as key-value pairs.
+- **Connection Arguments (Optional)**: Enter any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as key-value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/athena.mdx b/v2.0.x-SNAPSHOT/connectors/database/athena.mdx
index 147c171a2..e84297d93 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/athena.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/athena.mdx
@@ -4,12 +4,8 @@ sidebarTitle: Overview
description: Connect Amazon Athena to OpenMetadata effortlessly. Complete setup guide, configuration steps, and best practices for database metadata ingestion.
---
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
If you have external services other than glue and facing permission issues, add the permissions to the list above.
+
## Metadata Ingestion
-
-## Connection Details
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
+
+To ingest metadata from Athena, you need to create a service connection. The service connects Athena with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Athena** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Athena services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Athena. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+You can find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
@@ -159,11 +188,11 @@ You can find more information on [Using temporary credentials with AWS resources
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
@@ -182,11 +211,69 @@ Find more information about the [Role Session Name](https://docs.aws.amazon.com/
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
- **S3 Staging Directory**: The S3 staging directory is an optional parameter. Enter a staging directory to override the default staging directory for AWS Athena.
- **Athena Workgroup**: The Athena workgroup is an optional parameter. If you wish to have your Athena connection related to an existing AWS workgroup add your workgroup name here.
-
-
-
-
-
-
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection Scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/azuresql.mdx b/v2.0.x-SNAPSHOT/connectors/database/azuresql.mdx
index fa27403aa..64a599ff0 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/azuresql.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/azuresql.mdx
@@ -4,11 +4,7 @@ description: Learn how to connect Azure SQL OpenMetadata to OpenMetadata with st
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Options
-
-
+
+To ingest metadata from AzureSQL, you need to create a service connection. The service connects AzureSQL with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **AzureSQL** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other AzureSQL services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for AzureSQL. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to AzureSQL. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to AzureSQL.
- **Host and Port**: Enter the fully qualified hostname and port number for your AzureSQL deployment in the Host and Port field.
- **Database**: The database of the data source is an optional parameter, if you would like to restrict the metadata reading to a single database. If left blank, OpenMetadata ingestion attempts to scan all the databases.
- **Driver**: Connecting to AzureSQL requires ODBC driver to be installed. Specify ODBC driver name in the field.
You can download the ODBC driver from [here](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16). In case of docker or kubernetes deployment this driver comes out of the box with version `ODBC Driver 18 for SQL Server`.
-**Authentication Mode**:
+
+#### Authentication Mode
- **Authentication**:
- The `authentication` parameter determines the method of authentication when connecting to AzureSQL using ODBC (Open Database Connectivity).
- If you select **"Active Directory Password"**, you'll need to provide the password associated with your Azure Active Directory account.
@@ -63,10 +92,69 @@ You can download the ODBC driver from [here](https://learn.microsoft.com/en-us/s
- **Connection Timeout**:
- The `connectionTimeout` parameter specifies the maximum time (in seconds) that your application will wait while attempting to establish a connection to AzureSQL.
- If the connection cannot be established within this timeframe, an error will be raised.
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection Scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/bigquery.mdx b/v2.0.x-SNAPSHOT/connectors/database/bigquery.mdx
index 88fd3984d..a9853177d 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/bigquery.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/bigquery.mdx
@@ -5,13 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
import PartitionedTables from '/snippets/connectors/database/partitioned-tables.mdx'
-
-import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Supported Authentication Types:**
-- **GCP Credentials Values** — Direct service account key values (type, project ID, private key, client email, etc.)
+- **GCP Credentials Values** — Direct service account key values (type, project ID, private key, client email, and so on)
- **GCP Credentials Path** — Path to a local service account key JSON file
- **GCP External Account** — Workload Identity Federation for non-GCP environments
- **GCP Application Default Credentials (ADC)** — Automatic credential discovery in GCP environments or via `gcloud auth application-default login`
@@ -33,24 +27,15 @@ In this section, we provide guides and references to use the BigQuery connector.
Configure and schedule BigQuery metadata and profiler workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- - [Incremental Extraction](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/metadata/incremental-extraction/bigquery)
-- [Query Usage](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage)
-- [Data Profiler](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow)
-- [Data Quality](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality/configure)
-- [Lineage](/v2.0.x-SNAPSHOT/connectors/ingestion/lineage)
-- [dbt Integration](/v2.0.x-SNAPSHOT/connectors/database/dbt)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/bigquery/troubleshooting)
-
## Requirements
-You need to create an service account in order to ingest metadata from bigquery refer [this](/v2.0.x-SNAPSHOT/connectors/database/bigquery/create-credentials) guide on how to create service account.
-
-
- Check out this documentation on how to create a custom role and assign it to the service account.
-
-
+You need to create an service account in order to ingest metadata from bigquery refer. For more information about how to create a service account, see [Create Custom GCP Role](/v2.0.x-SNAPSHOT/connectors/database/bigquery/create-credentials).
+
+
### Data Catalog API Permissions
+
- Enable the `datacatalog.googleapis.com` service in your GCP project. See the [Data Catalog API reference](https://docs.cloud.google.com/data-catalog/docs/reference/rest).
- Select the `GCP Project ID` that you want to enable the `Data Catalog API` on.
- Click on `Enable API` which will enable the data catalog api on the respective project.
@@ -84,16 +69,54 @@ You can learn more about the benefits of partitioned tables [here](https://cloud
If you want to convert your existing sharded tables to partitioned tables, you can follow the steps in this [guide](https://cloud.google.com/bigquery/docs/creating-partitioned-tables#convert-date-sharded-tables).
This will help you simplify your data management and optimize your performance in BigQuery.
+
## Metadata Ingestion
-
-### Connection Details
-
-
+To ingest metadata from BigQuery, you need to create a service connection. The service connects BigQuery with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **BigQuery** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other BigQuery services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for BigQuery. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: The ingestion runtime uses the BigQuery API endpoint `bigquery.googleapis.com`. Custom `hostPort` values are not currently applied.
-- **Billing Project ID** (Optional): The GCP project that should be charged for the BigQuery jobs OpenMetadata runs. Use this when the project that pays for metadata, usage, or lineage queries is different from the project ID or IDs you ingest from.
-- **GCP Credentials**: You can authenticate with your bigquery instance using either `GCP Credentials Path` where you can specify the file path of the service account key or you can pass the values directly by choosing the `GCP Credentials Values` from the service account key file. You can check out [this](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) documentation on how to create the service account keys and download it.
+- **GCP Credentials**:
+You can authenticate with your bigquery instance using either `GCP Credentials Path` where you can specify the file path of the service account key or you can pass the values directly by choosing the `GCP Credentials Values` from the service account key file.
+You can check out [this](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) documentation on how to create the service account keys and download it.
- **GCP Credentials Values**: Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery:
- **Credentials type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
+ - **Billing Project ID (Optional)**: The GCP project that should be charged for the BigQuery jobs OpenMetadata runs. Use this when the project that pays for metadata, usage, or lineage queries is different from the project ID or IDs you ingest from.
- **Project ID**: The BigQuery project ID or IDs that OpenMetadata should scan for datasets, tables, and other metadata. For service account credentials, this usually comes from the `project_id` value in the key file. You can also pass multiple project IDs to ingest metadata from different BigQuery projects into one service.
- **Private Key ID**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
- **Private Key**: This is the private key associated with the service account that is used to authenticate and authorize access to BigQuery. To fetch this key, look for the value associated with the `private_key` key in the service account file.
@@ -104,41 +127,112 @@ This will help you simplify your data management and optimize your performance i
- **Authentication Provider X509 Certificate URL**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
- **Client X509Cert URL**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
- **GCP Credentials Path**: Passing a local file path that contains the credentials.
- - **GCP Impersonate Service Account Configuration** (Optional): Enable the authenticated service account to impersonate another service account, instead of ingesting directly with the credentials above.
+ - **GCP Impersonate Service Account Configuration (Optional)**: Enable the authenticated service account to impersonate another service account, instead of ingesting directly with the credentials above.
- **Target Service Account Email**: The email of the service account to impersonate.
- **Lifetime**: Number of seconds the delegated credential should remain valid. Defaults to `3600`.
-
- `Project ID` (above) tells OpenMetadata where to read metadata from. `Billing Project ID` tells BigQuery which project should pay for the queries.
-
- - Same-project setup: if your data lives in `analytics-prod` and that same project should pay for the queries, use `analytics-prod` as the `Project ID` and either leave `Billing Project ID` empty or set it to `analytics-prod`.
- - Cross-project billing setup: if your data lives in `marketing-prod` and `finance-prod`, but all query costs should be charged to `central-billing`, use `marketing-prod` and `finance-prod` as `Project ID` values and set `Billing Project ID` to `central-billing`.
-
-
- **Application Default Credentials (ADC) Authentication**
-
- If you want to use [ADC authentication](https://cloud.google.com/docs/authentication#adc) for BigQuery, configure the GCP credentials with type `gcp_adc`:
- ```yaml
- credentials:
- gcpConfig:
- type: gcp_adc
- projectId: ["your-project-id"] # Optional: specify project(s) for data access
- ```
- **Using ADC with Billing Project ID**: When using ADC authentication, you can still specify a **Billing Project ID** to control which project pays for the BigQuery queries OpenMetadata runs. This is particularly useful when:
- - Your service account has access to multiple projects
- - You want to bill queries to a specific project different from the one containing your data
- - You're running queries that span multiple projects
-
- **ADC Setup**: ADC authentication works automatically when running in Google Cloud environments (GKE, Compute Engine, Cloud Run) or when you've configured it locally using `gcloud auth application-default login`.
-
-- **Include Policy Tags** (Optional): Option to include policy tags as part of the column description. Enabled by default.
-- **Taxonomy Project ID** (Optional): Bigquery uses taxonomies to create hierarchical groups of policy tags. To apply access controls to BigQuery columns, tag the columns with policy tags. Learn more about how yo can create policy tags and set up column-level access control [here](https://cloud.google.com/bigquery/docs/column-level-security). If you have attached policy tags to the columns of table available in Bigquery, then OpenMetadata will fetch those tags and attach it to the respective columns. In this field you need to specify the id of project in which the taxonomy was created.
-- **Taxonomy Location** (Optional): Bigquery uses taxonomies to create hierarchical groups of policy tags. To apply access controls to BigQuery columns, tag the columns with policy tags. Learn more about how yo can create policy tags and set up column-level access control [here](https://cloud.google.com/bigquery/docs/column-level-security). If you have attached policy tags to the columns of table available in Bigquery, then OpenMetadata will fetch those tags and attach it to the respective columns. In this field you need to specify the location/region in which the taxonomy was created.
-- **Usage Location** (Optional): Location used to query `INFORMATION_SCHEMA.JOBS_BY_PROJECT` to fetch usage data. You can pass multi-regions, such as `us` or `eu`, or your specific region such as `us-east1`. Australia and Asia multi-regions are not yet supported.
-- **Cost Per TiB** (Optional): The cost (in USD) per tebibyte (TiB) of data processed during BigQuery usage analysis. This value is used to estimate query costs when analyzing usage metrics from `INFORMATION_SCHEMA.JOBS_BY_PROJECT`. This setting does **not** affect actual billing—it is only used for internal reporting and visualization of estimated costs. The default value is $6.25 per TiB; adjust it according to your organization's negotiated rates or flat-rate pricing model.
-
-
-
-
-
-
+
+ `Project ID` tells OpenMetadata where to read metadata from. `Billing Project ID` tells BigQuery which project should pay for the queries.
+
+ - Same-project setup: if your data lives in `analytics-prod` and that same project should pay for the queries, use `analytics-prod` as the `Project ID` and either leave `Billing Project ID` empty or set it to `analytics-prod`.
+ - Cross-project billing setup: if your data lives in `marketing-prod` and `finance-prod`, but all query costs should be charged to `central-billing`, use `marketing-prod` and `finance-prod` as `Project ID` values and set `Billing Project ID` to `central-billing`.
+
+- **Include Policy Tags (Optional)**: Option to include policy tags as part of the column description. Enabled by default.
+- **Taxonomy Project ID (Optional)**: Bigquery uses taxonomies to create hierarchical groups of policy tags. To apply access controls to BigQuery columns, tag the columns with policy tags. Learn more about how yo can create policy tags and set up column-level access control [here](https://cloud.google.com/bigquery/docs/column-level-security)
+If you have attached policy tags to the columns of table available in Bigquery, then OpenMetadata will fetch those tags and attach it to the respective columns.
+In this field you need to specify the id of project in which the taxonomy was created.
+- **Taxonomy Location (Optional)**: Bigquery uses taxonomies to create hierarchical groups of policy tags. To apply access controls to BigQuery columns, tag the columns with policy tags. Learn more about how yo can create policy tags and set up column-level access control [here](https://cloud.google.com/bigquery/docs/column-level-security)
+If you have attached policy tags to the columns of table available in Bigquery, then OpenMetadata will fetch those tags and attach it to the respective columns.
+In this field you need to specify the location/region in which the taxonomy was created.
+- **Usage Location (Optional)**:
+Location used to query `INFORMATION_SCHEMA.JOBS_BY_PROJECT` to fetch usage data. You can pass multi-regions, such as `us` or `eu`, or your specific region such as `us-east1`. Australia and Asia multi-regions are not yet supported.
+- **Cost Per TiB (Optional)**:
+The cost (in USD) per tebibyte (TiB) of data processed during BigQuery usage analysis. This value is used to estimate query costs when analyzing usage metrics from `INFORMATION_SCHEMA.JOBS_BY_PROJECT`.
+This setting does **not** affect actual billing, it is only used for internal reporting and visualization of estimated costs.
+The default value is $6.25 per TiB; adjust it according to your organization's negotiated rates or flat-rate pricing model.
+
+
+**Application Default Credentials (ADC) Authentication**
+
+If you want to use [ADC authentication](https://cloud.google.com/docs/authentication#adc) for BigQuery, configure the GCP credentials with type `gcp_adc`:
+```yaml
+credentials:
+ gcpConfig:
+ type: gcp_adc
+ projectId: ["your-project-id"] # Optional: specify project(s) for data access
+```
+**Using ADC with Billing Project ID**: When using ADC authentication, you can still specify a **Billing Project ID** to control which project pays for the BigQuery queries OpenMetadata runs. This is particularly useful when:
+- Your service account has access to multiple projects
+- You want to bill queries to a specific project different from the one containing your data
+- You're running queries that span multiple projects
+
+**ADC Setup**: ADC authentication works automatically when running in Google Cloud environments (GKE, Compute Engine, Cloud Run) or when you've configured it locally using `gcloud auth application-default login`.
+
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection Scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+## Cross Project Lineage
+OpenMetadata supports cross-project lineage, but the data must be ingested within a single service. This means you need to perform lineage ingestion for just one service while including multiple projects.
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/bigtable.mdx b/v2.0.x-SNAPSHOT/connectors/database/bigtable.mdx
index 27a958a72..2eb356458 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/bigtable.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/bigtable.mdx
@@ -4,12 +4,8 @@ description: Connect Google Cloud Bigtable to OpenMetadata with our comprehensiv
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
## Requirements
+
+Before configuring the BigTable connector, make sure the following requirements are in place.
+
### BigTable Admin API Permissions
- Enable the `bigtableadmin.googleapis.com` service in your GCP project. See the [Cloud Bigtable Admin API reference](https://docs.cloud.google.com/bigtable/docs/reference/admin/rest).
- Select the `GCP Project ID`.
@@ -47,33 +45,131 @@ To execute metadata extraction workflow successfully the user or the service acc
Checkout this documentation on how to create a custom role and assign it to the service account.
+
## Metadata Ingestion
-
-# Connection Options
-
-
-**GCP Credentials**:
+
+To ingest metadata from BigTable, you need to create a service connection. The service connects BigTable with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **BigTable** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other BigTable services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for BigTable. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **GCP Credentials**:
You can authenticate with your BigTable instance using either `GCP Credentials Path` where you can specify the file path of the service account key or you can pass the values directly by choosing the `GCP Credentials Values` from the service account key file.
You can checkout [this](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) documentation on how to create the service account keys and download it.
-**GCP Credentials Values**: Passing the raw credential values provided by BigTable. This requires us to provide the following information, all provided by BigTable:
-- **Credentials type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
-- **Project ID**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigTable projects into one service.
-- **Private Key ID**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
-- **Private Key**: This is the private key associated with the service account that is used to authenticate and authorize access to BigTable. To fetch this key, look for the value associated with the `private_key` key in the service account file.
-- **Client Email**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
-- **Client ID**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
-- **Auth URI**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
-- **Token URI**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. The default token URI is `https://oauth2.googleapis.com/token`.
-- **Authentication Provider X509 Certificate URL**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
-- **Client X509Cert URL**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
-**GCP Credentials Path**: Passing a local file path that contains the credentials.
+ - **GCP Credentials Values**: Passing the raw credential values provided by BigTable. This requires us to provide the following information, all provided by BigTable:
+ - **Credentials type**: Credentials Type is the type of the account, for a service account the value of this field is `service_account`. To fetch this key, look for the value associated with the `type` key in the service account key file.
+ - **Project ID**: A project ID is a unique string used to differentiate your project from all others in Google Cloud. To fetch this key, look for the value associated with the `project_id` key in the service account key file. You can also pass multiple project id to ingest metadata from different BigTable projects into one service.
+ - **Private Key ID**: This is a unique identifier for the private key associated with the service account. To fetch this key, look for the value associated with the `private_key_id` key in the service account file.
+ - **Private Key**: This is the private key associated with the service account that is used to authenticate and authorize access to BigTable. To fetch this key, look for the value associated with the `private_key` key in the service account file.
+ - **Client Email**: This is the email address associated with the service account. To fetch this key, look for the value associated with the `client_email` key in the service account key file.
+ - **Client ID**: This is a unique identifier for the service account. To fetch this key, look for the value associated with the `client_id` key in the service account key file.
+ - **Auth URI**: This is the URI for the authorization server. To fetch this key, look for the value associated with the `auth_uri` key in the service account key file. The default value to Auth URI is https://accounts.google.com/o/oauth2/auth.
+ - **Token URI**: The Google Cloud Token URI is a specific endpoint used to obtain an OAuth 2.0 access token from the Google Cloud IAM service. This token allows you to authenticate and access various Google Cloud resources and APIs that require authorization. To fetch this key, look for the value associated with the `token_uri` key in the service account credentials file. The default token URI is `https://oauth2.googleapis.com/token`.
+ - **Authentication Provider X509 Certificate URL**: This is the URL of the certificate that verifies the authenticity of the authorization server. To fetch this key, look for the value associated with the `auth_provider_x509_cert_url` key in the service account key file. The Default value for Auth Provider X509Cert URL is https://www.googleapis.com/oauth2/v1/certs
+ - **Client X509Cert URL**: This is the URL of the certificate that verifies the authenticity of the service account. To fetch this key, look for the value associated with the `client_x509_cert_url` key in the service account key file.
+ - **GCP Credentials Path**: Passing a local file path that contains the credentials.
+
If you want to use [ADC authentication](https://cloud.google.com/docs/authentication#adc) for BigTable you can just leave
the GCP credentials empty. This is why they are not marked as required.
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection Scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/burstiq.mdx b/v2.0.x-SNAPSHOT/connectors/database/burstiq.mdx
index 39988cb98..cfdb58aea 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/burstiq.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/burstiq.mdx
@@ -1,16 +1,12 @@
---
title: BurstIQ Connector | OpenMetadata LifeGraph Database Guide
-description: Connect BurstIQ LifeGraph to OpenMetadata with our comprehensive database connector guide. Step-by-step setup, configuration, and metadata extraction for seamless integration.
+description: Connect BurstIQ LifeGraph to OpenMetadata with this database connector guide. Step-by-step setup, configuration, and metadata extraction.
slug: /connectors/database/burstiq
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+To ingest metadata from BurstIQ, you need to create a service connection. The service connects BurstIQ with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **BurstIQ** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other BurstIQ services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for BurstIQ. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Username**: Username to connect to BurstIQ. This user should have privileges to read all the metadata in BurstIQ LifeGraph.
+- **Password**: Password to connect to BurstIQ.
+- **Realm Name**: BurstIQ Keycloak realm name (for example, 'ems' from https://auth.burstiq.com/realms/ems). This identifies your organization's authentication realm.
+- **BurstIQ SDZ Name**: BurstIQ Secure Data Zone (SDZ) name for API requests.
+- **BurstIQ Customer Name**: BurstIQ customer name for API requests.
+- **BurstIQ System Wallet ID**: BurstIQ system wallet ID sent as the `biq_system_wallet_id` header. Required for profiler data access.
+- **Table Filter Pattern** *(Optional)*: Regex to only include/exclude dictionaries (tables) that match the pattern.
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection Scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
-## Connection Options
+When both Include and Exclude are set, Exclude takes priority.
-
-
- - **Username**: Username to connect to BurstIQ. This user should have privileges to read all the metadata in BurstIQ LifeGraph.
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
- - **Password**: Password to connect to BurstIQ.
+**Filter Options**
- - **Realm Name**: BurstIQ Keycloak realm name (e.g., 'ems' from https://auth.burstiq.com/realms/ems). This identifies your organization's authentication realm.
+The Database, Schema, and Table sections each include the following filter options:
- - **BurstIQ SDZ Name**: BurstIQ Secure Data Zone (SDZ) name for API requests.
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
- - **BurstIQ Customer Name**: BurstIQ customer name for API requests.
+Each section provides the following controls:
- - **BurstIQ System Wallet ID**: BurstIQ system wallet ID sent as the `biq_system_wallet_id` header. Required for profiler data access.
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
- - **Table Filter Pattern** *(Optional)*: Regex to only include/exclude dictionaries (tables) that match the pattern.
-
+### Step 6: Create & Deploy
-
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
-
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
-
+## Configure Metadata Agent and Schedule Ingestion
-
-
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/cassandra.mdx b/v2.0.x-SNAPSHOT/connectors/database/cassandra.mdx
index 16e609423..1177c7048 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/cassandra.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/cassandra.mdx
@@ -4,10 +4,7 @@ description: Connect Cassandra to OpenMetadata with our database connector. Step
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Cassandra, you need to create a service connection. The service connects Cassandra with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Cassandra** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Cassandra services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Cassandra. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Username to connect to Cassandra. This user must have the necessary permissions to perform metadata extraction and table queries.
-- **Host Port**: When using the `cassandra` connecion schema, the hostPort parameter specifies the host and port of the Cassandra. This should be specified as a string in the format `hostname:port`. E.g., `localhost:9042`.- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
-**Auth Type**: Following authentication types are supported:
+- **Host Port**: When using the `cassandra` connecion schema, the hostPort parameter specifies the host and port of the Cassandra. This should be specified as a string in the format `hostname:port`. For example, `localhost:9042`.
+- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
+- **Auth Type**: Following authentication types are supported:
1. **Basic Authentication**:
We'll use the user credentials to connect to Cassandra
- **password**: Password of the user.
@@ -51,7 +79,8 @@ Configuration for connecting to DataStax Astra DB in the cloud.
- **requestTimeout**: Timeout in seconds for individual Cassandra requests.
- **token**: The Astra DB application token used for authentication.
- **secureConnectBundle**: File path to the Secure Connect Bundle (.zip) used for a secure connection to DataStax Astra DB.
-**SSL Modes**
+
+#### SSL Modes
There are a couple of types of SSL modes that Cassandra supports which can be added to ConnectionArguments, they are as follows:
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
@@ -59,15 +88,70 @@ There are a couple of types of SSL modes that Cassandra supports which can be ad
- **require**: SSL is required.
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified, and the server hostname must match the hostname attribute on the certificate.
-**SSL Configuration**
+
+#### SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under sslConfig which is placed in the source.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Cassandra Connection with SSL in OpenMetadata
To establish secure connections between OpenMetadata and a Cassandra database, you can use any SSL mode provided by Cassandra, except disable.
Under `Advanced Config`, after selecting the SSL mode, provide the CA certificate, SSL certificate and SSL key.
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/clickhouse.mdx b/v2.0.x-SNAPSHOT/connectors/database/clickhouse.mdx
index b12a064fc..2c9c09890 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/clickhouse.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/clickhouse.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Clickhouse connector.
Configure and schedule Clickhouse metadata and profiler workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- - [Profiler \& Data Quality](#profiler-&-data-quality)
- - [Usage \& Lineage](#usage-&-lineage)
- [Metadata Ingestion](#metadata-ingestion)
- - [Service Name](#metadata-ingestion)
- - [Connection Options](#connection-options)
- - [Metadata Ingestion Options](#metadata-ingestion-options)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/clickhouse/troubleshooting)
- - [Workflow Deployment Error](/v2.0.x-SNAPSHOT/connectors/database/clickhouse/troubleshooting#workflow-deployment-error)
-- [Related](#related)
## Requirements
Clickhouse user must grant `SELECT` privilege on `system.*` and schema/tables to fetch the metadata of tables and views.
@@ -50,22 +39,117 @@ GRANT SELECT ON .* to ;
Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
### Usage & Lineage
For the usage and lineage workflow, the user will need `SELECT` privilege. You can find more information on the usage workflow [here](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage) and the lineage workflow [here](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage).
+
## Metadata Ingestion
-
-# Connection Options
-
-
+
+To ingest metadata from ClickHouse, you need to create a service connection. The service connects ClickHouse with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **ClickHouse** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other ClickHouse services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for ClickHouse. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Clickhouse. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Clickhouse.
- **Host and Port**: Enter the fully qualified hostname and port number for your Clickhouse deployment in the Host and Port field.
- **Use HTTPS Protocol**: Enable this flag when the when the Clickhouse instance is hosted via HTTPS protocol. This flag is useful when you are using `clickhouse+http` connection scheme.
- **Secure Connection**: Establish secure connection with ClickHouse. ClickHouse supports secure communication over SSL/TLS to protect data in transit, by checking this option, it establishes secure connection with ClickHouse. This flag is useful when you are using `clickhouse+native` connection scheme.
- **Key File**: The key file path is the location when ClickHouse looks for a file containing the private key needed for secure communication over SSL/TLS. By default, ClickHouse will look for the key file in the `/etc/clickhouse-server directory`, with the file name `server.key`. However, this can be customized in the ClickHouse configuration file (`config.xml`). This flag is useful when you are using `clickhouse+native` connection scheme and the secure connection flag is enabled.
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection Scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/cockroach.mdx b/v2.0.x-SNAPSHOT/connectors/database/cockroach.mdx
index b5edc1b05..a02895131 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/cockroach.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/cockroach.mdx
@@ -4,10 +4,7 @@ description: Set up CockroachDB connector in OpenMetadata to discover, catalog,
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from CockroachDB, you need to create a service connection. The service connects CockroachDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **CockroachDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other CockroachDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for CockroachDB. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Cockroach. It should have enough privileges to read all the metadata.
- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
- **Basic Auth**:
- **Password**: Password to connect to Cockroach
-
- **Host and Port**: Enter the fully qualified hostname and port number for your Cockroach deployment in the Host and Port field.
-
-
-
-
-
-
-**SSL Modes**
+
+#### SSL Modes
There are a couple of types of SSL modes that Cockroach supports which can be added to ConnectionArguments, they are as follows:
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
@@ -57,10 +78,70 @@ There are a couple of types of SSL modes that Cockroach supports which can be ad
- **require**: SSL is required.
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified, and the server hostname must match the hostname attribute on the certificate.
-**SSL Configuration**
+
+#### SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under sslConfig which is placed in the source.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Cockroach Connection with SSL in OpenMetadata
To establish secure connections between OpenMetadata and a Cockroach database, you can configure SSL using different SSL modes provided by Cockroach, each offering varying levels of security.
Under `Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that Cockroach requires only the CA certificate for SSL validation.
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/couchbase.mdx b/v2.0.x-SNAPSHOT/connectors/database/couchbase.mdx
index 51014fb99..1031dddbb 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/couchbase.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/couchbase.mdx
@@ -4,11 +4,7 @@ description: Connect Couchbase to OpenMetadata effortlessly. Complete setup guid
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Couchbase, you need to create a service connection. The service connects Couchbase with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Couchbase** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Couchbase services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Couchbase. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Username to connect to Couchbase.
- **Password**: Password to connect to Couchbase.
-- **Hostport**: If couchbase is hosted on cloud then the hostport parameter specifies the connection string and if you are using couchbase server then the hostport parameter specifies hostname of the Couchbase. This should be specified as a string in the format `hostname` or `xyz.cloud.couchbase.com`. E.g., `localhost`.
-- **bucketName**: Optional name to give to the bucket in OpenMetadata. If left blank, If left blank, we will ingest all the bucket names.
-
-
-
-
-
-
+- **Hostport**: If Couchbase is hosted on the cloud, the hostport parameter specifies the connection string. If you are using a Couchbase server, the hostport parameter specifies the hostname of the Couchbase instance. This should be specified as a string in the format `hostname` or `xyz.cloud.couchbase.com`, for example, `localhost`.
+- **bucketName**: Optional name to give to the bucket in OpenMetadata. If left blank, OpenMetadata ingests all bucket names.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Options (Optional)**: Enter any additional connection options that can be sent to the database during the connection. These details must be added as key-value pairs.
+- **Connection Arguments (Optional)**: Enter any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as key-value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/databricks.mdx b/v2.0.x-SNAPSHOT/connectors/database/databricks.mdx
index 94965cb71..398af5069 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/databricks.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/databricks.mdx
@@ -6,11 +6,7 @@ sidebarTitle: Overview
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Supported Authentication Types:**
-- **Personal Access Token** — Token-based workspace authentication generated from User Settings in Databricks
-- **Databricks OAuth** — OAuth2 Machine-to-Machine authentication using Service Principal credentials
-- **Azure AD Setup** — Azure Active Directory authentication using Azure Service Principal (for Azure Databricks workspaces)
+- **Personal Access Token**: Token-based workspace authentication generated from User Settings in Databricks
+- **Databricks OAuth**: OAuth2 Machine-to-Machine authentication using Service Principal credentials
+- **Azure AD Setup**: Azure Active Directory authentication using Azure Service Principal (for Azure Databricks workspaces)
Configure and schedule Databricks metadata and profiler workflows from the OpenMetadata UI:
@@ -43,10 +39,17 @@ Configure and schedule Databricks metadata and profiler workflows from the OpenM
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/databricks/troubleshooting)
+
## Requirements
+
+Before configuring the Databricks connector, make sure the following requirements are in place.
+
### Permission Requirement
+
To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
### Metadata and Profiling Permissions
+
These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
```sql
GRANT USE CATALOG ON CATALOG TO ``;
@@ -54,7 +57,9 @@ GRANT USE SCHEMA ON SCHEMA TO ``;
GRANT SELECT ON TABLE TO ``;
```
Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
### Usage and Lineage
+
These permissions enable OpenMetadata to extract query history and construct lineage information.
```sql
-- Query history for usage analytics and SQL-based lineage
@@ -69,12 +74,14 @@ GRANT USE SCHEMA ON SCHEMA system.access TO ``;
These permissions allow access to Databricks system tables that track query activity and lineage relationships, enabling lineage and usage statistics generation.
### View Definitions
+
To extract view definitions, the user needs access to the information schema:
```sql
GRANT SELECT ON INFORMATION_SCHEMA.VIEWS TO ``;
```
### Tags (Databricks 13.3+)
+
To extract Databricks tags on catalogs, schemas, tables, and columns, the following permissions are required:
```sql
GRANT SELECT ON ``.information_schema.catalog_tags TO ``;
@@ -89,36 +96,118 @@ Tag extraction requires Databricks Runtime 13.3 or higher. If your cluster is ru
Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
## Unity Catalog
+
If you are using Unity Catalog in Databricks, then checkout the [Unity Catalog](/v2.0.x-SNAPSHOT/connectors/database/unity-catalog) connector.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Databricks, you need to create a service connection. The service connects Databricks with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Databricks** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Databricks services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Databricks. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: Enter the fully qualified hostname and port number for your Databricks deployment in the Host and Port field.
- **Authentication Type**: Choose one of the following authentication methods:
- - **Personal Access Token** — Provide a `token` generated from User Settings → Developer → Access Tokens in your Databricks workspace.
- - **Databricks OAuth** — Provide a `clientId` and `clientSecret` for a Service Principal created in your Databricks Account Console.
- - **Azure AD Setup** — Provide `azureClientId`, `azureClientSecret`, and `azureTenantId` for an Azure Service Principal registered in Azure Active Directory (for Azure Databricks workspaces only).
+ - **Personal Access Token**: Provide a `token` generated from **User Settings** > **Developer** > **Access Tokens** in your Databricks workspace.
+ - **Databricks OAuth**: Provide a `clientId` and `clientSecret` for a Service Principal created in your Databricks Account Console.
+ - **Azure AD Setup**: Provide `azureClientId`, `azureClientSecret`, and `azureTenantId` for an Azure Service Principal registered in Azure Active Directory (for Azure Databricks workspaces only).
- **HTTP Path**: Databricks compute resources URL.
- **Connection Timeout**: The maximum amount of time (in seconds) to wait for a successful connection to the data source. If the connection attempt takes longer than this timeout period, an error will be returned.
- **Catalog**: Catalog of the data source (Example: hive_metastore). This is an optional parameter, if you would like to restrict the metadata reading to a single catalog. When left blank, OpenMetadata Ingestion attempts to scan all the catalogs.
- **DatabaseSchema**: Database schema of the data source. This is an optional parameter, if you would like to restrict the metadata reading to a single database schema. When left blank, OpenMetadata Ingestion attempts to scan all the database schemas.
- **Query History Table**: Table name to fetch the query history from. Defaults to `system.query.history`.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/db2.mdx b/v2.0.x-SNAPSHOT/connectors/database/db2.mdx
index b2bd9bb9b..cb12994dc 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/db2.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/db2.mdx
@@ -4,11 +4,7 @@ description: Connect DB2 databases to OpenMetadata with our comprehensive connec
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Username**: Specify the User to connect to DB2. It should have enough privileges to read all the metadata.
+
+To ingest metadata from DB2, you need to create a service connection. The service connects DB2 with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **DB2** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other DB2 services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for DB2. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Username**: Specify the user to connect to DB2. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to DB2.
- **database**: Database of the data source.
- **Host and Port**: Enter the fully qualified hostname and port number for your DB2 deployment in the Host and Port field.
- **License File Name**: License file name in case the license is required for connection.
- **License**: Contents of your license file if applicable, make sure to replace new lines with `\n` before pasting it here.
+
If you are using DB2 for IBM i:
-- From advanced config you need to chose `ibmi` scheme
-- In Host and Port you should not add the Port Number.
+- From advanced config, choose the `ibmi` scheme.
+- In Host and Port, don't add the port number.
If you have a **db2jcc_license_cisuz.jar** file, it will not work with **ibm_db**. This file is a **Db2 Connect** license for the Java Driver.
For **non-Java drivers**, such as the Python Client used in OpenMetadata ingestion, a **Db2 Connect** client-side license is required, typically named **db2con*.lic**.
The **db2jcc_license_cisuz.jar** is specifically for Java-based clients, whereas OpenMetadata ingestion operates with a Python Client, making the `.jar` file incompatible.
-For activating a **non-Java license** for Db2 Connect **Application Server Edition**, **Advanced Application Server Edition**, **Enterprise Edition**, or **Trial**, follow these steps:
+To activate a non-Java license for Db2 Connect **Application Server Edition**, **Advanced Application Server Edition**, **Enterprise Edition**, or **Trial**, follow these steps:
- Download the **license activation kit** from IBM Passport Advantage: [IBM PPA](https://www.ibm.com/software/passportadvantage/pao_customer.html).
-- Unzip the package and locate the **non-Java license file** (e.g., `db2consv_ee.lic`).
+- Unzip the package and locate the **non-Java license file** (for example, `db2consv_ee.lic`).
- Apply the `.lic` file to activate the license.
-For further reference, check this IBM post: [Everything About Db2 Connect Licensing](https://community.ibm.com/community/user/datamanagement/blogs/shilu-mathai2/2023/05/05/everything-about-db2-connect-licensing).
+For more information, see [Everything About Db2 Connect Licensing](https://community.ibm.com/community/user/datamanagement/blogs/shilu-mathai2/2023/05/05/everything-about-db2-connect-licensing).
+
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Options (Optional)**: Enter any additional connection options that can be sent to the database during the connection. These details must be added as key-value pairs.
+- **Connection Arguments (Optional)**: Enter any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as key-value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
-
-
-
-
-
-
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/dbt/configure-dbt-workflow.mdx b/v2.0.x-SNAPSHOT/connectors/database/dbt/configure-dbt-workflow.mdx
index 2e5da5a61..fd8e963c6 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/dbt/configure-dbt-workflow.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/dbt/configure-dbt-workflow.mdx
@@ -59,13 +59,13 @@ No additional manual configuration is required in the UI after a successful dbt
We can create a workflow that will obtain the dbt information from the dbt files and feed it to OpenMetadata. The dbt Ingestion will be in charge of obtaining this data.
-### 1. Add a dbt Ingestion
+### Step 1: Add a dbt Ingestion
-From the Service Page, go to the Ingestions tab to add a new ingestion and click on Add dbt Ingestion.
+In the Service details page, navigate to the **Agents** tab and click **Add Agent** > **DBT**.
-
+
-### 2. Configure the dbt Ingestion
+### Step 2: Configure the dbt Ingestion
Here you can enter the configuration required for OpenMetadata to get the dbt files (manifest.json, catalog.json and run_results.json) required to extract the dbt metadata.
Select any one of the source from below from where the dbt files can be fetched:
@@ -76,8 +76,6 @@ Only the `manifest.json` file is required for dbt ingestion.
-### dbt Core
-
#### AWS S3 Buckets
OpenMetadata connects to the AWS s3 bucket via the credentials provided and scans the AWS s3 buckets for `manifest.json`, `catalog.json` and `run_results.json` files.
@@ -86,7 +84,7 @@ The name of the s3 bucket and prefix path to the folder in which the dbt files a
Follow the link [here](/v2.0.x-SNAPSHOT/connectors/database/dbt/setup-multiple-dbt-projects) for instructions on setting up multiple dbt projects.
-
+
#### Google Cloud Storage Buckets
@@ -96,16 +94,16 @@ The name of the GCS bucket and prefix path to the folder in which the dbt files
GCS credentials can be stored in two ways:
-**1.** Entering the credentials directly into the form
+- Entering the credentials directly into the form
Follow the link [here](/v2.0.x-SNAPSHOT/connectors/database/dbt/setup-multiple-dbt-projects) for instructions on setting up multiple dbt projects.
-
+ 
-**2.** Entering the path of file in which the GCS bucket credentials are stored.
+- Entering the path of file in which the GCS bucket credentials are stored.
-
+ 
For more information on Google Cloud Storage authentication click [here](https://cloud.google.com/docs/authentication/getting-started#create-service-account-console).
@@ -118,19 +116,19 @@ The Azure Storage account, container name, and optional folder (prefix) path whe
Follow the link [here](/v2.0.x-SNAPSHOT/connectors/database/dbt/setup-multiple-dbt-projects) for instructions on setting up multiple dbt projects.
-
+
#### Local Storage
Path of the `manifest.json`, `catalog.json` and `run_results.json` files stored in the local system or in the container in which OpenMetadata server is running can be directly provided.
-
+
#### File Server
File server path of the `manifest.json`, `catalog.json` and `run_results.json` files stored on a file server directly provided.
-
+
### dbt Cloud
@@ -149,7 +147,7 @@ Refer to the code [here](https://github.com/open-metadata/OpenMetadata/blob/main
-
+
@@ -164,4 +162,4 @@ To know how to get the values for `Dbt Cloud Account Id`, `Dbt Cloud Project Id`
### 3. Schedule and Deploy
After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the lineage pipeline being added to the Service Ingestions.
-
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/dbt/ingest-dbt-owner.mdx b/v2.0.x-SNAPSHOT/connectors/database/dbt/ingest-dbt-owner.mdx
index 082f5eb4f..65abf30f6 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/dbt/ingest-dbt-owner.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/dbt/ingest-dbt-owner.mdx
@@ -79,26 +79,25 @@ models:
- **User Validation**: Ensure the specified owners (e.g., John Doe, jane@gmail.com) exist in OpenMetadata as valid user or team entities.
- **Overwrite Behavior**: Use the dbtUpdateOwners parameter in your ingestion configuration to control whether this list overwrites existing owners or only applies to unowned assets.
-#### Following steps shows adding a User to OpenMetadata:
-**1.** Click on the `Users` section from homepage
+#### Add a User to OpenMetadata:
-
-
+1. Navigate to **Settings** > **Team & User Management** > **Users**.
-**2.** Click on the `Add User` button
+ 
-
+2. Click **Add User**.
+ 
-3. Enter the details as shown for the user
+3. Enter the details and click **Create**.
-
+
-If the owner's name in `manifest.json` or `catalog.json` file is `openmetadata`, you need to enter `openmetadata@youremail.com` in the email id section of add user form as shown below.
+ If the owner's name in `manifest.json` or `catalog.json` file is `openmetadata`, you need to enter `openmetadata@youremail.com` in the email id section of add user form as shown below.
-
+
-
+
#### Following steps shows adding a Team to OpenMetadata:
diff --git a/v2.0.x-SNAPSHOT/connectors/database/deltalake.mdx b/v2.0.x-SNAPSHOT/connectors/database/deltalake.mdx
index c7ca5486f..8c3d8a8ea 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/deltalake.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/deltalake.mdx
@@ -4,11 +4,7 @@ description: Connect OpenMetadata to Delta Lake with our comprehensive database
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
Bucket Policy in AWS requires at least these permissions:
```json
{
@@ -48,11 +48,49 @@ To execute metadata extraction AWS account should have enough access to fetch re
]
}
```
+
## Metadata Ingestion
-
-## Connection Details
-
-
+
+To ingest metadata from Delta Lake, you need to create a service connection. The service connects Delta Lake with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **DeltaLake** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Delta Lake services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Delta Lake. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection Details for MetastoreConfig**
- **Metastore Host Port**: Enter the Host & Port of Hive Metastore Service to configure the Spark Session. Either
of `metastoreHostPort`, `metastoreDb` or `metastoreFilePath` is required.
- **Metastore File Path**: Enter the file path to local Metastore in case Spark cluster is running locally. Either
@@ -60,68 +98,128 @@ To execute metadata extraction AWS account should have enough access to fetch re
- **Metastore DB**: The JDBC connection to the underlying Hive metastore DB. Either
of `metastoreHostPort`, `metastoreDb` or `metastoreFilePath` is required.
- **appName (Optional)**: Enter the app name of spark session.
-- **Connection Arguments (Optional)**: Key-Value pairs that will be used to pass extra `config` elements to the Spark Session builder.
-We are internally running with `pyspark` 3.X and `delta-lake` 2.0.0. This means that we need to consider Spark configuration options for 3.X.
+- **Connection Arguments (Optional)**: Key-value pairs used to pass extra `config` elements to the Spark Session builder.
+
+OpenMetadata internally runs `pyspark` 3.X and `delta-lake` 2.0.0, so Spark configuration options must target 3.X.
+
**Metastore Host Port**
-When connecting to an External Metastore passing the parameter `Metastore Host Port`, we will be preparing a Spark Session with the configuration
+
+When connecting to an external metastore using the `Metastore Host Port` parameter, OpenMetadata prepares a Spark Session with the following configuration:
```
.config("hive.metastore.uris", "thrift://{connection.metastoreHostPort}")
```
-Then, we will be using the `catalog` functions from the Spark Session to pick up the metadata exposed by the Hive Metastore.
+OpenMetadata then uses the `catalog` functions from the Spark Session to pick up the metadata exposed by the Hive Metastore.
+
**Metastore File Path**
-If instead we use a local file path that contains the metastore information (e.g., for local testing with the default `metastore_db` directory), we will set
+
+If a local file path contains the metastore information instead (for example, for local testing with the default `metastore_db` directory), set the following:
```
.config("spark.driver.extraJavaOptions", "-Dderby.system.home={connection.metastoreFilePath}")
```
-To update the `Derby` information. More information about this in a great [SO thread](https://stackoverflow.com/questions/38377188/how-to-get-rid-of-derby-log-metastore-db-from-spark-shell).
-- You can find all supported configurations [here](https://spark.apache.org/docs/latest/configuration.html)
-- If you need further information regarding the Hive metastore, you can find it [here](https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html),
- and in The Internals of Spark SQL [book](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-hive-metastore.html).
+This updates the `Derby` information. For more information, see this [Stack Overflow thread](https://stackoverflow.com/questions/38377188/how-to-get-rid-of-derby-log-metastore-db-from-spark-shell).
+- For all supported configurations, see the [Spark configuration documentation](https://spark.apache.org/docs/latest/configuration.html).
+- For more information about the Hive metastore, see the [Spark Hive tables documentation](https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html)
+ and *The Internals of Spark SQL* [book](https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-hive-metastore.html).
+
**Metastore Database**
-You can also connect to the metastore by directly pointing to the Hive Metastore db, e.g., `jdbc:mysql://localhost:3306/demo_hive`.
-Here, we will need to inform all the common database settings (url, username, password), and the driver class name for JDBC metastore.
-You will need to provide the driver to the ingestion image, and pass the `classpath` which will be used in the Spark Configuration under `spark.driver.extraClassPath`.
-#### Connection Details for StorageConfig - S3
+
+Connect to the metastore directly by pointing to the Hive Metastore db, for example, `jdbc:mysql://localhost:3306/demo_hive`.
+This requires the common database settings (url, username, password) and the driver class name for the JDBC metastore.
+Provide the driver to the ingestion image and pass the `classpath` used in the Spark Configuration under `spark.driver.extraClassPath`.
+
+**Connection Details for StorageConfig - S3**
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
-Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
-You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+ Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
+ You must use both the access key ID and secret access key together to authenticate your requests.
+ You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
-Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
-services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+ As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
+ Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
+ services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
+ You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
-You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
+ You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
-Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+ Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
-You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+ You can inform this field if you'd like to use a profile other than `default`.
+ Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
-A user who wants to access a role in a different account must also have permissions that are delegated from the account
-administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
-This is a required field if you'd like to `AssumeRole`.
-Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+ A user who wants to access a role in a different account must also have permissions that are delegated from the account
+ administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
+ This is a required field if you'd like to `AssumeRole`.
+ Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
-By default, we'll use the name `OpenMetadataSession`.
-Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
+ By default, OpenMetadata uses the name `OpenMetadataSession`.
+ Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
-Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
-
+ Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/domo-database.mdx b/v2.0.x-SNAPSHOT/connectors/database/domo-database.mdx
index fb02397dc..718397a53 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/domo-database.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/domo-database.mdx
@@ -4,10 +4,7 @@ description: Connect Domo Database to OpenMetadata with our comprehensive connec
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Domo Database, you need to create a service connection. The service connects Domo Database with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Domo Database** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Domo Database services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Domo Database. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Client ID**: Client ID for DOMO Database.
- **Secret Token**: Secret Token to Connect DOMO Database.
- **Access Token**: Access to Connect to DOMO Database.
- **Api Host**: API Host to Connect to DOMO Database instance.
- **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/doris.mdx b/v2.0.x-SNAPSHOT/connectors/database/doris.mdx
index 720ff9fce..da656e035 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/doris.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/doris.mdx
@@ -4,11 +4,7 @@ description: Connect Apache Doris to OpenMetadata with our comprehensive databas
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
= 1.2.0, Data Profiler: Doris >= 2.0.2
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Doris, you need to create a service connection. The service connects Doris with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Doris** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Doris services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Doris. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Doris. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Doris.
- **Host and Port**: Enter the fully qualified hostname and port number for your Doris deployment in the Host and Port field.
-- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
+- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, OpenMetadata uses default as the database name.
- **databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
- **caCertificate**: Provide the path to ssl ca file.
- **sslCertificate**: Provide the path to ssl client certificate file (ssl_cert).
- **sslKey**: Provide the path to ssl client certificate file (ssl_key).
-
-
-
-
-
-
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Options (Optional)**: Enter any additional connection options that can be sent to the database during the connection. These details must be added as key-value pairs.
+- **Connection Arguments (Optional)**: Enter any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as key-value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Doris Connection with SSL in OpenMetadata
-To establish secure connections between OpenMetadata and Doris, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client’s SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server’s certificate.
+
+To establish secure connections between OpenMetadata and Doris, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server's certificate.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/druid.mdx b/v2.0.x-SNAPSHOT/connectors/database/druid.mdx
index 6bc1f99be..64f0173a3 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/druid.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/druid.mdx
@@ -4,11 +4,7 @@ description: Learn how to connect Apache Druid to OpenMetadata with our comprehe
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Druid, you need to create a service connection. The service connects Druid with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Druid** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Druid services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Druid. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Druid. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Druid.
- **Host and Port**: Enter the fully qualified hostname and port number for your Druid deployment in the Host and Port field.
-- **Database Name**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
-
-
-
-
-
-
+- **Database Name**: Optional name to give to the database in OpenMetadata. If left blank, OpenMetadata uses default as the database name.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Options (Optional)**: Enter any additional connection options that can be sent to the database during the connection. These details must be added as key-value pairs.
+- **Connection Arguments (Optional)**: Enter any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as key-value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/dynamodb.mdx b/v2.0.x-SNAPSHOT/connectors/database/dynamodb.mdx
index b99c8f7ae..c78915dec 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/dynamodb.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/dynamodb.mdx
@@ -4,10 +4,7 @@ description: Connect DynamoDB to OpenMetadata with our comprehensive database co
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from DynamoDB, you need to create a service connection. The service connects DynamoDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **DynamoDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other DynamoDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for DynamoDB. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
-Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
-You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+ Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
+ You must use both the access key ID and secret access key together to authenticate your requests.
+ You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
-Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
-services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+ As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+ Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
+ services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
+ You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
-You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
+ You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
-Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
+ Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
-You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+ You can inform this field if you'd like to use a profile other than `default`.
+ Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
-A user who wants to access a role in a different account must also have permissions that are delegated from the account
-administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
-This is a required field if you'd like to `AssumeRole`.
-Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
-
-When using Assume Role authentication, ensure you provide the following details:
-- **AWS Region**: Specify the AWS region for your deployment.
-- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
-
+ A user who wants to access a role in a different account must also have permissions that are delegated from the account
+ administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
+ This is a required field if you'd like to `AssumeRole`.
+ Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
+
+ When using Assume Role authentication, ensure you provide the following details:
+ - **AWS Region**: Specify the AWS region for your deployment.
+ - **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
+
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
-By default, we'll use the name `OpenMetadataSession`.
-Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
+ By default, OpenMetadata uses the name `OpenMetadataSession`.
+ Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
-Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
+ Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/epic.mdx b/v2.0.x-SNAPSHOT/connectors/database/epic.mdx
index a57abfb0e..e7c63915c 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/epic.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/epic.mdx
@@ -1,16 +1,10 @@
---
-title: Epic FHIR Connector | Collate Healthcare Integration Guide
-description: Connect Epic FHIR to Collate to automatically discover, catalog, and manage your Epic FHIR metadata. Step-by-step configuration guide.
+title: Epic FHIR Connector | OpenMetadata Database Integration Guide
+description: Connect Epic FHIR to OpenMetadata to automatically discover, catalog, and manage your Epic FHIR metadata. Step-by-step configuration guide.
sidebarTitle: Overview
-Collate: true
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+unavailableFeatures={["Data Profiler", "Data Quality", "dbt", "Lineage", "Column-level Lineage", "Query Usage", "Owners", "Tags", "Sample Data", "Auto-Classification", "Stored Procedures"]} />
In this section, we provide guides and references to use the Epic FHIR connector.
-Configure and schedule Epic metadata workflows from the Collate UI:
+Configure and schedule Epic metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/epic/troubleshooting)
-
+
## Requirements
+
To fetch metadata from Epic FHIR into OpenMetadata you will need:
-1. An accessible Epic FHIR base URL (e.g. `https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR`).
+
+1. An accessible Epic FHIR base URL (for example, `https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR`).
2. The FHIR version supported by your Epic server. Supported values are: `R4`, `STU3`, and `DSTU2`.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Epic, you need to create a service connection. The service connects Epic with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Epic** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Epic services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Epic. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **FHIR Server URL**: Base URL of the Epic FHIR server.
- **FHIR Version**: FHIR specification version supported by the server (`R4`, `STU3`, or `DSTU2`).
- **Database Name**: Optional; name that will be shown inside OpenMetadata. Defaults to `epic`.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/epic/yaml.mdx b/v2.0.x-SNAPSHOT/connectors/database/epic/yaml.mdx
index c9d2643e8..81392709d 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/epic/yaml.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/epic/yaml.mdx
@@ -21,7 +21,7 @@ icon='/public/images/connectors/epic.png'
name="Epic"
stage="BETA"
availableFeatures={["Metadata"]}
-unavailableFeatures={["Data Profiler", "Data Quality", "dbt", "Lineage", "Column-level Lineage", "Query Usage", "Owners", "Tags", "Sample Data", "Reverse Metadata (Collate Only)", "Auto-Classification", "Stored Procedures"]} />
+unavailableFeatures={["Data Profiler", "Data Quality", "dbt", "Lineage", "Column-level Lineage", "Query Usage", "Owners", "Tags", "Sample Data", "Auto-Classification", "Stored Procedures"]} />
In this section, we provide guides and references to use the Epic FHIR connector.
Configure and schedule Epic metadata workflows from the Collate UI:
- [Requirements](#requirements)
diff --git a/v2.0.x-SNAPSHOT/connectors/database/exasol.mdx b/v2.0.x-SNAPSHOT/connectors/database/exasol.mdx
index 634b2108a..e33b28a24 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/exasol.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/exasol.mdx
@@ -4,12 +4,8 @@ description: Connect Exasol database to OpenMetadata with our comprehensive conn
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references for using the Exasol connector.
-
-Configure and schedule Exasol metadata and other workflows from the OpenMetadata UI:
+Configure and schedule Exasol metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
-- [Lineage](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage)
-- [Query Usage](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/exasol/troubleshooting)
-
-
## Requirements
+
Note that we support an Exasol database version 8 or later. Ensure your Exasol instance meets this minimum version requirement before proceeding.
### Metadata
+
To ingest basic metadata, an Exasol user must have the following privileges:
- `USAGE` privilege on the schema
- `SELECT` privilege on the tables
@@ -56,6 +49,7 @@ Use these broader privileges only when access to all schemas and tables is inten
### Lineage & Query Usage
+
For the lineage & query usage workflows, OpenMetadata fetches query logs by querying
`EXA_STATISTICS.EXA_DBA_AUDIT_SQL` and joining with `EXA_STATISTICS.EXA_DBA_AUDIT_SESSIONS`.
To access these system tables, an Exasol user must have the `SELECT ANY DICTIONARY` system
@@ -69,10 +63,46 @@ GRANT SELECT ANY DICTIONARY TO ;
```
## Metadata Ingestion
-
-# Connection Options
-
-
+
+To ingest metadata from Exasol, you need to create a service connection. The service connects Exasol with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Exasol** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Exasol services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Exasol. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Connection Scheme**: Specifies the SQLAlchemy driver scheme options required to connect to Exasol.
- **Username**: The username used to connect to the Exasol database. Ensure that this user has sufficient privileges to read all the metadata from Exasol.
- **Password**: The password associated with the user connecting to Exasol.
@@ -88,9 +118,62 @@ GRANT SELECT ANY DICTIONARY TO ;
This mode should **never** be used in production and should only be used in debugging scenarios.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/gcs-datalake.mdx b/v2.0.x-SNAPSHOT/connectors/database/gcs-datalake.mdx
index 9a0b1de52..3e1004c9e 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/gcs-datalake.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/gcs-datalake.mdx
@@ -4,11 +4,7 @@ description: Connect your Google Cloud Storage data lake to OpenMetadata with ou
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
The GCS Datalake connector supports extracting metadata from file types `JSON`, `CSV`, `TSV` & `Parquet`.
+
## Metadata Ingestion
-
-## Connection Details
-
-
-- **Bucket Name**: A bucket name in DataLake is a unique identifier used to organize and store data objects.
- It's similar to a folder name, but it's used for object storage rather than file storage.
+
+To ingest metadata from GCS Datalake, you need to create a service connection. The service connects GCS Datalake with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **GCS Datalake** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other GCS Datalake services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for GCS Datalake. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Bucket Name**: A bucket name in DataLake is a unique identifier used to organize and store data objects. It's similar to a folder name, but it's used for object storage rather than file storage.
- **Prefix**: The prefix of a data source in datalake refers to the first part of the data path that identifies the source or origin of the data. It's used to organize and categorize data within the datalake, and can help users easily locate and access the data they need.
-**GCS Credentials**
-We support two ways of authenticating to GCS:
-1. Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery:
- 1. Credentials type, e.g. `service_account`.
- 2. Project ID
- 3. Private Key ID
- 4. Private Key
- 5. Client Email
- 6. Client ID
- 7. Auth URI, [https://accounts.google.com/o/oauth2/auth](https://accounts.google.com/o/oauth2/auth) by default
- 8. Token URI, `https://oauth2.googleapis.com/token` by default
- 9. Authentication Provider X509 Certificate URL, [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default
- 10. Client X509 Certificate URL
-
-
-
-
-
-
+- **GCS Credentials**: We support two ways of authenticating to GCS:
+ - Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery:
+ - **Credentials type**, for example, `service_account`.
+ - **Project ID**
+ - **Private Key ID**
+ - **Private Key**
+ - **Client Email**
+ - **Client ID**
+ - **Auth URI**, [https://accounts.google.com/o/oauth2/auth](https://accounts.google.com/o/oauth2/auth) by default
+ - **Token URI**, `https://oauth2.googleapis.com/token` by default
+ - **Authentication Provider X509 Certificate URL**, `https://www.googleapis.com/oauth2/v1/certs` by default
+ - **Client X509 Certificate URL**
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/glue.mdx b/v2.0.x-SNAPSHOT/connectors/database/glue.mdx
index 114da45f1..4b244dc26 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/glue.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/glue.mdx
@@ -4,10 +4,7 @@ description: Learn how to integrate AWS Glue with OpenMetadata using our compreh
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Glue, you need to create a service connection. The service connects Glue with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Glue** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Glue services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Glue. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+You can find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
@@ -55,11 +83,11 @@ You can find more information on [Using temporary credentials with AWS resources
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
@@ -78,9 +106,62 @@ Find more information about the [Role Session Name](https://docs.aws.amazon.com/
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/greenplum.mdx b/v2.0.x-SNAPSHOT/connectors/database/greenplum.mdx
index 48bf79398..f4792797e 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/greenplum.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/greenplum.mdx
@@ -4,11 +4,7 @@ description: Connect Greenplum to OpenMetadata with our comprehensive database c
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Greenplum, you need to create a service connection. The service connects Greenplum with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Greenplum** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Greenplum services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Greenplum. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Greenplum. It should have enough privileges to read all the metadata.
- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
- **Basic Auth**:
@@ -52,12 +86,12 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
- You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+ You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
- You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+ You can find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
@@ -89,27 +123,89 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
- **Host and Port**: Enter the fully qualified hostname and port number for your Greenplum deployment in the Host and Port field.
+
**SSL Modes**
+
There are a couple of types of SSL modes that Greenplum supports which can be added to ConnectionArguments, they are as follows:
+
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
- **prefer**: SSL is used if the server supports it.
- **require**: SSL is required.
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified, and the server hostname must match the hostname attribute on the certificate.
+
**SSL Configuration**
+
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under sslConfig which is placed in the source.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Greenplum Connection with SSL in OpenMetadata
+
To establish secure connections between OpenMetadata and a Greenplum database, you can configure SSL using different SSL modes provided by Greenplum, each offering varying levels of security.
+
Under `Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that Greenplum requires only the CA certificate for SSL validation.
+
For IAM authentication, it is recommended to choose the `allow` mode or another SSL mode that fits your specific requirements.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/hive.mdx b/v2.0.x-SNAPSHOT/connectors/database/hive.mdx
index e8be028ef..1d7be4bfb 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/hive.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/hive.mdx
@@ -4,11 +4,7 @@ description: Connect Apache Hive to OpenMetadata with our comprehensive database
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Hive, you need to create a service connection. The service connects Hive with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Hive** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Hive services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Hive. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Hive. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Hive.
- **Host and Port**: This parameter specifies the host and port of the Hive server instance. This should be specified as a string in the format `hostname:port`. For example, you might set the hostPort parameter to `myhivehost:10000`.
@@ -52,27 +83,86 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
- **Kerberos Service Name**: This parameter specifies the Kerberos service name to use for authentication. This should only be specified if using Kerberos authentication. The default value is `hive`.
- **Database Schema**: Schema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single schema. When left blank, OpenMetadata Ingestion attempts to scan all the schemas.
- **Database Name**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
-#### For MySQL Metastore Connection
+
+**For MySQL Metastore Connection**
+
You can also ingest the metadata using Mysql metastore. This step is optional if metastore details are not provided then we will query the hive server directly.
+
- **Username**: Specify the User to connect to MySQL Metastore. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to MySQL.
- **Host and Port**: Enter the fully qualified hostname and port number for your MySQL Metastore deployment in the Host and Port field in the format `hostname:port`.
- **databaseSchema**: Enter the database schema which is associated with the metastore.
-
-
-#### For Postgres Metastore Connection
+
+**For Postgres Metastore Connection**
+
You can also ingest the metadata using Postgres metastore. This step is optional if metastore details are not provided then we will query the hive server directly.
+
- **Username**: Specify the User to connect to Postgres Metastore. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Postgres.
- **Host and Port**: Enter the fully qualified hostname and port number for your Postgres deployment in the Host and Port field in the format `hostname:port`.
- **Database**: Initial Postgres database to connect to. Specify the name of database associated with metastore instance.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Hive Connection with SSL in OpenMetadata
-To configure SSL for secure connections between OpenMetadata and a Hive database, you need to add `ssl_cert` as a key and the path to the CA certificate as its value under `connectionArguments`. Ensure that the certificate is accessible by the server. If you use a Docker or Kubernetes deployment, update the CA certificate in the Open Metadata server.
+
+To configure SSL for secure connections between OpenMetadata and a Hive database, you need to add `ssl_cert` as a key and the path to the CA certificate as its value under `connectionArguments`. Ensure that the certificate is accessible by the server. If you use a Docker or Kubernetes deployment, update the CA certificate in the OpenMetadata server.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/impala.mdx b/v2.0.x-SNAPSHOT/connectors/database/impala.mdx
index 8937188cf..2a39e65f2 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/impala.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/impala.mdx
@@ -4,11 +4,7 @@ description: Connect Apache Impala to OpenMetadata seamlessly. Complete setup gu
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Impala connector.
Configure and schedule Impala metadata and profiler workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Data Profiler](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow)
-- [Lineage](/v2.0.x-SNAPSHOT/how-to-guides/data-lineage/workflow)
- [Data Quality](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality)
- [dbt Integration](/v2.0.x-SNAPSHOT/connectors/database/dbt)
- [Enable Security](#securing-impala-connection-with-ssl-in-openmetadata)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/impala/troubleshooting)
+
+## Requirements
+
+Before configuring the Impala connector, make sure the following requirements are in place.
+
+### Metadata
+
+To extract metadata, the user needs to have privileges to perform `SELECT`, `SHOW`, and `DESCRIBE` operations on the databases and tables.
+
+### Profiler & Data Quality
+
+For profiler workflow and data quality tests, the user needs `SELECT` permission on the tables/schemas where the profiler/tests will be executed.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Impala, you need to create a service connection. The service connects Impala with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Impala** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Impala services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Impala. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **username**: Specify the User to connect to Impala. It should have enough privileges to read all the metadata.
- **password**: Password to connect to Impala.
- **hostPort**: Enter the fully qualified hostname and port number for your Impala deployment in the Host and Port field.
@@ -49,13 +86,68 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
`Database Service > Database > Schema > Table`. In the case of Impala, we won't have a Database as such. If you'd like to see your data in a database named something other than `default`, you can specify the name in this field.
- **useSSL**: Establish secure connection with Impala. Enables SSL for the connector.
- **authOptions**: Enter the auth options string for impala connection.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Impala Connection with SSL in OpenMetadata
+
To configure SSL for secure connections between OpenMetadata and an Impala database, add the key `use_ssl` with a value of `true` to the `connectionArguments` to enable SSL. Additionally, include the key `ca_cert` with the path to the CA certificate file as its value. Ensure that the certificate file is accessible by the server, and if deploying via Docker or Kubernetes, update the CA certificate in the OpenMetadata server to reflect these changes.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/iomete.mdx b/v2.0.x-SNAPSHOT/connectors/database/iomete.mdx
index 4d6c9b218..6e45ac157 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/iomete.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/iomete.mdx
@@ -4,10 +4,7 @@ description: Connect IOMETE to OpenMetadata with our comprehensive database conn
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-- **Host and Port**: Host of the IOMETE service. You can optionally include the port using the `host:port` format (e.g. `dev.iomete.cloud:443`). If no port is specified, port `443` is used by default.
-- **Username**: Username to connect to IOMETE. This user should have privileges to read all the metadata in IOMETE.
+
+To ingest metadata from IOMETE, you need to create a service connection. The service connects IOMETE with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **IOMETE** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other IOMETE services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify where ingestion runs, provide your source credentials, and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for IOMETE. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Host and Port**: Host and port of the IOMETE service, for example `dev.iomete.cloud:443`.
+- **Username**: Username to connect to IOMETE.
+- **Cluster**: IOMETE lakehouse cluster name to connect to.
+- **Data Plane**: IOMETE data plane name.
+
+**Authentication**
- **Password**: Password to connect to IOMETE.
-- **Cluster**: IOMETE lakehouse cluster name to connect to. This is passed as the `cluster` query parameter in the connection URL.
-- **Data Plane**: IOMETE data plane name. This is passed as the `data_plane` query parameter in the connection URL (e.g. `default`).
-- **Catalog** *(Optional)*: Catalog of the data source (e.g. `spark_catalog`). If left blank, OpenMetadata uses the default catalog.
-- **Database Schema** *(Optional)*: IOMETE database (schema) to restrict metadata ingestion to (e.g. `default`, `finance_db`). If left blank, OpenMetadata attempts to scan all schemas in the catalog.
-- **Storage Config for Sample Data** *(Optional)*: Configure where OpenMetadata stores sample data rows collected during profiling.
-
-
-
-
-
-
\ No newline at end of file
+
+**Scope & Options**
+- **Catalog**: Catalog of the data source (for example, `spark_catalog`). This is an optional parameter; if left blank, OpenMetadata uses the default catalog.
+- **Database Schema**: IOMETE database to restrict metadata ingestion to (for example, `default`, `finance_db`). This is an optional parameter; if left blank, OpenMetadata attempts to scan all databases in the catalog.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Storage Config for Sample Data**: Configure where OpenMetadata stores sample data rows collected during profiling.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/iomete/yaml.mdx b/v2.0.x-SNAPSHOT/connectors/database/iomete/yaml.mdx
index 9f7a37935..9dfaa36db 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/iomete/yaml.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/iomete/yaml.mdx
@@ -64,7 +64,7 @@ Configure the source type and service name for your IOMETE connector.
-**hostPort**: Host of the IOMETE service. You can optionally include the port using the `host:port` format (e.g. `dev.iomete.cloud:443`). If no port is specified, port `443` is used by default.
+**hostPort**: Host of the IOMETE service. You can optionally include the port using the `host:port` format (for example, `dev.iomete.cloud:443`). If no port is specified, port `443` is used by default.
@@ -88,13 +88,13 @@ Configure the source type and service name for your IOMETE connector.
-**dataPlane**: IOMETE data plane name. This is passed as the `data_plane` query parameter in the connection URL (e.g. `default`).
+**dataPlane**: IOMETE data plane name. This is passed as the `data_plane` query parameter in the connection URL (for example, `default`).
-**catalog** (Optional): Catalog of the data source (e.g. `spark_catalog`). If left blank, OpenMetadata uses the default catalog.
+**catalog** (Optional): Catalog of the data source (for example, `spark_catalog`). If left blank, OpenMetadata uses the default catalog.
**databaseSchema** (Optional): IOMETE database (schema) to restrict metadata ingestion to. If left blank, OpenMetadata attempts to scan all schemas in the catalog.
diff --git a/v2.0.x-SNAPSHOT/connectors/database/mariadb.mdx b/v2.0.x-SNAPSHOT/connectors/database/mariadb.mdx
index 79dee9e4a..2ee724ae8 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/mariadb.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/mariadb.mdx
@@ -4,11 +4,7 @@ description: Connect MariaDB to OpenMetadata with our comprehensive database con
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
[@] IDENTIFIED BY '';
@@ -38,30 +40,113 @@ GRANT SELECT ON world.* TO '';
-- Grant select on a specific object
GRANT SELECT ON world.hello TO '';
```
+
### Profiler & Data Quality
+
Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from MariaDB, you need to create a service connection. The service connects MariaDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **MariaDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other MariaDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for MariaDB. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to MariaDB. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to MariaDB.
- **Host and Port**: Enter the fully qualified hostname and port number for your MariaDB deployment in the Host and Port field.
- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
- **databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/mongodb.mdx b/v2.0.x-SNAPSHOT/connectors/database/mongodb.mdx
index cc04aa38e..552df97cf 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/mongodb.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/mongodb.mdx
@@ -1,14 +1,10 @@
---
title: MongoDB Connector | OpenMetadata NoSQL Database Integration
-description: Connect MongoDB to OpenMetadata with our comprehensive database connector guide. Step-by-step setup, configuration, and metadata extraction for seamless integration.
+description: Connect MongoDB to OpenMetadata with this database connector guide. Step-by-step setup, configuration, and metadata extraction.
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
+
+To ingest metadata from MongoDB, you need to create a service connection. The service connects MongoDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **MongoDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other MongoDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for MongoDB. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Username**: Username to connect to MongoDB. This user must have access to perform the `find` operation on collections and the `listCollection` operation on databases available in MongoDB.
+- **Password**: Password to connect to MongoDB.
+- **Host Port**: When using the `mongodb` connection schema, the hostPort parameter specifies the host and port of MongoDB. Specify it as a string in the format `hostname:port`. For example, `localhost:27017`. When using the `mongodb+srv` connection schema, the hostPort parameter specifies the host of MongoDB. Specify it as a string in the format `hostname`. For example, `cluster0-abcde.mongodb.net`.
+- **Database Name**: Optional name to give to the database in OpenMetadata. If left blank, OpenMetadata uses the default database name.
+
+Using Atlas? Follow the [MongoDB Atlas connection string guide](https://www.mongodb.com/docs/guides/atlas/connection-string/) to get the connection string.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
-- **Username**: Username to connect to Mongodb. This user must have access to perform `find` operation on collection and `listCollection` operations on database available in MongoDB.
-- **Password**: Password to connect to MongoDB.
-- **Host Port**: When using the `mongodb` connecion schema, the hostPort parameter specifies the host and port of the MongoDB. This should be specified as a string in the format `hostname:port`. E.g., `localhost:27017`. When using the `mongodb+srv` connection schema, the hostPort parameter specifies the host and port of the MongoDB. This should be specified as a string in the format `hostname`. E.g., `cluster0-abcde.mongodb.net`.
-- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
-Using Atlas? Follow [this guide](https://www.mongodb.com/docs/guides/atlas/connection-string/) to get the connection string.
-
-
-
-
-
-
-
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose one of the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips. An asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+## Data Profiler
+
+[Profiler deployment](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow)
+
### Limitations
-The MongodDB data profiler current supports only the following features:
-1. **Row count**: The number of rows in the collection. Sampling or custom query is not supported.
-2. **Sample data:** If a custom query is defined it will be used for sample data.
+
+The MongoDB data profiler currently supports only the following features:
+1. **Row count**: The number of rows in the collection. Sampling or a custom query is not supported.
+2. **Sample data**: If a custom query is defined, it will be used for sample data.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx b/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx
index 74b2da365..497b56359 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/mssql.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the MSSQL connector.
Configure and schedule MSSQL metadata and profiler workflows from the OpenMetadata UI:
-- [Remote-Connection](#for-remote-connection)
+- [Remote Connection](#for-remote-connection)
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Query Usage](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage)
@@ -31,6 +27,7 @@ Configure and schedule MSSQL metadata and profiler workflows from the OpenMetada
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/mssql/troubleshoot)
## Requirements
+
MSSQL User must grant `SELECT` privilege to fetch the metadata of tables and views.
```sql
-- Create a new user
@@ -41,6 +38,8 @@ GRANT SELECT TO Mary;
```
### View Definitions
+Capturing view lineage depends on an additional server or database permission.
+
**View lineage requires** the `VIEW DEFINITION` (database-level) or `VIEW ANY DEFINITION` (server-level) permission. Without it, SQL Server returns each view's definition as `NULL` and raises no error, so views are ingested without their SQL and **no view lineage is created**. Table metadata is not affected.
@@ -51,13 +50,13 @@ GRANT VIEW DEFINITION TO Mary;
-- Or server-level: grant to the login (not the contained user) to cover all databases
GRANT VIEW ANY DEFINITION TO [login_name];
```
-### Usage & Lineage consideration
+### Usage & Lineage Consideration
To perform the query analysis for Usage and Lineage computation, the connector reads query history from SQL Server's [Query Store](https://learn.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store) when it is enabled on a database. Query Store exposes this history through `sys.query_store_query`, `sys.query_store_query_text`, `sys.query_store_plan`, and `sys.query_store_runtime_stats`. Query Store keeps a durable, on-disk record of executed queries. The plan-cache Dynamic Management Views (DMVs) (`sys.dm_exec_cached_plans`, `sys.dm_exec_query_stats`, and `sys.dm_exec_sql_text`) are evicted on server restart, memory pressure, or plan recompilation, and can silently return incomplete history.
Query Store is auto-detected per database. No configuration is needed. When `ingestAllDatabases` is enabled, each database is read from its own best source independently: databases with Query Store use it, and the rest fall back to the plan-cache DMVs, so one database without Query Store never downgrades the others.
-**Tip:** Enable Query Store on the databases you ingest, for reliable and durable Usage and Lineage history.
+**Tip**: Enable Query Store on the databases you ingest, for reliable and durable Usage and Lineage history.
To check whether Query Store is enabled on a database, and how it is currently operating:
@@ -91,46 +90,139 @@ GRANT VIEW SERVER STATE TO YourUser;
```
Exact grant names and tiers can differ on Azure SQL Database and Azure SQL Managed Instance. Check `sys.database_query_store_options` and the [Microsoft documentation](https://learn.microsoft.com/en-us/sql/relational-databases/system-catalog-views/sys-query-store-query-text-transact-sql) for your instance.
### For Remote Connection
-#### 1. SQL Server running
-Make sure the SQL server that you are trying to connect is in running state.
-#### 2. Allow remote connection on MSSMS(Microsoft SQL Server Management Studio)
-This step allow the sql server to accept remote connection request.
+
+If the ingestion runner connects to MSSQL over the network rather than locally, work through the following checks in order.
+
+#### 1. SQL Server Running
+Make sure the SQL Server instance you are trying to connect to is in a running state.
+#### 2. Allow Remote Connection on SSMS (Microsoft SQL Server Management Studio)
+This step allows the SQL Server to accept remote connection requests.
+
+
#### 3. Configure Windows Firewall
-If you are using SQL server on windows, you must configure the firewall on the computer running SQL Server to allow access.
-**Step 1**: On the Start menu, select Run, type WF.msc, and then select OK.
-**Step 2**: In the Windows Firewall with Advanced Security, in the left pane, right-click Inbound Rules, and then select New Rule in the action pane.
-**Step 3**: In the Rule Type dialog box, select Port, and then select Next.
-**Step 4**: In the Protocol and Ports dialog box, select TCP. Select Specific local ports, and then type the port number of the instance of the Database Engine, such as 1433 for the default instance. Select Next.
-**Step 5**: In the Action dialog box, select Allow the connection, and then select Next.
-**Step 6**: In the Profile dialog box, select any profiles that describe the computer connection environment when you want to connect to the Database Engine, and then select Next.
-**Step 7**: In the Name dialog box, type a name and description for this rule, and then select Finish.
-For details step please refer to this [link](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-a-windows-firewall-for-database-engine-access?view=sql-server-ver15).
+If you are using SQL Server on Windows, you must configure the firewall on the computer running SQL Server to allow access.
+1. On the Start menu, select Run, type WF.msc, and then select OK.
+2. In the Windows Firewall with Advanced Security, in the left pane, right-click Inbound Rules, and then select New Rule in the action pane.
+3. In the Rule Type dialog box, select Port, and then select Next.
+4. In the Protocol and Ports dialog box, select TCP. Select Specific local ports, and then type the port number of the instance of the Database Engine, such as 1433 for the default instance. Select Next.
+5. In the Action dialog box, select Allow the connection, and then select Next.
+6. In the Profile dialog box, select any profiles that describe the computer connection environment when you want to connect to the Database Engine, and then select Next.
+7. In the Name dialog box, type a name and description for this rule, and then select Finish.
+For details, see this [link](https://docs.microsoft.com/en-us/sql/database-engine/configure-windows/configure-a-windows-firewall-for-database-engine-access?view=sql-server-ver15).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Connection Scheme**: Defines how to connect to MSSQL. We support `mssql+pytds`, `mssql+pyodbc`, and `mssql+pymssql`. (If you are using windows authentication from a linux deployment please use pymssql)
+
+To ingest metadata from MSSQL, you need to create a service connection. The service connects MSSQL with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **MSSQL** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other MSSQL services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for MSSQL. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Connection Scheme**: Defines how to connect to MSSQL. OpenMetadata supports `mssql+pytds`, `mssql+pyodbc`, and `mssql+pymssql`. If you are using Windows Authentication from a Linux deployment, use `pymssql`.
- **Username**: Specify the User to connect to MSSQL. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to MSSQL.
- **Host and Port**: Enter the fully qualified hostname and port number for your MSSQL deployment in the Host and Port field.
- **URI String**: In case of a `pyodbc` connection.
- **Database**: The initial database to establish a connection to the data source.
-- **Ingest All Databases**: If you need to ingest multiple databases - aside from the initial one above - you can enable this option.
-
-
-
-
-
-
+- **Ingest All Databases**: If you need to ingest multiple databases, aside from the initial one above, enable this option.
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/mysql.mdx b/v2.0.x-SNAPSHOT/connectors/database/mysql.mdx
index 55662c158..49de85ef1 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/mysql.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/mysql.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
is omitted, defaults to '%'
-- More details https://dev.mysql.com/doc/refman/8.0/en/create-user.html
@@ -62,9 +61,9 @@ GRANT SELECT ON mysql.slow_log TO ''@'';
The `mysql.general_log` table grows continuously as it stores query logs. This can consume significant storage space over time and affect the execution time of lineage and usage procedures.
- Note: We recommend cleaning up log tables only after successful execution of Usage & Lineage workflows to ensure no loss of query data during extraction. Once cleanup occurs, the query history is lost.
Here are some important considerations and best practices:
-### Create Manual Schedule to rotate logs
+### Create Manual Schedule to Rotate Logs
When you rotate log tables manually, the current log table is copied to a backup log table and the entries in the current log table are removed. If the backup log table already exists, then it is deleted before the current log table is copied to the backup. You can query the backup log table if needed. The backup log table for the `mysql.general_log` table is named `mysql.general_log_backup`.
-The backup log table for the `mysql.slow_log table` is named `mysql.slow_log_backup`
+The backup log table for the `mysql.slow_log` table is named `mysql.slow_log_backup`.
```sql
-- rotate general logs
CREATE PROCEDURE rotate_general_log()
@@ -88,19 +87,19 @@ END
-- call this procedure
CALL rotate_slow_log();
```
-You can also check table size by running below query
+You can also check table size by running the query below:
```sql
SELECT table_name, round(data_length/1024/1024, 2) AS size_in_mb
FROM information_schema.tables
WHERE table_schema = 'mysql' AND table_name IN ('general_log', 'slow_log', 'general_log_backup', 'slow_log_backup');
```
-### Create Automatic Event to clear older logs
-You can also create automatic event like showed below which runs every week to clear older logs.
+### Create Automatic Event to Clear Older Logs
+You can also create an automatic event like the one shown below, which runs every week to clear older logs.
```sql
CREATE EVENT mysql.cleanup_general_log ON SCHEDULE EVERY 7 DAY DO DELETE FROM mysql.general_log WHERE event_time < NOW() - INTERVAL 7 DAY;
```
-Note: If you are using rds then you can rotate the `mysql.general_log` table manually by calling the `mysql.rds_rotate_general_log` procedure. You can rotate the `mysql.slow_log` table by calling the `mysql.rds_rotate_slow_log` procedure.
-You can also check below docs about more info on logs & its rotation methods.
+Note: If you are using RDS, you can rotate the `mysql.general_log` table manually by calling the `mysql.rds_rotate_general_log` procedure. You can rotate the `mysql.slow_log` table by calling the `mysql.rds_rotate_slow_log` procedure.
+You can also check the docs below for more info on logs and their rotation methods:
- [Rotating mysql query logs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-stored-proc-logging.html)
- [RDS for MySQL database logs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.MySQL.LogFileSize.html#USER_LogAccess.MySQL.Generallog)
- [Aurora for MySQL database logs](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_LogAccess.MySQL.LogFileSize.html#USER_LogAccess.MySQL.Generallog)
@@ -110,20 +109,49 @@ You can also check below docs about more info on logs & its rotation methods.
- Consider automating log cleanup after DAG execution
- Keep logging enabled only when needed for lineage extraction
### Profiler & Data Quality
-Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+Executing the profiler workflow or data quality tests will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from MySQL, you need to create a service connection. The service connects MySQL with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **MySQL** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other MySQL services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for MySQL. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to MySQL. It should have enough privileges to read all the metadata.
- **Auth Type**: Basic Auth, IAM based auth, or Azure auth.
- **Basic Auth**:
@@ -136,7 +164,7 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
- As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+ As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
@@ -179,14 +207,74 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
- **caCertificate**: Provide the path to ssl ca file.
- **sslCertificate**: Provide the path to ssl client certificate file (ssl_cert).
- **sslKey**: Provide the path to ssl client certificate file (ssl_key).
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips. An asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing MySQL Connection with SSL in OpenMetadata
-To establish secure connections between OpenMetadata and MySQL, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server's certificate.
+To establish secure connections between OpenMetadata and MySQL, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server's certificate.
diff --git a/v2.0.x-SNAPSHOT/connectors/database/oracle.mdx b/v2.0.x-SNAPSHOT/connectors/database/oracle.mdx
index df8c76db1..2d9b7ef6d 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/oracle.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/oracle.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Oracle, you need to create a service connection. The service connects Oracle with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Oracle** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Oracle services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Oracle. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Oracle. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Oracle.
- **Host and Port**: Enter the fully qualified hostname and port number for your Oracle deployment in the Host and Port field.
-- **Database Name**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name. It is recommended to use the database name same as the SID, This ensures accurate results and proper identification of tables during profiling, data quality checks and dbt workflow.
-- **Oracle Connection Type** : Select the Oracle Connection Type. The type can either be `Oracle Service Name` or `Database Schema`
- - **Oracle Service Name**: The Oracle Service name is the TNS alias that you give when you remotely connect to your database and this Service name is recorded in tnsnames.
+- **Database Name**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name. It is recommended to use the database name same as the SID, this ensures accurate results and proper identification of tables during profiling, data quality checks, and dbt workflow.
+- **Oracle Connection Type**: Select the Oracle Connection Type. The type can either be `Oracle Service Name` or `Database Schema`.
+ - **Oracle Service Name**: The Oracle Service name is the TNS alias that you give when you remotely connect to your database and this service name is recorded in `tnsnames.ora`.
- **Database Schema**: The name of the database schema available in Oracle that you want to connect with.
- **Oracle instant client directory**: The directory pointing to where the `instantclient` binaries for Oracle are located. In the ingestion Docker image we
- provide them by default at `/instantclient`. If this parameter is informed (it is by default), we will run the [thick oracle client](https://python-oracledb.readthedocs.io/en/latest/user_guide/initialization.html#initializing-python-oracledb).
- We are shipping the binaries for ARM and AMD architectures from [here](https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html)
+ provide them by default at `/instantclient`. If this parameter is informed (it is by default), we will run the [thick Oracle client](https://python-oracledb.readthedocs.io/en/latest/user_guide/initialization.html#initializing-python-oracledb).
+ We ship the binaries for ARM and AMD architectures from [here](https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html)
and [here](https://www.oracle.com/database/technologies/instant-client/linux-arm-aarch64-downloads.html) for the instant client version 19.
-- **Use DBA Table (toggle)**: When enabled, connector uses `DBA_TABLES` to retrieve metadata for all database objects. This requires the Oracle user to have DBA privileges. When disabled, connector uses `ALL_TABLES`, which only returns metadata for objects accessible to the current user.
-
-
-
-
-
-
+- **Use DBA Tables**: Enabled by default. When enabled, the connector uses `DBA_TABLES` to retrieve metadata for all database objects, which requires the Oracle user to have DBA privileges. When disabled, the connector uses `ALL_TABLES`, which only returns metadata for objects accessible to the current user.
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose one of the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips. An asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/pinotdb.mdx b/v2.0.x-SNAPSHOT/connectors/database/pinotdb.mdx
index 11a19cb23..ed5a07c9f 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/pinotdb.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/pinotdb.mdx
@@ -3,12 +3,9 @@ title: PinotDB Connector | OpenMetadata Real-Time Analytics Guide
description: Connect Apache Pinot to OpenMetadata with our comprehensive database connector guide. Setup instructions, configuration examples, and metadata extraction tips.
sidebarTitle: Overview
---
+
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the PinotDB connector.
Configure and schedule PinotDB metadata and profiler workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Data Profiler](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow)
- [Data Quality](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality)
- [dbt Integration](/v2.0.x-SNAPSHOT/connectors/database/dbt)
- [Lineage](/v2.0.x-SNAPSHOT/how-to-guides/data-lineage/workflow)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/pinotdb/troubleshooting)
+
+## Requirements
+
+Before configuring the PinotDB connector, make sure the following requirements are in place.
+
+### Metadata
+To extract metadata, the user needs access to the Pinot broker and controller APIs. The user should have permissions to query the controller for table schemas and the broker for query execution.
+### Profiler & Data Quality
+Executing the profiler workflow or data quality tests will require the user to have query execution permissions on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from PinotDB, you need to create a service connection. The service connects PinotDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **PinotDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other PinotDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for PinotDB. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to PinotDB. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to PinotDB.
-- **Host and Port**: Enter the fully qualified hostname and port number for your PinotDB deployment in the Host and Port field. Unlike broker host, prefix http:// or https:// must be added to controller host. For example, pinot broker host can be set to `localhost:8099` and pinot controller host can be set to `http://localhost:9000`.
+- **Host and Port**: Enter the fully qualified hostname and port number for your PinotDB deployment in the Host and Port field. Unlike the broker host, the prefix `http://` or `https://` must be added to the controller host. For example, the Pinot broker host can be set to `localhost:8099` and the Pinot controller host can be set to `http://localhost:9000`.
- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
- **databaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
- **caCertificate**: Provide the path to ssl ca file.
- **sslCertificate**: Provide the path to ssl client certificate file (ssl_cert).
- **sslKey**: Provide the path to ssl client certificate file (ssl_key).
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/postgres.mdx b/v2.0.x-SNAPSHOT/connectors/database/postgres.mdx
index f9b7b8ae8..3104330dd 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/postgres.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/postgres.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
Note that we only support officially supported PostgreSQL versions. You can check the version list [here](https://www.postgresql.org/support/versioning/).
@@ -57,33 +56,33 @@ GRANT pg_read_all_stats TO your_user;
You can find more information on the usage workflow [here](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage) and the lineage workflow [here](/v2.0.x-SNAPSHOT/connectors/ingestion/lineage).
-`pg_stat_statements` is **not a query log** — it is a fixed-size, in-memory statistics collector. This has important implications for lineage and usage completeness:
+`pg_stat_statements` is **not a query log**, it is a fixed-size, in-memory statistics collector. This has important implications for lineage and usage completeness:
- **Entry eviction**: It stores entries in a hash table controlled by `pg_stat_statements.max` (default: **5000**). When full, the least-executed entries are silently evicted. Queries can disappear before OpenMetadata reads them.
-- **Query normalization**: Queries are deduplicated by shape — literal values are replaced with placeholders (e.g., `SELECT * FROM users WHERE id = $1`). Individual executions are not stored.
-- **No timestamps**: There is no time data. Only cumulative `calls` count and `total_exec_time` since the last reset are tracked. The **query log duration** setting in OpenMetadata will have no impact — only the **query limit** matters.
+- **Query normalization**: Queries are deduplicated by shape, literal values are replaced with placeholders (for example, `SELECT * FROM users WHERE id = $1`). Individual executions are not stored.
+- **No timestamps**: There is no time data. Only cumulative `calls` count and `total_exec_time` since the last reset are tracked. The **query log duration** setting in OpenMetadata will have no impact, only the **query limit** matters.
- **Resets clear everything**: A server restart or `pg_stat_statements_reset()` wipes all entries.
To get the most complete lineage and usage data, we recommend:
- **Increase `pg_stat_statements.max`** to at least `10000` (or higher) in `postgresql.conf` to reduce entry eviction.
- **Set `pg_stat_statements.track = 'all'`** in `postgresql.conf` to capture queries inside functions and procedures.
-- **Schedule frequent ingestion runs** (e.g., every 1–2 hours) to capture queries before they are evicted.
+- **Schedule frequent ingestion runs** (for example, every 1-2 hours) to capture queries before they are evicted.
- **Avoid calling `pg_stat_statements_reset()`** before ingestion. If periodic resets are needed, schedule them after ingestion completes.
-If your organization restricts direct access to `pg_stat_statements`, or if you need more control over query retention, you can set the **Query Statement Source** connection property to a custom view or table (e.g., `my_schema.custom_query_history`). The custom source must expose the same columns that OpenMetadata reads from `pg_stat_statements`: `userid` (oid), `dbid` (oid), `query` (text), and either `total_exec_time` (double precision, PostgreSQL 13+) or `total_time` (double precision, PostgreSQL < 13). If not set, OpenMetadata defaults to `pg_stat_statements`.
+If your organization restricts direct access to `pg_stat_statements`, or if you need more control over query retention, you can set the **Query Statement Source** connection property to a custom view or table (for example, `my_schema.custom_query_history`). The custom source must expose the same columns that OpenMetadata reads from `pg_stat_statements`: `userid` (oid), `dbid` (oid), `query` (text), and either `total_exec_time` (double precision, PostgreSQL 13+) or `total_time` (double precision, PostgreSQL < 13). If not set, OpenMetadata defaults to `pg_stat_statements`.
### IAM Authentication
In order to be able to connect via IAM, you need to have the following:
-1. Database is configured to use IAM authentication
+1. Database is configured to use IAM authentication.
Ensure that the RDS has IAM DB authentication enabled. Otherwise, you can click on Modify to enable it.
-2. The user has the necessary IAM permissions
-Even if you use IAM to connect to postgres, you need to specify a user to prepare the connection. You need to create a user as follows:
+2. The user has the necessary IAM permissions.
+Even if you use IAM to connect to PostgreSQL, you need to specify a user to prepare the connection. You need to create a user as follows:
```sql
CREATE USER iam_user WITH LOGIN;
GRANT rds_iam TO iam_user;
```
-3. The AWS Role has the necessary permissions
-The role that is going to be used to perform the ingestion, needs to have the following permissions:
+3. The AWS Role has the necessary permissions.
+The role that is going to be used to perform the ingestion needs to have the following permissions:
```json
{
"Version": "2012-10-17",
@@ -106,33 +105,62 @@ PAM authentication failed for user "<user>"
When executing stored procedures in PostgreSQL, lineage extraction relies on capturing the SQL queries executed within the procedure. However, by default, PostgreSQL does not track the internal queries of a stored procedure in `pg_stat_statements`.
### Enabling Query Tracking for Lineage
To ensure OpenMetadata captures lineage from stored procedures, follow these steps:
-1. **Enable Logging for All Statements**
+1. **Enable Logging for All Statements**.
Modify the `postgresql.conf` file and set:
- ```ini
- log_statement = 'all'
- ```
- This will log all executed SQL statements, including those inside stored procedures.
-2. **Configure `pg_stat_statements` to Track Nested Queries**
+ ```ini
+ log_statement = 'all'
+ ```
+This will log all executed SQL statements, including those inside stored procedures.
+2. **Configure `pg_stat_statements` to Track Nested Queries**.
By default, `pg_stat_statements` may only capture top-level procedure calls and not the internal queries. To change this behavior, update:
- ```ini
- pg_stat_statements.track = 'all'
- ```
+ ```ini
+ pg_stat_statements.track = 'all'
+ ```
This ensures that statements executed within procedures are recorded.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from PostgreSQL, you need to create a service connection. The service connects PostgreSQL with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **PostgreSQL** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other PostgreSQL services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for PostgreSQL. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to PostgreSQL. It should have enough privileges to read all the metadata.
-- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud rds.
+- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud RDS.
- **Basic Auth**:
- **Password**: Password to connect to PostgreSQL.
- **IAM Based Auth**:
@@ -143,7 +171,7 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
- As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+ As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
@@ -178,7 +206,8 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
- **Host and Port**: Enter the fully qualified hostname and port number for your PostgreSQL deployment in the Host and Port field.
-**SSL Modes**
+
+#### SSL Modes
There are a couple of types of SSL modes that PostgreSQL supports which can be added to ConnectionArguments, they are as follows:
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
@@ -186,14 +215,75 @@ There are a couple of types of SSL modes that PostgreSQL supports which can be a
- **require**: SSL is required.
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified, and the server hostname must match the hostname attribute on the certificate.
-**SSL Configuration**
+
+#### SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under sslConfig which is placed in the source.
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing PostgreSQL Connection with SSL in OpenMetadata
To establish secure connections between OpenMetadata and a PostgreSQL database, you can configure SSL using different SSL modes provided by PostgreSQL, each offering varying levels of security.
Under `Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that PostgreSQL requires only the CA certificate for SSL validation.
diff --git a/v2.0.x-SNAPSHOT/connectors/database/presto.mdx b/v2.0.x-SNAPSHOT/connectors/database/presto.mdx
index 3f67fdeed..bfc89fbbe 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/presto.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/presto.mdx
@@ -1,14 +1,11 @@
---
title: Presto Connector | OpenMetadata Distributed SQL Guide
-description: Connect Presto to OpenMetadata with our comprehensive database connector guide. Step-by-step setup, configuration, and metadata extraction for seamless integration.
+description: Connect Presto to OpenMetadata with this database connector guide. Step-by-step setup, configuration, and metadata extraction.
sidebarTitle: Overview
---
+
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Options
-
-
+
+To ingest metadata from Presto, you need to create a service connection. The service connects Presto with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Presto** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Presto services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Presto. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Presto. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Presto.
- **Host and Port**: Enter the fully qualified hostname and port number for your Presto deployment in the Host and Port field.
- **Catalog**: Presto offers a catalog feature where all the databases are stored.
- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/questdb.mdx b/v2.0.x-SNAPSHOT/connectors/database/questdb.mdx
index 9de486d6b..faacd931f 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/questdb.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/questdb.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+**Note**: On QuestDB Enterprise, role-based access control (RBAC) is enforced at the connection endpoint. In addition to valid credentials, the service account must be granted PGWIRE endpoint access, or the connection is refused even though authentication succeeds:
+
+```sql
+GRANT PGWIRE TO '';
+```
+
## Metadata Ingestion
-
+To ingest metadata from QuestDB, you need to create a service connection. The service connects QuestDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
-# Connection Details
+### Step 1: Add New Service
-
-
+1. Navigate to **Settings** > **Services**.
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields such as passwords, API keys, or private keys must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **QuestDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other QuestDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for QuestDB. The right-hand panel in the UI displays inline help for each field.
+
+
- **Username**: Username to connect to QuestDB. The user should have enough privileges to read all the metadata.
- **Password**: Password for the QuestDB user.
-- **Host and Port**: Host and port of the QuestDB service. QuestDB exposes the PostgreSQL wire protocol on port **8812** by default (e.g., `localhost:8812`).
-- **Connection Options (Optional)**: Additional connection options to pass to the database during connection. These details must be added as Key-Value pairs.
-- **Connection Arguments (Optional)**: Additional connection arguments such as security or protocol configs. These details must be added as Key-Value pairs.
-
-
-
-
-
-
-
+- **Host and Port**: Host and port of the QuestDB service. QuestDB exposes the PostgreSQL wire protocol on port **8812** by default (for example, `localhost:8812`).
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/redshift.mdx b/v2.0.x-SNAPSHOT/connectors/database/redshift.mdx
index 506c7ab3b..2c0bbaff1 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/redshift.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/redshift.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Supported Authentication Types:**
-- **Basic Auth** — Username and password authentication
-- **IAM Auth** — AWS IAM-based authentication with automatic temporary credential retrieval (supports both Provisioned Clusters and Serverless Workgroups)
+- **Basic Auth**: Username and password authentication
+- **IAM Auth**: AWS IAM-based authentication with automatic temporary credential retrieval (supports both Provisioned Clusters and Serverless Workgroups)
Configure and schedule Redshift metadata and profiler workflows from the OpenMetadata UI:
@@ -39,6 +35,9 @@ Configure and schedule Redshift metadata and profiler workflows from the OpenMet
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/redshift/troubleshooting)
## Requirements
+
+Before configuring the Redshift connector, make sure the following requirements are in place.
+
### Metadata
Redshift user must grant `SELECT` privilege on table [SVV_TABLE_INFO](https://docs.aws.amazon.com/redshift/latest/dg/r_SVV_TABLE_INFO.html) to fetch the metadata of tables and views. For more information visit [here](https://docs.aws.amazon.com/redshift/latest/dg/c_visibility-of-data.html).
```sql
@@ -58,7 +57,7 @@ GRANT USAGE ON SCHEMA "" TO ;
```
### Profiler & Data Quality
-Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. For system-level profiling metrics on Provisioned Clusters, the user also needs:
+Executing the profiler workflow or data quality tests will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. The user should also be allowed to view information in `SVV_TABLE_INFO` for all objects in the database. For system-level profiling metrics on Provisioned Clusters, the user also needs:
```sql
GRANT SELECT ON TABLE pg_catalog.stl_insert TO ;
GRANT SELECT ON TABLE pg_catalog.stl_delete TO ;
@@ -130,35 +129,67 @@ To use IAM authentication instead of basic username/password, ensure the IAM rol
The connector automatically detects whether your Redshift endpoint is Provisioned or Serverless based on the hostname pattern and uses the appropriate API for credential retrieval.
+
## Metadata Ingestion
-
+
+To ingest metadata from Redshift, you need to create a service connection. The service connects Redshift with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Redshift** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Redshift services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Redshift. The right-hand panel in the UI displays inline help for each field.
+
+
+
It is recommended to exclude the schema "information_schema" from the metadata ingestion as it contains system tables and views.
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
- **Username**: Specify the User to connect to Redshift. It should have enough privileges to read all the metadata.
- **Auth Type**: Choose the authentication method:
- **Basic Auth**: Use username and password for authentication.
- **Password**: Password to connect to Redshift.
- **IAM Auth**: Use AWS IAM for authentication. This retrieves temporary credentials automatically.
- - **AWS Region**: AWS region where your Redshift cluster/workgroup is located (e.g., `us-east-1`).
- - **AWS Access Key ID** (Optional): Access key for the IAM user/role. If not provided, the connector uses the default AWS credentials chain (environment variables, instance profile, etc.).
+ - **AWS Region**: AWS region where your Redshift cluster/workgroup is located (for example, `us-east-1`).
+ - **AWS Access Key ID** (Optional): Access key for the IAM user/role. If not provided, the connector uses the default AWS credentials chain (environment variables, instance profile, and so on).
- **AWS Secret Access Key** (Optional): Secret key corresponding to the Access Key ID.
- **AWS Session Token** (Optional): Session token for temporary credentials.
- **Assume Role ARN** (Optional): ARN of a role to assume for cross-account access.
- **Assume Role Session Name** (Optional): Session name when assuming a role.
- **Database**: The database of the data source is an optional parameter, if you would like to restrict the metadata reading to a single database. If left blank, OpenMetadata ingestion attempts to scan all the databases.
-**SSL Configuration**
+
+#### SSL Configuration
In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under connectionArguments which is placed in the source.
-**SSL Modes**
+
+#### SSL Modes
There are a couple of types of SSL modes that Redshift supports which can be added to ConnectionArguments, they are as follows:
- **disable**: SSL is disabled and the connection is not encrypted.
- **allow**: SSL is used if the server requires it.
@@ -167,12 +198,72 @@ There are a couple of types of SSL modes that Redshift supports which can be add
- **verify-ca**: SSL must be used and the server certificate must be verified.
- **verify-full**: SSL must be used. The server certificate must be verified and the server hostname must match the hostname attribute on the certificate.
For more information, you can visit [Redshift SSL documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-ssl-support.html)
-
-
-
-
-
-
+
+#### Advanced Configuration
+
+Database Services have an Advanced Configuration section, where you can pass extra arguments to the connector and, if needed, change the connection scheme. This would only be required to handle advanced connectivity scenarios or customizations.
+
+- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to database during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Redshift Connection with SSL in OpenMetadata
To establish secure connections between OpenMetadata and a Redshift database, you can configure SSL using different SSL modes provided by Redshift, each offering varying levels of security.
Under `Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that Redshift requires only the CA certificate for SSL validation.
diff --git a/v2.0.x-SNAPSHOT/connectors/database/s3-datalake.mdx b/v2.0.x-SNAPSHOT/connectors/database/s3-datalake.mdx
index 03b5adc3c..6686bdd5b 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/s3-datalake.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/s3-datalake.mdx
@@ -4,11 +4,7 @@ description: Connect your S3 data lake to OpenMetadata with our comprehensive da
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
The S3 Datalake connector supports extracting metadata from file types `JSON`, `CSV`, `TSV` & `Parquet`.
+
### S3 Permissions
-To execute metadata extraction AWS account should have enough access to fetch required data. The Bucket Policy in AWS requires at least these permissions:
+
+To execute metadata extraction AWS account should have enough access to fetch required data. The **Bucket Policy** in AWS requires at least these permissions:
+
```json
{
"Version": "2012-10-17",
@@ -48,22 +49,59 @@ To execute metadata extraction AWS account should have enough access to fetch re
]
}
```
+
## Metadata Ingestion
-
-## Connection Details
-
-
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
+
+To ingest metadata from S3 Datalake, you need to create a service connection. The service connects S3 Datalake with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **S3 Datalake** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other S3 Datalake services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for S3 Datalake. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+You can find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
@@ -71,11 +109,11 @@ You can find more information on [Using temporary credentials with AWS resources
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
@@ -94,10 +132,62 @@ Find more information about the [Role Session Name](https://docs.aws.amazon.com/
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/salesforce.mdx b/v2.0.x-SNAPSHOT/connectors/database/salesforce.mdx
index f989d84bc..d3b0eccd4 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/salesforce.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/salesforce.mdx
@@ -3,13 +3,8 @@ title: Salesforce Connector | OpenMetadata CRM Integration Guide
description: Connect Salesforce to OpenMetadata with our comprehensive database connector guide. Setup instructions, configuration tips, and metadata extraction made easy.
sidebarTitle: Overview
---
-import Troubleshooting from '/snippets/connectors/troubleshooting.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Salesforce connector.
-Configure and schedule Salesforce metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Salesforce metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Enable Security](#securing-salesforce-connection-with-ssl-in-openmetadata)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/salesforce/troubleshooting)
+
## Requirements
+
These are the permissions you will require to fetch the metadata from Salesforce.
- **API Access**: You must have the API Enabled permission in your Salesforce organization.
- **Object Permissions**: You must have read access to the Salesforce objects that you want to ingest.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields – such as passwords, API keys, or private keys – must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Salesforce, you need to create a service connection. The service connects Salesforce with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Salesforce** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Salesforce services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Salesforce. The right-hand panel in the UI displays inline help for each field.
+
+
+
**Authentication**
OpenMetadata supports two authentication methods for Salesforce:
-- **OAuth 2.0 (Recommended)** – Authenticate using a Salesforce Connected App (External Client App) with a Consumer Key and Consumer Secret. See the [Salesforce External Client Apps documentation](https://help.salesforce.com/s/articleView?id=xcloud.external_client_apps.htm&type=5) to set one up.
+- **OAuth 2.0 (Recommended)**: Authenticate using a Salesforce Connected App (External Client App) with a Consumer Key and Consumer Secret. See the [Salesforce External Client Apps documentation](https://help.salesforce.com/s/articleView?id=xcloud.external_client_apps.htm&type=5) to set one up.
- **Consumer Key**: Salesforce Consumer Key (Client ID) for OAuth 2.0 authentication. Obtained from your Salesforce Connected App configuration. Provide this together with Consumer Secret to use OAuth 2.0.
- **Consumer Secret**: Salesforce Consumer Secret (Client Secret) for OAuth 2.0 authentication. Obtained from your Salesforce Connected App configuration. Provide this together with Consumer Key to use OAuth 2.0.
@@ -53,7 +79,7 @@ OpenMetadata supports two authentication methods for Salesforce:
**Which OAuth flow applies depends on your Salesforce Domain setting:**
- **OAuth password flow** (most common): Provide Consumer Key and Consumer Secret **together with** Username and Password. Works with `login` or `test` as the Salesforce Domain.
- - **Client credentials flow** (no username/password): Provide only Consumer Key and Consumer Secret. Requires your **Salesforce My Domain** as the domain value (e.g., `mycompany.my`). Does **not** work with `login` or `test`.
+ - **Client credentials flow** (no username/password): Provide only Consumer Key and Consumer Secret. Requires your **Salesforce My Domain** as the domain value (for example, `mycompany.my`). Does **not** work with `login` or `test`.
- **Username / Password (Legacy)**
@@ -62,24 +88,24 @@ OpenMetadata supports two authentication methods for Salesforce:
**Important**: Salesforce is deprecating direct username/password API access. Migrate to OAuth 2.0 using a Connected App (External Client App) to avoid future disruption. See [Salesforce OAuth documentation](https://help.salesforce.com/s/articleView?id=xcloud.remoteaccess_authenticate.htm&type=5).
- - **Username**: Username to connect to the Salesforce. This user should have the access as defined in requirements.
+ - **Username**: Username to connect to Salesforce. This user should have the access as defined in requirements.
- **Password**: Password to connect to Salesforce.
- **Security Token**: Salesforce Security Token is required for username/password authentication. See the [Salesforce Security Token documentation](https://help.salesforce.com/s/articleView?id=sf.user_security_token.htm&type=5) to get your security token.
**Common Fields**
-- **Organization ID**: Salesforce Organization ID is the unique identifier for your Salesforce identity. You can check out [this doc](https://help.salesforce.com/s/articleView?id=000385215&type=1) on how to get the your Salesforce Organization ID.
+- **Organization ID**: Salesforce Organization ID is the unique identifier for your Salesforce identity. You can check out [this doc](https://help.salesforce.com/s/articleView?id=000385215&type=1) on how to get your Salesforce Organization ID.
- **Note**: You need to provide `15` digit organization id in this section. for e.g. `00DIB000004nDEq`, which you can find by following the steps mentioned in above doc (`Salesforce dashboard->Setup->Company Profile->Company Information->Salesforce.com Organization Id`).
+ You need to provide the `15` digit organization id in this section, for example `00DIB000004nDEq`, which you can find by following the steps mentioned in the doc above (`Salesforce dashboard -> Setup -> Company Profile -> Company Information -> Salesforce.com Organization Id`).
- **Note**: If you want to access salesforce metadata without token(only by using organization id), you will need to setup your ip in trusted ip ranges. You can go (`Salesforce dashboard->Setup->Security->Network Access->Trusted IP Ranges`) to configure this. You can check [here](https://help.salesforce.com/s/articleView?id=sf.security_networkaccess.htm&type=5) to configure your ip in trusted ip ranges.
+ If you want to access Salesforce metadata without a token (only by using organization id), you will need to set up your IP in trusted IP ranges. Go to `Salesforce dashboard -> Setup -> Security -> Network Access -> Trusted IP Ranges` to configure this. You can check [here](https://help.salesforce.com/s/articleView?id=sf.security_networkaccess.htm&type=5) to configure your ip in trusted ip ranges.
- **Salesforce Object Names**: List of Salesforce Object Names to ingest. If left blank, all objects will be ingested (subject to filter patterns).
- **Database Name**: Optional name to give to the database in OpenMetadata. Defaults to `default` if left blank.
- **Salesforce API Version**: The API version to use when connecting to Salesforce. The correct value depends on your authentication method:
- **OAuth 2.0 (password flow or client credentials)**: Use any current Salesforce-supported version, including `65.0` or higher.
- - **Username / Password (Legacy)**: Use `64.0` or lower. Salesforce removed the SOAP `login()` endpoint in API version `65.0`, which the legacy username/password flow relies on — setting `65.0` or higher will cause login failures. Note that API versions `31.0` through `64.0` are also scheduled to be retired by Salesforce in mid-2027, so we recommend migrating to OAuth 2.0 before that date.
+ - **Username / Password (Legacy)**: Use `64.0` or lower. Salesforce removed the SOAP `login()` endpoint in API version `65.0`, which the legacy username/password flow relies on, so setting `65.0` or higher will cause login failures. Note that API versions `31.0` through `64.0` are also scheduled to be retired by Salesforce in mid-2027, so we recommend migrating to OAuth 2.0 before that date.
Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to find your org's API version.
@@ -104,15 +130,70 @@ OpenMetadata supports two authentication methods for Salesforce:
**SSL Configuration**
-In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under sslConfig which is placed in the source.
-
-
-
-
-
-
+
+In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under `sslConfig` which is placed in the source.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Salesforce Connection with SSL in OpenMetadata
-To establish secure connections between OpenMetadata and Salesforce, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client’s SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server’s certificate.
+
+To establish secure connections between OpenMetadata and Salesforce, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl_key`, `ssl_cert`, and `ssl_ca`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `ssl_ca` for the CA certificate to validate the server's certificate.
+
-
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/sap-erp.mdx b/v2.0.x-SNAPSHOT/connectors/database/sap-erp.mdx
index 645d31c10..936a3968b 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/sap-erp.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/sap-erp.mdx
@@ -4,11 +4,7 @@ description: Connect SAP ERP to OpenMetadata with our comprehensive database con
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from SAP ERP, you need to create a service connection. The service connects SAP ERP with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **SAP ERP** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SAP ERP services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SAP ERP. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: This parameter specifies the host and port of the SAP ERP instance. This should be specified as a string in the format `https://hostname.com`.
- **API Key**: API Key to authenticate the SAP ERP APIs.
- **database**: Optional name to give to the database in OpenMetadata. If left blank, we will use `default` as the database name.
- **databaseSchema**: Optional name to give to the database schema in OpenMetadata. If left blank, we will use `default` as the database schema name.
- **paginationLimit**: Pagination limit used while querying the SAP ERP APIs for fetching the entities.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/sap-hana.mdx b/v2.0.x-SNAPSHOT/connectors/database/sap-hana.mdx
index 7777195f4..0270529c7 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/sap-hana.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/sap-hana.mdx
@@ -4,11 +4,7 @@ description: Connect SAP HANA to OpenMetadata effortlessly with our comprehensiv
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
The connector is compatible with HANA or HANA express versions since HANA SPS 2.
+
### Metadata
+
To extract metadata the user used in the connection needs to have access to the `SYS` schema.
You can create a new user to run the ingestion with:
```SQL
@@ -46,38 +48,121 @@ permissions on the `_SYS_BIC` schema. You can grant the required permissions to
GRANT SELECT ON SCHEMA _SYS_BIC TO ;
```
The same applies to the `_SYS_REPO` schema, required for lineage extraction.
+
### Profiler & Data Quality
-Executing the profiler Workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. The user should also be allowed to view information in `tables` for all objects in the database. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+
+Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. The user should also be allowed to view information in `tables` for all objects in the database. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from SAP HANA, you need to create a service connection. The service connects SAP HANA with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **SAP HANA** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SAP HANA services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SAP HANA. The right-hand panel in the UI displays inline help for each field.
+
+
+
We support two possible connection types:
-1. **SQL Connection**, where you will the username, password and host.
-2. **HDB User Store** [connection](https://help.sap.com/docs/SAP_HANA_PLATFORM/b3ee5778bc2e4a089d3299b82ec762a7/dd95ac9dbb571014a7d7f0234d762fdb.html?version=2.0.05&locale=en-US).
- Note that the HDB Store will need to be locally available to the instance running the ingestion process.
- If you are unsure about this setting, you can run the ingestion process passing the usual SQL connection details.
+1. **SQL Connection**, where you will provide the username, password and host.
+2. **HDB User Store** [connection](https://help.sap.com/docs/SAP_HANA_PLATFORM/b3ee5778bc2e4a089d3299b82ec762a7/dd95ac9dbb571014a7d7f0234d762fdb.html?version=2.0.05&locale=en-US). Note that the HDB Store will need to be locally available to the instance running the ingestion process. If you are unsure about this setting, you can run the ingestion process passing the usual SQL connection details.
+
**SQL Connection**
-- **Host and Port**: Host and port of the SAP HANA service. This should be specified as a string in the format `hostname:port`. E.g., `localhost:39041`, `host.docker.internal:39041`.
+- **Host and Port**: Host and port of the SAP HANA service. This should be specified as a string in the format `hostname:port`. For example, `localhost:39041`, `host.docker.internal:39041`.
- **Username**: Specify the User to connect to SAP HANA. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to SAP HANA.
- **database**: Optional parameter to connect to a specific database.
-- **databaseSchema**: databaseSchema of the data source. This is an optional parameter, if you would like to restrict the metadata reading to a single schema. When left blank, OpenMetadata Ingestion attempts to scan all the schemas.
-**HDB USet Store**
+- **databaseSchema**: databaseSchema of the data source. This is an optional parameter, if you would like to restrict the metadata reading to a single schema. When left blank, OpenMetadata ingestion attempts to scan all the schemas.
+
+**HDB User Store**
- **User Key**: HDB Store User Key generated from the command `hdbuserstore SET `.
-
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/sas.mdx b/v2.0.x-SNAPSHOT/connectors/database/sas.mdx
index 35f9bf96b..3b8ab1d9d 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/sas.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/sas.mdx
@@ -3,11 +3,9 @@ title: SAS Connector | OpenMetadata Analytics Integration Guide
description: Connect SAS databases to OpenMetadata with our comprehensive connector guide. Step-by-step setup, configuration, and metadata extraction instructions.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/metadata/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
To deploy OpenMetadata, check the Deployment guides.
-
+
+
## Metadata Ingestion
-Prepare the SAS Service and configure the Ingestion:
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from SAS, you need to create a service connection. The service connects SAS with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **SAS** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SAS services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SAS. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **ServerHost**: Host and port of the SAS Viya deployment.
- **Username**: Username to connect to SAS Viya. This user should have privileges to read all the metadata in SAS Information Catalog.
- **Password**: Password to connect to SAS Viya.
- **Filter**: A filter expression specifying items for import. For more information, see the [SAS Catalog REST API](https://developer.sas.com/rest-apis/catalog).
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/singlestore.mdx b/v2.0.x-SNAPSHOT/connectors/database/singlestore.mdx
index f50fd0846..d47226621 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/singlestore.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/singlestore.mdx
@@ -4,11 +4,7 @@ description: Learn how to connect SingleStore database to OpenMetadata with step
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
';
-- Grant select on a specific object
GRANT SELECT ON world.hello TO '';
```
+
### Profiler & Data Quality
+
Executing the profiler workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow) and data quality tests [here](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from SingleStore, you need to create a service connection. The service connects SingleStore with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **SingleStore** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SingleStore services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SingleStore. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to SingleStore. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to SingleStore.
- **Host and Port**: Enter the fully qualified hostname and port number for your SingleStore deployment in the Host and Port field.
-- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.
-
-
-
-
-
-
+- **databaseName**: Optional name to give to the database in OpenMetadata. If left blank, we will use `default` as the database name.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/snowflake.mdx b/v2.0.x-SNAPSHOT/connectors/database/snowflake.mdx
index 811edaa3b..a5f6b43cb 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/snowflake.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/snowflake.mdx
@@ -5,11 +5,7 @@ sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Supported Authentication Types:**
-- **Basic Auth** — Username and password authentication
-- **Key Pair Auth** — Private key authentication with optional passphrase (see [Snowflake Key Pair Auth docs](https://docs.snowflake.com/en/user-guide/key-pair-auth))
-- **SSO** — Single-Sign-On via the `authenticator` connection argument
+- **Basic Auth**: Username and password authentication
+- **Key Pair Auth**: Private key authentication with optional passphrase (see [Snowflake Key Pair Auth docs](https://docs.snowflake.com/en/user-guide/key-pair-auth))
+- **SSO**: Single-Sign-On via the `authenticator` connection argument
Configure and schedule Snowflake metadata and profiler workflows from the OpenMetadata UI:
@@ -39,12 +35,13 @@ Configure and schedule Snowflake metadata and profiler workflows from the OpenMe
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/snowflake/troubleshooting)
## Requirements
+
To ingest basic metadata snowflake user must have the following privileges:
- `USAGE` Privilege on Warehouse
- `USAGE` Privilege on Database
- `USAGE` Privilege on Schema
- `SELECT` Privilege on Tables
-
+
Before you grant privileges, replace these placeholders with your own values:
| Placeholder | Description |
@@ -78,33 +75,63 @@ GRANT SELECT ON ALL FUTURE VIEWS IN DATABASE TO ROLE
GRANT SELECT ON ALL VIEWS IN DATABASE TO ROLE ;
GRANT SELECT ON ALL FUTURE DYNAMIC TABLES IN DATABASE TO ROLE ;
GRANT SELECT ON ALL DYNAMIC TABLES IN DATABASE TO ROLE ;
--- Grant IMPORTED PRIVILEGES on all Schemas of SNOWFLAKE DB to new role created above.
+-- Grant IMPORTED PRIVILEGES on all Schemas of SNOWFLAKE DB to new role created above.
-- This is optional but required for usage, lineage and stored procedure ingestion
GRANT IMPORTED PRIVILEGES ON ALL SCHEMAS IN DATABASE SNOWFLAKE TO ROLE ;
```
+
### Additional Privileges
+
The following workflows require additional grants:
- **Incremental Extraction**: OpenMetadata fetches the information by querying `snowflake.account_usage.tables`.
- **Ingesting Tags**: OpenMetadata fetches the information by querying `snowflake.account_usage.tag_references`.
- **Ingesting Stored Procedures**: OpenMetadata fetches the information by querying `snowflake.account_usage.procedures` & `snowflake.account_usage.functions`.
-- **Lineage & Usage Workflow**: For lineage, OpenMetadata uses Snowflake's `ACCOUNT_USAGE.ACCESS_HISTORY` view by default, which provides Snowflake-computed table- and column-level lineage (including for queries OpenMetadata cannot parse). If the configured role cannot read `ACCESS_HISTORY`, ingestion automatically falls back to parsing query logs. For the Usage workflow—and as the lineage fallback—OpenMetadata fetches query logs by querying `snowflake.account_usage.query_history`. Both `ACCESS_HISTORY` and `QUERY_HISTORY` require the Snowflake user to have the `ACCOUNTADMIN` role or a role with `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database.
+- **Lineage & Usage Workflow**: For lineage, OpenMetadata uses Snowflake's `ACCOUNT_USAGE.ACCESS_HISTORY` view by default, which provides Snowflake-computed table- and column-level lineage (including for queries OpenMetadata cannot parse). If the configured role cannot read `ACCESS_HISTORY`, ingestion automatically falls back to parsing query logs. For the Usage workflow, and as the lineage fallback, OpenMetadata fetches query logs by querying `snowflake.account_usage.query_history`. Both `ACCESS_HISTORY` and `QUERY_HISTORY` require the Snowflake user to have the `ACCOUNTADMIN` role or a role with `IMPORTED PRIVILEGES` on the `SNOWFLAKE` database.
For more information about the `account_usage` schema, see [Account Usage](https://docs.snowflake.com/en/sql-reference/account-usage).
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Snowflake, you need to create a service connection. The service connects Snowflake with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Snowflake** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Snowflake services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Snowflake. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Specify the User to connect to Snowflake. It should have enough privileges to read all the metadata.
- **Password**: Password to connect to Snowflake.
- **Account**: Snowflake account identifier uniquely identifies a Snowflake account within your organization, as well as throughout the global network of Snowflake-supported cloud platforms and cloud regions. If the Snowflake URL is `https://xyz1234.us-east-1.gcp.snowflakecomputing.com`, then the account is `xyz1234.us-east-1.gcp`.
@@ -112,12 +139,12 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
- **Database (Optional)**: The database of the data source is an optional parameter, if you would like to restrict the metadata reading to a single database. If left blank, OpenMetadata ingestion attempts to scan all the databases.
- **Warehouse**: Snowflake warehouse is required for executing queries to fetch the metadata. Enter the name of warehouse against which you would like to execute these queries.
- **Query Tag (Optional)**: Session query tag used to monitor usage on Snowflake. To use a query tag, the Snowflake user should have enough privileges to alter the session.
-- **Account Usage Schema Name**: Full name of account usage schema, used in case your user does not have direct access to `SNOWFLAKE.ACCOUNT_USAGE` schema. In such case you can replicate tables `QUERY_HISTORY`, `TAG_REFERENCES`, `PROCEDURES`, `FUNCTIONS`, `ACCESS_HISTORY` to a custom schema let's say `CUSTOM_DB.CUSTOM_SCHEMA` and provide the same name in this field.
+- **Account Usage Schema Name**: Full name of account usage schema, used in case your user does not have direct access to `SNOWFLAKE.ACCOUNT_USAGE` schema. In such case you can replicate tables `QUERY_HISTORY`, `TAG_REFERENCES`, `PROCEDURES`, `FUNCTIONS`, `ACCESS_HISTORY` to a custom schema, let's say `CUSTOM_DB.CUSTOM_SCHEMA`, and provide the same name in this field.
When using this field, make sure you have all these tables available within your custom schema `QUERY_HISTORY`, `TAG_REFERENCES`, `PROCEDURES`, `FUNCTIONS`, `ACCESS_HISTORY`.
- **Private Key (Optional)**: If you have configured the key pair authentication for the given user you will have to pass the private key associated with the user in this field. For more information about key-pair authentication, see [Key-pair authentication and key-pair rotation](https://docs.snowflake.com/en/user-guide/key-pair-auth).
Ensure your private key is formatted correctly before passing it. The key must be a single line with all line breaks replaced by `\n`.
-
+
For example, if you have the following multi-line key (raw format):
```
@@ -139,27 +166,76 @@ When using this field, make sure you have all these tables available within your
Replace every newline character in your key with a literal `\n`, including after the final line.
- **Snowflake Passphrase Key (Optional)**: If you have configured the encrypted key pair authentication for the given user you will have to pass the passphrase associated with the private key in this field. You can check out [this](https://docs.snowflake.com/en/user-guide/key-pair-auth) doc to get more details about key-pair authentication.
-- **Include Transient Tables**:
-Optional configuration for ingestion of `TRANSIENT` and `TEMPORARY` tables. Enabled by default, so these tables are ingested alongside permanent ones. Disable it to skip them.
-- **Include Streams**:
-Optional configuration for ingestion of streams, By default, it will skip the streams.
-- **Include Stages**:
-Optional configuration for ingestion of Snowflake stages (internal and external). By default, stages are not ingested.
+- **Include Transient Tables**: Optional configuration for ingestion of `TRANSIENT` and `TEMPORARY` tables. Enabled by default, so these tables are ingested alongside permanent ones. Disable it to skip them.
+- **Include Streams**: Optional configuration for ingestion of streams. By default, it will skip the streams.
+- **Include Stages**: Optional configuration for ingestion of Snowflake stages (internal and external). By default, stages are not ingested.
- **Client Session Keep Alive**: Optional Configuration to keep the session active in case the ingestion job runs for longer duration.
- **Cost of Credit (Optional)**: Cost of a Snowflake credit for your account, used to estimate warehouse and query costs. Defaults to `3.3`.
- **Snowflake Source Host (Optional)**: Snowflake account host used to build direct links back to Snowflake, for example from lineage or query details. Defaults to `app.snowflake.com`.
-
-
-
-In addition to the fields above, the Snowflake connector's Advanced Configuration includes:
- - **Use Access History for Lineage**: Uses Snowflake's `ACCOUNT_USAGE.ACCESS_HISTORY` view as the source of query lineage. `ACCESS_HISTORY` provides Snowflake-computed table- and column-level lineage, including for queries OpenMetadata cannot parse. Enabled by default. If the configured role cannot read `ACCESS_HISTORY`, ingestion automatically falls back to the legacy query-log parser.
- - **Access History Chunk Size (Days)**: Number of days of `ACCESS_HISTORY` scanned per query when "Use Access History for Lineage" is enabled. The lineage time window is split into chunks of this size to keep each query bounded and avoid timeouts on busy accounts. Defaults to `2`.
-
-
-
-
-
-### Incomplete Column Level for Views
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Use Access History for Lineage**: Uses Snowflake's `ACCOUNT_USAGE.ACCESS_HISTORY` view as the source of query lineage. `ACCESS_HISTORY` provides Snowflake-computed table- and column-level lineage, including for queries OpenMetadata cannot parse. Enabled by default. If the configured role cannot read `ACCESS_HISTORY`, ingestion automatically falls back to the legacy query-log parser.
+- **Access History Chunk Size (Days)**: Number of days of `ACCESS_HISTORY` scanned per query when "Use Access History for Lineage" is enabled. The lineage time window is split into chunks of this size to keep each query bounded and avoid timeouts on busy accounts. Defaults to `2`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+## Incomplete Column Level for Views
+
For views with a tag or policy, you may see incorrect lineage, this can be because user may not have enough access to fetch those policies or tags. You need to grant the following privileges in order to fix it.
Check out the [Snowflake docs](https://docs.snowflake.com/en/sql-reference/functions/get_ddl#usage-notes) for further details.
```
diff --git a/v2.0.x-SNAPSHOT/connectors/database/sqlite.mdx b/v2.0.x-SNAPSHOT/connectors/database/sqlite.mdx
index 3632b37c6..86ce6d0da 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/sqlite.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/sqlite.mdx
@@ -4,11 +4,7 @@ description: Learn to Configure OpenMetadata'sSQLite database connector for seam
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-In this section, we provide guides and references to use the Presto connector.
-Configure and schedule Presto metadata and profiler workflows from the OpenMetadata UI:
+In this section, we provide guides and references to use the SQLite connector.
+Configure and schedule SQLite metadata and profiler workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Data Profiler](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/profiler/profiler-workflow)
@@ -26,31 +22,117 @@ Configure and schedule Presto metadata and profiler workflows from the OpenMetad
- [Data Quality](/v2.0.x-SNAPSHOT/how-to-guides/data-quality-observability/quality)
- [dbt Integration](/v2.0.x-SNAPSHOT/connectors/database/dbt)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/sqlite/troubleshooting)
+
## Requirements
+
+Before configuring the SQLite connector, make sure the following requirements are in place.
+
### Metadata
+
To extract metadata, the user needs to be able to perform `.tables`, `.schema`, on database you wish to extract metadata from and have `SELECT` permission on the `sqlite_temp_master`. Access to resources will be different based on the connector used.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from SQLite, you need to create a service connection. The service connects SQLite with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **SQLite** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SQLite services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SQLite. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Username**: Username to connect to SQLite. Blank for in-memory database.
- **Password**: Password to connect to SQLite. Blank for in-memory database.
- **Host and Port**: Enter the fully qualified hostname and port number for your SQLite deployment in the Host and Port field.
- **Database**: The database of the data source is an optional parameter, if you would like to restrict the metadata reading to a single database. If left blank, OpenMetadata ingestion attempts to scan all the databases.
-- **Database Mode**: How to run the SQLite database. :memory: by default.
-
-
-
-
-
-
+- **Database Mode**: How to run the SQLite database. `:memory:` by default.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/starrocks.mdx b/v2.0.x-SNAPSHOT/connectors/database/starrocks.mdx
index 5ed2a36ef..c9c5b7471 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/starrocks.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/starrocks.mdx
@@ -1,15 +1,11 @@
---
title: StarRocks Connector | OpenMetadata Database Guide
-description: Connect StarRocks to OpenMetadata with our comprehensive database connector guide. Step-by-step setup, configuration, and metadata extraction for seamless integration.
+description: Connect StarRocks to OpenMetadata with this database connector guide. Step-by-step setup, configuration, and metadata extraction.
slug: /connectors/database/starrocks
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-## Connection Options
-
-
- - **Scheme**: SQLAlchemy driver scheme options. Defaults to `mysql+pymysql`.
- - **Username**: Username to connect to StarRocks. This user should have privileges to read all the metadata in StarRocks.
- - **Password**: Password to connect to StarRocks.
- - **Host and Port**: Host and port of the StarRocks FE service (e.g., `localhost:9030`).
- - **Database Name** *(Optional)*: Name to give to the database in OpenMetadata. If left blank, `default` will be used.
- - **Database Schema** *(Optional)*: Restrict metadata reading to a single schema. When left blank, OpenMetadata scans all schemas.
- - **SSL Config** *(Optional)*: SSL certificate configuration for secure connections.
- - **Schema Filter Pattern** *(Optional)*: Regex to only include/exclude schemas that match the pattern. System schemas (`information_schema`, `_statistics_`, `sys`) are excluded by default.
- - **Table Filter Pattern** *(Optional)*: Regex to only include/exclude tables that match the pattern.
- - **Database Filter Pattern** *(Optional)*: Regex to only include/exclude databases that match the pattern.
-
-
-
-
-
-
+
+To ingest metadata from StarRocks, you need to create a service connection. The service connects StarRocks with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **StarRocks** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other StarRocks services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for StarRocks. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Scheme**: SQLAlchemy driver scheme options. Defaults to `mysql+pymysql`.
+- **Username**: Username to connect to StarRocks. This user should have privileges to read all the metadata in StarRocks.
+- **Host and Port**: Host and port of the StarRocks FE service (for example, `localhost:9030`).
+
+**Authentication**
+- **Password**: Password to connect to StarRocks.
+
+**Scope & Options**
+- **Database Name** *(Optional)*: Name to give to the database in OpenMetadata. If left blank, `default` is used.
+- **Database Schema** *(Optional)*: Restrict metadata reading to a single schema. When left blank, OpenMetadata scans all schemas.
+- **SSL Config** *(Optional)*: SSL certificate configuration for secure connections.
+- **Schema Filter Pattern** *(Optional)*: Regex to only include/exclude schemas that match the pattern. System schemas (`information_schema`, `_statistics_`, `sys`) are excluded by default.
+- **Table Filter Pattern** *(Optional)*: Regex to only include/exclude tables that match the pattern.
+- **Database Filter Pattern** *(Optional)*: Regex to only include/exclude databases that match the pattern.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/teradata.mdx b/v2.0.x-SNAPSHOT/connectors/database/teradata.mdx
index 94e9a0d47..a3e4faa16 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/teradata.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/teradata.mdx
@@ -4,11 +4,7 @@ description: Connect Teradata databases to OpenMetadata with our comprehensive c
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
To deploy OpenMetadata, check the Deployment guides.
-
-Connector was tested on Teradata DBS version 17.20. Since there are no significant changes in metadata objects, so it should work with 15.x, 16.x versions.
+
+
+Connector was tested on Teradata DBS version 17.20. Since there are no significant changes in metadata objects, it should also work with 15.x and 16.x versions.
+
+By default, all valid users in a Teradata DB have full access to metadata objects, so there are no specific requirements for user privileges.
+
## Metadata Ingestion
-By default, all valid users in Teradata DB has full access to metadata objects, so there are no any specific requirements to user privileges.
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Username**: Specify the User to connect to Teradata.
-- **Password**: Password to connect to Teradata
-- **Logmech**: Specifies the logon authentication method. Possible values are TD2 (the default), JWT, LDAP, KRB5 for Kerberos, or TDNEGO.
+
+To ingest metadata from Teradata, you need to create a service connection. The service connects Teradata with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Teradata** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Teradata services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Teradata. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Username**: Specify the user to connect to Teradata.
+- **Host and Port**: Enter the fully qualified hostname and port number (default port for Teradata is `1025`) for your Teradata deployment in the Host and Port field.
+
+**Authentication**
+- **Password**: Password to connect to Teradata.
+- **Logmech**: Specifies the logon authentication method. Possible values are `TD2` (the default), `JWT`, `LDAP`, `KRB5` for Kerberos, or `TDNEGO`.
- **LOGDATA**: Specifies additional data needed by a logon mechanism, such as a secure token, Distinguished Name, or a domain/realm name. LOGDATA values are specific to each logon mechanism.
-- **Host and Port**: Enter the fully qualified hostname and port number (default port for Teradata is 1025) for your Teradata deployment in the Host and Port field.
-- **Transaction Mode**: Specifies the transaction mode for the connection. Possible values are DEFAULT (the default), ANSI, or TERA.
+
+**Scope & Options**
+- **Transaction Mode**: Specifies the transaction mode for the connection. Possible values are `DEFAULT` (the default), `ANSI`, or `TERA`.
- **Teradata Database Account**: Specifies an account string to override the default account string defined for the database user. Accounts are used by the database for workload management and resource usage monitoring.
-- **Connection Options** and **Connection Arguments**: additional connection parameters. For more information please view teradatasql [docs](https://pypi.org/project/teradatasql/).
-
-
-
-
-
-
+- **Connection Options** and **Connection Arguments**: Additional connection parameters. For more information, see the teradatasql [documentation](https://pypi.org/project/teradatasql/).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/timescale.mdx b/v2.0.x-SNAPSHOT/connectors/database/timescale.mdx
index 4878010e9..feae886d1 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/timescale.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/timescale.mdx
@@ -4,15 +4,10 @@ description: Connect TimescaleDB to OpenMetadata with our comprehensive database
sidebarTitle: Overview
collate: true
---
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
-
Starting from OpenMetadata **version 1.6.5**, support for **Stored Procedures Lineage** has been introduced. This feature enables tracking the relationships and dependencies between stored procedures and other database objects, enhancing lineage visibility and data traceability.
@@ -62,17 +60,15 @@ TimescaleDB is built on PostgreSQL. Note that we only support officially support
### Usage and Lineage considerations
-When extracting lineage and usage information from TimescaleDB we base our finding on the `pg_stat_statements` table.
-You can find more information about it on the official [docs](https://www.postgresql.org/docs/current/pgstatstatements.html#id-1.11.7.39.6).
+When extracting lineage and usage information from TimescaleDB, OpenMetadata bases its findings on the `pg_stat_statements` table. You can find more information about it in the official [documentation](https://www.postgresql.org/docs/current/pgstatstatements.html#id-1.11.7.39.6).
-Another interesting consideration here is explained in the following SO [question](https://stackoverflow.com/questions/50803147/what-is-the-timeframe-for-pg-stat-statements).
-As a summary:
-- The `pg_stat_statements` has no time data embedded in it.
-- It will show all queries from the last reset (one can call `pg_stat_statements_reset()`).
+Another interesting consideration is explained in the following Stack Overflow [question](https://stackoverflow.com/questions/50803147/what-is-the-timeframe-for-pg-stat-statements). As a summary:
+- The `pg_stat_statements` table has no time data embedded in it.
+- It shows all queries since the last reset (one can call `pg_stat_statements_reset()`).
-Then, when extracting usage and lineage data, the query log duration will have no impact, only the query limit.
+When extracting usage and lineage data, the query log duration has no impact, only the query limit.
-**Note:** For usage and lineage grant your user `pg_read_all_stats` permission.
+**Note:** For usage and lineage, grant your user the `pg_read_all_stats` permission.
```sql
GRANT pg_read_all_stats TO your_user;
@@ -80,21 +76,18 @@ GRANT pg_read_all_stats TO your_user;
### IAM Authentication
-In order to be able to connect via IAM, you need to have the following:
+In order to connect via IAM, you need the following:
-1. Database is configured to use IAM authentication
-Ensure that the RDS has IAM DB authentication enabled. Otherwise, you can click on Modify to enable it.
+1. **Database is configured to use IAM authentication.** Ensure that the RDS instance has IAM DB authentication enabled. Otherwise, click **Modify** to enable it.
-2. The user has the necessary IAM permissions
-Even if you use IAM to connect to TimescaleDB, you need to specify a user to prepare the connection. You need to create a user as follows:
+2. **The user has the necessary IAM permissions.** Even if you use IAM to connect to TimescaleDB, you need to specify a user to prepare the connection. Create a user as follows:
```sql
CREATE USER iam_user WITH LOGIN;
GRANT rds_iam TO iam_user;
```
-3. The AWS Role has the necessary permissions
-The role that is going to be used to perform the ingestion, needs to have the following permissions:
+3. **The AWS Role has the necessary permissions.** The role used to perform the ingestion needs the following permissions:
```json
{
@@ -112,24 +105,66 @@ The role that is going to be used to perform the ingestion, needs to have the fo
]
}
```
-Otherwise, you might be finding issues such as
+Otherwise, you might encounter an error such as:
+
+```
PAM authentication failed for user "user"
+```
## Metadata Ingestion
-
-## Connection Details
-
-
+To ingest metadata from TimescaleDB, you need to create a service connection. The service connects TimescaleDB with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **TimescaleDB** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other TimescaleDB services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for TimescaleDB. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
- **Username**: Specify the user to connect to TimescaleDB. It should have enough privileges to read all the metadata.
-- **Auth Type**: Basic Auth or IAM based auth to connect to instances / cloud RDS.
+- **Host and Port**: Enter the fully qualified hostname and port number for your TimescaleDB deployment.
+
+**Authentication**
+- **Auth Type**: Basic Auth or IAM-based auth to connect to instances or cloud RDS.
- **Basic Auth**: Provide the password to connect to TimescaleDB.
- **IAM Based Auth**:
- **AWS Access Key ID** & **AWS Secret Access Key**: Required when interacting with AWS programmatically ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
- **AWS Region**: Required to resolve service endpoints ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
- - **AWS Session Token (optional)**: Needed when using temporary credentials ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html)).
- - **Endpoint URL (optional)**: Override the default AWS endpoint if needed ([docs](https://docs.aws.amazon.com/general/latest/gr/rande.html)).
+ - **AWS Session Token** *(Optional)*: Needed when using temporary credentials ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html)).
+ - **Endpoint URL** *(Optional)*: Override the default AWS endpoint if needed ([docs](https://docs.aws.amazon.com/general/latest/gr/rande.html)).
- **Profile Name**: Use a named profile other than `default` if desired ([docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html)).
- **Assume Role Arn**: ARN of the role to assume for cross-account access ([docs](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)).
- **Assume Role Session Name**: Identifier for the assumed role session ([docs](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Role%20Session%20Name)).
@@ -137,29 +172,73 @@ PAM authentication failed for user "user"
When using Assume Role authentication, provide the AWS Region and the Assume Role ARN used for ingestion.
-- **Host and Port**: Enter the fully qualified hostname and port number for your TimescaleDB deployment.
-
-
-
-
-
-## Securing TimescaleDB Connection with SSL in OpenMetadata
+#### Test Connection
-To establish secure connections between OpenMetadata and a TimescaleDB database, you can configure SSL using different SSL modes provided by PostgreSQL, each offering varying levels of security.
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
-Under `Advanced Config`, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that TimescaleDB (PostgreSQL) requires only the CA certificate for SSL validation.
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
-For IAM authentication, it is recommended to choose the `allow` mode or another SSL mode that fits your specific requirements.
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
-
+**Filter Options**
+
+The Database, Schema, Table, and Stored Procedure sections each include the following filter options:
-
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+- **Stored Procedure**: Controls which stored procedures are included in metadata ingestion.
+Each section provides the following controls:
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
-
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+## Securing TimescaleDB Connection with SSL in OpenMetadata
+
+To establish secure connections between OpenMetadata and a TimescaleDB database, you can configure SSL using different SSL modes provided by PostgreSQL, each offering varying levels of security.
+
+Under **Advanced Config**, specify the SSL mode appropriate for your connection, such as `prefer`, `verify-ca`, `allow`, and others. After selecting the SSL mode, provide the CA certificate used for SSL validation (`caCertificate`). Note that TimescaleDB (PostgreSQL) requires only the CA certificate for SSL validation.
+
+
+For IAM authentication, it is recommended to choose the `allow` mode or another SSL mode that fits your specific requirements.
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/trino.mdx b/v2.0.x-SNAPSHOT/connectors/database/trino.mdx
index e69b3117b..46b38adad 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/trino.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/trino.mdx
@@ -1,15 +1,10 @@
---
title: Trino Connector | OpenMetadata Distributed SQL Integration
-description: Configure OpenMetadata'sTrino database connector to seamlessly integrate your distributed SQL engine with automated metadata discovery and lineage tracking.
+description: Configure OpenMetadata's Trino database connector to connect your distributed SQL engine with automated metadata discovery and lineage tracking.
sidebarTitle: Overview
---
-
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Username**: Specify the User to connect to Trino. It should have enough privileges to read all the metadata.
-- **Auth Config**: User can authenticate the Trino Instance with auth type as `Basic Authentication` i.e. Password **or** by using `JWT Authentication`.
+
+To ingest metadata from Trino, you need to create a service connection. The service connects Trino with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Trino** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Trino services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Trino. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Username**: Specify the user to connect to Trino. It should have enough privileges to read all the metadata.
+- **Host and Port**: Enter the fully qualified hostname and port number for your Trino deployment in the Host and Port field.
+- **Catalog**: Trino offers a catalog feature where all the databases are stored.
+- **Database Schema**: Database schema of the data source. This is an optional parameter, if you would like to restrict the metadata reading to a single database schema. When left blank, OpenMetadata attempts to scan all the database schemas.
+
+**Authentication**
+
+Authenticate the Trino instance with **Basic Authentication** (password) or **JWT Authentication**:
- **Basic Auth**:
- **Password**: Password to connect to Trino.
- **JWT Auth Config**:
- - **JWT**: JWT can be used to authenticate with trino. Follow the steps in the [official trino](https://trino.io/docs/current/security/jwt.html) documentation to setup trino with jwt.
+ - **JWT**: JWT can be used to authenticate with Trino. Follow the steps in the [official Trino documentation](https://trino.io/docs/current/security/jwt.html) to set up Trino with JWT.
- **Azure**:
- **Client ID**: To get the Client ID (also known as application ID), follow these steps:
1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
- 2. Search for `App registrations` and select the `App registrations link`.
+ 2. Search for `App registrations` and select the `App registrations` link.
3. Select the `Azure AD` app you're using for Trino.
4. From the Overview section, copy the `Application (client) ID`.
- **Client Secret**: To get the client secret, follow these steps:
1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
- 2. Search for `App registrations` and select the `App registrations link`.
+ 2. Search for `App registrations` and select the `App registrations` link.
3. Select the `Azure AD` app you're using for Trino.
4. Under `Manage`, select `Certificates & secrets`.
5. Under `Client secrets`, select `New client secret`.
@@ -69,35 +105,86 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
7. From the `Client secrets` section, copy the string in the `Value` column of the newly created application secret.
- **Tenant ID**: To get the tenant ID, follow these steps:
1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
- 2. Search for `App registrations` and select the `App registrations link`.
+ 2. Search for `App registrations` and select the `App registrations` link.
3. Select the `Azure AD` app you're using for Trino.
4. From the `Overview` section, copy the `Directory (tenant) ID`.
- - **Scopes**: To let OM use the Trino Auth APIs using your Azure AD app, you'll need to add the scope
+ - **Scopes**: To let OpenMetadata use the Trino Auth APIs using your Azure AD app, add the scope:
1. Log into [Microsoft Azure](https://ms.portal.azure.com/#allservices).
- 2. Search for `App registrations` and select the `App registrations link`.
+ 2. Search for `App registrations` and select the `App registrations` link.
3. Select the `Azure AD` app you're using for Trino.
- 4. From the `Expose an API` section, copy the `Application ID URI`
- 5. Make sure the URI ends with `/.default` in case it does not, you can append the same manually
-- **Host and Port**: Enter the fully qualified hostname and port number for your Trino deployment in the Host and Port field.
-- **Catalog**: Trino offers a catalog feature where all the databases are stored.
-- **DatabaseSchema**: DatabaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
-- **proxies**: Proxies for the connection to Trino data source
-- **params**: URL parameters for connection to the Trino data source
-- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Trino during the connection. These details must be added as Key-Value pairs.
-- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Trino during the connection. These details must be added as Key-Value pairs.
- - In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`
+ 4. From the `Expose an API` section, copy the `Application ID URI`.
+ 5. Make sure the URI ends with `/.default`; if it does not, append it manually.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **proxies**: Proxies for the connection to the Trino data source.
+- **params**: URL parameters for the connection to the Trino data source.
+- **Connection Options** *(Optional)*: Enter the details for any additional connection options that can be sent to Trino during the connection. These details must be added as key-value pairs.
+- **Connection Arguments** *(Optional)*: Enter the details for any additional connection arguments, such as security or protocol configs, that can be sent to Trino during the connection. These details must be added as key-value pairs.
+ - If you are using Single Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a key-value pair, for example `"authenticator" : "sso_login_url"`.
+
**SSL Configuration**
-In order to integrate SSL in the Metadata Ingestion Config, the user will have to add the SSL config under **connectionArguments** which is placed in source.
-**SSL Modes**
-There are a couple of types of SSL modes that redshift supports which can be added to ConnectionArguments, they are as follows:
-- **false**: In order to disable SSL verification, set the `verify` parameter to `False`.
-- **\**: To use self-signed certificates, specify a path to the certificate in `verify` parameter.
-Find more details in [the Python requests library documentation](https://requests.readthedocs.io/en/latest/user/advanced.html?highlight=ssl#ssl-cert-verification).
-
-
-
-
-
-
+
+To integrate SSL in the Metadata Ingestion config, add the SSL config under `connectionArguments`, which is placed in `source`. There are a couple of SSL modes supported, which can be added to `connectionArguments`:
+- **false**: To disable SSL verification, set the `verify` parameter to `False`.
+- **``**: To use self-signed certificates, specify a path to the certificate in the `verify` parameter.
+
+Find more details in the [Python requests library documentation](https://requests.readthedocs.io/en/latest/user/advanced.html?highlight=ssl#ssl-cert-verification).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/unity-catalog.mdx b/v2.0.x-SNAPSHOT/connectors/database/unity-catalog.mdx
index 8120e4c5f..aaf35068a 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/unity-catalog.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/unity-catalog.mdx
@@ -3,14 +3,9 @@ title: Unity Catalog Connector | OpenMetadata Data Governance
description: Learn how to connect Unity Catalog to OpenMetadata with our comprehensive connector guide. Setup instructions, configuration tips, and metadata integrat...
sidebarTitle: Overview
---
-
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Supported Authentication Types:**
-- **Personal Access Token** — Generated token from your Databricks workspace (User Settings → Developer → Access Tokens)
-- **Databricks OAuth** — OAuth2 Machine-to-Machine authentication using Service Principal credentials
-- **Azure AD Setup** — Azure Active Directory authentication using Service Principal for Azure Databricks workspaces
+- **Personal Access Token** — Generated token from your Databricks workspace (**User Settings** > **Developer** > **Access Tokens**).
+- **Databricks OAuth** — OAuth2 Machine-to-Machine authentication using Service Principal credentials.
+- **Azure AD Setup** — Azure Active Directory authentication using Service Principal for Azure Databricks workspaces.
Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
@@ -38,10 +33,17 @@ Configure and schedule Unity Catalog metadata workflow from the OpenMetadata UI:
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/database/unity-catalog/troubleshooting)
+
## Requirements
+
+Before configuring the Unity Catalog connector, make sure the following requirements are in place.
+
### Permission Requirement
+
To enable full functionality of metadata extraction, profiling, usage, and lineage features in OpenMetadata, the following permissions must be granted to the relevant users in your Databricks environment.
+
### Metadata and Profiling Permissions
+
These permissions are required on the catalogs, schemas, and tables from which metadata and profiling information will be ingested.
```sql
GRANT USE CATALOG ON CATALOG TO ``;
@@ -49,13 +51,17 @@ GRANT USE SCHEMA ON SCHEMA TO ``;
GRANT SELECT ON TABLE TO ``;
```
Ensure these grants are applied to all relevant tables for metadata ingestion and profiling operations.
+
### Usage
+
These permissions enable OpenMetadata to extract query history for usage statistics.
```sql
GRANT SELECT ON SYSTEM.QUERY.HISTORY TO ``;
GRANT USE SCHEMA ON SCHEMA system.query TO ``;
```
+
### Lineage
+
These permissions enable OpenMetadata to extract table and column-level lineage from Databricks system tables.
```sql
GRANT SELECT ON system.access.table_lineage TO ``;
@@ -66,7 +72,9 @@ For external table lineage (linking tables to their storage locations), the foll
```sql
GRANT SELECT ON system.information_schema.tables TO ``;
```
+
### Tags
+
These permissions enable OpenMetadata to extract Unity Catalog tags from catalogs, schemas, tables, and columns.
```sql
GRANT SELECT ON .information_schema.catalog_tags TO ``;
@@ -74,36 +82,123 @@ GRANT SELECT ON .information_schema.schema_tags TO ``;
GRANT SELECT ON .information_schema.table_tags TO ``;
GRANT SELECT ON .information_schema.column_tags TO ``;
```
+
Adjust <user>, <catalog_name>, <schema_name>, and <table_name> according to your specific deployment and security requirements.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Unity Catalog, you need to create a service connection. The service connects Unity Catalog with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Unity Catalog** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Unity Catalog services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Unity Catalog. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
- **Host and Port**: Enter the fully qualified hostname and port number for your Databricks deployment in the Host and Port field.
-- **Authentication Type**: Choose one of the following authentication methods:
- - **Personal Access Token**: Provide the token generated from your Databricks workspace (User Settings → Developer → Access Tokens).
- - **Databricks OAuth**: Provide the Service Principal `Client ID` and `Client Secret` created in your Databricks Account Console.
- - **Azure AD Setup**: Provide the `Azure Client ID`, `Azure Client Secret`, and `Azure Tenant ID` from your Azure Active Directory Service Principal.
- **HTTP Path**: Databricks compute resources URL.
-- **connectionTimeout**: The maximum amount of time (in seconds) to wait for a successful connection to the data source. If the connection attempt takes longer than this timeout period, an error will be returned.
-- **Catalog**: Catalog of the data source(Example: hive_metastore). This is optional parameter, if you would like to restrict the metadata reading to a single catalog. When left blank, OpenMetadata Ingestion attempts to scan all the catalog.
-- **DatabaseSchema**: databaseSchema of the data source. This is optional parameter, if you would like to restrict the metadata reading to a single databaseSchema. When left blank, OpenMetadata Ingestion attempts to scan all the databaseSchema.
-
-
-
-
-
-
+
+**Authentication**
+
+Choose one of the following authentication methods:
+- **Personal Access Token**: Provide the token generated from your Databricks workspace (**User Settings** > **Developer** > **Access Tokens**).
+- **Databricks OAuth**: Provide the Service Principal `Client ID` and `Client Secret` created in your Databricks Account Console.
+- **Azure AD Setup**: Provide the `Azure Client ID`, `Azure Client Secret`, and `Azure Tenant ID` from your Azure Active Directory Service Principal.
+
+**Scope & Options**
+- **connectionTimeout**: The maximum amount of time (in seconds) to wait for a successful connection to the data source. If the connection attempt takes longer than this timeout period, an error is returned.
+- **Catalog**: Catalog of the data source (for example, `hive_metastore`). This is an optional parameter, if you would like to restrict the metadata reading to a single catalog. When left blank, OpenMetadata attempts to scan all catalogs.
+- **Database Schema**: Database schema of the data source. This is an optional parameter, if you would like to restrict the metadata reading to a single database schema. When left blank, OpenMetadata attempts to scan all database schemas.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/database/vertica.mdx b/v2.0.x-SNAPSHOT/connectors/database/vertica.mdx
index c13208c57..11f0f07dc 100644
--- a/v2.0.x-SNAPSHOT/connectors/database/vertica.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/database/vertica.mdx
@@ -4,11 +4,7 @@ description: Connect Vertica database to OpenMetadata with our comprehensive con
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentDatabase from '/snippets/connectors/metadata-agent/database.mdx'
import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
.public" DEFAULT INCLUDE PRIVILEGES;
```
+
#### Lineage and Usage
-If you also want to run the Lineage and Usage workflows, then the user needs to be granted permissions to the
-`V_MONITOR` schema:
+
+If you also want to run the Lineage and Usage workflows, the user needs to be granted permissions to the `V_MONITOR` schema:
```sql
GRANT SELECT ON ALL TABLES IN SCHEMA V_MONITOR TO openmetadata;
```
-Note that this setting might only grant visibility to the queries executed by this user. A more complete approach
-will be to grant the `SYSMONITOR` role to the `openmetadata` user:
+Note that this setting might only grant visibility to the queries executed by this user. A more complete approach is to grant the `SYSMONITOR` role to the `openmetadata` user:
```sql
GRANT SYSMONITOR TO openmetadata;
ALTER USER openmetadata DEFAULT ROLE SYSMONITOR;
```
+
#### Profiler
-To run the profiler, it's not enough to have `USAGE` permissions to the schema as we need to `SELECT` the tables
-in there. Therefore, you'll need to grant `SELECT` on all tables for the schemas:
+
+To run the profiler, `USAGE` permissions on the schema are not enough, since OpenMetadata needs to `SELECT` the tables in there. Grant `SELECT` on all tables for the schemas:
```sql
GRANT SELECT ON ALL TABLES IN SCHEMA TO openmetadata;
```
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Vertica, you need to create a service connection. The service connects Vertica with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Database Services**, then click the **Vertica** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Vertica services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Vertica. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Username**: Specify the user to connect to Vertica. It should have enough privileges to read all the metadata.
+- **Host and Port**: Enter the fully qualified hostname and port number for your Vertica deployment in the Host and Port field.
+
+**Authentication**
+- **Password**: Password to connect to Vertica.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your database service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
-- **Username**: Specify the User to connect to Vertica. It should have enough privileges to read all the metadata.
-- **Password**: Password to connect to Vertica.
-- **Host and Port**: Enter the fully qualified hostname and port number for your Vertica deployment in the Host and Port field.
-
-
-
-
-
-
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/drive/custom-drive.mdx b/v2.0.x-SNAPSHOT/connectors/drive/custom-drive.mdx
index 669acb7ca..c5daf52c8 100644
--- a/v2.0.x-SNAPSHOT/connectors/drive/custom-drive.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/drive/custom-drive.mdx
@@ -3,10 +3,11 @@ title: Custom Drive Connector | Build & Extend OpenMetadata Easily
description: Learn how to create a Custom Drive Connector to ingest metadata from file system-like services into OpenMetadata using the ingestion framework.
sidebarTitle: Overview
---
+import MetadataAgentDrive from '/snippets/connectors/metadata-agent/drive.mdx'
# Custom Drive Connector
-Just like other services in OpenMetadata (Database, Pipelines, Dashboards, Messaging, etc.), it’s possible to create a **Custom Drive Connector** to bring metadata from a storage or file system-like service into OpenMetadata.
+Just like other services in OpenMetadata (Database, Pipelines, Dashboards, Messaging, and so on), it’s possible to create a **Custom Drive Connector** to bring metadata from a storage or file system-like service into OpenMetadata.
In this guide, we'll walk through how to implement your own Custom Drive Connector by extending the ingestion framework. The implementation pattern follows other service types closely, making the transition smooth for anyone familiar with the ecosystem.
@@ -22,6 +23,8 @@ Watch OpenMetadata's Webinar on Custom Connectors to get more context on how to
## Steps to Set Up a Custom Drive Connector
+Follow these steps to build and register a Custom Drive Connector.
+
### Step 1 - Prepare Your Drive Connector
A Custom Drive Connector is a Python class that inherits from:
@@ -142,7 +145,7 @@ python setup.py sdist
### Step 4 - Update the Ingestion Image
-To run your Custom Drive Connector inside Docker (e.g., with Airflow or directly from the UI), the ingestion image must include your module.
+To run your Custom Drive Connector inside Docker (for example, with Airflow or directly from the UI), the ingestion image must include your module.
**Dockerfile example:**
@@ -193,3 +196,7 @@ sink:
```
Run `metadata ingest -c `.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/drive/googledrive.mdx b/v2.0.x-SNAPSHOT/connectors/drive/googledrive.mdx
index 220925bb0..b75c36708 100644
--- a/v2.0.x-SNAPSHOT/connectors/drive/googledrive.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/drive/googledrive.mdx
@@ -1,17 +1,11 @@
---
title: Google Drive Connector | OpenMetadata Drive Guide
description: Connect Google Drive to OpenMetadata with our comprehensive drive connector guide. Step-by-step setup, configuration, and metadata extraction for files, folders, and Google Sheets.
-slug: /connectors/drive/googledrive
sidebarTitle: Overview
-collate: true
---
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import MetadataAgentDrive from '/snippets/connectors/metadata-agent/drive.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
**Service Accounts**
3. Click **Create Service Account**
4. Provide a name and description for the service account
5. Grant the service account the necessary roles (at minimum: **Viewer** role)
6. Click **Done**
+
#### 2. Enable Google Drive API
+
1. In the Google Cloud Console, navigate to **APIs & Services** > **Library**
2. Search for "Google Drive API"
3. Click on it and enable the API
+
#### 3. Create and Download Service Account Key
+
1. Navigate back to **IAM & Admin** > **Service Accounts**
2. Click on your newly created service account
3. Go to the **Keys** tab
4. Click **Add Key** > **Create new key**
5. Choose **JSON** as the key type
6. Download the JSON key file
+
#### 4. Domain-Wide Delegation (Optional)
+
If you need to access files owned by other users in your organization:
+
1. In the service account details, click **Show Domain-Wide Delegation**
2. Enable **Enable Google Workspace Domain-wide Delegation**
3. Note the **Client ID**
@@ -58,36 +66,135 @@ If you need to access files owned by other users in your organization:
- `https://www.googleapis.com/auth/drive.metadata.readonly`
- `https://www.googleapis.com/auth/spreadsheets.readonly` (if processing Google Sheets)
5. Specify the delegated email address in the connection configuration
+
### Permissions Required
+
The service account needs:
- Read access to the Google Drive files and folders you want to ingest
- If using shared drives: access to the specific shared drive
- If using domain-wide delegation: domain-wide delegation enabled with the scopes above
+
### Profiler & Data Quality
+
Google Drive is a file storage connector. Profiler and data quality workflows are not supported for this connector.
+
## Metadata Ingestion
-
-## Connection Options
-
-
- - **GCP Credentials**: Provide the GCP credentials to access the Google Drive API, either as a **GCP Credentials Path** (path to the service account key file) or **GCP Credentials Values** (paste the key file's contents directly). Leave empty to use [ADC authentication](https://cloud.google.com/docs/authentication#adc).
- - **Delegated Email** *(Optional)*: Email address to impersonate using domain-wide delegation. Required to access files owned by other users in your organization.
- - **Drive ID** *(Optional)*: Specific shared drive ID to connect to. Leave empty to process all drives accessible to the service account.
- - **Include Team Drives**: Include shared/team drives in metadata extraction. Defaults to `true`.
- - **Include Google Sheets**: Extract metadata for Google Sheets files, including spreadsheet structure and worksheet information. Defaults to `false`.
- - **Directory Filter Pattern** *(Optional)*: Regex to only include/exclude directories that match the pattern.
- - **File Filter Pattern** *(Optional)*: Regex to only include/exclude files that match the pattern.
- - **Spreadsheet Filter Pattern** *(Optional)*: Regex to only include/exclude spreadsheets that match the pattern. Only applies when **Include Google Sheets** is enabled.
- - **Worksheet Filter Pattern** *(Optional)*: Regex to only include/exclude worksheets within spreadsheets that match the pattern. Only applies when **Include Google Sheets** is enabled.
-
-
-
-
-
-
+
+To ingest metadata from Google Drive, you need to create a service connection. The service connects Google Drive with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Drive Services**, then click the **Google Drive** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Google Drive services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Google Drive. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **GCP Credentials**: Provide the GCP credentials to access Google Drive API. You can provide the credentials in one of the following ways:
+ - **GCP Credentials Path**: Path to the GCP service account credentials JSON file
+ - **GCP Credentials Values**: Paste the content of the GCP service account credentials JSON file directly
+
+- **Delegated Email (Optional)**: Email address to impersonate using domain-wide delegation. This is required if you want to access files owned by other users in your organization using domain-wide delegation.
+
+- **Drive ID (Optional)**: Specific shared drive ID to connect to. If provided, only this shared drive will be processed. Leave empty to process all accessible drives.
+
+- **Include Team Drives**: Enable to include shared/team drives in metadata extraction. Default is `true`.
+
+- **Include Google Sheets**: Enable to extract metadata only for Google Sheets files. When enabled, only Google Sheets will be processed. Default is `false`.
+
+- **Directory Filter Pattern (Optional)**: Regex pattern to include or exclude directories from metadata extraction.
+
+- **File Filter Pattern (Optional)**: Regex pattern to include or exclude files from metadata extraction.
+
+- **Spreadsheet Filter Pattern (Optional)**: Regex pattern to include or exclude spreadsheets from metadata extraction (only applies when Include Google Sheets is enabled).
+
+- **Worksheet Filter Pattern (Optional)**: Regex pattern to include or exclude worksheets within spreadsheets from metadata extraction (only applies when Include Google Sheets is enabled).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your drive service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all directories, files, spreadsheets, and worksheets available in the source.
+
+
+**Filter Options**
+
+The Directory, File, Spreadsheet, and Worksheet sections each include the following filter options:
+
+- **Directory**: Controls which directories (folders) OpenMetadata ingests from the source.
+- **File**: Controls which individual files within the ingested directories are included.
+- **Spreadsheet**: Controls which spreadsheet files are included in metadata ingestion.
+- **Worksheet**: Controls which individual worksheets (tabs within a spreadsheet) are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
\ No newline at end of file
diff --git a/v2.0.x-SNAPSHOT/connectors/drive/googledrive/yaml.mdx b/v2.0.x-SNAPSHOT/connectors/drive/googledrive/yaml.mdx
index 59b2dc1af..bd44f1453 100644
--- a/v2.0.x-SNAPSHOT/connectors/drive/googledrive/yaml.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/drive/googledrive/yaml.mdx
@@ -31,6 +31,9 @@ Configure and schedule Google Drive metadata workflows from the OpenMetadata UI:
- [Metadata Ingestion](#metadata-ingestion)
## Requirements
+
+Before configuring the Google Drive connector, make sure the following requirements are in place.
+
### Python Requirements
To run the Google Drive ingestion, you will need to install:
@@ -131,7 +134,13 @@ source:
### Configuration Examples
+
+The following examples show common Google Drive configurations.
+
#### Service Account with Domain-Wide Delegation
+
+This example uses domain-wide delegation to access shared drives across the organization.
+
```yaml
source:
type: googledrive
@@ -157,7 +166,11 @@ source:
delegatedEmail: shared-drives@yourcompany.com
includeTeamDrives: true
```
+
#### Google Sheets Ingestion with Filtering
+
+This example restricts ingestion to Google Sheets and applies spreadsheet and worksheet filter patterns.
+
```yaml
source:
type: googledrive
diff --git a/v2.0.x-SNAPSHOT/connectors/drive/sftp.mdx b/v2.0.x-SNAPSHOT/connectors/drive/sftp.mdx
index 8307a92ff..cfeb24b79 100644
--- a/v2.0.x-SNAPSHOT/connectors/drive/sftp.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/drive/sftp.mdx
@@ -3,15 +3,10 @@ title: SFTP Connector | OpenMetadata Drive Guide
description: Connect SFTP servers to OpenMetadata with our comprehensive drive connector guide. Step-by-step setup, configuration, and metadata extraction for secure file transfer protocol servers.
slug: /connectors/drive/sftp
sidebarTitle: Overview
-collate: true
---
-import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
-import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import MetadataAgentDrive from '/snippets/connectors/metadata-agent/drive.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-## Connection Options
-
-
- - **Host**: SFTP server hostname or IP address.
- - **Port**: SFTP server port number. Defaults to `22`.
- - **Authentication Type**: Choose between:
- - **Username/Password**: Authenticate using a username and password.
- - **Private Key**: Authenticate using an SSH private key in PEM format. Supports RSA, Ed25519, ECDSA, and DSS keys. An optional passphrase can be provided if the key is encrypted.
- - **Root Directories** *(Optional)*: List of root directories to scan for files and subdirectories. Defaults to `/` (the user's home directory).
- - **Directory Filter Pattern** *(Optional)*: Regex to only include/exclude directories that match the pattern.
- - **File Filter Pattern** *(Optional)*: Regex to only include/exclude files that match the pattern.
- - **Structured Data Files Only** *(Optional)*: When enabled, only catalog structured data files (CSV, TSV) that can have schema extracted. Non-structured files like images, PDFs, and videos will be skipped. Defaults to `false`.
- - **Extract Sample Data** *(Optional)*: When enabled, extract sample data from structured files (CSV, TSV). Disabled by default to avoid performance overhead.
-
-
-
-
-
-
+
+To ingest metadata from SFTP, you need to create a service connection. The service connects SFTP with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Drive Services**, then click the **SFTP** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SFTP services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SFTP. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Host**: SFTP server hostname or IP address.
+- **Port**: SFTP server port number. Defaults to `22`.
+
+**Authentication**
+- **Authentication Type**: Choose between:
+ - **Username/Password**: Authenticate using a username and password.
+ - **Private Key**: Authenticate using an SSH private key in PEM format. Supports RSA, Ed25519, ECDSA, and DSS keys. An optional passphrase can be provided if the key is encrypted.
+
+**Scope & Options**
+- **Root Directories** *(Optional)*: List of root directories to scan for files and subdirectories. Defaults to `/` (the user's home directory).
+- **Directory Filter Pattern** *(Optional)*: Regex to only include/exclude directories that match the pattern.
+- **File Filter Pattern** *(Optional)*: Regex to only include/exclude files that match the pattern.
+- **Structured Data Files Only** *(Optional)*: When enabled, only catalog structured data files (CSV, TSV) that can have schema extracted. Non-structured files like images, PDFs, and videos will be skipped. Defaults to `false`.
+- **Extract Sample Data** *(Optional)*: When enabled, extract sample data from structured files (CSV, TSV). Disabled by default to avoid performance overhead.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Options** *(Optional)*: Enter the details for any additional connection options that can be sent to the source during the connection. These details must be added as Key-Value pairs.
+- **Connection Arguments** *(Optional)*: Enter the details for any additional connection arguments such as security or protocol configs that can be sent during the connection. These details must be added as Key-Value pairs.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your drive service. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all directories, files, spreadsheets, and worksheets available in the source.
+
+
+**Filter Options**
+
+The Directory, File, Spreadsheet, and Worksheet sections each include the following filter options:
+
+- **Directory**: Controls which directories (folders) OpenMetadata ingests from the source.
+- **File**: Controls which individual files within the ingested directories are included.
+- **Spreadsheet**: Controls which spreadsheet files are included in metadata ingestion.
+- **Worksheet**: Controls which individual worksheets (tabs within a spreadsheet) are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/export-lineage.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/export-lineage.mdx
index eaca3bffd..7e5b60f4a 100644
--- a/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/export-lineage.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/export-lineage.mdx
@@ -5,8 +5,23 @@ sidebarTitle: Export Lineage
# Export Lineage
-You can now export the lineage information in an CSV file which makes it easier to do an advance analysis on the lineage information.
+You can export the lineage information for a data asset as a CSV, PNG, or PDF file, which makes it easier to do advanced analysis on the lineage information or share a visual snapshot.
-To export you need to navigate to the `Lineage` tab of the table, and then click on the export icon from the lineage editor.
+**To export the lineage:**
+1. Navigate to the **Explore** page and select a data asset.
+2. Click the **Lineage** tab.
+3. Click the **Export** icon, and do the following:
-
+ 
+
+ - Select an **Export Type** from *CSV*, *PNG*, or *PDF*.
+
+
+ **Note**: **PNG** and **PDF** export are currently in **Beta**.
+
+
+ - Edit the **File Name**.
+
+ 
+
+4. Click **Export**.
diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/spark-lineage.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/spark-lineage.mdx
index 2cd9233ae..73df22688 100644
--- a/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/spark-lineage.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ingestion/lineage/spark-lineage.mdx
@@ -1,113 +1,135 @@
---
-title: Spark Lineage Ingestion
+title: "Spark Lineage Ingestion"
+description: "Capture table- and column-level lineage from Apache Spark jobs in OpenMetadata using the native OpenLineage Spark integration. Includes guides for both Spark 3.x and Spark 2.4.x."
sidebarTitle: Spark Lineage
---
# Spark Lineage Ingestion
-A spark job may involve movement/transfer of data which may result into a data lineage, to capture such lineages you can make use of `OpenMetadata Spark Agent` which you can configure with your spark session and capture these spark lineages into your OpenMetadata instance.
-
-In this guide we will explain how you can make use of the `OpenMetadata Spark Agent` to capture such lineage.
-
-- [Requirements](#requirement)
-- [Configuration](#configuration)
-- [Using Spark Agent with Databricks](#using-spark-agent-with-databricks)
-- [Using Spark Agent with Glue](#using-spark-agent-with-glue)
-
-## Requirement
-
-To use the `OpenMetadata Spark Agent`, you will have to download the latest jar from [here](https://github.com/open-metadata/openmetadata-spark-agent/releases).
-
-We support spark version 3.1 and above.
-
-
-## Configuration
-
-While configuring the spark session, in this guide we will make use of PySpark to demonstrate the use of `OpenMetadata Spark Agent`
-
-
-
-
-Once you have downloaded the jar from [here](https://github.com/open-metadata/openmetadata-spark-agent/releases) in your spark configuration you will have to add the path to your `openmetadata-spark-agent.jar` along with other required jars to run your spark job, in this example it is `mysql-connector-java.jar`
-
-
-`openmetadata-spark-agent.jar` comes with a custom spark listener i.e. `io.openlineage.spark.agent.OpenLineageSparkListener` you will need to add this as `extraListeners` spark configuration.
-
-
-`spark.openmetadata.transport.hostPort`: Specify the host & port of the instance where your OpenMetadata is hosted.
-
-
-`spark.openmetadata.transport.type` is required configuration with value as `openmetadata`.
-
-
-`spark.openmetadata.transport.jwtToken`: Specify your OpenMetadata Jwt token here. Checkout [this](/v2.0.x-SNAPSHOT/deployment/security/enable-jwt-tokens#generate-token) documentation on how you can generate a jwt token in OpenMetadata.
-
-
-`spark.openmetadata.transport.pipelineServiceName`: This spark job will be creating a new pipeline service of type `Spark`, use this configuration to customize the pipeline service name.
-
-Note: If the pipeline service with the specified name already exists then we will be updating/using the same pipeline service.
-
-
-`spark.openmetadata.transport.pipelineName`: This spark job will also create a new pipeline within the pipeline service defined above. Use this configuration to customize the name of pipeline.
-
-Note: If the pipeline with the specified name already exists then we will be updating/using the same pipeline.
-
-
-`spark.openmetadata.transport.pipelineSourceUrl`: You can use this configuration to provide additional context to your pipeline by specifying a url related to the pipeline.
-
-
-`spark.openmetadata.transport.pipelineDescription`: Provide pipeline description using this spark configuration.
-
-
-`spark.openmetadata.transport.databaseServiceNames`: Provide the comma separated list of database service names which contains the source tables used in this job. If you do not provide this configuration then we will be searching through all the services available in openmetadata.
-
-
-`spark.openmetadata.transport.timeout`: Provide the timeout to communicate with OpenMetadata APIs.
-
-
-In this job we are reading data from `employee` table and moving it to another table `employee_new` of within same mysql source.
-
-
+A Spark job often moves or transforms data between tables, producing data lineage. OpenMetadata captures this lineage directly from the **standard [OpenLineage](https://openlineage.io/) Spark integration**: the open-source `openlineage-spark` listener used across the data ecosystem, with no custom agent required.
+
+
+**Important**:
+**The `openmetadata-spark-agent.jar` is superseded by the OpenLineage Spark listener — use the listener directly.**
+
+Earlier versions of this guide asked you to download and configure the custom `openmetadata-spark-agent.jar`. That JAR is **no longer required**. OpenMetadata now consumes OpenLineage events natively over HTTP at `POST /api/v1/openlineage/lineage`, so you can point the upstream `io.openlineage:openlineage-spark` listener straight at OpenMetadata.
+
+If you're still running the custom Spark Agent, migrate by changing two things in your Spark configuration:
+- Swap the agent JAR (`openmetadata-spark-agent.jar`) for the upstream `io.openlineage:openlineage-spark` package.
+- Replace the `spark.openmetadata.transport.*` properties with the `spark.openlineage.transport.*` HTTP transport properties shown below.
+
+
+## How it works
+
+1. You add the open-source `openlineage-spark` listener to your Spark session.
+2. The listener emits OpenLineage run events as your job reads and writes data.
+3. You point the listener's **HTTP transport** at OpenMetadata's OpenLineage endpoint (`/api/v1/openlineage/lineage`), authenticating with an OpenMetadata bot JSON Web Token (JWT).
+4. OpenMetadata resolves the datasets in each event to existing tables, builds table- and column-level lineage, and records a pipeline for the Spark job.
+
+- [Requirements](#requirements)
+- [Choosing the right JAR](#choosing-the-right-jar)
+- [Spark 3.x Configuration](#spark-3-x-configuration)
+- [Spark 2.4.x Configuration](#spark-2-4-x-configuration)
+- [Using OpenLineage with Databricks](#using-openlineage-with-databricks)
+- [Using OpenLineage with Glue](#using-openlineage-with-glue)
+
+## Requirements
+
+- A Spark cluster running **Spark 2.4.x** or **Spark 3.x** (see the version-specific sections below for the right JAR).
+- Network access from your Spark driver to your OpenMetadata instance.
+- An OpenMetadata **bot JWT token** for authentication. See [Generate Token](/v2.0.x-SNAPSHOT/deployment/security/enable-jwt-tokens#generate-token) for how to generate one.
+
+
+**Resolving datasets to your tables.** OpenMetadata matches the datasets in OpenLineage events to existing tables using the dataset namespace. If your Spark sources don't resolve automatically, configure the **namespace-to-service mapping** (and, optionally, auto-creation of missing entities and the default pipeline service) in OpenMetadata's OpenLineage settings. This is the server-side replacement for the old `spark.openmetadata.transport.databaseServiceNames` option from the Spark Agent.
+
+
+## Choosing the right JAR
+
+The `openlineage-spark` listener ships as a single package that you add to your Spark job. The version you use depends on your Spark version:
+
+| Spark version | `openlineage-spark` version | Maven coordinate |
+|---|---|---|
+| **Spark 3.x** | Latest release (`1.37.0` or newer) | `io.openlineage:openlineage-spark_2.12:1.37.0` |
+| **Spark 2.4.x** | **`1.36.0`** (last release to support Spark 2) | `io.openlineage:openlineage-spark_2.12:1.36.0` |
+
+
+**Note**: OpenLineage **dropped Spark 2.x support in release `1.37.0`**: from that release onward the minimum supported version is Spark 3.x. If you are on **Spark 2.4.x, pin `openlineage-spark` to `1.36.0`**. Match the Scala suffix (`_2.12` or `_2.13`) to the Scala version your Spark build was compiled with.
+
+
+## Spark 3.x Configuration
+
+The example below uses PySpark. It adds the OpenLineage listener and points its HTTP transport at OpenMetadata.
+
+### 1. Add the openlineage-spark package
+
+Add `io.openlineage:openlineage-spark_2.12:` to `spark.jars.packages` (or add a downloaded JAR to `spark.jars`) along with any other JARs your job needs, in this example the MySQL connector.
+
+### 2. Register the OpenLineage listener
+
+`openlineage-spark` ships a Spark listener, `io.openlineage.spark.agent.OpenLineageSparkListener`. Register it as a `spark.extraListeners`.
+
+### 3. spark.openlineage.transport.type
+
+Set `spark.openlineage.transport.type` to `http` so events are pushed to OpenMetadata's REST endpoint.
+
+### 4. spark.openlineage.transport.url
+
+Set `spark.openlineage.transport.url` to the base URL of your OpenMetadata instance, for example, `https://`.
+
+### 5. spark.openlineage.transport.endpoint
+
+Set `spark.openlineage.transport.endpoint` to `/api/v1/openlineage/lineage`, OpenMetadata's native OpenLineage ingestion endpoint.
+
+### 6. spark.openlineage.transport.auth
+
+Authenticate with an OpenMetadata bot JWT: set `spark.openlineage.transport.auth.type` to `api_key` and `spark.openlineage.transport.auth.apiKey` to your token. See [Generate Token](/v2.0.x-SNAPSHOT/deployment/security/enable-jwt-tokens#generate-token).
+
+### 7. spark.openlineage.namespace
+
+Set `spark.openlineage.namespace` to the **OpenLineage job namespace** that identifies the source of these events. This is **not** the pipeline service name. OpenMetadata combines the namespace with the job name to build the pipeline that represents this Spark job, and places it under the pipeline service configured as `defaultPipelineService` in OpenMetadata's OpenLineage settings (which defaults to a service named `openlineage`). To group these jobs under a specific service, set `defaultPipelineService` to an existing pipeline service rather than changing the namespace.
+
+### 8. spark.openlineage.parentJobName (optional)
+
+Optionally set `spark.openlineage.parentJobName` (and a `spark.openlineage.parentRunId`) to give the Spark job a stable OpenLineage job name and run identity. When OpenMetadata auto-creates a pipeline, it names it `-` under the configured `defaultPipelineService`.
+
+### 9. Run your job
+
+In this job we read data from the `employee` table and write it to another table, `employee_new`, within the same MySQL source.
```py
from pyspark.sql import SparkSession
+from uuid import uuid4
spark = (
SparkSession.builder.master("local")
.appName("localTestApp")
-.config(
- "spark.jars",
- "path/to/openmetadata-spark-agent.jar,path/to/mysql-connector-java-8.0.30.jar",
+ .config(
+ "spark.jars.packages",
+ # Spark 3.x: use the latest openlineage-spark release (1.37.0+)
+ "io.openlineage:openlineage-spark_2.12:1.37.0,mysql:mysql-connector-java:8.0.30",
)
-.config(
+ .config(
"spark.extraListeners",
"io.openlineage.spark.agent.OpenLineageSparkListener",
)
-.config("spark.openmetadata.transport.hostPort", "http://localhost:8585")
-.config("spark.openmetadata.transport.type", "openmetadata")
-.config("spark.openmetadata.transport.jwtToken", "")
-.config(
- "spark.openmetadata.transport.pipelineServiceName", "my_pipeline_service"
- )
-.config("spark.openmetadata.transport.pipelineName", "my_pipeline_name")
-.config(
- "spark.openmetadata.transport.pipelineSourceUrl",
- "http://your.org/path/to/pipeline",
- )
-.config(
- "spark.openmetadata.transport.pipelineDescription", "My ETL Pipeline"
+ .config("spark.openlineage.transport.type", "http")
+ .config("spark.openlineage.transport.url", "https://")
+ .config(
+ "spark.openlineage.transport.endpoint",
+ "/api/v1/openlineage/lineage",
)
-.config(
- "spark.openmetadata.transport.databaseServiceNames",
- "random,local_mysql",
+ .config("spark.openlineage.transport.auth.type", "api_key")
+ .config(
+ "spark.openlineage.transport.auth.apiKey",
+ "",
)
-.config("spark.openmetadata.transport.timeout", "30")
-.getOrCreate()
+ .config("spark.openlineage.namespace", "my_spark_namespace")
+ .config("spark.openlineage.parentJobName", "my_pipeline_name")
+ .config("spark.openlineage.parentRunId", str(uuid4()))
+ .getOrCreate()
)
-# Read table using jdbc()
-
-# Read from MySQL Table
+# Read from MySQL table
employee_df = (
spark.read.format("jdbc")
.option("url", "jdbc:mysql://localhost:3306/openmetadata_db")
@@ -134,127 +156,145 @@ employee_df = (
spark.stop()
```
+Once this PySpark job finishes, OpenMetadata records the lineage between `employee` and `employee_new`. The Spark job appears as a pipeline named `my_spark_namespace-my_pipeline_name`, built from the namespace and job name, under the **`openlineage`** pipeline service (the default `defaultPipelineService`). To land these pipelines in a different service, point `defaultPipelineService` at an existing pipeline service in OpenMetadata's OpenLineage settings.
-Once this pyspark job get finished you will see a new pipeline service with name `my_pipeline_service` generated in your openmetadata instance which would contain a pipeline with name `my_pipeline` as per the above example and you should also see lineage between the table `employee` and `employee_new` via `my_pipeline`.
+
+
-
+
+## Spark 2.4.x Configuration
-
+The configuration is **identical to Spark 3.x except for the `openlineage-spark` version**, which must be pinned to **`1.36.0`**: the last OpenLineage release that supports Spark 2.x. Everything else (the listener class and all `spark.openlineage.transport.*` properties) is the same.
+### 1. Pin openlineage-spark to 1.36.0
+Use `io.openlineage:openlineage-spark_2.12:1.36.0`. Versions `1.37.0` and later require Spark 3.x and will not work on Spark 2.4.x. Make sure your Spark 2.4 build is compiled with Scala 2.12 to match the `_2.12` artifact.
-
+### 2. Use the same transport configuration
-## Using Spark Agent with Databricks
+The listener class and all `spark.openlineage.transport.*` / `spark.openlineage.namespace` properties are exactly the same as in the Spark 3.x section above.
-Follow the below steps in order to use OpenMetadata Spark Agent with databricks.
+```py
+from pyspark.sql import SparkSession
+from uuid import uuid4
-### 1. Upload the jar to compute cluster
+spark = (
+ SparkSession.builder.master("local")
+ .appName("localTestApp")
+ .config(
+ "spark.jars.packages",
+ # Spark 2.4.x: pin to 1.36.0 (last release supporting Spark 2)
+ "io.openlineage:openlineage-spark_2.12:1.36.0,mysql:mysql-connector-java:8.0.30",
+ )
+ .config(
+ "spark.extraListeners",
+ "io.openlineage.spark.agent.OpenLineageSparkListener",
+ )
+ .config("spark.openlineage.transport.type", "http")
+ .config("spark.openlineage.transport.url", "https://")
+ .config(
+ "spark.openlineage.transport.endpoint",
+ "/api/v1/openlineage/lineage",
+ )
+ .config("spark.openlineage.transport.auth.type", "api_key")
+ .config(
+ "spark.openlineage.transport.auth.apiKey",
+ "",
+ )
+ .config("spark.openlineage.namespace", "my_spark_namespace")
+ .config("spark.openlineage.parentJobName", "my_pipeline_name")
+ .config("spark.openlineage.parentRunId", str(uuid4()))
+ .getOrCreate()
+)
-To use the `OpenMetadata Spark Agent`, you will have to download the latest jar from [here](https://github.com/open-metadata/openmetadata-spark-agent/releases) and upload it to your databricks compute cluster.
+# ... your read/write logic, identical to the Spark 3.x example ...
-To upload the jar you can visit the compute details page and then go to the libraries tab
+spark.stop()
+```
-
+## Using OpenLineage with Databricks
-Click on the "Install Now" button and choose `dbfs` mode and upload the `OpenMetadata Spark Agent` jar.
+Follow the steps below to capture lineage from Databricks Spark jobs into OpenMetadata using the OpenLineage listener.
-
+### 1. Install the openlineage-spark library on the cluster
-Once your jar is uploaded copy the path of the jar for the next steps.
+The simplest approach is to add `openlineage-spark` as a **Maven library** on your cluster:
-
+1. Open the compute (cluster) details page and go to the **Libraries** tab.
+2. Click **Install new** > **Maven**.
+3. Enter the coordinate for your Databricks Runtime's Spark version:
+ - **Spark 3.x** (DBR 7.3+): `io.openlineage:openlineage-spark_2.12:1.37.0` (or newer)
+ - **Spark 2.4.x** (DBR 6.x): `io.openlineage:openlineage-spark_2.12:1.36.0`
+4. Click **Install** and wait for the library to attach.
+
-### 2. Create Initialization Script
+
+**Tip**: If your environment cannot reach Maven Central, download the `openlineage-spark` JAR, upload it to DBFS, and either install it as a DBFS library or copy it into `/databricks/jars` via a cluster init script.
+
+### 2. Configure the cluster Spark settings
-Once your jar is uploaded you need to create a initialization script in your workspace.
+Go to **Advanced options** > **Spark** > **Spark config** and add the OpenLineage listener and HTTP transport settings:
+
+
```
-#!/bin/bash
+spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener
+spark.openlineage.transport.type http
+spark.openlineage.transport.url https://
+spark.openlineage.transport.endpoint /api/v1/openlineage/lineage
+spark.openlineage.transport.auth.type api_key
+spark.openlineage.transport.auth.apiKey
+spark.openlineage.namespace databricks_spark
+```
-STAGE_DIR_JAR=""
+### 3. (Optional) Set the listener via an init script
-echo "BEGIN: Upload Spark Listener JARs"
-cp -f $STAGE_DIR_JAR /mnt/driver-daemon/jars || { echo "Error copying Spark Listener library file"; exit 1;}
-echo "END: Upload Spark Listener JARs"
+If you prefer to register the listener through an init script, for example to guarantee it loads before the driver starts, create the script in your workspace and attach it under **Advanced options** > **Init Scripts**:
+
+```bash
+#!/bin/bash
-echo "BEGIN: Modify Spark config settings"
cat << 'EOF' > /databricks/driver/conf/openlineage-spark-driver-defaults.conf
[driver] {
"spark.extraListeners" = "io.openlineage.spark.agent.OpenLineageSparkListener"
}
EOF
-echo "END: Modify Spark config settings"
```
-Note: The copied path would look like this `dbfs:/FileStore/jars/....` you need to modify it like `/dbfs/FileStore/jars/...` this.
+
-
+After the library and configuration are in place, start (or restart) the cluster. Your Databricks Spark jobs will now push lineage to OpenMetadata.
+## Using OpenLineage with Glue
+Follow the steps below to capture lineage from AWS Glue Spark jobs into OpenMetadata.
-### 3. Configure Initialization Script
+
+**Note**: Match the `openlineage-spark` version to your Glue version's Spark runtime: Glue 3.0/4.0/5.0 run Spark 3.x (use `1.37.0` or newer), while Glue 2.0 runs Spark 2.4.x (use `1.36.0`).
+
-Once you have created a initialization script, you will need to attach this script to your compute instance, to do that you can go to advanced config > init scripts and add your script path.
+### 1. Provide the openlineage-spark JAR
-
-
-
-
-
-
-### 4. Configure Spark
-
-After configuring the init script, you will need to update the spark config as well.
-
-
-
-
-these are the possible configurations that you can do, please refer the `Configuration` section above to get the detailed information about the same.
-
-```
-spark.extraListeners io.openlineage.spark.agent.OpenLineageSparkListener
-spark.openmetadata.transport.type openmetadata
-spark.openmetadata transport.pipelineSourceUrl http://
-spark.openmetadata transport.pipelineDescription "your pipeline description"
-spark.openmetadata.transport.hostPort https://
-spark openmetadata transport.pipelineServiceName demo_pipeline
-spark.openmetadata transport.pipelineName demo_pipeline
-spark.openmetadata transport.databaseServiceNames db-service-name1,db-service-name2
-spark.openmetadata.transport.jwtToken
-spark.openmetadata.transport.timeout 30
-```
-
-After all these steps are completed you can start/restart your compute instance and you are ready to extract the lineage from spark to OpenMetadata.
-
-
-## Using Spark Agent with Glue
-
-Follow the below steps in order to use OpenMetadata Spark Agent with glue.
-
-### 1. Specify the OpenMetadata Spark Agent JAR URL
-
-1. Upload the OpenMetadata Spark Agent Jar to S3
-2. Navigate to the glue job,In the Job details tab, navigate to Advanced properties → Libraries → Dependent Jars path
-3. Add the S3 url of OpenMetadata Spark Agent Jar in the Dependent Jars path.
+1. Download the `openlineage-spark` JAR for your Spark version and upload it to S3.
+2. Open the Glue job and, in the **Job details** tab, go to **Advanced properties** > **Libraries** > **Dependent JARs path**.
+3. Add the S3 URL of the `openlineage-spark` JAR to the **Dependent JARs path**.
+### 2. Add the Spark configuration in Job parameters
-### 2. Add Spark configuration in Job Parameters
-
-In the same Job details tab, add a new property under Job parameters:
+In the same **Job details** tab, add a job parameter:
-1. Add the `--conf` property with following value, make sure to customize this configuration as described in the above documentation.
+1. Add a `--conf` parameter with the following value (customize the host, token, and namespace):
```
-spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener --conf spark.openmetadata.transport.hostPort=https://your-org.host:port --conf spark.openmetadata.transport.type=openmetadata --conf spark.openmetadata.transport.jwtToken= --conf spark.openmetadata.transport.pipelineServiceName=glue_spark_pipeline_service --conf spark.openmetadata.transport.pipelineName=glue_pipeline_name --conf spark.openmetadata.transport.timeout=30
+spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener --conf spark.openlineage.transport.type=http --conf spark.openlineage.transport.url=https:// --conf spark.openlineage.transport.endpoint=/api/v1/openlineage/lineage --conf spark.openlineage.transport.auth.type=api_key --conf spark.openlineage.transport.auth.apiKey= --conf spark.openlineage.namespace=glue_spark
```
-2. Add the `--user-jars-first` parameter and set its value to `true`
+2. Add the `--user-jars-first` parameter and set its value to `true`.
diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage.mdx
index 064188369..308712022 100644
--- a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/lineage.mdx
@@ -30,41 +30,74 @@ Learn how to do so 👇
Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Entity Lineage information.
-This will populate the Lineage tab from the Table Entity Page.
+This will populate the **Lineage** tab from the Table Entity Page.
-
+
+We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Lineage Agent will be in charge of obtaining this data.
-We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Lineage Ingestion will be in charge of obtaining this data.
+### Step 1: Add a Lineage Agent
-### 1. Add a Lineage Ingestion
+Navigate to your service and launch the Lineage Agent wizard.
-From the Service Page, go to the Ingestions tab to add a new ingestion and click on Add Lineage Ingestion.
+1. Navigate to **Settings** > **Services** and select the service type.
+2. Click the service you have added.
+3. Open the **Agents** tab and click **Add Agent** > **Lineage**.
-
+ 
-### 2. Configure the Lineage Ingestion
+4. Give the agent a recognisable name and click **Next**.
-Here you can enter the Lineage Ingestion details:
+### Step 2: Configure the Agent
-
+Configure the core parameters for this agent.
-### Lineage Options
+
-**Query Log Duration**
+**Agent Setup**
-Specify the duration in days for which the lineage should capture lineage data from the query logs. For example, if you specify 2 as the value for the duration, the data lineage will capture lineage information for 48 hours prior to when the ingestion workflow is run.
+| Field | Default | Description |
+|-------|---------|-------------|
+| **Query Log Duration** | `1` | How many days back to look in the query log. |
+| **Query Log File Path** | — | Path to a query log file. Use this instead of reading directly from the database. |
+| **Result Limit** | `1000` | Maximum number of query log rows to process per run. |
+| **Parsing Timeout Limit** | `300` | Timeout in seconds for parsing a single SQL query. |
+| **Query Parser Type** | `Auto` | SQL parser for lineage extraction. **Auto** tries SqlGlot first, then SqlFluff, then SqlParse. **SqlGlot** offers high performance with excellent dialect support. **SqlFluff** provides comprehensive dialect support. |
+| **Filter Condition** | — | SQL condition applied to the query history result set to exclude specific queries. See [Usage Query Filtering](/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage/filter-query-set). |
+| **Number of Threads** | `1` | Number of threads to use for parallel lineage ingestion. |
+| **Cross Database Service Names** | — | Names of other OpenMetadata services to search when resolving table references during lineage extraction. List specific service names. The underlying lookup returns a limited number of matches, so `*` can miss the correct service or resolve to the wrong table when many services or tables share names. |
+| **Override View Lineage** | Off | When enabled, overwrites existing view lineage on each run. |
+| **Process View Lineage** | On | Parse view definitions for lineage. |
+| **Process Query Lineage** | On | Parse query logs for lineage. |
+| **Process Stored Procedure Lineage** | On | Parse stored procedure definitions for lineage. |
+| **Process Cross Database Lineage** | Off | Build lineage across tables in different databases or projects (for example, across BigQuery projects). Also set **Cross Database Service Names** when enabling this option. See [Cross Database Lineage](/v2.0.x-SNAPSHOT/how-to-guides/data-lineage/cross-database-lineage). |
+| **Enable Temp Table Lineage** | Off | Handle lineage for Snowflake temporary and transient tables. |
+| **Incremental Lineage Processing** | On | Process only new or changed lineage data on each run. |
-**Result Limit**
+**Filter Patterns**
-Set the limit for the query log results to be run at a time.
+Apply regex patterns to include or exclude specific databases, schemas, tables, or stored procedures.
+**Scope & Behaviour**
-### 3. Schedule and Deploy
+| Field | Description |
+|-------|-------------|
+| **Enable Debug Log** | Turn on verbose logging to help troubleshoot ingestion issues. |
-After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the lineage pipeline being added to the Service Ingestions.
+### Step 3: Schedule and Deploy
-
+Choose how the lineage agent should run:
+
+- **Schedule**: Runs repeatedly based on a defined interval. Set the **Frequency** (Hourly, Daily, Weekly, Monthly, or Custom) and the **Time** to run.
+- **On Demand**: Runs once when manually triggered.
+
+Configure the remaining options:
+- **Number of Retries**: How many times to retry if the ingestion fails.
+- **Raise on Error**: Toggle on to surface errors immediately rather than silently skipping them.
+
+Click **Add & Deploy** to save and activate the lineage agent.
+
+
## YAML Configuration
diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/metadata/incremental-extraction/unity-catalog.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/metadata/incremental-extraction/unity-catalog.mdx
index 8c5beeb3a..5680a6358 100644
--- a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/metadata/incremental-extraction/unity-catalog.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/metadata/incremental-extraction/unity-catalog.mdx
@@ -28,20 +28,25 @@ A table that appears in both the changed and deleted sets is kept, not deleted.
This covers tables that were dropped and recreated within the window, since
`information_schema` only lists currently existing tables.
+
+ Incremental extraction only runs for catalogs whose name matches
+ `[A-Za-z0-9_]+`. Catalogs with names outside this pattern (for example,
+ quoted names like `cat-a-log`) are skipped with a warning: no changed or
+ deleted tables are returned, and no exception is raised.
+
+
## Prerequisites
-Before enabling incremental extraction, ensure the ingestion user has the
-necessary permissions on the Unity Catalog system tables. The following grants
-are the minimum required for each detection mode.
+Before enabling incremental extraction, review the following visibility and
+permission requirements for the Unity Catalog system tables used at runtime.
### Changed table detection
-The ingestion user must have `SELECT` on the catalog's
-`information_schema.tables`:
-
-```sql
-GRANT SELECT ON ``.information_schema.tables TO ``;
-```
+No explicit grant is required for `information_schema.tables`: Databricks
+automatically scopes its results to objects the querying principal can
+already see through existing Unity Catalog privileges. Ensure the ingestion
+service principal has standard read privileges (`USE CATALOG`, `USE SCHEMA`,
+`SELECT`) on the tables you want incrementally tracked.
### Deleted table detection (optional)
diff --git a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage.mdx b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage.mdx
index b35c64c1b..1efe16ed7 100644
--- a/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage.mdx
@@ -35,46 +35,51 @@ Learn how to do so 👇
Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Query Usage information.
-This will populate the Queries tab from the Table Entity Page.
+This will populate the **Queries** tab from the Table Entity Page.
-
+
-We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Usage Ingestion will be in charge of obtaining this data.
+We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Usage Agent will be in charge of obtaining this data.
-### 1. Add a Usage Ingestion
+### Step 1: Add a Usage Agent
-From the Service Page, go to the Ingestions tab to add a new ingestion and click on Add Usage Ingestion.
+1. Navigate to **Settings** > **Services** and select the service type.
+2. Click the service you have added.
+3. Open the **Agents** tab and click **Add Agent** > **Usage**.
-
+ 
+4. Give the agent a recognisable name and click **Next**.
-### 2. Configure the Usage Ingestion
+### Step 2: Configure the Agent
-Here you can enter the Usage Ingestion details:
+Configure the core parameters for the Usage agent:
-
+
+| **Field** | **Description** |
+|:--- |:--- |
+| **Query Log Duration** | How many days back to look in query logs to process usage data. Default is `1`. |
+| **Stage File Location** | Absolute file path for the temporary file used to store query logs before processing. Default is `/tmp/query_log`. |
+| **Filter Condition** | Optional condition to filter which queries are included from the query history. |
+| **Result Limit** | Maximum number of query log results to process at a time. Default is `1000`. |
+| **Query Log File Path** | File path to the query logs, if reading from a file rather than querying the source directly. |
+| **Process Query Cost** | Toggle to enable processing of query cost data alongside usage. |
-### Usage Options
+### Step 3: Schedule and Deploy
-**Query Log Duration**
+Choose how the usage agent should run:
-Specify the duration in days for which the usage should capture usage data from the query logs. For example, if you specify 2 as the value for the duration, the data usage will capture usage information for 48 hours prior to when the ingestion workflow is run.
+
-**Stage File Location**
+- **Schedule**: Runs repeatedly based on a defined interval. Set the **Frequency** (Hourly, Daily, Weekly, Monthly, or Custom) and the **Time** to run.
+- **On Demand**: Runs once when manually triggered.
-Mention the absolute file path of the temporary file name to store the query logs before processing.
+Configure the remaining options:
+- **Number of Retries**: How many times to retry if the ingestion fails.
+- **Raise on Error**: Toggle on to surface errors immediately rather than silently skipping them.
-**Result Limit**
-
-Set the limit for the query log results to be run at a time.
-
-
-### 3. Schedule and Deploy
-
-After clicking Next, you will be redirected to the Scheduling form. This will be the same as the Metadata Ingestion. Select your desired schedule and click on Deploy to find the usage pipeline being added to the Service Ingestions.
-
-
+Click **Add & Deploy** to save and activate the usage agent.
## YAML Configuration
diff --git a/v2.0.x-SNAPSHOT/connectors/messaging/kafka.mdx b/v2.0.x-SNAPSHOT/connectors/messaging/kafka.mdx
index 17d67bd2e..82e03804d 100644
--- a/v2.0.x-SNAPSHOT/connectors/messaging/kafka.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/messaging/kafka.mdx
@@ -3,11 +3,9 @@ title: Kafka Connector | OpenMetadata Messaging Integration
description: Connect Kafka to OpenMetadata effortlessly with our comprehensive connector guide. Set up messaging metadata ingestion, configuration, and monitoring in minutes.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/messaging/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMessaging from '/snippets/connectors/metadata-agent/messaging.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Kafka, you need to create a service connection. The service connects Kafka with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Messaging Services**, then click the **Kafka** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Kafka services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Kafka. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Bootstrap Servers**: List of brokers as comma separated values of broker `host` or `host:port`. Example: `host1:9092,host2:9092`
- **Schema Registry URL**: URL of the Schema Registry used to ingest the schemas of the topics.
- **SASL Username**: SASL username for use with the PLAIN and SASL-SCRAM mechanisms.
- **SASL Password**: SASL password for use with the PLAIN and SASL-SCRAM mechanisms.
- **SASL Mechanism**: SASL mechanism to use for authentication.
- **Basic Auth User Info**: Schema Registry Client HTTP credentials in the form of `username:password`. By default, user info is extracted from the URL if present.
-- **Consumer Config**: The accepted additional values for the consumer configuration can be found in the following [link](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.mdx).
-If you are using Confluent kafka and SSL encryption is enabled you need to add `security.protocol` as key and `SASL_SSL` as value under Consumer Config
+- **Consumer Config**: The accepted additional values for the consumer configuration can be found in the following [link](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).
+If you are using Confluent Kafka and SSL encryption is enabled, you need to add `security.protocol` as key and `SASL_SSL` as value under Consumer Config
- **Schema Registry Config**: The accepted additional values for the Schema Registry configuration can be found in the following [link](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#schemaregistryclient).
+
To ingest the topic schema `Schema Registry URL` must be passed
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your messaging service. Filter patterns use regular expressions applied to topic names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all topics available in the source.
+
+
+**Filter Options**
+
+The Topic section includes the following filter options:
+
+- **Topic**: Controls which topics (Kafka topics, Pulsar topics, or equivalent messaging channels) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Securing Kafka Connection with SSL in OpenMetadata
-To establish secure connections between OpenMetadata and Kafka, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl key`, `ssl cert`, and `caCertificate`. In this case, `ssl_cert` is used for the client’s SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `caCertificate` for the CA certificate to validate the server’s certificate.
- 
+
+To establish secure connections between OpenMetadata and Kafka, navigate to the `Advanced Config` section. Here, you can provide the CA certificate used for SSL validation by specifying the `caCertificate`. Alternatively, if both client and server require mutual authentication, you'll need to use all three parameters: `ssl key`, `ssl cert`, and `caCertificate`. In this case, `ssl_cert` is used for the client's SSL certificate, `ssl_key` for the private key associated with the SSL certificate, and `caCertificate` for the CA certificate to validate the server's certificate.
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/messaging/kinesis.mdx b/v2.0.x-SNAPSHOT/connectors/messaging/kinesis.mdx
index 1239d14a6..e2bc7874b 100644
--- a/v2.0.x-SNAPSHOT/connectors/messaging/kinesis.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/messaging/kinesis.mdx
@@ -3,11 +3,9 @@ title: Kinesis Connector | OpenMetadata Streaming Integration
description: Connect Amazon Kinesis to OpenMetadata with our comprehensive connector guide. Set up real-time streaming data ingestion and metadata extraction easily.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/messaging/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMessaging from '/snippets/connectors/metadata-agent/messaging.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
+
+To ingest metadata from Kinesis, you need to create a service connection. The service connects Kinesis with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Messaging Services**, then click the **Kinesis** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Kinesis services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Kinesis. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+You can find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
@@ -75,22 +106,24 @@ You can find more information on [Using temporary credentials with AWS resources
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
+
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
@@ -98,8 +131,60 @@ Find more information about the [Role Session Name](https://docs.aws.amazon.com/
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your messaging service. Filter patterns use regular expressions applied to topic names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all topics available in the source.
+
+
+**Filter Options**
+
+The Topic section includes the following filter options:
+
+- **Topic**: Controls which topics (Kafka topics, Pulsar topics, or equivalent messaging channels) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/messaging/pubsub.mdx b/v2.0.x-SNAPSHOT/connectors/messaging/pubsub.mdx
index 3959337b0..81fe4960b 100644
--- a/v2.0.x-SNAPSHOT/connectors/messaging/pubsub.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/messaging/pubsub.mdx
@@ -3,12 +3,9 @@ title: Google Pub/Sub Connector | OpenMetadata Messaging Integration
description: Connect Google Cloud Pub/Sub to OpenMetadata with our comprehensive connector guide. Set up topic metadata ingestion, schema registry, and subscription configuration in minutes.
sidebarTitle: Overview
---
-
-import ConfigureIngestion from '/snippets/connectors/messaging/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMessaging from '/snippets/connectors/metadata-agent/messaging.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+To ingest metadata from Google Pub/Sub, you need to create a service connection. The service connects Pub/Sub with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
-# Connection Details
+### Step 1: Add New Service
-
-
+1. Navigate to **Settings** > **Services**.
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
-- **GCP Credentials**: GCP service account credentials for authenticating with Pub/Sub. Provide a service account key in JSON format, or use Application Default Credentials when running on GCP infrastructure (GCE, GKE, Cloud Run). See [Creating a GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) for details.
+2. Click **Add New Service**.
-- **Project ID** (optional): GCP Project ID where Pub/Sub topics are located. If not specified, the project ID is read from the service account credentials.
+
-- **Host and Port** (optional): Pub/Sub API endpoint URL. Defaults to `pubsub.googleapis.com`. When connecting to a local **Pub/Sub emulator**, set this to the emulator address (e.g., `localhost:8085`) and enable **Use Emulator**.
+### Step 2: Select a Service and Connector
-- **Use Emulator** (optional): Connect to a local Pub/Sub emulator instead of the production service. Useful for development and testing. When enabled, `hostPort` must be set to the emulator address (not the default `pubsub.googleapis.com`).
+From the service type dropdown, select **Messaging Services**, then click the **Pub/Sub** connector tile.
-- **Enable Schema Registry** (optional, default: `true`): Fetch topic schemas from the Pub/Sub Schema Registry. Supports Avro and Protocol Buffer schema types. Disable if your project does not use the Schema Registry.
+
+
+### Step 3: Add Service Name and Description
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Pub/Sub services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Google Pub/Sub. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **GCP Credentials**: GCP service account credentials for authenticating with Pub/Sub. Provide a service account key in JSON format, or use Application Default Credentials when running on GCP infrastructure (GCE, GKE, Cloud Run). See [Creating a GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) for details.
+- **Project ID** (optional): GCP Project ID where Pub/Sub topics are located. If not specified, the project ID is read from the service account credentials.
+- **Host and Port** (optional): Pub/Sub API endpoint URL. Defaults to `pubsub.googleapis.com`. When connecting to a local **Pub/Sub emulator**, set this to the emulator address (for example, `localhost:8085`) and enable **Use Emulator**.
+- **Use Emulator** (optional): Connect to a local Pub/Sub emulator instead of the production service. Useful for development and testing. When enabled, `hostPort` must be set to the emulator address (not the default `pubsub.googleapis.com`).
+- **Enable Schema Registry** (optional, default: `true`): Fetch topic schemas from the Pub/Sub Schema Registry. Supports Avro and Protocol Buffer schema types. Disable if your project does not use the Schema Registry.
- **Include Subscriptions** (optional, default: `true`): Include subscription metadata for each topic. When enabled, subscription names, acknowledgment deadlines, retention durations, push endpoints, dead letter policies, and BigQuery export configurations are captured.
@@ -98,11 +117,53 @@ When a subscription has a **BigQuery export configuration**, OpenMetadata automa
- **Include Dead Letter Topics** (optional, default: `false`): Include dead letter topics in metadata extraction. By default, dead letter topics are detected via subscription policies and excluded to keep the topic list focused on primary business topics.
-
- **Topic Filter Pattern** (optional): Regex pattern to selectively include or exclude topics by name. Use `includes` for an allow-list and `excludes` for a deny-list. Example: exclude internal topics with `excludes: ["^_.*"]`.
-
-
-
-
-
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which topics OpenMetadata ingests from your Pub/Sub service. Filter patterns use regular expressions applied to topic names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only topics whose names match at least one expression. Leave blank to include all topics.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any topic whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match topic names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `orders` matches `my_orders_topic` and `orders_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_events` and `prod_orders`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all topics available in the source.
+
+
+**Filter Options**
+
+- **Topic**: Controls which topics OpenMetadata ingests from the source.
+- **Scan Mode**: Choose between **Scan all** (default) or **Only specific** to enable include rules.
+- **Always exclude**: Add permanent exclusion rules. Topics matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/messaging/redpanda.mdx b/v2.0.x-SNAPSHOT/connectors/messaging/redpanda.mdx
index c91fd8617..f22d489fe 100644
--- a/v2.0.x-SNAPSHOT/connectors/messaging/redpanda.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/messaging/redpanda.mdx
@@ -3,11 +3,9 @@ title: Redpanda Connector | OpenMetadata Messaging Services
description: Connect OpenMetadata to Redpanda with our official messaging connector. Stream metadata, automate discovery, and integrate your Kafka-compatible platform seamlessly.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/messaging/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMessaging from '/snippets/connectors/metadata-agent/messaging.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Redpanda, you need to create a service connection. The service connects Redpanda with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Messaging Services**, then click the **Redpanda** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Redpanda services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Redpanda. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Bootstrap Servers**: List of brokers as comma separated values of broker `host` or `host:port`. Example: `host1:9092,host2:9092`
- **Schema Registry URL**: URL of the Schema Registry used to ingest the schemas of the topics.
- **SASL Username**: SASL username for use with the PLAIN and SASL-SCRAM mechanisms.
- **SASL Password**: SASL password for use with the PLAIN and SASL-SCRAM mechanisms.
- **SASL Mechanism**: SASL mechanism to use for authentication.
- **Basic Auth User Info**: Schema Registry Client HTTP credentials in the form of `username:password`. By default, user info is extracted from the URL if present.
-- **Consumer Config**: The accepted additional values for the consumer configuration can be found in the following [link](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.mdx).
+- **Consumer Config**: The accepted additional values for the consumer configuration can be found in the following [link](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).
- **Schema Registry Config**: The accepted additional values for the Schema Registry configuration can be found in the following [link](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#schemaregistryclient).
- **Redpanda Admin API URL** *(Optional)*: URL of the Redpanda Admin API, typically running on port `9644`. When configured, OpenMetadata will extract data transform metadata to create topic-to-topic lineage. Example: `http://localhost:9644`
+
To ingest the topic schema `Schema Registry URL` must be passed
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your messaging service. Filter patterns use regular expressions applied to topic names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all topics available in the source.
+
+
+**Filter Options**
+
+The Topic section includes the following filter options:
+
+- **Topic**: Controls which topics (Kafka topics, Pulsar topics, or equivalent messaging channels) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
## Securing Redpanda Connection with SSL
@@ -64,7 +144,7 @@ You can configure SSL separately for the **Consumer Config** (broker connection)
Redpanda and Kafka connectors support extracting consumer group metadata for topics. When enabled, each topic will display the active consumer groups consuming from it, including:
-- Group ID and current state (Stable, Empty, PreparingRebalance, etc.)
+- Group ID and current state (Stable, Empty, PreparingRebalance, and so on)
- Member details (client IDs, hosts, assigned partitions)
- Partition assignment strategy
@@ -76,9 +156,11 @@ Redpanda supports native [WebAssembly data transforms](https://docs.redpanda.com
## Kafka Connect Lineage
-Redpanda is wire-compatible with Apache Kafka, so the existing **Kafka Connect** pipeline connector works with Redpanda out of the box. If you have Kafka Connect sink connectors running against Redpanda (e.g., JDBC Sink, BigQuery Sink, S3 Sink), you can set up a [Kafka Connect pipeline](/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect) in OpenMetadata to automatically extract topic-to-table lineage.
+Redpanda is wire-compatible with Apache Kafka, so the existing **Kafka Connect** pipeline connector works with Redpanda out of the box. If you have Kafka Connect sink connectors running against Redpanda (for example, JDBC Sink, BigQuery Sink, S3 Sink), you can set up a [Kafka Connect pipeline](/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect) in OpenMetadata to automatically extract topic-to-table lineage.
To configure this:
1. Set up a Redpanda messaging service in OpenMetadata (this connector)
2. Set up a **Kafka Connect** pipeline service, pointing to your Kafka Connect REST API
3. The pipeline connector will automatically detect sink connectors and create lineage from Redpanda topics to destination tables
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/metadata/alationsink.mdx b/v2.0.x-SNAPSHOT/connectors/metadata/alationsink.mdx
index 0d12ef6d1..1bb6c394a 100644
--- a/v2.0.x-SNAPSHOT/connectors/metadata/alationsink.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/metadata/alationsink.mdx
@@ -3,12 +3,10 @@ title: Alation Sink | OpenMetadata Metadata Integration
description: Connect Alation to OpenMetadata seamlessly with our AlationSink connector. Complete setup guide, configuration steps, and metadata sync instructions.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/metadata/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMetadata from '/snippets/connectors/metadata-agent/metadata.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+
## Requirements
+
The connector uses `POST` requests to write the data into Alation.
-Hence, an user credentials or an access token with `Source Admin` or `Catalog Admin` or `Server Admin` permissions will be required.
-Follow the link [here](https://developer.alation.com/dev/docs/authentication-into-alation-apis#create-via-ui) to create the access token.
+Hence, user credentials or an access token with `Source Admin`, `Catalog Admin`, or `Server Admin` permissions will be required.
+Follow the [Alation API authentication guide](https://developer.alation.com/dev/docs/authentication-into-alation-apis#create-via-ui) to create the access token.
+
## Data Mapping and Assumptions
-Following entities are supported and will be mapped to the from OpenMetadata to the entities in Alation.
+
+Following entities are supported and will be mapped from OpenMetadata to the entities in Alation.
+
| Alation Entity | OpenMetadata Entity |
| :----------------------------| :--------------------------- |
| Data Source (OCF) | Database |
| Schema | Schema |
| Table | Table |
| Columns | Columns |
+
## Metadata Ingestion
-Then, prepare the Alation Sink Service and configure the Ingestion:
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from AlationSink, you need to create a service connection. The service connects AlationSink with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Metadata Services**, then click the **AlationSink** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other AlationSink services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for AlationSink. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: Host and port of the Alation service.
- **Authentication Types**:
1. Basic Authentication
- Username: The name of the user whose credentials will be used to sign in.
- Password: The password of the user.
2. Access Token Authentication
- The access token created using the steps mentioned [here](https://developer.alation.com/dev/docs/authentication-into-alation-apis#create-via-ui) can directly be entered. We'll use that directly to authenticate the Alation APIs
+ The access token created using the steps in the [Alation API authentication guide](https://developer.alation.com/dev/docs/authentication-into-alation-apis#create-via-ui) can directly be entered. We'll use that directly to authenticate the Alation APIs.
- accessToken: Generated access token
-- **Project Name**: Project name to create the refreshToken. Can be anything.
-- **Pagination Limit**: Pagination limit used for Alation APIs pagination
+- **Project Name**: Project name used to create the refresh token. It can be anything.
+- **Pagination Limit**: Pagination limit used for Alation APIs pagination.
- **DataSource Links**: Add a custom mapping between OpenMetadata databases and Alation DataSources.
-If this mapping is present the connector will only look for the datasource in Alation to create other entities inside it. It will not create the datasource in Alation and it'll need to be created beforehand.
-The mapping needs to be of the format `alation_datasource_id: openmetadata_database_fqn`
-Here `alation_datasource_id` corresponds to the numerical id of the datasource in alation.
+If this mapping is present, the connector will only look for the data source in Alation to create other entities inside it. It will not create the data source in Alation, so it needs to be created beforehand.
+The mapping needs to be of the format `alation_datasource_id: openmetadata_database_fqn`.
+Here `alation_datasource_id` corresponds to the numerical ID of the data source in Alation.
And `openmetadata_database_fqn` corresponds to the fullyQualifiedName of the database in OpenMetadata.
Below is an example of the mapping:
+
```yaml
datasourceLinks: {
"23": "sample_data.ecommerce_db",
"15": "mysql_prod.customers_db",
}
```
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your source. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/metadata/amundsen.mdx b/v2.0.x-SNAPSHOT/connectors/metadata/amundsen.mdx
index 95b500a96..3ecd99c75 100644
--- a/v2.0.x-SNAPSHOT/connectors/metadata/amundsen.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/metadata/amundsen.mdx
@@ -3,11 +3,9 @@ title: Amundsen Connector | OpenMetadata Metadata Services
description: Connect Amundsen to OpenMetadata effortlessly with our comprehensive connector guide. Step-by-step setup, configuration tips, and metadata integration b...
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/metadata/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMetadata from '/snippets/connectors/metadata-agent/metadata.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+In this section, we provide guides and references to use the Amundsen connector.
+Configure and schedule Amundsen metadata workflow from the OpenMetadata UI:
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
+- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/metadata/amundsen/troubleshooting)
+
+## Requirements
+
+To connect to Amundsen, you will need:
+- **Username**: A user account with privileges to read metadata from Amundsen.
+- **Password**: The password for the user account.
+- **Host and Port**: The URI of your Amundsen Neo4j Connection (for example, `bolt://localhost:7687`).
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Amundsen, you need to create a service connection. The service connects Amundsen with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Metadata Services**, then click the **Amundsen** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Amundsen services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Amundsen. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Username**: Enter the username of your Amundsen user in the Username field. The specified user should be authorized to read all databases you want to include in the metadata ingestion workflow.
+- **Password**: Enter the password for your Amundsen user in the Password field.
+- **Host and Port**: Host and port of the Amundsen Neo4j Connection. This expects a URI format like `bolt://localhost:7687`.
+- **Max Connection Life Time** *(optional)*: Maximum connection lifetime for the Amundsen Neo4j Connection.
+- **Validate SSL** *(optional)*: Enable SSL validation for the Amundsen Neo4j Connection.
+- **Encrypted** *(optional)*: Enable encryption for the Amundsen Neo4j Connection.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your source. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
-- **username**: Enter the username of your Amundsen user in the Username field. The specified user should be authorized to read all databases you want to include in the metadata ingestion workflow.
-- **password**: Enter the password for your amundsen user in the Password field.
-- **hostPort**: Host and port of the Amundsen Neo4j Connection. This expect a URI format like: bolt://localhost:7687.
-- **maxConnectionLifeTime (optional)**: Maximum connection lifetime for the Amundsen Neo4j Connection
-- **validateSSL (optional)**: Enable SSL validation for the Amundsen Neo4j Connection.
-- **encrypted (Optional)**: Enable encryption for the Amundsen Neo4j Connection.
-
-
-
-
-
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/metadata/atlas.mdx b/v2.0.x-SNAPSHOT/connectors/metadata/atlas.mdx
index c39a07a8b..9bde3ef12 100644
--- a/v2.0.x-SNAPSHOT/connectors/metadata/atlas.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/metadata/atlas.mdx
@@ -3,11 +3,9 @@ title: Atlas Connector | OpenMetadata Metadata Integration
description: Connect Apache Atlas to OpenMetadata seamlessly. Complete setup guide, configuration steps, and best practices for Atlas metadata connector integration.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/metadata/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMetadata from '/snippets/connectors/metadata-agent/metadata.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+In this section, we provide guides and references to use the Atlas connector.
+Configure and schedule Atlas metadata workflow from the OpenMetadata UI:
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
+- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/metadata/atlas/troubleshooting)
+
## Requirements
+
+Before configuring the Atlas connector, make sure the following requirements are in place.
+
Every table ingested will have a tag name `AtlasMetadata.atlas_table`. You can find all tags under
Governance > Classification.
-## 1. Create Database & Messaging Services
-You need to create at least a Database Service before ingesting the metadata from Atlas. Make sure to note down the name, since
-we will use it to create Atlas Service.
+
+### Create Database and Messaging Services
+
+You need to create at least a Database Service before ingesting the metadata from Atlas. Make sure to note down the name, since OpenMetadata uses it to create the Atlas service.
For example, to create a Hive Service you can follow these steps:
-
-## 2. Atlas Metadata Ingestion
-Then, prepare the Atlas Service and configure the Ingestion:
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+
+
+
+
+
+
+## Metadata Ingestion
+
+To ingest metadata from Atlas, you need to create a service connection. The service connects Atlas with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Metadata Services**, then click the **Atlas** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Atlas services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Atlas. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: Host and port of the Atlas service.
-- **Username**: username to connect to the Atlas. This user should have privileges to read all the metadata in Atlas.
-- **Password**: password to connect to the Atlas.
-- **databaseServiceName**: source database of the data source. This is the service we created before: e.g., `local_hive`)
-- **messagingServiceName**: messaging service source of the data source.
+- **Username**: Username to connect to Atlas. This user should have privileges to read all the metadata in Atlas.
+- **Password**: Password to connect to Atlas.
+- **Database Service Name**: Source database of the data source. This is the service we created before, for example, `local_hive`.
+- **Messaging Service Name**: Messaging service source of the data source.
- **Entity Type**: Name of the entity type in Atlas.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your source. Filter patterns use regular expressions applied to asset names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave all filter patterns empty to ingest all databases, schemas, and tables available in the source.
+
+
+**Filter Options**
+
+The Database, Schema, and Table sections each include the following filter options:
+
+- **Database**: Controls which databases OpenMetadata ingests from the source.
+- **Schema**: Controls which schemas within the ingested databases are included.
+- **Table**: Controls which tables and views within the ingested schemas are included.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/ml-model/mlflow.mdx b/v2.0.x-SNAPSHOT/connectors/ml-model/mlflow.mdx
index edfa5cc22..d05d709f6 100644
--- a/v2.0.x-SNAPSHOT/connectors/ml-model/mlflow.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ml-model/mlflow.mdx
@@ -3,11 +3,9 @@ title: MLflow Connector | OpenMetadata ML Model Integration
description: Connect MLflow to OpenMetadata seamlessly with our comprehensive connector guide. Learn setup, configuration, and ML model metadata integration in minutes.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/ml-model/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMlModel from '/snippets/connectors/metadata-agent/ml-model.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
+
+To ingest metadata from MLflow, you need to create a service connection. The service connects MLflow with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **ML Model Services**, then click the **MLflow** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other MLflow services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for MLflow. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Tracking URI**: MLflow experiment tracking URI, for example `http://localhost:5000`.
+- **Registry URI**: MLflow model registry backend, for example `mysql+pymysql://mlflow:password@localhost:3307/experiments`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your ML model service. Filter patterns use regular expressions applied to model names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all ML models available in the source.
-- **trackingUri**: Mlflow Experiment tracking URI. E.g., `http://localhost:5000`
-- **registryUri**: Mlflow Model registry backend. E.g., `mysql+pymysql://mlflow:password@localhost:3307/experiments`
-
-
-
-
-
+
+**Filter Options**
+
+The ML Model section includes the following filter options:
+
+- **ML Model**: Controls which ML models OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/ml-model/sagemaker.mdx b/v2.0.x-SNAPSHOT/connectors/ml-model/sagemaker.mdx
index 99b978a57..f96ce864b 100644
--- a/v2.0.x-SNAPSHOT/connectors/ml-model/sagemaker.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/ml-model/sagemaker.mdx
@@ -3,11 +3,9 @@ title: Sagemaker Connector | OpenMetadata ML Model Services
description: Connect your AWS SageMaker ML models to OpenMetadata for complete model lineage, metadata management, and data governance. Easy setup guide included.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/ml-model/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentMlModel from '/snippets/connectors/metadata-agent/ml-model.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
- permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
- authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
-Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
-You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
-- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
-Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
-services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
-- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
- and AWS Secrets Access Key. Also, these will include an AWS Session Token.
-You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
-- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
- entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
- default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
-Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
- When you specify a profile to run a command, the settings and credentials are used to run that command.
- Multiple named profiles can be stored in the config and credentials files.
-You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
-- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
- `ARN` (Amazon Resource Name) of the policy of the other account.
-A user who wants to access a role in a different account must also have permissions that are delegated from the account
-administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
-This is a required field if you'd like to `AssumeRole`.
-Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
+To ingest metadata from SageMaker, you need to create a service connection. The service connects SageMaker with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **ML Model Services**, then click the **SageMaker** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other SageMaker services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for SageMaker. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Authentication**
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). Access keys consist of two parts: an **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). You must use both the access key ID and secret access key together to authenticate your requests. You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+- **AWS Session Token** (optional): If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID and AWS Secret Access Key. These will also include an AWS Session Token. Find more information on [using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
+
+**Connection**
+- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). As AWS can have instances in multiple regions, OpenMetadata needs to know the region the service you want to reach belongs to. The AWS Region is the only required parameter when configuring a connection. Find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+- **Endpoint URL** (optional): To connect programmatically to an AWS service, you use an endpoint. An endpoint is the URL of the entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region, but you can specify an alternate endpoint for your API requests. Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. Multiple named profiles can be stored in the config and credentials files. Provide this field if you'd like to use a profile other than `default`. Find more information about [named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+
+**Advanced Config** *(assume-role options; most connections don't need these)*
+- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field, set the `ARN` (Amazon Resource Name) of the policy of the other account. A user who wants to access a role in a different account must also have permissions that are delegated from the account administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account. This is a required field if you'd like to `AssumeRole`. Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
+
+ When using Assume Role authentication, ensure you provide the following details:
+ - **AWS Region**: Specify the AWS region for your deployment.
+ - **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
+
+
+- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. By default, OpenMetadata uses the name `OpenMetadataSession`. Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role. Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your ML model service. Filter patterns use regular expressions applied to model names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using Assume Role authentication, ensure you provide the following details:
-- **AWS Region**: Specify the AWS region for your deployment.
-- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
+Leave the filter pattern empty to ingest all ML models available in the source.
-- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
- is assumed by different principals or for different reasons.
-By default, we'll use the name `OpenMetadataSession`.
-Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
-- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
- information in AWS CloudTrail logs to determine who took actions with a role.
-Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
+
+**Filter Options**
+
+The ML Model section includes the following filter options:
+
+- **ML Model**: Controls which ML models OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/airbyte.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/airbyte.mdx
index e93b95d8b..1c5697f83 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/airbyte.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/airbyte.mdx
@@ -3,11 +3,9 @@ title: Airbyte Connector | OpenMetadata Pipeline Integration
description: Connect Airbyte data pipelines to OpenMetadata for comprehensive data lineage tracking, metadata discovery, and pipeline monitoring. Setup guide included.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Airbyte connector.
-Configure and schedule Airbyte metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Airbyte metadata workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/airbyte/troubleshooting)
+
+## Requirements
+
+To extract metadata, you will need access to the Airbyte instance with valid credentials (username and password).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Host and Port**: Pipeline Service Management UI URL
+
+To ingest metadata from Airbyte, you need to create a service connection. The service connects Airbyte with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Airbyte** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Airbyte services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Airbyte. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
+- **Host and Port**: Pipeline Service Management UI URL.
+- **API Version**: Version of the Airbyte REST API, by default `api/v1`.
+
+**Authentication**
- **Username**: Username to connect to Airbyte.
- **Password**: Password to connect to Airbyte.
-- **API Version**: Version of the Airbyte REST API by default `api/v1`.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Airbyte service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Airbyte.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/airflow.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/airflow.mdx
index 26983df94..a12878de9 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/airflow.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/airflow.mdx
@@ -4,10 +4,8 @@ description: Connect Apache Airflow to OpenMetadata effortlessly. Complete setup
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Airflow connector.
+
Configure and schedule Airflow metadata workflow from the OpenMetadata UI:
- [REST API Connection](/v2.0.x-SNAPSHOT/connectors/pipeline/airflow/rest-api-connection)
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/airflow/troubleshooting)
-
+
Run the ingestion framework externally!
@@ -28,30 +29,64 @@ Run the ingestion framework externally!
Run the ingestion from GCP Composer.
+
## Requirements
+
+Before configuring the Airflow connector, make sure the following requirements are in place.
+
We only support officially supported Airflow versions.
You can check the version list [here](https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html).
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Airflow, you need to create a service connection. The service connects Airflow with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Airflow** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Airflow services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Airflow. A help panel displays inline guidance for each field.
+
+
+
- **Host and Port**: URL to the Airflow instance.
-- **Number of Status**: Number of status we want to look back to in every ingestion (e.g., Past executions from a DAG).
+- **Number of Status**: Number of status we want to look back to in every ingestion (for example, Past executions from a DAG).
- **Connection**: Airflow metadata database connection. See these [docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html)
for supported backends.
+
In terms of `connection` we support the following selections:
-- `RestAPI`: Connects to Airflow via the web server REST API. No direct database access required — ideal for managed
+- `RestAPI`: Connects to Airflow via the web server REST API. No direct database access required, ideal for managed
deployments (Astronomer, Cloud Composer, MWAA) or any setup where database access is unavailable. Supports
Basic Auth, Access Token, GCP Service Account, and MWAA authentication methods.
See the [REST API Connection guide](/v2.0.x-SNAPSHOT/connectors/pipeline/airflow/rest-api-connection) for full setup instructions.
@@ -59,8 +94,59 @@ In terms of `connection` we support the following selections:
by running the ingestion from a DAG. It will use the current Airflow SQLAlchemy connection to extract the data.
- `MySQL`, `Postgres`, and `SQLite`: Pass the required credentials to reach out each of these services. We
will create a connection to the pointed database and read Airflow data from there.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Airflow service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines (DAGs) OpenMetadata ingests from Airflow.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/dagster.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/dagster.mdx
index ba2ba6dd9..226669e9b 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/dagster.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/dagster.mdx
@@ -3,11 +3,9 @@ title: Dagster Connector | OpenMetadata Pipeline Integration
description: Connect Dagster pipelines to OpenMetadata with our comprehensive integration guide. Learn setup, configuration, and metadata extraction in minutes.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Dagster connector.
-Configure and schedule Dagster metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Dagster metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- - [Dagster Versions](#dagster-versions)
- [Metadata Ingestion](#metadata-ingestion)
- - [Service Name](#metadata-ingestion)
- - [Connection Details](#connection-details)
- - [Metadata Ingestion Options](#metadata-ingestion-options)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/dagster/troubleshooting)
- - [Workflow Deployment Error](/v2.0.x-SNAPSHOT/connectors/pipeline/dagster/troubleshooting#workflow-deployment-error)
+
## Requirements
+
+Before configuring the Dagster connector, make sure the following requirements are in place.
+
### Dagster Versions
-OpenMetadata is integrated with dagster up to version [1.0.13](https://docs.dagster.io/getting-started) and will continue to work for future dagster versions.
-The ingestion framework uses [dagster graphql python client](https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) to connect to the dagster instance and perform the API calls
+
+OpenMetadata is integrated with Dagster up to version [1.0.13](https://docs.dagster.io/getting-started) and will continue to work for future Dagster versions.
+The ingestion framework uses the [Dagster GraphQL Python client](https://docs.dagster.io/_apidocs/libraries/dagster-graphql#dagster_graphql.DagsterGraphQLClient) to connect to the Dagster instance and perform the API calls.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Dagster, you need to create a service connection. The service connects Dagster with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Dagster** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Dagster services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Dagster. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host**: Host of the Dagster instance, for example `https://localhost:3000` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod`.
+- **Token**: Pass a token if connecting to a `dagster cloud` instance:
+ 1. Log in to your Dagster account.
+ 2. Click **Settings** in the top navigation bar.
+ 3. Click the **API Keys** tab.
+ 4. Click **Create a New API Key**.
+ 5. Give your API key a name and click **Create API Key**.
+ 6. Copy the generated API key to your clipboard and paste it in the field.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Dagster service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all pipelines available in the source.
-- **Host**: Host of the dagster eg.`https://localhost:300` or `https://127.0.0.1:3000` or `https://.dagster.cloud/prod`
-- **Token** : Need pass token if connecting to `dagster cloud` instance
- - Log in to your Dagster account.
- - Click on the "Settings" link in the top navigation bar.
- - Click on the "API Keys" tab.
- - Click on the "Create a New API Key" button.
- - Give your API key a name and click on the "Create API Key" button.
- - Copy the generated API key to your clipboard and paste it in the field.
-
-
-
-
-
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Dagster.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/databricks-pipeline.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/databricks-pipeline.mdx
index ee9597cf7..535d10996 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/databricks-pipeline.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/databricks-pipeline.mdx
@@ -3,11 +3,9 @@ title: Databricks Pipeline | OpenMetadata Integration Guide
description: Configure metadata ingestion from Databricks pipelines for job flows, dependencies, and structured datasets.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Databricks Pipeline connector.
Configure and schedule Databricks Pipeline metadata workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/databricks-pipeline/troubleshooting)
+
+## Requirements
+
+To extract metadata, you will need access to the Databricks Pipeline instance with valid credentials.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from Databricks Pipeline, you need to create a service connection. The service connects Databricks Pipeline with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Databricks Pipeline** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Databricks Pipeline services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Databricks Pipeline. The right-hand panel in the UI displays inline help for each field.
+
+
+
- **Host and Port**: Enter the fully qualified hostname and port number for your Databricks Pipeline deployment in the Host and Port field.
-- **Token**: Generated Token to connect to Databricks Pipeline.
+- **Token**: Generated token to connect to Databricks Pipeline.
- **HTTP Path**: Databricks Pipeline compute resources URL.
-- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Databricks during the connection. These details must be added as Key-Value pairs.
+
+**Advanced Config** *(driver-level options; most connections don't need these)*
+- **Connection Arguments** *(Optional)*: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Databricks during the connection. These details must be added as Key-Value pairs.
- In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Databricks Pipeline service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Databricks Pipeline.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/dbtcloud.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/dbtcloud.mdx
index 01e93d822..2ffaeb4b2 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/dbtcloud.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/dbtcloud.mdx
@@ -3,11 +3,9 @@ title: dbt Cloud Connector | OpenMetadata Pipeline Integration
description: Connect dbt Cloud to OpenMetadata with our comprehensive pipeline connector guide. Setup instructions, configuration examples, and troubleshooting tips.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the dbt Cloud connector.
-Configure and schedule dbt Cloud metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule dbt Cloud metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- - [dbt Cloud Versions](#dbt-cloud-versions)
- [Metadata Ingestion](#metadata-ingestion)
- - [Service Name](#metadata-ingestion)
- - [Connection Details](#connection-details)
- - [Metadata Ingestion Options](#metadata-ingestion-options)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/dbtcloud/troubleshooting)
- - [Workflow Deployment Error](/v2.0.x-SNAPSHOT/connectors/pipeline/dbtcloud/troubleshooting#workflow-deployment-error)
+
## Requirements
+
+Before configuring the dbt Cloud connector, make sure the following requirements are in place.
+
### dbt Cloud Versions
+
OpenMetadata is integrated with dbt Cloud up to version [1.8](https://docs.getdbt.com/docs/get-started-dbt) and will continue to work for future dbt Cloud versions.
-The Ingestion framework uses [dbt Cloud APIs](https://docs.getdbt.com/dbt-cloud/api-v2#/) to connect to dbt Cloud and fetch metadata.
+The ingestion framework uses [dbt Cloud APIs](https://docs.getdbt.com/dbt-cloud/api-v2#/) to connect to dbt Cloud and fetch metadata.
+
### dbt Cloud Permissions
+
The dbt Cloud API User token or Service account token must have the permission to fetch metadata.
To know more about permissions required refer [here](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens#permissions-for-service-account-tokens).
+
### dbt Cloud Account
+
- dbt Cloud [multi-tenant](https://docs.getdbt.com/docs/cloud/about-cloud/tenancy#multi-tenant) or [single tenant](https://docs.getdbt.com/docs/cloud/about-cloud/tenancy#single-tenant) account is required.
- You must be on a [Team or Enterprise plan](https://www.getdbt.com/pricing/).
- Your projects must be on dbt version 1.0 or later. Refer to [Upgrade dbt version in Cloud](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud) to upgrade.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from dbt Cloud, you need to create a service connection. The service connects dbt Cloud with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **dbt Cloud** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other dbt Cloud services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for dbt Cloud. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host**: dbt Cloud Access URL, for example `https://abc12.us1.dbt.com`. Go to your dbt Cloud account settings to find your Access URL.
+- **Discovery API URL**: dbt Cloud Discovery API URL, for example `https://metadata.cloud.getdbt.com/graphql`. Go to your dbt Cloud account settings to find your Discovery API URL. Make sure you have `/graphql` at the end of your URL.
+- **Account Id**: The Account ID of your dbt Cloud Project. Go to your dbt Cloud account settings to find your Account Id. This is a numeric value, but OpenMetadata parses it as a string.
+- **Job Ids** *(Optional)*: Job IDs of your dbt Cloud Jobs in your Project to fetch metadata for. Look for the segment after "jobs" in the URL. For instance, in a URL like `https://cloud.getdbt.com/accounts/123/projects/87477/jobs/73659994`, the job ID is `73659994`. This is a numeric value, but OpenMetadata parses it as a string. If not passed, all Jobs under the Account Id will be ingested.
+- **Project Ids** *(Optional)*: Project IDs of your dbt Cloud Account to fetch metadata for. Look for the segment after "projects" in the URL. For instance, in a URL like `https://cloud.getdbt.com/accounts/123/projects/87477/jobs/73659994`, the project ID is `87477`. This is a numeric value, but OpenMetadata parses it as a string. If not passed, all Projects under the Account Id will be ingested.
+
+ Note that if both `Job Ids` and `Project Ids` are passed, jobs from the passed projects are filtered. Any `Job Ids` not belonging to the `Project Ids` are also filtered out.
+- **Token**: The Authentication Token of your dbt Cloud API Account. To get your access token, follow the docs [here](https://docs.getdbt.com/docs/dbt-cloud-apis/authentication). Make sure the token has the necessary permissions to run GraphQL queries and get job and run details.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your dbt Cloud service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all pipelines available in the source.
-- **Host**: dbt Cloud Access URL eg.`https://abc12.us1.dbt.com`. Go to your dbt Cloud account settings to know your Access URL.
-- **Discovery API URL** : dbt Cloud Access URL eg. `https://metadata.cloud.getdbt.com/graphql`. Go to your dbt Cloud account settings to know your Discovery API url. Make sure you have `/graphql` at the end of your URL.
-- **Account Id** : The Account ID of your dbt Cloud Project. Go to your dbt Cloud account settings to know your Account Id. This will be a numeric value but in openmetadata we parse it as a string.
-- **Job Ids** : Optional. Job IDs of your dbt Cloud Jobs in your Project to fetch metadata for. Look for the segment after "jobs" in the URL. For instance, in a URL like `https://cloud.getdbt.com/accounts/123/projects/87477/jobs/73659994`, the job ID is `73659994`. This will be a numeric value but in openmetadata we parse it as a string. If not passed all Jobs under the Account id will be ingested.
-- **Project Ids** : Optional. Project IDs of your dbt Cloud Account to fetch metadata for. Look for the segment after "projects" in the URL. For instance, in a URL like `https://cloud.getdbt.com/accounts/123/projects/87477/jobs/73659994`, the job ID is `87477`. This will be a numeric value but in openmetadata we parse it as a string. If not passed all Projects under the Account id will be ingested.
-Note that if both `Job Ids` and `Project Ids` are passed then it will filter out the jobs from the passed projects. any `Job Ids` not belonging to the `Project Ids` will also be filtered out.
-- **Token** : The Authentication Token of your dbt Cloud API Account. To get your access token you can follow the docs [here](https://docs.getdbt.com/docs/dbt-cloud-apis/authentication).
-Make sure you have the necessary permissions on the token to run graphql queries and get job and run details.
-
-
-
-
-
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from dbt Cloud.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Displaying Lineage Information
+
Steps to retrieve and display the lineage information for a dbt Cloud service. Note that only the metadata from the last run will be used for lineage.
-1. Ingest Source and Sink Database Metadata: Identify both the source and sink database used by the dbt Cloud service for example Redshift. Ingest metadata for these database.
-2. Ingest dbt Cloud Service Metadata: Finally, Ingest your dbt Cloud service.
+
+1. **Ingest Source and Sink Database Metadata**: Identify both the source and sink database used by the dbt Cloud service, for example Redshift. Ingest metadata for these databases.
+2. **Ingest dbt Cloud Service Metadata**: Finally, ingest your dbt Cloud service.
+
By successfully completing these steps, the lineage information for the service will be displayed.
+
+
### Missing Lineage
+
If lineage information is not displayed for a dbt Cloud service, follow these steps to diagnose the issue.
-1. *dbt Cloud Account*: Make sure that the dbt Cloud instance you are ingesting have the necessary permissions to fetch jobs and run graphql queries over the API.
-2. *Metadata Ingestion*: Ensure that metadata for both the source and sink database is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
-3. *Last Run Successful*: Ensure that the Last Run for a Job is successful as OpenMetadata gets the metadata required to build the lineage using the last Run under a Job.
+
+1. **dbt Cloud Account**: Make sure that the dbt Cloud instance you are ingesting has the necessary permissions to fetch jobs and run GraphQL queries over the API.
+2. **Metadata Ingestion**: Ensure that metadata for both the source and sink database is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
+3. **Last Run Successful**: Ensure that the Last Run for a Job is successful, as OpenMetadata gets the metadata required to build the lineage using the last Run under a Job.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/domo-pipeline.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/domo-pipeline.mdx
index a145474de..654ca8225 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/domo-pipeline.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/domo-pipeline.mdx
@@ -3,11 +3,9 @@ title: Domo Pipeline | OpenMetadata Data Pipeline Services
description: Ingest metadata from Domo pipelines to capture ETL flows and integrate with governance and reporting tools.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-In this section, we provide guides and references to use the Domo-Pipeline connector.
-Configure and schedule Domo-Pipeline metadata and profiler workflows from the OpenMetadata UI:
+In this section, we provide guides and references to use the Domo Pipeline connector.
+Configure and schedule Domo Pipeline metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/domo-pipeline/troubleshooting)
+
## Requirements
-For metadata ingestion, make sure to add at least `data` scopes to the clientId provided.
+
+For metadata ingestion, make sure to add at least `data` scopes to the Client Id provided.
For questions related to scopes, click [here](https://developer.domo.com/portal/1845fc11bbe5d-api-authentication).
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Domo Pipeline, you need to create a service connection. The service connects Domo Pipeline with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Domo Pipeline** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Domo Pipeline services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Domo Pipeline. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Client ID**: Client ID for Domo Pipeline.
+- **Secret Token**: Secret Token to connect to Domo Pipeline.
+- **Access Token**: Access Token to connect to Domo Pipeline.
+- **API Host**: API Host to connect to Domo Pipeline.
+- **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Domo Pipeline service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all pipelines available in the source.
-- **Client ID**: Client Id for DOMO Pipeline.
-- **Secret Token**: Secret Token to Connect to DOMO Pipeline.
-- **Access Token**: Access to Connect to DOMO Pipeline.
-- **API Host**: API Host to Connect to DOMO Pipeline.
-- **Instance Domain**: URL to connect to your Domo instance UI. For example `https://.domo.com`.
-
-
-
-
-
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Domo Pipeline.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/fivetran.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/fivetran.mdx
index 866998a03..6e1a09500 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/fivetran.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/fivetran.mdx
@@ -3,11 +3,9 @@ title: Fivetran Connector | OpenMetadata Data Integration
description: Connect Fivetran pipelines to OpenMetadata for automated data lineage tracking, metadata discovery, and comprehensive pipeline monitoring and governance.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Fivetran connector.
-Configure and schedule Fivetran metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Fivetran metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/fivetran/troubleshooting)
+
## Requirements
+
To access Fivetran APIs, a Fivetran account on a Standard, Enterprise, or Business Critical plan is required.
### Permissions
+
Generate the API key and API secret from a Fivetran user with an account-level role that has read access to all destinations and connections, such as **Account Reviewer**. OpenMetadata needs to discover every destination and connector in the account. A destination-scoped role (for example, View Destination) is not enough, because it only covers destinations the user was explicitly invited to.
For detailed, per-phase Pipeline Status (Extract/Process/Load), OpenMetadata also reads the `fivetran_metadata.log` table that Fivetran automatically publishes in each destination's warehouse, using the Database Service already connected to that warehouse in OpenMetadata. The database user in that connection needs `SELECT` access to the `fivetran_metadata` schema. If this table isn't reachable, OpenMetadata automatically falls back to Fivetran's REST sync-history API for pipeline status.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Fivetran, you need to create a service connection. The service connects Fivetran with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Fivetran** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Fivetran services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Fivetran. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **API Key**: Fivetran API Key. Follow the steps below to generate the Fivetran API key and API secret:
+ 1. Click your user name in your Fivetran dashboard.
+ 2. Click **API Key**.
+ 3. Click **Generate API key**. (If you already have an API key, the button text is **Generate new API key**.)
+ 4. Make a note of the key and secret, as they won't be displayed once you close the page or navigate away.
+
+ For more detailed documentation, visit [here](https://fivetran.com/docs/rest-api/getting-started).
+- **API Secret**: Fivetran API Secret. Copy this from the same step where the API key is generated.
+- **Host and Port**: Host and port of the Fivetran instance. By default, OpenMetadata uses `https://api.fivetran.com` to connect to the Fivetran APIs.
+- **Limit**: Fivetran API limit for pagination. This is the maximum number of records returned in a single page of results when using Fivetran's API for pagination.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Fivetran service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all pipelines available in the source.
-**apiKey**: Fivetran API Key.
-Follow the steps mentioned below to generate the Fivetran API key and API secret:
-- Click your user name in your Fivetran dashboard.
-- Click API Key.
-- Click Generate API key. (If you already have an API key, then the button text is Generate new API key.)
-- Make a note of the key and secret as they won't be displayed once you close the page or navigate away.
-For more detailed documentation visit [here](https://fivetran.com/docs/rest-api/getting-started).
-- **apiSecret**: Fivetran API Secret. From the above step where the API key is generated copy the API secret
-- **hostPort**: HostPort of the Fivetran instance. By default, OpenMetadata will use `https://api.fivetran.com` to connect to the Fivetran APIs.
-- **limit**: Fivetran API Limit For Pagination. This refers to the maximum number of records that can be returned in a single page of results when using Fivetran's API for pagination.
-
-
-
-
-
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Fivetran.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/flink.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/flink.mdx
index cc516c7ef..1adacc945 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/flink.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/flink.mdx
@@ -3,11 +3,9 @@ title: Apache Flink Connector | OpenMetadata Streaming Guide
description: Connect Apache Flink pipelines to OpenMetadata with our comprehensive connector guide. Step-by-step setup, configuration, and metadata extraction instructions.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Apache Flink connector.
-Configure and schedule Flink metadata from the OpenMetadata UI:
+Configure and schedule Flink metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- - [Versions](#versions)
- [Metadata Ingestion](#metadata-ingestion)
- - [Service Name](#metadata-ingestion)
- - [Connection Details](#connection-details)
- - [Metadata Ingestion Options](#metadata-ingestion-options)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/flink/troubleshooting)
- - [Workflow Deployment Error](/v2.0.x-SNAPSHOT/connectors/pipeline/flink/troubleshooting#workflow-deployment-error)
+
## Requirements
+
+Before configuring the Flink connector, make sure the following requirements are in place.
+
### Versions
+
OpenMetadata is integrated with Flink up to version [1.19.0](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/gettingstarted/) and will continue to work for future Flink versions.
-The ingestion framework uses flink REST APIs to connect to the instance and perform the API calls
+The ingestion framework uses Flink REST APIs to connect to the instance and perform the API calls.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Flink, you need to create a service connection. The service connects Flink with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Flink** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Flink services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Flink. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **Host and Port**: The hostname or IP address of the Flink Connect worker with the REST API enabled, for example `http://localhost:8081` or `https://127.0.0.1:8081`.
+- **Flink Connect Config**: OpenMetadata supports SSL config:
+ - **caCertificate**: Authorized certificate for the SSL-configured server.
+ - **sslCertificate**: SSL certificate for the server.
+ - **sslKey**: Server root key for the connection.
+- **verifySSL**: Whether SSL verification should be performed when authenticating.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Flink service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all pipelines available in the source.
-- **Host and Port**: The hostname or IP address of the flink Connect worker with the REST API enabled eg.`http://localhost:8081` or `https://127.0.0.1:8081`.
-- **Flink Connect Config**: OpenMetadata supports SSL config.
- 1. SSL config
- - caCertificate: Authorized certificate for ssl configured server.
- - sslCertificate: SSL certificate for the server.
- - sslKey: Server root key for the connection.
-- **verifySSL** : Whether SSL verification should be perform when authenticating.
-
-
-
-
-
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Flink.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/glue-pipeline.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/glue-pipeline.mdx
index 9ad0a015a..ec7b11ef5 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/glue-pipeline.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/glue-pipeline.mdx
@@ -3,11 +3,9 @@ title: Glue Pipeline | OpenMetadata Data Integration Pipeline
description: Set up metadata ingestion from Glue pipelines to track source-to-target flows, logic, and job parameters.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the Glue connector.
-Configure and schedule Glue metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule Glue metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/glue-pipeline/troubleshooting)
+
## Requirements
+
The Glue connector ingests metadata through AWS [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html) Client.
-We will ingest Workflows, its jobs and their run status.
+OpenMetadata ingests workflows, their jobs, and run status.
The user must have the following permissions for the ingestion to run successfully:
- `glue:ListWorkflows`
- `glue:GetWorkflow`
- `glue:GetJobRuns`
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
+
+To ingest metadata from Glue Pipeline, you need to create a service connection. The service connects Glue Pipeline with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Glue Pipeline** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Glue Pipeline services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Glue Pipeline. A help panel displays inline guidance for each field.
+
+
+
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
@@ -58,11 +87,11 @@ You can find more information on [Using temporary credentials with AWS resources
entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
@@ -81,8 +110,59 @@ Find more information about the [Role Session Name](https://docs.aws.amazon.com/
- **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Glue Pipeline service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which workflows OpenMetadata ingests from Glue.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect.mdx
index b2c7c49e6..2d684301c 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect.mdx
@@ -3,11 +3,9 @@ title: KafkaConnect | OpenMetadata Messaging Pipeline Connector
description: Configure Kafka Connect for metadata ingestion from real-time event streams, schema updates, and topic usage.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
In this section, we provide guides and references to use the KafkaConnect connector.
-Configure and schedule KafkaConnect metadata and profiler workflows from the OpenMetadata UI:
+Configure and schedule KafkaConnect metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- - [KafkaConnect Versions](#kafkaconnect-versions)
- [Metadata Ingestion](#metadata-ingestion)
- - [Service Name](#metadata-ingestion)
- - [Connection Details](#connection-details)
- - [Metadata Ingestion Options](#metadata-ingestion-options)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect/troubleshooting)
- - [Workflow Deployment Error](/v2.0.x-SNAPSHOT/connectors/pipeline/kafkaconnect/troubleshooting#workflow-deployment-error)
+
## Requirements
+
+Before configuring the KafkaConnect connector, make sure the following requirements are in place.
+
### KafkaConnect Versions
+
OpenMetadata is integrated with Kafka Connect up to version [3.6.1](https://kafka.apache.org/36/kafka-connect/) and will continue to work with future Kafka Connect versions.
-The ingestion framework uses [kafkaconnect python client](https://libraries.io/pypi/kafka-connect-py) to connect to the kafkaconnect instance and perform the API calls
+The ingestion framework uses the [kafkaconnect python client](https://libraries.io/pypi/kafka-connect-py) to connect to the Kafka Connect instance and perform the API calls.
+
## Metadata Ingestion
-
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Host and Port**: The hostname or IP address of the Kafka Connect worker with the REST API enabled eg.`https://localhost:8083` or `https://127.0.0.1:8083` or `https://`
+
+To ingest metadata from KafkaConnect, you need to create a service connection. The service connects KafkaConnect with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **KafkaConnect** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other KafkaConnect services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for KafkaConnect. A help panel displays inline guidance for each field.
+
+
+
+- **Host and Port**: The hostname or IP address of the Kafka Connect worker with the REST API enabled, for example, `https://localhost:8083` or `https://127.0.0.1:8083` or `https://`.
- **Kafka Connect Config**: OpenMetadata supports username/password.
1. Basic Authentication
- - Username: Username to connect to Kafka Connect. This user should be able to send request to the Kafka Connect API and access the [Rest API](https://docs.confluent.io/platform/current/connect/references/restapi.html) GET endpoints.
+ - Username: Username to connect to Kafka Connect. This user should be able to send requests to the Kafka Connect API and access the [Rest API](https://docs.confluent.io/platform/current/connect/references/restapi.html) GET endpoints.
- Password: Password to connect to Kafka Connect.
-- **verifySSL** : Whether SSL verification should be perform when authenticating.
-- **Kafka Service Name** : The Service Name of the ingested [Kafka](/v2.0.x-SNAPSHOT/connectors/messaging/kafka#metadata-ingestion) instance associated with this KafkaConnect instance.
-
-
-
-
-
+- **verifySSL**: Whether SSL verification should be performed when authenticating.
+- **Kafka Service Name**: The Service Name of the ingested [Kafka](/v2.0.x-SNAPSHOT/connectors/messaging/kafka#metadata-ingestion) instance associated with this KafkaConnect instance.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your KafkaConnect service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines (connectors) OpenMetadata ingests from Kafka Connect.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
## Debezium CDC Support
The KafkaConnect connector provides **full support for Debezium CDC connectors** with intelligent column extraction and accurate lineage tracking.
@@ -62,29 +139,39 @@ The KafkaConnect connector provides **full support for Debezium CDC connectors**
When you ingest Debezium connectors, OpenMetadata automatically:
-1. **Detects CDC Envelope Structures** - Identifies Debezium's CDC format with `op`, `before`, and `after` fields
-2. **Extracts Real Table Columns** - Parses actual database columns from the CDC payload instead of CDC envelope metadata
-3. **Creates Accurate Column-Level Lineage** - Maps lineage from source database tables → Kafka topics → target systems
+1. **Detects CDC Envelope Structures**: Identifies Debezium's CDC format with `op`, `before`, and `after` fields.
+2. **Extracts Real Table Columns**: Parses actual database columns from the CDC payload instead of CDC envelope metadata.
+3. **Creates Accurate Column-Level Lineage**: Maps lineage from source database tables to Kafka topics to target systems.
### Recognized Configuration Parameters
OpenMetadata recognizes the following Debezium configuration parameters for intelligent CDC detection:
-- `database.server.name` - Server identifier (Debezium V1)
-- `topic.prefix` - Topic prefix (Debezium V2)
-- `table.include.list` - Tables to capture (e.g., `mydb.customers,mydb.orders`)
+- `database.server.name`: Server identifier (Debezium V1).
+- `topic.prefix`: Topic prefix (Debezium V2).
+- `table.include.list`: Tables to capture (for example, `mydb.customers,mydb.orders`).
+
+
## Supported Connectors
+
Currently, the following source and sink connectors for Kafka Connect are supported for lineage tracking:
* [MySQL](/v2.0.x-SNAPSHOT/connectors/database/mysql)
* [PostgreSQL](/v2.0.x-SNAPSHOT/connectors/database/postgres)
* [MSSQL](/v2.0.x-SNAPSHOT/connectors/database/mssql)
* [MongoDB](/v2.0.x-SNAPSHOT/connectors/database/mongodb)
* [Amazon S3](/v2.0.x-SNAPSHOT/connectors/storage/s3)
+
For these connectors, lineage information can be obtained provided they are configured with a source or sink and the corresponding metadata ingestion is enabled.
-**Note:** All supported database connectors listed above work seamlessly with **Debezium CDC connectors** for enhanced column-level lineage tracking. When using Debezium, OpenMetadata automatically detects the CDC envelope structure and extracts actual table columns for accurate lineage mapping.
+
+**Note**: All supported database connectors listed above work seamlessly with **Debezium CDC connectors** for enhanced column-level lineage tracking. When using Debezium, OpenMetadata automatically detects the CDC envelope structure and extracts actual table columns for accurate lineage mapping.
+
### Missing Lineage
+
If lineage information is not displayed for a Kafka Connect service, follow these steps to diagnose the issue.
-1. *Kafka Service Association*: Make sure the Kafka service that the data is being ingested from is associated with this Kafka Connect service. Additionally, verify that the correct name is passed on in the Kafka Service Name field during configuration. This field helps establish the lineage between the Kafka service and the Kafka Connect flow.
-2. *Source and Sink Configuration*: Verify that the Kafka Connect connector associated with the service is configured with a source and/or sink database or storage system. Connectors without a defined source or sink cannot provide lineage data.
-3. *Metadata Ingestion*: Ensure that metadata for both the source and sink database/storage systems is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
+
+1. **Kafka Service Association**: Make sure the Kafka service that the data is being ingested from is associated with this Kafka Connect service. Additionally, verify that the correct name is passed on in the Kafka Service Name field during configuration. This field helps establish the lineage between the Kafka service and the Kafka Connect flow.
+2. **Source and Sink Configuration**: Verify that the Kafka Connect connector associated with the service is configured with a source and/or sink database or storage system. Connectors without a defined source or sink cannot provide lineage data.
+3. **Metadata Ingestion**: Ensure that metadata for both the source and sink database/storage systems is ingested and passed to the lineage system. This typically involves configuring the relevant connectors to capture and transmit this information.
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/nifi.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/nifi.mdx
index 6eb1519a9..6f34095d8 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/nifi.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/nifi.mdx
@@ -3,11 +3,9 @@ title: Nifi Connector | OpenMetadata Data Pipeline Integration
description: Connect Apache NiFi to OpenMetadata seamlessly. Complete setup guide for NiFi pipeline connector with step-by-step configuration and metadata extraction.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **Host and Port**: Pipeline Service Management/UI URI. This should be specified as a string in the format 'hostname:port'.
+
+To ingest metadata from NiFi, you need to create a service connection. The service connects NiFi with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **NiFi** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other NiFi services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for NiFi. A help panel displays inline guidance for each field.
+
+
+
+- **Host and Port**: Pipeline Service Management/UI URI. This should be specified as a string in the format `hostname:port`.
- **NiFi Config**: OpenMetadata supports username/password or client certificate authentication.
- 1. Basic Authentication
- - Username: Username to connect to NiFi. This user should be able to send request to the Nifi API and access the `Resources` endpoint.
+ - **Basic Authentication**
+ - Username: Username to connect to NiFi. This user should be able to send requests to the NiFi API and access the `Resources` endpoint.
- Password: Password to connect to NiFi.
- - Verify SSL: Whether SSL verification should be perform when authenticating.
- 2. Client Certificate Authentication
- - Certificate Authority Path: Path to the certificate authority (CA) file. This is the certificate used to store and issue your digital certificate. This is an optional parameter. If omitted SSL verification will be skipped; this can present some sever security issue.
- **important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using OpenMetadata Ingestion Docker container, this file should be in this container.
+ - Verify SSL: Whether SSL verification should be performed when authenticating.
+ - **Client Certificate Authentication**
+ - Certificate Authority Path: Path to the certificate authority (CA) file. This is the certificate used to store and issue your digital certificate. This is an optional parameter. If omitted, SSL verification will be skipped; this can present a severe security issue.
+ **Important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using the OpenMetadata Ingestion Docker container, this file should be in this container.
- Client Certificate Path: Path to the certificate client file.
- **important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using OpenMetadata Ingestion Docker container, this file should be in this container.
+ **Important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using the OpenMetadata Ingestion Docker container, this file should be in this container.
- Client Key Path: Path to the client key file.
- **important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using OpenMetadata Ingestion Docker container, this file should be in this container.
-
-
-
-
-
+ **Important**: This file should be accessible from where the ingestion workflow is running. For example, if you are using the OpenMetadata Ingestion Docker container, this file should be in this container.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your NiFi service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines (process groups) OpenMetadata ingests from NiFi.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/openlineage.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/openlineage.mdx
index 846c17c33..ce991b2b4 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/openlineage.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/openlineage.mdx
@@ -1,13 +1,11 @@
---
title: OpenLineage | OpenMetadata Data Lineage Pipeline
-description: Connect your data pipelines with OpenMetadata'sOpenLineage connector. Track data lineage, monitor pipeline metadata, and gain end-to-end visibility.
+description: Connect your data pipelines with OpenMetadata's OpenLineage connector. Track data lineage, monitor pipeline metadata, and gain end-to-end visibility.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-## Connection Details
-
-
-
-
-
+
+To ingest metadata from OpenLineage, you need to create a service connection. The service connects OpenLineage with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **OpenLineage** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other OpenLineage services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for OpenLineage. A help panel displays inline guidance for each field.
+
+
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your OpenLineage service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all pipelines available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes one filter option, **Pipeline**, which controls which pipelines OpenMetadata ingests from OpenLineage.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose one of the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
### Providing Connection Details Programmatically via API
+You can also create and configure the OpenLineage service using the Python SDK instead of the UI.
+
#### 1. Preparing the Client
+
+Configure the SDK client with your OpenMetadata host and JWT token.
+
```python
from metadata.sdk import configure
@@ -98,6 +196,9 @@ configure(host="http://localhost:8585/api", jwt_token="")
```
#### 2. Creating the OpenLineage Pipeline Service (Kafka)
+
+Create the pipeline service pointing at your Kafka broker.
+
```python
from metadata.sdk import client
from metadata.generated.schema.api.services.createPipelineService import CreatePipelineServiceRequest
@@ -143,6 +244,9 @@ client().ometa.create_or_update(openlineage_service_request)
```
#### 3. Creating the OpenLineage Pipeline Service (Kinesis)
+
+Create the pipeline service pointing at your Kinesis stream instead.
+
```python
from metadata.sdk import client
from metadata.generated.schema.api.services.createPipelineService import CreatePipelineServiceRequest
@@ -180,3 +284,5 @@ openlineage_service_request = CreatePipelineServiceRequest(
)
client().ometa.create_or_update(openlineage_service_request)
```
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/prefect.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/prefect.mdx
new file mode 100644
index 000000000..8ee17ec06
--- /dev/null
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/prefect.mdx
@@ -0,0 +1,156 @@
+---
+title: Prefect Connector | OpenMetadata Pipeline Guide
+description: Connect Prefect to OpenMetadata for comprehensive data lineage tracking, metadata discovery, and pipeline monitoring. Setup guide included.
+sidebarTitle: Overview
+---
+import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
+
+
+In this section, we provide guides and references to use the Prefect connector.
+
+The connector ingests **flows** as Pipelines, each flow's **tasks** (from its most recent run) as Tasks, and **flow run history** as Pipeline Status. It connects to either Prefect Cloud or a self-hosted Prefect Server over the Prefect REST API.
+
+Configure and schedule Prefect metadata workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
+- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/troubleshooting)
+
+## Requirements
+
+The Prefect connector reads flows, deployments, and run history through the Prefect REST API. The connector requires no database-level grants, access is controlled entirely by the credentials used to call the API.
+
+### Prefect Cloud
+
+The API key must belong to a [service account or user](https://docs.prefect.io/v3/how-to-guides/cloud/manage-users/api-keys) with at least **Viewer** access to the workspace identified by the configured Account ID and Workspace ID. Viewer access is sufficient to list flows and read their deployments, flow runs, task runs, and asset materializations (for lineage).
+
+### Self-Hosted Prefect Server
+
+If the server has Basic Auth enabled via `PREFECT_SERVER_API_AUTH_STRING`, provide the same `user:password` value as the **Basic Auth String**. If the server has no auth enabled, leave the field empty and requests are sent without credentials.
+
+### Lineage
+
+Table-to-table lineage is built from two sources, tried in order:
+
+1. **Assets API** (Prefect Cloud only): exact upstream and downstream table pairs read from a flow run's asset materializations, when the flow uses `@materialize` or `asset_deps`.
+2. **Lineage tags** (Cloud and self-hosted): tag a flow, deployment, or task run with `om-source:..` and `om-destination:..`. Every detected source is linked to every detected destination. This requires the **Database Service Name(s)** to be configured in the pipeline service's lineage settings so the tagged tables can be resolved to existing OpenMetadata Table entities.
+
+## Metadata Ingestion
+
+To ingest metadata from Prefect, you need to create a service connection. The service connects Prefect with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Prefect** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Prefect services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Prefect. A help panel displays inline guidance for each field.
+
+
+
+- **Host and Port** (required): Prefect API base URL. Use `https://api.prefect.cloud` for Prefect Cloud, or your self-hosted server URL, for example `http://localhost:4200`.
+- **Authentication** (required): Choose the authentication type that matches your Prefect deployment:
+ - **Prefect Cloud**:
+ - **Prefect API Key**: API key for Prefect Cloud authentication.
+ - **Account ID**: Found in the URL: `app.prefect.cloud/account/{accountId}`.
+ - **Workspace ID**: Found in the URL after `/workspaces/{workspaceId}`.
+ - **Prefect Server**:
+ - **Basic Auth String** (optional): The `PREFECT_SERVER_API_AUTH_STRING` value in `user:password` format. Leave empty if the server has no auth enabled.
+- **Number of Status** (optional, default `10`, max `100`): Number of past flow run statuses to ingest per flow.
+- **Verify SSL** (optional): Client SSL verification mode for the Prefect API connection. Choose from:
+ - **No SSL** (default): Plain HTTP connection.
+ - **Ignore**: Accept HTTPS without validating the certificate (not recommended outside of testing).
+ - **Validate**: Validate the certificate using the CA cert provided in **SSL Configuration**.
+- **SSL Configuration** (optional): CA certificate (and, for mutual TLS, client certificate and key) used when **Verify SSL** is set to **Validate**.
+- **Default Pipeline Filter Pattern** (optional): Regex pattern to include or exclude flows by name. Supports **includes** and **excludes** arrays.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Prefect service. Filter patterns use regular expressions applied to flow names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all flows available in the source.
+
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which flows OpenMetadata ingests from Prefect.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips. An asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/troubleshooting.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/troubleshooting.mdx
new file mode 100644
index 000000000..716d13868
--- /dev/null
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/troubleshooting.mdx
@@ -0,0 +1,87 @@
+---
+title: Prefect Connector Troubleshooting Guide | OpenMetadata Support
+description: Resolve Prefect pipeline ingestion issues such as authentication failures, missing flows, rate limits, and lineage gaps.
+sidebarTitle: Troubleshooting
+---
+
+import Troubleshooting from '/snippets/connectors/troubleshooting.mdx'
+
+
+
+## Prefect-Specific Issues
+
+The following issues are specific to the Prefect connector's REST API integration.
+
+### Authentication Failed on Prefect Cloud
+
+**Issue**: The Test Connection step fails with a 401 (authentication failed) or 403 (access denied) error.
+
+**Cause**: The API key is invalid, expired, or does not belong to the configured Account ID and Workspace ID, or lacks Viewer access to the workspace.
+
+**Solution**: Verify the API key is valid and unexpired in your [Prefect Cloud account settings](https://docs.prefect.io/v3/how-to-guides/cloud/manage-users/api-keys). Confirm the Account ID and Workspace ID by checking the workspace URL: `app.prefect.cloud/account//workspace/`. If both are correct, confirm the API key belongs to that account and can read that workspace.
+
+### Endpoint Not Found (404) on Prefect Cloud
+
+**Issue**: The Test Connection step fails with a 404 error.
+
+**Cause**: Host and Port, Account ID, and Workspace ID together build the API path Prefect answered 404 for, so one of the three is wrong.
+
+**Solution**: Double-check the Account ID and Workspace ID against the workspace URL, and confirm Host and Port is `https://api.prefect.cloud`.
+
+### Authentication Failed on Self-Hosted Server
+
+**Issue**: The Test Connection step fails with a 401 or 403 error against a self-hosted Prefect Server.
+
+**Cause**: The Basic Auth String does not match the server's `PREFECT_SERVER_API_AUTH_STRING`, or the field was left empty while the server has auth enabled.
+
+**Solution**: Confirm the `user:password` value configured as the Basic Auth String matches the `PREFECT_SERVER_API_AUTH_STRING` environment variable on the server. If the server has no auth enabled, leave the field empty.
+
+### Endpoint Not Found (404) on a Self-Hosted Server
+
+**Issue**: The Test Connection step fails with a 404 error against a self-hosted server.
+
+**Cause**: Host and Port build the API path Prefect answered 404 for, this usually means Host and Port points at the Prefect UI URL instead of the API base URL.
+
+**Solution**: Check Host and Port for typos and confirm it points at the Prefect API base URL (for example `http://localhost:4200`), not the UI.
+
+### Rate Limited (429)
+
+**Issue**: The Test Connection step or a metadata ingestion run fails or is delayed with a 429 error.
+
+**Cause**: Prefect throttled the request.
+
+**Solution**: Retry after a few minutes. If this happens consistently, reduce ingestion frequency or the **Number of Status** value to reduce the number of API calls per run.
+
+### Connection Timed Out or Host Unreachable
+
+**Issue**: The Test Connection step fails with a timeout, or a "cannot reach the host" / connection error.
+
+**Cause**: A firewall or network ACL is blocking egress from where ingestion runs to Host and Port, or Host and Port has a typo and does not resolve.
+
+**Solution**: Confirm Host and Port is correct and reachable from wherever the ingestion agent runs, and that any firewall or network ACL allows egress to it.
+
+### TLS Verification Failed
+
+**Issue**: The Test Connection step fails with an SSL/TLS certificate verification error.
+
+**Cause**: The Prefect server's certificate could not be verified with the current SSL configuration, or a TLS-intercepting proxy sits between the ingestion agent and Prefect.
+
+**Solution**: Provide the CA certificate under **SSL Configuration** with **Verify SSL** set to `validate`, or set **Verify SSL** to `ignore` for a self-signed certificate (not recommended outside of testing). If a TLS-intercepting proxy is present, make sure its certificate is trusted where ingestion runs.
+
+### No Flows Ingested
+
+**Issue**: The connector runs successfully but no Pipeline entities are created in OpenMetadata.
+
+**Cause**: The configured workspace (Cloud) or server (self-hosted) has no flows, or the Pipeline Filter Pattern excludes all of them. Test Connection can succeed even when the workspace has zero flows, since it only proves that authentication and read access to the workspace succeeded.
+
+**Solution**: Confirm flows exist in the target workspace or server. Review the Pipeline Filter Pattern `includes` and `excludes` rules and ensure at least one flow name matches an include rule (or that the filter is left empty to include all).
+
+### No Lineage Between Tables
+
+**Issue**: Flows and their run history are ingested, but no table-to-table lineage edges appear in OpenMetadata.
+
+**Cause**: On Prefect Cloud, the flow's runs have no asset materializations (the flow doesn't use `@materialize` or `asset_deps`). On both Cloud and self-hosted, the fallback lineage tags are missing, malformed, or the **Database Service Name(s)** is not configured in the pipeline service's lineage settings.
+
+**Solution**:
+1. For asset-based lineage (Prefect Cloud only), confirm the flow uses Prefect's [Assets API](https://docs.prefect.io/v3/concepts/assets) with `@materialize` or `asset_deps`.
+2. For tag-based lineage, confirm the tag format is exactly `om-source:..` and `om-destination:..` (three dot-separated parts). Verify the tagged tables exist in OpenMetadata under the database service(s) configured in the pipeline service's lineage settings.
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/yaml.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/yaml.mdx
new file mode 100644
index 000000000..539591c73
--- /dev/null
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/prefect/yaml.mdx
@@ -0,0 +1,147 @@
+---
+title: Run the Prefect Connector Externally
+description: Use YAML to ingest metadata from Prefect Cloud or a self-hosted Prefect Server, including flows, tasks, run history, and lineage.
+sidebarTitle: Run Externally
+mode: "wide"
+---
+import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
+import { CodePreview, ContentPanel, ContentSection, CodePanel } from '/snippets/components/CodePreview.jsx'
+import SourceConfigDef from '/snippets/connectors/yaml/pipeline/source-config-def.mdx'
+import SourceConfig from '/snippets/connectors/yaml/pipeline/source-config.mdx'
+import IngestionSinkDef from '/snippets/connectors/yaml/ingestion-sink-def.mdx'
+import WorkflowConfigDef from '/snippets/connectors/yaml/workflow-config-def.mdx'
+import IngestionCli from '/snippets/connectors/yaml/ingestion-cli.mdx'
+import PythonRequirements from '/snippets/connectors/python-requirements.mdx'
+import ExternalIngestionDeployment from '/snippets/v2.0.x-SNAPSHOT/connectors/external-ingestion-deployment.mdx'
+import IngestionSink from '/snippets/connectors/yaml/ingestion-sink.mdx'
+import WorkflowConfig from '/snippets/connectors/yaml/workflow-config.mdx'
+
+
+In this section, we provide guides and references to use the Prefect connector.
+
+The connector ingests **flows** as Pipelines, each flow's **tasks** (from its most recent run) as Tasks, and **flow run history** as Pipeline Status. It connects to either Prefect Cloud or a self-hosted Prefect Server over the Prefect REST API.
+
+Configure and schedule Prefect metadata workflows from the OpenMetadata UI:
+- [Requirements](#requirements)
+- [Metadata Ingestion](#metadata-ingestion)
+
+## Requirements
+
+The Prefect connector reads flows, deployments, and run history through the Prefect REST API. It requires no database-level grants, access is controlled entirely by the credentials used to call the API.
+
+On Prefect Cloud, the API key must belong to a [service account or user](https://docs.prefect.io/v3/how-to-guides/cloud/manage-users/api-keys) with at least **Viewer** access to the workspace identified by the Account ID and Workspace ID. On a self-hosted Prefect Server, if `PREFECT_SERVER_API_AUTH_STRING` is set, provide the same `user:password` value as the Basic Auth String; if the server has no auth enabled, leave it empty.
+
+### Python Requirements
+
+Prefect only depends on the `requests` library, which already ships with the base package. To run the Prefect ingestion, you will need to install:
+```bash
+pip3 install "openmetadata-ingestion"
+```
+## Metadata Ingestion
+All connectors are defined as JSON Schemas.
+[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/pipeline/prefectConnection.json)
+you can find the structure to create a connection to Prefect.
+In order to create and run a Metadata Ingestion workflow, we will follow
+the steps to create a YAML configuration able to connect to the source,
+process the Entities if needed, and reach the OpenMetadata server.
+The workflow is modeled around the following
+[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)
+### 1. Define the YAML Config
+This is a sample config for Prefect:
+
+
+
+
+
+Configure the source type and service name for your Prefect connector.
+
+
+
+
+
+**hostPort**: Prefect API base URL. Use `https://api.prefect.cloud` for Prefect Cloud, or your self-hosted server's URL, for example, `http://localhost:4200`.
+
+
+
+
+
+**authType**: Choose the authentication type that matches your Prefect deployment:
+- **Prefect Cloud**: `apiKey`, `accountId`, and `workspaceId` are all required. The Account ID and Workspace ID can be found in the workspace URL: `app.prefect.cloud/account//workspace/`.
+- **Prefect Server**: `authString` holds the `PREFECT_SERVER_API_AUTH_STRING` value in `user:password` format. Leave it empty if the server has no auth enabled.
+
+
+
+
+
+**numberOfStatus** (Optional): Number of past flow run statuses to ingest per flow. Default: `10`, maximum `100`.
+
+
+
+
+
+**verifySSL** (Optional): Client SSL verification mode for the Prefect API connection: `no-ssl` (default), `ignore`, or `validate`.
+**sslConfig** (Optional): CA certificate (and, for mutual TLS, client certificate and key) used when `verifySSL` is set to `validate`.
+
+
+
+
+
+**pipelineFilterPattern** (Optional): Regex to only include/exclude flows that match the pattern.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+```yaml
+source:
+ type: prefect
+ serviceName: prefect_source
+ serviceConnection:
+ config:
+ type: Prefect
+ hostPort: https://api.prefect.cloud # REQUIRED - or your self-hosted server URL, for example, http://localhost:4200
+ authType:
+ apiKey: # REQUIRED for Prefect Cloud
+ accountId: # REQUIRED for Prefect Cloud
+ workspaceId: # REQUIRED for Prefect Cloud
+ # #
+ # authString: ":" # Prefect Server (PREFECT_SERVER_API_AUTH_STRING); leave empty if no auth
+ # numberOfStatus: 10
+ # verifySSL: no-ssl
+ # pipelineFilterPattern:
+ # includes:
+ # -
+ # excludes:
+ # -
+```
+
+
+
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/pipeline/spline.mdx b/v2.0.x-SNAPSHOT/connectors/pipeline/spline.mdx
index d8d5abd11..00de52f9d 100644
--- a/v2.0.x-SNAPSHOT/connectors/pipeline/spline.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/pipeline/spline.mdx
@@ -3,11 +3,9 @@ title: Spline Connector | OpenMetadata Data Lineage Integration
description: Connect Spline data lineage to OpenMetadata with our pipeline connector. Track data flow, automate metadata ingestion, and enhance governance seamlessly.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/pipeline/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentPipeline from '/snippets/connectors/metadata-agent/pipeline.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
Currently, we do not support data source of type AWS S3 or any other cloud storage, which also means that the lineage for external tables from Databricks will not be extracted.
+
You can refer [this](https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-databricks) documentation on how to configure Databricks with Spline.
+
## Metadata Ingestion
-
-# Connection Details
-
-
+
+To ingest metadata from Spline, you need to create a service connection. The service connects Spline with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Pipeline Services**, then click the **Spline** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other Spline services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for Spline. A help panel displays inline guidance for each field.
+
+
+
+- **Spline REST Server Host & Port**: OpenMetadata uses Spline REST Server APIs to extract the execution details from Spline to generate lineage. This should be specified as a URI string in the format `scheme://hostname:port`. For example, `http://localhost:8080`, `http://host.docker.internal:8080`.
+- **Spline UI Host & Port**: Spline UI Host & Port is an optional field which is used for generating a redirection URL from OpenMetadata to the Spline Portal. This should be specified as a URI string in the format `scheme://hostname:port`. For example, `http://localhost:9090`, `http://host.docker.internal:9090`.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your Spline service. Filter patterns use regular expressions applied to pipeline names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all pipelines available in the source.
-- **Spline REST Server Host & Port**: OpenMetadata uses Spline REST Server APIs to extract the execution details from spline to generate lineage. This should be specified as a URI string in the format `scheme://hostname:port`. E.g., `http://localhost:8080`, `http://host.docker.internal:8080`.
-- **Spline UI Host & Port**: Spline UI Host & Port is an optional field which is used for generating redirection URL from OpenMetadata to Spline Portal. This should be specified as a URI string in the format `scheme://hostname:port`. E.g., `http://localhost:9090`, `http://host.docker.internal:9090`.
-
-
-
-
-
+
+#### Filter Options
+
+The Pipeline section includes the following filter options:
+
+- **Pipeline**: Controls which pipelines OpenMetadata ingests from Spline.
+
+Each section provides the following controls:
+
+- **Scan Mode**: Choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/search/elasticsearch.mdx b/v2.0.x-SNAPSHOT/connectors/search/elasticsearch.mdx
index b887cf92a..b3e747e1e 100644
--- a/v2.0.x-SNAPSHOT/connectors/search/elasticsearch.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/search/elasticsearch.mdx
@@ -3,11 +3,9 @@ title: Elasticsearch Connector | OpenMetadata Search Services
description: Connect Elasticsearch to OpenMetadata with our comprehensive connector guide. Step-by-step setup, configuration, and integration instructions for seamless search.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/search/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentSearch from '/snippets/connectors/metadata-agent/search.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from ElasticSearch, you need to create a service connection. The service connects ElasticSearch with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Search Services**, then click the **ElasticSearch** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other ElasticSearch services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for ElasticSearch. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
- **Host and Port**: This parameter specifies the host and port of the Elasticsearch instance. This should be specified as a URI string in the format `http://hostname:port` or `https://hostname:port`. For example, you might set it to `https://localhost:9200`.
-- **Authentication Types**:
- 1. Basic Authentication
+- **Connection Timeout in Seconds**: Connection timeout configuration for communicating with Elasticsearch APIs.
+
+**Authentication**
+- **Basic Authentication**:
- Username: Username to connect to Elasticsearch required when Basic Authentication is enabled on Elasticsearch.
- Password: Password of the user account to connect with Elasticsearch.
- 2. API Key Authentication
+- **API Key Authentication**:
- API Key: API Key to connect to Elasticsearch required when API Key Authentication is enabled on Elasticsearch.
- - API Key Id: Enter API Key ID In case of API Key Authentication if there is any API Key ID associated with the API Key, otherwise this field can be left blank.
-- **SSL Certificates**:
- 1. SSL Certificates By Path
- - CA Certificate Path: This field specifies the path of CA certificate required for authentication.
- - Client Certificate Path: This field specifies the path of Clint certificate required for authentication.
- - Private Key Path: This field specifies the path of Clint Key/Private Key required for authentication.
-
- 2. SSL Certificates By Value
- - CA Certificate Value: This field specifies the value of CA certificate required for authentication.
- - Client Certificate Value: This field specifies the value of Clint certificate required for authentication.
- - Private Key Value: This field specifies the value of Clint Key/Private Key required for authentication.
- - Staging Directory Path: This field specifies the path to temporary staging directory, where the certificates will be stored temporarily during the ingestion process, which will de deleted once the ingestion job is over.
- - when you are using this approach make sure you are passing the key in a correct format. If your certificate looks like this:
+ - API Key Id: Enter API Key ID in case of API Key Authentication if there is any API Key ID associated with the API Key, otherwise this field can be left blank.
+
+**Advanced Config** *(SSL options; most connections don't need these)*
+- **SSL Certificates By Path**:
+ - CA Certificate Path: This field specifies the path of the CA certificate required for authentication.
+ - Client Certificate Path: This field specifies the path of the client certificate required for authentication.
+ - Private Key Path: This field specifies the path of the client key/private key required for authentication.
+- **SSL Certificates By Value**:
+ - CA Certificate Value: This field specifies the value of the CA certificate required for authentication.
+ - Client Certificate Value: This field specifies the value of the client certificate required for authentication.
+ - Private Key Value: This field specifies the value of the client key/private key required for authentication.
+ - Staging Directory Path: This field specifies the path to a temporary staging directory where the certificates are stored during the ingestion process. The directory is cleared once the ingestion job completes.
+
+ When using the "By Value" approach, make sure you pass the key in the correct format. If your certificate looks like this:
```
-----BEGIN CERTIFICATE-----
MII..
@@ -66,12 +102,64 @@ For a complete guide on managing secrets in hybrid setups, see the [Hybrid Inges
h+4=
-----END CERTIFICATE-----
```
- You will have to replace new lines with `\n` and the final value that you need to pass should look like this:
+ Replace new lines with `\n` so the final value looks like this:
```
-----BEGIN CERTIFICATE-----\nMII..\nMBQ...\nCgU..\n8Lt..\n...\nh+4=\n-----END CERTIFICATE-----\n
-- **Connection Timeout in Seconds**: Connection timeout configuration for communicating with Elasticsearch APIs.
-
-
-
-
-
+ ```
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your search service. Filter patterns use regular expressions applied to index names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all search indexes available in the source.
+
+
+**Filter Options**
+
+The Search Index section includes the following filter options:
+
+- **Search Index**: Controls which search indexes (Elasticsearch indexes, OpenSearch indexes, or equivalent) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/search/opensearch.mdx b/v2.0.x-SNAPSHOT/connectors/search/opensearch.mdx
index 91759d304..915dccc59 100644
--- a/v2.0.x-SNAPSHOT/connectors/search/opensearch.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/search/opensearch.mdx
@@ -3,11 +3,9 @@ title: OpenSearch Connector | OpenMetadata Search Integration
description: Connect OpenMetadata to OpenSearch with our comprehensive connector guide. Setup instructions, configuration options, and troubleshooting tips included.
sidebarTitle: Overview
---
-import ConfigureIngestion from '/snippets/connectors/search/configure-ingestion.mdx'
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
-import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
+import MetadataAgentSearch from '/snippets/connectors/metadata-agent/search.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-# Connection Details
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
+
+To ingest metadata from OpenSearch, you need to create a service connection. The service connects OpenSearch with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Search Services**, then click the **OpenSearch** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other OpenSearch services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for OpenSearch. The right-hand panel in the UI displays inline help for each field.
+
+
+
+**Connection**
- **Host and Port**: This parameter specifies the host and port of the OpenSearch instance. This should be specified as a URI string in the format `http://hostname:port` or `https://hostname:port`. For example, you might set it to `https://localhost:9200`.
-- **Authentication Types**:
- 1. Basic Authentication
+- **Verify SSL**: Client SSL verification. Make sure to configure the SSL Config if enabled. Possible values:
+ - `validate`: Validate the certificate using the public certificate (recommended).
+ - `ignore`: Ignore the certification validation (not recommended for production).
+ - `no-ssl`: SSL validation is not needed.
+- **Connection Timeout in Seconds**: Connection timeout configuration for communicating with OpenSearch APIs.
+
+**Authentication**
+- **Basic Authentication**:
- Username: Username to connect to OpenSearch required when Basic Authentication is enabled on OpenSearch.
- Password: Password of the user account to connect with OpenSearch.
- 2. IAM based Authentication
- - **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
- permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
- authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
- Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
- You must use both the access key ID and secret access key together to authenticate your requests.
- You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- - **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
- As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
- Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
- services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
- You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- - **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
- and AWS Secrets Access Key. Also, these will include an AWS Session Token.
- You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
- - **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
- entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
- default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
- Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
- - **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
- When you specify a profile to run a command, the settings and credentials are used to run that command.
- Multiple named profiles can be stored in the config and credentials files.
- You can inform this field if you'd like to use a profile other than `default`.
- Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- - **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
- `ARN` (Amazon Resource Name) of the policy of the other account.
- A user who wants to access a role in a different account must also have permissions that are delegated from the account
- administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
- This is a required field if you'd like to `AssumeRole`.
- Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
- - **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
- is assumed by different principals or for different reasons.
- By default, we'll use the name `OpenMetadataSession`.
- Find more information about the [Role Session Name](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=An%20identifier%20for%20the%20assumed%20role%20session.).
- - **Assume Role Source Identity**: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity
- information in AWS CloudTrail logs to determine who took actions with a role.
- Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
-- **Verify SSL**:
-Client SSL verification. Make sure to configure the SSLConfig if enabled.
-Possible values:
- * `validate`: Validate the certificate using the public certificate (recommended).
- * `ignore`: Ignore the certification validation (not recommended for production).
- * `no-ssl`: SSL validation is not needed.
-- **SSL Certificates**:
- 1. SSL Certificates By Path
- - CA Certificate Path: This field specifies the path of CA certificate required for authentication.
- - Client Certificate Path: This field specifies the path of Clint certificate required for authentication.
- - Private Key Path: This field specifies the path of Clint Key/Private Key required for authentication.
-
- 2. SSL Certificates By Value
- - CA Certificate Value: This field specifies the value of CA certificate required for authentication.
- - Client Certificate Value: This field specifies the value of Clint certificate required for authentication.
- - Private Key Value: This field specifies the value of Clint Key/Private Key required for authentication.
- - Staging Directory Path: This field specifies the path to temporary staging directory, where the certificates will be stored temporarily during the ingestion process, which will de deleted once the ingestion job is over.
- - when you are using this approach make sure you are passing the key in a correct format. If your certificate looks like this:
- ```
- -----BEGIN CERTIFICATE-----
- MII..
- MBQ...
- CgU..
- 8Lt..
- ...
- h+4=
- -----END CERTIFICATE-----
- ```
- You will have to replace new lines with `\n` and the final value that you need to pass should look like this:
- ```
- -----BEGIN CERTIFICATE-----\nMII..\nMBQ...\nCgU..\n8Lt..\n...\nh+4=\n-----END CERTIFICATE-----\n
-- **Connection Timeout in Seconds**: Connection timeout configuration for communicating with OpenSearch APIs.
-
-
-
-
-
+- **IAM Based Authentication**:
+ - AWS Access Key ID and AWS Secret Access Key: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)). Access keys consist of two parts: an **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`). You must use both together to authenticate your requests. Find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+ - AWS Region: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)). As AWS can have instances in multiple regions, OpenMetadata needs to know the region the service you want to reach belongs to. The AWS Region is the only required parameter when configuring a connection. Find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+ - AWS Session Token (optional): If you are using temporary credentials to access your services, you will need to provide the AWS Access Key ID and AWS Secret Access Key along with an AWS Session Token. Find more information on [using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
+ - Endpoint URL (optional): To connect programmatically to an AWS service, you use an endpoint. An endpoint is the URL of the entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region, but you can specify an alternate endpoint for your API requests. Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
+ - Profile Name: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command. Provide this field if you'd like to use a profile other than `default`. Find more information about [named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+ - Assume Role Arn: Typically, you use `AssumeRole` within your account or for cross-account access. In this field, set the `ARN` (Amazon Resource Name) of the policy of the other account. A user who wants to access a role in a different account must also have permissions delegated from the account administrator. Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+ - Assume Role Session Name: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role is assumed by different principals or for different reasons. By default, OpenMetadata uses the name `OpenMetadataSession`.
+ - Assume Role Source Identity: The source identity specified by the principal that is calling the `AssumeRole` operation. You can use source identity information in AWS CloudTrail logs to determine who took actions with a role.
+
+**Advanced Config** *(SSL options; most connections don't need these)*
+- **SSL Certificates By Path**:
+ - CA Certificate Path: This field specifies the path of the CA certificate required for authentication.
+ - Client Certificate Path: This field specifies the path of the client certificate required for authentication.
+ - Private Key Path: This field specifies the path of the client key/private key required for authentication.
+- **SSL Certificates By Value**:
+ - CA Certificate Value: This field specifies the value of the CA certificate required for authentication.
+ - Client Certificate Value: This field specifies the value of the client certificate required for authentication.
+ - Private Key Value: This field specifies the value of the client key/private key required for authentication.
+ - Staging Directory Path: This field specifies the path to a temporary staging directory where the certificates are stored during the ingestion process. The directory is cleared once the ingestion job completes.
+
+ When using the "By Value" approach, make sure you pass the key in the correct format. If your certificate looks like this:
+ ```
+ -----BEGIN CERTIFICATE-----
+ MII..
+ MBQ...
+ CgU..
+ 8Lt..
+ ...
+ h+4=
+ -----END CERTIFICATE-----
+ ```
+ Replace new lines with `\n` so the final value looks like this:
+ ```
+ -----BEGIN CERTIFICATE-----\nMII..\nMBQ...\nCgU..\n8Lt..\n...\nh+4=\n-----END CERTIFICATE-----\n
+ ```
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your search service. Filter patterns use regular expressions applied to index names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only assets whose names match at least one expression. Leave blank to include all assets.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any asset whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all search indexes available in the source.
+
+
+**Filter Options**
+
+The Search Index section includes the following filter options:
+
+- **Search Index**: Controls which search indexes (Elasticsearch indexes, OpenSearch indexes, or equivalent) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/storage/gcs.mdx b/v2.0.x-SNAPSHOT/connectors/storage/gcs.mdx
index abe098d91..e5f1436c2 100644
--- a/v2.0.x-SNAPSHOT/connectors/storage/gcs.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/storage/gcs.mdx
@@ -4,10 +4,9 @@ description: Connect OpenMetadata to Google Cloud Storage with our comprehensive
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
import Manifest from '/snippets/connectors/storage/manifest.mdx'
-import ConfigureIngestion from '/snippets/connectors/storage/configure-ingestion.mdx'
+import MetadataAgentStorage from '/snippets/connectors/metadata-agent/storage.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-This page contains the setup guide and reference information for the GCS connector.
+In this section, we provide guides and references to use the GCS connector.
Configure and schedule GCS metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/storage/gcs/troubleshooting)
+
## Requirements
+
We need the following permissions in GCP:
+
### GCS Permissions
+
For all the buckets that we want to ingest, we need to provide the following:
- `storage.buckets.get`
- `storage.buckets.list`
- `storage.objects.get`
- `storage.objects.list`
+
### OpenMetadata Manifest
+
In any other connector, extracting metadata happens automatically. In this case, we will be able to extract high-level
metadata from buckets, but in order to understand their internal structure we need users to provide an `openmetadata.json`
file at the bucket root.
+
`Supported File Formats: [ "csv", "tsv", "avro", "parquet", "json", "json.gz", "json.zip" ]`
+
You can learn more about this [here](/v2.0.x-SNAPSHOT/connectors/storage). Keep reading for an example on the shape of the manifest file.
+
+
## Metadata Ingestion
-
-
- The first step is ingesting the metadata from your sources. Under
-Settings, you will find a Services link an external source system to
-OpenMetadata. Once a service is created, it can be used to configure
-metadata, usage, and profiler workflows.
-To visit the Services page, select Services from the Settings menu.
- 
-
-
- Click on the 'Add New Service' button to start the Service creation.
- 
-
-
- Select GCS as the service type and click Next.
- 
-
-
- Provide a name and description for your service.
-#### Service Name
-OpenMetadata uniquely identifies services by their Service Name. Provide
-a name that distinguishes your deployment from other services, including
-the other Storage services that you might be ingesting metadata
-from.
- 
-
-
- In this step, we will configure the connection settings required for
-this connector. Please follow the instructions below to ensure that
-you've configured the connector to read from your GCS service as
-desired.
- 
-
-
+
+To ingest metadata from GCS, you need to create a service connection. The service connects GCS with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Storage Services**, then click the **GCS** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other GCS services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for GCS. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **GCP Credentials**: You can authenticate with your GCS instance using either **GCP Credentials Path**, where you specify the file path of the service account key, or **GCP Credentials Values**, where you pass the credential values directly from the service account key file. You can check out [this](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) documentation on how to create the service account keys and download it.
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your storage service. Filter patterns use regular expressions applied to container names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only containers whose names match at least one expression. Leave blank to include all containers.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any container whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
+Leave the filter pattern empty to ingest all containers (S3 buckets, ADLS containers, GCS buckets) available in the source.
-
-
-
-
-
+
+**Filter Options**
+
+The Container section includes the following filter options:
+
+- **Container**: Controls which top-level storage containers (S3 buckets, ADLS containers, or GCS buckets) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
diff --git a/v2.0.x-SNAPSHOT/connectors/storage/s3.mdx b/v2.0.x-SNAPSHOT/connectors/storage/s3.mdx
index dbb7e0519..04086f1f7 100644
--- a/v2.0.x-SNAPSHOT/connectors/storage/s3.mdx
+++ b/v2.0.x-SNAPSHOT/connectors/storage/s3.mdx
@@ -4,10 +4,9 @@ description: Connect your S3 storage to OpenMetadata with our comprehensive conn
sidebarTitle: Overview
---
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
-import TestConnection from '/snippets/connectors/test-connection.mdx'
-import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
import Manifest from '/snippets/connectors/storage/manifest.mdx'
-import ConfigureIngestion from '/snippets/connectors/storage/configure-ingestion.mdx'
+import MetadataAgentStorage from '/snippets/connectors/metadata-agent/storage.mdx'
+import Related from '/snippets/v2.0.x-SNAPSHOT/connectors/database/related.mdx'
-This page contains the setup guide and reference information for the S3 connector.
+In this section, we provide guides and references to use the S3 connector.
Configure and schedule S3 metadata workflows from the OpenMetadata UI:
- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Troubleshooting](/v2.0.x-SNAPSHOT/connectors/storage/s3/troubleshooting)
+
## Requirements
+
We need the following permissions in AWS:
+
### S3 Permissions
+
For all the buckets that we want to ingest, we need to provide the following:
- `s3:ListBucket`
- `s3:GetObject`
- `s3:GetBucketLocation`
- `s3:ListAllMyBuckets`
+
Note that the `Resources` should be all the buckets that you'd like to scan. A possible policy could be:
+
```json
{
"Version": "2012-10-17",
@@ -48,11 +53,15 @@ Note that the `Resources` should be all the buckets that you'd like to scan. A p
]
}
```
+
### CloudWatch Permissions
+
Which is used to fetch the total size in bytes for a bucket and the total number of files. It requires:
- `cloudwatch:GetMetricData`
- `cloudwatch:ListMetrics`
+
The policy would look like:
+
```json
{
"Version": "2012-10-17",
@@ -69,90 +78,96 @@ The policy would look like:
]
}
```
+
### OpenMetadata Manifest
+
In any other connector, extracting metadata happens automatically. In this case, we will be able to extract high-level
metadata from buckets, but in order to understand their internal structure we need users to provide an `openmetadata.json`
file at the bucket root.
+
`Supported File Formats: [ "csv", "tsv", "avro", "parquet", "json", "json.gz", "json.zip" ]`
+
You can learn more about this [here](/v2.0.x-SNAPSHOT/connectors/storage). Keep reading for an example on the shape of the manifest file.
+
+
## Metadata Ingestion
-
-
- The first step is ingesting the metadata from your sources. Under
-Settings, you will find a Services link an external source system to
-OpenMetadata. Once a service is created, it can be used to configure
-metadata, usage, and profiler workflows.
-To visit the Services page, select Services from the Settings menu.
-
-
-
- Click on the 'Add New Service' button to start the Service creation.
-
-
-
- Select S3 as the service type and click Next.
- 
-
-
- Provide a name and description for your service.
-#### Service Name
-OpenMetadata uniquely identifies services by their Service Name. Provide
-a name that distinguishes your deployment from other services, including
-the other Storage services that you might be ingesting metadata
-from.
- 
-
-
- In this step, we will configure the connection settings required for
-this connector. Please follow the instructions below to ensure that
-you've configured the connector to read from your S3 service as
-desired.
- 
-
-
-
-When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
-```
-password: secret:/my/database/password
-```
-This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
-For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
-
-- **AWS Access Key ID** & **AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
+
+To ingest metadata from S3, you need to create a service connection. The service connects S3 with OpenMetadata. Once you create a service, OpenMetadata automatically starts ingesting metadata.
+
+### Step 1: Add New Service
+
+1. Navigate to **Settings** > **Services**.
+
+
+
+2. Click **Add New Service**.
+
+
+
+### Step 2: Select a Service and Connector
+
+From the service type dropdown, select **Storage Services**, then click the **S3** connector tile.
+
+
+
+### Step 3: Add Service Name and Description
+
+- Enter a unique, descriptive **Service Name**. OpenMetadata identifies services by their service name. Enter a name that distinguishes this deployment from other S3 services you are ingesting metadata from.
+- Optional: Enter a **Description** for the service.
+
+
+
+
+**Note**: The service name cannot be changed after it is set.
+
+
+### Step 4: Configure Connection Options
+
+Specify your source credentials and verify the connection.
+
+#### Enter Connection Details
+
+Enter the connection details for S3. The right-hand panel in the UI displays inline help for each field.
+
+
+
+- **AWS Access Key ID and AWS Secret Access Key**: When you interact with AWS, you specify your AWS security credentials to verify who you are and whether you have
permission to access the resources that you are requesting. AWS uses the security credentials to authenticate and
authorize your requests ([docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html)).
Access keys consist of two parts: An **access key ID** (for example, `AKIAIOSFODNN7EXAMPLE`), and a **secret access key** (for example, `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY`).
You must use both the access key ID and secret access key together to authenticate your requests.
-You can find further information on how to manage your access keys [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
+You can find further information on [how to manage your access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
- **AWS Region**: Each AWS Region is a separate geographic area in which AWS clusters data centers ([docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)).
-As AWS can have instances in multiple regions, we need to know the region the service you want reach belongs to.
+As AWS can have instances in multiple regions, we need to know the region the service you want to reach belongs to.
Note that the AWS Region is the only required parameter when configuring a connection. When connecting to the
services programmatically, there are different ways in which we can extract and use the rest of AWS configurations.
-You can find further information about configuring your credentials [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
+You can find further information about [configuring your credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials).
- **AWS Session Token (optional)**: If you are using temporary credentials to access your services, you will need to inform the AWS Access Key ID
and AWS Secrets Access Key. Also, these will include an AWS Session Token.
You can find more information on [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html).
- **Endpoint URL (optional)**: To connect programmatically to an AWS service, you use an endpoint. An *endpoint* is the URL of the
- entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically use the
+ entry point for an AWS web service. The AWS SDKs and the AWS Command Line Interface (AWS CLI) automatically uses the
default endpoint for each service in an AWS Region. But you can specify an alternate endpoint for your API requests.
Find more information on [AWS service endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
-- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to a AWS CLI command.
+- **Profile Name**: A named profile is a collection of settings and credentials that you can apply to an AWS CLI command.
When you specify a profile to run a command, the settings and credentials are used to run that command.
Multiple named profiles can be stored in the config and credentials files.
You can inform this field if you'd like to use a profile other than `default`.
-Find here more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
+Find more information about [Named profiles for the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
- **Assume Role Arn**: Typically, you use `AssumeRole` within your account or for cross-account access. In this field you'll set the
`ARN` (Amazon Resource Name) of the policy of the other account.
A user who wants to access a role in a different account must also have permissions that are delegated from the account
administrator. The administrator must attach a policy that allows the user to call `AssumeRole` for the `ARN` of the role in the other account.
This is a required field if you'd like to `AssumeRole`.
Find more information on [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
+
When using Assume Role authentication, ensure you provide the following details:
- **AWS Region**: Specify the AWS region for your deployment.
- **Assume Role ARN**: Provide the ARN of the role in your AWS account that OpenMetadata will assume.
+
- **Assume Role Session Name**: An identifier for the assumed role session. Use the role session name to uniquely identify a session when the same role
is assumed by different principals or for different reasons.
By default, we'll use the name `OpenMetadataSession`.
@@ -161,8 +176,60 @@ Find more information about the [Role Session Name](https://docs.aws.amazon.com/
information in AWS CloudTrail logs to determine who took actions with a role.
Find more information about [Source Identity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html#:~:text=Required%3A%20No-,SourceIdentity,-The%20source%20identity).
- **Bucket Names (Optional)**: Provide the names of buckets that you would want to ingest, if you want to ingest metadata from all buckets or apply a filter to ingest buckets then leave this field empty.
-
-
-
-
-
+
+#### Test Connection
+
+Once the credentials have been added, click on **Test Connection** and **Save** the changes.
+
+
+
+### Step 5: Configure Ingestion Options
+
+In the **What to Ingest** step, use filter patterns to control which assets OpenMetadata ingests from your storage service. Filter patterns use regular expressions applied to container names.
+
+#### How Filter Patterns Work
+
+- **Include**: Add one or more comma-separated regular expressions. OpenMetadata ingests only containers whose names match at least one expression. Leave blank to include all containers.
+- **Exclude**: Add one or more comma-separated regular expressions. OpenMetadata skips any container whose name matches an expression. Leave blank to exclude nothing.
+
+Rules match asset names using one of five expressions:
+
+- **contains**: matches any name containing the value. For example, `sales` matches `my_sales_data` and `sales_2024`.
+- **starts with**: matches names beginning with the value. For example, `prod_` matches `prod_db` and `prod_schema`.
+- **ends with**: matches names ending with the value. For example, `_raw` matches `events_raw` and `logs_raw`.
+- **is exactly**: matches the exact name only. For example, `analytics` matches only `analytics`.
+- **matches regex**: matches names using a regular expression. For example, `^prod_.*_v\d+$` matches `prod_events_v1`.
+
+When both Include and Exclude are set, Exclude takes priority.
+
+
+Leave the filter pattern empty to ingest all containers (S3 buckets, ADLS containers, GCS buckets) available in the source.
+
+
+**Filter Options**
+
+The Container section includes the following filter options:
+
+- **Container**: Controls which top-level storage containers (S3 buckets, ADLS containers, or GCS buckets) OpenMetadata ingests from the source.
+
+Each section provides the following controls:
+
+- **Scan Mode**: You can choose between the following scan modes:
+ - **Scan all**: Ingests every asset of that type the connector can access. This is the default.
+ - **Only specific**: Enables include rules so only assets matching at least one rule are ingested.
+- **Exclude system toggle**: Use this toggle to automatically filter out system-reserved names defined by the connector, for example, **Exclude system databases** for the Databases section.
+- **Always exclude**: Add permanent exclusion rules (shown in red). Assets matching these rules are never ingested, regardless of include rules.
+- **Preview**: Shows a real-time summary of what will be in scope based on your current rules.
+- **Include rules** *(available only in **Only specific** mode)*: Click **+ Add** to define a rule. Added rules appear as chips; an asset is included if it matches any rule.
+
+### Step 6: Create & Deploy
+
+Click **Create & Deploy** to deploy the agent and start the first metadata ingestion run. OpenMetadata saves the service configuration and immediately begins pulling metadata from the source.
+
+To monitor ingestion progress or view the service you just added, go to **Settings** > **Services** and select your service.
+
+## Configure Metadata Agent and Schedule Ingestion
+
+
+
+
+
+2. Click the service you have added.
+
+3. Select the **Agents** tab and click **Add Agent** > **Metadata**.
+
+ 
+
+ For some services, the dropdown is not available and clicking **Add Agent** takes you directly to the agent configuration page.
+
+4. On the **Configure Ingestion** page, do the following and click **Next**.
+
+ - **Name this Ingestion**: Enter a unique recognizable name for this ingestion pipeline.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which API collections and endpoints this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Mark Deleted API Collections | On | Soft-delete API collections in OpenMetadata when they are removed from the source. Associated entities like lineage are also deleted. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+6. Click **Add** to deploy the agent.
diff --git a/snippets/connectors/metadata-agent/dashboard.mdx b/snippets/connectors/metadata-agent/dashboard.mdx
new file mode 100644
index 000000000..df82720b8
--- /dev/null
+++ b/snippets/connectors/metadata-agent/dashboard.mdx
@@ -0,0 +1,62 @@
+The **Metadata Agent** extracts dashboards, charts, data models, and other structural metadata from your source and keeps your OpenMetadata catalog in sync. It powers discovery, lineage, and governance across your data assets.
+
+When you click **Create & Deploy**, OpenMetadata automatically deploys a Metadata Agent for this service and triggers the first ingestion run. View its status and run history from the **Agents** tab on the service detail page.
+
+To configure the additional Metadata Agent and schedule ingestion, follow these steps:
+
+1. Navigate to **Settings** > **Services** and select the service type.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which dashboards, charts, data models, and projects this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Include Current Owners | Off | When enabled, replaces the current owner with a new owner from the source if the current owner is null. Recommended to keep enabled during the first metadata ingestion to capture owner information. |
+ | Include Tags | On | Ingest tags from the source. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+ | Override Lineage | Off | When on, existing lineage is replaced with newly extracted lineage on each run. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which databases, schemas, tables, and stored procedures this agent ingests. For more information about various filter options, see **Step 6: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control how the agent handles metadata during ingestion. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Include Tables | On | Ingest table metadata from the source. Turn off to skip tables entirely. |
+ | Include Tags | On | Ingest tags from the source. |
+ | Include Stored Procedures | On | Ingest stored procedure metadata. |
+ | Include DDL Statements | Off | Ingest DDL statements alongside table metadata. |
+ | Include Owners | Off | Assign owners to ingested entities when the source owner's email matches a user in OpenMetadata. Does not overwrite an existing owner. |
+ | Include Custom Properties | Off | Ingest connector-specific custom properties onto entities. |
+ | Mark Deleted Tables | On | Soft-delete tables in OpenMetadata when they are removed from the source. Applies only within the currently ingested schema. |
+ | Mark Deleted Stored Procedures | On | Soft-delete stored procedures in OpenMetadata when they are removed from the source. |
+ | Mark Deleted Schemas | Off | Soft-delete schemas and all their child assets when removed from the source. |
+ | Mark Deleted Databases | Off | Soft-delete databases and all their child assets when removed from the source. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+
+ 
+
+ - **Advanced Config**: Optional connector-specific settings such as Include Views and Extract JSON Schema.
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which directories, files, spreadsheets, and worksheets this agent ingests. Use FQN for filtering to apply regex on fully qualified names (for example, `service_name.directory_name.file_name`) instead of raw names. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Include Tags | On | Ingest tags from the source. |
+ | Include Owners | Off | Assign owners from the source when the owner email matches an OpenMetadata user. Does not overwrite an existing owner. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which topics this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which databases, schemas, tables, and stored procedures this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Mark Deleted Tables | On | Soft-delete tables in OpenMetadata when they are removed from the source. Applies only within the currently ingested schema. |
+ | Mark Deleted Stored Procedures | On | Soft-delete stored procedures in OpenMetadata when they are removed from the source. Associated entities like lineage are also deleted. |
+ | Mark Deleted Schemas | Off | Soft-delete schemas and all their child assets when removed from the source. |
+ | Mark Deleted Databases | Off | Soft-delete databases and all their child assets when removed from the source. |
+ | Include Tables | On | Ingest table metadata from the source. Turn off to skip tables entirely. |
+ | Include Tags | On | Ingest tags from the source. |
+ | Include Custom Properties | Off | Ingest connector-specific custom properties onto entities. |
+ | Include Owners | Off | Assign owners to ingested entities when the source owner's email matches a user in OpenMetadata. Does not overwrite an existing owner. |
+ | Include Stored Procedures | On | Ingest stored procedure metadata. |
+ | Include DDL Statements | Off | Ingest DDL statements alongside table metadata. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which ML models this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+ | Override Lineage | Off | When on, existing lineage is replaced with newly extracted lineage on each run. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which pipelines this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+ | Override Lineage | Off | When on, existing lineage is replaced with newly extracted lineage on each run. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 
+
+ - **Filter Patterns**: Apply include or exclude rules to scope which search indexes this agent ingests. These follow the same filter options described in **Step 5: Configure Ingestion Options**.
+
+ 
+
+ - **Scope & Behaviour**: Control what metadata to include and how to handle deletions. Toggle each option on or off based on your needs:
+
+ | Toggle | Default | Description |
+ |---|---|---|
+ | Enable Debug Log | Off | Sets the ingestion log level to DEBUG. Useful for troubleshooting. |
+ | Override Metadata | Off | When on, source values overwrite existing descriptions, tags, owners, and display names in OpenMetadata. When off, OpenMetadata only updates fields that have no existing value. |
+
+ 
+
+5. On the **Schedule Interval** page, set when the agent runs:
+ - **Schedule**: Choose a preset interval (Hourly, Daily, Weekly, Monthly) or enter a custom cron expression.
+ - **On-Demand**: No automatic schedule; trigger the agent manually when needed.
+
+ 