From 4c7b4c2e14a950f43b7b095538cb38e1b1c04cbe Mon Sep 17 00:00:00 2001 From: Esteban Pastorino Date: Thu, 2 Jul 2026 17:13:55 +0100 Subject: [PATCH 1/2] Document object storage ClickPipe settings --- .../clickpipes/object-storage/amazon-s3/01_overview.md | 8 +++++--- .../object-storage/amazon-s3/03_unordered-mode.md | 8 +++++--- .../object-storage/azure-blob-storage/01_overview.md | 3 ++- .../object-storage/google-cloud-storage/01_overview.md | 6 ++++-- .../google-cloud-storage/03_unordered-mode.md | 10 ++++++---- 5 files changed, 22 insertions(+), 13 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md b/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md index f5fd231d7a6..cfe83fd4c40 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md @@ -55,7 +55,7 @@ When continuous ingestion is enabled, ClickPipes continuously ingests data from By default, the S3 ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file **must** be lexically greater than the last ingested file. For example, files named `file1`, `file2`, and `file3` will be ingested sequentially, but if a new `file 0` is added to the bucket, it will be **ignored** because the file name isn't lexically greater than the last ingested file. -In this mode, the S3 ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the specified path. +In this mode, the S3 ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). To begin continuous ingestion after a specific file, configure **Start after**. ClickPipes will skip files that are lexicographically less than or equal to the configured value and continue ingesting newer files from that point. #### Any order {#continuous-ingestion-any-order} @@ -69,13 +69,13 @@ It's possible to configure an S3 ClickPipe to ingest files that don't have an im Unordered mode is **only** supported for Amazon S3 and is **not** supported for public buckets or S3-compatible services. It requires setting up an [Amazon SQS](https://aws.amazon.com/sqs/) queue connected to the bucket, optionally using [Amazon EventBridge](https://aws.amazon.com/eventbridge/) as an event router. ::: -In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. +In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. To ingest only files delivered through the queue, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the SQS queue. :::note Setting a prefix/postfix for events is optional. If you do, make sure it matches the path set for the clickpipe. S3 doesn't allow multiple overlapping notification rules for the same event types. ::: -Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path. If a DLQ is configured, failed messages will be reenqueued and reprocessed up to the number of times configured in the DLQ `maxReceiveCount` parameter. +Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). If a DLQ is configured, failed messages will be reenqueued and reprocessed up to the number of times configured in the DLQ `maxReceiveCount` parameter. :::tip We strongly recommend configuring a **Dead-Letter-Queue (DLQ)** for the SQS queue, so it's easier to debug and retry failed messages. @@ -198,6 +198,8 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca | `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. | | `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). | | `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. | +| `Start after` | None | For continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | +| `Skip initial load` | false | For unordered continuous ingestion, skips the initial scan of existing files and processes only object notifications from the configured SQS queue. | | `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). | | `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). | | `Use cluster function` | true | Whether to process files in parallel across multiple nodes. | diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/03_unordered-mode.md b/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/03_unordered-mode.md index 7771645406e..171a77275b7 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/03_unordered-mode.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/03_unordered-mode.md @@ -23,7 +23,9 @@ Unordered mode is **only** supported for Amazon S3 and is **not** supported for ## How it works {#how-it-works} -In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, a file not matching the path, or an event of a different type will be **ignored**. Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path. +In this mode, the S3 ClickPipe does an initial load of **all files** in the selected path, and then listens for `ObjectCreated:*` events in the queue that match the specified path. Any message for a previously seen file, a file not matching the path, or an event of a different type will be **ignored**. Files are ingested once the threshold configured in `max insert bytes` or `max file count` is reached, or after a configurable interval (by default, 30 seconds). + +To ingest only files delivered through the queue, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the SQS queue. Various types of failures can occur when ingesting data, which can result in partial inserts or duplicate data. Object Storage ClickPipes are resilient to insert failures and provide exactly-once semantics using temporary staging tables. Data is first inserted into a staging table; if something goes wrong, the staging table is truncated and the insert is retried from a clean state. Only once an insert completes successfully are the partitions moved to the target table. @@ -200,10 +202,10 @@ S3 doesn't allow multiple overlapping notification rules for the same event type **1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe** and select **Amazon S3**. Enter the details to connect to your S3 bucket. Under **Authentication method**, choose **IAM role** and provide the ARN of the role you created in the previous step. -**2.** Under **Incoming data**, toggle on **Continuous ingestion**. Select **Any order** as the ingestion mode and provide the **SQS queue URL** for the queue connected to your bucket. +**2.** Under **Incoming data**, toggle on **Continuous ingestion**. Select **Any order** as the ingestion mode and provide the **SQS queue URL** for the queue connected to your bucket. To process only files from the queue and skip the initial scan of existing files in the selected path, enable **Skip initial load**. **3.** Under **Parse information**, define a **Sorting key** for the target table. Make any necessary adjustments to the mapped schema, then configure a role for the ClickPipes database user. -**4.** Review the configuration and click **Create ClickPipe**. ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as new `ObjectCreated:*` events arrive in the queue. +**4.** Review the configuration and click **Create ClickPipe**. Unless **Skip initial load** is enabled, ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as matching `ObjectCreated:*` events are delivered from the queue. diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md b/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md index 234b47ce26b..baf1eacf3c1 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md @@ -36,7 +36,7 @@ When continuous ingestion is enabled, ClickPipes continuously ingests data from The ABS ClickPipe assumes files are added to a container in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file **must** be lexically greater than the last ingested file. For example, files named `file1`, `file2`, and `file3` will be ingested sequentially, but if a new `file 0` is added to the container, it will be **ignored** because the file name isn't lexically greater than the last ingested file. -In this mode, the ABS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the specified path. +In this mode, the ABS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). To begin continuous ingestion after a specific file, configure **Start after**. ClickPipes will skip files that are lexicographically less than or equal to the configured value and continue ingesting newer files from that point. ### File pattern matching {#file-pattern-matching} @@ -142,6 +142,7 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca | `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. | | `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). | | `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. | +| `Start after` | None | For continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | | `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). | | `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). | | `Use cluster function` | true | Whether to process files in parallel across multiple nodes. | diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md index ef37664bfa6..db0b4bf6875 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md @@ -39,7 +39,7 @@ When continuous ingestion is enabled, ClickPipes continuously ingests data from The GCS ClickPipe assumes files are added to a bucket in lexicographical order, and relies on this implicit order to ingest files sequentially. This means that any new file **must** be lexically greater than the last ingested file. For example, files named `file1`, `file2`, and `file3` will be ingested sequentially, but if a new `file 0` is added to the bucket, it will be **ignored** because the file name isn't lexically greater than the last ingested file. -In this mode, the GCS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the specified path. +In this mode, the GCS ClickPipe does an initial load of **all files** in the specified path, and then polls for new files at a configurable interval (by default, 30 seconds). To begin continuous ingestion after a specific file, configure **Start after**. ClickPipes will skip files that are lexicographically less than or equal to the configured value and continue ingesting newer files from that point. #### Any order {#continuous-ingestion-any-order} @@ -53,7 +53,7 @@ It's possible to configure a GCS ClickPipe to ingest files that don't have an im Unordered mode is **not** supported for public buckets. It requires **Service Account** authentication and a [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) subscription connected to the bucket. ::: -In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for `OBJECT_FINALIZE` notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path. +In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for `OBJECT_FINALIZE` notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. To ingest only files delivered through the subscription, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the Pub/Sub subscription. ### File pattern matching {#file-pattern-matching} @@ -154,6 +154,8 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca | `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. | | `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). | | `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. | +| `Start after` | None | For continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | +| `Skip initial load` | false | For unordered continuous ingestion, skips the initial scan of existing files and processes only object notifications from the configured Pub/Sub subscription. | | `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). | | `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). | | `Use cluster function` | true | Whether to process files in parallel across multiple nodes. | diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md index 3a3f3ec4eb0..787f1aa2433 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md @@ -18,7 +18,9 @@ Unordered mode is **not** supported for public buckets. It requires **Service Ac ## How it works {#how-it-works} -In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for object notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. It is **not possible** to start ingestion from a specific file or point in time — ClickPipes will always load all files in the selected path. +In this mode, the GCS ClickPipe does an initial load of **all files** in the selected path, and then listens for object notifications via the Pub/Sub subscription that match the specified path. Any message for a previously seen file, file not matching the path, or event of a different type will be **ignored**. + +To ingest only files delivered through the subscription, enable **Skip initial load**. When enabled, ClickPipes skips the initial scan of existing files in the selected path and processes only files delivered through the Pub/Sub subscription. Various types of failures can occur when ingesting data, which can result in partial inserts or duplicate data. Object Storage ClickPipes are resilient to insert failures and provide exactly-once semantics using temporary staging tables. Data is first inserted into a staging table; if something goes wrong, the staging table is truncated and the insert is retried from a clean state. Only once an insert completes successfully are the partitions moved to the target table. @@ -70,7 +72,7 @@ Various types of failures can occur when ingesting data, which can result in par **1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe** and select **Google Cloud Storage**. Enter the details to connect to your GCS bucket. Under **Authentication method**, choose **Service Account** and provide the `.json` service account key. -**2.** Toggle on **Continuous ingestion**, then select **Any order** as the ingestion mode and provide the **Pub/Sub subscription** name for the subscription connected to your bucket. The subscription name must follow the following format: +**2.** Toggle on **Continuous ingestion**, then select **Any order** as the ingestion mode and provide the **Pub/Sub subscription** name for the subscription connected to your bucket. To process only files from the subscription and skip the initial scan of existing files in the selected path, enable **Skip initial load**. The subscription name must follow the following format: ```text projects/${YOUR_PROJECT_ID}/subscriptions/${YOUR_SUBSCRIPTION_NAME} @@ -78,6 +80,6 @@ projects/${YOUR_PROJECT_ID}/subscriptions/${YOUR_SUBSCRIPTION_NAME} **3.** Click **Incoming data**. Define a **Sorting key** for the target table. Make any necessary adjustments to the mapped schema, then configure a role for the ClickPipes database user. -**4.** Review the configuration and click **Create ClickPipe**. ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as new `OBJECT_FINALIZE` events arrive in the topic. +**4.** Review the configuration and click **Create ClickPipe**. Unless **Skip initial load** is enabled, ClickPipes will perform an initial scan of your bucket to load all existing files that match the specified path, and will then begin processing files as matching `OBJECT_FINALIZE` events are delivered from the subscription. - \ No newline at end of file + From eac84af0b6ad83ae961a68cbcf7df7d695e4d4fd Mon Sep 17 00:00:00 2001 From: Esteban Pastorino Date: Fri, 3 Jul 2026 10:36:41 +0100 Subject: [PATCH 2/2] Address object storage ClickPipe docs review --- .../clickpipes/object-storage/amazon-s3/01_overview.md | 2 +- .../clickpipes/object-storage/azure-blob-storage/01_overview.md | 2 +- .../object-storage/google-cloud-storage/01_overview.md | 2 +- .../object-storage/google-cloud-storage/03_unordered-mode.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md b/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md index cfe83fd4c40..50d2b36b4c2 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/01_overview.md @@ -198,7 +198,7 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca | `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. | | `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). | | `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. | -| `Start after` | None | For continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | +| `Start after` | None | For ordered continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | | `Skip initial load` | false | For unordered continuous ingestion, skips the initial scan of existing files and processes only object notifications from the configured SQS queue. | | `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). | | `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). | diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md b/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md index baf1eacf3c1..29c0795725e 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/azure-blob-storage/01_overview.md @@ -142,7 +142,7 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca | `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. | | `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). | | `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. | -| `Start after` | None | For continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | +| `Start after` | None | For ordered continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | | `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). | | `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). | | `Use cluster function` | true | Whether to process files in parallel across multiple nodes. | diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md index db0b4bf6875..5fba1b70418 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/01_overview.md @@ -154,7 +154,7 @@ ClickPipes provides sensible defaults that cover the requirements of most use ca | `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. | | `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). | | `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. | -| `Start after` | None | For continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | +| `Start after` | None | For ordered continuous ingestion, skips files that are lexicographically less than or equal to the configured filename or path. | | `Skip initial load` | false | For unordered continuous ingestion, skips the initial scan of existing files and processes only object notifications from the configured Pub/Sub subscription. | | `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). | | `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). | diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md index 787f1aa2433..70e9a2b037a 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage/google-cloud-storage/03_unordered-mode.md @@ -72,7 +72,7 @@ Various types of failures can occur when ingesting data, which can result in par **1.** In the ClickHouse Cloud console, navigate to **Data Sources > Create ClickPipe** and select **Google Cloud Storage**. Enter the details to connect to your GCS bucket. Under **Authentication method**, choose **Service Account** and provide the `.json` service account key. -**2.** Toggle on **Continuous ingestion**, then select **Any order** as the ingestion mode and provide the **Pub/Sub subscription** name for the subscription connected to your bucket. To process only files from the subscription and skip the initial scan of existing files in the selected path, enable **Skip initial load**. The subscription name must follow the following format: +**2.** Toggle on **Continuous ingestion**, then select **Any order** as the ingestion mode and provide the **Pub/Sub subscription** name for the subscription connected to your bucket. To process only files from the subscription and skip the initial scan of existing files in the selected path, enable **Skip initial load**. The subscription name must use the following format: ```text projects/${YOUR_PROJECT_ID}/subscriptions/${YOUR_SUBSCRIPTION_NAME}