From 6d0a94f92c94005579745b7e15be33e6ba59494b Mon Sep 17 00:00:00 2001 From: morsapaes Date: Fri, 3 Jul 2026 17:29:17 +0100 Subject: [PATCH 1/2] clickpipes: add new CDC resource utilization metrics --- .../data-ingestion/clickpipes/monitoring.md | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/monitoring.md b/docs/integrations/data-ingestion/clickpipes/monitoring.md index 2ca3a589cb5..94bab1bcce4 100644 --- a/docs/integrations/data-ingestion/clickpipes/monitoring.md +++ b/docs/integrations/data-ingestion/clickpipes/monitoring.md @@ -7,7 +7,6 @@ doc_type: 'reference' keywords: ['ClickPipes', 'monitoring', 'metrics', 'Prometheus', 'observability'] --- - In addition to in-console monitoring, ClickPipes exposes metrics to a [Prometheus-compatible endpoint](/integrations/prometheus) for scraping. These metrics are published with other ClickHouse Cloud service metrics, and allow you to integrate ClickPipes monitoring with your existing observability stack (e.g., [Grafana](/integrations/prometheus#integrating-with-grafana), [Datadog](/integrations/prometheus#integrating-with-datadog)). ## Metric labels {#metric-labels} @@ -89,14 +88,27 @@ ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickho | `ClickPipes_Latency` | Gauge (_avg_) | Time in milliseconds between when a record was produced at the source and when it was written to ClickHouse. | | `ClickPipes_SourceReplicationLatency_MiB` | Gauge (_avg_) | **CDC ClickPipes** Replication lag at the source in MiB. For Postgres ClickPipes, this reflects the replication slot lag. | -### Resource usage {#metrics-resources} +### Resource utilization {#metrics-resources} | Metric | Type | Description | |----------------------------------|----------------|-------------| -| `ClickPipes_Replica_CPUUsage` | Gauge (_avg_) | Average CPU usage of the ingestion replicas, in millicores. | -| `ClickPipes_Replica_CPULimit` | Gauge (_last_) | Configured CPU limit of the ingestion replicas, in millicores. | -| `ClickPipes_Replica_MemoryUsage` | Gauge (_avg_) | Average memory usage of the ingestion replicas, in bytes. | -| `ClickPipes_Replica_MemoryLimit` | Gauge (_last_) | Configured memory limit of the ingestion replicas, in bytes. | +| `ClickPipes_Replica_CPUUsage` | Gauge (_avg_) | Average CPU usage of the ingestion replicas, in millicores. | +| `ClickPipes_Replica_CPULimit` | Gauge (_last_) | Configured CPU limit of the ingestion replicas, in millicores. | +| `ClickPipes_Replica_MemoryUsage` | Gauge (_avg_) | Average memory usage of the ingestion replicas, in bytes. | +| `ClickPipes_Replica_MemoryLimit` | Gauge (_last_) | Configured memory limit of the ingestion replicas, in bytes. | +| `ClickPipes_CDC_CPUUsage` | Gauge (_avg_) | **CDC ClickPipes** Average CPU usage, in millicores. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_CPULimit` | Gauge (_last_) | **CDC ClickPipes** Configured CPU limit, in millicores. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_MemoryUsage` | Gauge (_avg_) | **CDC ClickPipes** Average memory usage, in bytes. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_MemoryLimit` | Gauge (_last_) | **CDC ClickPipes** Configured memory limit, in bytes. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_NetworkReceiveBytes` | Gauge | **CDC ClickPipes** Inbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service. | + +#### Granularity {#metrics-resources-granularity} + +These metrics report resource utilization at different levels of granularity, based on the underlying implementation of each category of connectors: + +* `ClickPipes_Replica_*` report the resource usage of a **single ingestion replica**. Each of these ClickPipes has its own dedicated compute, so the values belong to that one ClickPipe. + +* `ClickPipes_CDC_*` report resource usage for **CDC ClickPipes** (Postgres, MySQL, and MongoDB). All CDC ClickPipes in the same service **share** compute, so these values reflect the combined usage of every CDC ClickPipe in the service, not a single one. ### State and progress {#metrics-state} From b92c77835ebad2e7e8a2b87aaa5e5ecf8cbac765 Mon Sep 17 00:00:00 2001 From: morsapaes Date: Sat, 4 Jul 2026 10:05:41 +0100 Subject: [PATCH 2/2] Split into sections --- .../data-ingestion/clickpipes/monitoring.md | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/monitoring.md b/docs/integrations/data-ingestion/clickpipes/monitoring.md index 94bab1bcce4..e9dffba93dd 100644 --- a/docs/integrations/data-ingestion/clickpipes/monitoring.md +++ b/docs/integrations/data-ingestion/clickpipes/monitoring.md @@ -90,25 +90,28 @@ ClickPipes_Latency{clickhouse_org="11dfa1ec-767d-43cb-bfad-618ce2aaf959",clickho ### Resource utilization {#metrics-resources} +#### Streaming and object storage ClickPipes {#metrics-resources-streaming-os} + +`ClickPipes_Replica_*` metrics report resource utilization for streaming and object storage ClickPipes. Each ClickPipe has **dedicated compute**, so these values reflect utilization for individual ClickPipes. + | Metric | Type | Description | |----------------------------------|----------------|-------------| | `ClickPipes_Replica_CPUUsage` | Gauge (_avg_) | Average CPU usage of the ingestion replicas, in millicores. | | `ClickPipes_Replica_CPULimit` | Gauge (_last_) | Configured CPU limit of the ingestion replicas, in millicores. | | `ClickPipes_Replica_MemoryUsage` | Gauge (_avg_) | Average memory usage of the ingestion replicas, in bytes. | | `ClickPipes_Replica_MemoryLimit` | Gauge (_last_) | Configured memory limit of the ingestion replicas, in bytes. | -| `ClickPipes_CDC_CPUUsage` | Gauge (_avg_) | **CDC ClickPipes** Average CPU usage, in millicores. Shared across all CDC ClickPipes in the service. | -| `ClickPipes_CDC_CPULimit` | Gauge (_last_) | **CDC ClickPipes** Configured CPU limit, in millicores. Shared across all CDC ClickPipes in the service. | -| `ClickPipes_CDC_MemoryUsage` | Gauge (_avg_) | **CDC ClickPipes** Average memory usage, in bytes. Shared across all CDC ClickPipes in the service. | -| `ClickPipes_CDC_MemoryLimit` | Gauge (_last_) | **CDC ClickPipes** Configured memory limit, in bytes. Shared across all CDC ClickPipes in the service. | -| `ClickPipes_CDC_NetworkReceiveBytes` | Gauge | **CDC ClickPipes** Inbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service. | - -#### Granularity {#metrics-resources-granularity} -These metrics report resource utilization at different levels of granularity, based on the underlying implementation of each category of connectors: +#### CDC ClickPipes {#metrics-resources-cdc} -* `ClickPipes_Replica_*` report the resource usage of a **single ingestion replica**. Each of these ClickPipes has its own dedicated compute, so the values belong to that one ClickPipe. +`ClickPipes_CDC_*` metrics report resource utilization for CDC ClickPipes. All CDC ClickPipes in the same service **share compute**, so these values reflect the combined usage of every CDC ClickPipe in the service. -* `ClickPipes_CDC_*` report resource usage for **CDC ClickPipes** (Postgres, MySQL, and MongoDB). All CDC ClickPipes in the same service **share** compute, so these values reflect the combined usage of every CDC ClickPipe in the service, not a single one. +| Metric | Type | Description | +|----------------------------------|----------------|-------------| +| `ClickPipes_CDC_CPUUsage` | Gauge (_avg_) | Average CPU usage, in millicores. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_CPULimit` | Gauge (_last_) | Configured CPU limit, in millicores. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_MemoryUsage` | Gauge (_avg_) | Average memory usage, in bytes. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_MemoryLimit` | Gauge (_last_) | Configured memory limit, in bytes. Shared across all CDC ClickPipes in the service. | +| `ClickPipes_CDC_NetworkReceiveBytes` | Gauge | Inbound network bandwidth, in bytes (delta per scrape window). Shared across all CDC ClickPipes in the service. | ### State and progress {#metrics-state}