From 313fa555661863d220daaac37e026143016390cf Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 12:46:47 +0200 Subject: [PATCH 01/59] Improve geo-replication documentation --- docs/administration-geo.md | 97 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 5 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 3c6fd4c98fdf..de0ec977a3bd 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -22,7 +22,39 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) -Any message published on *any* topic in that namespace is replicated to all clusters in the specified set. +### Replication configuration settings + +The target clusters for replication of a message are determined by a hierarchy of settings at the tenant, namespace, topic, and message level: + +* **Tenant `allowed-clusters`**: Specifies which clusters the tenant is permitted to use for replication. An empty value means all clusters are allowed. Note that this setting cannot be modified once the tenant has existing namespaces, so it must be configured before creating any namespaces under the tenant. + +* **Namespace `clusters`**: Defines the default set of clusters that messages in the namespace are replicated to. + +* **Namespace `allowed-clusters`**: Further restricts which clusters are permitted for replication at the namespace level, overriding the tenant-level setting. Introduced in [PIP-321](https://github.com/apache/pulsar/blob/master/pip/pip-321.md). + +* **Topic-level policies**: Can override the namespace-level `clusters` setting for a specific topic. Topic policies can be **local** (applying only to the local cluster) or **global** (replicated to all clusters in the geo-replication set, see [PIP-92](https://github.com/apache/pulsar/blob/master/pip/pip-92.md)). Note that `allowed-clusters` has not been implemented at the topic level; [PIP-321](https://github.com/apache/pulsar/blob/master/pip/pip-321.md) mentions that it could be added in the future. + +* **Message-level replication control**: Producers can override which clusters a specific message is replicated to using the [`replicationClusters`](/api/client/4.1.x/org/apache/pulsar/client/api/TypedMessageBuilder.html#replicationClusters(java.util.List)) method in the client API, or disable replication entirely for a message using [`disableReplication`](/api/client/4.1.x/org/apache/pulsar/client/api/TypedMessageBuilder.html#disableReplication()) (see [Selective replication](#selective-replication)). Note that these settings cannot override the `allowed-clusters` configuration — messages can only be routed to clusters that are permitted by the resolved `allowed-clusters` settings. + +The `clusters` and `allowed-clusters` settings are resolved hierarchically. When the tenant-level `allowed-clusters` is non-empty, all clusters specified in namespace-level `allowed-clusters` must be a subset of it — this is validated when `allowed-clusters` is modified at the namespace level. Namespace-level `allowed-clusters` can further restrict the tenant-level configuration, and topic-level policies can override the namespace-level `clusters` setting for a specific topic. + +### 1-way and 2-way geo-replication + +Geo-replication can be configured as 1-way or 2-way: + +* **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). + +* **1-way replication**: Messages flow in one direction only (for example, `us-west` → `us-east`). Replication in the other direction can be prevented using two mechanisms: + * **`allowed-clusters`**: Forcefully prevents replication to clusters not in the resolved `allowed-clusters` configuration. This cannot be overridden by producers using the message-level `replicationClusters` setting. + * **`clusters`**: Sets the default replication targets. If the reverse direction cluster is not included, replication is 1-way by default, though producers can still override this per-message using the `replicationClusters` setting. + + :::note + The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. + ::: + +When using a **shared configuration store**, tenant-level and namespace-level configuration is shared across all clusters. If both clusters are listed in the shared namespace `clusters` setting and neither is excluded by `allowed-clusters`, replication is 2-way. + +When using **separate metadata stores** (no shared configuration store), each cluster has its own metadata store and you must configure geo-replication independently on each cluster. To achieve 2-way replication, both clusters must be configured so that geo-replication is enabled for the topic on both sides — the tenant's `allowed-clusters`, the namespace `clusters`, and the namespace `allowed-clusters` must all include both clusters on each cluster. ## Local persistence and forwarding @@ -203,6 +235,20 @@ You can explicitly disable topic garbage collection by setting `brokerDeleteInac To delete a geo-replication topic, close all producers and consumers on the topic, and delete all of its local subscriptions in every replication cluster. When Pulsar determines that no valid subscription for the topic remains across the system, it will garbage collect the topic. +## Propagated deletions in geo-replication + +In a geo-replicated setup, topic and namespace deletions will propagate to peer clusters. The direction of propagation follows the direction of replication: in 1-way replication, deletions propagate in one direction; in 2-way replication, they propagate in both directions. + +When you remove a cluster from the `clusters` replication configuration at the namespace level, Pulsar automatically deletes the sub-topics on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), the same cascading deletion applies at the topic level when you remove a cluster from a topic-level `clusters` policy. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. + +:::warning + +Removing a cluster from the `clusters` configuration will delete that cluster's sub-topics automatically. If you want to stop replication to a cluster without deleting its data, do not remove the cluster from the replication list. + +::: + +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if data durability across accidental deletions is a requirement. + ## Replicated subscriptions Pulsar supports replicated subscriptions, so you can keep the subscription state in sync, within a sub-second timeframe, in the context of a topic that is being asynchronously replicated across multiple geographical regions. @@ -211,11 +257,17 @@ In case of failover, a consumer can restart consuming from the failure point in ### Enable replicated subscription +:::note + +Replicated subscriptions require [2-way geo-replication](#1-way-and-2-way-geo-replication) to be properly configured between all participating clusters. See [1-way and 2-way geo-replication](#1-way-and-2-way-geo-replication) for configuration requirements. + +::: + If you want to use replicated subscriptions in Pulsar: -- For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). +* For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). -- For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. +* For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. ```java Consumer consumer = client.newConsumer(Schema.STRING) @@ -239,14 +291,49 @@ The advantages of replicated subscription are as follows. The limitations of replicated subscription are as follows. * When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. -* Only the base line cursor position is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. +* Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. +* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may not be processed at all if the replication snapshot reaches the other cluster before the messages are consumed there. It is recommended to route processing to a single active cluster when using replicated subscriptions. +* Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. :::note -* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions#storing-snapshots) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. +* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions#storing-snapshots) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. ::: +### Replicated subscriptions snapshot configuration and tuning + +Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions). + +Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are always required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. + +A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. + +After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. + +The following broker settings control snapshot behavior: + +| Setting | Default | Description | +| --- | --- | --- | +| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | + +For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. + +### Observability + +Observability for replicated subscriptions is limited. For debugging, debug-level logs are available in `org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController`, though these are not suitable for production operations. + +The following broker-level metrics are available for monitoring snapshot health. Note that these metrics are aggregated across all topics on a broker and do not include per-topic labels. + +| Metric | OpenTelemetry name | Description | +| --- | --- | --- | +| `pulsar_replicated_subscriptions_pending_snapshots` | `pulsar.broker.replication.subscription.snapshot.operation.count` | Number of currently pending snapshots. | +| `pulsar_replicated_subscriptions_timedout_snapshots` | `pulsar.broker.replication.subscription.snapshot.operation.duration` | Number of snapshots that have timed out. | + +Topic stats and internal stats can be used to inspect the state of subscriptions. The cursor's mark-delete position is particularly useful, as subscription state can only be replicated up to that position. + ## Migrate data between clusters using geo-replication Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. From 0f240bdafc77be263425ab132fd53e21d612df7b Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 21:36:46 +0200 Subject: [PATCH 02/59] Add clarifications --- docs/administration-geo.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index de0ec977a3bd..216450767ee0 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -235,19 +235,26 @@ You can explicitly disable topic garbage collection by setting `brokerDeleteInac To delete a geo-replication topic, close all producers and consumers on the topic, and delete all of its local subscriptions in every replication cluster. When Pulsar determines that no valid subscription for the topic remains across the system, it will garbage collect the topic. -## Propagated deletions in geo-replication +## Cascading topic deletions when modifying the replication clusters configuration -In a geo-replicated setup, topic and namespace deletions will propagate to peer clusters. The direction of propagation follows the direction of replication: in 1-way replication, deletions propagate in one direction; in 2-way replication, they propagate in both directions. +Tenant, namespace, and topic configurations can be shared or synchronized across clusters in two ways: -When you remove a cluster from the `clusters` replication configuration at the namespace level, Pulsar automatically deletes the sub-topics on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), the same cascading deletion applies at the topic level when you remove a cluster from a topic-level `clusters` policy. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. +* **Shared configuration store**: Tenant and namespace metadata is stored in a shared configuration store accessible to all clusters. +* **`configurationMetadataSyncEventTopic`**: Tenant, namespace, and topic configuration changes are synchronized across clusters via geo-replication. The direction of synchronization follows the direction of replication — with 1-way replication, configuration updates flow in one direction only; with 2-way replication, they flow in both directions. + +Topic policies are also shared via geo-replication when the namespace has geo-replication enabled, with support for both local (single-cluster) and global (all-clusters) policies. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). + +When the namespace `clusters` configuration is shared or synchronized across clusters and a cluster is removed from that list, Pulsar automatically deletes all topics for that namespace on the excluded cluster. + +When a topic-level `clusters` policy is shared or synchronized across clusters and a cluster is removed from that policy, Pulsar automatically deletes the topic and all its partitions on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), this cascading deletion is supported at the topic level in addition to the namespace level. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, policy updates flow only toward the destination cluster. :::warning -Removing a cluster from the `clusters` configuration will delete that cluster's sub-topics automatically. If you want to stop replication to a cluster without deleting its data, do not remove the cluster from the replication list. +When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. It is not possible to stop replication to a cluster without triggering these deletions when the configuration is shared or synchronized. ::: -Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if data durability across accidental deletions is a requirement. +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. When namespace or topic configuration is shared or synchronized across clusters, a misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ## Replicated subscriptions @@ -292,7 +299,7 @@ The limitations of replicated subscription are as follows. * When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. * Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. -* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may not be processed at all if the replication snapshot reaches the other cluster before the messages are consumed there. It is recommended to route processing to a single active cluster when using replicated subscriptions. +* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may be processed in either cluster if the replication subscription update reaches the other cluster before the messages are consumed there. It is recommended to process messages in a single cluster when using replicated subscriptions. * Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. :::note From 91008940f258ade888386127e373dcf650f3985d Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 21:39:44 +0200 Subject: [PATCH 03/59] Fix links --- docs/administration-geo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 216450767ee0..66fda971ad93 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -304,13 +304,13 @@ The limitations of replicated subscription are as follows. :::note -* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions#storing-snapshots) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. +* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. The snapshot is created only when new messages have been produced to the topic after the last snapshot creation begun. ::: ### Replicated subscriptions snapshot configuration and tuning -Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions). +Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are always required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. From 01ccfd895723597bfffea584b530ebc6be37a890 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 22:53:42 +0200 Subject: [PATCH 04/59] Document deletion to include PIP-422 details --- docs/administration-geo.md | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 66fda971ad93..695147b20ceb 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -221,19 +221,31 @@ bin/pulsar-admin topics stats persistent://my-tenant/my-namespace/my-topic Each cluster reports its own local stats, including the incoming and outgoing replication rates and backlogs. -#### Delete a geo-replication topic +#### Geo-replication topic deletion -Given that geo-replication topics exist in multiple regions, directly deleting a geo-replication topic is not possible. Instead, you should rely on automatic topic garbage collection. +**Topic deletion** -In Pulsar, a topic is automatically deleted when the topic meets the following three conditions: -- no producers or consumers are connected to it; -- no subscriptions to it; -- no more messages are kept for retention. -For geo-replication topics, each region uses a fault-tolerant mechanism to decide when deleting the topic locally is safe. +The recommended procedure for deleting a geo-replication topic from all clusters is: -You can explicitly disable topic garbage collection by setting `brokerDeleteInactiveTopicsEnabled` to `false` in your [broker configuration](reference-configuration.md#broker). +1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. +2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. +4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. -To delete a geo-replication topic, close all producers and consumers on the topic, and delete all of its local subscriptions in every replication cluster. When Pulsar determines that no valid subscription for the topic remains across the system, it will garbage collect the topic. +Without this procedure, forcefully deleting a topic on one cluster leaves it orphaned — it still exists on peer clusters and geo-replication from those clusters remains active. If auto-topic creation is enabled on the cluster where the topic was deleted, the topic may be recreated through auto-creation or because `createTopicToRemoteClusterForReplication=true` is set on a peer cluster. + +When namespace or topic configuration is shared via a shared configuration store or synchronized via `configurationMetadataSyncEventTopic`, forcefully deleting a topic propagates the deletion to all clusters that share or receive the configuration. However, replication delays may cause the topic to be recreated before the deletion takes effect everywhere. For this reason, it is recommended to follow the procedure above. + +To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. + +**Garbage collection** + +A geo-replication topic is also automatically deleted by garbage collection when `brokerDeleteInactiveTopicsEnabled=true` and no producers or consumers are connected to it. The additional conditions depend on the `brokerDeleteInactiveTopicsMode` setting: + +- `delete_when_no_subscriptions`: the topic is deleted when there are no subscriptions. +- `delete_when_subscriptions_caught_up`: the topic is deleted when all subscriptions have caught up and there is no backlog. + +Each region independently decides when it is safe to delete the topic locally. To trigger garbage collection, close all producers and consumers on the topic and delete all local subscriptions in every replication cluster. When Pulsar determines that no valid subscription remains across the system, it garbage collects the topic. ## Cascading topic deletions when modifying the replication clusters configuration @@ -250,7 +262,9 @@ When a topic-level `clusters` policy is shared or synchronized across clusters a :::warning -When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. It is not possible to stop replication to a cluster without triggering these deletions when the configuration is shared or synchronized. +When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. + +To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. ::: From 3d953bdb4e830f4607d4fda54007ab12b256ecdd Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 23:27:26 +0200 Subject: [PATCH 05/59] Document deleting from all but one cluster --- docs/administration-geo.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 695147b20ceb..641f419e404f 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -238,6 +238,8 @@ When namespace or topic configuration is shared via a shared configuration store To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. +To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. + **Garbage collection** A geo-replication topic is also automatically deleted by garbage collection when `brokerDeleteInactiveTopicsEnabled=true` and no producers or consumers are connected to it. The additional conditions depend on the `brokerDeleteInactiveTopicsMode` setting: From df53be160c7762087d1c7b7d35302bd42ccc1e72 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 23:37:28 +0200 Subject: [PATCH 06/59] Grammar --- docs/administration-geo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 641f419e404f..d00d3d4f2144 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -320,7 +320,7 @@ The limitations of replicated subscription are as follows. :::note -* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. The snapshot is created only when new messages have been produced to the topic after the last snapshot creation begun. +* Replicated subscriptions add a special snapshot message to the topic every second. If there are inactive subscriptions on the topic, this increases the backlog on both the source and destination clusters. The [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is only created when new messages have been produced to the topic since the last snapshot was taken. ::: From 9edefd95d602da846f6135f9b117a37e65e50840 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 23:43:10 +0200 Subject: [PATCH 07/59] Minor revisits --- docs/administration-geo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index d00d3d4f2144..fefa1eed6f19 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -223,7 +223,7 @@ Each cluster reports its own local stats, including the incoming and outgoing re #### Geo-replication topic deletion -**Topic deletion** +**Explicit topic deletion** The recommended procedure for deleting a geo-replication topic from all clusters is: @@ -240,7 +240,7 @@ To remove a topic from a specific cluster only, set a global topic-level `cluste To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. -**Garbage collection** +**Deletion by garbage collection** A geo-replication topic is also automatically deleted by garbage collection when `brokerDeleteInactiveTopicsEnabled=true` and no producers or consumers are connected to it. The additional conditions depend on the `brokerDeleteInactiveTopicsMode` setting: @@ -328,7 +328,7 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are always required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. From ed235f3d78516c1de38fac6f8d77552ffca916dc Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Thu, 5 Mar 2026 23:46:35 +0200 Subject: [PATCH 08/59] Sync to 4.0.x and 4.1.x docs --- .../version-4.0.x/administration-geo.md | 138 ++++++++++++++++-- .../version-4.1.x/administration-geo.md | 138 ++++++++++++++++-- 2 files changed, 248 insertions(+), 28 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 3c6fd4c98fdf..fefa1eed6f19 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -22,7 +22,39 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) -Any message published on *any* topic in that namespace is replicated to all clusters in the specified set. +### Replication configuration settings + +The target clusters for replication of a message are determined by a hierarchy of settings at the tenant, namespace, topic, and message level: + +* **Tenant `allowed-clusters`**: Specifies which clusters the tenant is permitted to use for replication. An empty value means all clusters are allowed. Note that this setting cannot be modified once the tenant has existing namespaces, so it must be configured before creating any namespaces under the tenant. + +* **Namespace `clusters`**: Defines the default set of clusters that messages in the namespace are replicated to. + +* **Namespace `allowed-clusters`**: Further restricts which clusters are permitted for replication at the namespace level, overriding the tenant-level setting. Introduced in [PIP-321](https://github.com/apache/pulsar/blob/master/pip/pip-321.md). + +* **Topic-level policies**: Can override the namespace-level `clusters` setting for a specific topic. Topic policies can be **local** (applying only to the local cluster) or **global** (replicated to all clusters in the geo-replication set, see [PIP-92](https://github.com/apache/pulsar/blob/master/pip/pip-92.md)). Note that `allowed-clusters` has not been implemented at the topic level; [PIP-321](https://github.com/apache/pulsar/blob/master/pip/pip-321.md) mentions that it could be added in the future. + +* **Message-level replication control**: Producers can override which clusters a specific message is replicated to using the [`replicationClusters`](/api/client/4.1.x/org/apache/pulsar/client/api/TypedMessageBuilder.html#replicationClusters(java.util.List)) method in the client API, or disable replication entirely for a message using [`disableReplication`](/api/client/4.1.x/org/apache/pulsar/client/api/TypedMessageBuilder.html#disableReplication()) (see [Selective replication](#selective-replication)). Note that these settings cannot override the `allowed-clusters` configuration — messages can only be routed to clusters that are permitted by the resolved `allowed-clusters` settings. + +The `clusters` and `allowed-clusters` settings are resolved hierarchically. When the tenant-level `allowed-clusters` is non-empty, all clusters specified in namespace-level `allowed-clusters` must be a subset of it — this is validated when `allowed-clusters` is modified at the namespace level. Namespace-level `allowed-clusters` can further restrict the tenant-level configuration, and topic-level policies can override the namespace-level `clusters` setting for a specific topic. + +### 1-way and 2-way geo-replication + +Geo-replication can be configured as 1-way or 2-way: + +* **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). + +* **1-way replication**: Messages flow in one direction only (for example, `us-west` → `us-east`). Replication in the other direction can be prevented using two mechanisms: + * **`allowed-clusters`**: Forcefully prevents replication to clusters not in the resolved `allowed-clusters` configuration. This cannot be overridden by producers using the message-level `replicationClusters` setting. + * **`clusters`**: Sets the default replication targets. If the reverse direction cluster is not included, replication is 1-way by default, though producers can still override this per-message using the `replicationClusters` setting. + + :::note + The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. + ::: + +When using a **shared configuration store**, tenant-level and namespace-level configuration is shared across all clusters. If both clusters are listed in the shared namespace `clusters` setting and neither is excluded by `allowed-clusters`, replication is 2-way. + +When using **separate metadata stores** (no shared configuration store), each cluster has its own metadata store and you must configure geo-replication independently on each cluster. To achieve 2-way replication, both clusters must be configured so that geo-replication is enabled for the topic on both sides — the tenant's `allowed-clusters`, the namespace `clusters`, and the namespace `allowed-clusters` must all include both clusters on each cluster. ## Local persistence and forwarding @@ -189,19 +221,56 @@ bin/pulsar-admin topics stats persistent://my-tenant/my-namespace/my-topic Each cluster reports its own local stats, including the incoming and outgoing replication rates and backlogs. -#### Delete a geo-replication topic +#### Geo-replication topic deletion + +**Explicit topic deletion** + +The recommended procedure for deleting a geo-replication topic from all clusters is: + +1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. +2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. +4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. + +Without this procedure, forcefully deleting a topic on one cluster leaves it orphaned — it still exists on peer clusters and geo-replication from those clusters remains active. If auto-topic creation is enabled on the cluster where the topic was deleted, the topic may be recreated through auto-creation or because `createTopicToRemoteClusterForReplication=true` is set on a peer cluster. + +When namespace or topic configuration is shared via a shared configuration store or synchronized via `configurationMetadataSyncEventTopic`, forcefully deleting a topic propagates the deletion to all clusters that share or receive the configuration. However, replication delays may cause the topic to be recreated before the deletion takes effect everywhere. For this reason, it is recommended to follow the procedure above. + +To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. + +To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. + +**Deletion by garbage collection** + +A geo-replication topic is also automatically deleted by garbage collection when `brokerDeleteInactiveTopicsEnabled=true` and no producers or consumers are connected to it. The additional conditions depend on the `brokerDeleteInactiveTopicsMode` setting: + +- `delete_when_no_subscriptions`: the topic is deleted when there are no subscriptions. +- `delete_when_subscriptions_caught_up`: the topic is deleted when all subscriptions have caught up and there is no backlog. + +Each region independently decides when it is safe to delete the topic locally. To trigger garbage collection, close all producers and consumers on the topic and delete all local subscriptions in every replication cluster. When Pulsar determines that no valid subscription remains across the system, it garbage collects the topic. + +## Cascading topic deletions when modifying the replication clusters configuration -Given that geo-replication topics exist in multiple regions, directly deleting a geo-replication topic is not possible. Instead, you should rely on automatic topic garbage collection. +Tenant, namespace, and topic configurations can be shared or synchronized across clusters in two ways: -In Pulsar, a topic is automatically deleted when the topic meets the following three conditions: -- no producers or consumers are connected to it; -- no subscriptions to it; -- no more messages are kept for retention. -For geo-replication topics, each region uses a fault-tolerant mechanism to decide when deleting the topic locally is safe. +* **Shared configuration store**: Tenant and namespace metadata is stored in a shared configuration store accessible to all clusters. +* **`configurationMetadataSyncEventTopic`**: Tenant, namespace, and topic configuration changes are synchronized across clusters via geo-replication. The direction of synchronization follows the direction of replication — with 1-way replication, configuration updates flow in one direction only; with 2-way replication, they flow in both directions. -You can explicitly disable topic garbage collection by setting `brokerDeleteInactiveTopicsEnabled` to `false` in your [broker configuration](reference-configuration.md#broker). +Topic policies are also shared via geo-replication when the namespace has geo-replication enabled, with support for both local (single-cluster) and global (all-clusters) policies. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). -To delete a geo-replication topic, close all producers and consumers on the topic, and delete all of its local subscriptions in every replication cluster. When Pulsar determines that no valid subscription for the topic remains across the system, it will garbage collect the topic. +When the namespace `clusters` configuration is shared or synchronized across clusters and a cluster is removed from that list, Pulsar automatically deletes all topics for that namespace on the excluded cluster. + +When a topic-level `clusters` policy is shared or synchronized across clusters and a cluster is removed from that policy, Pulsar automatically deletes the topic and all its partitions on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), this cascading deletion is supported at the topic level in addition to the namespace level. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, policy updates flow only toward the destination cluster. + +:::warning + +When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. + +To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. + +::: + +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. When namespace or topic configuration is shared or synchronized across clusters, a misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ## Replicated subscriptions @@ -211,11 +280,17 @@ In case of failover, a consumer can restart consuming from the failure point in ### Enable replicated subscription +:::note + +Replicated subscriptions require [2-way geo-replication](#1-way-and-2-way-geo-replication) to be properly configured between all participating clusters. See [1-way and 2-way geo-replication](#1-way-and-2-way-geo-replication) for configuration requirements. + +::: + If you want to use replicated subscriptions in Pulsar: -- For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). +* For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). -- For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. +* For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. ```java Consumer consumer = client.newConsumer(Schema.STRING) @@ -239,14 +314,49 @@ The advantages of replicated subscription are as follows. The limitations of replicated subscription are as follows. * When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. -* Only the base line cursor position is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. +* Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. +* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may be processed in either cluster if the replication subscription update reaches the other cluster before the messages are consumed there. It is recommended to process messages in a single cluster when using replicated subscriptions. +* Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. :::note -* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions#storing-snapshots) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. +* Replicated subscriptions add a special snapshot message to the topic every second. If there are inactive subscriptions on the topic, this increases the backlog on both the source and destination clusters. The [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is only created when new messages have been produced to the topic since the last snapshot was taken. ::: +### Replicated subscriptions snapshot configuration and tuning + +Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). + +Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. + +A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. + +After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. + +The following broker settings control snapshot behavior: + +| Setting | Default | Description | +| --- | --- | --- | +| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | + +For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. + +### Observability + +Observability for replicated subscriptions is limited. For debugging, debug-level logs are available in `org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController`, though these are not suitable for production operations. + +The following broker-level metrics are available for monitoring snapshot health. Note that these metrics are aggregated across all topics on a broker and do not include per-topic labels. + +| Metric | OpenTelemetry name | Description | +| --- | --- | --- | +| `pulsar_replicated_subscriptions_pending_snapshots` | `pulsar.broker.replication.subscription.snapshot.operation.count` | Number of currently pending snapshots. | +| `pulsar_replicated_subscriptions_timedout_snapshots` | `pulsar.broker.replication.subscription.snapshot.operation.duration` | Number of snapshots that have timed out. | + +Topic stats and internal stats can be used to inspect the state of subscriptions. The cursor's mark-delete position is particularly useful, as subscription state can only be replicated up to that position. + ## Migrate data between clusters using geo-replication Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 3c6fd4c98fdf..fefa1eed6f19 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -22,7 +22,39 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) -Any message published on *any* topic in that namespace is replicated to all clusters in the specified set. +### Replication configuration settings + +The target clusters for replication of a message are determined by a hierarchy of settings at the tenant, namespace, topic, and message level: + +* **Tenant `allowed-clusters`**: Specifies which clusters the tenant is permitted to use for replication. An empty value means all clusters are allowed. Note that this setting cannot be modified once the tenant has existing namespaces, so it must be configured before creating any namespaces under the tenant. + +* **Namespace `clusters`**: Defines the default set of clusters that messages in the namespace are replicated to. + +* **Namespace `allowed-clusters`**: Further restricts which clusters are permitted for replication at the namespace level, overriding the tenant-level setting. Introduced in [PIP-321](https://github.com/apache/pulsar/blob/master/pip/pip-321.md). + +* **Topic-level policies**: Can override the namespace-level `clusters` setting for a specific topic. Topic policies can be **local** (applying only to the local cluster) or **global** (replicated to all clusters in the geo-replication set, see [PIP-92](https://github.com/apache/pulsar/blob/master/pip/pip-92.md)). Note that `allowed-clusters` has not been implemented at the topic level; [PIP-321](https://github.com/apache/pulsar/blob/master/pip/pip-321.md) mentions that it could be added in the future. + +* **Message-level replication control**: Producers can override which clusters a specific message is replicated to using the [`replicationClusters`](/api/client/4.1.x/org/apache/pulsar/client/api/TypedMessageBuilder.html#replicationClusters(java.util.List)) method in the client API, or disable replication entirely for a message using [`disableReplication`](/api/client/4.1.x/org/apache/pulsar/client/api/TypedMessageBuilder.html#disableReplication()) (see [Selective replication](#selective-replication)). Note that these settings cannot override the `allowed-clusters` configuration — messages can only be routed to clusters that are permitted by the resolved `allowed-clusters` settings. + +The `clusters` and `allowed-clusters` settings are resolved hierarchically. When the tenant-level `allowed-clusters` is non-empty, all clusters specified in namespace-level `allowed-clusters` must be a subset of it — this is validated when `allowed-clusters` is modified at the namespace level. Namespace-level `allowed-clusters` can further restrict the tenant-level configuration, and topic-level policies can override the namespace-level `clusters` setting for a specific topic. + +### 1-way and 2-way geo-replication + +Geo-replication can be configured as 1-way or 2-way: + +* **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). + +* **1-way replication**: Messages flow in one direction only (for example, `us-west` → `us-east`). Replication in the other direction can be prevented using two mechanisms: + * **`allowed-clusters`**: Forcefully prevents replication to clusters not in the resolved `allowed-clusters` configuration. This cannot be overridden by producers using the message-level `replicationClusters` setting. + * **`clusters`**: Sets the default replication targets. If the reverse direction cluster is not included, replication is 1-way by default, though producers can still override this per-message using the `replicationClusters` setting. + + :::note + The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. + ::: + +When using a **shared configuration store**, tenant-level and namespace-level configuration is shared across all clusters. If both clusters are listed in the shared namespace `clusters` setting and neither is excluded by `allowed-clusters`, replication is 2-way. + +When using **separate metadata stores** (no shared configuration store), each cluster has its own metadata store and you must configure geo-replication independently on each cluster. To achieve 2-way replication, both clusters must be configured so that geo-replication is enabled for the topic on both sides — the tenant's `allowed-clusters`, the namespace `clusters`, and the namespace `allowed-clusters` must all include both clusters on each cluster. ## Local persistence and forwarding @@ -189,19 +221,56 @@ bin/pulsar-admin topics stats persistent://my-tenant/my-namespace/my-topic Each cluster reports its own local stats, including the incoming and outgoing replication rates and backlogs. -#### Delete a geo-replication topic +#### Geo-replication topic deletion + +**Explicit topic deletion** + +The recommended procedure for deleting a geo-replication topic from all clusters is: + +1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. +2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. +4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. + +Without this procedure, forcefully deleting a topic on one cluster leaves it orphaned — it still exists on peer clusters and geo-replication from those clusters remains active. If auto-topic creation is enabled on the cluster where the topic was deleted, the topic may be recreated through auto-creation or because `createTopicToRemoteClusterForReplication=true` is set on a peer cluster. + +When namespace or topic configuration is shared via a shared configuration store or synchronized via `configurationMetadataSyncEventTopic`, forcefully deleting a topic propagates the deletion to all clusters that share or receive the configuration. However, replication delays may cause the topic to be recreated before the deletion takes effect everywhere. For this reason, it is recommended to follow the procedure above. + +To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. + +To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. + +**Deletion by garbage collection** + +A geo-replication topic is also automatically deleted by garbage collection when `brokerDeleteInactiveTopicsEnabled=true` and no producers or consumers are connected to it. The additional conditions depend on the `brokerDeleteInactiveTopicsMode` setting: + +- `delete_when_no_subscriptions`: the topic is deleted when there are no subscriptions. +- `delete_when_subscriptions_caught_up`: the topic is deleted when all subscriptions have caught up and there is no backlog. + +Each region independently decides when it is safe to delete the topic locally. To trigger garbage collection, close all producers and consumers on the topic and delete all local subscriptions in every replication cluster. When Pulsar determines that no valid subscription remains across the system, it garbage collects the topic. + +## Cascading topic deletions when modifying the replication clusters configuration -Given that geo-replication topics exist in multiple regions, directly deleting a geo-replication topic is not possible. Instead, you should rely on automatic topic garbage collection. +Tenant, namespace, and topic configurations can be shared or synchronized across clusters in two ways: -In Pulsar, a topic is automatically deleted when the topic meets the following three conditions: -- no producers or consumers are connected to it; -- no subscriptions to it; -- no more messages are kept for retention. -For geo-replication topics, each region uses a fault-tolerant mechanism to decide when deleting the topic locally is safe. +* **Shared configuration store**: Tenant and namespace metadata is stored in a shared configuration store accessible to all clusters. +* **`configurationMetadataSyncEventTopic`**: Tenant, namespace, and topic configuration changes are synchronized across clusters via geo-replication. The direction of synchronization follows the direction of replication — with 1-way replication, configuration updates flow in one direction only; with 2-way replication, they flow in both directions. -You can explicitly disable topic garbage collection by setting `brokerDeleteInactiveTopicsEnabled` to `false` in your [broker configuration](reference-configuration.md#broker). +Topic policies are also shared via geo-replication when the namespace has geo-replication enabled, with support for both local (single-cluster) and global (all-clusters) policies. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). -To delete a geo-replication topic, close all producers and consumers on the topic, and delete all of its local subscriptions in every replication cluster. When Pulsar determines that no valid subscription for the topic remains across the system, it will garbage collect the topic. +When the namespace `clusters` configuration is shared or synchronized across clusters and a cluster is removed from that list, Pulsar automatically deletes all topics for that namespace on the excluded cluster. + +When a topic-level `clusters` policy is shared or synchronized across clusters and a cluster is removed from that policy, Pulsar automatically deletes the topic and all its partitions on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), this cascading deletion is supported at the topic level in addition to the namespace level. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, policy updates flow only toward the destination cluster. + +:::warning + +When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. + +To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. + +::: + +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. When namespace or topic configuration is shared or synchronized across clusters, a misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ## Replicated subscriptions @@ -211,11 +280,17 @@ In case of failover, a consumer can restart consuming from the failure point in ### Enable replicated subscription +:::note + +Replicated subscriptions require [2-way geo-replication](#1-way-and-2-way-geo-replication) to be properly configured between all participating clusters. See [1-way and 2-way geo-replication](#1-way-and-2-way-geo-replication) for configuration requirements. + +::: + If you want to use replicated subscriptions in Pulsar: -- For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). +* For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). -- For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. +* For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. ```java Consumer consumer = client.newConsumer(Schema.STRING) @@ -239,14 +314,49 @@ The advantages of replicated subscription are as follows. The limitations of replicated subscription are as follows. * When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. -* Only the base line cursor position is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. +* Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. +* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may be processed in either cluster if the replication subscription update reaches the other cluster before the messages are consumed there. It is recommended to process messages in a single cluster when using replicated subscriptions. +* Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. :::note -* This replicated subscription will add a new special message every second, it will contains the [snapshot](https://github.com/apache/pulsar/wiki/PIP-33:-Replicated-subscriptions#storing-snapshots) of the subscription. That means, if there are inactive subscriptions over the topic there can be an increase in backlog in source and destination clusters. +* Replicated subscriptions add a special snapshot message to the topic every second. If there are inactive subscriptions on the topic, this increases the backlog on both the source and destination clusters. The [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is only created when new messages have been produced to the topic since the last snapshot was taken. ::: +### Replicated subscriptions snapshot configuration and tuning + +Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). + +Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. + +A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. + +After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. + +The following broker settings control snapshot behavior: + +| Setting | Default | Description | +| --- | --- | --- | +| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | + +For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. + +### Observability + +Observability for replicated subscriptions is limited. For debugging, debug-level logs are available in `org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController`, though these are not suitable for production operations. + +The following broker-level metrics are available for monitoring snapshot health. Note that these metrics are aggregated across all topics on a broker and do not include per-topic labels. + +| Metric | OpenTelemetry name | Description | +| --- | --- | --- | +| `pulsar_replicated_subscriptions_pending_snapshots` | `pulsar.broker.replication.subscription.snapshot.operation.count` | Number of currently pending snapshots. | +| `pulsar_replicated_subscriptions_timedout_snapshots` | `pulsar.broker.replication.subscription.snapshot.operation.duration` | Number of snapshots that have timed out. | + +Topic stats and internal stats can be used to inspect the state of subscriptions. The cursor's mark-delete position is particularly useful, as subscription state can only be replicated up to that position. + ## Migrate data between clusters using geo-replication Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. From b488eb5c217807c0f7c1f561344d22e6dfa17e8e Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 11:30:08 +0200 Subject: [PATCH 09/59] Update config sharing details --- docs/administration-geo.md | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index fefa1eed6f19..7b3a0687710f 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -10,7 +10,6 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ```` - ## Enable geo-replication for a namespace You must enable geo-replication on a [per-tenant basis](#concepts-multi-tenancy) in Pulsar. For example, you can enable geo-replication between two specific clusters only when a tenant has access to both clusters. @@ -22,6 +21,38 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) +### Sharing cluster configuration across geo-replicated clusters + +In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. The way this sharing is configured determines how configuration changes and deletions propagate between clusters, with important implications for replication behavior and topic lifecycle. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. + +Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. + +There are three approaches for sharing configuration: + +#### No shared or synchronized configuration (default) + +By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster manages its tenants, namespaces, partitioned topic metadata, and their policies independently. Geo-replication is configured on a per-namespace basis by creating the namespace and setting the same replication policies explicitly on each participating cluster. + +#### Shared configuration store + +Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that reference it share the same tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. + +#### Synchronized configuration via `configurationMetadataSyncEventTopic` + +When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. + +#### Creation of topic partitions in geo-replication + +Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. + +For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, it is important to ensure partitioned topic metadata exists on all clusters before producers connect, to avoid orphaned partitions being created without corresponding metadata. + +For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. + +#### Topic policies + +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the above approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). + ### Replication configuration settings The target clusters for replication of a message are determined by a hierarchy of settings at the tenant, namespace, topic, and message level: @@ -38,9 +69,9 @@ The target clusters for replication of a message are determined by a hierarchy o The `clusters` and `allowed-clusters` settings are resolved hierarchically. When the tenant-level `allowed-clusters` is non-empty, all clusters specified in namespace-level `allowed-clusters` must be a subset of it — this is validated when `allowed-clusters` is modified at the namespace level. Namespace-level `allowed-clusters` can further restrict the tenant-level configuration, and topic-level policies can override the namespace-level `clusters` setting for a specific topic. -### 1-way and 2-way geo-replication +### 1-way (unidirectional) and 2-way (bidirectional) geo-replication -Geo-replication can be configured as 1-way or 2-way: +Geo-replication can be configured as 1-way (unidirectional) or 2-way (bidirectional): * **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). From ce3495f8c03859ad825a9338c01fbace471e7144 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 11:32:49 +0200 Subject: [PATCH 10/59] Remove duplication in cascading topic deletions section --- docs/administration-geo.md | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 7b3a0687710f..0899c0e8bdd0 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -282,26 +282,19 @@ Each region independently decides when it is safe to delete the topic locally. T ## Cascading topic deletions when modifying the replication clusters configuration -Tenant, namespace, and topic configurations can be shared or synchronized across clusters in two ways: +When namespace configuration is shared or synchronized across clusters (see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), updating the namespace `clusters` configuration to remove a cluster automatically deletes all topics in that namespace on the excluded cluster. -* **Shared configuration store**: Tenant and namespace metadata is stored in a shared configuration store accessible to all clusters. -* **`configurationMetadataSyncEventTopic`**: Tenant, namespace, and topic configuration changes are synchronized across clusters via geo-replication. The direction of synchronization follows the direction of replication — with 1-way replication, configuration updates flow in one direction only; with 2-way replication, they flow in both directions. - -Topic policies are also shared via geo-replication when the namespace has geo-replication enabled, with support for both local (single-cluster) and global (all-clusters) policies. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). - -When the namespace `clusters` configuration is shared or synchronized across clusters and a cluster is removed from that list, Pulsar automatically deletes all topics for that namespace on the excluded cluster. - -When a topic-level `clusters` policy is shared or synchronized across clusters and a cluster is removed from that policy, Pulsar automatically deletes the topic and all its partitions on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), this cascading deletion is supported at the topic level in addition to the namespace level. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, policy updates flow only toward the destination cluster. +When a topic-level `clusters` policy is set to a subset of the replication clusters, the topic and all its partitions are automatically deleted on any cluster excluded. :::warning -When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. +Removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. Setting a topic-level `clusters` policy that excludes a cluster deletes that topic and all its partitions on the excluded cluster. To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. ::: -Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. When namespace or topic configuration is shared or synchronized across clusters, a misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ## Replicated subscriptions From 7ac49451ef043796ac4f4be27927dac0219dfef0 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 11:37:08 +0200 Subject: [PATCH 11/59] Polish --- docs/administration-geo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 0899c0e8bdd0..5a1a78afc464 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -41,7 +41,7 @@ Clusters can share a dedicated [configuration store](concepts-architecture-overv When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. -#### Creation of topic partitions in geo-replication +#### Creation of topics in geo-replication Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. @@ -51,7 +51,7 @@ For **non-partitioned topics**, topic auto-creation must be enabled at the broke #### Topic policies -Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the above approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). ### Replication configuration settings From eff65c2db37eaf68d94705f0ec0ab8f2bd1d231c Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 11:47:35 +0200 Subject: [PATCH 12/59] Fix details in 1-way / 2-way geo-replication configuration --- docs/administration-geo.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 5a1a78afc464..ea79691584c9 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -71,21 +71,25 @@ The `clusters` and `allowed-clusters` settings are resolved hierarchically. When ### 1-way (unidirectional) and 2-way (bidirectional) geo-replication -Geo-replication can be configured as 1-way (unidirectional) or 2-way (bidirectional): +Geo-replication can be configured as 1-way (unidirectional) or 2-way (bidirectional). The available options depend on whether a shared configuration store is used. -* **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). +#### Replication direction with a shared configuration store -* **1-way replication**: Messages flow in one direction only (for example, `us-west` → `us-east`). Replication in the other direction can be prevented using two mechanisms: - * **`allowed-clusters`**: Forcefully prevents replication to clusters not in the resolved `allowed-clusters` configuration. This cannot be overridden by producers using the message-level `replicationClusters` setting. - * **`clusters`**: Sets the default replication targets. If the reverse direction cluster is not included, replication is 1-way by default, though producers can still override this per-message using the `replicationClusters` setting. +When a shared configuration store is used, namespace configuration is shared across all clusters, so geo-replication is always 2-way at the namespace level. Individual topics can be made unidirectional by adding a local topic-level `clusters` policy on a specific cluster that includes only the local cluster. This prevents messages produced on that cluster from being replicated to others. However, since topic policies do not include `allowed-clusters`, producers can still override this using the message-level `replicationClusters` setting, so true enforcement of 1-way replication is not possible with a shared configuration store. - :::note - The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. - ::: +#### Replication direction with separate metadata stores -When using a **shared configuration store**, tenant-level and namespace-level configuration is shared across all clusters. If both clusters are listed in the shared namespace `clusters` setting and neither is excluded by `allowed-clusters`, replication is 2-way. +When each cluster uses its own metadata store, 1-way or 2-way replication is determined by the namespace `clusters` and `allowed-clusters` settings on each cluster: -When using **separate metadata stores** (no shared configuration store), each cluster has its own metadata store and you must configure geo-replication independently on each cluster. To achieve 2-way replication, both clusters must be configured so that geo-replication is enabled for the topic on both sides — the tenant's `allowed-clusters`, the namespace `clusters`, and the namespace `allowed-clusters` must all include both clusters on each cluster. +* **2-way replication**: Both clusters include each other in their namespace `clusters` and `allowed-clusters` settings, so messages flow in both directions. + +* **1-way replication**: On the cluster that should not replicate outbound, set both `clusters` and `allowed-clusters` to include only the local cluster. The remote cluster can still replicate inbound to this cluster if it is configured to do so. + * **`allowed-clusters`**: Forcefully prevents replication to any cluster not listed. This cannot be overridden by producers using the message-level `replicationClusters` setting. + * **`clusters`**: Sets the default replication targets. If only the local cluster is listed, outbound replication is disabled by default, though producers can still override this per-message using `replicationClusters` unless `allowed-clusters` also restricts it. + +:::note +The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. +::: ## Local persistence and forwarding From 6b5bb6eff80f0878cfcb38c3382bdd6dd52d9b73 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 11:51:48 +0200 Subject: [PATCH 13/59] Update cascading deletion details --- docs/administration-geo.md | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index ea79691584c9..d35b0677d4c6 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -286,19 +286,29 @@ Each region independently decides when it is safe to delete the topic locally. T ## Cascading topic deletions when modifying the replication clusters configuration -When namespace configuration is shared or synchronized across clusters (see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), updating the namespace `clusters` configuration to remove a cluster automatically deletes all topics in that namespace on the excluded cluster. +:::warning -When a topic-level `clusters` policy is set to a subset of the replication clusters, the topic and all its partitions are automatically deleted on any cluster excluded. +Modifying the `clusters` configuration at the namespace or topic level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. -:::warning +::: -Removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. Setting a topic-level `clusters` policy that excludes a cluster deletes that topic and all its partitions on the excluded cluster. +### Namespace-level deletions -To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. +When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. -::: +If the namespace-level `allowed-clusters` is modified to exclude a cluster, topics on that cluster are also deleted regardless of any topic-level `clusters` policy, since `clusters` must be a subset of `allowed-clusters`. + +### Topic-level deletions + +Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. + +### Retaining specific topics + +To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level `clusters` policy for that topic. Since it is not currently possible to override the namespace-level `allowed-clusters` in a topic policy, this protection does not apply if `allowed-clusters` is also changed to exclude a cluster. + +There is no single configuration that protects all topics in a namespace — the policy must be applied to each topic individually. For additional protection against the global topic-level policy itself being modified to exclude a cluster, a local topic-level `clusters` policy can also be set on that cluster to include only the local cluster. -Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. ## Replicated subscriptions From f128b769625ab192e9374d7595d8ace255b99eb6 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:24:27 +0200 Subject: [PATCH 14/59] Improve creation of topics --- docs/administration-geo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index d35b0677d4c6..ecde56cd9217 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -45,9 +45,9 @@ When a shared configuration store is not used, tenant, namespace, and partitione Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. -For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, it is important to ensure partitioned topic metadata exists on all clusters before producers connect, to avoid orphaned partitions being created without corresponding metadata. +For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. -For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. +For **non-partitioned topics**, there are two options: enable topic auto-creation at the broker level (the default) or in the namespace policy, or create the topic explicitly in each cluster. #### Topic policies From 84a7d603d238ccacd13115426a8f6160e0e104b7 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:36:24 +0200 Subject: [PATCH 15/59] Clarify --- docs/administration-geo.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index ecde56cd9217..d33ee9bc7f17 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -23,7 +23,9 @@ Complete the following tasks to enable geo-replication for a namespace: ### Sharing cluster configuration across geo-replicated clusters -In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. The way this sharing is configured determines how configuration changes and deletions propagate between clusters, with important implications for replication behavior and topic lifecycle. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. + +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. @@ -47,7 +49,7 @@ Topic partitions are local to each cluster and are not part of the configuration For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. -For **non-partitioned topics**, there are two options: enable topic auto-creation at the broker level (the default) or in the namespace policy, or create the topic explicitly in each cluster. +For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. #### Topic policies From 02bfabdd45f5cd9bdf2ba3bf3cac480af6815bc9 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:40:01 +0200 Subject: [PATCH 16/59] Add warning --- docs/administration-geo.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index d33ee9bc7f17..c224fc22a969 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -401,6 +401,12 @@ Topic stats and internal stats can be used to inspect the state of subscriptions Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. +:::warning + +Please note that replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. + +::: + 1. Create your new cluster. 2. Add the new cluster to your old cluster. From c146bad2ac42bcb23fec708383d8aaf80d4492c4 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:42:05 +0200 Subject: [PATCH 17/59] Sync 4.0.x and 4.1.x docs --- .../version-4.0.x/administration-geo.md | 92 ++++++++++++++----- .../version-4.1.x/administration-geo.md | 92 ++++++++++++++----- 2 files changed, 138 insertions(+), 46 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index fefa1eed6f19..c224fc22a969 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -10,7 +10,6 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ```` - ## Enable geo-replication for a namespace You must enable geo-replication on a [per-tenant basis](#concepts-multi-tenancy) in Pulsar. For example, you can enable geo-replication between two specific clusters only when a tenant has access to both clusters. @@ -22,6 +21,40 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) +### Sharing cluster configuration across geo-replicated clusters + +In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. + +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. + +Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. + +There are three approaches for sharing configuration: + +#### No shared or synchronized configuration (default) + +By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster manages its tenants, namespaces, partitioned topic metadata, and their policies independently. Geo-replication is configured on a per-namespace basis by creating the namespace and setting the same replication policies explicitly on each participating cluster. + +#### Shared configuration store + +Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that reference it share the same tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. + +#### Synchronized configuration via `configurationMetadataSyncEventTopic` + +When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. + +#### Creation of topics in geo-replication + +Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. + +For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. + +For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. + +#### Topic policies + +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). + ### Replication configuration settings The target clusters for replication of a message are determined by a hierarchy of settings at the tenant, namespace, topic, and message level: @@ -38,23 +71,27 @@ The target clusters for replication of a message are determined by a hierarchy o The `clusters` and `allowed-clusters` settings are resolved hierarchically. When the tenant-level `allowed-clusters` is non-empty, all clusters specified in namespace-level `allowed-clusters` must be a subset of it — this is validated when `allowed-clusters` is modified at the namespace level. Namespace-level `allowed-clusters` can further restrict the tenant-level configuration, and topic-level policies can override the namespace-level `clusters` setting for a specific topic. -### 1-way and 2-way geo-replication +### 1-way (unidirectional) and 2-way (bidirectional) geo-replication + +Geo-replication can be configured as 1-way (unidirectional) or 2-way (bidirectional). The available options depend on whether a shared configuration store is used. -Geo-replication can be configured as 1-way or 2-way: +#### Replication direction with a shared configuration store -* **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). +When a shared configuration store is used, namespace configuration is shared across all clusters, so geo-replication is always 2-way at the namespace level. Individual topics can be made unidirectional by adding a local topic-level `clusters` policy on a specific cluster that includes only the local cluster. This prevents messages produced on that cluster from being replicated to others. However, since topic policies do not include `allowed-clusters`, producers can still override this using the message-level `replicationClusters` setting, so true enforcement of 1-way replication is not possible with a shared configuration store. -* **1-way replication**: Messages flow in one direction only (for example, `us-west` → `us-east`). Replication in the other direction can be prevented using two mechanisms: - * **`allowed-clusters`**: Forcefully prevents replication to clusters not in the resolved `allowed-clusters` configuration. This cannot be overridden by producers using the message-level `replicationClusters` setting. - * **`clusters`**: Sets the default replication targets. If the reverse direction cluster is not included, replication is 1-way by default, though producers can still override this per-message using the `replicationClusters` setting. +#### Replication direction with separate metadata stores - :::note - The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. - ::: +When each cluster uses its own metadata store, 1-way or 2-way replication is determined by the namespace `clusters` and `allowed-clusters` settings on each cluster: -When using a **shared configuration store**, tenant-level and namespace-level configuration is shared across all clusters. If both clusters are listed in the shared namespace `clusters` setting and neither is excluded by `allowed-clusters`, replication is 2-way. +* **2-way replication**: Both clusters include each other in their namespace `clusters` and `allowed-clusters` settings, so messages flow in both directions. -When using **separate metadata stores** (no shared configuration store), each cluster has its own metadata store and you must configure geo-replication independently on each cluster. To achieve 2-way replication, both clusters must be configured so that geo-replication is enabled for the topic on both sides — the tenant's `allowed-clusters`, the namespace `clusters`, and the namespace `allowed-clusters` must all include both clusters on each cluster. +* **1-way replication**: On the cluster that should not replicate outbound, set both `clusters` and `allowed-clusters` to include only the local cluster. The remote cluster can still replicate inbound to this cluster if it is configured to do so. + * **`allowed-clusters`**: Forcefully prevents replication to any cluster not listed. This cannot be overridden by producers using the message-level `replicationClusters` setting. + * **`clusters`**: Sets the default replication targets. If only the local cluster is listed, outbound replication is disabled by default, though producers can still override this per-message using `replicationClusters` unless `allowed-clusters` also restricts it. + +:::note +The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. +::: ## Local persistence and forwarding @@ -251,26 +288,29 @@ Each region independently decides when it is safe to delete the topic locally. T ## Cascading topic deletions when modifying the replication clusters configuration -Tenant, namespace, and topic configurations can be shared or synchronized across clusters in two ways: +:::warning -* **Shared configuration store**: Tenant and namespace metadata is stored in a shared configuration store accessible to all clusters. -* **`configurationMetadataSyncEventTopic`**: Tenant, namespace, and topic configuration changes are synchronized across clusters via geo-replication. The direction of synchronization follows the direction of replication — with 1-way replication, configuration updates flow in one direction only; with 2-way replication, they flow in both directions. +Modifying the `clusters` configuration at the namespace or topic level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. -Topic policies are also shared via geo-replication when the namespace has geo-replication enabled, with support for both local (single-cluster) and global (all-clusters) policies. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). +::: -When the namespace `clusters` configuration is shared or synchronized across clusters and a cluster is removed from that list, Pulsar automatically deletes all topics for that namespace on the excluded cluster. +### Namespace-level deletions -When a topic-level `clusters` policy is shared or synchronized across clusters and a cluster is removed from that policy, Pulsar automatically deletes the topic and all its partitions on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), this cascading deletion is supported at the topic level in addition to the namespace level. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, policy updates flow only toward the destination cluster. +When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. -:::warning +If the namespace-level `allowed-clusters` is modified to exclude a cluster, topics on that cluster are also deleted regardless of any topic-level `clusters` policy, since `clusters` must be a subset of `allowed-clusters`. -When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. +### Topic-level deletions -To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. +Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. -::: +### Retaining specific topics + +To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level `clusters` policy for that topic. Since it is not currently possible to override the namespace-level `allowed-clusters` in a topic policy, this protection does not apply if `allowed-clusters` is also changed to exclude a cluster. -Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. When namespace or topic configuration is shared or synchronized across clusters, a misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +There is no single configuration that protects all topics in a namespace — the policy must be applied to each topic individually. For additional protection against the global topic-level policy itself being modified to exclude a cluster, a local topic-level `clusters` policy can also be set on that cluster to include only the local cluster. + +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. ## Replicated subscriptions @@ -361,6 +401,12 @@ Topic stats and internal stats can be used to inspect the state of subscriptions Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. +:::warning + +Please note that replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. + +::: + 1. Create your new cluster. 2. Add the new cluster to your old cluster. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index fefa1eed6f19..c224fc22a969 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -10,7 +10,6 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ```` - ## Enable geo-replication for a namespace You must enable geo-replication on a [per-tenant basis](#concepts-multi-tenancy) in Pulsar. For example, you can enable geo-replication between two specific clusters only when a tenant has access to both clusters. @@ -22,6 +21,40 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) +### Sharing cluster configuration across geo-replicated clusters + +In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. + +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. + +Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. + +There are three approaches for sharing configuration: + +#### No shared or synchronized configuration (default) + +By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster manages its tenants, namespaces, partitioned topic metadata, and their policies independently. Geo-replication is configured on a per-namespace basis by creating the namespace and setting the same replication policies explicitly on each participating cluster. + +#### Shared configuration store + +Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that reference it share the same tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. + +#### Synchronized configuration via `configurationMetadataSyncEventTopic` + +When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. + +#### Creation of topics in geo-replication + +Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. + +For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. + +For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. + +#### Topic policies + +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). + ### Replication configuration settings The target clusters for replication of a message are determined by a hierarchy of settings at the tenant, namespace, topic, and message level: @@ -38,23 +71,27 @@ The target clusters for replication of a message are determined by a hierarchy o The `clusters` and `allowed-clusters` settings are resolved hierarchically. When the tenant-level `allowed-clusters` is non-empty, all clusters specified in namespace-level `allowed-clusters` must be a subset of it — this is validated when `allowed-clusters` is modified at the namespace level. Namespace-level `allowed-clusters` can further restrict the tenant-level configuration, and topic-level policies can override the namespace-level `clusters` setting for a specific topic. -### 1-way and 2-way geo-replication +### 1-way (unidirectional) and 2-way (bidirectional) geo-replication + +Geo-replication can be configured as 1-way (unidirectional) or 2-way (bidirectional). The available options depend on whether a shared configuration store is used. -Geo-replication can be configured as 1-way or 2-way: +#### Replication direction with a shared configuration store -* **2-way replication**: Messages flow in both directions (`us-west` ↔ `us-east`). For 2-way replication to work, the resolved hierarchical `allowed-clusters` configuration must include both clusters (so neither is blocked from replicating to the other), and the resolved hierarchical `clusters` configuration must include both clusters (so messages are sent to both by default). +When a shared configuration store is used, namespace configuration is shared across all clusters, so geo-replication is always 2-way at the namespace level. Individual topics can be made unidirectional by adding a local topic-level `clusters` policy on a specific cluster that includes only the local cluster. This prevents messages produced on that cluster from being replicated to others. However, since topic policies do not include `allowed-clusters`, producers can still override this using the message-level `replicationClusters` setting, so true enforcement of 1-way replication is not possible with a shared configuration store. -* **1-way replication**: Messages flow in one direction only (for example, `us-west` → `us-east`). Replication in the other direction can be prevented using two mechanisms: - * **`allowed-clusters`**: Forcefully prevents replication to clusters not in the resolved `allowed-clusters` configuration. This cannot be overridden by producers using the message-level `replicationClusters` setting. - * **`clusters`**: Sets the default replication targets. If the reverse direction cluster is not included, replication is 1-way by default, though producers can still override this per-message using the `replicationClusters` setting. +#### Replication direction with separate metadata stores - :::note - The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. - ::: +When each cluster uses its own metadata store, 1-way or 2-way replication is determined by the namespace `clusters` and `allowed-clusters` settings on each cluster: -When using a **shared configuration store**, tenant-level and namespace-level configuration is shared across all clusters. If both clusters are listed in the shared namespace `clusters` setting and neither is excluded by `allowed-clusters`, replication is 2-way. +* **2-way replication**: Both clusters include each other in their namespace `clusters` and `allowed-clusters` settings, so messages flow in both directions. -When using **separate metadata stores** (no shared configuration store), each cluster has its own metadata store and you must configure geo-replication independently on each cluster. To achieve 2-way replication, both clusters must be configured so that geo-replication is enabled for the topic on both sides — the tenant's `allowed-clusters`, the namespace `clusters`, and the namespace `allowed-clusters` must all include both clusters on each cluster. +* **1-way replication**: On the cluster that should not replicate outbound, set both `clusters` and `allowed-clusters` to include only the local cluster. The remote cluster can still replicate inbound to this cluster if it is configured to do so. + * **`allowed-clusters`**: Forcefully prevents replication to any cluster not listed. This cannot be overridden by producers using the message-level `replicationClusters` setting. + * **`clusters`**: Sets the default replication targets. If only the local cluster is listed, outbound replication is disabled by default, though producers can still override this per-message using `replicationClusters` unless `allowed-clusters` also restricts it. + +:::note +The [replicated subscription](#replicated-subscriptions) feature requires 2-way geo-replication and is not available when geo-replication is configured as 1-way. +::: ## Local persistence and forwarding @@ -251,26 +288,29 @@ Each region independently decides when it is safe to delete the topic locally. T ## Cascading topic deletions when modifying the replication clusters configuration -Tenant, namespace, and topic configurations can be shared or synchronized across clusters in two ways: +:::warning -* **Shared configuration store**: Tenant and namespace metadata is stored in a shared configuration store accessible to all clusters. -* **`configurationMetadataSyncEventTopic`**: Tenant, namespace, and topic configuration changes are synchronized across clusters via geo-replication. The direction of synchronization follows the direction of replication — with 1-way replication, configuration updates flow in one direction only; with 2-way replication, they flow in both directions. +Modifying the `clusters` configuration at the namespace or topic level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. -Topic policies are also shared via geo-replication when the namespace has geo-replication enabled, with support for both local (single-cluster) and global (all-clusters) policies. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). +::: -When the namespace `clusters` configuration is shared or synchronized across clusters and a cluster is removed from that list, Pulsar automatically deletes all topics for that namespace on the excluded cluster. +### Namespace-level deletions -When a topic-level `clusters` policy is shared or synchronized across clusters and a cluster is removed from that policy, Pulsar automatically deletes the topic and all its partitions on the excluded cluster. Since [PIP-422](https://github.com/apache/pulsar/blob/master/pip/pip-422.md), this cascading deletion is supported at the topic level in addition to the namespace level. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, policy updates flow only toward the destination cluster. +When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. -:::warning +If the namespace-level `allowed-clusters` is modified to exclude a cluster, topics on that cluster are also deleted regardless of any topic-level `clusters` policy, since `clusters` must be a subset of `allowed-clusters`. -When the namespace `clusters` configuration is shared or synchronized across clusters, removing a cluster from the list automatically deletes all topics in that namespace on the excluded cluster. When a topic-level `clusters` policy is shared or synchronized, removing a cluster deletes that topic and all its partitions on the excluded cluster. +### Topic-level deletions -To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level policy for that topic and is the only way to stop replication for a specific topic without triggering deletions on other clusters when namespace configuration is shared or synchronized. +Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. -::: +### Retaining specific topics + +To prevent a specific topic from being deleted when the namespace `clusters` configuration changes, set a global topic-level `clusters` policy for that topic listing the clusters where it should be retained. This overrides the namespace-level `clusters` policy for that topic. Since it is not currently possible to override the namespace-level `allowed-clusters` in a topic policy, this protection does not apply if `allowed-clusters` is also changed to exclude a cluster. -Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. When namespace or topic configuration is shared or synchronized across clusters, a misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +There is no single configuration that protects all topics in a namespace — the policy must be applied to each topic individually. For additional protection against the global topic-level policy itself being modified to exclude a cluster, a local topic-level `clusters` policy can also be set on that cluster to include only the local cluster. + +Geo-replication is designed for high availability and disaster recovery, not as a substitute for backups. A misconfigured `clusters` policy can trigger cascading topic deletions on peer clusters. ## Replicated subscriptions @@ -361,6 +401,12 @@ Topic stats and internal stats can be used to inspect the state of subscriptions Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. +:::warning + +Please note that replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. + +::: + 1. Create your new cluster. 2. Add the new cluster to your old cluster. From b5b86b89da4dc870b1c75cf4cf5350128fe0f241 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:50:21 +0200 Subject: [PATCH 18/59] Clarify deletion --- docs/administration-geo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index c224fc22a969..cdcdb3551d99 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -267,7 +267,7 @@ The recommended procedure for deleting a geo-replication topic from all clusters 1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. 2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. 3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. -4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. +4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. If active producers or consumers are still present at this point, the topic may be recreated and geo-replication re-enabled, which is why step 1 is a prerequisite. Without this procedure, forcefully deleting a topic on one cluster leaves it orphaned — it still exists on peer clusters and geo-replication from those clusters remains active. If auto-topic creation is enabled on the cluster where the topic was deleted, the topic may be recreated through auto-creation or because `createTopicToRemoteClusterForReplication=true` is set on a peer cluster. From 38b1921f3e82277b8b2b41ae90a243f288d04bc8 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:50:40 +0200 Subject: [PATCH 19/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 +- versioned_docs/version-4.1.x/administration-geo.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index c224fc22a969..cdcdb3551d99 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -267,7 +267,7 @@ The recommended procedure for deleting a geo-replication topic from all clusters 1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. 2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. 3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. -4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. +4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. If active producers or consumers are still present at this point, the topic may be recreated and geo-replication re-enabled, which is why step 1 is a prerequisite. Without this procedure, forcefully deleting a topic on one cluster leaves it orphaned — it still exists on peer clusters and geo-replication from those clusters remains active. If auto-topic creation is enabled on the cluster where the topic was deleted, the topic may be recreated through auto-creation or because `createTopicToRemoteClusterForReplication=true` is set on a peer cluster. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index c224fc22a969..cdcdb3551d99 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -267,7 +267,7 @@ The recommended procedure for deleting a geo-replication topic from all clusters 1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. 2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. 3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. -4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. +4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. If active producers or consumers are still present at this point, the topic may be recreated and geo-replication re-enabled, which is why step 1 is a prerequisite. Without this procedure, forcefully deleting a topic on one cluster leaves it orphaned — it still exists on peer clusters and geo-replication from those clusters remains active. If auto-topic creation is enabled on the cluster where the topic was deleted, the topic may be recreated through auto-creation or because `createTopicToRemoteClusterForReplication=true` is set on a peer cluster. From 413a9858d82b49d517cd78e2475db4a971071300 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:56:30 +0200 Subject: [PATCH 20/59] Add missing detail --- docs/administration-geo.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index cdcdb3551d99..7447a0ca573e 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -284,6 +284,8 @@ A geo-replication topic is also automatically deleted by garbage collection when - `delete_when_no_subscriptions`: the topic is deleted when there are no subscriptions. - `delete_when_subscriptions_caught_up`: the topic is deleted when all subscriptions have caught up and there is no backlog. +The `brokerDeleteInactiveTopicsMode` setting can be overridden at the namespace level with the `inactive-topic-policies`. + Each region independently decides when it is safe to delete the topic locally. To trigger garbage collection, close all producers and consumers on the topic and delete all local subscriptions in every replication cluster. When Pulsar determines that no valid subscription remains across the system, it garbage collects the topic. ## Cascading topic deletions when modifying the replication clusters configuration From 33761ad2aa270ac93898fe02c26aa946b1dcdd7d Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 12:56:45 +0200 Subject: [PATCH 21/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 ++ versioned_docs/version-4.1.x/administration-geo.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index cdcdb3551d99..7447a0ca573e 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -284,6 +284,8 @@ A geo-replication topic is also automatically deleted by garbage collection when - `delete_when_no_subscriptions`: the topic is deleted when there are no subscriptions. - `delete_when_subscriptions_caught_up`: the topic is deleted when all subscriptions have caught up and there is no backlog. +The `brokerDeleteInactiveTopicsMode` setting can be overridden at the namespace level with the `inactive-topic-policies`. + Each region independently decides when it is safe to delete the topic locally. To trigger garbage collection, close all producers and consumers on the topic and delete all local subscriptions in every replication cluster. When Pulsar determines that no valid subscription remains across the system, it garbage collects the topic. ## Cascading topic deletions when modifying the replication clusters configuration diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index cdcdb3551d99..7447a0ca573e 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -284,6 +284,8 @@ A geo-replication topic is also automatically deleted by garbage collection when - `delete_when_no_subscriptions`: the topic is deleted when there are no subscriptions. - `delete_when_subscriptions_caught_up`: the topic is deleted when all subscriptions have caught up and there is no backlog. +The `brokerDeleteInactiveTopicsMode` setting can be overridden at the namespace level with the `inactive-topic-policies`. + Each region independently decides when it is safe to delete the topic locally. To trigger garbage collection, close all producers and consumers on the topic and delete all local subscriptions in every replication cluster. When Pulsar determines that no valid subscription remains across the system, it garbage collects the topic. ## Cascading topic deletions when modifying the replication clusters configuration From 6b36ac6c7fd47af3b9bd373c82180639776f7cd7 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:05:43 +0200 Subject: [PATCH 22/59] Update warnings --- docs/administration-geo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 7447a0ca573e..995a43971c68 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -292,7 +292,7 @@ Each region independently decides when it is safe to delete the topic locally. T :::warning -Modifying the `clusters` configuration at the namespace or topic level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +Modifying the `clusters` configuration at the namespace or topic policy level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ::: @@ -405,7 +405,7 @@ Using geo-replication to migrate data between clusters is a special use case of :::warning -Please note that replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +Replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. ::: From 2310a19d14f9f35653aaf89bc93208ae815bd72c Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:06:02 +0200 Subject: [PATCH 23/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 4 ++-- versioned_docs/version-4.1.x/administration-geo.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 7447a0ca573e..995a43971c68 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -292,7 +292,7 @@ Each region independently decides when it is safe to delete the topic locally. T :::warning -Modifying the `clusters` configuration at the namespace or topic level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +Modifying the `clusters` configuration at the namespace or topic policy level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ::: @@ -405,7 +405,7 @@ Using geo-replication to migrate data between clusters is a special use case of :::warning -Please note that replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +Replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. ::: diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 7447a0ca573e..995a43971c68 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -292,7 +292,7 @@ Each region independently decides when it is safe to delete the topic locally. T :::warning -Modifying the `clusters` configuration at the namespace or topic level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. +Modifying the `clusters` configuration at the namespace or topic policy level can automatically trigger topic deletions on excluded clusters. Always maintain independent backups if protection against accidental deletions is a requirement. ::: @@ -405,7 +405,7 @@ Using geo-replication to migrate data between clusters is a special use case of :::warning -Please note that replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +Replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. ::: From 4728f1e5897ce27b000bcf1d2c4d9f357e0d593f Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:07:44 +0200 Subject: [PATCH 24/59] Sync the other warning --- docs/administration-geo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 995a43971c68..fc1a85798bfb 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -25,7 +25,7 @@ Complete the following tasks to enable geo-replication for a namespace: In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. -The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. From d0f18056dd831347bbfe188e13ab6b557eca63ec Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:08:20 +0200 Subject: [PATCH 25/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 +- versioned_docs/version-4.1.x/administration-geo.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 995a43971c68..fc1a85798bfb 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -25,7 +25,7 @@ Complete the following tasks to enable geo-replication for a namespace: In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. -The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 995a43971c68..fc1a85798bfb 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -25,7 +25,7 @@ Complete the following tasks to enable geo-replication for a namespace: In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. -The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of keeping a snapshot of data — is not supported. Removing the cluster from the configuration will trigger cascading deletion of all topics in the namespace on that cluster. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. From e7a3d200d28a5ca6edb873de9b80e420dd68601f Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:21:35 +0200 Subject: [PATCH 26/59] Improve configuration store section --- docs/administration-geo.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index fc1a85798bfb..e3fa2beddb7d 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -21,27 +21,27 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) -### Sharing cluster configuration across geo-replicated clusters +### Configuration store and geo-replication setup -In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster. -The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +There are three approaches for managing the configuration store in a geo-replicated setup: -Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. +#### Independent configuration stores (default) -There are three approaches for sharing configuration: +By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster independently manages its cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, each participating cluster must be registered and the tenant and namespace must be created with the same replication policies on every cluster. -#### No shared or synchronized configuration (default) +#### Shared configuration store -By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster manages its tenants, namespaces, partitioned topic metadata, and their policies independently. Geo-replication is configured on a per-namespace basis by creating the namespace and setting the same replication policies explicitly on each participating cluster. +Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that use it share the same cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. -#### Shared configuration store +#### Configuration synchronization via `configurationMetadataSyncEventTopic` -Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that reference it share the same tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. +When independent configuration stores are used on each cluster, configuration store metadata can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. To bootstrap this setup, each participating cluster must be independently configured with its own cluster registration, plus a dedicated tenant and namespace to hold the sync topic. Once geo-replication is active for that namespace, configuration store metadata — including subsequent cluster registrations, tenants, namespaces, and their policies — is automatically synchronized across all participating clusters via that topic. -#### Synchronized configuration via `configurationMetadataSyncEventTopic` +#### Topic policies -When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). #### Creation of topics in geo-replication @@ -51,9 +51,9 @@ For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. -#### Topic policies +#### Cascading topic deletions -Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. ### Replication configuration settings From 6e0dec9dc496ae3a7cb85927d38b80557fadaa6b Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:22:03 +0200 Subject: [PATCH 27/59] Sync versioned docs --- .../version-4.0.x/administration-geo.md | 26 +++++++++---------- .../version-4.1.x/administration-geo.md | 26 +++++++++---------- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index fc1a85798bfb..e3fa2beddb7d 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -21,27 +21,27 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) -### Sharing cluster configuration across geo-replicated clusters +### Configuration store and geo-replication setup -In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster. -The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +There are three approaches for managing the configuration store in a geo-replicated setup: -Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. +#### Independent configuration stores (default) -There are three approaches for sharing configuration: +By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster independently manages its cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, each participating cluster must be registered and the tenant and namespace must be created with the same replication policies on every cluster. -#### No shared or synchronized configuration (default) +#### Shared configuration store -By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster manages its tenants, namespaces, partitioned topic metadata, and their policies independently. Geo-replication is configured on a per-namespace basis by creating the namespace and setting the same replication policies explicitly on each participating cluster. +Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that use it share the same cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. -#### Shared configuration store +#### Configuration synchronization via `configurationMetadataSyncEventTopic` -Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that reference it share the same tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. +When independent configuration stores are used on each cluster, configuration store metadata can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. To bootstrap this setup, each participating cluster must be independently configured with its own cluster registration, plus a dedicated tenant and namespace to hold the sync topic. Once geo-replication is active for that namespace, configuration store metadata — including subsequent cluster registrations, tenants, namespaces, and their policies — is automatically synchronized across all participating clusters via that topic. -#### Synchronized configuration via `configurationMetadataSyncEventTopic` +#### Topic policies -When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). #### Creation of topics in geo-replication @@ -51,9 +51,9 @@ For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. -#### Topic policies +#### Cascading topic deletions -Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. ### Replication configuration settings diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index fc1a85798bfb..e3fa2beddb7d 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -21,27 +21,27 @@ Complete the following tasks to enable geo-replication for a namespace: * [Enable a geo-replication namespace](#enable-geo-replication-at-namespace-level) * [Configure that namespace to replicate across two or more provisioned clusters](admin-api-namespaces.md#configure-replication-clusters) -### Sharing cluster configuration across geo-replicated clusters +### Configuration store and geo-replication setup -In a geo-replicated setup, tenants, namespaces, partitioned topic metadata, and their policies can be shared across clusters. +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster. -The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. Unintentional deletions due to misconfiguration can impact disaster recovery if topics are unexpectedly absent on a remote cluster. For example, replicating data to a remote cluster and then removing that cluster from the namespace replication configuration — with the intention of retaining an independent data snapshot — is not a supported use case. Removing the cluster from the configuration triggers cascading deletion of all topics on that cluster in certain cases, risking data loss when the goal is to keep an independent copy of the data. While it is possible to prevent cascading deletions, doing so has caveats. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +There are three approaches for managing the configuration store in a geo-replicated setup: -Note that the configuration store contains tenants, namespaces, their policies, and partitioned topic metadata (the topic name and partition count), but not the individual topic partitions themselves. Topic partitions and non-partitioned topics are local to each cluster. +#### Independent configuration stores (default) -There are three approaches for sharing configuration: +By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster independently manages its cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, each participating cluster must be registered and the tenant and namespace must be created with the same replication policies on every cluster. -#### No shared or synchronized configuration (default) +#### Shared configuration store -By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster manages its tenants, namespaces, partitioned topic metadata, and their policies independently. Geo-replication is configured on a per-namespace basis by creating the namespace and setting the same replication policies explicitly on each participating cluster. +Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that use it share the same cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. -#### Shared configuration store +#### Configuration synchronization via `configurationMetadataSyncEventTopic` -Clusters can share a dedicated [configuration store](concepts-architecture-overview.md#configuration-store) that is separate from each cluster's local metadata store. A shared configuration store is typically deployed across multiple regions or zones for fault tolerance. All clusters that reference it share the same tenants, namespaces, partitioned topic metadata, and their policies, so any change made on one cluster is immediately visible to all others. +When independent configuration stores are used on each cluster, configuration store metadata can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. To bootstrap this setup, each participating cluster must be independently configured with its own cluster registration, plus a dedicated tenant and namespace to hold the sync topic. Once geo-replication is active for that namespace, configuration store metadata — including subsequent cluster registrations, tenants, namespaces, and their policies — is automatically synchronized across all participating clusters via that topic. -#### Synchronized configuration via `configurationMetadataSyncEventTopic` +#### Topic policies -When a shared configuration store is not used, tenant, namespace, and partitioned topic configuration changes can still be synchronized across clusters using the `configurationMetadataSyncEventTopic` setting. When using this approach, geo-replication must be configured explicitly on every participating cluster for the namespace that holds the `configurationMetadataSyncEventTopic` topic. Once set up, configuration updates are synchronized across clusters via that topic. +Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). #### Creation of topics in geo-replication @@ -51,9 +51,9 @@ For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. -#### Topic policies +#### Cascading topic deletions -Topic policies are shared via geo-replication when the namespace has geo-replication enabled, regardless of which of the configuration store approaches is used. Both local (single-cluster) and global (all-clusters) policies are supported. Global topic policies apply to all clusters unless it has been overridden with a local topic policy in a specific cluster. Topic policies require `topicLevelPoliciesEnabled=true` in broker configuration (enabled by default). +The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. ### Replication configuration settings From bc6800de0d1e31be66105136b14be4e464d05993 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:24:04 +0200 Subject: [PATCH 28/59] Clarify --- docs/administration-geo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index e3fa2beddb7d..12d952246a76 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -23,7 +23,7 @@ Complete the following tasks to enable geo-replication for a namespace: ### Configuration store and geo-replication setup -Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster. +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and the metadata is stored in the [metadata store](concepts-architecture-overview.md#metadata-store). There are three approaches for managing the configuration store in a geo-replicated setup: From 16c049e2f744aca1861d5b573763c85122acc75f Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:24:25 +0200 Subject: [PATCH 29/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 +- versioned_docs/version-4.1.x/administration-geo.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index e3fa2beddb7d..12d952246a76 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -23,7 +23,7 @@ Complete the following tasks to enable geo-replication for a namespace: ### Configuration store and geo-replication setup -Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster. +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and the metadata is stored in the [metadata store](concepts-architecture-overview.md#metadata-store). There are three approaches for managing the configuration store in a geo-replicated setup: diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index e3fa2beddb7d..12d952246a76 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -23,7 +23,7 @@ Complete the following tasks to enable geo-replication for a namespace: ### Configuration store and geo-replication setup -Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster. +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and the metadata is stored in the [metadata store](concepts-architecture-overview.md#metadata-store). There are three approaches for managing the configuration store in a geo-replicated setup: From c3f11eb63c03254bea57a79d3024fcceda2cf081 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:29:49 +0200 Subject: [PATCH 30/59] Improve --- docs/administration-geo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 12d952246a76..9e16f8cb4b22 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -23,13 +23,13 @@ Complete the following tasks to enable geo-replication for a namespace: ### Configuration store and geo-replication setup -Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and the metadata is stored in the [metadata store](concepts-architecture-overview.md#metadata-store). +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and tracked in the cluster's [metadata store](concepts-architecture-overview.md#metadata-store). A shared configuration store is not required for geo-replication. There are three approaches for managing the configuration store in a geo-replicated setup: #### Independent configuration stores (default) -By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster independently manages its cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, each participating cluster must be registered and the tenant and namespace must be created with the same replication policies on every cluster. +By default, each Pulsar cluster uses its [metadata store](concepts-architecture-overview.md#metadata-store) as the [configuration store](concepts-architecture-overview.md#configuration-store) as well, so each cluster independently manages its own cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, all participating clusters must be registered in each other and the tenant and namespace must be created with matching replication policies on every cluster. #### Shared configuration store From 0eaced19a183cdeb078aa1cd879072610ffbf162 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:30:15 +0200 Subject: [PATCH 31/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 4 ++-- versioned_docs/version-4.1.x/administration-geo.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 12d952246a76..9e16f8cb4b22 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -23,13 +23,13 @@ Complete the following tasks to enable geo-replication for a namespace: ### Configuration store and geo-replication setup -Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and the metadata is stored in the [metadata store](concepts-architecture-overview.md#metadata-store). +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and tracked in the cluster's [metadata store](concepts-architecture-overview.md#metadata-store). A shared configuration store is not required for geo-replication. There are three approaches for managing the configuration store in a geo-replicated setup: #### Independent configuration stores (default) -By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster independently manages its cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, each participating cluster must be registered and the tenant and namespace must be created with the same replication policies on every cluster. +By default, each Pulsar cluster uses its [metadata store](concepts-architecture-overview.md#metadata-store) as the [configuration store](concepts-architecture-overview.md#configuration-store) as well, so each cluster independently manages its own cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, all participating clusters must be registered in each other and the tenant and namespace must be created with matching replication policies on every cluster. #### Shared configuration store diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 12d952246a76..9e16f8cb4b22 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -23,13 +23,13 @@ Complete the following tasks to enable geo-replication for a namespace: ### Configuration store and geo-replication setup -Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and the metadata is stored in the [metadata store](concepts-architecture-overview.md#metadata-store). +Geo-replication setup — including cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies — is stored in the [configuration store](concepts-architecture-overview.md#configuration-store). Individual topic partitions and non-partitioned topics are not part of the configuration store; they are local to each cluster and tracked in the cluster's [metadata store](concepts-architecture-overview.md#metadata-store). A shared configuration store is not required for geo-replication. There are three approaches for managing the configuration store in a geo-replicated setup: #### Independent configuration stores (default) -By default, each Pulsar cluster uses its own [metadata store](concepts-architecture-overview.md#metadata-store) for both local cluster state and configuration data. Each cluster independently manages its cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, each participating cluster must be registered and the tenant and namespace must be created with the same replication policies on every cluster. +By default, each Pulsar cluster uses its [metadata store](concepts-architecture-overview.md#metadata-store) as the [configuration store](concepts-architecture-overview.md#configuration-store) as well, so each cluster independently manages its own cluster registrations, tenants, namespaces, partitioned topic metadata, and their policies. To set up geo-replication, all participating clusters must be registered in each other and the tenant and namespace must be created with matching replication policies on every cluster. #### Shared configuration store From 59ebe9e45ab592347cd2777eb861e467246a168d Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:34:45 +0200 Subject: [PATCH 32/59] Improve --- docs/administration-geo.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 9e16f8cb4b22..1eaabebf3c56 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -45,12 +45,10 @@ Topic policies are shared via geo-replication when the namespace has geo-replica #### Creation of topics in geo-replication -Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. - -For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. - For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. +For **partitioned topics**, note that partitioned topic metadata (the topic name and partition count) is stored in the configuration store, while the individual topic partitions themselves are local to each cluster. When `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. + #### Cascading topic deletions The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. From 3d5d484990dc88038bda3908328b7b235b26240c Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:34:52 +0200 Subject: [PATCH 33/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 6 ++---- versioned_docs/version-4.1.x/administration-geo.md | 6 ++---- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 9e16f8cb4b22..1eaabebf3c56 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -45,12 +45,10 @@ Topic policies are shared via geo-replication when the namespace has geo-replica #### Creation of topics in geo-replication -Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. - -For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. - For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. +For **partitioned topics**, note that partitioned topic metadata (the topic name and partition count) is stored in the configuration store, while the individual topic partitions themselves are local to each cluster. When `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. + #### Cascading topic deletions The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 9e16f8cb4b22..1eaabebf3c56 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -45,12 +45,10 @@ Topic policies are shared via geo-replication when the namespace has geo-replica #### Creation of topics in geo-replication -Topic partitions are local to each cluster and are not part of the configuration store. The following applies regardless of which configuration sharing approach is used. - -For **partitioned topics**, when `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. - For **non-partitioned topics**, topic auto-creation must be enabled at the broker level (the default) or in the namespace policy, or the topic must be created explicitly in each cluster. +For **partitioned topics**, note that partitioned topic metadata (the topic name and partition count) is stored in the configuration store, while the individual topic partitions themselves are local to each cluster. When `createTopicToRemoteClusterForReplication=true` (the default), it is sufficient to create the topic in a single cluster — Pulsar automatically creates the matching partitioned topic metadata in remote clusters. Despite its name, this setting applies only to partitioned topics. If `createTopicToRemoteClusterForReplication` is disabled and clusters do not share a configuration store, partitioned topics must be created explicitly in each cluster. In that case, partitioned topic metadata must exist on all clusters before any clients connect. If it is missing, a consumer may auto-create a non-partitioned topic on the cluster lacking the metadata, resulting in incompatible topic types across clusters. Additionally, replication may create individual topic partitions on the target cluster without the corresponding partitioned topic metadata, leaving those partitions orphaned. For this reason, keeping `createTopicToRemoteClusterForReplication` enabled is recommended. + #### Cascading topic deletions The configuration approach also determines how changes to the replication clusters configuration propagate between clusters. In particular, certain configuration changes can trigger automatic topic deletions on remote clusters. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. From c60135e285036ed2ff67c3b759201abba494310a Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:45:09 +0200 Subject: [PATCH 34/59] Improve --- docs/administration-geo.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 1eaabebf3c56..83399da6423f 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -263,7 +263,7 @@ Each cluster reports its own local stats, including the incoming and outgoing re The recommended procedure for deleting a geo-replication topic from all clusters is: 1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. -2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic (including topic partitions in the case of a partitioned topic) and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. 3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. 4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. If active producers or consumers are still present at this point, the topic may be recreated and geo-replication re-enabled, which is why step 1 is a prerequisite. @@ -271,7 +271,7 @@ Without this procedure, forcefully deleting a topic on one cluster leaves it orp When namespace or topic configuration is shared via a shared configuration store or synchronized via `configurationMetadataSyncEventTopic`, forcefully deleting a topic propagates the deletion to all clusters that share or receive the configuration. However, replication delays may cause the topic to be recreated before the deletion takes effect everywhere. For this reason, it is recommended to follow the procedure above. -To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. +To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic (including topic partitions in the case of a partitioned topic) on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. @@ -296,13 +296,13 @@ Modifying the `clusters` configuration at the namespace or topic policy level ca ### Namespace-level deletions -When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. +When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Configuration store and geo-replication setup](#configuration-store-and-geo-replication-setup)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. If the namespace-level `allowed-clusters` is modified to exclude a cluster, topics on that cluster are also deleted regardless of any topic-level `clusters` policy, since `clusters` must be a subset of `allowed-clusters`. ### Topic-level deletions -Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. +Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last topic partition is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. ### Retaining specific topics From 9589661564709c4c41de00d4e31b3accce162ee3 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 13:45:15 +0200 Subject: [PATCH 35/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 8 ++++---- versioned_docs/version-4.1.x/administration-geo.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 1eaabebf3c56..83399da6423f 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -263,7 +263,7 @@ Each cluster reports its own local stats, including the incoming and outgoing re The recommended procedure for deleting a geo-replication topic from all clusters is: 1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. -2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic (including topic partitions in the case of a partitioned topic) and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. 3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. 4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. If active producers or consumers are still present at this point, the topic may be recreated and geo-replication re-enabled, which is why step 1 is a prerequisite. @@ -271,7 +271,7 @@ Without this procedure, forcefully deleting a topic on one cluster leaves it orp When namespace or topic configuration is shared via a shared configuration store or synchronized via `configurationMetadataSyncEventTopic`, forcefully deleting a topic propagates the deletion to all clusters that share or receive the configuration. However, replication delays may cause the topic to be recreated before the deletion takes effect everywhere. For this reason, it is recommended to follow the procedure above. -To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. +To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic (including topic partitions in the case of a partitioned topic) on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. @@ -296,13 +296,13 @@ Modifying the `clusters` configuration at the namespace or topic policy level ca ### Namespace-level deletions -When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. +When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Configuration store and geo-replication setup](#configuration-store-and-geo-replication-setup)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. If the namespace-level `allowed-clusters` is modified to exclude a cluster, topics on that cluster are also deleted regardless of any topic-level `clusters` policy, since `clusters` must be a subset of `allowed-clusters`. ### Topic-level deletions -Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. +Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last topic partition is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. ### Retaining specific topics diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 1eaabebf3c56..83399da6423f 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -263,7 +263,7 @@ Each cluster reports its own local stats, including the incoming and outgoing re The recommended procedure for deleting a geo-replication topic from all clusters is: 1. Ensure there are no active producers or consumers on the topic across all clusters before proceeding. If any are present when the next step is performed, the topic will be forcefully deleted from under them. If auto-topic creation is also enabled, the topic may be immediately recreated. -2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic's sub-topics and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. +2. Set a global topic-level `clusters` policy to include only the local cluster. This triggers the cascading deletion mechanism to remove the topic (including topic partitions in the case of a partitioned topic) and clean up schemas and local topic policies on all excluded clusters. Producers and consumers connected to an excluded cluster will be rejected from reconnecting. See [Cascading topic deletions when modifying the replication clusters configuration](#cascading-topic-deletions-when-modifying-the-replication-clusters-configuration) for details. 3. Delete the topic. Geo-replication is now disabled, so the deletion only affects the local cluster. 4. Run `pulsar-admin topicPolicies delete ` on each cluster to remove the remaining topic-level policy state. If active producers or consumers are still present at this point, the topic may be recreated and geo-replication re-enabled, which is why step 1 is a prerequisite. @@ -271,7 +271,7 @@ Without this procedure, forcefully deleting a topic on one cluster leaves it orp When namespace or topic configuration is shared via a shared configuration store or synchronized via `configurationMetadataSyncEventTopic`, forcefully deleting a topic propagates the deletion to all clusters that share or receive the configuration. However, replication delays may cause the topic to be recreated before the deletion takes effect everywhere. For this reason, it is recommended to follow the procedure above. -To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic's sub-topics on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. +To remove a topic from a specific cluster only, set a global topic-level `clusters` policy that excludes that cluster. The broker automatically deletes the topic (including topic partitions in the case of a partitioned topic) on the excluded cluster. Do not remove the global topic-level policy afterward, as this would allow the namespace-level `clusters` policy to take effect and potentially re-enable replication. To later delete the topic from all clusters, follow the full procedure above. To retain the topic in a specific cluster while removing it from all others, follow the procedure above on the cluster where the topic should be retained, but omit steps 3 and 4. @@ -296,13 +296,13 @@ Modifying the `clusters` configuration at the namespace or topic policy level ca ### Namespace-level deletions -When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Sharing cluster configuration across geo-replicated clusters](#sharing-cluster-configuration-across-geo-replicated-clusters)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. +When namespace configuration is shared or synchronized across clusters (via a shared configuration store or `configurationMetadataSyncEventTopic`, see [Configuration store and geo-replication setup](#configuration-store-and-geo-replication-setup)), removing a cluster from the namespace `clusters` configuration automatically deletes all topics in that namespace on the excluded cluster. When namespace configuration is not shared or synchronized, namespace-level policy changes remain local and do not trigger cascading deletions on remote clusters. If the namespace-level `allowed-clusters` is modified to exclude a cluster, topics on that cluster are also deleted regardless of any topic-level `clusters` policy, since `clusters` must be a subset of `allowed-clusters`. ### Topic-level deletions -Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last sub-topic is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. +Since topic policies are always shared via geo-replication when the namespace has geo-replication enabled (see [Topic policies](#topic-policies)), updating a global topic-level `clusters` policy to exclude a cluster deletes that topic and all its partitions on the excluded cluster, regardless of whether namespace configuration is shared or synchronized. Schemas and local topic policies are cleaned up after the last topic partition is deleted. Topic-level policy updates follow the replication direction of the namespace — with 1-way replication, updates flow only toward the destination cluster. ### Retaining specific topics From 147399c779292fb693eb778fb67e011b516e6c8e Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 14:57:34 +0200 Subject: [PATCH 36/59] Add mermaidjs support --- docusaurus.config.ts | 4 +- package.json | 3 +- yarn.lock | 1361 ++++++++++++++++++++++++++++++++++++++---- 3 files changed, 1262 insertions(+), 106 deletions(-) diff --git a/docusaurus.config.ts b/docusaurus.config.ts index c817ee4429b9..36e8b41529f8 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -134,6 +134,7 @@ module.exports = async function createConfigAsync() { projectName: "pulsar", trailingSlash: true, markdown: { + mermaid: true, preprocessor: ({ filePath, fileContent }) => { return fileContent.replaceAll(/{@inject:([^}]+)}/g, (_, p1) => { const p1Trimmed = p1.trim(); @@ -150,6 +151,7 @@ module.exports = async function createConfigAsync() { }); }, }, + themes: ['@docusaurus/theme-mermaid'], themeConfig: /** @type {import('@docusaurus/preset-classic').ThemeConfig} */ ({ @@ -168,7 +170,7 @@ module.exports = async function createConfigAsync() { disableSwitch: true, }, zoom: { - selector: '.markdown img', + selector: '.markdown img, .markdown svg', background: { light: '#fff', dark: '#111' diff --git a/package.json b/package.json index c7bce8d1678b..398aa686dd4a 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,7 @@ "@docusaurus/plugin-google-analytics": "3.8.1", "@docusaurus/preset-classic": "3.8.1", "@docusaurus/theme-classic": "^3.8.1", + "@docusaurus/theme-mermaid": "3.8.1", "@emotion/react": "^11.13.3", "@emotion/styled": "^11.13.0", "@mdx-js/react": "^3.0.0", @@ -35,7 +36,7 @@ "chokidar": "^4.0.1", "clsx": "^2.1.1", "compare-versions": "4.1.3", - "docusaurus-plugin-image-zoom": "^2.0.0", + "docusaurus-plugin-image-zoom": "^3.0.1", "execa": "^9.4.1", "file-loader": "^6.2.0", "font-awesome": "^4.7.0", diff --git a/yarn.lock b/yarn.lock index 493efaa60e7d..09eba2aff14c 100644 --- a/yarn.lock +++ b/yarn.lock @@ -361,6 +361,16 @@ __metadata: languageName: node linkType: hard +"@antfu/install-pkg@npm:^1.1.0": + version: 1.1.0 + resolution: "@antfu/install-pkg@npm:1.1.0" + dependencies: + package-manager-detector: ^1.3.0 + tinyexec: ^1.0.1 + checksum: e20b7cd1c37eff832cc878cddd794f8c3779175681cf6d75c4cc1ae1475526126a4c1f71fa027161aa1ee35a8850782be9ca0ec01b621893defebe97ba9dc70e + languageName: node + linkType: hard + "@babel/code-frame@npm:^7.0.0, @babel/code-frame@npm:^7.25.9, @babel/code-frame@npm:^7.26.0, @babel/code-frame@npm:^7.26.2": version: 7.26.2 resolution: "@babel/code-frame@npm:7.26.2" @@ -1665,6 +1675,55 @@ __metadata: languageName: node linkType: hard +"@braintree/sanitize-url@npm:^7.1.1": + version: 7.1.2 + resolution: "@braintree/sanitize-url@npm:7.1.2" + checksum: ca787264bbea2a3c02ce5cd5e08a1debf1ffff1c87b954ed2522d816fc3f3a9b93a5a55056dfe394d86c780ced35ee2fc06abbffd0f5f2e95264f7b62cb29d66 + languageName: node + linkType: hard + +"@chevrotain/cst-dts-gen@npm:11.1.2": + version: 11.1.2 + resolution: "@chevrotain/cst-dts-gen@npm:11.1.2" + dependencies: + "@chevrotain/gast": 11.1.2 + "@chevrotain/types": 11.1.2 + lodash-es: 4.17.23 + checksum: fb09d03de5a8c0339cfbdcb5487f244f332fe4bf4362911712e292594f73b50b7d1157d74674a8835d0225f57050fa8b359ea3e775a45ddb071ba5c321664a0f + languageName: node + linkType: hard + +"@chevrotain/gast@npm:11.1.2": + version: 11.1.2 + resolution: "@chevrotain/gast@npm:11.1.2" + dependencies: + "@chevrotain/types": 11.1.2 + lodash-es: 4.17.23 + checksum: 62e9869886deb86e4b6f8379c3d289918605a61cfd1f837a1c31591f813fecda4f718d403f1791c177e291602e01a62390e20bf95a54f5e90170e7fbf5fe7cbb + languageName: node + linkType: hard + +"@chevrotain/regexp-to-ast@npm:11.1.2": + version: 11.1.2 + resolution: "@chevrotain/regexp-to-ast@npm:11.1.2" + checksum: ddf9c8b5cd2f390f154642aa9eafbd968a3ee3c2573ac47285e002ed4561191ed7acebd9481b1492359e4d7a6c7a00e073f7f7a1a4c844042b9456e07daabaa7 + languageName: node + linkType: hard + +"@chevrotain/types@npm:11.1.2": + version: 11.1.2 + resolution: "@chevrotain/types@npm:11.1.2" + checksum: 4c948b8559c94329bae5fa5b087e20ebfbac3fa73ff32ee7e3752716ab565da1c9efc2103eb1dbee2bf492d68231d4386836283f0bb7cca63f4185788808af70 + languageName: node + linkType: hard + +"@chevrotain/utils@npm:11.1.2": + version: 11.1.2 + resolution: "@chevrotain/utils@npm:11.1.2" + checksum: 88f2a75ec80d4a8b0ebaf53d99214f69c797ffa3895003a5ac61624dfcb412e753327edb5203cd3bae02d824843b374ba74e821273d133b183f34ac7d0cfe629 + languageName: node + linkType: hard + "@colors/colors@npm:1.5.0": version: 1.5.0 resolution: "@colors/colors@npm:1.5.0" @@ -3126,6 +3185,24 @@ __metadata: languageName: node linkType: hard +"@docusaurus/theme-mermaid@npm:3.8.1": + version: 3.8.1 + resolution: "@docusaurus/theme-mermaid@npm:3.8.1" + dependencies: + "@docusaurus/core": 3.8.1 + "@docusaurus/module-type-aliases": 3.8.1 + "@docusaurus/theme-common": 3.8.1 + "@docusaurus/types": 3.8.1 + "@docusaurus/utils-validation": 3.8.1 + mermaid: ">=11.6.0" + tslib: ^2.6.0 + peerDependencies: + react: ^18.0.0 || ^19.0.0 + react-dom: ^18.0.0 || ^19.0.0 + checksum: f596e30d7c62e2840d38363aa03f79e0d1c252a16236b4976434331243f2c99b49ccbfc83cecb36f8cb8fa6c4e14365e236e509d434dddcc1085d6136f4a7a0d + languageName: node + linkType: hard + "@docusaurus/theme-search-algolia@npm:3.8.1": version: 3.8.1 resolution: "@docusaurus/theme-search-algolia@npm:3.8.1" @@ -3445,6 +3522,24 @@ __metadata: languageName: node linkType: hard +"@iconify/types@npm:^2.0.0": + version: 2.0.0 + resolution: "@iconify/types@npm:2.0.0" + checksum: 029f58542c160e9d4a746869cf2e475b603424d3adf3994c5cc8d0406c47e6e04a3b898b2707840c1c5b9bd5563a1660a34b110d89fce43923baca5222f4e597 + languageName: node + linkType: hard + +"@iconify/utils@npm:^3.0.1": + version: 3.1.0 + resolution: "@iconify/utils@npm:3.1.0" + dependencies: + "@antfu/install-pkg": ^1.1.0 + "@iconify/types": ^2.0.0 + mlly: ^1.8.0 + checksum: b4055a322a13289740b2ecef424a1807ccb1567a200b716e4d1e2f40ad24dd9e24fa7b9a1aa1a275eea30ef3f08a32a4640a1a66f013d32cfe31117ac76b4075 + languageName: node + linkType: hard + "@isaacs/cliui@npm:^8.0.2": version: 8.0.2 resolution: "@isaacs/cliui@npm:8.0.2" @@ -3611,6 +3706,15 @@ __metadata: languageName: node linkType: hard +"@mermaid-js/parser@npm:^1.0.0": + version: 1.0.0 + resolution: "@mermaid-js/parser@npm:1.0.0" + dependencies: + langium: ^4.0.0 + checksum: 5d12454adf3d38fcb41e08da09d6147d600bf109512294f1448a593402e6c38b0f8ed6040ef49941bbe4d868aaa8ffa0d6d09c41ba76930d737211ce0e553460 + languageName: node + linkType: hard + "@mui/base@npm:^5.0.0-beta.60": version: 5.0.0-dev.20240529-082515-213b5e33ab resolution: "@mui/base@npm:5.0.0-dev.20240529-082515-213b5e33ab" @@ -4512,6 +4616,278 @@ __metadata: languageName: node linkType: hard +"@types/d3-array@npm:*": + version: 3.2.2 + resolution: "@types/d3-array@npm:3.2.2" + checksum: 72e8e2abe0911cb431d6f3fe0a1f71b915356b679d4d9c826f52941bb30210c0fe8299dde066b08d9986754c620f031b13b13ab6dfc60d404eceab66a075dd5d + languageName: node + linkType: hard + +"@types/d3-axis@npm:*": + version: 3.0.6 + resolution: "@types/d3-axis@npm:3.0.6" + dependencies: + "@types/d3-selection": "*" + checksum: ea1065d9e6d134c04427763603cbe9d549b8b5785b8ae0d002b5b14a362619d5b8f5ee3c2fda8b36b7e5a413cbcd387e1a2d89898b919a9f0cc91ad4e67b5ab5 + languageName: node + linkType: hard + +"@types/d3-brush@npm:*": + version: 3.0.6 + resolution: "@types/d3-brush@npm:3.0.6" + dependencies: + "@types/d3-selection": "*" + checksum: e5166bc53e5c914b1fed0a6ce55ca14d76ae11c5afd16b724b8ae47989e977c4af02bb07496d1ccd0a77f4ccd9a2ca7345e1d289bcfce16490fe4b39a9e0d170 + languageName: node + linkType: hard + +"@types/d3-chord@npm:*": + version: 3.0.6 + resolution: "@types/d3-chord@npm:3.0.6" + checksum: b511cf372ed8a0086d37a715c0d4aca811b614454e1f7c1561fbcd46863beaccdb115d274a7a992a30a8218393fbc3e1fdd7ca6e9d572e729a4570002c327083 + languageName: node + linkType: hard + +"@types/d3-color@npm:*": + version: 3.1.3 + resolution: "@types/d3-color@npm:3.1.3" + checksum: 8a0e79a709929502ec4effcee2c786465b9aec51b653ba0b5d05dbfec3e84f418270dd603002d94021885061ff592f614979193bd7a02ad76317f5608560e357 + languageName: node + linkType: hard + +"@types/d3-contour@npm:*": + version: 3.0.6 + resolution: "@types/d3-contour@npm:3.0.6" + dependencies: + "@types/d3-array": "*" + "@types/geojson": "*" + checksum: 83c13eb0567e95d6675d6d81cbeab38d0899c5af70a7c69354e23e0860ddb2f3e911d2cacd33a8baa60ce7846b38785a337b2d7c8d2763a1340bfb999b4bd2ab + languageName: node + linkType: hard + +"@types/d3-delaunay@npm:*": + version: 6.0.4 + resolution: "@types/d3-delaunay@npm:6.0.4" + checksum: 502fe0eb91f7d05b0f57904d68028c24348a54b1e5458009caf662de995d0e59bd82cd701b4af0087d614ee9e456d415fe32d63c25272ca753bf12b3f27b2d77 + languageName: node + linkType: hard + +"@types/d3-dispatch@npm:*": + version: 3.0.7 + resolution: "@types/d3-dispatch@npm:3.0.7" + checksum: ce7ab5a7d5c64aacf563797c0c61f3862b9ff687cb35470fe462219f09e402185646f51707339beede616586d92ded6974c3958dbeb15e35a85b1ecfafdf13a8 + languageName: node + linkType: hard + +"@types/d3-drag@npm:*": + version: 3.0.7 + resolution: "@types/d3-drag@npm:3.0.7" + dependencies: + "@types/d3-selection": "*" + checksum: 1107cb1667ead79073741c06ea4a9e8e4551698f6c9c60821e327a6aa30ca2ba0b31a6fe767af85a2e38a22d2305f6c45b714df15c2bba68adf58978223a5fc5 + languageName: node + linkType: hard + +"@types/d3-dsv@npm:*": + version: 3.0.7 + resolution: "@types/d3-dsv@npm:3.0.7" + checksum: 5025e01459827d09d14e0d00281995a04042ce9e3e76444c5a65466c1d29649d82cbfaa9251e33837bf576f5c587525d8d8ff5aacc6bd3b831824d54449261b9 + languageName: node + linkType: hard + +"@types/d3-ease@npm:*": + version: 3.0.2 + resolution: "@types/d3-ease@npm:3.0.2" + checksum: 0885219966294bfc99548f37297e1c75e75da812a5f3ec941977ebb57dcab0a25acec5b2bbd82d09a49d387daafca08521ca269b7e4c27ddca7768189e987b54 + languageName: node + linkType: hard + +"@types/d3-fetch@npm:*": + version: 3.0.7 + resolution: "@types/d3-fetch@npm:3.0.7" + dependencies: + "@types/d3-dsv": "*" + checksum: e60cf60b25cbc49b2066ac2a3638f610c7379000562b0f499dd90fd57a8cb9740c24667a70496c2a66456d42867afeffb1722a75b26d95e7d7ee8667d96b0b36 + languageName: node + linkType: hard + +"@types/d3-force@npm:*": + version: 3.0.10 + resolution: "@types/d3-force@npm:3.0.10" + checksum: 0faf1321ddd85f7bf25769ee97513b380a897791ad1cd6c4282f09e0108e566132fad80f4c73cdb592a352139b22388d3c77458298a00f92ef72e27019fb33c7 + languageName: node + linkType: hard + +"@types/d3-format@npm:*": + version: 3.0.4 + resolution: "@types/d3-format@npm:3.0.4" + checksum: e69421cd93861a0c080084b0b23d4a5d6a427497559e46898189002fb756dae2c7c858b465308f6bcede7272b90e39ce8adab810bded2309035a5d9556c59134 + languageName: node + linkType: hard + +"@types/d3-geo@npm:*": + version: 3.1.0 + resolution: "@types/d3-geo@npm:3.1.0" + dependencies: + "@types/geojson": "*" + checksum: a4b2daa8a64012912ce7186891e8554af123925dca344c111b771e168a37477e02d504c6c94ee698440380e8c4f3f373d6755be97935da30eae0904f6745ce40 + languageName: node + linkType: hard + +"@types/d3-hierarchy@npm:*": + version: 3.1.7 + resolution: "@types/d3-hierarchy@npm:3.1.7" + checksum: 69746b3a65e0fe0ceb3ffcb1a8840a61e271eadb32eccb5034f0fce036d24801aef924ee45b99246580c9f7c81839ab0555f776a11773d82e860d522a2ff1c0e + languageName: node + linkType: hard + +"@types/d3-interpolate@npm:*": + version: 3.0.4 + resolution: "@types/d3-interpolate@npm:3.0.4" + dependencies: + "@types/d3-color": "*" + checksum: efd2770e174e84fc7316fdafe03cf3688451f767dde1fa6211610137f495be7f3923db7e1723a6961a0e0e9ae0ed969f4f47c038189fa0beb1d556b447922622 + languageName: node + linkType: hard + +"@types/d3-path@npm:*": + version: 3.1.1 + resolution: "@types/d3-path@npm:3.1.1" + checksum: fee8f6b0d3b28a3611c7d7fda3bf2f79392ded266f54b03a220f205c42117644bdcd33dcbf4853da3cca02229f1c669d2a60d5d297a24ce459ba8271ccb26c03 + languageName: node + linkType: hard + +"@types/d3-polygon@npm:*": + version: 3.0.2 + resolution: "@types/d3-polygon@npm:3.0.2" + checksum: 7cf1eadb54f02dd3617512b558f4c0f3811f8a6a8c887d9886981c3cc251db28b68329b2b0707d9f517231a72060adbb08855227f89bef6ef30caedc0a67cab2 + languageName: node + linkType: hard + +"@types/d3-quadtree@npm:*": + version: 3.0.6 + resolution: "@types/d3-quadtree@npm:3.0.6" + checksum: 631fb1a50dbe4fb0c97574891b180ec3d6a0f524bbd8aee8dfd44eda405e7ed1ca2b03d5568a35f697d09e5e4b598117e149236874b0c8764979a3d6242bb0bc + languageName: node + linkType: hard + +"@types/d3-random@npm:*": + version: 3.0.3 + resolution: "@types/d3-random@npm:3.0.3" + checksum: 33285b57768a724d2466ac1deec002432805c9df3e475ffb7f7fec66681cfe3e18d2f68b7f8ba45f400b274907bbebfe8adff14c9a97ef1987e476135e784925 + languageName: node + linkType: hard + +"@types/d3-scale-chromatic@npm:*": + version: 3.1.0 + resolution: "@types/d3-scale-chromatic@npm:3.1.0" + checksum: cb7b86deac077c7c217a52a3f658cdfb812cff8708404fbfe54918c03ead545e1df87df377e9c4eab21c9d6c1aeee6471320e02a5b6b27e2e3f786a12a82ab02 + languageName: node + linkType: hard + +"@types/d3-scale@npm:*": + version: 4.0.9 + resolution: "@types/d3-scale@npm:4.0.9" + dependencies: + "@types/d3-time": "*" + checksum: c44265a38e538983686b1b8d159abfb4e81c09b33316f3a68f0f372d38400fa950ad531644d25230cc7b48ea5adb50270fc54823f088979ade62dcd0225f7aa3 + languageName: node + linkType: hard + +"@types/d3-selection@npm:*": + version: 3.0.11 + resolution: "@types/d3-selection@npm:3.0.11" + checksum: 4b76630f76dffdafc73cdc786d73e7b4c96f40546483074b3da0e7fe83fd7f5ed9bc6c50f79bcef83595f943dcc9ed6986953350f39371047af644cc39c41b43 + languageName: node + linkType: hard + +"@types/d3-shape@npm:*": + version: 3.1.8 + resolution: "@types/d3-shape@npm:3.1.8" + dependencies: + "@types/d3-path": "*" + checksum: 659d51882dccc85d24817bdbcd50589212d12e24eb2aad19bae073665ed25443026e120966faa8523f0412f8a30f7c16002499cea3eb87d25b3011e0ee42e6a2 + languageName: node + linkType: hard + +"@types/d3-time-format@npm:*": + version: 4.0.3 + resolution: "@types/d3-time-format@npm:4.0.3" + checksum: e981fc9780697a9d8c5d1ddf1167d9c6bc28e4e610afddff1384fe55e6eb52cb65309b2a0a1d4cf817413b0a80b9f1a652fe0b2cb8054ace4eafff80a6093aa5 + languageName: node + linkType: hard + +"@types/d3-time@npm:*": + version: 3.0.4 + resolution: "@types/d3-time@npm:3.0.4" + checksum: 0c296884571ce70c4bbd4ea9cd1c93c0c8aee602c6c806b056187dd4ee49daf70c2f41da94b25ba0d796edf8ca83cbb87fe6d1cdda7ca669ab800170ece1c12b + languageName: node + linkType: hard + +"@types/d3-timer@npm:*": + version: 3.0.2 + resolution: "@types/d3-timer@npm:3.0.2" + checksum: 1643eebfa5f4ae3eb00b556bbc509444d88078208ec2589ddd8e4a24f230dd4cf2301e9365947e70b1bee33f63aaefab84cd907822aae812b9bc4871b98ab0e1 + languageName: node + linkType: hard + +"@types/d3-transition@npm:*": + version: 3.0.9 + resolution: "@types/d3-transition@npm:3.0.9" + dependencies: + "@types/d3-selection": "*" + checksum: c8608b1ac7cf09acfe387f3d41074631adcdfd7f2c8ca2efb378309adf0e9fc8469dbcf0d7a8c40fd1f03f2d2bf05fcda0cde7aa356ae8533a141dcab4dff221 + languageName: node + linkType: hard + +"@types/d3-zoom@npm:*": + version: 3.0.8 + resolution: "@types/d3-zoom@npm:3.0.8" + dependencies: + "@types/d3-interpolate": "*" + "@types/d3-selection": "*" + checksum: a1685728949ed39faf8ce162cc13338639c57bc2fd4d55fc7902b2632cad2bc2a808941263e57ce6685647e8a6a0a556e173386a52d6bb74c9ed6195b68be3de + languageName: node + linkType: hard + +"@types/d3@npm:^7.4.3": + version: 7.4.3 + resolution: "@types/d3@npm:7.4.3" + dependencies: + "@types/d3-array": "*" + "@types/d3-axis": "*" + "@types/d3-brush": "*" + "@types/d3-chord": "*" + "@types/d3-color": "*" + "@types/d3-contour": "*" + "@types/d3-delaunay": "*" + "@types/d3-dispatch": "*" + "@types/d3-drag": "*" + "@types/d3-dsv": "*" + "@types/d3-ease": "*" + "@types/d3-fetch": "*" + "@types/d3-force": "*" + "@types/d3-format": "*" + "@types/d3-geo": "*" + "@types/d3-hierarchy": "*" + "@types/d3-interpolate": "*" + "@types/d3-path": "*" + "@types/d3-polygon": "*" + "@types/d3-quadtree": "*" + "@types/d3-random": "*" + "@types/d3-scale": "*" + "@types/d3-scale-chromatic": "*" + "@types/d3-selection": "*" + "@types/d3-shape": "*" + "@types/d3-time": "*" + "@types/d3-time-format": "*" + "@types/d3-timer": "*" + "@types/d3-transition": "*" + "@types/d3-zoom": "*" + checksum: 12234aa093c8661546168becdd8956e892b276f525d96f65a7b32fed886fc6a569fe5a1171bff26fef2a5663960635f460c9504a6f2d242ba281a2b6c8c6465c + languageName: node + linkType: hard + "@types/debug@npm:^4.0.0": version: 4.1.12 resolution: "@types/debug@npm:4.1.12" @@ -4614,6 +4990,13 @@ __metadata: languageName: node linkType: hard +"@types/geojson@npm:*": + version: 7946.0.16 + resolution: "@types/geojson@npm:7946.0.16" + checksum: d66e5e023f43b3e7121448117af1930af7d06410a32a585a8bc9c6bb5d97e0d656cd93d99e31fa432976c32e98d4b780f82bf1fd1acd20ccf952eb6b8e39edf2 + languageName: node + linkType: hard + "@types/gtag.js@npm:^0.0.12": version: 0.0.12 resolution: "@types/gtag.js@npm:0.0.12" @@ -4930,6 +5313,13 @@ __metadata: languageName: node linkType: hard +"@types/trusted-types@npm:^2.0.7": + version: 2.0.7 + resolution: "@types/trusted-types@npm:2.0.7" + checksum: 8e4202766a65877efcf5d5a41b7dd458480b36195e580a3b1085ad21e948bc417d55d6f8af1fd2a7ad008015d4117d5fdfe432731157da3c68678487174e4ba3 + languageName: node + linkType: hard + "@types/unist@npm:*, @types/unist@npm:^3.0.0": version: 3.0.3 resolution: "@types/unist@npm:3.0.3" @@ -5192,6 +5582,15 @@ __metadata: languageName: node linkType: hard +"acorn@npm:^8.16.0": + version: 8.16.0 + resolution: "acorn@npm:8.16.0" + bin: + acorn: bin/acorn + checksum: bbfa466cd0dbd18b4460a85e9d0fc2f35db999380892403c573261beda91f23836db2aa71fd3ae65e94424ad14ff8e2b7bd37c7a2624278fd89137cd6e448c41 + languageName: node + linkType: hard + "address@npm:^1.0.1": version: 1.2.2 resolution: "address@npm:1.2.2" @@ -6058,6 +6457,31 @@ __metadata: languageName: node linkType: hard +"chevrotain-allstar@npm:~0.3.1": + version: 0.3.1 + resolution: "chevrotain-allstar@npm:0.3.1" + dependencies: + lodash-es: ^4.17.21 + peerDependencies: + chevrotain: ^11.0.0 + checksum: 5f5213693886d03ca04ffacc57f7424b5c8015e7a62de3c193c3bc94ae7472f113e9fab7f4e92ce0553c181483950a170576897d7b695aac6196ce32b988475e + languageName: node + linkType: hard + +"chevrotain@npm:~11.1.1": + version: 11.1.2 + resolution: "chevrotain@npm:11.1.2" + dependencies: + "@chevrotain/cst-dts-gen": 11.1.2 + "@chevrotain/gast": 11.1.2 + "@chevrotain/regexp-to-ast": 11.1.2 + "@chevrotain/types": 11.1.2 + "@chevrotain/utils": 11.1.2 + lodash-es: 4.17.23 + checksum: 9b5f1abe42db8af0e772e85b74ff730367741cd0dbead6a4cef9cc65fe6023ed891daf3a32a8d65b91a367b652b57a23fb222185851a25c9bdb6172ed1d9874d + languageName: node + linkType: hard + "chokidar@npm:^3.3.0, chokidar@npm:^3.5.3": version: 3.6.0 resolution: "chokidar@npm:3.6.0" @@ -6279,6 +6703,13 @@ __metadata: languageName: node linkType: hard +"commander@npm:7, commander@npm:^7.2.0": + version: 7.2.0 + resolution: "commander@npm:7.2.0" + checksum: 53501cbeee61d5157546c0bef0fedb6cdfc763a882136284bed9a07225f09a14b82d2a84e7637edfd1a679fb35ed9502fd58ef1d091e6287f60d790147f68ddc + languageName: node + linkType: hard + "commander@npm:^10.0.0": version: 10.0.1 resolution: "commander@npm:10.0.1" @@ -6300,13 +6731,6 @@ __metadata: languageName: node linkType: hard -"commander@npm:^7.2.0": - version: 7.2.0 - resolution: "commander@npm:7.2.0" - checksum: 53501cbeee61d5157546c0bef0fedb6cdfc763a882136284bed9a07225f09a14b82d2a84e7637edfd1a679fb35ed9502fd58ef1d091e6287f60d790147f68ddc - languageName: node - linkType: hard - "commander@npm:^8.3.0": version: 8.3.0 resolution: "commander@npm:8.3.0" @@ -6378,6 +6802,13 @@ __metadata: languageName: node linkType: hard +"confbox@npm:^0.1.8": + version: 0.1.8 + resolution: "confbox@npm:0.1.8" + checksum: 5c7718ab22cf9e35a31c21ef124156076ae8c9dc65e6463d54961caf5a1d529284485a0fdf83fd23b27329f3b75b0c8c07d2e36c699f5151a2efe903343f976a + languageName: node + linkType: hard + "config-chain@npm:^1.1.11": version: 1.1.13 resolution: "config-chain@npm:1.1.13" @@ -6519,6 +6950,24 @@ __metadata: languageName: node linkType: hard +"cose-base@npm:^1.0.0": + version: 1.0.3 + resolution: "cose-base@npm:1.0.3" + dependencies: + layout-base: ^1.0.0 + checksum: 3f3d592316df74adb215ca91e430f1c22b6e890bc0025b32ae1f6464c73fdb9614816cb40a8d38b40c6a3e9e7b8c64eda90d53fb9a4a6948abec17dad496f30b + languageName: node + linkType: hard + +"cose-base@npm:^2.2.0": + version: 2.2.0 + resolution: "cose-base@npm:2.2.0" + dependencies: + layout-base: ^2.0.0 + checksum: 2e694f340bf216c71fc126d237578a4168e138720011d0b48c88bf9bfc7fd45f912eff2c603ef3d1307d6e3ce6f465ed382285a764a3a6620db590c5457d2557 + languageName: node + linkType: hard + "cosmiconfig@npm:^7.0.0": version: 7.1.0 resolution: "cosmiconfig@npm:7.1.0" @@ -6730,127 +7179,519 @@ __metadata: languageName: node linkType: hard -"css-what@npm:^6.0.1, css-what@npm:^6.1.0": - version: 6.1.0 - resolution: "css-what@npm:6.1.0" - checksum: b975e547e1e90b79625918f84e67db5d33d896e6de846c9b584094e529f0c63e2ab85ee33b9daffd05bff3a146a1916bec664e18bb76dd5f66cbff9fc13b2bbe +"css-what@npm:^6.0.1, css-what@npm:^6.1.0": + version: 6.1.0 + resolution: "css-what@npm:6.1.0" + checksum: b975e547e1e90b79625918f84e67db5d33d896e6de846c9b584094e529f0c63e2ab85ee33b9daffd05bff3a146a1916bec664e18bb76dd5f66cbff9fc13b2bbe + languageName: node + linkType: hard + +"cssdb@npm:^8.2.1": + version: 8.2.2 + resolution: "cssdb@npm:8.2.2" + checksum: 0098a659654991aa8866eea9b32c0f072898d002f48bb8b25dbb8210e458a3758ce9614d3e328973c08b574df5900f4c1e1bb8d84e9229aa17d9e633dbd81306 + languageName: node + linkType: hard + +"cssdb@npm:^8.4.0": + version: 8.4.0 + resolution: "cssdb@npm:8.4.0" + checksum: 5fa7af956ddce6059165344bde949b13d35b6dc1d94fd01d7d49576672dfad78752786b8444f479b71d924e67887f2c4c10b990fc7d9e76e785be0a9c1ffa239 + languageName: node + linkType: hard + +"cssesc@npm:^3.0.0": + version: 3.0.0 + resolution: "cssesc@npm:3.0.0" + bin: + cssesc: bin/cssesc + checksum: f8c4ababffbc5e2ddf2fa9957dda1ee4af6048e22aeda1869d0d00843223c1b13ad3f5d88b51caa46c994225eacb636b764eb807a8883e2fb6f99b4f4e8c48b2 + languageName: node + linkType: hard + +"cssnano-preset-advanced@npm:^6.1.2": + version: 6.1.2 + resolution: "cssnano-preset-advanced@npm:6.1.2" + dependencies: + autoprefixer: ^10.4.19 + browserslist: ^4.23.0 + cssnano-preset-default: ^6.1.2 + postcss-discard-unused: ^6.0.5 + postcss-merge-idents: ^6.0.3 + postcss-reduce-idents: ^6.0.3 + postcss-zindex: ^6.0.2 + peerDependencies: + postcss: ^8.4.31 + checksum: cf70e27915947412730abb3075587efb66bcea58d7f1b906a7225bb4a40c9ca40150251a2ac33363d4f55bbdeb9ba000c242fa6244ee36cba2477ac07fbbe791 + languageName: node + linkType: hard + +"cssnano-preset-default@npm:^6.1.2": + version: 6.1.2 + resolution: "cssnano-preset-default@npm:6.1.2" + dependencies: + browserslist: ^4.23.0 + css-declaration-sorter: ^7.2.0 + cssnano-utils: ^4.0.2 + postcss-calc: ^9.0.1 + postcss-colormin: ^6.1.0 + postcss-convert-values: ^6.1.0 + postcss-discard-comments: ^6.0.2 + postcss-discard-duplicates: ^6.0.3 + postcss-discard-empty: ^6.0.3 + postcss-discard-overridden: ^6.0.2 + postcss-merge-longhand: ^6.0.5 + postcss-merge-rules: ^6.1.1 + postcss-minify-font-values: ^6.1.0 + postcss-minify-gradients: ^6.0.3 + postcss-minify-params: ^6.1.0 + postcss-minify-selectors: ^6.0.4 + postcss-normalize-charset: ^6.0.2 + postcss-normalize-display-values: ^6.0.2 + postcss-normalize-positions: ^6.0.2 + postcss-normalize-repeat-style: ^6.0.2 + postcss-normalize-string: ^6.0.2 + postcss-normalize-timing-functions: ^6.0.2 + postcss-normalize-unicode: ^6.1.0 + postcss-normalize-url: ^6.0.2 + postcss-normalize-whitespace: ^6.0.2 + postcss-ordered-values: ^6.0.2 + postcss-reduce-initial: ^6.1.0 + postcss-reduce-transforms: ^6.0.2 + postcss-svgo: ^6.0.3 + postcss-unique-selectors: ^6.0.4 + peerDependencies: + postcss: ^8.4.31 + checksum: 51d93e52df7141143947dc4695b5087c04b41ea153e4f4c0282ac012b62c7457c6aca244f604ae94fa3b4840903a30a1e7df38f8610e0b304d05e3065375ee56 + languageName: node + linkType: hard + +"cssnano-utils@npm:^4.0.2": + version: 4.0.2 + resolution: "cssnano-utils@npm:4.0.2" + peerDependencies: + postcss: ^8.4.31 + checksum: f04c6854e75d847c7a43aff835e003d5bc7387ddfc476f0ad3a2d63663d0cec41047d46604c1717bf6b5a8e24e54bb519e465ff78d62c7e073c7cbe2279bebaf + languageName: node + linkType: hard + +"cssnano@npm:^6.0.1, cssnano@npm:^6.1.2": + version: 6.1.2 + resolution: "cssnano@npm:6.1.2" + dependencies: + cssnano-preset-default: ^6.1.2 + lilconfig: ^3.1.1 + peerDependencies: + postcss: ^8.4.31 + checksum: 65aad92c5ee0089ffd4cd933c18c65edbf7634f7c3cd833a499dc948aa7e4168be22130dfe83bde07fcdc87f7c45a02d09040b7f439498208bc90b8d5a9abcc8 + languageName: node + linkType: hard + +"csso@npm:^5.0.5": + version: 5.0.5 + resolution: "csso@npm:5.0.5" + dependencies: + css-tree: ~2.2.0 + checksum: 0ad858d36bf5012ed243e9ec69962a867509061986d2ee07cc040a4b26e4d062c00d4c07e5ba8d430706ceb02dd87edd30a52b5937fd45b1b6f2119c4993d59a + languageName: node + linkType: hard + +"csstype@npm:^3.0.2, csstype@npm:^3.1.3": + version: 3.1.3 + resolution: "csstype@npm:3.1.3" + checksum: 8db785cc92d259102725b3c694ec0c823f5619a84741b5c7991b8ad135dfaa66093038a1cc63e03361a6cd28d122be48f2106ae72334e067dd619a51f49eddf7 + languageName: node + linkType: hard + +"cytoscape-cose-bilkent@npm:^4.1.0": + version: 4.1.0 + resolution: "cytoscape-cose-bilkent@npm:4.1.0" + dependencies: + cose-base: ^1.0.0 + peerDependencies: + cytoscape: ^3.2.0 + checksum: bea6aa139e21bf4135b01b99f8778eed061e074d1a1689771597e8164a999d66f4075d46be584b0a88a5447f9321f38c90c8821df6a9322faaf5afebf4848d97 + languageName: node + linkType: hard + +"cytoscape-fcose@npm:^2.2.0": + version: 2.2.0 + resolution: "cytoscape-fcose@npm:2.2.0" + dependencies: + cose-base: ^2.2.0 + peerDependencies: + cytoscape: ^3.2.0 + checksum: 94ffe6f131f9c08c2a0a7a6ce1c6c5e523a395bf8d84eba6d4a5f85e23f33788ea3ff807540861a5f78a6914a27729e06a7e6f66784f4f28ea1c030acf500121 + languageName: node + linkType: hard + +"cytoscape@npm:^3.29.3": + version: 3.33.1 + resolution: "cytoscape@npm:3.33.1" + checksum: 4ebb9551ecb868fc6e831f523933bf96bd107d09b984d6d44db45adfd0a0f82f3383d7d0d5bc2053267ab2e8da47ce5ea280159643e818a4f2534affee248db8 + languageName: node + linkType: hard + +"d3-array@npm:1 - 2": + version: 2.12.1 + resolution: "d3-array@npm:2.12.1" + dependencies: + internmap: ^1.0.0 + checksum: 97853b7b523aded17078f37c67742f45d81e88dda2107ae9994c31b9e36c5fa5556c4c4cf39650436f247813602dfe31bf7ad067ff80f127a16903827f10c6eb + languageName: node + linkType: hard + +"d3-array@npm:2 - 3, d3-array@npm:2.10.0 - 3, d3-array@npm:2.5.0 - 3, d3-array@npm:3, d3-array@npm:^3.2.0": + version: 3.2.4 + resolution: "d3-array@npm:3.2.4" + dependencies: + internmap: 1 - 2 + checksum: a5976a6d6205f69208478bb44920dd7ce3e788c9dceb86b304dbe401a4bfb42ecc8b04c20facde486e9adcb488b5d1800d49393a3f81a23902b68158e12cddd0 + languageName: node + linkType: hard + +"d3-axis@npm:3": + version: 3.0.0 + resolution: "d3-axis@npm:3.0.0" + checksum: 227ddaa6d4bad083539c1ec245e2228b4620cca941997a8a650cb0af239375dc20271993127eedac66f0543f331027aca09385e1e16eed023f93eac937cddf0b + languageName: node + linkType: hard + +"d3-brush@npm:3": + version: 3.0.0 + resolution: "d3-brush@npm:3.0.0" + dependencies: + d3-dispatch: 1 - 3 + d3-drag: 2 - 3 + d3-interpolate: 1 - 3 + d3-selection: 3 + d3-transition: 3 + checksum: 1d042167769a02ac76271c71e90376d7184206e489552b7022a8ec2860209fe269db55e0a3430f3dcbe13b6fec2ff65b1adeaccba3218991b38e022390df72e3 + languageName: node + linkType: hard + +"d3-chord@npm:3": + version: 3.0.1 + resolution: "d3-chord@npm:3.0.1" + dependencies: + d3-path: 1 - 3 + checksum: ddf35d41675e0f8738600a8a2f05bf0858def413438c12cba357c5802ecc1014c80a658acbbee63cbad2a8c747912efb2358455d93e59906fe37469f1dc6b78b + languageName: node + linkType: hard + +"d3-color@npm:1 - 3, d3-color@npm:3": + version: 3.1.0 + resolution: "d3-color@npm:3.1.0" + checksum: 4931fbfda5d7c4b5cfa283a13c91a954f86e3b69d75ce588d06cde6c3628cebfc3af2069ccf225e982e8987c612aa7948b3932163ce15eb3c11cd7c003f3ee3b + languageName: node + linkType: hard + +"d3-contour@npm:4": + version: 4.0.2 + resolution: "d3-contour@npm:4.0.2" + dependencies: + d3-array: ^3.2.0 + checksum: 56aa082c1acf62a45b61c8d29fdd307041785aa17d9a07de7d1d848633769887a33fb6823888afa383f31c460d0f21d24756593e84e334ddb92d774214d32f1b + languageName: node + linkType: hard + +"d3-delaunay@npm:6": + version: 6.0.4 + resolution: "d3-delaunay@npm:6.0.4" + dependencies: + delaunator: 5 + checksum: ce6d267d5ef21a8aeadfe4606329fc80a22ab6e7748d47bc220bcc396ee8be84b77a5473033954c5ac4aa522d265ddc45d4165d30fe4787dd60a15ea66b9bbb4 + languageName: node + linkType: hard + +"d3-dispatch@npm:1 - 3, d3-dispatch@npm:3": + version: 3.0.1 + resolution: "d3-dispatch@npm:3.0.1" + checksum: fdfd4a230f46463e28e5b22a45dd76d03be9345b605e1b5dc7d18bd7ebf504e6c00ae123fd6d03e23d9e2711e01f0e14ea89cd0632545b9f0c00b924ba4be223 + languageName: node + linkType: hard + +"d3-drag@npm:2 - 3, d3-drag@npm:3": + version: 3.0.0 + resolution: "d3-drag@npm:3.0.0" + dependencies: + d3-dispatch: 1 - 3 + d3-selection: 3 + checksum: d297231e60ecd633b0d076a63b4052b436ddeb48b5a3a11ff68c7e41a6774565473a6b064c5e9256e88eca6439a917ab9cea76032c52d944ddbf4fd289e31111 + languageName: node + linkType: hard + +"d3-dsv@npm:1 - 3, d3-dsv@npm:3": + version: 3.0.1 + resolution: "d3-dsv@npm:3.0.1" + dependencies: + commander: 7 + iconv-lite: 0.6 + rw: 1 + bin: + csv2json: bin/dsv2json.js + csv2tsv: bin/dsv2dsv.js + dsv2dsv: bin/dsv2dsv.js + dsv2json: bin/dsv2json.js + json2csv: bin/json2dsv.js + json2dsv: bin/json2dsv.js + json2tsv: bin/json2dsv.js + tsv2csv: bin/dsv2dsv.js + tsv2json: bin/dsv2json.js + checksum: 5fc0723647269d5dccd181d74f2265920ab368a2868b0b4f55ffa2fecdfb7814390ea28622cd61ee5d9594ab262879509059544e9f815c54fe76fbfb4ffa4c8a + languageName: node + linkType: hard + +"d3-ease@npm:1 - 3, d3-ease@npm:3": + version: 3.0.1 + resolution: "d3-ease@npm:3.0.1" + checksum: 06e2ee5326d1e3545eab4e2c0f84046a123dcd3b612e68858219aa034da1160333d9ce3da20a1d3486d98cb5c2a06f7d233eee1bc19ce42d1533458bd85dedcd + languageName: node + linkType: hard + +"d3-fetch@npm:3": + version: 3.0.1 + resolution: "d3-fetch@npm:3.0.1" + dependencies: + d3-dsv: 1 - 3 + checksum: 382dcea06549ef82c8d0b719e5dc1d96286352579e3b51b20f71437f5800323315b09cf7dcfd4e1f60a41e1204deb01758470cea257d2285a7abd9dcec806984 + languageName: node + linkType: hard + +"d3-force@npm:3": + version: 3.0.0 + resolution: "d3-force@npm:3.0.0" + dependencies: + d3-dispatch: 1 - 3 + d3-quadtree: 1 - 3 + d3-timer: 1 - 3 + checksum: 6c7e96438cab62fa32aeadb0ade3297b62b51f81b1b38b0a60a5ec9fd627d74090c1189654d92df2250775f31b06812342f089f1d5947de9960a635ee3581def + languageName: node + linkType: hard + +"d3-format@npm:1 - 3, d3-format@npm:3": + version: 3.1.2 + resolution: "d3-format@npm:3.1.2" + checksum: 2ce13417b3186311df3fd924028cd516ec3e96d7c3eb6df9c83f6c2ed43de1717e6c5119a385b7744ef84e2b8a4c678ad95a2b2998391803ceb0d809e235cff4 + languageName: node + linkType: hard + +"d3-geo@npm:3": + version: 3.1.1 + resolution: "d3-geo@npm:3.1.1" + dependencies: + d3-array: 2.5.0 - 3 + checksum: 3cc4bb50af5d2d4858d2df1729a1777b7fd361854079d9faab1166186c988d2cba0d11911da0c4598d5e22fae91d79113ed262a9f98cabdbc6dbf7c30e5c0363 + languageName: node + linkType: hard + +"d3-hierarchy@npm:3": + version: 3.1.2 + resolution: "d3-hierarchy@npm:3.1.2" + checksum: 0fd946a8c5fd4686d43d3e11bbfc2037a145fda29d2261ccd0e36f70b66af6d7638e2c0c7112124d63fc3d3127197a00a6aecf676bd5bd392a94d7235a214263 + languageName: node + linkType: hard + +"d3-interpolate@npm:1 - 3, d3-interpolate@npm:1.2.0 - 3, d3-interpolate@npm:3": + version: 3.0.1 + resolution: "d3-interpolate@npm:3.0.1" + dependencies: + d3-color: 1 - 3 + checksum: a42ba314e295e95e5365eff0f604834e67e4a3b3c7102458781c477bd67e9b24b6bb9d8e41ff5521050a3f2c7c0c4bbbb6e187fd586daa3980943095b267e78b + languageName: node + linkType: hard + +"d3-path@npm:1": + version: 1.0.9 + resolution: "d3-path@npm:1.0.9" + checksum: d4382573baf9509a143f40944baeff9fead136926aed6872f7ead5b3555d68925f8a37935841dd51f1d70b65a294fe35c065b0906fb6e42109295f6598fc16d0 + languageName: node + linkType: hard + +"d3-path@npm:1 - 3, d3-path@npm:3, d3-path@npm:^3.1.0": + version: 3.1.0 + resolution: "d3-path@npm:3.1.0" + checksum: 2306f1bd9191e1eac895ec13e3064f732a85f243d6e627d242a313f9777756838a2215ea11562f0c7630c7c3b16a19ec1fe0948b1c82f3317fac55882f6ee5d8 + languageName: node + linkType: hard + +"d3-polygon@npm:3": + version: 3.0.1 + resolution: "d3-polygon@npm:3.0.1" + checksum: 0b85c532517895544683849768a2c377cee3801ef8ccf3fa9693c8871dd21a0c1a2a0fc75ff54192f0ba2c562b0da2bc27f5bf959dfafc7fa23573b574865d2c + languageName: node + linkType: hard + +"d3-quadtree@npm:1 - 3, d3-quadtree@npm:3": + version: 3.0.1 + resolution: "d3-quadtree@npm:3.0.1" + checksum: 5469d462763811475f34a7294d984f3eb100515b0585ca5b249656f6b1a6e99b20056a2d2e463cc9944b888896d2b1d07859c50f9c0cf23438df9cd2e3146066 + languageName: node + linkType: hard + +"d3-random@npm:3": + version: 3.0.1 + resolution: "d3-random@npm:3.0.1" + checksum: a70ad8d1cabe399ebeb2e482703121ac8946a3b336830b518da6848b9fdd48a111990fc041dc716f16885a72176ffa2898f2a250ca3d363ecdba5ef92b18e131 + languageName: node + linkType: hard + +"d3-sankey@npm:^0.12.3": + version: 0.12.3 + resolution: "d3-sankey@npm:0.12.3" + dependencies: + d3-array: 1 - 2 + d3-shape: ^1.2.0 + checksum: df1cb9c9d02dd8fd14040e89f112f0da58c03bd7529fa001572a6925a51496d1d82ff25d9fedb6c429a91645fbd2476c19891e535ac90c8bc28337c33ee21c87 languageName: node linkType: hard -"cssdb@npm:^8.2.1": - version: 8.2.2 - resolution: "cssdb@npm:8.2.2" - checksum: 0098a659654991aa8866eea9b32c0f072898d002f48bb8b25dbb8210e458a3758ce9614d3e328973c08b574df5900f4c1e1bb8d84e9229aa17d9e633dbd81306 +"d3-scale-chromatic@npm:3": + version: 3.1.0 + resolution: "d3-scale-chromatic@npm:3.1.0" + dependencies: + d3-color: 1 - 3 + d3-interpolate: 1 - 3 + checksum: ab6324bd8e1f708e731e02ab44e09741efda2b174cea1d8ca21e4a87546295e99856bc44e2fd3890f228849c96bccfbcf922328f95be6a7df117453eb5cf22c9 languageName: node linkType: hard -"cssdb@npm:^8.4.0": - version: 8.4.0 - resolution: "cssdb@npm:8.4.0" - checksum: 5fa7af956ddce6059165344bde949b13d35b6dc1d94fd01d7d49576672dfad78752786b8444f479b71d924e67887f2c4c10b990fc7d9e76e785be0a9c1ffa239 +"d3-scale@npm:4": + version: 4.0.2 + resolution: "d3-scale@npm:4.0.2" + dependencies: + d3-array: 2.10.0 - 3 + d3-format: 1 - 3 + d3-interpolate: 1.2.0 - 3 + d3-time: 2.1.1 - 3 + d3-time-format: 2 - 4 + checksum: a9c770d283162c3bd11477c3d9d485d07f8db2071665f1a4ad23eec3e515e2cefbd369059ec677c9ac849877d1a765494e90e92051d4f21111aa56791c98729e languageName: node linkType: hard -"cssesc@npm:^3.0.0": +"d3-selection@npm:2 - 3, d3-selection@npm:3": version: 3.0.0 - resolution: "cssesc@npm:3.0.0" - bin: - cssesc: bin/cssesc - checksum: f8c4ababffbc5e2ddf2fa9957dda1ee4af6048e22aeda1869d0d00843223c1b13ad3f5d88b51caa46c994225eacb636b764eb807a8883e2fb6f99b4f4e8c48b2 + resolution: "d3-selection@npm:3.0.0" + checksum: f4e60e133309115b99f5b36a79ae0a19d71ee6e2d5e3c7216ef3e75ebd2cb1e778c2ed2fa4c01bef35e0dcbd96c5428f5bd6ca2184fe2957ed582fde6841cbc5 languageName: node linkType: hard -"cssnano-preset-advanced@npm:^6.1.2": - version: 6.1.2 - resolution: "cssnano-preset-advanced@npm:6.1.2" +"d3-shape@npm:3": + version: 3.2.0 + resolution: "d3-shape@npm:3.2.0" dependencies: - autoprefixer: ^10.4.19 - browserslist: ^4.23.0 - cssnano-preset-default: ^6.1.2 - postcss-discard-unused: ^6.0.5 - postcss-merge-idents: ^6.0.3 - postcss-reduce-idents: ^6.0.3 - postcss-zindex: ^6.0.2 - peerDependencies: - postcss: ^8.4.31 - checksum: cf70e27915947412730abb3075587efb66bcea58d7f1b906a7225bb4a40c9ca40150251a2ac33363d4f55bbdeb9ba000c242fa6244ee36cba2477ac07fbbe791 + d3-path: ^3.1.0 + checksum: de2af5fc9a93036a7b68581ca0bfc4aca2d5a328aa7ba7064c11aedd44d24f310c20c40157cb654359d4c15c3ef369f95ee53d71221017276e34172c7b719cfa languageName: node linkType: hard -"cssnano-preset-default@npm:^6.1.2": - version: 6.1.2 - resolution: "cssnano-preset-default@npm:6.1.2" +"d3-shape@npm:^1.2.0": + version: 1.3.7 + resolution: "d3-shape@npm:1.3.7" dependencies: - browserslist: ^4.23.0 - css-declaration-sorter: ^7.2.0 - cssnano-utils: ^4.0.2 - postcss-calc: ^9.0.1 - postcss-colormin: ^6.1.0 - postcss-convert-values: ^6.1.0 - postcss-discard-comments: ^6.0.2 - postcss-discard-duplicates: ^6.0.3 - postcss-discard-empty: ^6.0.3 - postcss-discard-overridden: ^6.0.2 - postcss-merge-longhand: ^6.0.5 - postcss-merge-rules: ^6.1.1 - postcss-minify-font-values: ^6.1.0 - postcss-minify-gradients: ^6.0.3 - postcss-minify-params: ^6.1.0 - postcss-minify-selectors: ^6.0.4 - postcss-normalize-charset: ^6.0.2 - postcss-normalize-display-values: ^6.0.2 - postcss-normalize-positions: ^6.0.2 - postcss-normalize-repeat-style: ^6.0.2 - postcss-normalize-string: ^6.0.2 - postcss-normalize-timing-functions: ^6.0.2 - postcss-normalize-unicode: ^6.1.0 - postcss-normalize-url: ^6.0.2 - postcss-normalize-whitespace: ^6.0.2 - postcss-ordered-values: ^6.0.2 - postcss-reduce-initial: ^6.1.0 - postcss-reduce-transforms: ^6.0.2 - postcss-svgo: ^6.0.3 - postcss-unique-selectors: ^6.0.4 - peerDependencies: - postcss: ^8.4.31 - checksum: 51d93e52df7141143947dc4695b5087c04b41ea153e4f4c0282ac012b62c7457c6aca244f604ae94fa3b4840903a30a1e7df38f8610e0b304d05e3065375ee56 + d3-path: 1 + checksum: 46566a3ab64a25023653bf59d64e81e9e6c987e95be985d81c5cedabae5838bd55f4a201a6b69069ca862eb63594cd263cac9034afc2b0e5664dfe286c866129 languageName: node linkType: hard -"cssnano-utils@npm:^4.0.2": - version: 4.0.2 - resolution: "cssnano-utils@npm:4.0.2" - peerDependencies: - postcss: ^8.4.31 - checksum: f04c6854e75d847c7a43aff835e003d5bc7387ddfc476f0ad3a2d63663d0cec41047d46604c1717bf6b5a8e24e54bb519e465ff78d62c7e073c7cbe2279bebaf +"d3-time-format@npm:2 - 4, d3-time-format@npm:4": + version: 4.1.0 + resolution: "d3-time-format@npm:4.1.0" + dependencies: + d3-time: 1 - 3 + checksum: 7342bce28355378152bbd4db4e275405439cabba082d9cd01946d40581140481c8328456d91740b0fe513c51ec4a467f4471ffa390c7e0e30ea30e9ec98fcdf4 languageName: node linkType: hard -"cssnano@npm:^6.0.1, cssnano@npm:^6.1.2": - version: 6.1.2 - resolution: "cssnano@npm:6.1.2" +"d3-time@npm:1 - 3, d3-time@npm:2.1.1 - 3, d3-time@npm:3": + version: 3.1.0 + resolution: "d3-time@npm:3.1.0" dependencies: - cssnano-preset-default: ^6.1.2 - lilconfig: ^3.1.1 - peerDependencies: - postcss: ^8.4.31 - checksum: 65aad92c5ee0089ffd4cd933c18c65edbf7634f7c3cd833a499dc948aa7e4168be22130dfe83bde07fcdc87f7c45a02d09040b7f439498208bc90b8d5a9abcc8 + d3-array: 2 - 3 + checksum: 613b435352a78d9f31b7f68540788186d8c331b63feca60ad21c88e9db1989fe888f97f242322ebd6365e45ec3fb206a4324cd4ca0dfffa1d9b5feb856ba00a7 languageName: node linkType: hard -"csso@npm:^5.0.5": - version: 5.0.5 - resolution: "csso@npm:5.0.5" +"d3-timer@npm:1 - 3, d3-timer@npm:3": + version: 3.0.1 + resolution: "d3-timer@npm:3.0.1" + checksum: 1cfddf86d7bca22f73f2c427f52dfa35c49f50d64e187eb788dcad6e927625c636aa18ae4edd44d084eb9d1f81d8ca4ec305dae7f733c15846a824575b789d73 + languageName: node + linkType: hard + +"d3-transition@npm:2 - 3, d3-transition@npm:3": + version: 3.0.1 + resolution: "d3-transition@npm:3.0.1" dependencies: - css-tree: ~2.2.0 - checksum: 0ad858d36bf5012ed243e9ec69962a867509061986d2ee07cc040a4b26e4d062c00d4c07e5ba8d430706ceb02dd87edd30a52b5937fd45b1b6f2119c4993d59a + d3-color: 1 - 3 + d3-dispatch: 1 - 3 + d3-ease: 1 - 3 + d3-interpolate: 1 - 3 + d3-timer: 1 - 3 + peerDependencies: + d3-selection: 2 - 3 + checksum: cb1e6e018c3abf0502fe9ff7b631ad058efb197b5e14b973a410d3935aead6e3c07c67d726cfab258e4936ef2667c2c3d1cd2037feb0765f0b4e1d3b8788c0ea languageName: node linkType: hard -"csstype@npm:^3.0.2, csstype@npm:^3.1.3": - version: 3.1.3 - resolution: "csstype@npm:3.1.3" - checksum: 8db785cc92d259102725b3c694ec0c823f5619a84741b5c7991b8ad135dfaa66093038a1cc63e03361a6cd28d122be48f2106ae72334e067dd619a51f49eddf7 +"d3-zoom@npm:3": + version: 3.0.0 + resolution: "d3-zoom@npm:3.0.0" + dependencies: + d3-dispatch: 1 - 3 + d3-drag: 2 - 3 + d3-interpolate: 1 - 3 + d3-selection: 2 - 3 + d3-transition: 2 - 3 + checksum: 8056e3527281cfd1ccbcbc458408f86973b0583e9dac00e51204026d1d36803ca437f970b5736f02fafed9f2b78f145f72a5dbc66397e02d4d95d4c594b8ff54 + languageName: node + linkType: hard + +"d3@npm:^7.9.0": + version: 7.9.0 + resolution: "d3@npm:7.9.0" + dependencies: + d3-array: 3 + d3-axis: 3 + d3-brush: 3 + d3-chord: 3 + d3-color: 3 + d3-contour: 4 + d3-delaunay: 6 + d3-dispatch: 3 + d3-drag: 3 + d3-dsv: 3 + d3-ease: 3 + d3-fetch: 3 + d3-force: 3 + d3-format: 3 + d3-geo: 3 + d3-hierarchy: 3 + d3-interpolate: 3 + d3-path: 3 + d3-polygon: 3 + d3-quadtree: 3 + d3-random: 3 + d3-scale: 4 + d3-scale-chromatic: 3 + d3-selection: 3 + d3-shape: 3 + d3-time: 3 + d3-time-format: 4 + d3-timer: 3 + d3-transition: 3 + d3-zoom: 3 + checksum: 1c0e9135f1fb78aa32b187fafc8b56ae6346102bd0e4e5e5a5339611a51e6038adbaa293fae373994228100eddd87320e930b1be922baeadc07c9fd43d26d99b + languageName: node + linkType: hard + +"dagre-d3-es@npm:7.0.13": + version: 7.0.13 + resolution: "dagre-d3-es@npm:7.0.13" + dependencies: + d3: ^7.9.0 + lodash-es: ^4.17.21 + checksum: 4a6e5aeb8d4a643c19241b58b8507a7e8aadae8c470ecb9b0133b3af0d6244356cb2cda643fe2f4ef5f872800eb873d7287977b4c5965962396b1757afb0b5de languageName: node linkType: hard @@ -6887,6 +7728,13 @@ __metadata: languageName: node linkType: hard +"dayjs@npm:^1.11.18": + version: 1.11.19 + resolution: "dayjs@npm:1.11.19" + checksum: dfafcca2c67cc6e542fd880d77f1d91667efd323edc28f0487b470b184a11cc97696163ed5be1142ea2a031045b27a0d0555e72f60a63275e0e0401ac24bea5d + languageName: node + linkType: hard + "debounce@npm:^1.2.1": version: 1.2.1 resolution: "debounce@npm:1.2.1" @@ -6992,6 +7840,15 @@ __metadata: languageName: node linkType: hard +"delaunator@npm:5": + version: 5.0.1 + resolution: "delaunator@npm:5.0.1" + dependencies: + robust-predicates: ^3.0.2 + checksum: 69ee43ec649b4a13b7f33c8a027fb3e8dfcb09266af324286118da757e04d3d39df619b905dca41421405c311317ccf632ecfa93db44519bacec3303c57c5a0b + languageName: node + linkType: hard + "delayed-stream@npm:~1.0.0": version: 1.0.0 resolution: "delayed-stream@npm:1.0.0" @@ -7095,15 +7952,15 @@ __metadata: languageName: node linkType: hard -"docusaurus-plugin-image-zoom@npm:^2.0.0": - version: 2.0.0 - resolution: "docusaurus-plugin-image-zoom@npm:2.0.0" +"docusaurus-plugin-image-zoom@npm:^3.0.1": + version: 3.0.1 + resolution: "docusaurus-plugin-image-zoom@npm:3.0.1" dependencies: - medium-zoom: ^1.0.8 + medium-zoom: ^1.1.0 validate-peer-dependencies: ^2.2.0 peerDependencies: "@docusaurus/theme-classic": ">=3.0.0" - checksum: 35090848fbdfc41191acf266ce90ba66b80c227c26457c490938dae3439242d9f2554a417ca9b8e0cf78b2bbecc1225d0c893159d52c4e5d486b95c4bc0f2931 + checksum: 55cc967cd025eafb78f5422d7f6988f179baf3917c8f1286a8a795106a12b15ad04c4ff8cb70809bc06c481029d9f861516b53e788b1114ce4c955dfb03e6ac4 languageName: node linkType: hard @@ -7199,6 +8056,18 @@ __metadata: languageName: node linkType: hard +"dompurify@npm:^3.2.5": + version: 3.3.2 + resolution: "dompurify@npm:3.3.2" + dependencies: + "@types/trusted-types": ^2.0.7 + dependenciesMeta: + "@types/trusted-types": + optional: true + checksum: 27856958c4088de2e2279b9514fcf3427c925e3cedf9d176957d9469a5199c39b572931a441cbb2025d1910c2890644280d0db8d5180b1366164cac309da08e5 + languageName: node + linkType: hard + "domutils@npm:^1.5.1": version: 1.7.0 resolution: "domutils@npm:1.7.0" @@ -8524,6 +9393,13 @@ __metadata: languageName: node linkType: hard +"hachure-fill@npm:^0.5.2": + version: 0.5.2 + resolution: "hachure-fill@npm:0.5.2" + checksum: 01cf2ac6b787ec73ced3d6eb393a0f989d55f32431d1e8a1c1c864769d1b8763c9cb6aa1d45fb1c237a065de90167491c6a46193690b688ea6c25f575f84586c + languageName: node + linkType: hard + "handle-thing@npm:^2.0.0": version: 2.0.1 resolution: "handle-thing@npm:2.0.1" @@ -9120,7 +9996,7 @@ __metadata: languageName: node linkType: hard -"iconv-lite@npm:^0.6.2": +"iconv-lite@npm:0.6, iconv-lite@npm:^0.6.2": version: 0.6.3 resolution: "iconv-lite@npm:0.6.3" dependencies: @@ -9300,6 +10176,20 @@ __metadata: languageName: node linkType: hard +"internmap@npm:1 - 2": + version: 2.0.3 + resolution: "internmap@npm:2.0.3" + checksum: 7ca41ec6aba8f0072fc32fa8a023450a9f44503e2d8e403583c55714b25efd6390c38a87161ec456bf42d7bc83aab62eb28f5aef34876b1ac4e60693d5e1d241 + languageName: node + linkType: hard + +"internmap@npm:^1.0.0": + version: 1.0.1 + resolution: "internmap@npm:1.0.1" + checksum: 9d00f8c0cf873a24a53a5a937120dab634c41f383105e066bb318a61864e6292d24eb9516e8e7dccfb4420ec42ca474a0f28ac9a6cc82536898fa09bbbe53813 + languageName: node + linkType: hard + "invariant@npm:^2.2.4": version: 2.2.4 resolution: "invariant@npm:2.2.4" @@ -10070,6 +10960,17 @@ __metadata: languageName: node linkType: hard +"katex@npm:^0.16.22": + version: 0.16.35 + resolution: "katex@npm:0.16.35" + dependencies: + commander: ^8.3.0 + bin: + katex: cli.js + checksum: 7b97abdd28c193e47c03d3cdbd00862179af387e4c8fa4b9f9e87f6ffc67e069ef8d0be23ee4d8a58ab5cc68c66ffbe440a77c4355267952f2a7e5b47c3e5002 + languageName: node + linkType: hard + "keyv@npm:^4.5.3": version: 4.5.4 resolution: "keyv@npm:4.5.4" @@ -10079,6 +10980,13 @@ __metadata: languageName: node linkType: hard +"khroma@npm:^2.1.0": + version: 2.1.0 + resolution: "khroma@npm:2.1.0" + checksum: b34ba39d3a9a52d388110bded8cb1c12272eb69c249d8eb26feab12d18a96a9bc4ceec4851d2afa43de4569f7d5ea78fa305965a3d0e96a38e02fe77c53677da + languageName: node + linkType: hard + "kind-of@npm:^6.0.0, kind-of@npm:^6.0.2": version: 6.0.3 resolution: "kind-of@npm:6.0.3" @@ -10093,6 +11001,19 @@ __metadata: languageName: node linkType: hard +"langium@npm:^4.0.0": + version: 4.2.1 + resolution: "langium@npm:4.2.1" + dependencies: + chevrotain: ~11.1.1 + chevrotain-allstar: ~0.3.1 + vscode-languageserver: ~9.0.1 + vscode-languageserver-textdocument: ~1.0.11 + vscode-uri: ~3.1.0 + checksum: ec46b533abddc4c8f9c4d1da578742d31ca452a26a08922e3c220da37cd5171f14d50da5d48bdbde179f877f935377d67e3f2a1580490abe941fba30e11eec94 + languageName: node + linkType: hard + "latest-version@npm:^7.0.0": version: 7.0.0 resolution: "latest-version@npm:7.0.0" @@ -10112,6 +11033,20 @@ __metadata: languageName: node linkType: hard +"layout-base@npm:^1.0.0": + version: 1.0.2 + resolution: "layout-base@npm:1.0.2" + checksum: e4c312765ac4fa13b49c940e701461309c7a0aa07f784f81d31f626b945dced90a8abf83222388a5af16b7074271f745501a90ef5a3af676abb2e7eb16d55b2e + languageName: node + linkType: hard + +"layout-base@npm:^2.0.0": + version: 2.0.1 + resolution: "layout-base@npm:2.0.1" + checksum: ef93baf044f67c3680f4f3a6d628bf4c7faba0f70f3e0abb16e4811bed087045208560347ca749e123d169cbf872505ad84e11fb21b0be925997227e042c7f43 + languageName: node + linkType: hard + "leven@npm:^3.1.0": version: 3.1.0 resolution: "leven@npm:3.1.0" @@ -10293,6 +11228,13 @@ __metadata: languageName: node linkType: hard +"lodash-es@npm:4.17.23, lodash-es@npm:^4.17.21, lodash-es@npm:^4.17.23": + version: 4.17.23 + resolution: "lodash-es@npm:4.17.23" + checksum: b1bd1d141bbde8ffc72978e34b364065675806b0ca42ab99477d247fb2ae795faeed81db9283bf18ae1f096c2b6611ec0589e0503fa9724bf82e3dce947bad69 + languageName: node + linkType: hard + "lodash.debounce@npm:^4.0.8": version: 4.0.8 resolution: "lodash.debounce@npm:4.0.8" @@ -10458,6 +11400,15 @@ __metadata: languageName: node linkType: hard +"marked@npm:^16.2.1": + version: 16.4.2 + resolution: "marked@npm:16.4.2" + bin: + marked: bin/marked.js + checksum: 8749bc6228ff59eb63f82c7310750336eb85c42c2b37d0d24f86807cf9e7b441bf8a20ed1bbcadfcd7a2db41d1b6069642286d4403815b90c2ce5be6aa00124c + languageName: node + linkType: hard + "marked@npm:^2.0.3": version: 2.1.3 resolution: "marked@npm:2.1.3" @@ -10773,7 +11724,7 @@ __metadata: languageName: node linkType: hard -"medium-zoom@npm:^1.0.8": +"medium-zoom@npm:^1.1.0": version: 1.1.0 resolution: "medium-zoom@npm:1.1.0" checksum: c58ea2d3335c3b5d4625b4fa70d09925e35d2eb15e9906f41586e7016aa3059ca9b01656ae6d35e70219c7e3e94dda0e9528cd354fcd2f6614231b529654b0b9 @@ -10817,6 +11768,34 @@ __metadata: languageName: node linkType: hard +"mermaid@npm:>=11.6.0": + version: 11.12.3 + resolution: "mermaid@npm:11.12.3" + dependencies: + "@braintree/sanitize-url": ^7.1.1 + "@iconify/utils": ^3.0.1 + "@mermaid-js/parser": ^1.0.0 + "@types/d3": ^7.4.3 + cytoscape: ^3.29.3 + cytoscape-cose-bilkent: ^4.1.0 + cytoscape-fcose: ^2.2.0 + d3: ^7.9.0 + d3-sankey: ^0.12.3 + dagre-d3-es: 7.0.13 + dayjs: ^1.11.18 + dompurify: ^3.2.5 + katex: ^0.16.22 + khroma: ^2.1.0 + lodash-es: ^4.17.23 + marked: ^16.2.1 + roughjs: ^4.6.6 + stylis: ^4.3.6 + ts-dedent: ^2.2.0 + uuid: ^11.1.0 + checksum: c49efc4f10b0cca1fe417a3ff59cc1c400093632ff6cfecc12314f1986422c344deb7727be980da60f2e25e8dac9b434f01dadceff7dfed943aed59b5ed5b817 + languageName: node + linkType: hard + "methods@npm:~1.1.2": version: 1.1.2 resolution: "methods@npm:1.1.2" @@ -11589,6 +12568,18 @@ __metadata: languageName: node linkType: hard +"mlly@npm:^1.7.4, mlly@npm:^1.8.0": + version: 1.8.1 + resolution: "mlly@npm:1.8.1" + dependencies: + acorn: ^8.16.0 + pathe: ^2.0.3 + pkg-types: ^1.3.1 + ufo: ^1.6.3 + checksum: 31bd5dda2cb3939771da6cfeda08d0033a87e94311b927ef80539a9a9cf269215cef801a896db3204c2e4c1de73ab56501ca2d0509f5f2e9304b859ef7e779d7 + languageName: node + linkType: hard + "moment@npm:^2.30.1": version: 2.30.1 resolution: "moment@npm:2.30.1" @@ -12213,6 +13204,13 @@ __metadata: languageName: node linkType: hard +"package-manager-detector@npm:^1.3.0": + version: 1.6.0 + resolution: "package-manager-detector@npm:1.6.0" + checksum: 154d55225e70e32582f59b5d4a46d25716f0730a14d7e4b6f0fd76c870c720cc6f448d2becca06a15f2042492f0293cf26e5ad8fcd85d0eab0af3b9b46c0b43a + languageName: node + linkType: hard + "pacote@npm:^19.0.0, pacote@npm:^19.0.1": version: 19.0.1 resolution: "pacote@npm:19.0.1" @@ -12382,6 +13380,13 @@ __metadata: languageName: node linkType: hard +"path-data-parser@npm:0.1.0, path-data-parser@npm:^0.1.0": + version: 0.1.0 + resolution: "path-data-parser@npm:0.1.0" + checksum: a23a214adb38074576a8873d25e8dea7e090b8396d86f58f83f3f6c6298ff56b06adc694147b67f0ed22f14dc478efa1d525710d3ec7b2d7b1efbac57e3fafe6 + languageName: node + linkType: hard + "path-exists@npm:^5.0.0": version: 5.0.0 resolution: "path-exists@npm:5.0.0" @@ -12487,6 +13492,13 @@ __metadata: languageName: node linkType: hard +"pathe@npm:^2.0.1, pathe@npm:^2.0.3": + version: 2.0.3 + resolution: "pathe@npm:2.0.3" + checksum: 0602bdd4acb54d91044e0c56f1fb63467ae7d44ab3afea1f797947b0eb2b4d1d91cf0d58d065fdb0a8ab0c4acbbd8d3a5b424983eaf10dd5285d37a16f6e3ee9 + languageName: node + linkType: hard + "picocolors@npm:^1.0.0, picocolors@npm:^1.0.1, picocolors@npm:^1.1.0, picocolors@npm:^1.1.1": version: 1.1.1 resolution: "picocolors@npm:1.1.1" @@ -12517,6 +13529,34 @@ __metadata: languageName: node linkType: hard +"pkg-types@npm:^1.3.1": + version: 1.3.1 + resolution: "pkg-types@npm:1.3.1" + dependencies: + confbox: ^0.1.8 + mlly: ^1.7.4 + pathe: ^2.0.1 + checksum: 4fa4edb2bb845646cdbd04c5c6bc43cdbc8f02ed4d1c28bfcafb6e65928aece789bcf1335e4cac5f65dfdc376e4bd7435bd509a35e9ec73ef2c076a1b88e289c + languageName: node + linkType: hard + +"points-on-curve@npm:0.2.0, points-on-curve@npm:^0.2.0": + version: 0.2.0 + resolution: "points-on-curve@npm:0.2.0" + checksum: 05e87d6839e3d869cfac0e63c2b1ca700fc8f1083e3f9ae80841cc50379fd31204f9e1f221407df1a90afcb8bfa98404aee0b0fa00330b7b3b328d33be21cf47 + languageName: node + linkType: hard + +"points-on-path@npm:^0.2.1": + version: 0.2.1 + resolution: "points-on-path@npm:0.2.1" + dependencies: + path-data-parser: 0.1.0 + points-on-curve: 0.2.0 + checksum: 5564dd84d15699579bf07bd33adfd0dc1a5e717c0d36ee11f0832b6b6890941e25e9ea68d15f7858698a9b5ec509f60e6472a0346624bb9dd9c2100cf568ac8f + languageName: node + linkType: hard + "possible-typed-array-names@npm:^1.0.0": version: 1.0.0 resolution: "possible-typed-array-names@npm:1.0.0" @@ -14753,6 +15793,25 @@ __metadata: languageName: node linkType: hard +"robust-predicates@npm:^3.0.2": + version: 3.0.2 + resolution: "robust-predicates@npm:3.0.2" + checksum: 36854c1321548ceca96d36ad9d6e0a5a512986029ec6929ad6ed3ec1612c22cc8b46cc72d2c5674af42e8074a119d793f6f0ea3a5b51373e3ab926c64b172d7a + languageName: node + linkType: hard + +"roughjs@npm:^4.6.6": + version: 4.6.6 + resolution: "roughjs@npm:4.6.6" + dependencies: + hachure-fill: ^0.5.2 + path-data-parser: ^0.1.0 + points-on-curve: ^0.2.0 + points-on-path: ^0.2.1 + checksum: ec4b8266ac4a50c7369e337d8ddff3b2d970506229cac5425ddca56f4e6b29fca07dded4300e9e392bb608da4ba618d349fd241283affb25055cab7c2fe48f8f + languageName: node + linkType: hard + "rtlcss@npm:^4.1.0": version: 4.3.0 resolution: "rtlcss@npm:4.3.0" @@ -14776,6 +15835,13 @@ __metadata: languageName: node linkType: hard +"rw@npm:1": + version: 1.3.3 + resolution: "rw@npm:1.3.3" + checksum: c20d82421f5a71c86a13f76121b751553a99cd4a70ea27db86f9b23f33db941f3f06019c30f60d50c356d0bd674c8e74764ac146ea55e217c091bde6fba82aa3 + languageName: node + linkType: hard + "safe-array-concat@npm:^1.1.2": version: 1.1.2 resolution: "safe-array-concat@npm:1.1.2" @@ -15620,6 +16686,13 @@ __metadata: languageName: node linkType: hard +"stylis@npm:^4.3.6": + version: 4.3.6 + resolution: "stylis@npm:4.3.6" + checksum: 4f56a087caace85b34c3a163cf9d662f58f42dc865b2447af5c3ee3588eebaffe90875fe294578cce26f172ff527cad2b01433f6e1ae156400ec38c37c79fd61 + languageName: node + linkType: hard + "supports-color@npm:^7.1.0": version: 7.2.0 resolution: "supports-color@npm:7.2.0" @@ -15828,6 +16901,13 @@ __metadata: languageName: node linkType: hard +"tinyexec@npm:^1.0.1": + version: 1.0.2 + resolution: "tinyexec@npm:1.0.2" + checksum: af22de2191cc70bb782eef29bbba7cf6ac16664e550b547b0db68804f988eeb2c70e12fbb7d2d688ee994b28ba831d746e9eded98c3d10042fd3a9b8de208514 + languageName: node + linkType: hard + "tinypool@npm:^1.0.2": version: 1.1.1 resolution: "tinypool@npm:1.1.1" @@ -15890,6 +16970,13 @@ __metadata: languageName: node linkType: hard +"ts-dedent@npm:^2.2.0": + version: 2.2.0 + resolution: "ts-dedent@npm:2.2.0" + checksum: 93ed8f7878b6d5ed3c08d99b740010eede6bccfe64bce61c5a4da06a2c17d6ddbb80a8c49c2d15251de7594a4f93ffa21dd10e7be75ef66a4dc9951b4a94e2af + languageName: node + linkType: hard + "ts-node@npm:^10.9.2": version: 10.9.2 resolution: "ts-node@npm:10.9.2" @@ -16080,6 +17167,13 @@ __metadata: languageName: node linkType: hard +"ufo@npm:^1.6.3": + version: 1.6.3 + resolution: "ufo@npm:1.6.3" + checksum: a23eff86bbbef0b9cc69c19c653c703b656c2328938576d3a60e05e246ef5a78d88b17c710afa146311c5b855950ccfee60ba8f6c8845e8d1ed6b5a9086ddad1 + languageName: node + linkType: hard + "unbox-primitive@npm:^1.0.2": version: 1.0.2 resolution: "unbox-primitive@npm:1.0.2" @@ -16419,6 +17513,15 @@ __metadata: languageName: node linkType: hard +"uuid@npm:^11.1.0": + version: 11.1.0 + resolution: "uuid@npm:11.1.0" + bin: + uuid: dist/esm/bin/uuid + checksum: 840f19758543c4631e58a29439e51b5b669d5f34b4dd2700b6a1d15c5708c7a6e0c3e2c8c4a2eae761a3a7caa7e9884d00c86c02622ba91137bd3deade6b4b4a + languageName: node + linkType: hard + "uuid@npm:^8.3.2": version: 8.3.2 resolution: "uuid@npm:8.3.2" @@ -16506,6 +17609,55 @@ __metadata: languageName: node linkType: hard +"vscode-jsonrpc@npm:8.2.0": + version: 8.2.0 + resolution: "vscode-jsonrpc@npm:8.2.0" + checksum: f302a01e59272adc1ae6494581fa31c15499f9278df76366e3b97b2236c7c53ebfc71efbace9041cfd2caa7f91675b9e56f2407871a1b3c7f760a2e2ee61484a + languageName: node + linkType: hard + +"vscode-languageserver-protocol@npm:3.17.5": + version: 3.17.5 + resolution: "vscode-languageserver-protocol@npm:3.17.5" + dependencies: + vscode-jsonrpc: 8.2.0 + vscode-languageserver-types: 3.17.5 + checksum: dfb42d276df5dfea728267885b99872ecff62f6c20448b8539fae71bb196b420f5351c5aca7c1047bf8fb1f89fa94a961dce2bc5bf7e726198f4be0bb86a1e71 + languageName: node + linkType: hard + +"vscode-languageserver-textdocument@npm:~1.0.11": + version: 1.0.12 + resolution: "vscode-languageserver-textdocument@npm:1.0.12" + checksum: 49415c8f065860693fdd6cb0f7b8a24470130dc941e887a396b6e6bbae93be132323a644aa1edd7d0eec38a730e05a2d013aebff6bddd30c5af374ef3f4cd9ab + languageName: node + linkType: hard + +"vscode-languageserver-types@npm:3.17.5": + version: 3.17.5 + resolution: "vscode-languageserver-types@npm:3.17.5" + checksum: 79b420e7576398d396579ca3a461c9ed70e78db4403cd28bbdf4d3ed2b66a2b4114031172e51fad49f0baa60a2180132d7cb2ea35aa3157d7af3c325528210ac + languageName: node + linkType: hard + +"vscode-languageserver@npm:~9.0.1": + version: 9.0.1 + resolution: "vscode-languageserver@npm:9.0.1" + dependencies: + vscode-languageserver-protocol: 3.17.5 + bin: + installServerIntoExtension: bin/installServerIntoExtension + checksum: 8b7dfda47fb64c3f48a9dabd3f01938cc8d39f3f068f1ee586eaf0a373536180a1047bdde8d876f965cfc04160d1587e99828b61b742b0342595fee67c8814ea + languageName: node + linkType: hard + +"vscode-uri@npm:~3.1.0": + version: 3.1.0 + resolution: "vscode-uri@npm:3.1.0" + checksum: d0f76a22f3d205dd2754d1c2820f55c1c9941c59b3baf1f4ed330fcdc006f2fc8b3c1338dafd6b30448f153173892651afe738a94b2c218ca4072d0e604c8d5e + languageName: node + linkType: hard + "walk-up-path@npm:^3.0.1": version: 3.0.1 resolution: "walk-up-path@npm:3.0.1" @@ -16716,6 +17868,7 @@ __metadata: "@docusaurus/plugin-google-analytics": 3.8.1 "@docusaurus/preset-classic": 3.8.1 "@docusaurus/theme-classic": ^3.8.1 + "@docusaurus/theme-mermaid": 3.8.1 "@docusaurus/tsconfig": 3.8.1 "@docusaurus/types": 3.8.1 "@emotion/react": ^11.13.3 @@ -16736,7 +17889,7 @@ __metadata: chokidar: ^4.0.1 clsx: ^2.1.1 compare-versions: 4.1.3 - docusaurus-plugin-image-zoom: ^2.0.0 + docusaurus-plugin-image-zoom: ^3.0.1 execa: ^9.4.1 file-loader: ^6.2.0 font-awesome: ^4.7.0 From f668c5710c0324ee9302d260c45e6bc058fd2380 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 14:57:49 +0200 Subject: [PATCH 37/59] Add more details and sequence diagram --- docs/administration-geo.md | 98 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 83399da6423f..aaa5f8c3a2bb 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -374,6 +374,10 @@ A known issue where the snapshot condition was not met under high message rates After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. +The improve snapshot cache expiration policy introduced in [PR #25044](https://github.com/apache/pulsar/pull/25044) doesn't address the challenges with messaged delivered with the delayed message delivery feature. The reason for this is that replication snapshots are added to the cache when the replicator for the particular topic reads the replication snapshot marker messages from the topic. When delayed message delivery is used, there might be a very large backlog and the topic might have been unloaded or moved to another broker in the cluster due to broker restarts or load shedding. In that case, there won't be any entries in the replication snapshot cache and it won't get restored since the replication cursor has already moved further ahead. + +This is the reason why replicated subscriptions don't provide consistent behavior when using delayed message delivery. In the case of shared or key-shared subscriptions, the mark delete position moves ahead fairly frequently compared to delayed message delivery and that's why the problem is addressed with PR #25044 and the new snapshot cache expiration policy. After a topic unload or topic getting moved to another broker the replication snapshot cache would eventually fill with suitable snapshots which can be used to replicate the subscription state. + The following broker settings control snapshot behavior: | Setting | Default | Description | @@ -384,6 +388,100 @@ The following broker settings control snapshot behavior: For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. +### Replicated subscriptions sequence diagrams + +This sequence diagram describes the interactions of the replicated subscription solution. It could help understand the detail of why the mark delete position of a replicated subscription might not get updated to the remote cluster. The role of the replication snapshot cache is visible in the diagram. + +
+
+```mermaid +sequenceDiagram + actor ProducerC1 + actor ConsumerC1 + + box Broker c1 + participant PT_C1 as PersistentTopicC1 + participant RSC_C1 as ReplicatedSubscriptionsControllerC1 + participant Disp_C1 as DispatcherC1 (for my-sub) + participant PS_C1 as PersistentSubscriptionC1 (sub: my-sub) + participant RSCache_C1 as ReplicatedSubscriptionSnapshotCacheC1 (for my-sub) + participant Repl_C1_to_C2 as Replicator_C1_to_C2 + end + + box Broker c2 + participant PT_C2 as PersistentTopicC2 + participant RSC_C2 as ReplicatedSubscriptionsControllerC2 + participant PS_C2 as PersistentSubscriptionC2 (sub: my-sub) + participant Repl_C2_to_C1 as Replicator_C2_to_C1 + end + + %% --- Periodic Snapshot Cycle (replicatedSubscriptionsSnapshotFrequencyMillis) --- + loop Snapshot Attempt (ID: S1) + RSC_C1 ->> RSC_C1: startNewSnapshot() + RSC_C1 ->> PT_C1: publishMessage(Marker: SNAPSHOT_REQUEST S1, from:c1) + PT_C1 -->> Repl_C1_to_C2: Read SNAPSHOT_REQUEST S1 + Repl_C1_to_C2 ->> PT_C2: Deliver SNAPSHOT_REQUEST S1 (writes to PT_C2) + + PT_C2 -->> RSC_C2: PT_C2.receivedReplicatedSubscriptionMarker(SNAPSHOT_REQUEST S1) + RSC_C2 ->> RSC_C2: receivedSnapshotRequest(S1) + RSC_C2 ->> PT_C2: PT_C2.publishMessage(Marker: SNAPSHOT_RESPONSE S1, cluster:c2, msgId:LastMsgID_c2_T0) + + PT_C2 -->> Repl_C2_to_C1: Read SNAPSHOT_RESPONSE S1 + Repl_C2_to_C1 ->> PT_C1: Deliver SNAPSHOT_RESPONSE S1 (writes to PT_C1) + + PT_C1 -->> RSC_C1: receivedReplicatedSubscriptionMarker(SNAPSHOT_RESPONSE S1, Pos_Marker_c1_T1) + RSC_C1 ->> RSC_C1: receivedSnapshotResponse(S1, c2, LastMsgID_c2_T0, Pos_Marker_c1_T1) + opt All Responses Received + RSC_C1 ->> PT_C1: publishMessage(Marker: REPLICATED_SUBSCRIPTION_SNAPSHOT S1, local_pos:Pos_Marker_c1_T1, c2_remote_pos:LastMsgID_c2_T0) + RSC_C1 ->> RSC_C1: snapshotCompleted(S1) + end + end + + PT_C1 -->> Disp_C1: Read REPLICATED_SUBSCRIPTION_SNAPSHOT S1 + Disp_C1 ->> Disp_C1: filterEntriesForConsumer / processReplicatedSubscriptionSnapshot(S1) + Disp_C1 ->> PS_C1: processReplicatedSubscriptionSnapshot(S1) + PS_C1 ->> RSCache_C1: addNewSnapshot(S1) + + %% --- Message Production on c1 & Replication to c2 --- + ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg1 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) + + ProducerC1 ->> PT_C1: publishMessage(Msg2) (at Pos_M2_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg2 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg2 (writes to PT_C2 at Pos_M2_c2) + + %% --- Consumption and Acknowledgement on c1 --- + PT_C1 -->> Disp_C1: Read Msg1 for dispatch + Disp_C1 -->> ConsumerC1: Dispatch Msg1 (Pos_M1_c1) + ConsumerC1 ->> PS_C1: acknowledgeMessage(Pos_M1_c1, Individual) + Note right of PS_C1: markDeletePosition may not advance yet + + PT_C1 -->> Disp_C1: Read Msg2 for dispatch + Disp_C1 -->> ConsumerC1: Dispatch Msg2 (Pos_M2_c1) + ConsumerC1 ->> PS_C1: acknowledgeMessage(Pos_M2_c1, Individual) + PS_C1 ->> PS_C1: cursor markDeletePosition advances to Pos_MDP_c1 + + %% --- Subscription State Replication (triggered by markDeletePosition advance on c1) --- + PS_C1 ->> RSCache_C1: advancedMarkDeletePosition(Pos_MDP_c1) + RSCache_C1 -->> PS_C1: Return Snapshot S1 (if Pos_MDP_c1 >= S1.local_pos) + Note right of PS_C1: Suitable snapshot S1 may not be present in the cache.
In that case, the subscription update won't be replicated. + + opt Snapshot S1 is present. This flow won't happen unless a suitable snapshot is present in the cache. + PS_C1 ->> RSC_C1: localSubscriptionUpdated("my-sub", S1) + RSC_C1 ->> PT_C1: publishMessage(Marker: REPLICATED_SUBSCRIPTION_UPDATE for "my-sub", c2_target_pos: S1.c2_remote_pos) + PT_C1 -->> Repl_C1_to_C2: Read SUBSCRIPTION_UPDATE + Repl_C1_to_C2 ->> PT_C2: Deliver SUBSCRIPTION_UPDATE (writes to PT_C2) + + PT_C2 -->> RSC_C2: PT_C2.receivedReplicatedSubscriptionMarker(SUBSCRIPTION_UPDATE) + RSC_C2 ->> RSC_C2: receiveSubscriptionUpdated("my-sub", S1.c2_remote_pos) + RSC_C2 ->> PS_C2: acknowledgeMessage([S1.c2_remote_pos], Cumulative, ...) + Note right of PS_C2: markDeletePosition on c2 advances + end +``` +
+
+ ### Observability Observability for replicated subscriptions is limited. For debugging, debug-level logs are available in `org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController`, though these are not suitable for production operations. From e78542a9033db235215388229fdb9e4c7ca7cfb4 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 16:30:37 +0200 Subject: [PATCH 38/59] Improve --- docs/administration-geo.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index aaa5f8c3a2bb..3a1b205cde98 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -368,15 +368,11 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. -A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. +A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This occurs with shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and with delayed message delivery. Previously, if the mark-delete position did not advance past a snapshot's local position before cached snapshots expired, no usable snapshot remained. A snapshot is usable when the mark-delete position has reached or passed the position at which the snapshot was created. -After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. - -The improve snapshot cache expiration policy introduced in [PR #25044](https://github.com/apache/pulsar/pull/25044) doesn't address the challenges with messaged delivered with the delayed message delivery feature. The reason for this is that replication snapshots are added to the cache when the replicator for the particular topic reads the replication snapshot marker messages from the topic. When delayed message delivery is used, there might be a very large backlog and the topic might have been unloaded or moved to another broker in the cluster due to broker restarts or load shedding. In that case, there won't be any entries in the replication snapshot cache and it won't get restored since the replication cursor has already moved further ahead. - -This is the reason why replicated subscriptions don't provide consistent behavior when using delayed message delivery. In the case of shared or key-shared subscriptions, the mark delete position moves ahead fairly frequently compared to delayed message delivery and that's why the problem is addressed with PR #25044 and the new snapshot cache expiration policy. After a topic unload or topic getting moved to another broker the replication snapshot cache would eventually fill with suitable snapshots which can be used to replicate the subscription state. +The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from just ahead of the current mark-delete position to the latest snapshot — when the cache is full. The nearest cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger cache, subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. The following broker settings control snapshot behavior: @@ -390,7 +386,7 @@ For setups with more than two clusters, it is recommended to increase `replicate ### Replicated subscriptions sequence diagrams -This sequence diagram describes the interactions of the replicated subscription solution. It could help understand the detail of why the mark delete position of a replicated subscription might not get updated to the remote cluster. The role of the replication snapshot cache is visible in the diagram. +This sequence diagram illustrates the interactions involved in replicating subscription state across two clusters. It shows how the mark-delete position on one cluster is propagated to the other, and makes the role of the replication snapshot cache visible — including why a subscription update may not be replicated if no suitable snapshot is available. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position.
From e287e2175af76eaa5e490b675bf66c3044cf5c9b Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 16:30:44 +0200 Subject: [PATCH 39/59] Sync versioned docs --- .../version-4.0.x/administration-geo.md | 100 +++++++++++++++++- .../version-4.1.x/administration-geo.md | 100 +++++++++++++++++- 2 files changed, 194 insertions(+), 6 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 83399da6423f..3a1b205cde98 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -368,11 +368,11 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. -A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. +A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This occurs with shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and with delayed message delivery. Previously, if the mark-delete position did not advance past a snapshot's local position before cached snapshots expired, no usable snapshot remained. A snapshot is usable when the mark-delete position has reached or passed the position at which the snapshot was created. -After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. +The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from just ahead of the current mark-delete position to the latest snapshot — when the cache is full. The nearest cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger cache, subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. The following broker settings control snapshot behavior: @@ -384,6 +384,100 @@ The following broker settings control snapshot behavior: For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. +### Replicated subscriptions sequence diagrams + +This sequence diagram illustrates the interactions involved in replicating subscription state across two clusters. It shows how the mark-delete position on one cluster is propagated to the other, and makes the role of the replication snapshot cache visible — including why a subscription update may not be replicated if no suitable snapshot is available. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position. + +
+
+```mermaid +sequenceDiagram + actor ProducerC1 + actor ConsumerC1 + + box Broker c1 + participant PT_C1 as PersistentTopicC1 + participant RSC_C1 as ReplicatedSubscriptionsControllerC1 + participant Disp_C1 as DispatcherC1 (for my-sub) + participant PS_C1 as PersistentSubscriptionC1 (sub: my-sub) + participant RSCache_C1 as ReplicatedSubscriptionSnapshotCacheC1 (for my-sub) + participant Repl_C1_to_C2 as Replicator_C1_to_C2 + end + + box Broker c2 + participant PT_C2 as PersistentTopicC2 + participant RSC_C2 as ReplicatedSubscriptionsControllerC2 + participant PS_C2 as PersistentSubscriptionC2 (sub: my-sub) + participant Repl_C2_to_C1 as Replicator_C2_to_C1 + end + + %% --- Periodic Snapshot Cycle (replicatedSubscriptionsSnapshotFrequencyMillis) --- + loop Snapshot Attempt (ID: S1) + RSC_C1 ->> RSC_C1: startNewSnapshot() + RSC_C1 ->> PT_C1: publishMessage(Marker: SNAPSHOT_REQUEST S1, from:c1) + PT_C1 -->> Repl_C1_to_C2: Read SNAPSHOT_REQUEST S1 + Repl_C1_to_C2 ->> PT_C2: Deliver SNAPSHOT_REQUEST S1 (writes to PT_C2) + + PT_C2 -->> RSC_C2: PT_C2.receivedReplicatedSubscriptionMarker(SNAPSHOT_REQUEST S1) + RSC_C2 ->> RSC_C2: receivedSnapshotRequest(S1) + RSC_C2 ->> PT_C2: PT_C2.publishMessage(Marker: SNAPSHOT_RESPONSE S1, cluster:c2, msgId:LastMsgID_c2_T0) + + PT_C2 -->> Repl_C2_to_C1: Read SNAPSHOT_RESPONSE S1 + Repl_C2_to_C1 ->> PT_C1: Deliver SNAPSHOT_RESPONSE S1 (writes to PT_C1) + + PT_C1 -->> RSC_C1: receivedReplicatedSubscriptionMarker(SNAPSHOT_RESPONSE S1, Pos_Marker_c1_T1) + RSC_C1 ->> RSC_C1: receivedSnapshotResponse(S1, c2, LastMsgID_c2_T0, Pos_Marker_c1_T1) + opt All Responses Received + RSC_C1 ->> PT_C1: publishMessage(Marker: REPLICATED_SUBSCRIPTION_SNAPSHOT S1, local_pos:Pos_Marker_c1_T1, c2_remote_pos:LastMsgID_c2_T0) + RSC_C1 ->> RSC_C1: snapshotCompleted(S1) + end + end + + PT_C1 -->> Disp_C1: Read REPLICATED_SUBSCRIPTION_SNAPSHOT S1 + Disp_C1 ->> Disp_C1: filterEntriesForConsumer / processReplicatedSubscriptionSnapshot(S1) + Disp_C1 ->> PS_C1: processReplicatedSubscriptionSnapshot(S1) + PS_C1 ->> RSCache_C1: addNewSnapshot(S1) + + %% --- Message Production on c1 & Replication to c2 --- + ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg1 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) + + ProducerC1 ->> PT_C1: publishMessage(Msg2) (at Pos_M2_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg2 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg2 (writes to PT_C2 at Pos_M2_c2) + + %% --- Consumption and Acknowledgement on c1 --- + PT_C1 -->> Disp_C1: Read Msg1 for dispatch + Disp_C1 -->> ConsumerC1: Dispatch Msg1 (Pos_M1_c1) + ConsumerC1 ->> PS_C1: acknowledgeMessage(Pos_M1_c1, Individual) + Note right of PS_C1: markDeletePosition may not advance yet + + PT_C1 -->> Disp_C1: Read Msg2 for dispatch + Disp_C1 -->> ConsumerC1: Dispatch Msg2 (Pos_M2_c1) + ConsumerC1 ->> PS_C1: acknowledgeMessage(Pos_M2_c1, Individual) + PS_C1 ->> PS_C1: cursor markDeletePosition advances to Pos_MDP_c1 + + %% --- Subscription State Replication (triggered by markDeletePosition advance on c1) --- + PS_C1 ->> RSCache_C1: advancedMarkDeletePosition(Pos_MDP_c1) + RSCache_C1 -->> PS_C1: Return Snapshot S1 (if Pos_MDP_c1 >= S1.local_pos) + Note right of PS_C1: Suitable snapshot S1 may not be present in the cache.
In that case, the subscription update won't be replicated. + + opt Snapshot S1 is present. This flow won't happen unless a suitable snapshot is present in the cache. + PS_C1 ->> RSC_C1: localSubscriptionUpdated("my-sub", S1) + RSC_C1 ->> PT_C1: publishMessage(Marker: REPLICATED_SUBSCRIPTION_UPDATE for "my-sub", c2_target_pos: S1.c2_remote_pos) + PT_C1 -->> Repl_C1_to_C2: Read SUBSCRIPTION_UPDATE + Repl_C1_to_C2 ->> PT_C2: Deliver SUBSCRIPTION_UPDATE (writes to PT_C2) + + PT_C2 -->> RSC_C2: PT_C2.receivedReplicatedSubscriptionMarker(SUBSCRIPTION_UPDATE) + RSC_C2 ->> RSC_C2: receiveSubscriptionUpdated("my-sub", S1.c2_remote_pos) + RSC_C2 ->> PS_C2: acknowledgeMessage([S1.c2_remote_pos], Cumulative, ...) + Note right of PS_C2: markDeletePosition on c2 advances + end +``` +
+
+ ### Observability Observability for replicated subscriptions is limited. For debugging, debug-level logs are available in `org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController`, though these are not suitable for production operations. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 83399da6423f..3a1b205cde98 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -368,11 +368,11 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires either one or two rounds of round-trips between the participating clusters. When more than two clusters are involved, two rounds are required. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. -A known issue where the snapshot condition was not met under high message rates with shared or key-shared subscriptions using individual acknowledgments was fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044). In that scenario, the cursor's mark-delete position advances slowly because all acknowledgment gaps must be filled before it can move forward. Previously, if the mark-delete position did not advance before cached snapshots expired, the subscription state would stop being replicated even after the position eventually moved forward. +A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This occurs with shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and with delayed message delivery. Previously, if the mark-delete position did not advance past a snapshot's local position before cached snapshots expired, no usable snapshot remained. A snapshot is usable when the mark-delete position has reached or passed the position at which the snapshot was created. -After PR #25044, when the snapshot cache is full, snapshots are retained spread across the full range from just ahead of the current mark-delete position to the latest snapshot. This means that when the cursor eventually advances, a usable snapshot is always available nearby. With a larger cache, the gap between a usable snapshot and the actual mark-delete position is smaller, which reduces both the replication lag and the number of potential duplicate messages on failover. +The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from just ahead of the current mark-delete position to the latest snapshot — when the cache is full. The nearest cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger cache, subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. The following broker settings control snapshot behavior: @@ -384,6 +384,100 @@ The following broker settings control snapshot behavior: For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. +### Replicated subscriptions sequence diagrams + +This sequence diagram illustrates the interactions involved in replicating subscription state across two clusters. It shows how the mark-delete position on one cluster is propagated to the other, and makes the role of the replication snapshot cache visible — including why a subscription update may not be replicated if no suitable snapshot is available. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position. + +
+
+```mermaid +sequenceDiagram + actor ProducerC1 + actor ConsumerC1 + + box Broker c1 + participant PT_C1 as PersistentTopicC1 + participant RSC_C1 as ReplicatedSubscriptionsControllerC1 + participant Disp_C1 as DispatcherC1 (for my-sub) + participant PS_C1 as PersistentSubscriptionC1 (sub: my-sub) + participant RSCache_C1 as ReplicatedSubscriptionSnapshotCacheC1 (for my-sub) + participant Repl_C1_to_C2 as Replicator_C1_to_C2 + end + + box Broker c2 + participant PT_C2 as PersistentTopicC2 + participant RSC_C2 as ReplicatedSubscriptionsControllerC2 + participant PS_C2 as PersistentSubscriptionC2 (sub: my-sub) + participant Repl_C2_to_C1 as Replicator_C2_to_C1 + end + + %% --- Periodic Snapshot Cycle (replicatedSubscriptionsSnapshotFrequencyMillis) --- + loop Snapshot Attempt (ID: S1) + RSC_C1 ->> RSC_C1: startNewSnapshot() + RSC_C1 ->> PT_C1: publishMessage(Marker: SNAPSHOT_REQUEST S1, from:c1) + PT_C1 -->> Repl_C1_to_C2: Read SNAPSHOT_REQUEST S1 + Repl_C1_to_C2 ->> PT_C2: Deliver SNAPSHOT_REQUEST S1 (writes to PT_C2) + + PT_C2 -->> RSC_C2: PT_C2.receivedReplicatedSubscriptionMarker(SNAPSHOT_REQUEST S1) + RSC_C2 ->> RSC_C2: receivedSnapshotRequest(S1) + RSC_C2 ->> PT_C2: PT_C2.publishMessage(Marker: SNAPSHOT_RESPONSE S1, cluster:c2, msgId:LastMsgID_c2_T0) + + PT_C2 -->> Repl_C2_to_C1: Read SNAPSHOT_RESPONSE S1 + Repl_C2_to_C1 ->> PT_C1: Deliver SNAPSHOT_RESPONSE S1 (writes to PT_C1) + + PT_C1 -->> RSC_C1: receivedReplicatedSubscriptionMarker(SNAPSHOT_RESPONSE S1, Pos_Marker_c1_T1) + RSC_C1 ->> RSC_C1: receivedSnapshotResponse(S1, c2, LastMsgID_c2_T0, Pos_Marker_c1_T1) + opt All Responses Received + RSC_C1 ->> PT_C1: publishMessage(Marker: REPLICATED_SUBSCRIPTION_SNAPSHOT S1, local_pos:Pos_Marker_c1_T1, c2_remote_pos:LastMsgID_c2_T0) + RSC_C1 ->> RSC_C1: snapshotCompleted(S1) + end + end + + PT_C1 -->> Disp_C1: Read REPLICATED_SUBSCRIPTION_SNAPSHOT S1 + Disp_C1 ->> Disp_C1: filterEntriesForConsumer / processReplicatedSubscriptionSnapshot(S1) + Disp_C1 ->> PS_C1: processReplicatedSubscriptionSnapshot(S1) + PS_C1 ->> RSCache_C1: addNewSnapshot(S1) + + %% --- Message Production on c1 & Replication to c2 --- + ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg1 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) + + ProducerC1 ->> PT_C1: publishMessage(Msg2) (at Pos_M2_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg2 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg2 (writes to PT_C2 at Pos_M2_c2) + + %% --- Consumption and Acknowledgement on c1 --- + PT_C1 -->> Disp_C1: Read Msg1 for dispatch + Disp_C1 -->> ConsumerC1: Dispatch Msg1 (Pos_M1_c1) + ConsumerC1 ->> PS_C1: acknowledgeMessage(Pos_M1_c1, Individual) + Note right of PS_C1: markDeletePosition may not advance yet + + PT_C1 -->> Disp_C1: Read Msg2 for dispatch + Disp_C1 -->> ConsumerC1: Dispatch Msg2 (Pos_M2_c1) + ConsumerC1 ->> PS_C1: acknowledgeMessage(Pos_M2_c1, Individual) + PS_C1 ->> PS_C1: cursor markDeletePosition advances to Pos_MDP_c1 + + %% --- Subscription State Replication (triggered by markDeletePosition advance on c1) --- + PS_C1 ->> RSCache_C1: advancedMarkDeletePosition(Pos_MDP_c1) + RSCache_C1 -->> PS_C1: Return Snapshot S1 (if Pos_MDP_c1 >= S1.local_pos) + Note right of PS_C1: Suitable snapshot S1 may not be present in the cache.
In that case, the subscription update won't be replicated. + + opt Snapshot S1 is present. This flow won't happen unless a suitable snapshot is present in the cache. + PS_C1 ->> RSC_C1: localSubscriptionUpdated("my-sub", S1) + RSC_C1 ->> PT_C1: publishMessage(Marker: REPLICATED_SUBSCRIPTION_UPDATE for "my-sub", c2_target_pos: S1.c2_remote_pos) + PT_C1 -->> Repl_C1_to_C2: Read SUBSCRIPTION_UPDATE + Repl_C1_to_C2 ->> PT_C2: Deliver SUBSCRIPTION_UPDATE (writes to PT_C2) + + PT_C2 -->> RSC_C2: PT_C2.receivedReplicatedSubscriptionMarker(SUBSCRIPTION_UPDATE) + RSC_C2 ->> RSC_C2: receiveSubscriptionUpdated("my-sub", S1.c2_remote_pos) + RSC_C2 ->> PS_C2: acknowledgeMessage([S1.c2_remote_pos], Cumulative, ...) + Note right of PS_C2: markDeletePosition on c2 advances + end +``` +
+
+ ### Observability Observability for replicated subscriptions is limited. For debugging, debug-level logs are available in `org.apache.pulsar.broker.service.persistent.ReplicatedSubscriptionsController`, though these are not suitable for production operations. From ecfe135eb090657708d32c4042e20cdd1f021d48 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 16:34:05 +0200 Subject: [PATCH 40/59] Improve --- docs/administration-geo.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 3a1b205cde98..f04f877983bb 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -411,6 +411,11 @@ sequenceDiagram participant Repl_C2_to_C1 as Replicator_C2_to_C1 end + %% --- Message Production on c1 & Replication to c2 --- + ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg1 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) + %% --- Periodic Snapshot Cycle (replicatedSubscriptionsSnapshotFrequencyMillis) --- loop Snapshot Attempt (ID: S1) RSC_C1 ->> RSC_C1: startNewSnapshot() @@ -438,11 +443,7 @@ sequenceDiagram Disp_C1 ->> PS_C1: processReplicatedSubscriptionSnapshot(S1) PS_C1 ->> RSCache_C1: addNewSnapshot(S1) - %% --- Message Production on c1 & Replication to c2 --- - ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) - PT_C1 -->> Repl_C1_to_C2: Read Msg1 - Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) - + %% --- Continued message Production on c1 & Replication to c2 --- ProducerC1 ->> PT_C1: publishMessage(Msg2) (at Pos_M2_c1) PT_C1 -->> Repl_C1_to_C2: Read Msg2 Repl_C1_to_C2 ->> PT_C2: Deliver Msg2 (writes to PT_C2 at Pos_M2_c2) From 4c798c053d58b0deb342c5f9f1e404946712390c Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 16:34:13 +0200 Subject: [PATCH 41/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 11 ++++++----- versioned_docs/version-4.1.x/administration-geo.md | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 3a1b205cde98..f04f877983bb 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -411,6 +411,11 @@ sequenceDiagram participant Repl_C2_to_C1 as Replicator_C2_to_C1 end + %% --- Message Production on c1 & Replication to c2 --- + ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg1 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) + %% --- Periodic Snapshot Cycle (replicatedSubscriptionsSnapshotFrequencyMillis) --- loop Snapshot Attempt (ID: S1) RSC_C1 ->> RSC_C1: startNewSnapshot() @@ -438,11 +443,7 @@ sequenceDiagram Disp_C1 ->> PS_C1: processReplicatedSubscriptionSnapshot(S1) PS_C1 ->> RSCache_C1: addNewSnapshot(S1) - %% --- Message Production on c1 & Replication to c2 --- - ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) - PT_C1 -->> Repl_C1_to_C2: Read Msg1 - Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) - + %% --- Continued message Production on c1 & Replication to c2 --- ProducerC1 ->> PT_C1: publishMessage(Msg2) (at Pos_M2_c1) PT_C1 -->> Repl_C1_to_C2: Read Msg2 Repl_C1_to_C2 ->> PT_C2: Deliver Msg2 (writes to PT_C2 at Pos_M2_c2) diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 3a1b205cde98..f04f877983bb 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -411,6 +411,11 @@ sequenceDiagram participant Repl_C2_to_C1 as Replicator_C2_to_C1 end + %% --- Message Production on c1 & Replication to c2 --- + ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) + PT_C1 -->> Repl_C1_to_C2: Read Msg1 + Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) + %% --- Periodic Snapshot Cycle (replicatedSubscriptionsSnapshotFrequencyMillis) --- loop Snapshot Attempt (ID: S1) RSC_C1 ->> RSC_C1: startNewSnapshot() @@ -438,11 +443,7 @@ sequenceDiagram Disp_C1 ->> PS_C1: processReplicatedSubscriptionSnapshot(S1) PS_C1 ->> RSCache_C1: addNewSnapshot(S1) - %% --- Message Production on c1 & Replication to c2 --- - ProducerC1 ->> PT_C1: publishMessage(Msg1) (at Pos_M1_c1) - PT_C1 -->> Repl_C1_to_C2: Read Msg1 - Repl_C1_to_C2 ->> PT_C2: Deliver Msg1 (writes to PT_C2 at Pos_M1_c2) - + %% --- Continued message Production on c1 & Replication to c2 --- ProducerC1 ->> PT_C1: publishMessage(Msg2) (at Pos_M2_c1) PT_C1 -->> Repl_C1_to_C2: Read Msg2 Repl_C1_to_C2 ->> PT_C2: Deliver Msg2 (writes to PT_C2 at Pos_M2_c2) From bdde7f52823b0c568dca222b55d717f2e985f0f9 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 16:58:08 +0200 Subject: [PATCH 42/59] Improve --- docs/administration-geo.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index f04f877983bb..0e8771bb6c2e 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -370,16 +370,20 @@ Replicated subscriptions use a periodic snapshotting mechanism to establish a co Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. -A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This occurs with shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and with delayed message delivery. Previously, if the mark-delete position did not advance past a snapshot's local position before cached snapshots expired, no usable snapshot remained. A snapshot is usable when the mark-delete position has reached or passed the position at which the snapshot was created. +The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. -The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from just ahead of the current mark-delete position to the latest snapshot — when the cache is full. The nearest cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger cache, subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. +A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This affects shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and topics using delayed message delivery. The original cache expiration policy kept only the `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` most recently created snapshots, evicting older ones. As a result, all cached snapshots could end up ahead of the mark-delete position if the mark-delete position had not advanced within the last `replicatedSubscriptionsSnapshotFrequencyMillis × replicatedSubscriptionsSnapshotMaxCachedPerSubscription` milliseconds (10 seconds with default settings), leaving no suitable snapshot available. + +The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from ahead of the current mark-delete position to the latest snapshot — when the cache is full. The first cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger snapshot cache size (`replicatedSubscriptionsSnapshotMaxCachedPerSubscription`), subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. + +[PR #25044](https://github.com/apache/pulsar/pull/25044) also reduced the memory footprint of each snapshot entry to approximately 200 bytes, making it practical to increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` well beyond the default of `30` when finer snapshot granularity is needed for large backlogs. The tradeoff is higher heap memory consumption, which should be accounted for when sizing the broker. The current implementation uses a fixed per-subscription limit rather than a global memory budget; a future improvement could cap total cache memory consumption and apply the same spread-based eviction strategy once the limit is reached. The following broker settings control snapshot behavior: | Setting | Default | Description | | --- | --- | --- | | `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | -| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. From 0411995b787893c51ef79cf2374f1bbe30ceb638 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 16:58:17 +0200 Subject: [PATCH 43/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 10 +++++++--- versioned_docs/version-4.1.x/administration-geo.md | 10 +++++++--- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index f04f877983bb..0e8771bb6c2e 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -370,16 +370,20 @@ Replicated subscriptions use a periodic snapshotting mechanism to establish a co Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. -A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This occurs with shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and with delayed message delivery. Previously, if the mark-delete position did not advance past a snapshot's local position before cached snapshots expired, no usable snapshot remained. A snapshot is usable when the mark-delete position has reached or passed the position at which the snapshot was created. +The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. -The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from just ahead of the current mark-delete position to the latest snapshot — when the cache is full. The nearest cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger cache, subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. +A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This affects shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and topics using delayed message delivery. The original cache expiration policy kept only the `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` most recently created snapshots, evicting older ones. As a result, all cached snapshots could end up ahead of the mark-delete position if the mark-delete position had not advanced within the last `replicatedSubscriptionsSnapshotFrequencyMillis × replicatedSubscriptionsSnapshotMaxCachedPerSubscription` milliseconds (10 seconds with default settings), leaving no suitable snapshot available. + +The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from ahead of the current mark-delete position to the latest snapshot — when the cache is full. The first cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger snapshot cache size (`replicatedSubscriptionsSnapshotMaxCachedPerSubscription`), subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. + +[PR #25044](https://github.com/apache/pulsar/pull/25044) also reduced the memory footprint of each snapshot entry to approximately 200 bytes, making it practical to increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` well beyond the default of `30` when finer snapshot granularity is needed for large backlogs. The tradeoff is higher heap memory consumption, which should be accounted for when sizing the broker. The current implementation uses a fixed per-subscription limit rather than a global memory budget; a future improvement could cap total cache memory consumption and apply the same spread-based eviction strategy once the limit is reached. The following broker settings control snapshot behavior: | Setting | Default | Description | | --- | --- | --- | | `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | -| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index f04f877983bb..0e8771bb6c2e 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -370,16 +370,20 @@ Replicated subscriptions use a periodic snapshotting mechanism to establish a co Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. -A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This occurs with shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and with delayed message delivery. Previously, if the mark-delete position did not advance past a snapshot's local position before cached snapshots expired, no usable snapshot remained. A snapshot is usable when the mark-delete position has reached or passed the position at which the snapshot was created. +The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. -The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from just ahead of the current mark-delete position to the latest snapshot — when the cache is full. The nearest cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger cache, subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. +A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This affects shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and topics using delayed message delivery. The original cache expiration policy kept only the `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` most recently created snapshots, evicting older ones. As a result, all cached snapshots could end up ahead of the mark-delete position if the mark-delete position had not advanced within the last `replicatedSubscriptionsSnapshotFrequencyMillis × replicatedSubscriptionsSnapshotMaxCachedPerSubscription` milliseconds (10 seconds with default settings), leaving no suitable snapshot available. + +The improved snapshot cache expiration policy introduced by [PR #25044](https://github.com/apache/pulsar/pull/25044) addresses this by retaining snapshots spread across the full backlog range — from ahead of the current mark-delete position to the latest snapshot — when the cache is full. The first cached snapshot is always retained until the mark-delete position advances past it, guaranteeing that progress is eventually made even with a large backlog. With a larger snapshot cache size (`replicatedSubscriptionsSnapshotMaxCachedPerSubscription`), subsequent snapshots are spread more densely, which allows replication to proceed more frequently and reduces both replication lag and the number of potential duplicate messages on failover. + +[PR #25044](https://github.com/apache/pulsar/pull/25044) also reduced the memory footprint of each snapshot entry to approximately 200 bytes, making it practical to increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` well beyond the default of `30` when finer snapshot granularity is needed for large backlogs. The tradeoff is higher heap memory consumption, which should be accounted for when sizing the broker. The current implementation uses a fixed per-subscription limit rather than a global memory budget; a future improvement could cap total cache memory consumption and apply the same spread-based eviction strategy once the limit is reached. The following broker settings control snapshot behavior: | Setting | Default | Description | | --- | --- | --- | | `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | -| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. From af9f65c7e0f7c599ab0cafcad2e84c31f4a6fa23 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 18:39:28 +0200 Subject: [PATCH 44/59] Improve --- docs/administration-geo.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 0e8771bb6c2e..9964285744c3 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -353,10 +353,10 @@ The advantages of replicated subscription are as follows. The limitations of replicated subscription are as follows. -* When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. -* Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. -* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may be processed in either cluster if the replication subscription update reaches the other cluster before the messages are consumed there. It is recommended to process messages in a single cluster when using replicated subscriptions. -* Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. +* Replicated subscriptions use periodic snapshots to establish a consistent association between message positions across clusters. Snapshots are taken every `replicatedSubscriptionsSnapshotFrequencyMillis` milliseconds (default: 1000 ms), but the effective granularity of mark-delete position updates on the remote cluster depends on how many messages are produced between snapshots. See [Replicated subscriptions snapshot configuration and tuning](#replicated-subscriptions-snapshot-configuration-and-tuning) for details. +* Only the mark-delete position (the baseline cursor position) is replicated — individual acknowledgments are not. Messages acknowledged out of order may be redelivered after a cluster failover. +* Replicated subscriptions do not provide consistent behavior when consumers are active on multiple clusters simultaneously. Most messages will be processed on both clusters (duplicate processing), and some may be processed on either cluster depending on replication timing. To avoid this, process messages on a single cluster at a time when using replicated subscriptions. +* Delayed message delivery impairs subscription replication. The mark-delete position does not advance until delayed messages have been delivered and acknowledged, so replication lags behind accordingly. :::note @@ -368,7 +368,11 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. If any participating cluster is offline, snapshots cannot be established and the mark-delete position will not be propagated until connectivity is restored. + +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. + +A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. @@ -378,6 +382,8 @@ The improved snapshot cache expiration policy introduced by [PR #25044](https:// [PR #25044](https://github.com/apache/pulsar/pull/25044) also reduced the memory footprint of each snapshot entry to approximately 200 bytes, making it practical to increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` well beyond the default of `30` when finer snapshot granularity is needed for large backlogs. The tradeoff is higher heap memory consumption, which should be accounted for when sizing the broker. The current implementation uses a fixed per-subscription limit rather than a global memory budget; a future improvement could cap total cache memory consumption and apply the same spread-based eviction strategy once the limit is reached. +If geo-replication is enabled on a namespace that already contains messages, no snapshot markers will be present in the existing backlog. The mark-delete position cannot be propagated until the replicator has read past the pre-existing messages and new snapshots have been written and processed — regardless of the snapshot cache size. Similarly, neither the improved cache eviction policy nor increasing the cache size addresses the long interval between snapshots under bursty traffic; that continues to affect the lag of mark-delete position updates to remote clusters. + The following broker settings control snapshot behavior: | Setting | Default | Description | @@ -386,7 +392,10 @@ The following broker settings control snapshot behavior: | `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | -For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. +Tuning recommendations: + +* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshots complete before timing out. +* **Large backlogs with slow mark-delete advancement** (shared or key-shared subscriptions with individual acknowledgments, or delayed message delivery): Increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to at least `50` so that cached snapshots are spread more densely and a suitable snapshot is more likely to be close to the mark-delete position when it advances. At a value of `50`, the snapshot cache consumes approximately 10 KB of heap memory per subscription; with the default of `30`, the footprint is approximately 6 KB per subscription. ### Replicated subscriptions sequence diagrams From 0aa137be27ec60b14b075eee4783c0afba37e2f4 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 18:39:35 +0200 Subject: [PATCH 45/59] Sync versioned docs --- .../version-4.0.x/administration-geo.md | 21 +++++++++++++------ .../version-4.1.x/administration-geo.md | 21 +++++++++++++------ 2 files changed, 30 insertions(+), 12 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 0e8771bb6c2e..9964285744c3 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -353,10 +353,10 @@ The advantages of replicated subscription are as follows. The limitations of replicated subscription are as follows. -* When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. -* Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. -* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may be processed in either cluster if the replication subscription update reaches the other cluster before the messages are consumed there. It is recommended to process messages in a single cluster when using replicated subscriptions. -* Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. +* Replicated subscriptions use periodic snapshots to establish a consistent association between message positions across clusters. Snapshots are taken every `replicatedSubscriptionsSnapshotFrequencyMillis` milliseconds (default: 1000 ms), but the effective granularity of mark-delete position updates on the remote cluster depends on how many messages are produced between snapshots. See [Replicated subscriptions snapshot configuration and tuning](#replicated-subscriptions-snapshot-configuration-and-tuning) for details. +* Only the mark-delete position (the baseline cursor position) is replicated — individual acknowledgments are not. Messages acknowledged out of order may be redelivered after a cluster failover. +* Replicated subscriptions do not provide consistent behavior when consumers are active on multiple clusters simultaneously. Most messages will be processed on both clusters (duplicate processing), and some may be processed on either cluster depending on replication timing. To avoid this, process messages on a single cluster at a time when using replicated subscriptions. +* Delayed message delivery impairs subscription replication. The mark-delete position does not advance until delayed messages have been delivered and acknowledged, so replication lags behind accordingly. :::note @@ -368,7 +368,11 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. If any participating cluster is offline, snapshots cannot be established and the mark-delete position will not be propagated until connectivity is restored. + +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. + +A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. @@ -378,6 +382,8 @@ The improved snapshot cache expiration policy introduced by [PR #25044](https:// [PR #25044](https://github.com/apache/pulsar/pull/25044) also reduced the memory footprint of each snapshot entry to approximately 200 bytes, making it practical to increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` well beyond the default of `30` when finer snapshot granularity is needed for large backlogs. The tradeoff is higher heap memory consumption, which should be accounted for when sizing the broker. The current implementation uses a fixed per-subscription limit rather than a global memory budget; a future improvement could cap total cache memory consumption and apply the same spread-based eviction strategy once the limit is reached. +If geo-replication is enabled on a namespace that already contains messages, no snapshot markers will be present in the existing backlog. The mark-delete position cannot be propagated until the replicator has read past the pre-existing messages and new snapshots have been written and processed — regardless of the snapshot cache size. Similarly, neither the improved cache eviction policy nor increasing the cache size addresses the long interval between snapshots under bursty traffic; that continues to affect the lag of mark-delete position updates to remote clusters. + The following broker settings control snapshot behavior: | Setting | Default | Description | @@ -386,7 +392,10 @@ The following broker settings control snapshot behavior: | `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | -For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. +Tuning recommendations: + +* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshots complete before timing out. +* **Large backlogs with slow mark-delete advancement** (shared or key-shared subscriptions with individual acknowledgments, or delayed message delivery): Increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to at least `50` so that cached snapshots are spread more densely and a suitable snapshot is more likely to be close to the mark-delete position when it advances. At a value of `50`, the snapshot cache consumes approximately 10 KB of heap memory per subscription; with the default of `30`, the footprint is approximately 6 KB per subscription. ### Replicated subscriptions sequence diagrams diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 0e8771bb6c2e..9964285744c3 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -353,10 +353,10 @@ The advantages of replicated subscription are as follows. The limitations of replicated subscription are as follows. -* When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is `1 second`. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the `broker.conf` file. -* Only the base line cursor position ("mark delete position") is synced in replicated subscriptions while the individual acknowledgments are not synced. This means the messages acknowledged out-of-order could end up getting delivered again, in the case of a cluster failover. -* Replicated subscriptions do not support consistent behavior when consumers are active on multiple clusters simultaneously. Most messages would be processed in both clusters (duplicate processing), and some may be processed in either cluster if the replication subscription update reaches the other cluster before the messages are consumed there. It is recommended to process messages in a single cluster when using replicated subscriptions. -* Delayed delivery prevents subscription replication because the cursor's mark-delete position does not advance until delayed messages have been delivered and processed. +* Replicated subscriptions use periodic snapshots to establish a consistent association between message positions across clusters. Snapshots are taken every `replicatedSubscriptionsSnapshotFrequencyMillis` milliseconds (default: 1000 ms), but the effective granularity of mark-delete position updates on the remote cluster depends on how many messages are produced between snapshots. See [Replicated subscriptions snapshot configuration and tuning](#replicated-subscriptions-snapshot-configuration-and-tuning) for details. +* Only the mark-delete position (the baseline cursor position) is replicated — individual acknowledgments are not. Messages acknowledged out of order may be redelivered after a cluster failover. +* Replicated subscriptions do not provide consistent behavior when consumers are active on multiple clusters simultaneously. Most messages will be processed on both clusters (duplicate processing), and some may be processed on either cluster depending on replication timing. To avoid this, process messages on a single cluster at a time when using replicated subscriptions. +* Delayed message delivery impairs subscription replication. The mark-delete position does not advance until delayed messages have been delivered and acknowledged, so replication lags behind accordingly. :::note @@ -368,7 +368,11 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. +Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. If any participating cluster is offline, snapshots cannot be established and the mark-delete position will not be propagated until connectivity is restored. + +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. + +A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. @@ -378,6 +382,8 @@ The improved snapshot cache expiration policy introduced by [PR #25044](https:// [PR #25044](https://github.com/apache/pulsar/pull/25044) also reduced the memory footprint of each snapshot entry to approximately 200 bytes, making it practical to increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` well beyond the default of `30` when finer snapshot granularity is needed for large backlogs. The tradeoff is higher heap memory consumption, which should be accounted for when sizing the broker. The current implementation uses a fixed per-subscription limit rather than a global memory budget; a future improvement could cap total cache memory consumption and apply the same spread-based eviction strategy once the limit is reached. +If geo-replication is enabled on a namespace that already contains messages, no snapshot markers will be present in the existing backlog. The mark-delete position cannot be propagated until the replicator has read past the pre-existing messages and new snapshots have been written and processed — regardless of the snapshot cache size. Similarly, neither the improved cache eviction policy nor increasing the cache size addresses the long interval between snapshots under bursty traffic; that continues to affect the lag of mark-delete position updates to remote clusters. + The following broker settings control snapshot behavior: | Setting | Default | Description | @@ -386,7 +392,10 @@ The following broker settings control snapshot behavior: | `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | -For setups with more than two clusters, it is recommended to increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` and `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to `50` to ensure that two-round snapshots complete before the timeout. The tradeoff is slightly higher memory consumption — with a value of `50`, the maximum heap memory consumed by the snapshot cache is approximately 10 KB per topic. +Tuning recommendations: + +* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshots complete before timing out. +* **Large backlogs with slow mark-delete advancement** (shared or key-shared subscriptions with individual acknowledgments, or delayed message delivery): Increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to at least `50` so that cached snapshots are spread more densely and a suitable snapshot is more likely to be close to the mark-delete position when it advances. At a value of `50`, the snapshot cache consumes approximately 10 KB of heap memory per subscription; with the default of `30`, the footprint is approximately 6 KB per subscription. ### Replicated subscriptions sequence diagrams From 873cb8ad54f0eb92e6e39776ebebbaa598912225 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 18:41:53 +0200 Subject: [PATCH 46/59] Zoom doesn't work for svg --- docusaurus.config.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docusaurus.config.ts b/docusaurus.config.ts index 36e8b41529f8..b11ebda7db8c 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -170,7 +170,7 @@ module.exports = async function createConfigAsync() { disableSwitch: true, }, zoom: { - selector: '.markdown img, .markdown svg', + selector: '.markdown img', background: { light: '#fff', dark: '#111' From 09ef5b4d04ed10ab92b2a06bfed9b1158c5279f4 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:00:04 +0200 Subject: [PATCH 47/59] Improve --- docs/administration-geo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 9964285744c3..9b10116fb7ca 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -360,7 +360,7 @@ The limitations of replicated subscription are as follows. :::note -* Replicated subscriptions add a special snapshot message to the topic every second. If there are inactive subscriptions on the topic, this increases the backlog on both the source and destination clusters. The [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is only created when new messages have been produced to the topic since the last snapshot was taken. +* A [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) cycle is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last cycle. Each cycle writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. ::: From dbf7dd4d0cce77ef59744f59661c15ac12020062 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:00:48 +0200 Subject: [PATCH 48/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 +- versioned_docs/version-4.1.x/administration-geo.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 9964285744c3..9b10116fb7ca 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -360,7 +360,7 @@ The limitations of replicated subscription are as follows. :::note -* Replicated subscriptions add a special snapshot message to the topic every second. If there are inactive subscriptions on the topic, this increases the backlog on both the source and destination clusters. The [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is only created when new messages have been produced to the topic since the last snapshot was taken. +* A [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) cycle is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last cycle. Each cycle writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. ::: diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 9964285744c3..9b10116fb7ca 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -360,7 +360,7 @@ The limitations of replicated subscription are as follows. :::note -* Replicated subscriptions add a special snapshot message to the topic every second. If there are inactive subscriptions on the topic, this increases the backlog on both the source and destination clusters. The [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is only created when new messages have been produced to the topic since the last snapshot was taken. +* A [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) cycle is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last cycle. Each cycle writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. ::: From 4e844f63cf2a08a5a51f10c90b5a9e83e7ca36dd Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:08:47 +0200 Subject: [PATCH 49/59] Improve --- docs/administration-geo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 9b10116fb7ca..6937cc4b4bf2 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -360,7 +360,7 @@ The limitations of replicated subscription are as follows. :::note -* A [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) cycle is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last cycle. Each cycle writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. +* A [snapshot attempt](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last attempt. Each attempt writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. ::: @@ -388,8 +388,8 @@ The following broker settings control snapshot behavior: | Setting | Default | Description | | --- | --- | --- | -| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | -| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often a snapshot attempt is started. A new attempt is initiated only when new messages have been produced since the last attempt completed. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot attempt can remain in progress before it is abandoned. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | Tuning recommendations: From 0508d53b2b4e649ce95fa6c22d79f3a0ef48de04 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:08:53 +0200 Subject: [PATCH 50/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 6 +++--- versioned_docs/version-4.1.x/administration-geo.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 9b10116fb7ca..6937cc4b4bf2 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -360,7 +360,7 @@ The limitations of replicated subscription are as follows. :::note -* A [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) cycle is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last cycle. Each cycle writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. +* A [snapshot attempt](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last attempt. Each attempt writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. ::: @@ -388,8 +388,8 @@ The following broker settings control snapshot behavior: | Setting | Default | Description | | --- | --- | --- | -| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | -| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often a snapshot attempt is started. A new attempt is initiated only when new messages have been produced since the last attempt completed. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot attempt can remain in progress before it is abandoned. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | Tuning recommendations: diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 9b10116fb7ca..6937cc4b4bf2 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -360,7 +360,7 @@ The limitations of replicated subscription are as follows. :::note -* A [snapshot](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) cycle is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last cycle. Each cycle writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. +* A [snapshot attempt](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot) is initiated every `replicatedSubscriptionsSnapshotFrequencyMillis` when new messages have been produced since the last attempt. Each attempt writes several marker messages to the topic — the snapshot request, the response from the remote cluster (replicated back), and the final snapshot marker. These markers increase the backlog for inactive subscriptions on both clusters. ::: @@ -388,8 +388,8 @@ The following broker settings control snapshot behavior: | Setting | Default | Description | | --- | --- | --- | -| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often snapshots are taken. | -| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot request can remain pending before it times out. | +| `replicatedSubscriptionsSnapshotFrequencyMillis` | `1000` | How often a snapshot attempt is started. A new attempt is initiated only when new messages have been produced since the last attempt completed. | +| `replicatedSubscriptionsSnapshotTimeoutSeconds` | `30` | How long a snapshot attempt can remain in progress before it is abandoned. | | `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | `30` (increased from 10 in PR #25044) | Maximum number of snapshots cached per subscription. Each entry consumes approximately 200 bytes of memory. | Tuning recommendations: From 13e0938fb38d5c667bd2a05132525e53c3e726ae Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:21:00 +0200 Subject: [PATCH 51/59] Improve --- docs/administration-geo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 6937cc4b4bf2..d4cfe8360c3e 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -368,9 +368,9 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. If any participating cluster is offline, snapshots cannot be established and the mark-delete position will not be propagated until connectivity is restored. +Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. Two-round attempts take roughly twice as long to complete, making snapshot timeout tuning particularly important in that case. If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. -The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot attempt time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. @@ -394,7 +394,7 @@ The following broker settings control snapshot behavior: Tuning recommendations: -* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshots complete before timing out. +* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshot attempts complete before timing out. * **Large backlogs with slow mark-delete advancement** (shared or key-shared subscriptions with individual acknowledgments, or delayed message delivery): Increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to at least `50` so that cached snapshots are spread more densely and a suitable snapshot is more likely to be close to the mark-delete position when it advances. At a value of `50`, the snapshot cache consumes approximately 10 KB of heap memory per subscription; with the default of `30`, the footprint is approximately 6 KB per subscription. ### Replicated subscriptions sequence diagrams From 70e946676550a7471af454e100e47b09e3063a1e Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:21:09 +0200 Subject: [PATCH 52/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 6 +++--- versioned_docs/version-4.1.x/administration-geo.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 6937cc4b4bf2..d4cfe8360c3e 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -368,9 +368,9 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. If any participating cluster is offline, snapshots cannot be established and the mark-delete position will not be propagated until connectivity is restored. +Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. Two-round attempts take roughly twice as long to complete, making snapshot timeout tuning particularly important in that case. If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. -The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot attempt time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. @@ -394,7 +394,7 @@ The following broker settings control snapshot behavior: Tuning recommendations: -* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshots complete before timing out. +* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshot attempts complete before timing out. * **Large backlogs with slow mark-delete advancement** (shared or key-shared subscriptions with individual acknowledgments, or delayed message delivery): Increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to at least `50` so that cached snapshots are spread more densely and a suitable snapshot is more likely to be close to the mark-delete position when it advances. At a value of `50`, the snapshot cache consumes approximately 10 KB of heap memory per subscription; with the default of `30`, the footprint is approximately 6 KB per subscription. ### Replicated subscriptions sequence diagrams diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 6937cc4b4bf2..d4cfe8360c3e 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -368,9 +368,9 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot requires one or two round-trips between the participating clusters — two when more than two clusters are involved. This increases the time needed for a snapshot to complete and makes snapshot timeout tuning more important. If any participating cluster is offline, snapshots cannot be established and the mark-delete position will not be propagated until connectivity is restored. +Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. Two-round attempts take roughly twice as long to complete, making snapshot timeout tuning particularly important in that case. If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. -The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot attempt time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. @@ -394,7 +394,7 @@ The following broker settings control snapshot behavior: Tuning recommendations: -* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshots complete before timing out. +* **More than two clusters:** Increase `replicatedSubscriptionsSnapshotTimeoutSeconds` to `60` to ensure that two-round snapshot attempts complete before timing out. * **Large backlogs with slow mark-delete advancement** (shared or key-shared subscriptions with individual acknowledgments, or delayed message delivery): Increase `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` to at least `50` so that cached snapshots are spread more densely and a suitable snapshot is more likely to be close to the mark-delete position when it advances. At a value of `50`, the snapshot cache consumes approximately 10 KB of heap memory per subscription; with the default of `30`, the footprint is approximately 6 KB per subscription. ### Replicated subscriptions sequence diagrams From e05642a8d4f991bb2549e82cdceecde5ed1e139c Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:58:28 +0200 Subject: [PATCH 53/59] Improve --- docs/administration-geo.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index d4cfe8360c3e..1d4355d824c1 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -368,13 +368,15 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. Two-round attempts take roughly twice as long to complete, making snapshot timeout tuning particularly important in that case. If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. +Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. Each round takes time because snapshot request and response markers are written into the topic as marker messages, and the replicator on each participating cluster must process all preceding messages in the topic before it can read them. Under high load, this can push the total snapshot attempt time beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, causing the attempt to time out without producing a completed snapshot — a risk that is greater when two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. -The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot attempt time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. +If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. + +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. With bursty or high-rate message production, many messages accumulate per snapshot interval, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests. This is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. -The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. +The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the subscription's snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. Completed snapshots are added to the cache as the subscription's dispatcher reads messages from the topic. The snapshot cache is cleared whenever all consumers disconnect — whether due to a topic unload, broker restart, load shedding, or explicit disconnection. When the first consumer reconnects, the dispatcher re-reads all unacknowledged messages and replication marker messages from the mark-delete position onward to restore the snapshot cache and redeliver the unacknowledged messages. A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This affects shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and topics using delayed message delivery. The original cache expiration policy kept only the `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` most recently created snapshots, evicting older ones. As a result, all cached snapshots could end up ahead of the mark-delete position if the mark-delete position had not advanced within the last `replicatedSubscriptionsSnapshotFrequencyMillis × replicatedSubscriptionsSnapshotMaxCachedPerSubscription` milliseconds (10 seconds with default settings), leaving no suitable snapshot available. From 02b30ba5349128154a456646d03fb45be677996b Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 19:58:35 +0200 Subject: [PATCH 54/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 8 +++++--- versioned_docs/version-4.1.x/administration-geo.md | 8 +++++--- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index d4cfe8360c3e..1d4355d824c1 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -368,13 +368,15 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. Two-round attempts take roughly twice as long to complete, making snapshot timeout tuning particularly important in that case. If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. +Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. Each round takes time because snapshot request and response markers are written into the topic as marker messages, and the replicator on each participating cluster must process all preceding messages in the topic before it can read them. Under high load, this can push the total snapshot attempt time beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, causing the attempt to time out without producing a completed snapshot — a risk that is greater when two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. -The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot attempt time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. +If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. + +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. With bursty or high-rate message production, many messages accumulate per snapshot interval, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests. This is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. -The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. +The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the subscription's snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. Completed snapshots are added to the cache as the subscription's dispatcher reads messages from the topic. The snapshot cache is cleared whenever all consumers disconnect — whether due to a topic unload, broker restart, load shedding, or explicit disconnection. When the first consumer reconnects, the dispatcher re-reads all unacknowledged messages and replication marker messages from the mark-delete position onward to restore the snapshot cache and redeliver the unacknowledged messages. A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This affects shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and topics using delayed message delivery. The original cache expiration policy kept only the `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` most recently created snapshots, evicting older ones. As a result, all cached snapshots could end up ahead of the mark-delete position if the mark-delete position had not advanced within the last `replicatedSubscriptionsSnapshotFrequencyMillis × replicatedSubscriptionsSnapshotMaxCachedPerSubscription` milliseconds (10 seconds with default settings), leaving no suitable snapshot available. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index d4cfe8360c3e..1d4355d824c1 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -368,13 +368,15 @@ The limitations of replicated subscription are as follows. Replicated subscriptions use a periodic snapshotting mechanism to establish a consistent association between message positions across clusters. The design is described in [PIP-33: Replicated subscriptions](https://github.com/apache/pulsar/blob/master/pip/pip-33.md#constructing-a-cursor-snapshot). -Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. Two-round attempts take roughly twice as long to complete, making snapshot timeout tuning particularly important in that case. If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. +Each snapshot attempt consists of one or two rounds, where each round is a snapshot request sent to all remote clusters followed by their responses. With two clusters, one round is sufficient; with more than two clusters, two rounds are required. Each round takes time because snapshot request and response markers are written into the topic as marker messages, and the replicator on each participating cluster must process all preceding messages in the topic before it can read them. Under high load, this can push the total snapshot attempt time beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, causing the attempt to time out without producing a completed snapshot — a risk that is greater when two rounds are required. When all rounds complete successfully, the final snapshot — containing the consistent cross-cluster position mapping — is written to the topic. -The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. A snapshot is only written to the topic once the full round-trip completes: the remote replicator must process all incoming messages up to the point where it receives the snapshot request, write a response, and have that response replicated back and processed on the originating cluster. With more than two clusters, two rounds of snapshotting are required, doubling the round-trip time. With bursty or high-rate message production, many messages can accumulate during this round-trip, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests — and under high load, the total snapshot attempt time can also push beyond the default `replicatedSubscriptionsSnapshotTimeoutSeconds` of `30` seconds, because the replicator must read through the high volume of regular messages in the topic before it reaches the snapshot response markers written by the remote cluster. The long interval between snapshots under bursty traffic is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. +If any participating cluster is offline, snapshot attempts will not be started and no snapshots will be created. As a result, mark-delete position updates cannot be propagated for any messages accumulated during the offline period, even after connectivity is restored, since those messages have no associated snapshots. + +The effective granularity of mark-delete position updates on the remote cluster is determined by how many messages are written between consecutive snapshots, not by `replicatedSubscriptionsSnapshotFrequencyMillis` alone. With bursty or high-rate message production, many messages accumulate per snapshot interval, resulting in much longer mark-delete position update intervals than the snapshot frequency setting suggests. This is particularly problematic when consumption is much slower than production — for example, when a batch job produces a large volume of messages within 30 seconds but consuming and acknowledging those messages takes minutes or hours. A potential future improvement would be to process snapshot requests and responses at message publish time rather than waiting for the replicator to drain its backlog. This would significantly reduce the snapshot round-trip time and snapshot lag under high-rate message production. -The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. +The subscription's mark-delete position can only be propagated to the remote cluster when there is a suitable snapshot in the subscription's snapshot cache. A snapshot is suitable when the mark-delete position has reached or passed the snapshot's local position — the position in the local cluster's topic at which the snapshot was created. Completed snapshots are added to the cache as the subscription's dispatcher reads messages from the topic. The snapshot cache is cleared whenever all consumers disconnect — whether due to a topic unload, broker restart, load shedding, or explicit disconnection. When the first consumer reconnects, the dispatcher re-reads all unacknowledged messages and replication marker messages from the mark-delete position onward to restore the snapshot cache and redeliver the unacknowledged messages. A known issue fixed in Pulsar 4.0.9 and 4.1.3 by [PR #25044](https://github.com/apache/pulsar/pull/25044) caused subscription state replication to stall in scenarios where the mark-delete position advances slowly. This affects shared or key-shared subscriptions using individual acknowledgments — where all acknowledgment gaps must be filled before the mark-delete position can advance — and topics using delayed message delivery. The original cache expiration policy kept only the `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` most recently created snapshots, evicting older ones. As a result, all cached snapshots could end up ahead of the mark-delete position if the mark-delete position had not advanced within the last `replicatedSubscriptionsSnapshotFrequencyMillis × replicatedSubscriptionsSnapshotMaxCachedPerSubscription` milliseconds (10 seconds with default settings), leaving no suitable snapshot available. From b4150be79b45c4f1d6d4e4a0ed09bab30e06bb42 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 20:19:13 +0200 Subject: [PATCH 55/59] Improve --- docs/administration-geo.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 1d4355d824c1..8f5e1d9a0faf 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -507,6 +507,8 @@ The following broker-level metrics are available for monitoring snapshot health. Topic stats and internal stats can be used to inspect the state of subscriptions. The cursor's mark-delete position is particularly useful, as subscription state can only be replicated up to that position. +A potential future improvement would be to expose pending snapshot attempt state including snapshot related topic-level counters in topic stats, and snapshot cache state in per-subscription stats. This would improve visibility when investigating replicated subscription issues. + ## Migrate data between clusters using geo-replication Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. From 869e0812c7a6e5961fbff6d93058b2ab9746c41f Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 20:19:19 +0200 Subject: [PATCH 56/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 ++ versioned_docs/version-4.1.x/administration-geo.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 1d4355d824c1..8f5e1d9a0faf 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -507,6 +507,8 @@ The following broker-level metrics are available for monitoring snapshot health. Topic stats and internal stats can be used to inspect the state of subscriptions. The cursor's mark-delete position is particularly useful, as subscription state can only be replicated up to that position. +A potential future improvement would be to expose pending snapshot attempt state including snapshot related topic-level counters in topic stats, and snapshot cache state in per-subscription stats. This would improve visibility when investigating replicated subscription issues. + ## Migrate data between clusters using geo-replication Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 1d4355d824c1..8f5e1d9a0faf 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -507,6 +507,8 @@ The following broker-level metrics are available for monitoring snapshot health. Topic stats and internal stats can be used to inspect the state of subscriptions. The cursor's mark-delete position is particularly useful, as subscription state can only be replicated up to that position. +A potential future improvement would be to expose pending snapshot attempt state including snapshot related topic-level counters in topic stats, and snapshot cache state in per-subscription stats. This would improve visibility when investigating replicated subscription issues. + ## Migrate data between clusters using geo-replication Using geo-replication to migrate data between clusters is a special use case of the [active-active replication pattern](concepts-replication.md#active-active-replication) when you don't have a large amount of data. From 099774a7bd819b4a2322b9ab2c63f5c4dfeb0835 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 20:44:27 +0200 Subject: [PATCH 57/59] Update browser list --- yarn.lock | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/yarn.lock b/yarn.lock index 09eba2aff14c..112ff752cee9 100644 --- a/yarn.lock +++ b/yarn.lock @@ -6348,17 +6348,10 @@ __metadata: languageName: node linkType: hard -"caniuse-lite@npm:^1.0.0, caniuse-lite@npm:^1.0.30001646, caniuse-lite@npm:^1.0.30001669": - version: 1.0.30001720 - resolution: "caniuse-lite@npm:1.0.30001720" - checksum: 97b9f9de842595ff9674001abb9c5bc093c03bb985d481ed97617ea48fc248bfb2cc1f1afe19da2bf20016f28793e495fa2f339e22080d8da3c9714fb7950926 - languageName: node - linkType: hard - -"caniuse-lite@npm:^1.0.30001702, caniuse-lite@npm:^1.0.30001737": - version: 1.0.30001741 - resolution: "caniuse-lite@npm:1.0.30001741" - checksum: 0f2e90e1418a0b35923735420a0a0f9d2aa91eb6e0e2ae955e386155b402892ed4aa9996aae644b9f0cf2cf6a2af52c9467d4f8fc1d1710e8e4c961f12bdec67 +"caniuse-lite@npm:^1.0.0, caniuse-lite@npm:^1.0.30001646, caniuse-lite@npm:^1.0.30001669, caniuse-lite@npm:^1.0.30001702, caniuse-lite@npm:^1.0.30001737": + version: 1.0.30001777 + resolution: "caniuse-lite@npm:1.0.30001777" + checksum: 962e97beb3a21d84ee7b5a04d68b5dc55cd3a84ff14f1ec6ad031a76ade8432a2d5d4a9697099d2f7e197f707d4f2698a01b756c8c68a30a13b37ded09c9b58a languageName: node linkType: hard From 070f1f0f91c24cbc4ef373e32ce5c18e0ce872a2 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 20:44:38 +0200 Subject: [PATCH 58/59] Update link --- docs/administration-geo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/administration-geo.md b/docs/administration-geo.md index 8f5e1d9a0faf..68dd01e11030 100644 --- a/docs/administration-geo.md +++ b/docs/administration-geo.md @@ -328,7 +328,7 @@ Replicated subscriptions require [2-way geo-replication](#1-way-and-2-way-geo-re If you want to use replicated subscriptions in Pulsar: -* For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). +* Ensure that `enableReplicatedSubscriptions` is `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/v4.1.3/conf/broker.conf#L728-L729). This is enabled by default. * For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. From 20f32e4d2d12358780f35914152881322a81ce25 Mon Sep 17 00:00:00 2001 From: Lari Hotari Date: Fri, 6 Mar 2026 20:44:46 +0200 Subject: [PATCH 59/59] Sync versioned docs --- versioned_docs/version-4.0.x/administration-geo.md | 2 +- versioned_docs/version-4.1.x/administration-geo.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-4.0.x/administration-geo.md b/versioned_docs/version-4.0.x/administration-geo.md index 8f5e1d9a0faf..68dd01e11030 100644 --- a/versioned_docs/version-4.0.x/administration-geo.md +++ b/versioned_docs/version-4.0.x/administration-geo.md @@ -328,7 +328,7 @@ Replicated subscriptions require [2-way geo-replication](#1-way-and-2-way-geo-re If you want to use replicated subscriptions in Pulsar: -* For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). +* Ensure that `enableReplicatedSubscriptions` is `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/v4.1.3/conf/broker.conf#L728-L729). This is enabled by default. * For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer. diff --git a/versioned_docs/version-4.1.x/administration-geo.md b/versioned_docs/version-4.1.x/administration-geo.md index 8f5e1d9a0faf..68dd01e11030 100644 --- a/versioned_docs/version-4.1.x/administration-geo.md +++ b/versioned_docs/version-4.1.x/administration-geo.md @@ -328,7 +328,7 @@ Replicated subscriptions require [2-way geo-replication](#1-way-and-2-way-geo-re If you want to use replicated subscriptions in Pulsar: -* For broker side: set `enableReplicatedSubscriptions` to `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/470b674016c8718f2dfd0a0f93cf02d49af0fead/conf/broker.conf#L592). +* Ensure that `enableReplicatedSubscriptions` is `true` in [`broker.conf`](https://github.com/apache/pulsar/blob/v4.1.3/conf/broker.conf#L728-L729). This is enabled by default. * For consumer side: replicated subscription is disabled by default. You can enable replicated subscriptions when creating a consumer.