diff --git a/docs/limitations.md b/docs/limitations.md index 6308ecd9..e5f749c8 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -31,7 +31,7 @@ The following limitations apply specifically to sharded cluster replication: * {{pcsm.short}} replicates the data and doesn't replicate metadata. This means that the following information is not preserved from the source cluster: * The primary shard name for a collection. The target cluster may have a different primary shard name. - * The chunk distribution information. The target cluster manages chunk distribution according to its own sharding configuration. See [Sharding support](sharding.md#limitations) for more information. + * Subsequent chunk distribution changes. For ranged shard keys, {{pcsm.short}} recreates the source chunk boundaries on the target before the initial clone starts, but later chunk distribution changes are not preserved. The target cluster then manages chunk distribution according to its own sharding configuration. See [Sharding support](sharding.md#chunk-distribution) for more information. * The configuration of [zones for sharded data :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/zone-sharding/). * During data replication, the following commands are not supported: `movePrimary`, `reshardCollecton`, `unshardCollection`, `refineCollectionShardKey`. Running them results in failed replication and you must start it anew, from the initial data sync stage. @@ -52,10 +52,8 @@ The following limitations apply specifically to sharded cluster replication: The following functionalities are not supported: -* Multiple source or multiple target clusters +* A single PCSM instance cannot have multiple source or target endpoints. Multiple isolated PCSM instances can share one source cluster and replicate to different targets, as described in [Replicate from one source to multiple targets](multiple-targets.md). * You cannot resume initial synchronization if an issue occurred. You must start it from scratch. * Arbitrary database upgrades during a sync are not supported. For supported staged upgrades from lower to higher MongoDB major versions, follow the cross-version replication procedure. * Reverse synchronization * External authentication via Kerberos, AWS and LDAP - - diff --git a/docs/multiple-targets.md b/docs/multiple-targets.md new file mode 100644 index 00000000..f942834f --- /dev/null +++ b/docs/multiple-targets.md @@ -0,0 +1,394 @@ +# Replicate from one source to multiple targets + +!!! admonition "Version added: 0.10.0" + +You can run multiple {{pcsm.full_name}} (PCSM) instances against the same source cluster and replicate different namespaces to different target clusters at the same time. This lets you split one cluster across several destinations in a single pass, with each instance moving only the data you assign to it. + +## How it works + +Each PCSM instance has: + +- The same source cluster +- Its own target cluster +- Its own namespace filter + +Every instance runs the full replication workflow on its own: clone, replication, and finalization. See [How PCSM works](intro.md) for what happens at each stage. + +!!! note + + Run each PCSM server in a separate container or host, or assign a unique `--port` when servers share a network namespace. Run every `start`, `status`, and `finalize` command in the corresponding container or host; for a shared host, pass that instance's `--port` to every subcommand. The examples below assume separate environments, where all instances can use the default port `2242`. See [Percona ClusterSync for MongoDB startup configuration](install/parameters.md) for the available options. + +## Before you begin + +Map out which instance owns which namespaces and which target before you start. You need that mapping again for every command you run, and it is the only record of which instance owns which data. + +!!! warning "Starting replication overwrites target collections" + `pcsm start` drops and recreates the collections that match your filter on the target, discarding any data already in them. Collections outside the filter stay as they are. Review each filter first, since a mistyped pattern affects only the target and leaves no trace on the source. + +The examples below use target clusters with no pre-existing application namespaces. They replicate `db_0` to the first target and `db_1` to the second. Select the tab that matches your deployment. + +=== "Replica set" + + ## Replicate from a replica set to two targets + + This example uses a source replica set `rs1` and two target replica sets, `rs2` and `rs3`. + + | **PCSM instance** | **Source** | **Target** | **Included namespaces** | + |-------------------|------------|------------|-------------------------| + | csync-a | rs1 | rs2 | `db_0.*` | + | csync-b | rs1 | rs3 | `db_1.*` | + + Follow these steps: + {.power-number} + + 1. Start `csync-a` with `rs1` as the source and `rs2` as the target: + + ```bash + pcsm \ + --source "mongodb://csync:@rs101:27017,rs102:27017,rs103:27017/?replicaSet=rs1" \ + --target "mongodb://csync:@rs201:27017,rs202:27017,rs203:27017/?replicaSet=rs2" + ``` + + 2. Start `csync-b` against the same source, with `rs3` as the target: + + ```bash + pcsm \ + --source "mongodb://csync:@rs101:27017,rs102:27017,rs103:27017/?replicaSet=rs1" \ + --target "mongodb://csync:@rs301:27017,rs302:27017,rs303:27017/?replicaSet=rs3" + ``` + + 3. Start replication on `csync-a`, filtered to the namespaces it replicates: + + ```bash + pcsm start --include-namespaces="db_0.*" + ``` + + ??? example "Expected output" + + ```{.json .no-copy} + { + "ok": true + } + ``` + + 4. Start replication on `csync-b` with its own filter: + + ```bash + pcsm start --include-namespaces="db_1.*" + ``` + + For information on how include and exclude filters interact, see [Start the filtered replication](install/usage.md#start-the-filtered-replication). For the full flag list, see [PCSM commands](pcsm-commands.md). You can also drive every step through the [PCSM HTTP API](api.md). + + 5. Check each instance and wait for the clone to complete and replication lag to reach an acceptable value. Look for `initialSync.completed` set to `true` and a low `lagTimeSeconds`: + + ```bash + pcsm status + ``` + + 6. Initiate finalization for each instance: + + + ```bash + pcsm finalize + ``` + + The command returns while the PCSM server continues creating the remaining indexes on the target. Use `pcsm status` to monitor the server and confirm when finalization is complete. + + !!! warning "Finalization cannot be undone" + You cannot resume an instance after you finalize it. Running `start` again begins a fresh initial sync and overwrites the target collections a second time. For a migration cutover, stop application writes to the namespaces the instance owns, wait for `lagTimeSeconds` to reach `0`, and finalize only then. Instances you are not cutting over yet keep replicating and are unaffected. + + 7. For each instance, run `pcsm status` until its server reports `state` as `finalized` and `finalization.completed` as `true`. + + The following output is from `csync-a`. The `csync-b` output has the same structure with its own operation time and finalization timestamps: + + + ```bash + pcsm status + ``` + + ??? example "Expected output" + + ```{.json .no-copy} + { + "ok": true, + "state": "finalized", + "info": "Finalized", + "lagTimeSeconds": 0, + "eventsRead": 0, + "eventsApplied": 0, + "lastReplicatedOpTime": { + "ts": "1787298593.1", + "isoDate": "2026-08-21T07:49:53Z" + }, + "initialSync": { + "estimatedCloneSizeBytes": 9877780, + "clonedSizeBytes": 9877780, + "completed": true, + "cloneCompleted": true + }, + "finalization": { + "completed": true, + "startedAt": "2026-08-21T07:49:53.633159569Z", + "completedAt": "2026-08-21T07:49:53.759444616Z" + } + } + ``` + + If the `finalization` object contains an `unsuccessfulIndexes` array, review it before you send traffic to that target. See [Unsuccessful indexes](install/usage.md#unsuccessful-indexes). + + ### Verify the result on replica set targets + + Connect to each target and confirm it holds only the namespaces that its instance replicated. + + ```sh + show databases + ``` + + You see `db_0` next to `admin`, `config`, and `percona_clustersync_mongodb`, which is where PCSM keeps its own replication metadata. The `db_1` database is absent, because `csync-a` never replicated it. + + + Counting documents confirms the same thing from the data side: + + ```javascript + db.getSiblingDB('db_0').docs.countDocuments({}) + db.getSiblingDB('db_1').docs.countDocuments({}) + ``` + + The first count matches the source. The second returns `0` rather than an error. + + + PCSM recreates the source indexes on the target during finalization, so check that they arrived: + + ```javascript + db.getSiblingDB('db_0').docs.getIndexes().map(i => i.name) + ``` + + ??? example "Expected output" + + ```{.text .no-copy} + [ + '_id_', + 'value_1', + 'value_1_uid_-1', + 'uid_1', + 'tag_text', + '_id_hashed', + 'created_at_1', + 'value_partial', + 'tag_sparse' + ] + ``` + + The collection replicated to the other target does not exist here, so querying it returns an error. + + ```javascript + db.getSiblingDB('db_1').docs.getIndexes().map(i => i.name) + ``` + + ```{.text .no-copy} + MongoServerError[NamespaceNotFound]: ns does not exist: db_1.docs + ``` + + Repeat the same three checks on `rs3` with the databases reversed. There, `db_1` holds the data and its indexes, and `db_0.docs` returns `ns does not exist: db_0.docs`. + +=== "Sharded cluster" + + !!! warning "Technical preview" + + Sharding support in PCSM is a technical preview and is not recommended for production. See [Sharding support in Percona ClusterSync for MongoDB](sharding.md). + + ## Replicate from a sharded cluster to two targets + + This example uses three sharded clusters, each with its own [mongos :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/sharded-cluster-query-router/){:target="_blank"}, config server, and two shards. One cluster is the source and two are targets. + + | **PCSM instance** | **Source** | **Target** | **Included namespaces** | + |-------------------|------------|------------|-------------------------| + | csync-a | mongos1 | mongos2 | `db_0.*` | + | csync-b | mongos1 | mongos3 | `db_1.*` | + + !!! note "Requirements for sharded deployments" + The source and both targets must be sharded clusters running the same MongoDB version, unless you are using [cross-version replication](version-compatibility.md). You do not need to disable the balancer on any of them. See [Sharding support in Percona ClusterSync for MongoDB](sharding.md). + + PCSM connects through `mongos` on both the source and the target, so you do not need to list individual shard members or config servers in the connection strings. + {.power-number} + + 1. Start `csync-a` against the source `mongos` and the first target `mongos`: + + ```bash + pcsm \ + --source "mongodb://csync:@mongos1:27017" \ + --target "mongodb://csync:@mongos2:27017" + ``` + + 2. Start `csync-b` against the same source `mongos` and the second target `mongos`: + + ```bash + pcsm \ + --source "mongodb://csync:@mongos1:27017" \ + --target "mongodb://csync:@mongos3:27017" + ``` + + 3. Start replication on `csync-a`: + + ```bash + pcsm start --include-namespaces="db_0.*" + ``` + + ??? example "Expected output" + + ```{.json .no-copy} + { + "ok": true + } + ``` + + Before the clone begins, PCSM checks which of the selected collections are sharded on the source and creates matching sharded collections on the target, carrying over the shard key and, for ranged shard keys, recreating the source chunk boundaries. + + + 4. Start replication on `csync-b`: + + ```bash + pcsm start --include-namespaces="db_1.*" + ``` + + For information on how include and exclude filters interact, see [Start the filtered replication](install/usage.md#start-the-filtered-replication). For the full flag list, see [PCSM commands](pcsm-commands.md). You can also drive every step through the [PCSM HTTP API](api.md). + + + 5. Check each instance and wait for the clone to complete and replication lag to reach an acceptable value. Look for `initialSync.completed` set to `true` and a low `lagTimeSeconds`: + + ```bash + pcsm status + ``` + + 6. Initiate finalization for each instance: + + ```bash + pcsm finalize + ``` + + The command returns while the PCSM server continues creating the remaining indexes on the target. Use `pcsm status` to monitor the server and confirm when finalization is complete. + + + !!! warning "Finalization cannot be undone" + You cannot resume an instance after you finalize it. Running `start` again begins a fresh initial sync and overwrites the target collections a second time. For a migration cutover, stop application writes to the namespaces the instance owns, wait for `lagTimeSeconds` to reach `0`, and finalize only then. + + 7. For each instance, run `pcsm status` until the server reports both `state: finalized` and `finalization.completed: true`. + + The following output is from `csync-a`: + + ```bash + pcsm status + ``` + + ??? example "Expected output" + + ```{.json .no-copy} + { + "ok": true, + "state": "finalized", + "info": "Finalized", + "lagTimeSeconds": 0, + "eventsRead": 6, + "eventsApplied": 5, + "lastReplicatedOpTime": { + "ts": "1787301347.3", + "isoDate": "2026-08-21T08:35:47Z" + }, + "initialSync": { + "estimatedCloneSizeBytes": 9877780, + "clonedSizeBytes": 9877780, + "completed": true, + "cloneCompleted": true + }, + "finalization": { + "completed": true, + "startedAt": "2026-08-21T08:35:47.949454942Z", + "completedAt": "2026-08-21T08:35:48.21823288Z" + } + } + ``` + + If the `finalization` object contains an `unsuccessfulIndexes` array, review it before you send traffic to that target. See [Unsuccessful indexes](install/usage.md#unsuccessful-indexes). + + + ### Verify the result on sharded targets + + Connect to the `mongos` of each target cluster, not to the shards directly. On `mongos2`, list the databases: + + + ```javascript + show databases + ``` + + ??? example "Expected output" + + ```{.text .no-copy} + admin 172.00 KiB + config 2.11 MiB + db_0 31.56 MiB + percona_clustersync_mongodb 168.00 KiB + ``` + + Count the documents. The `db_0` database returns the full count and `db_1` returns zero: + + ```javascript + db.getSiblingDB('db_0').docs.countDocuments({}) + db.getSiblingDB('db_1').docs.countDocuments({}) + ``` + + Check that the indexes PCSM recreated during finalization are present: + + ```javascript + db.getSiblingDB('db_0').docs.getIndexes().map(i => i.name) + ``` + + ??? example "Expected output" + + ```{.text .no-copy} + [ + '_id_', + 'value_1', + 'value_1_uid_-1', + 'uid_1', + 'tag_text', + '_id_hashed', + 'created_at_1', + 'value_partial', + 'tag_sparse' + ] + ``` + + The collection replicated to the other target does not exist here, so querying it returns an error. This is the expected result: + + ```javascript + db.getSiblingDB('db_1').docs.getIndexes().map(i => i.name) + ``` + + ```{.text .no-copy} + MongoServerError[NamespaceNotFound]: ns does not exist: db_1.docs + ``` + + If the source collection was sharded, confirm that the target collection is sharded too. + + + !!! note + For a ranged shard key, PCSM recreates the source chunk boundaries on the target before the clone. Later sharding metadata changes are not replicated, so the layouts diverge as each balancer works. That is expected. See [Chunk distribution](sharding.md#chunk-distribution). + + Run the same checks on `mongos3` with the databases reversed. There, `db_1` holds the data and its indexes, and `db_0.docs` returns `ns does not exist: db_0.docs`. + +## Check the logs + +Every instance logs separately, so check each one for errors before you decommission the source or send traffic to a target. + +- The PCSM server writes logs to `stdout`. +- Client subcommands write command responses to `stdout` and logs and errors to `stderr`. + +See [Logging in Percona ClusterSync for MongoDB](logging.md). + +If replication enters the `failed` state after initial sync while the PCSM server is still running, resolve the underlying issue and use `pcsm resume --from-failure`. If the PCSM process stops unexpectedly during real-time replication, restart it; PCSM resumes automatically from its last saved checkpoint. An interruption during initial sync cannot be resumed. Restart the clone as described in [Recover PCSM during initial data clone](troubleshooting.md#recover-pcsm-during-initial-data-clone). For command details, see [Resume the replication](pcsm-commands.md#resume). + +## Next steps + +- [Use Percona ClusterSync for MongoDB](./install/usage.md){.md-button} + +- [Sharding support in Percona ClusterSync for MongoDB](./sharding.md){.md-button} \ No newline at end of file diff --git a/docs/sharding.md b/docs/sharding.md index 34b13d88..3c6ee3bf 100644 --- a/docs/sharding.md +++ b/docs/sharding.md @@ -36,7 +36,7 @@ For detailed information about authentication and connection string configuratio ### Initial sync preparation -Before starting the initial sync, {{pcsm.short}} checks which collections are sharded on the source cluster and creates corresponding sharded collections on the destination cluster. The only sharding configuration preserved from the source cluster is the sharding key; all other sharding details are handled internally by the destination cluster. +Before starting the initial sync, {{pcsm.short}} checks which collections are sharded on the source cluster and creates corresponding sharded collections on the destination cluster. The sharding key is preserved from the source cluster. For ranged shard keys, {{pcsm.short}} also recreates the source chunk boundaries on the destination before the clone starts. Later sharding metadata changes are not replicated. ### Balancer operation @@ -44,9 +44,9 @@ Before starting the initial sync, {{pcsm.short}} checks which collections are sh ### Chunk distribution -{{pcsm.short}} does not preserve chunk distribution information from the source cluster. The target cluster manages chunk distribution internally through its balancer. This means that after replication, chunks may be distributed differently on the target cluster compared to the source cluster, which is expected behavior. +For ranged shard keys, {{pcsm.short}} recreates the source chunk boundaries on the target before the initial clone starts. After replication begins, later chunk distribution changes on the source cluster are not replicated. The target cluster manages chunk distribution internally through its balancer, so chunks may later be distributed differently on the target cluster compared to the source cluster, which is expected behavior. -Since the target cluster already has information about which collections are sharded, it handles sharding internally. {{pcsm.short}} does not interfere with the target cluster's sharding configuration or chunk distribution. +Since the target cluster already has information about which collections are sharded, it handles sharding internally. After the initial preparation, {{pcsm.short}} does not interfere with the target cluster's sharding configuration or chunk distribution. ## Usage diff --git a/mkdocs-base.yml b/mkdocs-base.yml index bb68e601..7fb1095e 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -180,6 +180,7 @@ nav: - install/parameters.md - Use PCSM: - Usage: install/usage.md + - Replicate from one source to multiple targets: multiple-targets.md - Cross-version replication: version-compatibility.md - pqs.md - PCSM commands: pcsm-commands.md