-
Notifications
You must be signed in to change notification settings - Fork 7
PCSM-283 Skip sharding operations for replica set #67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
47a43c4
542e25e
b789c70
b43c0bc
ed44f22
8cadb85
7a02a06
bccb242
675429f
ba79a33
9a972b7
15dd913
9266ff8
e84e92c
003e8e2
f67df55
5a290f0
d608fcc
090cd53
6cd6e66
2d56962
a993af9
895a309
b63a5ff
3f8f690
db72be9
19e103e
bc7fde8
91c9e95
271c764
4ff0fcb
5465fc3
d8a7af0
3196536
2b9c87e
644794e
ea37bd1
8589625
470c43d
5ef5a00
4453888
5438446
929cde3
7421f62
d43e402
f206a65
eda880e
4fd1663
929f26e
9a132e3
400488c
8dccd8e
915aafe
118ba7e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,166 @@ | ||
| # Replicate from a sharded cluster to a replica set | ||
|
|
||
| !!! admonition "Version added: 0.10.0" | ||
|
|
||
| {{pcsm.full_name}} (PCSM) supports replication from a sharded MongoDB cluster to a replica set. This lets you migrate data from a sharded deployment without having to recreate the source sharding configuration on the target. | ||
|
Check notice on line 5 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| For example, you can use this topology when moving data from a sharded MongoDB Atlas or MongoDB Enterprise deployment to a Percona Server for MongoDB replica set. | ||
|
|
||
| For information about sharded cluster support, see [Sharding support in Percona ClusterSync for MongoDB](sharding.md). | ||
|
|
||
| ## Overview | ||
|
|
||
| When the PCSM server starts, it detects that the source is sharded and the target is a replica set. | ||
|
Check notice on line 13 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| During the initial sync, PCSM creates every source collection selected for replication on the target, including the sharded ones, as a regular collection. It doesn't carry over the source shard key, because a replica set has no shards to distribute documents across and doesn't support [`shardCollection` :octicons-link-external-16:](https://www.mongodb.com/docs/manual/reference/command/shardCollection/){:target="_blank"}. | ||
|
Check notice on line 15 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| During change replication, PCSM skips `shardCollection` events coming from the source [change stream :octicons-link-external-16:](https://www.mongodb.com/docs/manual/changeStreams/){:target="_blank"} and keeps applying the data changes it supports. No additional configuration is required. | ||
|
Check notice on line 17 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| !!! note | ||
| A collection that is sharded on the source is created as a regular collection on the replica set target. The collection data is copied, but the target collection isn't sharded. | ||
|
Check notice on line 20 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| ## What is replicated | ||
|
Check notice on line 22 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| !!! warning | ||
| `_id` values must be unique across all source shards. PCSM does not verify their uniqueness. If duplicates exist, documents may be silently skipped or overwritten without causing the synchronization to fail. | ||
|
Check notice on line 25 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| | **On the source** | **On the replica set target** | | ||
| |---|---| | ||
| | Sharded collection | Created as a regular collection. The shard key isn't applied because it doesn't apply to a replica set. | | ||
| | Unsharded collection | Created and copied as in a replica set to replica set sync. | | ||
| | Chunk distribution and primary shard | Not preserved. PCSM replicates data, not cluster metadata. | | ||
|
Check notice on line 31 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| ## Before you start | ||
|
|
||
| * Ensure that `_id` values are unique across all source shards in every collection you want to replicate. | ||
| * Ensure the source and target MongoDB versions meet the version requirements. | ||
| * Configure authentication for both deployments. | ||
| * Configure the source connection string with the `mongos` hostname and port. Configure the target connection string with the replica set members. | ||
| * Verify that PCSM can connect to both the source sharded cluster and the target replica set. | ||
|
Check notice on line 39 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
| ## Connection string format | ||
|
|
||
| Point the source URI at the `mongos` hostname and port. Point the target URI at the replica set members and name the replica set: | ||
|
|
||
| **Example** | ||
|
|
||
| ```sh | ||
| PCSM_SOURCE_URI="mongodb://source-user:password@mongos-source:27017/admin" | ||
|
|
||
| PCSM_TARGET_URI="mongodb://target-user:password@target1:27017,target2:27017,target3:27017/admin?replicaSet=rs0" | ||
| ``` | ||
|
|
||
| In this example, `mongos-source:27017` is the `mongos` endpoint for the source sharded cluster. The target URI lists the replica set members and identifies the replica set as `rs0`. | ||
|
|
||
| ## Usage | ||
|
|
||
| The commands and API endpoints are the same as for any other topology. See [Percona ClusterSync for MongoDB commands](pcsm-commands.md) for the command reference. | ||
|
|
||
| ??? example "Walkthrough: sharded source to replica set target" | ||
|
|
||
| Follow these steps: | ||
| {.power-number} | ||
|
|
||
| 1. Create two clusters: one sharded source cluster and one target replica set. | ||
|
|
||
| 2. Create two collections on the sharded cluster: | ||
|
|
||
| 1. `sharded_coll` (sharded) | ||
| 2. `plain_collection` (unsharded) | ||
|
|
||
| 3. Add documents to both the collections. | ||
|
|
||
| 4. Start replication: | ||
|
|
||
| ```sh | ||
| pcsm start | ||
| ``` | ||
|
|
||
| 5. Check the replication status. `clonedSizeBytes` matches `estimatedCloneSizeBytes`, and the state is `running`: | ||
|
|
||
| ```{.text .no-copy} | ||
| pcsm status | ||
| { | ||
| "ok": true, | ||
| "state": "running", | ||
| "info": "Replicating Changes", | ||
| "lagTimeSeconds": 0, | ||
| "eventsRead": 0, | ||
| "eventsApplied": 0, | ||
| "lastReplicatedOpTime": { | ||
| "ts": "1787645813.1", | ||
| "isoDate": "2026-08-25T08:16:53Z" | ||
| }, | ||
| "initialSync": { | ||
| "estimatedCloneSizeBytes": 7490, | ||
| "clonedSizeBytes": 7490, | ||
| "completed": true, | ||
| "cloneCompleted": true | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 6. Run the same query against both deployments and compare the results to confirm that both collections are present and that document counts match: | ||
|
|
||
| ```javascript | ||
| db.sharded_coll.countDocuments() | ||
| db.plain_collection.countDocuments() | ||
| ``` | ||
|
|
||
| The collection that was sharded on the source appears here as a regular collection. That is expected. | ||
|
|
||
| 7. Finalize the sync: | ||
|
|
||
| ```{.bash data-prompt="$"} | ||
| $ pcsm finalize | ||
| ``` | ||
|
|
||
| 8. Check the status again until the state is `finalized`: | ||
|
|
||
| ```{.bash data-prompt="$"} | ||
| $ pcsm status | ||
| ``` | ||
|
|
||
| ```{.json .no-copy} | ||
| { | ||
| "ok": true, | ||
| "state": "finalized", | ||
| "info": "Finalized", | ||
| "lagTimeSeconds": 1, | ||
| "eventsRead": 0, | ||
| "eventsApplied": 0, | ||
| "lastReplicatedOpTime": { | ||
| "ts": "1787645817.1", | ||
| "isoDate": "2026-08-25T08:16:57Z" | ||
| }, | ||
| "initialSync": { | ||
| "estimatedCloneSizeBytes": 7490, | ||
| "clonedSizeBytes": 7490, | ||
| "completed": true, | ||
| "cloneCompleted": true | ||
| }, | ||
| "finalization": { | ||
| "completed": true, | ||
| "startedAt": "2026-08-25T08:16:57.539447026Z", | ||
| "completedAt": "2026-08-25T08:16:57.539557888Z" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 9. Check the replication logs and confirm that no errors were recorded. For details, see [Logging in Percona ClusterSync for MongoDB](logging.md). | ||
|
|
||
| 10. Confirm that the documents for both `plain_collection` and `sharded_coll` got copied to the destination cluster. | ||
|
|
||
|
|
||
| ## Next steps | ||
|
|
||
| * [Install {{pcsm.full_name}}](installation.md) | ||
| * [Configure authentication](install/authentication.md) | ||
| * [Start replication](install/usage.md) | ||
| * [Monitor replication status](install/usage.md#check-the-replication-status) | ||
| * [Monitor PCSM performance with Percona Monitoring and Management](pmm-setup.md) | ||
|
Check notice on line 161 in docs/sharded-source-to-replica-set-target.md
|
||
|
|
||
|
|
||
| ## Learn more | ||
|
|
||
| [Shard Keys :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/sharding-shard-key/){:target="_blank"} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,49 +4,57 @@ | |
|
|
||
| Sharding support is available starting with {{pcsm.full_name}} 0.7.0 and is currently in technical preview stage. We encourage you to try it out and share your feedback. This will help us improve the feature in future releases. | ||
|
|
||
| {{pcsm.full_name}} supports replication between sharded MongoDB clusters, enabling you to migrate or synchronize data from one sharded deployment to another. This capability allows you to migrate sharded clusters with minimal downtime and synchronize data between sharded clusters for testing or development purposes. | ||
| {{pcsm.full_name}} supports replication from a sharded MongoDB cluster to another sharded cluster or to a replica set. | ||
|
|
||
| With a sharded target, you can migrate or synchronize data between sharded deployments with minimal downtime. With a replica set target, {{pcsm.full_name}} copies the data and skips the source sharding configuration. | ||
|
rasika-chivate marked this conversation as resolved.
|
||
|
|
||
| For details about using a replica set as the target, see [Replicate from a sharded cluster to a replica set](sharded-source-to-replica-set-target.md). | ||
|
Comment on lines
+7
to
+11
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed in |
||
|
|
||
| ## Overview | ||
|
|
||
| The workflow for sharded clusters is similar to replica sets. See [How {{pcsm.full_name}} works](intro.md#replication-workflows) for the complete workflow overview. The key difference is that {{pcsm.short}} connects to `mongos` instances on both the source and target clusters instead of replica set members. | ||
| The workflow for sharded clusters is similar to replica sets. See [How {{pcsm.full_name}} works](intro.md#replication-workflows) for the complete workflow overview. The key difference is the target topology: when the target is a sharded cluster, {{pcsm.short}} connects through `mongos` on both the source and target. When the target is a replica set, it connects through the source `mongos` and then the target replica set members instead of a target `mongos`. | ||
|
|
||
| Since {{pcsm.short}} connects through `mongos`, the cluster topology doesn't matter. This means the source and target clusters can have different numbers of shards. | ||
| In both cases, the source must be a sharded MongoDB deployment. When the target is a sharded cluster, the source and target can have different numbers of shards. A replica set target does not require a target `mongos` instance. | ||
|
Check notice on line 17 in docs/sharding.md
|
||
|
|
||
| Also, {{pcsm.short}} replicates data and not metadata. This means chunk distribution as well as the primary shard name for a collection may differ on source and target clusters. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * {{pcsm.full_name}} version 0.7.0 or later | ||
| * Source and target clusters must be sharded MongoDB deployments | ||
| * If the target is a sharded MongoDB deployment, {{pcsm.full_name}} version 0.7.0 or later. | ||
| * If the target is a replica set, {{pcsm.full_name}} version 0.10.0 or later. | ||
| * The source must be a sharded MongoDB deployment. | ||
| * The target can be either a sharded MongoDB deployment or a replica set. | ||
|
rasika-chivate marked this conversation as resolved.
|
||
| * Both clusters must be running the same MongoDB version. Check [Version requirements](deployment.md#version-requirements) for more information about supported versions. | ||
|
|
||
| ## Connection string format | ||
|
|
||
| When connecting to sharded clusters, use the standard MongoDB connection string format but specify `mongos` hostname and port instead of replica set members: | ||
| When connecting to a sharded source or a sharded target, use the standard MongoDB connection string format but specify the `mongos` hostname and port instead of replica set members: | ||
|
|
||
| ```{.text .no-copy} | ||
| mongodb://user:pwd@mongos-host:port/[authdb]?[options] | ||
| ``` | ||
|
|
||
| Since {{pcsm.short}} connects through `mongos`, you don't need to specify individual shard members or config servers in the connection string. The `mongos` router handles routing to the appropriate shards. | ||
| When the target is a replica set, specify the target replica set members in the target connection string instead of a `mongos` URI. {{pcsm.short}} does not require a target `mongos` instance in that topology. | ||
|
Check notice on line 37 in docs/sharding.md
|
||
|
|
||
| For detailed information about authentication and connection string configuration, see [Configure authentication in MongoDB](install/authentication.md). | ||
|
|
||
| ## Sharding-specific behavior | ||
|
|
||
| The following behavior applies when both the source and target are sharded MongoDB deployments. For replica set targets, see [Replicate from a sharded cluster to a replica set](sharded-source-to-replica-set-target.md). | ||
|
Check notice on line 43 in docs/sharding.md
|
||
|
|
||
| ### Initial sync preparation | ||
|
|
||
| Before starting the initial sync, {{pcsm.short}} checks which collections are sharded on the source cluster and creates corresponding sharded collections on the destination cluster. The only sharding configuration preserved from the source cluster is the sharding key; all other sharding details are handled internally by the destination cluster. | ||
| Before starting the initial sync, {{pcsm.short}} checks which collections are sharded on the source cluster and creates corresponding sharded collections on the target sharded cluster. The only sharding configuration preserved from the source cluster is the sharding key. All other sharding details are handled internally by the target sharded cluster. | ||
|
Check notice on line 47 in docs/sharding.md
|
||
|
|
||
| ### Balancer operation | ||
|
|
||
| {{pcsm.full_name}} connects to source and target clusters via a `mongos` instance. Therefore, you do not need to disable the balancer on either the source or target cluster before starting replication. The target cluster's balancer continues to operate normally and manages chunk distribution according to its own sharding configuration and balancer settings. | ||
| {{pcsm.full_name}} connects to the sharded source through a `mongos` instance. When the target is also sharded, {{pcsm.full_name}} connects to it through `mongos`. You do not need to disable the balancer on either sharded cluster before starting replication. The target balancer continues to operate normally and manages chunk distribution according to the target cluster's sharding configuration and balancer settings. | ||
|
Check notice on line 51 in docs/sharding.md
|
||
|
|
||
| ### Chunk distribution | ||
|
|
||
| {{pcsm.short}} does not preserve chunk distribution information from the source cluster. The target cluster manages chunk distribution internally through its balancer. This means that after replication, chunks may be distributed differently on the target cluster compared to the source cluster, which is expected behavior. | ||
| For ranged shard keys, {{pcsm.short}} uses the source chunk boundaries to pre-split a sharded target before the clone begins. Collections with a hashed shard key keep the layout MongoDB creates. From there each cluster's balancer manages its own placement, so the layouts can differ. | ||
|
|
||
| Since the target cluster already has information about which collections are sharded, it handles sharding internally. {{pcsm.short}} does not interfere with the target cluster's sharding configuration or chunk distribution. | ||
| The target sharded cluster already knows which collections are sharded and manages their chunk distribution. {{pcsm.short}} does not interfere with the target cluster's sharding configuration or chunk distribution. | ||
|
Check notice on line 57 in docs/sharding.md
|
||
|
|
||
| ## Usage | ||
|
|
||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.