From e41be025b413bc8c734fa0566895965be256993c Mon Sep 17 00:00:00 2001 From: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 13:29:42 +0530 Subject: [PATCH 01/14] PCSM-93 HA for PCSM during repl phase --- docs/high-availability.md | 283 ++++++++++++++++++++++++++++++++++++++ mkdocs-base.yml | 1 + 2 files changed, 284 insertions(+) create mode 100644 docs/high-availability.md diff --git a/docs/high-availability.md b/docs/high-availability.md new file mode 100644 index 00000000..47ad4dbe --- /dev/null +++ b/docs/high-availability.md @@ -0,0 +1,283 @@ +# High availability during replication + +!!! admonition "Version added: 0.10.0" + +Percona ClusterSync for MongoDB (PCSM) supports active-standby high availability during replication. Run two or more instances against the same source and target, and one of them takes charge while the rest wait. If the active instance becomes unavailable, another takes over and resumes replication from the last checkpoint. + +High availability is always enabled and requires no configuration. A single instance behaves the same as in earlier versions. To enable failover, start a second instance with the same source and target. + +!!! info "Important" + High availability applies to the **replication phase after the initial clone completes**. + + The initial clone is not resumable. If the active PCSM instance fails during the clone, a standby becomes active, but the interrupted clone cannot continue. Start a new synchronization run to clone the data again. + +## How high availability works + +The instances coordinate through a lease stored on the target cluster, so the MongoDB deployment you already have is the only coordinator involved. Exactly one instance holds the lease at a time. That instance is `ACTIVE` and runs replication. The rest are `STANDBY` and do nothing until the lease expires. + +PCSM uses three mechanisms to ensure safe failover: + +### Lease election + +PCSM uses a lease to ensure that only one instance is ACTIVE at a time. Lease acquisition and renewal use atomic single-document operations. If several standby instances try to acquire an expired lease, only one can become active. + +PCSM evaluates lease expiration using the target MongoDB server clock. Differences between the clocks on PCSM hosts therefore do not affect the election. + +PCSM stores the lease as a single document in the `percona_clustersync_mongodb.lease` collection. For example: + + ```sh + { "_id": "lease", "term": 7, "instanceId": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", "electionDate": { "$date": "2026-07-17T09:14:02.190Z" }, "expiresAt": { "$date": "2026-07-17T09:20:41.882Z" } } + ``` + +Each instance also maintains a liveness document in the `percona_clustersync_mongodb.members` collection, refreshed on every heartbeat: + +```{.json .no-copy} +{ + "_id": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", + "group": "default", + "host": "pcsm0", + "port": 2242, + "role": "ACTIVE", + "term": 7, + "pcsmVersion": "0.10.0", + "startedAt": { "$date": "2026-07-17T09:14:02.113Z" }, + "lastHeartbeat": { "$date": "2026-07-17T09:20:31.882Z" } +} +``` +A member whose `lastHeartbeat` falls past the stale threshold is treated as dead and drops out of the group view. + +### Term fencing + +Each lease has a term value that increases whenever a new `ACTIVE` instance is elected. PCSM includes this value in every checkpoint written by the active instance. + +If a previous active instance resumes after losing its lease, its checkpoint writes contain an outdated term and are rejected. The instance then switches to `STANDBY`, which prevents it from overwriting the current replication state. + +### Checkpoint recovery + +When a standby becomes `ACTIVE`, PCSM resumes replication from the last persisted checkpoint. Failover uses the existing recovery mechanism and happens automatically. + +The timings are fixed: + +| **Setting** | **Value** | +|---------|-------| +| Lease TTL | 10 seconds | +| Lease renewal by the active instance | Every 3 seconds | +| Heartbeat interval | Every 3 seconds | +| Stale member threshold | 3 missed heartbeats | + +If the active instance stops unexpectedly, a standby can take over after the lease expires and continue replication from the latest checkpoint. + +## Set up high availability + +Run at least two PCSM instances on separate hosts, containers, or pods. Configure every instance with the same source and target clusters. + +For example, run the following command on each host: + +```bash +pcsm \ + --source "" \ + --target "" +``` + +No additional HA option is required. + +If you run multiple PCSM instances on the same host, configure a different `--port` for each instance. For protection against a host failure, run the instances on separate hosts or pods. + +See [Start PCSM](install/start-pcsm.md) for startup options and [Percona ClusterSync for MongoDB startup configuration](install/parameters.md) for the available parameters. + +### Identify the HA group + +Instances that share a group name coordinate as one active-standby group. Set the name with `--group-name` or the `PCSM_GROUP_NAME` environment variable: + +```sh +pcsm \ + --source "" \ + --target "" \ + --group-name migration-1 +``` + +The default group name is `default`. The name appears in member documents, in the API envelope, and as a label on the `..._ha_info` metric. + +## Failover during the initial clone + +High availability applies to the replication phase after the initial clone completes. The initial clone is not resumable because PCSM does not persist progress for individual collections. + +If the `ACTIVE` instance becomes unavailable during the initial clone, a standby is promoted. During recovery, the new `ACTIVE` detects that the clone was interrupted and stops the synchronization. PCSM reports the reason in the logs and through the /status endpoint: + +```sh +initial clone interrupted by failover and is not resumable; start a new run to re-clone from scratch +``` + +!!! info "Important" + + To recover, start a new synchronization run on the `ACTIVE` instance. PCSM starts the initial clone again from the beginning. Automatic recovery from the last checkpoint becomes available after the initial clone completes and PCSM enters the replication phase. + +See PCSM HTTP API for information about the /status endpoint and Start and manage synchronization for information about starting a new synchronization run. + +## Operate an HA deployment + +In an HA deployment, you need to know which PCSM instance is ACTIVE, direct operational commands to that instance, and monitor the health of all instances. PCSM provides API responses, metrics, and health endpoints to help you manage these tasks. + +### Check the active instance + +PCSM exposes the HA role through the `/metrics` endpoint. + +```bash +curl -sS http://localhost:2242/metrics | grep percona_clustersync_mongodb_ha_active +``` + +A value of `1` identifies the active instance. A value of `0` identifies a standby. + +When PCSM sees more than one live member, API responses can also include the me, role, and group fields. These fields identify the instance that handled the request and list the other members of the HA group. + +When PCSM sees more than one live member, API responses can also include the `me`, `role`, and `group` fields. These fields identify the instance that handled the request and list the other members of the HA group. For example, a `GET /status` response from the active instance: + +```{.json .no-copy} +{ + "ok": false, + "error": "not_active", + "me": { + "instanceId": "" + }, + "role": "STANDBY", + "group": { + "term": 7, + "members": [ + { + "instanceId": "", + "host": "pcsm0", + "port": 2242, + "role": "ACTIVE" + }, + { + "instanceId": "", + "host": "pcsm1", + "port": 2243, + "role": "STANDBY" + } + ] + } +} +``` +A request sent to a standby returns HTTP `409` with the `not_active` error: + +```{.json .no-copy} +{ + "ok": false, + "error": "not_active", + "me": { "instanceId": "6a2d8e10-4b3c-4f97-9c0a-2f7e1b4d6c88" }, + "role": "STANDBY", + "group": { + "name": "default", + "term": 7, + "members": [ + { "instanceId": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", "host": "pcsm0", "port": 2242, "role": "ACTIVE" }, + { "instanceId": "6a2d8e10-4b3c-4f97-9c0a-2f7e1b4d6c88", "host": "pcsm1", "port": 2243, "role": "STANDBY" } + ] + } +} +``` + +!!! info "Important" + + The `me`, `role`, and `group` fields are included only when the instance observes more than one live member. Applications that consume the PCSM API must therefore treat these fields as optional. + + A single PCSM instance continues to return API responses in the same format as earlier releases. + +See the [PCSM HTTP API](api.md) for endpoint details. + +### Operational commands on standby instances + +Replication commands must be sent to the active PCSM instance. + +The following endpoints return HTTP `409` with `error: "not_active"` when called on a standby: + +* `/status` +* `/start` +* `/pause` +* `/resume` +* `/finalize` + +The response identifies the standby and, when available, includes the HA member list so you can locate the active instance. + +The `/metrics` endpoint and `pprof` endpoints remain available on both active and standby instances. + + +See [PCSM commands](pcsm-commands.md) for information about managing a synchronization run. + +## Configure readiness probes + +Use `/metrics` for liveness and readiness probes in an HA deployment. This endpoint is available regardless of whether an instance is active or standby. + +Do not use `/status` for a readiness probe. A healthy standby returns HTTP `409` from this endpoint because replication status is available only from the active instance. + +!!! warning + + A healthy standby returns HTTP `409` from this endpoint because replication status is available only from the active instance. A probe pointed there marks every standby unhealthy. For Kubernetes deployments, see [Configure Liveness, Readiness and Startup Probes :octicons-link-external-16:](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/){:target="_blank"}. + +For monitoring configuration, see [Set up observability with Percona Monitoring and Management](pmm-setup.md). + +## High availability metrics + +PCSM exposes the following HA metrics through `/metrics`: + +| Metric | Description | +| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| `percona_clustersync_mongodb_ha_active` | Shows the current role. `1` means ACTIVE and `0` means STANDBY. | +| `percona_clustersync_mongodb_ha_term` | Shows the current HA lease term. | +| `percona_clustersync_mongodb_ha_role_transitions_total` | Counts role changes for the PCSM instance. | +| `percona_clustersync_mongodb_ha_info` | Reports instance information. The metric has a constant value of `1` and includes the `instance_id` and `group` labels. | + +You can use these metrics to identify the active instance, detect role changes, and monitor failover behavior. + +## Reset HA state + +PCSM provides commands to clear the stored HA membership or lease state. + +!!! warning + Stop all PCSM server instances that use the target cluster before running these commands. Do not reset HA state while PCSM is running. + +Clear the recorded member information: + +```bash +pcsm reset members +``` + +Clear the HA lease: + +```bash +pcsm reset lease +``` + +Use these commands only when you need to clear HA coordination state. To clear all PCSM state, use `pcsm reset`. + +## Upgrade from PCSM 0.9.0 + +Replication state created by PCSM 0.9.0 is not compatible with PCSM 0.10.0. + +Before starting PCSM 0.10.0: +{.power-number} + +1. Stop all PCSM 0.9.0 instances that use the target cluster. + +2.Rreset the stored PCSM state on the target: + + ```bash + pcsm reset --target "" + ``` + +3. Start PCSM 0.10.0 +4. Start a new synchronization run. + +!!! important + Do not run PCSM 0.9.0 and PCSM 0.10.0 against the same target at the same time. + +## Next steps + +[Manage synchronization with PCSM commands](pcsm-commands.md){.md-button} + +[Use the PCSM HTTP API](api.md){.md-button} + +[Set up observability with Percona Monitoring and Management](pmm-setup.md){.md-button} + + diff --git a/mkdocs-base.yml b/mkdocs-base.yml index bb68e601..654467c5 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -180,6 +180,7 @@ nav: - install/parameters.md - Use PCSM: - Usage: install/usage.md + - High availability: high-availability.md - Cross-version replication: version-compatibility.md - pqs.md - PCSM commands: pcsm-commands.md From 1ba857c3a6176ec314b442687ea95b5104f336c0 Mon Sep 17 00:00:00 2001 From: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:12:54 +0530 Subject: [PATCH 02/14] Update high-availability.md --- docs/high-availability.md | 110 ++++++++++++++++++++------------------ 1 file changed, 58 insertions(+), 52 deletions(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index 47ad4dbe..0d1116c6 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -13,7 +13,7 @@ High availability is always enabled and requires no configuration. A single inst ## How high availability works -The instances coordinate through a lease stored on the target cluster, so the MongoDB deployment you already have is the only coordinator involved. Exactly one instance holds the lease at a time. That instance is `ACTIVE` and runs replication. The rest are `STANDBY` and do nothing until the lease expires. +The instances coordinate through a lease stored on the target cluster, so the MongoDB deployment you already have is the only coordinator involved. Exactly one instance holds the lease at a time. That instance is `ACTIVE` and runs replication. PCSM uses three mechanisms to ensure safe failover: @@ -23,32 +23,25 @@ PCSM uses a lease to ensure that only one instance is ACTIVE at a time. Lease ac PCSM evaluates lease expiration using the target MongoDB server clock. Differences between the clocks on PCSM hosts therefore do not affect the election. -PCSM stores the lease as a single document in the `percona_clustersync_mongodb.lease` collection. For example: - - ```sh - { "_id": "lease", "term": 7, "instanceId": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", "electionDate": { "$date": "2026-07-17T09:14:02.190Z" }, "expiresAt": { "$date": "2026-07-17T09:20:41.882Z" } } - ``` +For more information about atomic single-document operations, see [Atomicity and Transactions :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/write-operations-atomicity/){="_blank"} in the MongoDB documentation. -Each instance also maintains a liveness document in the `percona_clustersync_mongodb.members` collection, refreshed on every heartbeat: +PCSM stores the lease as a single document in the `percona_clustersync_mongodb.lease` collection. For example: -```{.json .no-copy} -{ - "_id": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", - "group": "default", - "host": "pcsm0", - "port": 2242, - "role": "ACTIVE", - "term": 7, - "pcsmVersion": "0.10.0", - "startedAt": { "$date": "2026-07-17T09:14:02.113Z" }, - "lastHeartbeat": { "$date": "2026-07-17T09:20:31.882Z" } +```sh +{ "_id": "lease", + "group": "default", + "term": 7, + "instanceId": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", + "electionDate": { "$date": "2026-07-17T09:14:02.190Z" }, + "expiresAt": { "$date": "2026-07-17T09:20:41.882Z" } } ``` -A member whose `lastHeartbeat` falls past the stale threshold is treated as dead and drops out of the group view. + +The document identifies the active instance and records the current lease term, election time, and expiration time. ### Term fencing -Each lease has a term value that increases whenever a new `ACTIVE` instance is elected. PCSM includes this value in every checkpoint written by the active instance. +Each lease has a `term` value that increases whenever a new `ACTIVE` instance is elected. PCSM includes this value in every checkpoint written by the active instance. If a previous active instance resumes after losing its lease, its checkpoint writes contain an outdated term and are rejected. The instance then switches to `STANDBY`, which prevents it from overwriting the current replication state. @@ -67,6 +60,29 @@ The timings are fixed: If the active instance stops unexpectedly, a standby can take over after the lease expires and continue replication from the latest checkpoint. +### Instance membership + +Each PCSM instance records its identity and liveness information in the `percona_clustersync_mongodb.members` collection on the target cluster. The instance refreshes this information with each heartbeat. + + +For example: + +```sh +{ + "_id": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", + "group": "default", + "host": "pcsm0", + "port": 2242, + "role": "ACTIVE", + "term": 7, + "pcsmVersion": "0.10.0", + "startedAt": { "$date": "2026-07-17T09:14:02.113Z" }, + "lastHeartbeat": { "$date": "2026-07-17T09:20:31.882Z" } +} +``` + +A member that does not send a heartbeat within the stale-member threshold is removed from the current group view. + ## Set up high availability Run at least two PCSM instances on separate hosts, containers, or pods. Configure every instance with the same source and target clusters. @@ -87,7 +103,9 @@ See [Start PCSM](install/start-pcsm.md) for startup options and [Percona Cluster ### Identify the HA group -Instances that share a group name coordinate as one active-standby group. Set the name with `--group-name` or the `PCSM_GROUP_NAME` environment variable: +You can use `--group-name` or the `PCSM_GROUP_NAME` environment variable to assign a name that identifies the HA deployment in member information, API responses, metrics, and logs. + +For example: ```sh pcsm \ @@ -96,7 +114,13 @@ pcsm \ --group-name migration-1 ``` -The default group name is `default`. The name appears in member documents, in the API envelope, and as a label on the `..._ha_info` metric. +The default group name is `default`. + +!!! important + + In PCSM 0.10.0, the group name is used for identification and observability. It does not isolate HA coordination between different groups that use the same target cluster. + + Do not rely on different group names to create independent HA deployments against the same target. ## Failover during the initial clone @@ -116,11 +140,11 @@ See PCSM HTTP API for information about the /status endpoint and Start and manag ## Operate an HA deployment -In an HA deployment, you need to know which PCSM instance is ACTIVE, direct operational commands to that instance, and monitor the health of all instances. PCSM provides API responses, metrics, and health endpoints to help you manage these tasks. +During normal operation, you need to know which PCSM instance is active, send replication commands to that instance, and monitor all members of the deployment. PCSM exposes the information you need through its API and `/metrics` endpoint. ### Check the active instance -PCSM exposes the HA role through the `/metrics` endpoint. +Use the `percona_clustersync_mongodb_ha_active` metric to check the role of a PCSM instance: ```bash curl -sS http://localhost:2242/metrics | grep percona_clustersync_mongodb_ha_active @@ -130,7 +154,9 @@ A value of `1` identifies the active instance. A value of `0` identifies a stand When PCSM sees more than one live member, API responses can also include the me, role, and group fields. These fields identify the instance that handled the request and list the other members of the HA group. -When PCSM sees more than one live member, API responses can also include the `me`, `role`, and `group` fields. These fields identify the instance that handled the request and list the other members of the HA group. For example, a `GET /status` response from the active instance: +When PCSM sees more than one live member, API responses can also include the me, role, and group fields. The group information lists the live PCSM instances and their current roles. + +For example, an operational request sent to a standby returns HTTP `409` with `error: "not_active"`: ```{.json .no-copy} { @@ -159,24 +185,7 @@ When PCSM sees more than one live member, API responses can also include the `me } } ``` -A request sent to a standby returns HTTP `409` with the `not_active` error: - -```{.json .no-copy} -{ - "ok": false, - "error": "not_active", - "me": { "instanceId": "6a2d8e10-4b3c-4f97-9c0a-2f7e1b4d6c88" }, - "role": "STANDBY", - "group": { - "name": "default", - "term": 7, - "members": [ - { "instanceId": "b3f1c2a4-9d7e-4c11-8a2f-1e6b0d5c9a77", "host": "pcsm0", "port": 2242, "role": "ACTIVE" }, - { "instanceId": "6a2d8e10-4b3c-4f97-9c0a-2f7e1b4d6c88", "host": "pcsm1", "port": 2243, "role": "STANDBY" } - ] - } -} -``` +The response shows which instance handled the request and identifies the current active member. !!! info "Important" @@ -186,7 +195,7 @@ A request sent to a standby returns HTTP `409` with the `not_active` error: See the [PCSM HTTP API](api.md) for endpoint details. -### Operational commands on standby instances +### Send operational commands to the active instance Replication commands must be sent to the active PCSM instance. @@ -198,14 +207,11 @@ The following endpoints return HTTP `409` with `error: "not_active"` when called * `/resume` * `/finalize` -The response identifies the standby and, when available, includes the HA member list so you can locate the active instance. - -The `/metrics` endpoint and `pprof` endpoints remain available on both active and standby instances. - +When group information is available, the 409 response includes the member list so you can locate the active instance. See [PCSM commands](pcsm-commands.md) for information about managing a synchronization run. -## Configure readiness probes +## Configure health probes Use `/metrics` for liveness and readiness probes in an HA deployment. This endpoint is available regardless of whether an instance is active or standby. @@ -215,7 +221,7 @@ Do not use `/status` for a readiness probe. A healthy standby returns HTTP `409` A healthy standby returns HTTP `409` from this endpoint because replication status is available only from the active instance. A probe pointed there marks every standby unhealthy. For Kubernetes deployments, see [Configure Liveness, Readiness and Startup Probes :octicons-link-external-16:](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/){:target="_blank"}. -For monitoring configuration, see [Set up observability with Percona Monitoring and Management](pmm-setup.md). +For information about Kubernetes probes, see [Configure Liveness, Readiness and Startup Probes :octicons-link-external-16:](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/){="_blank"}. ## High availability metrics @@ -240,13 +246,13 @@ PCSM provides commands to clear the stored HA membership or lease state. Clear the recorded member information: ```bash -pcsm reset members +pcsm reset members --target " ``` Clear the HA lease: ```bash -pcsm reset lease +pcsm reset lease --target " ``` Use these commands only when you need to clear HA coordination state. To clear all PCSM state, use `pcsm reset`. From 02acd20784f5609d1970575a5b5becf0d33e58f3 Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:16:32 +0530 Subject: [PATCH 03/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index 0d1116c6..d3bd2f4e 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -266,7 +266,7 @@ Before starting PCSM 0.10.0: 1. Stop all PCSM 0.9.0 instances that use the target cluster. -2.Rreset the stored PCSM state on the target: +2. Reset the stored PCSM state on the target: ```bash pcsm reset --target "" From 1d0bf3b8768235b5776f7ae34615a6dca98d699e Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 11 Sep 2026 08:47:31 +0000 Subject: [PATCH 04/14] Document HA group startup parameter Co-authored-by: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> --- docs/install/parameters.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/install/parameters.md b/docs/install/parameters.md index af35f678..88c0817b 100644 --- a/docs/install/parameters.md +++ b/docs/install/parameters.md @@ -5,6 +5,7 @@ When [starting the `pcsm` process](start-pcsm.md), you can use the following opt - `--port`: The port on which the server will listen (default: 2242) - `--source`: The MongoDB connection string for the source cluster - `--target`: The MongoDB connection string for the target cluster +- `--group-name`: A name that identifies the HA deployment in member information, API responses, metrics, and logs (default: "default") - `--log-level`: The log level (default: "info") - `--log-json`: Output log in JSON format with disabled color - `--no-color`: Disable log ANSI color @@ -35,6 +36,7 @@ Alternatively, you can define the following environment variables: |----------|-------------|---------| | `PCSM_SOURCE_URI` | MongoDB connection string for the source cluster | - | | `PCSM_TARGET_URI` | MongoDB connection string for the target cluster | - | +| `PCSM_GROUP_NAME` | A name that identifies the HA deployment in member information, API responses, metrics, and logs. | `default` | | `PCSM_PORT` | Server port number | `2242` | | `PCSM_CLONE_NUM_PARALLEL_COLLECTIONS` | Number of collections cloned in parallel | `2` | | `PCSM_CLONE_NUM_READ_WORKERS` | Number of read workers for cloning | `NumCPU / 4` | @@ -46,4 +48,3 @@ Alternatively, you can define the following environment variables: | `PCSM_REPL_EVENT_QUEUE_SIZE` | Controls the size of the internal event queue used by the replication subsystem. | `5000` | | `PCSM_REPL_WORKER_QUEUE_SIZE` | Defines the maximum number of replication events that each replication worker thread can queue before processing. | `5000` | | `PCSM_REPL_BULK_OPS_SIZE` | Defines the maximum number of operations that can be grouped together into a single bulk apply batch during replication. | `5000` | - From 1b7f11c23e6e58ba2f8addcf5aedeea744a44c36 Mon Sep 17 00:00:00 2001 From: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:21:58 +0530 Subject: [PATCH 05/14] Update high-availability.md --- docs/high-availability.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index 0d1116c6..0fd95970 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -116,7 +116,7 @@ pcsm \ The default group name is `default`. -!!! important +!!! info "Important" In PCSM 0.10.0, the group name is used for identification and observability. It does not isolate HA coordination between different groups that use the same target cluster. @@ -275,15 +275,15 @@ Before starting PCSM 0.10.0: 3. Start PCSM 0.10.0 4. Start a new synchronization run. -!!! important +!!! info "Important" Do not run PCSM 0.9.0 and PCSM 0.10.0 against the same target at the same time. ## Next steps -[Manage synchronization with PCSM commands](pcsm-commands.md){.md-button} - [Use the PCSM HTTP API](api.md){.md-button} +[Manage synchronization with PCSM commands](pcsm-commands.md){.md-button} + [Set up observability with Percona Monitoring and Management](pmm-setup.md){.md-button} From 98a71e6c96fe166ba433517abb7c6525a6fc0024 Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:23:18 +0530 Subject: [PATCH 06/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index ba391c94..fe3a5360 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -15,7 +15,7 @@ High availability is always enabled and requires no configuration. A single inst The instances coordinate through a lease stored on the target cluster, so the MongoDB deployment you already have is the only coordinator involved. Exactly one instance holds the lease at a time. That instance is `ACTIVE` and runs replication. -PCSM uses three mechanisms to ensure safe failover: +PCSM uses the following mechanisms to ensure safe failover and track instance membership: ### Lease election From 68ae095de6d8be6c6ca43176fa8664a74350b011 Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:23:31 +0530 Subject: [PATCH 07/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index fe3a5360..c9fa671e 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -152,9 +152,7 @@ curl -sS http://localhost:2242/metrics | grep percona_clustersync_mongodb_ha_act A value of `1` identifies the active instance. A value of `0` identifies a standby. -When PCSM sees more than one live member, API responses can also include the me, role, and group fields. These fields identify the instance that handled the request and list the other members of the HA group. - -When PCSM sees more than one live member, API responses can also include the me, role, and group fields. The group information lists the live PCSM instances and their current roles. +When PCSM sees more than one live member, API responses can also include the `me`, `role`, and `group` fields. These fields identify the instance that handled the request and list the live PCSM instances and their current roles. For example, an operational request sent to a standby returns HTTP `409` with `error: "not_active"`: From 2b94b33b4e301260bf10c17138707c3a02f1e436 Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:23:55 +0530 Subject: [PATCH 08/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index c9fa671e..b0fead60 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -244,7 +244,7 @@ PCSM provides commands to clear the stored HA membership or lease state. Clear the recorded member information: ```bash -pcsm reset members --target " +pcsm reset members --target "" ``` Clear the HA lease: From 9328e70a78060667d849048ff65e6dbd0369c2d6 Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:24:23 +0530 Subject: [PATCH 09/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index b0fead60..0077adbc 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -219,7 +219,6 @@ Do not use `/status` for a readiness probe. A healthy standby returns HTTP `409` A healthy standby returns HTTP `409` from this endpoint because replication status is available only from the active instance. A probe pointed there marks every standby unhealthy. For Kubernetes deployments, see [Configure Liveness, Readiness and Startup Probes :octicons-link-external-16:](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/){:target="_blank"}. -For information about Kubernetes probes, see [Configure Liveness, Readiness and Startup Probes :octicons-link-external-16:](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/){="_blank"}. ## High availability metrics From 1fb890c1fc69d9a7a49a71733269ddac0fa38ea8 Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:24:58 +0530 Subject: [PATCH 10/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index 0077adbc..11d1419d 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -136,7 +136,7 @@ initial clone interrupted by failover and is not resumable; start a new run to r To recover, start a new synchronization run on the `ACTIVE` instance. PCSM starts the initial clone again from the beginning. Automatic recovery from the last checkpoint becomes available after the initial clone completes and PCSM enters the replication phase. -See PCSM HTTP API for information about the /status endpoint and Start and manage synchronization for information about starting a new synchronization run. +See the [PCSM HTTP API](api.md) for information about the `/status` endpoint and [Start the replication](install/usage.md#start-the-replication) for information about starting a new synchronization run. ## Operate an HA deployment From dfa19b3982556cddf21d234075984cc2f658d7ad Mon Sep 17 00:00:00 2001 From: Rasika Chivate <95711051+rasika-chivate@users.noreply.github.com> Date: Fri, 11 Sep 2026 14:25:07 +0530 Subject: [PATCH 11/14] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/high-availability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/high-availability.md b/docs/high-availability.md index 11d1419d..f723f276 100644 --- a/docs/high-availability.md +++ b/docs/high-availability.md @@ -23,7 +23,7 @@ PCSM uses a lease to ensure that only one instance is ACTIVE at a time. Lease ac PCSM evaluates lease expiration using the target MongoDB server clock. Differences between the clocks on PCSM hosts therefore do not affect the election. -For more information about atomic single-document operations, see [Atomicity and Transactions :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/write-operations-atomicity/){="_blank"} in the MongoDB documentation. +For more information about atomic single-document operations, see [Atomicity and Transactions :octicons-link-external-16:](https://www.mongodb.com/docs/manual/core/write-operations-atomicity/){:target="_blank"} in the MongoDB documentation. PCSM stores the lease as a single document in the `percona_clustersync_mongodb.lease` collection. For example: From 1c98a05fedfb05171798c664db4e31eb386cb971 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 11 Sep 2026 08:55:47 +0000 Subject: [PATCH 12/14] Document HA reset subcommands Co-authored-by: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> --- docs/pcsm-commands.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/pcsm-commands.md b/docs/pcsm-commands.md index 8244290e..5f1b0949 100644 --- a/docs/pcsm-commands.md +++ b/docs/pcsm-commands.md @@ -82,6 +82,22 @@ Resets the `PCSM` state and deletes the metadata collections from target deploym $ pcsm reset --target ``` +#### reset members + +Clears the recorded HA member information from the target deployment only. + +```{.bash data-prompt="$"$} +$ pcsm reset members --target "" +``` + +#### reset lease + +Clears the HA lease state from the target deployment only. + +```{.bash data-prompt="$"$} +$ pcsm reset lease --target "" +``` + ### finalize Finalize cluster replication. From 655e84e69d3f1e82e9a4a47dc0f3540f5f49f9bb Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 11 Sep 2026 08:56:27 +0000 Subject: [PATCH 13/14] Document HA API responses Co-authored-by: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> --- docs/api.md | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/docs/api.md b/docs/api.md index 21d9fc73..d696c162 100644 --- a/docs/api.md +++ b/docs/api.md @@ -60,6 +60,7 @@ curl -X POST "http://localhost:2242/start" \ - `ok`: Boolean indicating if the operation was successful. - `error` (optional): Error message if the operation failed. +- In HA deployments, this endpoint returns HTTP `409` with `error: "not_active"` when the request reaches a standby instance. See [HA responses](#ha-responses) for the optional `me`, `role`, and `group` fields that can also appear in this response. Example: @@ -83,6 +84,7 @@ curl -X POST http://localhost:2242/finalize - `ok`: Boolean indicating if the operation was successful. - `error` (optional): Error message if the operation failed. +- In HA deployments, this endpoint returns HTTP `409` with `error: "not_active"` when the request reaches a standby instance. See [HA responses](#ha-responses) for the optional `me`, `role`, and `group` fields that can also appear in this response. Example: @@ -106,6 +108,7 @@ curl -X POST http://localhost:2242/pause - `ok`: Boolean indicating if the operation was successful. - `error` (optional): Error message if the operation failed. +- In HA deployments, this endpoint returns HTTP `409` with `error: "not_active"` when the request reaches a standby instance. See [HA responses](#ha-responses) for the optional `me`, `role`, and `group` fields that can also appear in this response. Example: @@ -137,6 +140,7 @@ curl -X POST http://localhost:2242/resume -d '{ - `ok`: Boolean indicating if the operation was successful. - `error` (optional): Error message if the operation failed. +- In HA deployments, this endpoint returns HTTP `409` with `error: "not_active"` when the request reaches a standby instance. See [HA responses](#ha-responses) for the optional `me`, `role`, and `group` fields that can also appear in this response. Example: @@ -186,6 +190,8 @@ The following are response fields: | `finalization.unsuccessfulIndexes[].type` | string | Machine-readable failure category (`failed`, `incomplete`, `inconsistent`) | | `finalization.unsuccessfulIndexes[].reason` | string | Human-readable reason why finalization failed for this index | +In HA deployments, this endpoint returns HTTP `409` with `error: "not_active"` when the request reaches a standby instance. See [HA responses](#ha-responses) for the optional `me`, `role`, and `group` fields that can also appear in this response. + Example: ```json @@ -225,6 +231,51 @@ Example: } ``` +### HA responses + +The `/status`, `/start`, `/pause`, `/resume`, and `/finalize` endpoints can return the following optional HA fields when PCSM observes more than one live member: + +| Field | Type | Description | +|-------|------|-------------| +| `me.instanceId` | string | Identifier of the instance that handled the request | +| `role` | string | Role of the instance that handled the request (`ACTIVE` or `STANDBY`) | +| `group.term` | number | Current HA term | +| `group.members` | array | Live members observed by the instance | +| `group.members[].instanceId` | string | Identifier of the listed member | +| `group.members[].host` | string | Hostname of the listed member | +| `group.members[].port` | number | Port of the listed member | +| `group.members[].role` | string | Role of the listed member (`ACTIVE` or `STANDBY`) | + +When one of these requests reaches a standby instance, PCSM returns HTTP `409` with `error: "not_active"` and can include the HA fields shown above: + +```json +{ + "ok": false, + "error": "not_active", + "me": { + "instanceId": "" + }, + "role": "STANDBY", + "group": { + "term": 7, + "members": [ + { + "instanceId": "", + "host": "pcsm0", + "port": 2242, + "role": "ACTIVE" + }, + { + "instanceId": "", + "host": "pcsm1", + "port": 2243, + "role": "STANDBY" + } + ] + } +} +``` + ## Error handling The API uses standard HTTP status codes and returns error messages in the following format: @@ -242,4 +293,3 @@ Common error scenarios: - 404 Not Found: Endpoint not found - 500 Internal Server Error: Server-side issues - From 07be8263b0fe9f3fbbba2ec91e7e9b1a2bf860ea Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 14 Sep 2026 05:20:28 +0000 Subject: [PATCH 14/14] Add HA metrics to PMM metrics table Co-authored-by: rasika-chivate <95711051+rasika-chivate@users.noreply.github.com> --- docs/pmm-setup.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/pmm-setup.md b/docs/pmm-setup.md index 19a70eb9..483b3d08 100644 --- a/docs/pmm-setup.md +++ b/docs/pmm-setup.md @@ -929,3 +929,7 @@ You can collect and view the following PCSM metrics at the `/metrics` endpoint: | `percona_clustersync_mongodb_copy_insert_document_total` | Total count of the inserted documents | | `percona_clustersync_mongodb_copy_read_batch_duration_seconds` | Read batch duration time in seconds | | `percona_clustersync_mongodb_copy_insert_batch_duration_seconds` | Insert batch duration time in seconds | +| `percona_clustersync_mongodb_ha_active` | Shows the current role. `1` means ACTIVE and `0` means STANDBY. | +| `percona_clustersync_mongodb_ha_term` | Shows the current HA lease term. | +| `percona_clustersync_mongodb_ha_role_transitions_total` | Counts role changes for the PCSM instance. | +| `percona_clustersync_mongodb_ha_info` | Reports instance information. The metric has a constant value of `1` and includes the `instance_id` and `group` labels. |