[DPE-10203] fix(async-replication): recover async replication after a dead datacenter - #1913
Open
marceloneppel wants to merge 13 commits into
Open
[DPE-10203] fix(async-replication): recover async replication after a dead datacenter#1913marceloneppel wants to merge 13 commits into
marceloneppel wants to merge 13 commits into
Conversation
2 tasks
github-actions
Bot
added
the
Libraries: Out of sync
This was referenced Aug 21, 2026
After a dead-datacenter failover, re-establishing async replication to a new cluster failed on create-replication with "committing requested changes failed" / "secret with label async-replication-secret already exists". The offer/primary and consumer/standby sides shared one fixed Juju secret label. A cluster that had been a standby keeps that label reserved as a consumer alias; Juju does not release it when the dead primary's relation is torn down. When the cluster is later promoted and owns the shared secret, the owner-create reuses the same label and deadlocks: the label is simultaneously unreadable (the aliased remote secret is gone) and uncreatable (the alias still reserves it). Own the secret under a distinct label so an owner-create can never collide with the consumer alias; the consumer keeps reading by secret id, so the handoff is unchanged and a wedged cluster self-heals without a redeploy. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…eardown When a dead datacenter is force-removed, Juju may never deliver the relation-broken event for the cross-model relation, so the promoted-cluster -counter set during promotion is left behind. A later, newly-formed async relation then re-counts that stale value and create-replication wrongly fails with "There is already a replication set up", blocking recovery. Reconcile the orphaned counter from the update-status handler, deciding solely from peer data (no Patroni call) because Patroni is frequently unreachable right after a dead-DC promotion — exactly when this must still run. Also harden relation-broken: building the Patroni client can transiently fail during a force-removal (network-get, goal-state, the REST API), which crashed the hook and wedged both units in error forever, so update-status never ran and the cluster never reverted to standalone. Tolerate those failures so the counter is always cleared. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
After a dead-DC teardown (the primary datacenter force-killed and its offer cleared with remove-saas --force), the local cross-model replication relation lingers in a dying state whose databags return "permission denied" on every relation-get — yet get_relation still returns it and its active flag can still read True. The charm read that relation's data unconditionally in _get_primary_cluster (the remote app databag) and in _relation (used by every write path), so the unhandled ModelError crashed the charm in __init__ on every hook. All hooks then failed before update-status could run clear_stale_promotion, so a promoted primary never recovered and DPE-10203's create-replication could not proceed. Guard both reads: _get_primary_cluster skips a peer whose databag raises ModelError, and _relation probes a cheap own-unit read and treats an unreadable relation as absent so a promoted primary reconciles as a standalone cluster. Unit tests cover both guards. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
… forms The dead-DC recovery sequence (DPE-10203) offers the promoted cluster to a fresh cluster BEFORE running create-replication. clear_stale_promotion refused to reconcile the orphaned promoted-cluster-counter whenever ANY async relation existed, so once that new offer relation formed — the relation the old, force-removed one never delivered relation-broken for — the counter stayed positive in the peers app databag. create-replication then failed with "There is already a replication set up." on every retry: the cluster's own counter makes _get_primary_cluster report itself as the primary. A promotion writes the counter to both the async relation it was promoted under and the peers databag, so "live" is precisely "some current async relation mirrors the counter". Reconcile on that instead of mere relation existence: a relation formed after the promotion carries no mirror and the counter is stale exactly then. Verified on a live 3-model deployment of the ticket scenario (first Testflinger run since the PS7 outage): phases 1-2 passed, phase 3 wedged on this counter; with the reconciler fixed the counter clears from update-status and create-replication proceeds. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…he consumer The standby/consumer side read the shared offer secret with a fixed Juju label, which registers a consumer-side alias under that label. Juju keeps the alias reserved even after the remote secret is force-removed during a dead-DC teardown, so a former standby is left carrying a stale alias it can neither read nor release. The owner-side OFFER_SECRET_LABEL split already breaks the promotion deadlock, but the consumer alias remains an avoidable latent hazard whose safe reuse depends on unverified Juju relabel behaviour. Referencing the secret purely by the id already published in the relation databag — as the MySQL async-replication charm does — means no consumer alias is ever registered and nothing can go stale, closing the DPE-10203 failure class on the consumer side too. This is backward compatible: the id is already present in primary-cluster-data and no peer depends on the alias existing. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…on reads
After a dead-DC teardown the remote app/unit databag on the dying cross-model
async relation raises ModelError ("permission denied") on any read. Only
_get_primary_cluster was guarded; the sibling reads were not, so they crashed the
hooks that keep a promoted cluster reconciling — replication-offer-relation-joined
and database-peers-relation-changed both died, leaving units wedged in error and
blocking re-replication to a fresh cluster.
Route every async-relation databag read through a shared _safe_databag_get helper
(and a shared _remote_unit_addresses for the two identical endpoint list
comprehensions) that treats an unreadable databag as key-absent, matching the
existing _get_primary_cluster behaviour, so the teardown always reconciles instead
of crashing.
Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
ops implements Application.planned_units() via the goal-state hook command, which
fails ("saas application ... not found") while a cross-model SAAS force-removed
during a dead-DC teardown still lingers in goal-state. Every caller — the _patroni
property construction, the degraded-status check, the synchronous-node count and the
backups pre-checks — crashed its hook, cascading through the _patroni property to
wedge the promoted cluster and block re-replication to a fresh cluster.
Route them through a single guarded _planned_units property that falls back to the
current unit count when goal-state is unavailable, so hooks reconcile instead of
crashing (DPE-10203 Issue B).
Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
DPE-10203 rework of the owner-side secret handling: neither side of the
async-replication relation attaches any label to the shared
cluster-credentials secret. The owner creates it labelless and re-finds
it by the id persisted in app peer data; the consumer already references
it purely by the id published in the relation databag.
This replaces the distinct-offer-label workaround. Labels were the
collision surface in both deadlock directions Juju can produce: a stale
consumer alias blocks a later owner-create ('secret with label already
exists' while the label is unreadable), and an owner label cannot
survive the secret's own id churn — a refreshed owner that misses its
label lookup mints a second secret, and Juju refuses to rebind a
consumer label to the new id, wedging any consumer still running
label-attaching code.
A cluster refreshed from the legacy charm has no id in peer data but its
own offer-relation data still publishes the last-known secret id; the
owner adopts that secret instead of creating a second one, so the id
never changes across the refresh and old consumers keep working.
Also re-applies the stale DCS standby_cluster clearing that the 16/edge
single-kernel update_config adoption (#1849) had displaced: update_config
now patches standby_cluster to None when this cluster is the primary,
so a force-promoted cluster converges out of read-only standby-leader
state on the next reconciliation.
Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The no-label invariant and the failure history it explained belong to the commit messages and the PR description (DPE-10203); restating them at the module head duplicates that record where it goes stale as the code evolves. The code itself — no label constant defined, secret access strictly by id — already expresses the invariant. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
Three sites in test_on_create_replication wrote the PropertyMock onto the class (type(relation)._relation = ...) without cleanup, leaking it to every later test; two more tests then silently RELIED on that leak — their instance writes only worked through the leaked mock's __set__, so both fail with AttributeError when run standalone. Six further instance writes were silent no-ops against whichever mock happened to be leaked, their assertions passing by coincidence of branch equivalence. Convert every site to scoped patch.object contexts: cleanup is automatic, order-dependence is gone, previously-red solo runs pass, and _REAL_RELATION_PROPERTY capture/restore machinery is deleted. The leaked .app PropertyMocks in test_promote_to_primary covered a path production never reads and are dropped entirely. Verified: full file green, and solo runs of test__configure_standby_ cluster / test_handle_forceful_promotion / the dying-relation probe now pass in isolation. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
marceloneppel
force-pushed
the
dpe-10203-async-replication-recovery
branch
from
September 1, 2026 18:00
9700e8e to
336a4d6
Compare
create-replication and promote-to-primary read _get_primary_cluster, which counts an orphaned promoted-cluster-counter left in peer data by a dead-DC teardown whose relation-broken never fired. With the only reconciler in update-status, the action fails with "There is already a replication set up." (or promote-to-primary wrongly sees a primary) until an update-status cycle happens to run — which the dead-DC recovery window cannot rely on. Both actions now clear the stale counter before their guard; the mirror check inside clear_stale_promotion keeps live replications untouched. The update- status reconciler stays (idempotent, cheap, and covers paths that don't go through the actions). The dead-DC integration test drops its 10-minute retry loop, whose only purpose was absorbing that update- status latency. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
… constant The async-replication-secret-id peer-data key is read in one place and written in two; a module constant keeps the three references in sync with a single definition. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…n types Condense the multi-paragraph dead-DC comments to the repo's short single-purpose comments, trim the regression test module docstring to the same shape, and narrow two of the teardown exception handlers to their concrete failure types (DeployedWithoutTrustError, RetryError, ModelError). The get_standby_leader handler keeps Exception: it probes across three layers whose failure modes cannot be enumerated here, and a crash would defeat the counter-clear reconciliation. Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
This was referenced Sep 4, 2026
marceloneppel
marked this pull request as ready for review
September 5, 2026 14:16
marceloneppel
requested review from
carlcsaposs-canonical,
dragomirp,
juju-charm-bot and
taurus-forever
and removed request for
a team
September 5, 2026 14:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
DPE-10203: when the primary datacenter of an async-replication pair dies, the surviving cluster cannot re-establish replication to a fresh cluster. After force-promoting the standby and force-removing the dead consumed offer (
remove-saas --force, which never deliversrelation-broken),create-replicationfails: the promotion leaves an orphanedpromoted-cluster-counterthat makes the cluster report itself as an existing primary ("There is already a replication set up."), the Patroni DCS keeps a stalestandby_clusterthat keeps the promoted cluster read-only, and reads of the dying relation's databags crash every hook.Solution
Full fix set, landed as one revision because the pieces interlock:
promoted-cluster-counterfrom update-status, including after a new offer relation has already formed (the recovery sequence offers before runningcreate-replication), when no live relation mirrors the counter — andcreate-replication/promote-to-primaryalso clear it inline before their guards read_get_primary_cluster, so recovery does not wait for an update-status cycle to happen to run first (a counter mirrored on a live relation survives, both in the reconciler and the actions);standby_clusterwhen this cluster is the primary, so a force-promoted cluster converges out of read-only standby-leader state;planned_units()fails on the goal-state read — so hooks reconcile instead of crashing during teardown.The goal-state failure mode guarded was reproduced live against the current
16/edgebase charm (no guard): two clusters deployed with replication established, the primary's machines force-stopped, thenremove-saas --forceon the survivor — after which both units of the surviving cluster stayed inerrorstatus for the entire 12-minute observation window, hook-looping uncaught exceptions of the form:Regression test added through #1914.
Checklist