Skip to content

fix(reshard): fence maintenance through direct primary - #1015

Merged
bill-ph merged 4 commits into
mainfrom
codex/fix-reshard-cancel-e2e
Jul 30, 2026
Merged

fix(reshard): fence maintenance through direct primary#1015
bill-ph merged 4 commits into
mainfrom
codex/fix-reshard-cancel-e2e

Conversation

@bill-ph

@bill-ph bill-ph commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • investigate the flaky reshard cancellation/rollback failure seen on an unrelated pull request
  • establish the final operation-scoped maintenance session directly against the source CNPG primary
  • terminate all pooled maintenance sessions after NOLOGIN, then close the direct backend instead of returning it to PgBouncer
  • add regression coverage for source routing, credential isolation, fail-closed takeover behavior, and timeout budgeting

Root cause

The maintenance cleanup path connected through the shard's session-mode PgBouncer pool. pgx.Connect could establish the frontend before PgBouncer authenticated a PostgreSQL backend; after the maintenance role became NOLOGIN, the first SQL statement could fail with PgBouncer's cached server_login_retry error.

Merely forcing PgBouncer to assign a backend was incomplete: closing that frontend returns its still-authenticated backend to the pool, where a stale client with the operation credential could reuse it after NOLOGIN.

The final cleanup session now uses the source shard's direct -rw endpoint while retaining the operation-scoped maintenance username/password. It terminates every other maintenance-role backend and then closes its own direct PostgreSQL connection, leaving no authenticated backend for PgBouncer to recycle.

Endpoint resolution happens before any takeover re-enables a previously disabled role and has a separate timeout from the final disable/drain transition.

Reproduction

Validation

  • red/green coverage for direct source endpoint routing and maintenance credential preservation
  • red/green coverage for endpoint-resolution failure before NOLOGIN
  • red/green coverage for takeover preserving an existing NOLOGIN state when resolution fails
  • red/green coverage for separate endpoint-resolution and disable/drain timeout budgets
  • green: Kubernetes-tagged controlplane/provisioner suite
  • green: just lint
  • green: focused reshard e2e lane, including cancellation and forced-cutover rollback: https://github.com/PostHog/duckgres/actions/runs/30549063384/job/90893868444
  • green: full e2e lane: https://github.com/PostHog/duckgres/actions/runs/30549063384/job/90893868337
  • the broader local Kubernetes control-plane recipe cannot complete its Docker-backed admin tests because the local Docker runtime is unavailable

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 0 1 0
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +3 / -0
  • Assertions: +17 / -0
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

@bill-ph bill-ph changed the title fix(reshard): keep cancelled operations consistent with warehouse state fix(reshard): pin maintenance backend before NOLOGIN Jul 30, 2026
@bill-ph bill-ph changed the title fix(reshard): pin maintenance backend before NOLOGIN fix(reshard): authenticate maintenance backend before NOLOGIN Jul 30, 2026
@bill-ph
bill-ph marked this pull request as ready for review July 30, 2026 11:16
@bill-ph bill-ph changed the title fix(reshard): authenticate maintenance backend before NOLOGIN fix(reshard): fence maintenance through direct primary Jul 30, 2026
@bill-ph
bill-ph merged commit a057d8e into main Jul 30, 2026
31 checks passed
@bill-ph
bill-ph deleted the codex/fix-reshard-cancel-e2e branch July 30, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant