Skip to content

fix: make FSR rotation quota-safe - #1

Merged
heemankv merged 2 commits into
mainfrom
fsr-safe-rotation
Sep 11, 2026
Merged

heemankv merged 2 commits into
mainfrom
fsr-safe-rotation

Conversation

@heemankv

@heemankv heemankv commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Make Fast Snapshot Restore rotation safe under the five snapshot/AZ quota and recover automatically from partial or previously failed transitions.

Rotation contract

For snapshots N (newest), N-1, and N-2:

  1. Keep N-1 FSR-enabled while N is being prepared.
  2. Disable FSR on N-2 and older to free quota.
  3. Reconcile N from AWS state, enabling each missing AZ independently.
  4. Wait until N is enabled in every configured AZ and each AZ has at least one FastSnapshotRestoreCreditsBalance credit.
  5. Disable FSR on N-1 only after N is usable.
  6. Permit the normal retention controller to delete N-2 only after that readiness gate.

Why

The current controller submits all AZs in one request, does not retry a missing AZ, treats failed as terminal, and deletes expired snapshots before FSR reconciliation. With a quota of five, that left the Sep 10 Pathfinder snapshot enabled only in ap-northeast-1c while Sep 8 and Sep 9 consumed four slots.

Changes

  • Treat AWS state as authoritative and recover stale/failed annotations.
  • Enable missing AZs independently and idempotently.
  • Pre-drain FSR only from N-2 and older; preserve N-1 during handover.
  • Gate readiness on both FSR state and CloudWatch volume-creation credits.
  • Defer expired VolumeSnapshot deletion until the newest snapshot passes the readiness gate.
  • Treat disabling as not yet cold.
  • Add regression tests for quota saturation, partial enable recovery, credit gating, and deletion gating.
  • Add CI plus immutable GHCR image publishing for Karnot-owned builds.

Deployment artifacts

  • Image: ghcr.io/karnotxyz/gemini:sha-0e10120
  • Digest: sha256:d40faf04542b854b7bb7c35b38bf13606d428affc4676e8863f9e960286db073
  • IAM: karnotxyz/paradex-cloud-terraform#137
  • Mainnet GitOps: karnotxyz/paradex-cloud-argocd#846

Validation

  • go test ./...
  • go vet ./...
  • git diff --check
  • CI image build and anonymous image pull

No AWS or Kubernetes resources are changed by this PR.

@heemankv
heemankv merged commit 70ab76e into main Sep 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant