feat(#10 pr2c): FINAL ACCEPTANCE — READY→ACTIVE + Redis Sentinel failover + SIGKILL matrix + RPO/RTO - #22
Merged
Merged
Conversation
…ver, SIGKILL matrix, RPO/RTO The PR2c acceptance milestone that closes #10, per the merged design §5. Frozen scope, one draft. 1. READY→ACTIVE activation: `HaControlFencing.activate()` — the terminal signed forward-CAS on the cutover head; B becomes the live write authority. Carries the ratified READY binding forward as the durable go-live proof; idempotent only if it reproduces byte-identical signed ACTIVE evidence; ACTIVE is terminal (a new promotion needs a new command+epoch). `cutover()` reader already public. 2. Real Redis Sentinel/quorum (tsk-ha-sentinel-drill, 4/4): 1 master + 1 replica + 3 sentinels (quorum 2), AOF everysec, on a dedicated network with STATIC IPs so sentinels monitor the master by fixed IP and fail over even when the crashed master's container exits (Docker DNS drops). A real master crash (SHUTDOWN NOSAVE) → automatic promotion; the committed+replica-ack'd (WAIT) fence SURVIVES (RPO=0), stays monotonic (epoch 2 claims, epoch 1 refused — no rollback), RTO measured. Whole topology + drill run via docker compose (ci/redis-sentinel/, run.sh docker-waits on the drill so the intentional master exit doesn't abort the run). RedisFencingStore needed NO change — Sentinel is a client-wiring concern. 3. Child-process SIGKILL/restart matrix (tsk-cutover-sigkill-drill + tsk-cutover-worker, 11/11): the full cutover (bind→advance→import→ready→activate) driven through CHILD processes killed by a REAL SIGKILL. before-commit (SIGKILL inside the tx via the transactor's onBeforeCommit seam) → PostgreSQL rolls back, phase UNCHANGED, no torn state. after-commit → a fresh child re-runs the SAME transition and resumes IDEMPOTENTLY (phase advanced exactly once, byte-identical signed evidence). Reaches ACTIVE. advanceEpoch crashes on its 2nd tx (Redis claimed, FENCED not committed) and still resumes correctly. 4. Failover/partition recovery: covered by (2) master failover + (3) process-crash resume. 5. Measured per-fault RPO/RTO: printed by both drills (Sentinel: RPO=0, RTO measured; SIGKILL: RPO=0, RTO ~avg-ms per phase). CI: test:sigkill (3 independent PG16 + Redis) + test:sentinel (docker compose) wired fail-not-skip. No production code beyond `activate()`; no DDL change (ACTIVE was already in the cutover CHECK/terminal set) → HA_CONTROL_MANIFEST_DIGEST unchanged. Regression: control-cutover 10/10, ha-fencing 20/20, unit 23/23, typecheck clean. Handoff file (test key material) is gitignored + deleted by the drill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…node quorum + enforced WAIT; live partition Addresses codex-1's 3 frozen-acceptance blockers on 173c55d: CRIT — activation was incomplete (only control READY→ACTIVE). Added GOVERNED B source-authority activation: `HaControlFencing.activateSource()` requires the ACTIVE head for this exact command/epoch, re-verifies B's BFinalizedReceipt, asserts the ACTIVE evidence pins the SAME B (bReceiptDigest+n+bSystemId), then GUARD-signs (ed25519) B's genesis source lease AT THE PROMOTED epoch (leaseEpoch==targetEpoch, holder==B, commandId==cutover). B verifies it with the guard PUBLIC key (unforgeable). New drill tsk-b-activation-drill (3/3): B installs the governed lease, seeds its source checkpoint to the verified (N, head@N), mints its SourceFenceReadyToken, and ORIGINATES sequence N+1 through the real fenced outbox (chaining from head@N) — while the OLD A, at the prior epoch with a revoked lease, is DENIED (no split-brain). HIGH — Sentinel was 2 data nodes + a caller-side WAIT. Now 1 master + 2 REPLICAS + 3 sentinels (quorum 2), AOF everysec, min-replicas-to-write 1 + min-replicas-max-lag. RPO=0 is now ENFORCED IN THE CLAIM PATH: `RedisFencingStore` takes an optional FenceDurabilityPolicy {waitReplicas, waitTimeoutMs} and issues Redis WAIT after the epoch CAS, failing closed if the replica quorum did not ACK. Sentinel drill uses it (no manual WAIT); RTO measured against a fresh reconnecting client (data-plane, ~5s) not one socket's slow follow. HIGH — added a LIVE split-brain partition (not a crash). tsk-ha-partition-drill (4/4): the old master is isolated ALIVE (docker network disconnect); it REFUSES writes (min-replicas-to-write → NOREPLICAS, no progress) while the surviving quorum promotes a new writable+monotonic master (fence intact, RPO=0); on HEAL the old master is demoted to a replica and reconciles to the new epoch. Per-fault RPO/RTO reported. Topology now runs on the host (published ports + ioredis natMap) with static IPs + healthchecks (no startup false-failover); run.sh brings up per-drill and waits for replica sync. No DDL change (activateSource signs off-DB; ACTIVE unchanged) → HA_CONTROL_MANIFEST_DIGEST unchanged. Regression: control-cutover 10/10, sigkill 11/11, ha-fencing 20/20, redis 6/6, unit 23/23, typecheck clean. #10 stays OPEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…idence; typed WAIT; abrupt crash; RTO truth; secret hygiene Addresses codex-1's 5 bounded blockers on a2f3a7b: 1 (CRIT) activateSource authorization/recovery: it no longer accepts a per-call guard private key or caller-chosen holder/leaseId/expiry. It signs with the CONFIGURED control authority (policy.sourceGuard, held as deployment config), DERIVES the holder from the ratified B identity (== BFinalizedReceipt.bKeyId, which must equal the bKeyId ratified into the ACTIVE head), uses a deterministic leaseId, and PERSISTS the exact signed grant tuple+digest into new durable evidence `tsk_ha_source_activation`. A retry REHYDRATES the byte-identical grant; a different/conflicting activation for the stream is quarantined. HA_CONTROL manifest re-pinned on alpine (a80069c5…). Drill asserts rehydration + holder==ratified B. 2 (HIGH) Redis WAIT post-effect ambiguity: after a SUCCESSFUL CAS whose replica quorum did not ACK, claim() no longer returns an ordinary false (which means "CAS refused"). It throws a typed FenceDurabilityUncertainError carrying the exact stored tuple → the caller reconciles + fails closed. Hermetic regression (single node, no replicas) proves the typed throw AND that a refused CAS still returns plain false. 3 (HIGH) nondeterministic crash: the sentinel drill now crashes the master with an ABRUPT `docker kill -s KILL` (real crash, deterministic exit) wrapped for exit-code robustness, replacing the graceful/flaky redis-cli SHUTDOWN NOSAVE (which exited 137 and threw). 4 (HIGH) RTO truth: both fault drills now measure RTO from fault → NEW AUTHORITY DURABLY WRITABLE (a successful enforced-WAIT epoch-2 claim on the promoted master), with the fence-readable / old-master-refusal times reported as labelled SUB-times. A fresh reconnecting client is created AFTER promotion (data-plane RTO, not one socket's slow follow): sentinel ~4.5s, partition ~12s. 5 (HIGH) secret hygiene: the SIGKILL handoff (carries the control HMAC secret) is now created in the OS temp dir with mode 0600 and removed in an OUTER try/finally on EVERY exit path; a final check proves it is gone. No new architecture. Regression: b-activation 3/3, sentinel 3/3, partition 4/4, sigkill 12, control-cutover 10/10, ha-fencing 20/20, redis 7/7, receiver 10/10, unit 23/23, typecheck clean. #10 stays OPEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…connection (durability continuity) Addresses codex-1's final HIGH on 8d448ef (verified against official Redis docs: WAIT tracks prior writes in the context of the CURRENT connection / per-client replication offset, and ioredis auto-reconnects). RedisFencingStore.claim() previously ran the CAS eval() and the WAIT as two separate round-trips on the auto-reconnecting Sentinel client. If the socket/master switched between them, WAIT could bind a fresh server connection whose replication offset does not cover the prior CAS write — a durability hole the typed under-ack signal did not cover. Fix: the durable path now issues a SINGLE pipeline — [CLIENT ID, EVAL(CAS), WAIT, CLIENT ID] — written to ONE socket (a pipeline is not split across a reconnect), so WAIT's offset binds the same connection that ran the CAS. After the CAS wrote, ANY anomaly is durability-UNKNOWN → FenceDurabilityUncertainError: a WAIT-side error (cut between EVAL and WAIT), an aborted pipeline (connection lost), a changed CLIENT ID (path switch), or an under-ack. A refused CAS is still an ordinary false; the non-durable path is unchanged. Hermetic deterministic regression (stubbed pipeline exec vector) proves each path — WAIT-cut, identity change, under-ack, aborted pipeline → typed Uncertain; acked-same-connection → true; refused → false. redis 8/8. Real-cluster: sentinel 3/3 (RTO ~3.7s), partition 4/4 (RTO ~11.2s) — unchanged. No new architecture. #10 OPEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… CLIENT ID bracket is load-bearing Evidence-backed (Redis WAIT is per-connection; ioredis resends unfulfilled commands on reconnect by default) — the CLIENT ID bracket catches a WAIT resent on a fresh Sentinel-promoted connection. Comment only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…S+WAIT (close both continuity bypasses) The R3 pipeline + CLIENT-ID bracket did NOT satisfy "dedicated non-retrying physical connection" (codex-1, exact 9e925af). Two concrete bypasses remained: (1) pipeline.exec() REJECTION escaped raw before wrapping, not FenceDurabilityUncertainError; (2) ioredis can REPLAY the whole unfulfilled pipeline on a new master — both CLIENT IDs then match on the new connection, the replayed EVAL returns 0 (the first already wrote), and claim() returned an ordinary false before continuity validation. CLIENT IDs are server-local and can coincide across masters. Verified against official Redis WAIT docs (per-connection / per-client offset) — an ordinary auto-reconnecting ioredis client is insufficient. Surgical closure: the durable claim now acquires a DEDICATED physical connection to the current master via redis.duplicate() with ALL reconnect/retry/resend DISABLED (retryStrategy:()=>null, maxRetriesPerRequest:0, enableOfflineQueue:false, autoResendUnfulfilledCommands:false, reconnectOnError:()=>false, role:master, lazyConnect). It connects (pre-dispatch failure propagates — the claim never started), dispatches EVAL then WAIT on that ONE socket (WAIT binds this connection's offset, which covers the CAS), and treats ANY disconnect/rejection AFTER the CAS was dispatched as FenceDurabilityUncertainError; the connection is closed in finally. No pipeline, no CLIENT-ID bracket, no replay path — bypass 2 is structurally impossible. Regressions (hermetic, dedicated-connection stub): EVAL-reject-after-dispatch, WAIT-reject-after-dispatch, under-ack → typed Uncertain; acked-same-connection → true; genuinely-refused CAS0 → false; pre-dispatch connect failure PROPAGATES (not a silent false/uncertain); and a STRUCTURAL assertion that the connection is created with reconnect/retry/offline-queue/RESEND all disabled (so the replay-to-CAS0 path cannot occur). redis 8/8. Real cluster: sentinel 3/3 (RTO ~4.4s), partition 4/4 (RTO ~11.3s) — duplicate() resolves the promoted master correctly across failover. No new architecture. #10 stays OPEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rblake2320
marked this pull request as ready for review
July 19, 2026 16:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR2c — FINAL ACCEPTANCE (closes #10)
The acceptance milestone per the merged design §5. Frozen scope. Reviews R1 (
a2f3a7b) + R2 (8d448ef) close all acceptance gates codex-1 flagged.1. READY → ACTIVE + GOVERNED B source-authority activation
activate()— terminal signed forward-CAS; B becomes the live authority.activateSource()— mints B's unforgeable, governed source capability using the configured control authority (policy.sourceGuard, never a per-call key). It derives the holder from the ratified B identity (== BFinalizedReceipt.bKeyId, which must equal the bKeyId ratified into the ACTIVE head), uses a deterministic leaseId, and persists the exact signed grant into durable evidence (tsk_ha_source_activation). A retry rehydrates the byte-identical grant; a conflicting activation is quarantined.test:b-activation(3/3): B installs the governed lease, seeds its checkpoint to the verified(N, head@N), mints itsSourceFenceReadyToken, and originates sequence N+1 through the real fenced outbox while the old A (prior epoch, revoked lease) is DENIED.2. Real Redis Sentinel/quorum + ENFORCED, UNAMBIGUOUS WAIT —
test:sentinel(3/3)min-replicas-to-write 1.RedisFencingStore.claim()issues Redis WAIT after the epoch CAS. RPO=0 is enforced; a successful CAS whose replica quorum did not ACK throws a typedFenceDurabilityUncertainError(carrying the stored tuple to reconcile) — never conflated with an ordinaryfalse(CAS refused). Hermetic regression proves both.docker kill -s KILL(deterministic). RTO measured to durably writable.3. Live split-brain PARTITION —
test:partition(4/4)Old master isolated alive; refuses writes (
NOREPLICAS) while the quorum promotes a new durably writable + monotonic master (fence intact, RPO=0); on heal the old master demotes to a replica and reconciles.4. Child-process SIGKILL / restart matrix —
test:sigkill(12)Full cutover through child processes killed by a real SIGKILL: before-commit → PG rollback; after-commit → idempotent resume to ACTIVE. The handoff (control HMAC secret) is created 0600 in the OS temp dir and removed in an outer try/finally on every exit; a final check proves cleanup.
5. Measured per-fault RPO / RTO
Each fault drill measures RPO = 0 and RTO = fault → new authority durably writable (successful enforced-WAIT epoch-2 claim), with fence-readable / old-master-refusal reported as labelled sub-times (sentinel ~4.5s, partition ~12s).
Safety
No new architecture.
HA_CONTROL_MANIFEST_DIGESTre-pinned onpostgres:16-alpinefor the one added evidence table. Regression: control-cutover 10/10, ha-fencing 20/20, redis 7/7, receiver 10/10, unit 23/23, typecheck clean. Mechanism-only on a single host (processes, not independent physical failure domains).