You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Restore production health after the failed deployment #6433
🤖 Generated by the Agentic Engineer (interactive Codex session)
Problem
Production deployment lost Kubernetes API readiness after a node-update failure. Further application rollouts cannot be verified while recovery is incomplete.
Evidence
The deployment and restore-to-main run failed both the deployment and subsequent recovery. The recovery job completed unsuccessfully at 13:44 UTC on 16 November 2026 because the API readiness probe timed out.
The independent platform health helper also failed to read every required resource group and reported PLATFORM-HEALTH=UNKNOWN.
The public website still responds; that alone does not establish cluster health.
The existing reviewed recovery path completes and production is verified against the current default branch.
The Kubernetes API passes the existing stability check, and the platform health helper exits successfully.
Flux sources and reconciliation layers are current and healthy, with no unresolved speculative deployment.
Public website and portal entry checks pass at the restored revision.
Resume affected application rollouts only after the deployment and live-health evidence agree.
Keep credentials and persistence intact. Detailed operational evidence stays in the existing deployment logs; no customer data is required for this investigation.
The deployment failed during node scaling after provider placement retries were exhausted; the subsequent Kubernetes API stability check also failed.
The automatic restore-to-main job is now completed with failure. Its initial API readiness probe timed out, and its cluster-update and subsequent stability steps were skipped. The automatic recovery is no longer running.
The observed failure is reachability/readiness; these logs do not establish a credential rejection.
The public home page and portal entry page returned HTTP 200 in this check. That does not verify login, Flux health or restoration to the current main revision.
The next recovery check needs API reachability first, followed by the reviewed restore-to-main path and exact-revision reconciliation readback. Retrying the deployment alone would repeat the failed prerequisite. Detailed diagnostics remain in the Actions jobs; no cluster or provider changes were made by this check.
Read-only investigation now distinguishes the stable endpoint failure from cluster/API failure: direct control-plane Kubernetes readiness probes pass, observed nodes are Ready, and direct Talos etcd service reads are healthy. The production floating IP is attached in the provider to a different control plane from the unique node that currently has that address locally. A fresh stable-endpoint probe still times out.
The deployed KSail endpoint-reconcile path unconditionally assigns an existing floating IP to the first listed control plane. This can disagree with the node holding Talos’s VIP lease; it matches the attachment recorded in the failed deploy. I am preparing an upstream regression/fix and a narrowly bounded recovery plan. Production has not been mutated by these diagnostics. Ordinary merge-queue retries remain held; no credential rejection was found.
The permanent floating-IP preservation safeguard is merged in KSail #8417. The additional inventory/update-order safeguard in KSail #8418 has a clean current-head Codex review and is completing CI. No production provider assignment has been made.
Fresh recovery observations still match the prepared plan, but an older current-main CD attempt is queued with no jobs. Both cancellation endpoints refuse it as not yet queued while its read surfaces still report queued. This inconsistent run state is not deployment-lock clearance. Recovery is holding that prerequisite; deployment-dependent work remains parked until stable API readiness, fixed deployment-tool adoption and current-main recovery are verified.
The approved endpoint repair completed successfully. The provider action and assignment readback matched the prepared plan; readiness through the stable Kubernetes API now passes. All observed nodes are Ready and the canonical platform health check exits zero.
The old queued CD attempt is terminal cancelled, with no deployment applied. Automatic Restore Stranded Prod is temporarily paused as incident containment so it cannot reintroduce the old deployment tool. Normal CI and deployment checks remain in place. This pause must be removed after fixed-tool adoption and current-main recovery are verified.
Recovery is not complete. KSail #8418 still needs its same-head vulnerability rescan after a hosted runner lost communication; platform #6442 owns adoption of the release containing both safeguards. Production queues remain held until that adoption and the reviewed current-main recovery have their exact-revision readbacks.
Both permanent safeguards are merged in KSail #8417 and #8418. Native Release produced v7.215.4 at b39d383a52bc52a414e9a7c04f4621f8b7484fd9; native CD 43919875908 is packaging that combined version. Published CLI checksums and embedded-source proof, coherent platform adoption in #6442 and reviewed current-main recovery are still pending.
The approved endpoint repair is complete. Production queues remain held. Automatic Restore Stranded Prod remains temporarily paused and must be re-enabled after fixed-tool adoption and current-main recovery are verified. Further CI-producing rollout changes are being held while the critical publication/recovery path gets its runner evidence.
KSail v7.215.4 is now publicly published at b39d383a52bc52a414e9a7c04f4621f8b7484fd9; its native CD run 43919875908 completed successfully. Published checksum metadata, downloaded bytes and the CLI's embedded source identity agree.
Recovery remains held. Follow-up investigation reproduced an unsafe control-plane removal choice in the ordinary update path; KSail #8439 owns the bounded selection and lifecycle repair. Coherent adoption in #6442 also has full validation and configuration prerequisites to resolve without reducing enforcement. The failed-candidate healing path and application/identity compatibility must be safe before any production attempt.
The approved endpoint repair remains complete; no further production mutation has been made. Automatic Restore Stranded Prod is still temporarily paused and must be re-enabled and read back after fixed-tool current-main recovery is verified. Production queues remain held.
An intervening automated recovery changed the live baseline. The earlier cleanup plan has been withdrawn and its private evidence preserved; it must not be executed.
Fresh canonical platform health, datastore-quorum and API checks pass. Exact source, configuration and data acceptance and safe-tool delivery remain open, so recovery remains in progress.
A newer scheduled backup is still in progress. Its completion and data recoverability remain separate acceptance gates; the earlier health and source checks do not satisfy them.
The obsolete merge-group deployment and its automatic recovery job are now terminal. Both failed before executing any runner steps, and the run has no pending deployment approvals. The final native census found no active production writer, and the live baseline and application revision remained unchanged.
A temporary native production-entry hold remains verified. Restoring its original configuration and re-enabling Automatic Restore Stranded Prod with live readback are mandatory final gates after safe current-main recovery and acceptance.
Activity
chore: agent-claim #6433 nonce=4399a0233776b2382e417dc3a6164363
devantler commentedon Nov 16, 2026
Independent readback of run 43871939266:
The next recovery check needs API reachability first, followed by the reviewed restore-to-main path and exact-revision reconciliation readback. Retrying the deployment alone would repeat the failed prerequisite. Detailed diagnostics remain in the Actions jobs; no cluster or provider changes were made by this check.
devantler commentedon Nov 16, 2026
Read-only investigation now distinguishes the stable endpoint failure from cluster/API failure: direct control-plane Kubernetes readiness probes pass, observed nodes are Ready, and direct Talos etcd service reads are healthy. The production floating IP is attached in the provider to a different control plane from the unique node that currently has that address locally. A fresh stable-endpoint probe still times out.
The deployed KSail endpoint-reconcile path unconditionally assigns an existing floating IP to the first listed control plane. This can disagree with the node holding Talos’s VIP lease; it matches the attachment recorded in the failed deploy. I am preparing an upstream regression/fix and a narrowly bounded recovery plan. Production has not been mutated by these diagnostics. Ordinary merge-queue retries remain held; no credential rejection was found.
chore: agent-claim #6433 nonce=ae4e0b51d786c9811f793e37ed155d59
chore: agent-claim #6433 nonce=2b1e5e61d5aff00818cbabf5f13533e4
devantler commentedon Nov 16, 2026
The permanent floating-IP preservation safeguard is merged in KSail #8417. The additional inventory/update-order safeguard in KSail #8418 has a clean current-head Codex review and is completing CI. No production provider assignment has been made.
Fresh recovery observations still match the prepared plan, but an older current-main CD attempt is queued with no jobs. Both cancellation endpoints refuse it as not yet queued while its read surfaces still report queued. This inconsistent run state is not deployment-lock clearance. Recovery is holding that prerequisite; deployment-dependent work remains parked until stable API readiness, fixed deployment-tool adoption and current-main recovery are verified.
2 remaining items
devantler commentedon Nov 16, 2026
The approved endpoint repair completed successfully. The provider action and assignment readback matched the prepared plan; readiness through the stable Kubernetes API now passes. All observed nodes are Ready and the canonical platform health check exits zero.
The old queued CD attempt is terminal
cancelled, with no deployment applied. Automatic Restore Stranded Prod is temporarily paused as incident containment so it cannot reintroduce the old deployment tool. Normal CI and deployment checks remain in place. This pause must be removed after fixed-tool adoption and current-main recovery are verified.Recovery is not complete. KSail #8418 still needs its same-head vulnerability rescan after a hosted runner lost communication; platform #6442 owns adoption of the release containing both safeguards. Production queues remain held until that adoption and the reviewed current-main recovery have their exact-revision readbacks.
chore: agent-claim #6433 nonce=c16819f2c941b51fba71fdde3dfe51fe
devantler commentedon Nov 16, 2026
Both permanent safeguards are merged in KSail #8417 and #8418. Native Release produced
v7.215.4atb39d383a52bc52a414e9a7c04f4621f8b7484fd9; native CD 43919875908 is packaging that combined version. Published CLI checksums and embedded-source proof, coherent platform adoption in #6442 and reviewed current-main recovery are still pending.The approved endpoint repair is complete. Production queues remain held. Automatic Restore Stranded Prod remains temporarily paused and must be re-enabled after fixed-tool adoption and current-main recovery are verified. Further CI-producing rollout changes are being held while the critical publication/recovery path gets its runner evidence.
chore: agent-claim #6433 nonce=1fd0ede00a77163e08e88e3d2b988e64
devantler commentedon Nov 16, 2026
KSail
v7.215.4is now publicly published atb39d383a52bc52a414e9a7c04f4621f8b7484fd9; its native CD run 43919875908 completed successfully. Published checksum metadata, downloaded bytes and the CLI's embedded source identity agree.Recovery remains held. Follow-up investigation reproduced an unsafe control-plane removal choice in the ordinary update path; KSail #8439 owns the bounded selection and lifecycle repair. Coherent adoption in #6442 also has full validation and configuration prerequisites to resolve without reducing enforcement. The failed-candidate healing path and application/identity compatibility must be safe before any production attempt.
The approved endpoint repair remains complete; no further production mutation has been made. Automatic Restore Stranded Prod is still temporarily paused and must be re-enabled and read back after fixed-tool current-main recovery is verified. Production queues remain held.
chore: agent-claim #6433 nonce=39ff4d1d722d507774fa35c29a179bbb
chore: agent-claim #6433 nonce=57d6587215cbd915e89df596beca63ad
chore: agent-claim #6433 nonce=acc5ae3e1a0f5c058968cfb3dabd8cab
chore: agent-claim #6433 nonce=67dce13609b449ada2bc8ddfa82b22ca
devantler commentedon Nov 16, 2026
An intervening automated recovery changed the live baseline. The earlier cleanup plan has been withdrawn and its private evidence preserved; it must not be executed.
Fresh canonical platform health, datastore-quorum and API checks pass. Exact source, configuration and data acceptance and safe-tool delivery remain open, so recovery remains in progress.
A newer scheduled backup is still in progress. Its completion and data recoverability remain separate acceptance gates; the earlier health and source checks do not satisfy them.
The obsolete merge-group deployment and its automatic recovery job are now terminal. Both failed before executing any runner steps, and the run has no pending deployment approvals. The final native census found no active production writer, and the live baseline and application revision remained unchanged.
A temporary native production-entry hold remains verified. Restoring its original configuration and re-enabling Automatic Restore Stranded Prod with live readback are mandatory final gates after safe current-main recovery and acceptance.
chore: agent-claim #6433 nonce=def87dd830f673ae5452e134f1825e96
chore: agent-claim #6433 nonce=f13772f93287577aa15eabfc84c06393
chore: agent-claim #6433 nonce=303bb1c8047faba94e27cb7cf15266d1