Skip to content

Restore production health after the failed deployment #6433

Open
2 of 4 issues completed
Open

Description

@devantler

🤖 Generated by the Agentic Engineer (interactive Codex session)

Problem

Production deployment lost Kubernetes API readiness after a node-update failure. Further application rollouts cannot be verified while recovery is incomplete.

Evidence

  • The deployment and restore-to-main run failed both the deployment and subsequent recovery. The recovery job completed unsuccessfully at 13:44 UTC on 16 November 2026 because the API readiness probe timed out.
  • The independent platform health helper also failed to read every required resource group and reported PLATFORM-HEALTH=UNKNOWN.
  • The public website still responds; that alone does not establish cluster health.
  • Operator-access PR feat(portal): deliver the customer directory credential #6417 has been removed from the merge queue while recovery is unresolved.

Acceptance criteria

  • The existing reviewed recovery path completes and production is verified against the current default branch.
  • The Kubernetes API passes the existing stability check, and the platform health helper exits successfully.
  • Flux sources and reconciliation layers are current and healthy, with no unresolved speculative deployment.
  • Public website and portal entry checks pass at the restored revision.
  • Resume affected application rollouts only after the deployment and live-health evidence agree.

Keep credentials and persistence intact. Detailed operational evidence stays in the existing deployment logs; no customer data is required for this investigation.

Activity

  1. added theissue type on Nov 16, 2026
  2. moved this to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 16, 2026
  3. added a commit that references this issue on Nov 16, 2026
  4. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Independent readback of run 43871939266:

    • The deployment failed during node scaling after provider placement retries were exhausted; the subsequent Kubernetes API stability check also failed.
    • The automatic restore-to-main job is now completed with failure. Its initial API readiness probe timed out, and its cluster-update and subsequent stability steps were skipped. The automatic recovery is no longer running.
    • The observed failure is reachability/readiness; these logs do not establish a credential rejection.
    • The public home page and portal entry page returned HTTP 200 in this check. That does not verify login, Flux health or restoration to the current main revision.

    The next recovery check needs API reachability first, followed by the reviewed restore-to-main path and exact-revision reconciliation readback. Retrying the deployment alone would repeat the failed prerequisite. Detailed diagnostics remain in the Actions jobs; no cluster or provider changes were made by this check.

  5. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Read-only investigation now distinguishes the stable endpoint failure from cluster/API failure: direct control-plane Kubernetes readiness probes pass, observed nodes are Ready, and direct Talos etcd service reads are healthy. The production floating IP is attached in the provider to a different control plane from the unique node that currently has that address locally. A fresh stable-endpoint probe still times out.

    The deployed KSail endpoint-reconcile path unconditionally assigns an existing floating IP to the first listed control plane. This can disagree with the node holding Talos’s VIP lease; it matches the attachment recorded in the failed deploy. I am preparing an upstream regression/fix and a narrowly bounded recovery plan. Production has not been mutated by these diagnostics. Ordinary merge-queue retries remain held; no credential rejection was found.

  6. added a commit that references this issue on Nov 16, 2026
  7. added a commit that references this issue on Nov 16, 2026
  8. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The permanent floating-IP preservation safeguard is merged in KSail #8417. The additional inventory/update-order safeguard in KSail #8418 has a clean current-head Codex review and is completing CI. No production provider assignment has been made.

    Fresh recovery observations still match the prepared plan, but an older current-main CD attempt is queued with no jobs. Both cancellation endpoints refuse it as not yet queued while its read surfaces still report queued. This inconsistent run state is not deployment-lock clearance. Recovery is holding that prerequisite; deployment-dependent work remains parked until stable API readiness, fixed deployment-tool adoption and current-main recovery are verified.

  9. 2 remaining items

  10. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    The approved endpoint repair completed successfully. The provider action and assignment readback matched the prepared plan; readiness through the stable Kubernetes API now passes. All observed nodes are Ready and the canonical platform health check exits zero.

    The old queued CD attempt is terminal cancelled, with no deployment applied. Automatic Restore Stranded Prod is temporarily paused as incident containment so it cannot reintroduce the old deployment tool. Normal CI and deployment checks remain in place. This pause must be removed after fixed-tool adoption and current-main recovery are verified.

    Recovery is not complete. KSail #8418 still needs its same-head vulnerability rescan after a hosted runner lost communication; platform #6442 owns adoption of the release containing both safeguards. Production queues remain held until that adoption and the reviewed current-main recovery have their exact-revision readbacks.

  11. added a commit that references this issue on Nov 16, 2026
  12. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Both permanent safeguards are merged in KSail #8417 and #8418. Native Release produced v7.215.4 at b39d383a52bc52a414e9a7c04f4621f8b7484fd9; native CD 43919875908 is packaging that combined version. Published CLI checksums and embedded-source proof, coherent platform adoption in #6442 and reviewed current-main recovery are still pending.

    The approved endpoint repair is complete. Production queues remain held. Automatic Restore Stranded Prod remains temporarily paused and must be re-enabled after fixed-tool adoption and current-main recovery are verified. Further CI-producing rollout changes are being held while the critical publication/recovery path gets its runner evidence.

  13. added a commit that references this issue on Nov 16, 2026
  14. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    KSail v7.215.4 is now publicly published at b39d383a52bc52a414e9a7c04f4621f8b7484fd9; its native CD run 43919875908 completed successfully. Published checksum metadata, downloaded bytes and the CLI's embedded source identity agree.

    Recovery remains held. Follow-up investigation reproduced an unsafe control-plane removal choice in the ordinary update path; KSail #8439 owns the bounded selection and lifecycle repair. Coherent adoption in #6442 also has full validation and configuration prerequisites to resolve without reducing enforcement. The failed-candidate healing path and application/identity compatibility must be safe before any production attempt.

    The approved endpoint repair remains complete; no further production mutation has been made. Automatic Restore Stranded Prod is still temporarily paused and must be re-enabled and read back after fixed-tool current-main recovery is verified. Production queues remain held.

  15. added 4 commits that reference this issue on Nov 16, 2026
  16. devantler commented on Nov 16, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    An intervening automated recovery changed the live baseline. The earlier cleanup plan has been withdrawn and its private evidence preserved; it must not be executed.

    Fresh canonical platform health, datastore-quorum and API checks pass. Exact source, configuration and data acceptance and safe-tool delivery remain open, so recovery remains in progress.

    A newer scheduled backup is still in progress. Its completion and data recoverability remain separate acceptance gates; the earlier health and source checks do not satisfy them.

    The obsolete merge-group deployment and its automatic recovery job are now terminal. Both failed before executing any runner steps, and the run has no pending deployment approvals. The final native census found no active production writer, and the live baseline and application revision remained unchanged.

    A temporary native production-entry hold remains verified. Restoring its original configuration and re-enabling Automatic Restore Stranded Prod with live readback are mandatory final gates after safe current-main recovery and acceptance.

  17. added 3 commits that reference this issue on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions