From 4535f75e7b06bca29b34de4171378e1a33f94d7f Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 12:40:51 +0900 Subject: [PATCH 1/8] ci: restore the disposable workload pool before the contract suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The K8s contract tests have failed on every main commit since 2026-08-05, 6 of 21 tests red: no disposable workload node found — refusing to touch system nodes timed out after 180s: pod should be scheduled Nothing was wrong with the tests or the product. The `workload` node pool was at zero nodes, and that pool is the only place the suite may schedule: its nodes carry korvid.dev/disposable=true, and the node-operation tests fail closed rather than fall back to a system node — correct behaviour, and the reason the symptom is a hard failure rather than a dangerous fallback. The workflow started and stopped the cluster but never checked that it had capacity, so once something scaled the pool to zero (a perf run, a cost sweep, a cleanup) nothing put it back and the suite stayed red for days. Scale the pool to one node when it is found empty, before fetching credentials. The pool name is a fixed literal rather than a repository variable so drifting settings cannot aim the scale-up at the system pool. Verified against the real cluster: scaled the pool to zero, ran the new step verbatim, watched it report `workload was at 0` and restore one node; re-ran with the node present and confirmed it is a no-op. Full contract suite then passed 21/21 locally against the restored cluster. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 22 ++++++++++++++++++++++ docs/dev/contract-tests.md | 6 ++++++ 2 files changed, 28 insertions(+) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index f522b163..af08c90b 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -38,6 +38,10 @@ jobs: env: AKS_RESOURCE_GROUP: ${{ vars.AKS_RESOURCE_GROUP }} AKS_CLUSTER_NAME: ${{ vars.AKS_CLUSTER_NAME }} + # The only pool the suite may schedule onto; its nodes carry + # korvid.dev/disposable=true. Fixed rather than a variable so a drifting + # setting cannot point the scale-up at the system pool. + WORKLOAD_NODEPOOL: workload steps: - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 with: @@ -75,6 +79,24 @@ jobs: az aks start -g "$AKS_RESOURCE_GROUP" -n "$AKS_CLUSTER_NAME" fi + - name: Ensure the disposable workload pool has a node + run: | + # The node-ops and resize contracts need a node labelled + # korvid.dev/disposable=true; they refuse to touch system nodes. A + # pool scaled to zero (by a perf run, a cost sweep, or a previous + # cleanup) makes them fail with "no disposable workload node found" + # or time out waiting for a pod to schedule, and nothing in the + # workflow put it back — the suite stayed red for days that way. + count=$(az aks nodepool show -g "$AKS_RESOURCE_GROUP" \ + --cluster-name "$AKS_CLUSTER_NAME" -n "$WORKLOAD_NODEPOOL" \ + --query count -o tsv) + if [ "$count" -lt 1 ]; then + echo "::notice title=Scaling workload pool::$WORKLOAD_NODEPOOL was at $count; scaling to 1 for the contract suite." + az aks nodepool scale -g "$AKS_RESOURCE_GROUP" \ + --cluster-name "$AKS_CLUSTER_NAME" -n "$WORKLOAD_NODEPOOL" \ + --node-count 1 + fi + - uses: azure/use-kubelogin@0ce7c36141aa27d4934872cf00b0120804c98a29 # v1.3 with: kubelogin-version: v0.2.15 diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index 22d2248f..624f462c 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -42,6 +42,12 @@ Node pools: the only node the node-operation tests (cordon/drain/evict) will touch; tests fail rather than fall back to a system node. +Because that pool is the suite's single point of failure, the workflow scales +it to one node when it finds it at zero. Nothing else restores it, and a pool +left at zero — by a performance run, a cost sweep, or a cleanup — fails the +node-operation and resize contracts with `no disposable workload node found` +or a pod-scheduling timeout until someone notices. + The cluster has AAD + Azure RBAC enabled with local accounts disabled; the workflow identity holds a minimal custom role (read/start/stop/list-user- credentials) plus AKS RBAC Cluster Admin, both scoped to the one cluster. From 51eba5737f3764c50edba8cf1e03e1ebbe9adf94 Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 13:04:08 +0900 Subject: [PATCH 2/8] ci: check the workload pool instead of trying to scale it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review caught that the scale-up could never work: the workflow identity holds a deliberately minimal role with agentPools/read and no write, so `az aks nodepool scale` returns 403 in CI. My local verification ran under my own credentials and so proved nothing about the OIDC identity — confirmed by reading the deployed role, which lists read/start/stop/listClusterUserCredential /agentPools/read and nothing else. Granting write would fix the symptom and widen the blast radius: the same permission that scales the workload pool also reshapes the system pool, which every guard in this workflow and every node-operation test exists to keep out of reach. Not worth it to save one command. Check and fail instead, with the pool named and the repair command spelled out. An empty pool is drift, not a setting — cost at rest is handled by stopping the cluster, and nothing in the repo scales this pool down — so the right response is a loud, diagnosable failure rather than silent repair. Verified both paths against the real cluster: at one node the guard passes; at zero it emits the error and would exit 1; and the command in the error message restores the pool as written. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 24 ++++++++++++++---------- docs/dev/contract-tests.md | 21 ++++++++++++++++----- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index af08c90b..40e11330 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -79,22 +79,26 @@ jobs: az aks start -g "$AKS_RESOURCE_GROUP" -n "$AKS_CLUSTER_NAME" fi - - name: Ensure the disposable workload pool has a node + - name: Guard — the disposable workload pool must have a node run: | # The node-ops and resize contracts need a node labelled - # korvid.dev/disposable=true; they refuse to touch system nodes. A - # pool scaled to zero (by a perf run, a cost sweep, or a previous - # cleanup) makes them fail with "no disposable workload node found" - # or time out waiting for a pod to schedule, and nothing in the - # workflow put it back — the suite stayed red for days that way. + # korvid.dev/disposable=true; they refuse to touch system nodes. When + # the pool is empty they fail with "no disposable workload node + # found" or time out waiting for a pod to schedule — six failures + # spread over four files, none of which name the actual cause. The + # suite stayed red for eight commits that way. + # + # Checked, not repaired: the workflow identity holds a deliberately + # minimal role with agentPools/read and no write, so it cannot scale + # the pool, and widening that role to auto-heal would let this + # workflow reshape the system pool too. Cost at rest is handled by + # stopping the cluster, so an empty pool is drift, not a setting. count=$(az aks nodepool show -g "$AKS_RESOURCE_GROUP" \ --cluster-name "$AKS_CLUSTER_NAME" -n "$WORKLOAD_NODEPOOL" \ --query count -o tsv) if [ "$count" -lt 1 ]; then - echo "::notice title=Scaling workload pool::$WORKLOAD_NODEPOOL was at $count; scaling to 1 for the contract suite." - az aks nodepool scale -g "$AKS_RESOURCE_GROUP" \ - --cluster-name "$AKS_CLUSTER_NAME" -n "$WORKLOAD_NODEPOOL" \ - --node-count 1 + echo "::error title=Workload pool is empty::$WORKLOAD_NODEPOOL has $count nodes, so the node-operation and resize contracts have nowhere to schedule. Restore it with: az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1" + exit 1 fi - uses: azure/use-kubelogin@0ce7c36141aa27d4934872cf00b0120804c98a29 # v1.3 diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index 624f462c..cf3989b9 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -42,11 +42,22 @@ Node pools: the only node the node-operation tests (cordon/drain/evict) will touch; tests fail rather than fall back to a system node. -Because that pool is the suite's single point of failure, the workflow scales -it to one node when it finds it at zero. Nothing else restores it, and a pool -left at zero — by a performance run, a cost sweep, or a cleanup — fails the -node-operation and resize contracts with `no disposable workload node found` -or a pod-scheduling timeout until someone notices. +Because that pool is the suite's single point of failure, the workflow checks +it before doing anything expensive and fails with a named cause and the exact +repair command. It does not scale the pool itself: the workflow identity holds +`agentPools/read` and no write, and granting write so the job could self-heal +would also let it reshape the system pool. Cost at rest is handled by stopping +the cluster, so a pool sitting at zero is drift to be repaired, not a setting +to be respected: + +```sh +az aks nodepool scale -g rg-korvid-contract-test \ + --cluster-name aks-korvid-contract-test -n workload --node-count 1 +``` + +Without the check, an empty pool surfaces as six failures across four test +files — `no disposable workload node found` and pod-scheduling timeouts — none +of which name the pool. The cluster has AAD + Azure RBAC enabled with local accounts disabled; the workflow identity holds a minimal custom role (read/start/stop/list-user- From 7e265a4d2a3f2bcc6d640917482ed99b07798a2e Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 13:44:55 +0900 Subject: [PATCH 3/8] ci: ask the cluster for a disposable node, not Azure for a count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Chasing the review notes about the unquoted count comparison turned up something worse: the count itself is not evidence. A stopped cluster reports agentPools[].count as 0 for the workload pool even when the pool is populated — observed directly, stopped reads 0 and starting the same cluster reads 1 with no scale operation in between. `az aks start` also returns before nodes finish registering, so a count read just after it can be 0 for a healthy cluster. The guard I wrote would have failed good runs. Ask the cluster for the thing the tests actually need instead: a Ready node labelled korvid.dev/disposable=true, polled with kubectl wait for up to five minutes so a slow start is tolerated. That is immune to the reporting quirk, needs no extra Azure permission, and dumps node labels on failure so the next person can see what the cluster really had. Also fixes the review findings that led here: the empty-string comparison that died with "integer expression expected" is gone with the count check, the step now runs under set -euo pipefail, and the WORKLOAD_NODEPOOL comment no longer describes a scale-up this workflow never performs. Verified on the real cluster: passes with the disposable node Ready, and fails with the named error when the selector matches nothing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 59 +++++++++++++++++------------- docs/dev/contract-tests.md | 20 ++++++---- 2 files changed, 46 insertions(+), 33 deletions(-) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index 40e11330..f5070e91 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -38,9 +38,8 @@ jobs: env: AKS_RESOURCE_GROUP: ${{ vars.AKS_RESOURCE_GROUP }} AKS_CLUSTER_NAME: ${{ vars.AKS_CLUSTER_NAME }} - # The only pool the suite may schedule onto; its nodes carry - # korvid.dev/disposable=true. Fixed rather than a variable so a drifting - # setting cannot point the scale-up at the system pool. + # Named only in the guard's repair hint. Fixed rather than a variable so + # a drifting setting cannot send someone to scale the system pool. WORKLOAD_NODEPOOL: workload steps: - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 @@ -79,28 +78,6 @@ jobs: az aks start -g "$AKS_RESOURCE_GROUP" -n "$AKS_CLUSTER_NAME" fi - - name: Guard — the disposable workload pool must have a node - run: | - # The node-ops and resize contracts need a node labelled - # korvid.dev/disposable=true; they refuse to touch system nodes. When - # the pool is empty they fail with "no disposable workload node - # found" or time out waiting for a pod to schedule — six failures - # spread over four files, none of which name the actual cause. The - # suite stayed red for eight commits that way. - # - # Checked, not repaired: the workflow identity holds a deliberately - # minimal role with agentPools/read and no write, so it cannot scale - # the pool, and widening that role to auto-heal would let this - # workflow reshape the system pool too. Cost at rest is handled by - # stopping the cluster, so an empty pool is drift, not a setting. - count=$(az aks nodepool show -g "$AKS_RESOURCE_GROUP" \ - --cluster-name "$AKS_CLUSTER_NAME" -n "$WORKLOAD_NODEPOOL" \ - --query count -o tsv) - if [ "$count" -lt 1 ]; then - echo "::error title=Workload pool is empty::$WORKLOAD_NODEPOOL has $count nodes, so the node-operation and resize contracts have nowhere to schedule. Restore it with: az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1" - exit 1 - fi - - uses: azure/use-kubelogin@0ce7c36141aa27d4934872cf00b0120804c98a29 # v1.3 with: kubelogin-version: v0.2.15 @@ -116,6 +93,38 @@ jobs: chmod 600 "$KUBECONFIG" echo "KUBECONFIG=$KUBECONFIG" >> "$GITHUB_ENV" + - name: Guard — a schedulable disposable node must exist + run: | + set -euo pipefail + # The node-ops and resize contracts only touch nodes labelled + # korvid.dev/disposable=true and fail closed rather than fall back to + # a system node. With no such node they produce six failures across + # four files — "no disposable workload node found" and pod-scheduling + # timeouts — none of which names the cause. The suite stayed red for + # eight commits that way. + # + # Asked of the cluster rather than of Azure: a stopped or + # still-starting cluster reports agentPools[].count as 0 even when + # the pool is populated, so the control-plane number is not evidence. + # This asks for exactly what the tests need, and waits, because + # `az aks start` returns before nodes finish registering. + # + # Checked, not repaired: the workflow identity holds a deliberately + # minimal role with agentPools/read and no write, and widening it so + # this job could scale the pool would also let it reshape the system + # pool that every other guard here exists to protect. + for _ in $(seq 1 30); do + if kubectl wait --for=condition=Ready node \ + -l korvid.dev/disposable=true --timeout=10s >/dev/null 2>&1; then + kubectl get nodes -l korvid.dev/disposable=true + exit 0 + fi + sleep 10 + done + echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready node labelled korvid.dev/disposable=true after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If the $WORKLOAD_NODEPOOL pool is empty, restore it with: az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1" + kubectl get nodes --show-labels || true + exit 1 + - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 with: python-version: "3.13" diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index cf3989b9..5b7db8ee 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -42,20 +42,24 @@ Node pools: the only node the node-operation tests (cordon/drain/evict) will touch; tests fail rather than fall back to a system node. -Because that pool is the suite's single point of failure, the workflow checks -it before doing anything expensive and fails with a named cause and the exact -repair command. It does not scale the pool itself: the workflow identity holds -`agentPools/read` and no write, and granting write so the job could self-heal -would also let it reshape the system pool. Cost at rest is handled by stopping -the cluster, so a pool sitting at zero is drift to be repaired, not a setting -to be respected: +Because that pool is the suite's single point of failure, the workflow waits +for a Ready node carrying `korvid.dev/disposable=true` before installing +anything, and fails with a named cause and the repair command if none appears. +It asks the cluster rather than Azure: a stopped or still-starting cluster +reports `agentPools[].count` as 0 even when the pool is populated, so the +control-plane number is not evidence. + +It checks but does not repair. The workflow identity holds `agentPools/read` +and no write, and granting write so the job could scale the pool would also +let it reshape the system pool. Cost at rest is handled by stopping the +cluster, so a pool sitting at zero is drift to be repaired by hand: ```sh az aks nodepool scale -g rg-korvid-contract-test \ --cluster-name aks-korvid-contract-test -n workload --node-count 1 ``` -Without the check, an empty pool surfaces as six failures across four test +Without the check, a missing node surfaces as six failures across four test files — `no disposable workload node found` and pod-scheduling timeouts — none of which name the pool. From e94823aa26a296efcbf65ff6c7a8ca081109695c Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 14:18:01 +0900 Subject: [PATCH 4/8] ci: require both scheduling labels and stop misreporting the cause MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings on the wait loop, all of which left a way for the guard to be green while the suite is red, or to blame the wrong thing: The selector only asked for korvid.dev/disposable=true, but every test pod in tests/contract/conftest.py carries a korvid.dev/pool=workload nodeSelector. Those labels sharing a node is a convention of the pool setup, not something this workflow enforced, so label drift would pass the guard and leave pods Pending — exactly the undiagnosable failure being removed. Both are now required on the same node. The budget was a loop count, so a registered-but-NotReady node spent each attempt on its own 10s timeout plus a 10s sleep: 10 minutes against an error message promising 5. Replaced with a wall-clock deadline, which holds for both paths. Every kubectl failure was swallowed and read as "no node", so an expired token, an RBAC denial, or an unreachable API server would burn the deadline and then advise a scale command that cannot help. The node list is now proven reachable once up front, and its stderr is reported. Verified on the real cluster: passes with both labels present; fails within the deadline when one label drifts; and an unreachable API server fails in one second naming the connection error rather than the pool. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 45 +++++++++++++++++++++--------- docs/dev/contract-tests.md | 9 ++++-- 2 files changed, 39 insertions(+), 15 deletions(-) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index f5070e91..a13d667e 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -96,32 +96,51 @@ jobs: - name: Guard — a schedulable disposable node must exist run: | set -euo pipefail - # The node-ops and resize contracts only touch nodes labelled + # The suite needs one node that satisfies both of its requirements at + # once: the node-operation contracts only touch # korvid.dev/disposable=true and fail closed rather than fall back to - # a system node. With no such node they produce six failures across - # four files — "no disposable workload node found" and pod-scheduling - # timeouts — none of which names the cause. The suite stayed red for - # eight commits that way. + # a system node, while every test pod carries a + # korvid.dev/pool=workload nodeSelector. Those labels riding the same + # node is a convention of the pool setup, not something enforced + # here, so both are required together — a node with only one of them + # would let this guard pass and leave pods Pending. + # + # With no such node the suite produces six failures across four files + # — "no disposable workload node found" and pod-scheduling timeouts — + # none of which names the cause. It stayed red for eight commits. # # Asked of the cluster rather than of Azure: a stopped or # still-starting cluster reports agentPools[].count as 0 even when # the pool is populated, so the control-plane number is not evidence. - # This asks for exactly what the tests need, and waits, because - # `az aks start` returns before nodes finish registering. # # Checked, not repaired: the workflow identity holds a deliberately # minimal role with agentPools/read and no write, and widening it so # this job could scale the pool would also let it reshape the system # pool that every other guard here exists to protect. - for _ in $(seq 1 30); do - if kubectl wait --for=condition=Ready node \ - -l korvid.dev/disposable=true --timeout=10s >/dev/null 2>&1; then - kubectl get nodes -l korvid.dev/disposable=true + SELECTOR=korvid.dev/disposable=true,korvid.dev/pool=workload + + # Prove the node list is reachable first. Otherwise an expired token, + # an RBAC denial, or an unreachable API server spends the whole + # deadline and then blames an empty pool, sending the reader after a + # scale command that cannot help. + if ! nodes=$(kubectl get nodes -o name 2>&1); then + echo "::error title=Cannot query nodes::Listing nodes on $AKS_CLUSTER_NAME failed, so the disposable-node check could not run: $nodes" + exit 1 + fi + + # Wall-clock deadline, not an iteration count: a registered but + # NotReady node makes each attempt cost its own timeout as well as + # the sleep, which would silently double a loop-counted budget. + deadline=$(( SECONDS + 300 )) + while [ "$SECONDS" -lt "$deadline" ]; do + if kubectl wait --for=condition=Ready node -l "$SELECTOR" \ + --timeout=10s >/dev/null 2>&1; then + kubectl get nodes -l "$SELECTOR" exit 0 fi - sleep 10 + sleep 5 done - echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready node labelled korvid.dev/disposable=true after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If the $WORKLOAD_NODEPOOL pool is empty, restore it with: az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1" + echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready node matching $SELECTOR after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If the $WORKLOAD_NODEPOOL pool is empty, restore it with: az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1" kubectl get nodes --show-labels || true exit 1 diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index 5b7db8ee..94322166 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -43,8 +43,13 @@ Node pools: tests fail rather than fall back to a system node. Because that pool is the suite's single point of failure, the workflow waits -for a Ready node carrying `korvid.dev/disposable=true` before installing -anything, and fails with a named cause and the repair command if none appears. +for a Ready node carrying **both** `korvid.dev/disposable=true` and +`korvid.dev/pool=workload` before installing anything, and fails with a named +cause and the repair command if none appears within five minutes. Both labels +are required together: the node-operation tests select on the first and every +test pod's `nodeSelector` uses the second, so a node with only one of them +would pass a looser check and still leave pods Pending. + It asks the cluster rather than Azure: a stopped or still-starting cluster reports `agentPools[].count` as 0 even when the pool is populated, so the control-plane number is not evidence. From db2881c8a6c2cdb71e0a260bec8fe004c85f7dd9 Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 14:57:54 +0900 Subject: [PATCH 5/8] ci: need one Ready node, and print a repair sequence that runs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more ways the guard could report the wrong cause. `kubectl wait` requires *every* node matching the selector to satisfy the condition, but the suite needs only one somewhere to schedule. A second pool node, or one cycling through an image upgrade, would have burned the whole deadline and announced "no schedulable disposable node" with a healthy one sitting right there. Count Ready nodes among the matches instead. The repair command also could not run as printed. The stop-cluster job is `if: always()`, so it stops the cluster even when this guard fails, and Azure rejects everything except start on a stopped cluster — verified: `az aks nodepool scale` on the stopped cluster returns "Managed Cluster is in stopped state, no operations except for start are allowed". An operator pasting the hint got an error instead of a fix, which is the same dead end this step exists to remove. The hint is now start, scale, stop, and says why. Also verified that `az aks start` refuses on a running cluster, so the sequence is documented as the post-cleanup recovery rather than something to run mid-flight. Verified on the real cluster: passes in 3s with the node Ready, honours the deadline on label drift, and the printed sequence executes end to end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 18 +++++++++++------- docs/dev/contract-tests.md | 18 +++++++++++------- 2 files changed, 22 insertions(+), 14 deletions(-) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index a13d667e..d55c2ad1 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -128,19 +128,23 @@ jobs: exit 1 fi - # Wall-clock deadline, not an iteration count: a registered but - # NotReady node makes each attempt cost its own timeout as well as - # the sleep, which would silently double a loop-counted budget. + # `kubectl wait` requires *every* matching node to satisfy the + # condition, but this guard needs only one somewhere to schedule. A + # second pool node, or one cycling through an image upgrade, would + # otherwise burn the whole deadline and report "no schedulable node" + # while a perfectly good one was sitting there. deadline=$(( SECONDS + 300 )) - while [ "$SECONDS" -lt "$deadline" ]; do - if kubectl wait --for=condition=Ready node -l "$SELECTOR" \ - --timeout=10s >/dev/null 2>&1; then + while :; do + if kubectl get nodes -l "$SELECTOR" \ + -o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \ + 2>/dev/null | grep -q '=True$'; then kubectl get nodes -l "$SELECTOR" exit 0 fi + [ "$SECONDS" -lt "$deadline" ] || break sleep 5 done - echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready node matching $SELECTOR after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If the $WORKLOAD_NODEPOOL pool is empty, restore it with: az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1" + echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready node matching $SELECTOR after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If the $WORKLOAD_NODEPOOL pool is empty, restore it by starting the cluster first — this run's cleanup job stops it, and Azure rejects every operation except start on a stopped cluster: az aks start -g $AKS_RESOURCE_GROUP -n $AKS_CLUSTER_NAME && az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1 && az aks stop -g $AKS_RESOURCE_GROUP -n $AKS_CLUSTER_NAME" kubectl get nodes --show-labels || true exit 1 diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index 94322166..bbe9d7f3 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -43,12 +43,12 @@ Node pools: tests fail rather than fall back to a system node. Because that pool is the suite's single point of failure, the workflow waits -for a Ready node carrying **both** `korvid.dev/disposable=true` and -`korvid.dev/pool=workload` before installing anything, and fails with a named -cause and the repair command if none appears within five minutes. Both labels -are required together: the node-operation tests select on the first and every -test pod's `nodeSelector` uses the second, so a node with only one of them -would pass a looser check and still leave pods Pending. +for **at least one** Ready node carrying **both** `korvid.dev/disposable=true` +and `korvid.dev/pool=workload` before installing anything, and fails with a +named cause and the repair sequence if none appears within five minutes. Both +labels are required together: the node-operation tests select on the first and +every test pod's `nodeSelector` uses the second, so a node with only one of +them would pass a looser check and still leave pods Pending. It asks the cluster rather than Azure: a stopped or still-starting cluster reports `agentPools[].count` as 0 even when the pool is populated, so the @@ -57,11 +57,15 @@ control-plane number is not evidence. It checks but does not repair. The workflow identity holds `agentPools/read` and no write, and granting write so the job could scale the pool would also let it reshape the system pool. Cost at rest is handled by stopping the -cluster, so a pool sitting at zero is drift to be repaired by hand: +cluster, so a pool sitting at zero is drift to be repaired by hand. Start the +cluster first — the cleanup job stops it even when the guard fails, and Azure +rejects every operation except `start` on a stopped cluster: ```sh +az aks start -g rg-korvid-contract-test -n aks-korvid-contract-test az aks nodepool scale -g rg-korvid-contract-test \ --cluster-name aks-korvid-contract-test -n workload --node-count 1 +az aks stop -g rg-korvid-contract-test -n aks-korvid-contract-test ``` Without the check, a missing node surfaces as six failures across four test From 6d8a26d44ab42db2c9de01cfd8ed809ee11d7c44 Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 15:32:02 +0900 Subject: [PATCH 6/8] ci: reject a cordoned node instead of calling it schedulable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The step promised "schedulable" but only checked Ready, so a cordoned node passed it. That is not hypothetical: the janitor sweeps cordons precisely because a crashed node-operation test leaves one behind, and the janitor runs *after* this guard. A run following a crashed one would have gone green here and then timed out with every test pod Pending — the unnamed red this step exists to remove, reintroduced by the step itself. Reproduced it: cordoned the workload node, and the previous check reported the node as fine. Require spec.unschedulable to be unset alongside Ready, and say so in the failure message so an operator knows to uncordon rather than reach for the pool. Failure output is now `-o wide --show-labels` as well. Verified on the real cluster: cordoned node rejected, same node accepted after uncordon. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 23 ++++++++++++++--------- docs/dev/contract-tests.md | 15 +++++++++------ 2 files changed, 23 insertions(+), 15 deletions(-) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index d55c2ad1..18d5ff02 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -128,24 +128,29 @@ jobs: exit 1 fi - # `kubectl wait` requires *every* matching node to satisfy the - # condition, but this guard needs only one somewhere to schedule. A - # second pool node, or one cycling through an image upgrade, would - # otherwise burn the whole deadline and report "no schedulable node" - # while a perfectly good one was sitting there. + # `kubectl wait` requires *every* node matching the selector to + # satisfy the condition, but this guard needs only one somewhere to + # schedule. A second pool node, or one cycling through an image + # upgrade, would otherwise burn the whole deadline and report "no + # schedulable node" while a perfectly good one was sitting there. + # + # Ready alone is not enough: a crashed node-ops test leaves the node + # cordoned — that is why the janitor sweeps cordons — and the janitor + # runs after this step, so a cordoned node would pass here and leave + # every test pod Pending. Require unschedulable to be unset too. deadline=$(( SECONDS + 300 )) while :; do if kubectl get nodes -l "$SELECTOR" \ - -o jsonpath='{range .items[*]}{.metadata.name}{"="}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{"\n"}{end}' \ - 2>/dev/null | grep -q '=True$'; then + -o jsonpath='{range .items[*]}{.metadata.name}{"=ready:"}{range .status.conditions[?(@.type=="Ready")]}{.status}{end}{" unschedulable:"}{.spec.unschedulable}{"\n"}{end}' \ + 2>/dev/null | grep -q '=ready:True unschedulable:$'; then kubectl get nodes -l "$SELECTOR" exit 0 fi [ "$SECONDS" -lt "$deadline" ] || break sleep 5 done - echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready node matching $SELECTOR after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If the $WORKLOAD_NODEPOOL pool is empty, restore it by starting the cluster first — this run's cleanup job stops it, and Azure rejects every operation except start on a stopped cluster: az aks start -g $AKS_RESOURCE_GROUP -n $AKS_CLUSTER_NAME && az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1 && az aks stop -g $AKS_RESOURCE_GROUP -n $AKS_CLUSTER_NAME" - kubectl get nodes --show-labels || true + echo "::error title=No schedulable disposable node::$AKS_CLUSTER_NAME has no Ready, uncordoned node matching $SELECTOR after 5 minutes, so the node-operation and resize contracts have nowhere to schedule. If a previous run left the node cordoned, uncordon it. If the $WORKLOAD_NODEPOOL pool is empty, restore it by starting the cluster first — this run's cleanup job stops it, and Azure rejects every operation except start on a stopped cluster: az aks start -g $AKS_RESOURCE_GROUP -n $AKS_CLUSTER_NAME && az aks nodepool scale -g $AKS_RESOURCE_GROUP --cluster-name $AKS_CLUSTER_NAME -n $WORKLOAD_NODEPOOL --node-count 1 && az aks stop -g $AKS_RESOURCE_GROUP -n $AKS_CLUSTER_NAME" + kubectl get nodes -o wide --show-labels || true exit 1 - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index bbe9d7f3..b7b517de 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -43,12 +43,15 @@ Node pools: tests fail rather than fall back to a system node. Because that pool is the suite's single point of failure, the workflow waits -for **at least one** Ready node carrying **both** `korvid.dev/disposable=true` -and `korvid.dev/pool=workload` before installing anything, and fails with a -named cause and the repair sequence if none appears within five minutes. Both -labels are required together: the node-operation tests select on the first and -every test pod's `nodeSelector` uses the second, so a node with only one of -them would pass a looser check and still leave pods Pending. +for **at least one** node that is Ready, uncordoned, and carries **both** +`korvid.dev/disposable=true` and `korvid.dev/pool=workload` before installing +anything, and fails with a named cause and the repair sequence if none appears +within five minutes. Both labels are required together: the node-operation +tests select on the first and every test pod's `nodeSelector` uses the second, +so a node with only one of them would pass a looser check and still leave pods +Pending. `unschedulable` is checked because a crashed node-operation test +leaves the node cordoned — that is what the janitor's cordon sweep is for, and +the janitor runs *after* this guard. It asks the cluster rather than Azure: a stopped or still-starting cluster reports `agentPools[].count` as 0 even when the pool is populated, so the From ec9799cf941d0d27c7d19d4354b80b484e514c98 Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 16:03:00 +0900 Subject: [PATCH 7/8] ci: run the node guard after the janitor, not before it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both reviewers landed on the same flaw in my last commit from opposite directions, and they were right: adding the cordon check in front of the janitor converted a self-healing state into a hard failure. `_sweep_cordons` exists to uncordon disposable nodes left behind by a crashed node-ops test, and it did fix those runs silently. With the guard ahead of it, the same situation now spent five minutes and exited, killing the job before the janitor could run — and every later run stayed blocked until a human intervened. The goal is removing unnamed reds, not reds that used to fix themselves. Moved the guard after the janitor. It cannot move earlier instead: the janitor is a Python entry point that needs `uv sync` first. Cost is that a genuinely broken cluster now wastes the dependency install before failing, which is worth it — and a cluster too broken to reach fails at the janitor with its own error anyway. Verified end to end: cordoned the node, ran the janitor, watched it report `uncordoned 1 node(s)`, and the guard then passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/k8s-contract.yml | 41 +++++++++++++++--------------- docs/dev/contract-tests.md | 5 ++-- 2 files changed, 24 insertions(+), 22 deletions(-) diff --git a/.github/workflows/k8s-contract.yml b/.github/workflows/k8s-contract.yml index 18d5ff02..dbb84e30 100644 --- a/.github/workflows/k8s-contract.yml +++ b/.github/workflows/k8s-contract.yml @@ -93,6 +93,23 @@ jobs: chmod 600 "$KUBECONFIG" echo "KUBECONFIG=$KUBECONFIG" >> "$GITHUB_ENV" + - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 + with: + python-version: "3.13" + enable-cache: true + + - run: uv sync --locked --dev --all-extras + + - name: Install helm + run: | + HELM_VERSION=v3.19.0 + curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" \ + | tar -xz -C "$RUNNER_TEMP" linux-amd64/helm + sudo install "$RUNNER_TEMP/linux-amd64/helm" /usr/local/bin/helm + + - name: Janitor — remove stale fixtures from interrupted runs + run: uv run python -m tests.contract.janitor + - name: Guard — a schedulable disposable node must exist run: | set -euo pipefail @@ -135,9 +152,10 @@ jobs: # schedulable node" while a perfectly good one was sitting there. # # Ready alone is not enough: a crashed node-ops test leaves the node - # cordoned — that is why the janitor sweeps cordons — and the janitor - # runs after this step, so a cordoned node would pass here and leave - # every test pod Pending. Require unschedulable to be unset too. + # cordoned, and a cordoned node would pass a Ready-only check and + # leave every test pod Pending. This runs after the janitor, which + # uncordons disposable nodes, so anything still cordoned here is + # something the janitor could not fix and a human has to look at. deadline=$(( SECONDS + 300 )) while :; do if kubectl get nodes -l "$SELECTOR" \ @@ -153,23 +171,6 @@ jobs: kubectl get nodes -o wide --show-labels || true exit 1 - - uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 - with: - python-version: "3.13" - enable-cache: true - - - run: uv sync --locked --dev --all-extras - - - name: Install helm - run: | - HELM_VERSION=v3.19.0 - curl -fsSL "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" \ - | tar -xz -C "$RUNNER_TEMP" linux-amd64/helm - sudo install "$RUNNER_TEMP/linux-amd64/helm" /usr/local/bin/helm - - - name: Janitor — remove stale fixtures from interrupted runs - run: uv run python -m tests.contract.janitor - - name: Run contract suite timeout-minutes: 30 env: diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index b7b517de..32d5deee 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -50,8 +50,9 @@ within five minutes. Both labels are required together: the node-operation tests select on the first and every test pod's `nodeSelector` uses the second, so a node with only one of them would pass a looser check and still leave pods Pending. `unschedulable` is checked because a crashed node-operation test -leaves the node cordoned — that is what the janitor's cordon sweep is for, and -the janitor runs *after* this guard. +leaves the node cordoned. The guard runs *after* the janitor's cordon sweep, +so that case still heals itself; anything still cordoned at the guard is +something the janitor could not fix. It asks the cluster rather than Azure: a stopped or still-starting cluster reports `agentPools[].count` as 0 even when the pool is populated, so the From a35ca2b78bebb741c9f65fd0e8f91305b73a740b Mon Sep 17 00:00:00 2001 From: hellices Date: Fri, 7 Aug 2026 16:52:30 +0900 Subject: [PATCH 8/8] docs: stop claiming the node guard runs before installing anything Moving the guard behind the janitor made this sentence false, and it contradicted the next sentence in its own paragraph, which correctly says the guard runs after the janitor sweep. The real order is setup-uv, uv sync, helm, janitor, guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/dev/contract-tests.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/contract-tests.md b/docs/dev/contract-tests.md index 32d5deee..80dc7ee1 100644 --- a/docs/dev/contract-tests.md +++ b/docs/dev/contract-tests.md @@ -44,8 +44,8 @@ Node pools: Because that pool is the suite's single point of failure, the workflow waits for **at least one** node that is Ready, uncordoned, and carries **both** -`korvid.dev/disposable=true` and `korvid.dev/pool=workload` before installing -anything, and fails with a named cause and the repair sequence if none appears +`korvid.dev/disposable=true` and `korvid.dev/pool=workload` before running the +suite, and fails with a named cause and the repair sequence if none appears within five minutes. Both labels are required together: the node-operation tests select on the first and every test pod's `nodeSelector` uses the second, so a node with only one of them would pass a looser check and still leave pods