Skip to content

fix(ace-step): check in resting replicas 0 to stop ArgoCDSyncAlert refires - #911

Open
igou-hermes[bot] wants to merge 1 commit into
mainfrom
sre/ace-step-rest-replicas-0
Open

fix(ace-step): check in resting replicas 0 to stop ArgoCDSyncAlert refires#911
igou-hermes[bot] wants to merge 1 commit into
mainfrom
sre/ace-step-rest-replicas-0

Conversation

@igou-hermes

@igou-hermes igou-hermes Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Proposed by hermes-sre from ArgoCDSyncAlert (warning, firing since 2026-09-09T01:35Z on root-applications, openshift-gitops). No incident issue existed for this firing (prior ArgoCDSyncAlert issues #763/#799#804 covered other apps and are closed).

Diagnosis

  • root-applications sync has been stuck since 2026-09-09T00:22:52Z: waiting for healthy state of argoproj.io/Application/ace-step → app Progressing/OutOfSync → alert.
  • ace-step pod ace-step-54585d6f8d-55m2g (ns ace-step): Pending 3d3h, FailedScheduling ×9010/4 nodes are available: 4 node(s) didn't match Pod's node affinity/selector.
  • The pod pins node-role.kubernetes.io/burst + nvidia.com/gpu: 2 (PR feat(ace-step): schedule on casval with nvidia.com/gpu: 2 #879). The only burst node, casval (2× 4060 Ti), is provided by MachineSet casval-worker at replicas: 0 at rest (lease model), and the cluster currently advertises only the 2 GiB Quadro P620 on p330.
  • PR feat(ace-step): schedule on casval with nvidia.com/gpu: 2 #879 shipped the deployment checked in at replicas: 1, so the moment the casval lease ended the pod recreated and went Pending — holding the root app Progressing/OutOfSync and re-firing ArgoCDSyncAlert indefinitely.
  • The repo's own at-rest convention (comfyui, llmkube qwen3-35b) and the ace-step README scale-to-0 workflow both expect replicas: 0 at rest; ArgoCD ignores /spec/replicas so live scale-ups are never reverted.

Change

applications/ace-step/kustomization.yaml: replicas: 1replicas: 0 with an explanatory comment. No other field touched.

Validation

  • kustomize build --enable-helm applications/ace-step → Deployment ace-step renders replicas: 0
  • yamllint -c .yamllint clean

Not merging — human review requested. When merged, the root app should return Synced/Healthy and ArgoCDSyncAlert should resolve on the next reconcile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants