fix(hermes-sre): lower hermes-ws session pod CPU request 500m -> 100m - #909
Open
igou-hermes[bot] wants to merge 1 commit into
Open
fix(hermes-sre): lower hermes-ws session pod CPU request 500m -> 100m#909igou-hermes[bot] wants to merge 1 commit into
igou-hermes[bot] wants to merge 1 commit into
Conversation
Four concurrent session pods at 500m consume 2 full cores of requests; combined with the rest of the cluster this repeatedly crosses the KubeCPUOvercommit single-node-failure margin (38.922 requested vs 38.5 allowed when the largest node is lost). Real usage is ~0.1 core for the namespace, so a 100m request keeps the guarantee while leaving limits (4 CPU) untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed by hermes-sre from KubeCPUOvercommit (kube-system), firing 2026-09-09T02:23Z — incident issue igou-io/igou-inventory#718.
Diagnosis: The firing branch of KubeCPUOvercommit is the single-node-failure margin: OCP-side pod CPU requests must stay below total allocatable minus the largest node (
58 - 19.5 = 38.5). At 2026-09-09T02:23Z requests were38.922→ overcommit+0.422, exactly the alert value. Verified via Thanos with the OCP-only series split (the federated rk8s view sums to 98 allocatable and shows no overcommit; the alert evaluates onopenshift-monitoring/k8swhich sees only the 4 OCP nodes).Attribution: 48h per-namespace request drift on the OCP source shows exactly one mover:
hermes-sre+2.000— the fourhermes-ws-*session pods (MAX_SESSIONS=4), each requesting500m. Actual namespace usage is ~0.109cores over 1h — the requests are pure paper guarantee.Change:
applications/hermes-sre/hermesinstance.yaml— session workspace containerresources.requests.cpu:500m→100m(limits untouched at 4 CPU / 8Gi). This is the remediation already recommended in the #718 thread on 2026-09-08 when the same crossing happened; it recurred today. Worst-case burst (4 sessions × 4 CPU limit) is unchanged; only the reservation drops. After the change, 4 idle sessions reserve 0.4 instead of 2.0, keeping the single-node-failure margin positive during normal operation.Validation:
kustomize build applications/hermes-srerenders 46 docs and the CR carries the new value;yamllint -c .yamllintclean.Note: cursor-agent hit its provider usage limit, so this one-line value edit was applied directly by hermes-sre after its failure; diff reviewed and validated as above.
Not merging — human review requested. ArgoCD applies after merge.