Repository navigation
fix(k8s): retry ResourceQuota admission conflicts instead of treating them as existing objects - #2489
Merged
Merged
Conversation
… them as existing objects The API server answers HTTP 409 both for AlreadyExists and for a write it rejected on an optimistic-concurrency Conflict, notably ResourceQuota admission when several objects are created in one namespace at once. The runtime plugin image-pull secret and the sensitive-env secret paths treated every 409 as 'already exists', so a quota Conflict (which created nothing) was followed by a read/patch of an absent Secret, a 404, and a terminally failed plugin planning invocation -- failing the whole range when several ranges plan concurrently. Classify 409s by the Status reason and retry admission Conflicts with bounded full-jitter exponential backoff (6 attempts) in the image-pull secret, the sensitive-env secret, and the Job create path (which already retried, but only 3 times within ~0.3s). An exhausted Conflict still fails loudly.
Drops the unreachable trailing raise (and its trailing pragma comment, Sonar python:S139): the last attempt now propagates any error directly.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A CTF participant range failed during concurrent provisioning with
Runtime plugin planning failed. GKE audit logs for the plugin namespace at the failure time show:The API server returns HTTP 409 for both
AlreadyExistsand a write rejected on an optimistic-concurrencyConflict, here ResourceQuota admission racing when several plugin Jobs and their Secrets are created in one namespace at once. The two are separated only by the Statusreason._pull_secret(runtime plugin image-pull secret) treated every 409 as "already exists" and read the Secret back. After aConflictnothing was created, so the read 404'd, the invocation reconciler marked planning terminally failed, and the whole range failed._ensure_sensitive_secrethad the same conflation (patching an absent Secret).It is a race, so only some ranges failed.
Fix
shared/cloud/kubernetes/_helpers.py:is_admission_conflict(409 with Status reasonConflict) andcreate_with_admission_retry(6 attempts, full-jitter exponential backoff capped 0.1s→1.6s, so racing writers desynchronize).Conflictfails loudly and is never treated as "already exists". A realAlreadyExistskeeps its existing verify/patch recovery. A 409 with no parseable Status body keeps the historical handling.Tests
tests/engine/services/test_runtime_plugin_pull_secret.py(new;_pull_secrethad no direct coverage): classifier distinguishesConflict/AlreadyExists/missing body; conflicts are retried and never read back; exhausted conflicts raise; an existing matching Secret is reused. Verified the retry test fails without the fix.Conflictretries the create with no recovery patch; exhausted conflict fails.adr_guard --level ciclean.