feat(launch): --max-concurrent-auto derives sweep concurrency from real quota - #493
Merged
Merged
Conversation
…al quota The sweep orchestrator's wave mechanism (pkg/sweep + lambda/sweep-orchestrator) already polls active-instance count and launches min(available, remaining) -- a real batch/wait-for-room primitive. But its ceiling (state.MaxConcurrent) was always a number the caller had to already know, via --max-concurrent or the arbitrary default-10 fallback when a sweep auto-enables --detach. --max-concurrent-auto queries truffle's quota client for headroom (quota - current usage) per instance family, in the region the sweep is about to launch in, and converts vCPU headroom to an instance count via the real per-type vCPU count (truffle#134's new Capabilities.VCPUs, not a guessed size suffix). A heterogeneous sweep's derived ceiling is the MINIMUM across every distinct (instance type, spot/on-demand) combination present in the sweep, so a scarce family can't be silently outvoted by a roomier one. Real-world motivation: a 10-shard fleet launch with no concurrency guardrail hit an account's real ceiling (a G/VT Spot quota of 64 vCPUs, saturated by 8 running g7e.2xlarge instances) with zero prior warning; the actual launches then failed with MaxSpotInstanceCountExceeded. Not yet wired into `spawn resume`'s equivalent override -- tracked as a follow-up, since resume already has a recorded region and is a smaller retrofit. BLOCKED: go.mod carries a TEMPORARY local replace directive for truffle, since this code depends on truffle#133 (QuotaInfo.SpotUsage) and truffle#134 (Capabilities.VCPUs), both unreleased. Must be resolved to a real tag before merge. Fixes #492.
truffle v0.49.0 (carrying #133 QuotaInfo.SpotUsage and #134 Capabilities.VCPUs) is now released, so this PR no longer needs the local-path replace directive. go.mod now pins the real tag; verified the full build/test suite (including TestCatalogValid and the new sweep-quota tests) against it, not the local checkout.
scttfrdmn
marked this pull request as ready for review
August 10, 2026 22:38
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--max-concurrent-autoflag: derives the parameter-sweep concurrency ceiling from the account's real AWS quota headroom instead of a user-typed number.pkg/sweep+lambda/sweep-orchestrator'srunPollingLoop, which polls active-instance count and launchesmin(available, remaining)) — this closes the actual gap, which was that the ceiling was never derived from anything real.Capabilities.VCPUs(truffle v0.49.0).TestResolveAutoMaxConcurrent_TightestFamilyWins, which found real behavior different from what I initially assumed: an unseeded/zero-quota family correctly drags the WHOLE sweep's derived ceiling to 0 (and thus an error) rather than being silently outvoted by a roomier family. That's the safety-correct behavior — a sweep shouldn't launch instances of a type whose quota headroom was never actually verified.--max-concurrent(validated, matching the existing--detach/--no-detachpattern).--detachand that branch needs a concretemaxConcurrent > 0to take the Lambda-orchestrated path.Fixes #492.
Not in scope (follow-up)
spawn resume's--max-concurrentoverride has the same class of gap. Didn't extend this PR to cover it —resumealready has a recorded region from the original launch (an easier retrofit than the initial-launch path this PR touches), so it's a smaller, separate change.Test plan
TestSweepQuotaCombos_*(4 tests) — pure combo-extraction logic: dedup, base-config fallback, base-Spot inheritance, missing-instance-type/empty-params errorsTestResolveAutoMaxConcurrent_*(3 tests, Substrate-backed): derives the correct ceiling from a real seeded quota (32 vCPU Standard / 4 vCPU perc5.xlarge= 8), the tightest-family-wins safety behavior described above, and the region-must-be-resolved-first invariantgo build ./.../go test ./...(repo-wide) verified against the REAL releasedtruffle v0.49.0(not the local checkout the draft version used) — zero FAILs, andTestCatalogValid(previously blocked by a local-overlay artifact unrelated to this PR, see spawn#489/libs#38) now passes cleangolangci-lint run ./cmd/...— 19 pre-existing issues, all confirmed present in unmodified code, zero in the new filesmake check-docs— caught the new flag correctly,docs-gen/launch.mdregenerated and committed