Skip to content

feat(failure): distinguish quota-exceeded from the rest of FailureTerminal - #117

Merged
scttfrdmn merged 1 commit into
mainfrom
feat/116-quota-exceeded-signal
Aug 10, 2026
Merged

feat(failure): distinguish quota-exceeded from the rest of FailureTerminal#117
scttfrdmn merged 1 commit into
mainfrom
feat/116-quota-exceeded-signal

Conversation

@scttfrdmn

Copy link
Copy Markdown
Contributor

Summary

  • New pkg/failure.IsQuotaExceeded(err) bool — a finer-grained question than ClassifyFailure's FailureTerminal, which also covers bad AMI/IAM/malformed-request errors this correctly reports false for.
  • quotaErrorCodes (InstanceLimitExceeded, VcpuLimitExceeded, MaxSpotInstanceCountExceeded) split out of terminalErrorCodes into its own map, checked first in ClassifyFailure — so each code is listed exactly once, not duplicated across two maps.
  • pkg/watcher.IsQuotaExceeded aliases it, matching the existing alias pattern for every other pkg/failure export.
  • pkg/snipe.Status gains QuotaExceeded bool, set on every Options.Progress report that carries a LastErr — so a caller doesn't need to separately import pkg/failure and re-classify the error itself.

No default behavior changes. A quota error still classifies as FailureTerminal and Snipe still returns immediately on it, exactly as before — TestClassifyFailure, TestSnipe_TerminalStopsImmediately, etc. all pass unmodified, confirming the classification itself didn't move. This is purely an additive signal for a caller building its own backoff/reduce-concurrency logic on top.

Real-world motivation: several concurrent pkg/snipe.Snipe calls against the same account (one per fleet shard) each get MaxSpotInstanceCountExceeded and give up immediately when the account's own quota is momentarily saturated by the caller's OTHER in-flight requests — indistinguishable, before this change, from a hard account wall needing a support ticket.

Fixes #116.

Test plan

  • TestIsQuotaExceeded (11 cases) — all 3 quota codes report true; every other terminal code, every capacity code, an unknown code, and a plain non-AWS error all report false
  • TestIsQuotaExceeded_StillClassifiesTerminal — explicit regression guard that ClassifyFailure is unchanged for the 3 quota codes
  • TestSnipe_ProgressReportsQuotaExceeded / TestSnipe_ProgressDoesNotReportQuotaForOrdinaryTerminalStatus.QuotaExceeded correctly true/false via the real retry loop
  • watcher.TestIsQuotaExceeded — alias reaches the real classifier
  • Existing TestClassifyFailure (both pkg/failure and pkg/watcher copies) pass unmodified — proves quota codes still classify as FailureTerminal
  • go build ./..., go vet ./..., gofmt -l clean (root module + lambda/capacity-poller); full go test ./... green
  • golangci-lint run — 2 pre-existing issues in untouched files (pkg/watcher/notifier.go, pkg/watcher/store.go), zero in touched files
  • make check-docs — no drift

…minal

Every terminal error (bad AMI, IAM denial, malformed request, and an
exhausted account quota) previously looked identical to a caller: an
immediate give-up with no further signal. A caller running several
concurrent pkg/snipe.Snipe calls against the same account has no way
to tell "this quota ceiling might free up soon (it may be MY OWN other
in-flight requests)" apart from "nothing about this will ever work."

New pkg/failure.IsQuotaExceeded(err) checks a quota-specific error-code
set (split out of terminalErrorCodes into its own quotaErrorCodes map,
checked first in ClassifyFailure so a code is listed exactly once).
pkg/watcher.IsQuotaExceeded aliases it. pkg/snipe.Status gains
QuotaExceeded bool, set on every Progress report that carries a
LastErr.

Does NOT change any default retry behavior: a quota error still
classifies as FailureTerminal and Snipe still returns immediately on
it, unchanged. Purely additive -- existing TestClassifyFailure/
TestSnipe_TerminalStopsImmediately etc. pass unmodified, confirming
the classification itself didn't move.

Fixes #116.
@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@scttfrdmn
scttfrdmn merged commit efb2c14 into main Aug 10, 2026
6 checks passed
@scttfrdmn
scttfrdmn deleted the feat/116-quota-exceeded-signal branch August 10, 2026 22:02
scttfrdmn added a commit that referenced this pull request Aug 10, 2026
Bump spore-host/truffle v0.48.1 -> v0.49.0 in both the root module and
lambda/capacity-poller (separate go.mod, kept in sync per #43). Promotes
CHANGELOG Unreleased to 0.53.0 ahead of tagging, covering #117's additive
IsQuotaExceeded signal.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

snipe treats MaxSpotInstanceCountExceeded as a flat terminal failure — no signal for "reduce concurrency and retry" callers

1 participant