Background
generate_dstack_attestation reaches dstack through two get_with_backoff calls that pass None as max_retries:
None selects the without_max_times() branch, so both retry indefinitely with a 60 s maximum backoff and no per-call timeout. Neither is an error path the caller can observe: against an unreachable dstack socket they never return.
#4281 bounds the whole attestation round by the resubmission deadline, so this no longer wedges the submission loop. It does still mean every round burns its full budget before giving up, and the node cannot distinguish "dstack is slow" from "dstack is gone". The collateral fetch alongside them is already bounded, by PCCS_REQUEST_TIMEOUT per request with a single retry.
User Story
As a node operator, I want a broken dstack socket to surface quickly rather than consuming the whole attestation window on every cycle.
Acceptance Criteria
- The dstack
info and get_quote calls are bounded, either by a retry limit or a per-call timeout
- Exhausting that bound returns an
AttestationError rather than retrying forever
- A test covers the bounded path
Resources & Additional Notes
Raised by @haiyuechen-nearone in review of #4281: "I suggest that we add a timeout option for the dstack rpc call as well, but that is not blocking as it is adjacent to the issue fixed."
Background
generate_dstack_attestationreaches dstack through twoget_with_backoffcalls that passNoneasmax_retries:client.info()client.get_quote(..)Noneselects thewithout_max_times()branch, so both retry indefinitely with a 60 s maximum backoff and no per-call timeout. Neither is an error path the caller can observe: against an unreachable dstack socket they never return.#4281 bounds the whole attestation round by the resubmission deadline, so this no longer wedges the submission loop. It does still mean every round burns its full budget before giving up, and the node cannot distinguish "dstack is slow" from "dstack is gone". The collateral fetch alongside them is already bounded, by
PCCS_REQUEST_TIMEOUTper request with a single retry.User Story
As a node operator, I want a broken dstack socket to surface quickly rather than consuming the whole attestation window on every cycle.
Acceptance Criteria
infoandget_quotecalls are bounded, either by a retry limit or a per-call timeoutAttestationErrorrather than retrying foreverResources & Additional Notes
Raised by @haiyuechen-nearone in review of #4281: "I suggest that we add a timeout option for the dstack rpc call as well, but that is not blocking as it is adjacent to the issue fixed."