cloud: classify conditional-mutation failures by provider predicate code, retry S3 ConditionalRequestConflict, type 408 as timeout

[smiggleworth]

Gap cloud-28 · behavior-difference · confidence: verified · tracked in #242
Baseline: Midge 7d39f86 (0.1.1) vs Pants 2507c66. Midge paths are relative to the midge repo, Pants paths to this repo. Midge behavior is used as the proxy for the not-yet-extracted lsm-spec behavioral contract.

Prior: #94 (related, closed: renewal indeterminate path)

Midge: s3.rs:1423-1447 treats only 412 with <Code>PreconditionFailed</Code> as PreconditionFailed. azure.rs:1166-1190 accepts only
412 with ConditionNotMet or TargetConditionNotMet, and gcs.rs:1527-1563 accepts only 412 with conditionNotMet or PreconditionFailed. Everything
else maps through CloudError::from_http_status (cloud/mod.rs:204-212). Tests:
should_only_classify_s3_predicate_failure_as_precondition_conflict,
should_only_classify_azure_target_condition_as_precondition_conflict (412 LeaseIdMissing and 409 SnapshotsPresent are NOT conflicts),
should_only_classify_gcs_condition_reason_as_precondition_conflict (412 orgPolicyConstraintFailed is NOT a conflict).
S3 409 ConditionalRequestConflict/OperationAborted on an If-Match/If-None-Match PUT gets bounded retries
(executor.rs:86-92, 451-458, 511-521; should_retry_s3_conditional_request_conflict_when_creating_object s3.rs:2097).
HTTP 408 is a typed timeout (should_classify_http_408_as_timeout_without_relying_on_provider_detail_text).

Pants: PutAsync maps any 409 or 412 to false (condition not met) in S3ObjectStore.cs:221, AzureBlobObjectStore.cs:151,
and GcsObjectStore.cs:161. There is no body/code inspection, unlike DeleteAsync, which does check codes (S3 HasS3ErrorCodeAsync, Azure
HasAzurePredicateFailureAsync :545-560, GCS :722). Lease renewal turns false into PantsFencedException
(Leases/CloudLeaseCoordinator.cs:197-204), so a transient S3 409 ConditionalRequestConflict or an Azure 412 LeaseIdMissing
makes the writer fence itself. On reads, an exhausted 408 surfaces as generic PantsIOException ("HTTP 408"). Preflight then classifies it
as Provider, not Timeout (CloudConfigurationPreflight.cs:333-386, which parses message text).

Why it matters: when unrelated provider errors are reported as a lost CAS, the engine takes the fencing path (an availability loss) or
reports misleading conflicts. Immutable uploads are protected by readback (ProviderCloudPersistence.cs:113-128, 672-689), so
this is not a data-loss issue.

Acceptance criteria:

• PutAsync returns false only for provider predicate failures: S3 412 PreconditionFailed (plus the documented S3 409 case after retries are exhausted), Azure 412 ConditionNotMet/TargetConditionNotMet, and GCS 412 conditionNotMet. Azure If-None-Match:* returning 409 BlobAlreadyExists also needs a decision.
• Other 409/412 responses (LeaseIdMissing, SnapshotsPresent, orgPolicyConstraintFailed, OperationAborted after exhausted retries) raise a typed non-conflict error.
• S3 409 ConditionalRequestConflict on a conditional PUT is retried with a bound, and the test asserts the request count and the final outcome.
• An exhausted 408 surfaces as PantsTimeoutException for reads, and preflight reports it as Timeout.
• Provider error-code classification does not depend on the text of Pants's own exception messages.