Skip to content

fix(relay): bound the git conformance probe so a stalled backend cannot hang startup#2724

Open
webdevtodayjason wants to merge 1 commit into
block:mainfrom
webdevtodayjason:fix/git-probe-stall-hang
Open

fix(relay): bound the git conformance probe so a stalled backend cannot hang startup#2724
webdevtodayjason wants to merge 1 commit into
block:mainfrom
webdevtodayjason:fix/git-probe-stall-hang

Conversation

@webdevtodayjason

Copy link
Copy Markdown

Closes #2723.

Problem

The A3 conformance probe gates relay startup, but a backend connection that is accepted and never answered hangs it forever: the shared rust-s3 bucket client has no request timeout, the race phases join_all every racer (one stalled socket parks the phase), and run_conformance_probe has no overall deadline. Since main.rs awaits the probe before binding any listener, the deployment sits unhealthy indefinitely with nothing in the logs but silence.

Hit in production today on a fresh deploy/compose install (image ghcr.io/block/buzz:main, the bundle's pinned MinIO): two transport drops in round 0, then no output for 10+ minutes, relay never came up. Full evidence in #2723.

Fix

Two bounds, both env-overridable (naming matches the existing BUZZ_GIT_PROBE_WRITERS/_ROUNDS):

  • ProbeConfig.op_timeout (default 30s, BUZZ_GIT_PROBE_OP_TIMEOUT_SECS) — every probe operation is wrapped in tokio::time::timeout. A racer that elapses is classified as a transport drop under the probe's existing drop-and-floor rule: its outcome is unknown, exactly like S3Error::{Reqwest, Http, Io}, so it's dropped from the observer set and A3 semantics are unchanged (classified >= 2 and winners == 1 among observers still enforced, transport_drops still surfaced on the admission log line). Sequential/seed/etag-phase operations fail closed instead, with a diagnosable reason (no response within 30s (stalled backend connection)). Cleanup deletes are bounded too.
  • ProbeConfig.deadline (default 300s, BUZZ_GIT_PROBE_DEADLINE_SECS) — the whole run is wrapped in a timeout that fails closed with phase deadline and actionable guidance (investigate the store, or BUZZ_GIT_CONFORMANCE_PROBE=false to skip the gate). No future probe pathology can brick startup.

Considered and rejected: putting a request timeout on the shared Bucket itself. That client is also the pack/manifest data path, where large transfers make any fixed global timeout a foot-gun — scoping the bounds to the probe keeps the blast radius zero.

Relation to existing work: complementary to #2511 (clearer failure text + Helm disable toggle) and #2470 (GCS fails the probe visibly) — both assume the probe produces an outcome; this PR fixes the case where it produces nothing at all.

Tests

  • Two new no-infra regression tests drive the probe against a local tarpit listener (accepts connections, never responds — the exact production shape):
    • probe_fails_fast_when_backend_stalls — op-timeout path; asserts a diagnosable failure well under 10s.
    • probe_deadline_bounds_total_runtime — op_timeout set absurdly high (the old effectively-unbounded behavior), asserts the whole-probe deadline still fires.
    • Both hang forever without this patch.
  • Full api::git::store suite: 12 passed. cargo clippy clean. just ci green locally.
  • The gated live-MinIO probe tests (BUZZ_GIT_S3_PROBE=1) are unchanged apart from struct-literal completion.

Checklist

  • just ci passes (fmt + clippy + unit tests + mobile)
  • Bug fix includes regression tests
  • New public fields have doc comments
  • No new unwrap() in production code paths
  • No new unsafe blocks

@webdevtodayjason
webdevtodayjason requested a review from a team as a code owner July 24, 2026 15:44
…ang startup

The A3 conformance probe gates relay startup, but the shared rust-s3
bucket client has no request timeout and the race phases join_all every
racer — so one backend connection that is accepted and never answered
pends forever, the listeners never bind, and the deployment sits
unhealthy indefinitely (observed against MinIO from the deploy/compose
bundle: two transport drops in round 0, then silence).

ProbeConfig gains op_timeout (default 30s) and deadline (default 300s),
overridable via BUZZ_GIT_PROBE_OP_TIMEOUT_SECS / _DEADLINE_SECS. Racer
timeouts classify as transport drops under the existing drop-and-floor
rule (outcome unknown, dropped from the observer set — A3 semantics
unchanged); sequential-phase timeouts fail closed with a diagnosable
reason; the whole run is bounded by the deadline. Regression-tested with
a tarpit listener that reproduces the production hang shape.

Signed-off-by: webdevtodayjason <jason@webdevtoday.com>
@webdevtodayjason
webdevtodayjason force-pushed the fix/git-probe-stall-hang branch from 9006519 to eb21c72 Compare July 24, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

buzz-relay: git conformance probe hangs startup forever when a backend connection stalls (no request timeout, no deadline)

1 participant