fix(relay): bound the git conformance probe so a stalled backend cannot hang startup#2724
Open
webdevtodayjason wants to merge 1 commit into
Open
fix(relay): bound the git conformance probe so a stalled backend cannot hang startup#2724webdevtodayjason wants to merge 1 commit into
webdevtodayjason wants to merge 1 commit into
Conversation
…ang startup The A3 conformance probe gates relay startup, but the shared rust-s3 bucket client has no request timeout and the race phases join_all every racer — so one backend connection that is accepted and never answered pends forever, the listeners never bind, and the deployment sits unhealthy indefinitely (observed against MinIO from the deploy/compose bundle: two transport drops in round 0, then silence). ProbeConfig gains op_timeout (default 30s) and deadline (default 300s), overridable via BUZZ_GIT_PROBE_OP_TIMEOUT_SECS / _DEADLINE_SECS. Racer timeouts classify as transport drops under the existing drop-and-floor rule (outcome unknown, dropped from the observer set — A3 semantics unchanged); sequential-phase timeouts fail closed with a diagnosable reason; the whole run is bounded by the deadline. Regression-tested with a tarpit listener that reproduces the production hang shape. Signed-off-by: webdevtodayjason <jason@webdevtoday.com>
webdevtodayjason
force-pushed
the
fix/git-probe-stall-hang
branch
from
July 24, 2026 19:28
9006519 to
eb21c72
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2723.
Problem
The A3 conformance probe gates relay startup, but a backend connection that is accepted and never answered hangs it forever: the shared
rust-s3bucket client has no request timeout, the race phasesjoin_allevery racer (one stalled socket parks the phase), andrun_conformance_probehas no overall deadline. Sincemain.rsawaits the probe before binding any listener, the deployment sitsunhealthyindefinitely with nothing in the logs but silence.Hit in production today on a fresh
deploy/composeinstall (imageghcr.io/block/buzz:main, the bundle's pinned MinIO): two transport drops in round 0, then no output for 10+ minutes, relay never came up. Full evidence in #2723.Fix
Two bounds, both env-overridable (naming matches the existing
BUZZ_GIT_PROBE_WRITERS/_ROUNDS):ProbeConfig.op_timeout(default 30s,BUZZ_GIT_PROBE_OP_TIMEOUT_SECS) — every probe operation is wrapped intokio::time::timeout. A racer that elapses is classified as a transport drop under the probe's existing drop-and-floor rule: its outcome is unknown, exactly likeS3Error::{Reqwest, Http, Io}, so it's dropped from the observer set and A3 semantics are unchanged (classified >= 2andwinners == 1among observers still enforced,transport_dropsstill surfaced on the admission log line). Sequential/seed/etag-phase operations fail closed instead, with a diagnosable reason (no response within 30s (stalled backend connection)). Cleanup deletes are bounded too.ProbeConfig.deadline(default 300s,BUZZ_GIT_PROBE_DEADLINE_SECS) — the whole run is wrapped in a timeout that fails closed with phasedeadlineand actionable guidance (investigate the store, orBUZZ_GIT_CONFORMANCE_PROBE=falseto skip the gate). No future probe pathology can brick startup.Considered and rejected: putting a request timeout on the shared
Bucketitself. That client is also the pack/manifest data path, where large transfers make any fixed global timeout a foot-gun — scoping the bounds to the probe keeps the blast radius zero.Relation to existing work: complementary to #2511 (clearer failure text + Helm disable toggle) and #2470 (GCS fails the probe visibly) — both assume the probe produces an outcome; this PR fixes the case where it produces nothing at all.
Tests
probe_fails_fast_when_backend_stalls— op-timeout path; asserts a diagnosable failure well under 10s.probe_deadline_bounds_total_runtime— op_timeout set absurdly high (the old effectively-unbounded behavior), asserts the whole-probe deadline still fires.api::git::storesuite: 12 passed.cargo clippyclean.just cigreen locally.BUZZ_GIT_S3_PROBE=1) are unchanged apart from struct-literal completion.Checklist
just cipasses (fmt + clippy + unit tests + mobile)unwrap()in production code pathsunsafeblocks