perf(import): warmed first articles, hedged fast-fail STATs, dead-post fast path - #943
Merged
Merged
Conversation
http.ServeContent swallows the reader's error and stops writing, so a usenet fetch that gives up mid-stream reached the client as a truncated body with nothing in the server log to explain it. A bench playback died this way and the INFO log was empty. Wrap the file so the handler sees how the body ended and record the path, range, bytes served and error.
…store Import already fetches the first article of every file (WarmFirstSegments) but kept only a 16 KiB head for the parser, so the cold open that follows an import fetched the same article from the provider again: ~55 ms cold TTFB against ~5 ms warm in the bench, and a flat cost on every click→byte. The parser now hands the whole decoded body to the streaming segment store — the in-memory tier that is on by default, plus the disk cache when enabled — so the first read of a freshly imported file is a cache hit. Cold TTFB on the bench drops to 3–10 ms.
… a segment store is wired Clean-named videos skip the import-time first-segment fetch to save bandwidth, which also meant the one file a player opens first never reached the segment store and paid a provider round trip on its cold open (~85 ms vs 3-10 ms for everything else). Warm just the largest such file, only when a store can keep the article; the skip is unchanged for the rest and whenever caching is off.
A client abort cancels the request context and the reader surfaces that cancellation as its read error; a bench pass logged 36 of those as stream failures. Only a body cut short while the client was still listening is a server-side failure.
…iority lane A bench pass showed 9 of 64 probe STATs queued behind other traffic while the other 55 answered in ~150 ms — under the 90% threshold, so no hedge fired and the import rode the 2 s attempt ceiling again. Arm the hedge at 75% and send the re-issued STATs down the priority lane: on the normal lane they would only join the queue that made the originals straggle. A uniformly slow or dead release never reaches the threshold, so it is not hedged.
…f a reported fraction A debug import showed the probe's real straggler shape: 42 of 64 STATs answered in ~150 ms and 22 sat to the 2 s ceiling, then 19 of those answered instantly on retry and the last 3 were hedged and answered in 50-90 ms on the priority lane. The stragglers are STATs pipelined on slow or cold connections, not slow articles, and their number varies from one to a third of the probe — a 75% reported threshold missed the case that mattered. Once at least 8 answers are in, every answer restarts a lull timer sized from the observed median; when the sweep goes quiet with ids outstanding, all of them are re-issued on the priority lane. Both hedge decisions and answers are logged at debug level so the next unexplained 2 s import can be read from the log.
…m the last 7z volume's tail
The RAR and 7z header passes fetch through an import-scoped cache that
started empty, so the first volume's head — fetched moments earlier by the
warm-up — was fetched again ("pass=7z-header hits 1 misses 2"), and the 7z end
header at the tail of the last volume was a second serial round trip. Each
cost ~250 ms on the import's critical path for every 7z release.
The import cache now reads through to the streaming segment store the warm-up
publishes into, and the warm-up also fetches the last two articles of the
highest-numbered .7z.NNN volume when a store is wired. Both are no-ops when
caching is off. The archive processors take the store through an optional
setter so their interfaces and test doubles are unchanged.
… path After every first segment is in, the parse still made two provider round trips before anything could be written: the representative middle segment's yEnc header (release-wide part size) and the PAR2 index, read through a reader with no cache so its first segment — fetched by the warm-up seconds earlier — was fetched again. ~400 ms of a season pack's 0.9 s import. The warm-up now fetches the representative header alongside the probe, and the PAR2 index reader is given the segment store the warm-up publishes into. Fetches that exist only to fill the store (the largest video's first article, the 7z tail) no longer hold up the parse: they run detached from the warm-up wait, bounded by their own timeout.
…ATs of a full sample A healthy import sat for 4.5 s because one article of the 64 sampled stayed unanswered through two 2 s attempts — after the priority-lane hedge had been tried too, so the article is slow at the provider itself, not queued. The probe answers "is this post damaged?" from a sample; the thousands of articles it never sampled are handled at stream time, and so is this one. With every other sampled article healthy, the probe now passes after the first attempt when at most two of a sample of 32 or more remain unverified, logging what it left unchecked. Small samples and the per-file sweep — which maps exactly which files are broken — keep waiting.
… its per-file sweep Three dead releases in a row cost the imports after them: each was STAT-ed 64 times over three attempts and then swept per file, hundreds of STATs the provider answers as slow 430 spool lookups that stay pipelined on the connections — the next healthy import's probe queued behind them and came back inconclusive or took 14 s. Eight sampled articles all missing is already the verdict. The release probe now checks a first wave of 8 (the edges plus random middle articles) to completion; when those misses condemn the release (releaseLooksDead) every file is marked broken without the sweep, otherwise the rest of the sample is checked as before. A healthy post pays one extra STAT round trip.
…release; open with a small chunk A damaged post (first article of every volume gone) reaches the per-file sweep, which dispatched 64 STATs and waited the attempt out — 18 s on a provider answering 430s slowly — although the first dozen misses had already condemned it, and every 430 left in flight was a slow spool lookup pipelined on a connection for the next import's probe to queue behind. A sweep attempt now cancels itself the moment the definitive answers meet releaseLooksDead, returning what it has so the existing dead-release path condemns the rest, and the sweep's first chunk is capped at 16 STATs so a dead post leaves few of them in flight.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #941 (which merged the first two fast-fail changes). These are the remaining commits from the same benchmarking session against AIOStreams on the nzb-streaming-benchmarks harness, cherry-picked onto
main. Each is TDD'd.Fast-fail probe (
internal/importer/validation)Priority: true; decisions and answers are logged at debug.Cold open / import critical path
cache_path): cold TTFB 55 → 3–10 ms. Also warm the largest clean-named video when a store is wired.Observability (
internal/webdav)http.ServeContentswallows the reader's error. Client cancels excluded.Results (same-session head-to-head vs AIOStreams, 9 shared entries, 11 passes)
Ranges span the evening's link conditions; in like-for-like slots AltMount matched or led every column except memory.
Test plan
go test -race ./...green on top ofmain; new tests:fast_fail_hedge_test.go,fast_fail_deadwave_test.go,segment_cache_fallback_test.go,parser_warm_store_test.go,adapter_get_test.gogo build ./... && go vet ./...golangci-lint(local lint is broken against Go 1.27 export data)