fix(streaming): read the padded final AES block; stop rotating readers that make no progress - #940
Merged
Merged
Conversation
…s that make no progress createUsenetReader clamped a range end to FileSize-1 (#937). For AES-CBC encrypted RAR5/7z files whose size is not a multiple of 16 the stored segments cover the padded ciphertext (up to 15 bytes past FileSize) and the decryptor needs that final block, so the last plaintext bytes could never be produced: the AES reader hit EOF short of the range end and Read/ReadAtContext rebuilt the reader at the same offset in a tight loop, holding mvf.mu at ~2 cores until the stream read timeout, which cannot break the loop either. Clamp to the segment-covered extent instead of FileSize, and return io.ErrUnexpectedEOF when a rebuilt reader ends at the same offset again.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
createUsenetReaderclamped the range end toFileSize-1. For AES-CBC encrypted RAR5/7z files whose plaintext size is not a multiple of 16, the stored segments cover the padded ciphertext (up to 15 bytes pastFileSize, verified on a real.meta:usableTotal = FileSize + 8) and the decryptor needs that final block. With the clamp the AES reader hit EOF a few bytes short of the range end.Read/ReadAtContexttreatedEOF && hasMoreDataToRead()as "rotate to the next range" and rebuilt the reader at the same offset in a tight loop — ~1.85 cores, 187k segment-cache hits in 2 s, holdingmvf.mu— until the 2-minute stream read timeout fired, which cannot break that loop either. Surfaced in the nzb-streaming-benchmarks run as a 180 s hang on the last-2 MB integrity read of every encrypted archive withsize % 16 != 0, plus the CPU it stole from neighbouring measurements.segmentOffsetIndex.totalBytes()) instead ofFileSize— this still fixes fix(streaming): clamp a Range end past EOF instead of marking the file corrupted #937's over-EOF case — and add a no-progress guard in both rotate-on-EOF loops that returnsio.ErrUnexpectedEOFinstead of spinning.Test plan
aes_tail_read_test.go: tail read of an AES file withsize % 16 == 8returns the exact plaintext (hung before the fix); truncated-metadata variant returnsErrUnexpectedEOFpromptly instead of looping.range_past_eof_test.go(fix(streaming): clamp a Range end past EOF instead of marking the file corrupted #937) still passes.go test -race ./internal/nzbfilesystem/ ./internal/usenet/ ./internal/webdav/ ./internal/encryption/...7z-split-mediumpost: last-2 MB range now returns in ~5 s with the same sha256 every other app serves (3a59d975…); was a 180 s timeout.raw,altmount, smoke/core/stress): zero stream read timeouts; the two affected items' integrity wall 180 s → ~0.5 s, CPU 345 s → 15 s per item.make—golangci-lintcurrently fails locally on a Go 1.27 export-data version mismatch in untouched files; CI lint should confirm.