fix(par2): verify existing PAR2 sets describe the source before reusing them - #273
Merged
Merged
Conversation
…ng them PAR2 sets left in the shared temp dir by earlier uploads (never cleaned when the transfer rows were dropped by a database reset) were reused for any later file with the same basename, and volumes were collected with a loose prefix match that pulled in other uploads' .vol files. Both ended up in the NZB. Parse the main .par2 File Description packets and reuse a set only when it describes exactly the input files (name, size, hash of first 16 KiB), and match volumes strictly as <base>.volN+M.par2. Applies to the native and parpar executors, single-file and folder sets. Fixes the last report on #184.
This was referenced Sep 8, 2026
The transfer writer classified any .par2 path as generated_par2 by filename, so after a verified upload the cleaner deleted users' own PAR2 files that Postie had merely reused from the watch folder; standalone mode deleted them right after posting for the same reason. Par2Executor now returns a Result that separates Created from Reused files. Postie posts both, deletes only Created after posting, and tells the durable recorder which paths it generated; the recorder derives the file role from that instead of the filename, so reused PAR2 files are recorded as originals and follow the delete_original policy. Fixes #274.
Generated PAR2 sets written to par2.temp_dir were never removed once the transfer rows disappeared (database reset, old crash): the cleaner only runs for verified transfers and no OS reliably purges the temp root, least of all inside a Docker container. Generated files now go into <temp_dir>/postie-par2, a directory Postie owns, and a sweeper removes .par2 files there that are older than 24h, not referenced by any transfer_files row and not reserved by a running job. It runs five minutes after startup and hourly afterwards. Jobs reserve every PAR2 file they post, reused sets included, until they close, so a set can never be swept mid-upload. With an empty temp_dir PAR2 files still sit next to the sources and nothing is swept there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses the last comment on #184: after a database reset and re-upload, the NZB contained hundreds of extra
.par2entries (527 → 986 files in the reporter's screenshot). Also fixes #274 and the disk leak both stem from.1. Stale PAR2 sets reused from the shared temp dir (#184)
Root cause. Generated PAR2 sets go to
par2.temp_dir(default: OS temp dir, invisible inside Docker). In durable mode they are only removed by the transfer cleaner after verification, so a DB reset orphans them. On the next upload any set whose filename matched the source basename was reused without checking it was built from the same data, and volumes were collected with a loose prefix match that also picked up other uploads'<name>*.vol*.par2files.Fix
internal/par2/existing.go: parse the main.par2File Description packets and reuse an on-disk set only if it describes exactly the input files (name, size, MD5 of first 16 KiB, mirroring par2go). Otherwise warn and regenerate.<base>.volN+M.par2.2. Cleaner deleted users' own PAR2 files (#274)
Root cause.
transferwriter.roleFormarked any.par2path asgenerated_par2, so reused user files were deleted by the cleaner after verification (and by standalone mode right after posting).Fix
Par2Executorreturnspar2.Result{Created, Reused}; Postie postsAll(), deletes onlyCreated, and callsRecorder.MarkGeneratedfor created paths before posting.delete_originalpolicy.3. Orphaned sets are now garbage-collected
<temp_dir>/postie-par2, a directory Postie owns (emptytemp_dirstill means next to the sources, and nothing is swept there).internal/par2sweep: removes.par2files in that dir older than 24h that notransfer_filesrow references and no running job has reserved. A failed store lookup aborts the pass without deleting.temp_dirwill see PAR2 files appear in apostie-par2/subfolder. Leftovers already in the old location are not swept.Test plan
.par2rejectedResultsplit asserted in every existing reuse test (native + parpar executors); recorder role fromMarkGenerated; reused PAR2 survives posting in standalone mode.par2files; keeps everything on lookup failure; honours the initial delay. Reservation of created and reused sets released onClose.go test -race ./internal/... ./pkg/...green (cmd/webneeds the frontend bundle, unrelated)