Skip to content

fix(par2): verify existing PAR2 sets describe the source before reusing them - #273

Merged
javi11 merged 3 commits into
mainfrom
fix/par2-stale-reuse
Sep 8, 2026
Merged

javi11 merged 3 commits into
mainfrom
fix/par2-stale-reuse

Conversation

@javi11

@javi11 javi11 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Addresses the last comment on #184: after a database reset and re-upload, the NZB contained hundreds of extra .par2 entries (527 → 986 files in the reporter's screenshot). Also fixes #274 and the disk leak both stem from.

1. Stale PAR2 sets reused from the shared temp dir (#184)

Root cause. Generated PAR2 sets go to par2.temp_dir (default: OS temp dir, invisible inside Docker). In durable mode they are only removed by the transfer cleaner after verification, so a DB reset orphans them. On the next upload any set whose filename matched the source basename was reused without checking it was built from the same data, and volumes were collected with a loose prefix match that also picked up other uploads' <name>*.vol*.par2 files.

Fix

  • New internal/par2/existing.go: parse the main .par2 File Description packets and reuse an on-disk set only if it describes exactly the input files (name, size, MD5 of first 16 KiB, mirroring par2go). Otherwise warn and regenerate.
  • Volumes are matched strictly as <base>.volN+M.par2.
  • Shared by the native and parpar executors, for single-file and folder sets.

2. Cleaner deleted users' own PAR2 files (#274)

Root cause. transferwriter.roleFor marked any .par2 path as generated_par2, so reused user files were deleted by the cleaner after verification (and by standalone mode right after posting).

Fix

  • Par2Executor returns par2.Result{Created, Reused}; Postie posts All(), deletes only Created, and calls Recorder.MarkGenerated for created paths before posting.
  • The recorder derives the role from that set instead of the filename. Reused PAR2 files are recorded as originals and follow the delete_original policy.

3. Orphaned sets are now garbage-collected

  • Generated files go into <temp_dir>/postie-par2, a directory Postie owns (empty temp_dir still means next to the sources, and nothing is swept there).
  • New internal/par2sweep: removes .par2 files in that dir older than 24h that no transfer_files row references and no running job has reserved. A failed store lookup aborts the pass without deleting.
  • Runs 5 minutes after startup, then hourly. Jobs reserve every PAR2 file they post (created or reused) until they close.
  • Behaviour change: users with an explicit temp_dir will see PAR2 files appear in a postie-par2/ subfolder. Leftovers already in the old location are not swept.

Test plan

  • Regression tests: strict volume matching, stale set with same name/different content rejected, folder set rejected when files added/removed/changed, garbage .par2 rejected
  • Result split asserted in every existing reuse test (native + parpar executors); recorder role from MarkGenerated; reused PAR2 survives posting in standalone mode
  • Sweeper: removes only old, unreferenced, unreserved .par2 files; keeps everything on lookup failure; honours the initial delay. Reservation of created and reused sets released on Close.
  • go test -race ./internal/... ./pkg/... green (cmd/web needs the frontend bundle, unrelated)

…ng them

PAR2 sets left in the shared temp dir by earlier uploads (never cleaned when
the transfer rows were dropped by a database reset) were reused for any later
file with the same basename, and volumes were collected with a loose prefix
match that pulled in other uploads' .vol files. Both ended up in the NZB.

Parse the main .par2 File Description packets and reuse a set only when it
describes exactly the input files (name, size, hash of first 16 KiB), and
match volumes strictly as <base>.volN+M.par2. Applies to the native and
parpar executors, single-file and folder sets.

Fixes the last report on #184.
The transfer writer classified any .par2 path as generated_par2 by
filename, so after a verified upload the cleaner deleted users' own PAR2
files that Postie had merely reused from the watch folder; standalone mode
deleted them right after posting for the same reason.

Par2Executor now returns a Result that separates Created from Reused
files. Postie posts both, deletes only Created after posting, and tells
the durable recorder which paths it generated; the recorder derives the
file role from that instead of the filename, so reused PAR2 files are
recorded as originals and follow the delete_original policy.

Fixes #274.
Generated PAR2 sets written to par2.temp_dir were never removed once the
transfer rows disappeared (database reset, old crash): the cleaner only
runs for verified transfers and no OS reliably purges the temp root,
least of all inside a Docker container.

Generated files now go into <temp_dir>/postie-par2, a directory Postie
owns, and a sweeper removes .par2 files there that are older than 24h,
not referenced by any transfer_files row and not reserved by a running
job. It runs five minutes after startup and hourly afterwards. Jobs
reserve every PAR2 file they post, reused sets included, until they
close, so a set can never be swept mid-upload. With an empty temp_dir
PAR2 files still sit next to the sources and nothing is swept there.
@javi11
javi11 merged commit 6724364 into main Sep 8, 2026
4 checks passed
@javi11
javi11 deleted the fix/par2-stale-reuse branch September 8, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cleaner deletes user-supplied PAR2 files after verified upload (role inferred from filename)

1 participant