db/downloader: bound kept-snapshot seeding and rebuild malformed metainfo - #23459
db/downloader: bound kept-snapshot seeding and rebuild malformed metainfo#23459AskAlexSharov wants to merge 7 commits into
Conversation
…info Three fixes on the kept-local path, all reachable on a datadir that has snapshot data but no .torrent files (rsync'd or restored), where every preverified item takes the kept-local branch. Unbounded fan-out: addDownload spawned one goroutine per item, each running BuildFromFilePath, which SHA-1s a whole data file with a 2 MiB piece buffer. Thousands of those at once exhaust descriptors and thrash the disk. Bound it like BuildTorrentFilesIfNeed, and let the wait observe the downloader shutting down so a stop drains queued tasks instead of hashing every file first. Note the bound is scoped to d.ctx, not the batch: wait()'s defer runs abandon() on every path including success, and abandon cancels unconditionally, so a batch-scoped wait would drop seeding for a download that succeeded. Malformed metainfo: BuildTorrentIfNeed only checks that the .torrent path exists, so a corrupt one suppressed the derivation the kept data would supply. addCompleteTorrent then failed on the same bytes and the snapshot stayed out of torrentsByName -- absent from allActiveSnapshots and PublishLocalChainToml -- for the life of the process. Remove metainfo confirmed malformed so it is rebuilt from the data, keeping it when the file merely could not be read. Also: README described the complete path as dropping a mismatched .torrent and as never deriving one, both the opposite of what the code does; the maybeLoadMetainfoFromDisk docstring still promised a removal dropped in #18056; and the name assertions in the local-snapshot tests could not fail, since an earlier warn had already put the name in the buffer.
9052d38 to
c170688
Compare
|
Overlap check against #23472, which was opened for the same defect. #23472 bounds the fan-out with a downloader-wide What is only here: Suggested split: take the bound and the cancellation from #23472, and reduce this PR to the metainfo repair plus the docs. #23446 is blocked on both landing. |
|
Reviewed #23446 and QA'd it on n0; three of the findings land here. Checked each against this branch at 47b7552, so two of them are already yours — noting them only to say they're confirmed fixed. Already covered here. Still live1. Kept data with no local
That is exactly the datadir this PR names in its own description — snapshot data with no I don't think there's a free fix: with no local 2.
If seeding is meant to outlive the batch, it probably shouldn't be on Question, not a defectThe README now says a stale-but-well-formed |
AddNewSeedableFile stopped at addCompleteTorrent, so a kept snapshot kept DisallowDataUpload and carried no trackers.
# Conflicts: # db/downloader/download-batch.go # db/downloader/downloader.go # db/downloader/downloader_test.go
|
Rebased on main and reduced again: the What is left is the malformed-metainfo repair, the README/docstring corrections, and |
Follow-up to the review on #23446; lands on
mainsince that cherry-pick isfaithful. The seeding bound and cancellation went to #23472, the
AddNewSeedableFileupload fix to #23653. What is left:problem: on a datadir with snapshot data but no
.torrentfiles (rsync'd orrestored),
BuildTorrentIfNeedonly checks that the path exists. A corrupt.torrentsuppressed the derivation the kept data would supply, and the snapshotstayed out of
torrentsByName— hence out ofallActiveSnapshotsandPublishLocalChainToml— for the life of the process.Solution: remove and rebuild confirmed-malformed metainfo; keep a file that merely
could not be read.
TestKeptLocalSnapshotWithUnreadableMetainfoIsSeededverified red.Also: three README/docstring statements that were the opposite of the code.