Skip to content

feat: torrents download in the background, so MAM grabs aren't capped at the worker count - #68

Merged
mikaeltarquin merged 1 commit into
mainfrom
feat/torrents-download-in-background
Oct 7, 2026
Merged

mikaeltarquin merged 1 commit into
mainfrom
feat/torrents-download-in-background

Conversation

@mikaeltarquin

Copy link
Copy Markdown
Owner

Why

With 80/150 unsatisfied slots used, Shelfmark grabbed only 3 MAM torrents at a time, with long waits in between. The slot logic wasn't the limit: about 65 slots were free. The limit was Max Concurrent Downloads (default 3). A worker stayed with each torrent until the client finished downloading it, so only 3 torrents were ever in flight.

What changes

  • Torrents keep downloading in the client without a worker. This applies once a hash-keyed client (Deluge, qBittorrent, Transmission, rTorrent) has had a torrent for 30 seconds, or sooner if it stalls (this replaces fix: stalled torrents wait in the client instead of being cancelled; find them by hash #66's parking).
    • The worker is freed. The task stays downloading, showing the client's progress, which a background watcher reads every 30 seconds.
    • After 5 minutes without progress it says so: "No progress in Deluge for N minutes at X%".
    • When the client finishes the torrent, the task goes back to a worker at high priority, which imports the files (found by hash, with no .torrent fetch).
    • If the torrent leaves the client, the download fails saying so. Cancelling from Activity still works.
  • BookQueue.detach frees a downloading task's worker and runs the queue hook, which saves the retry data again with the client hash. So a retry after a restart finds the torrent in the client, even during a MAM freeze.
  • Importing isn't counted as a new MAM snatch.
    • A task's handoff is recorded only once per torrent in the client (DownloadTask.handoff_recorded; forget_handoff resets it after a refused grab).
    • pending_unsat_count skips tasks that are queued only to import.
  • MAM grabs are now limited by the free unsatisfied slots, less the slots kept free, with the existing safeguards from fix: don't grab past MAM's unsatisfied limit when its count lags; respect its download freeze #65. They are no longer limited by the worker count. Starts are still staggered a few seconds apart.
  • Changelog and troubleshooting docs are updated.

Testing

  • tests/download/test_wait_in_client.py, rewritten (18 tests):
    • freeing the worker while the task stays downloading, with the hash saved to the retry data
    • progress updates, the "no progress" message, import when finished, failure when the torrent is gone, dropping a cancelled task
    • the 30-second hand-over, and a handoff recorded only once
    • a stall outside a torrent client still cancels as before
  • New test that a torrent queued only to import isn't a pending snatch.
  • Full suite (not integration/e2e): 3780 passed. The only failures are the 7 entrypoint/hardlink tests that also fail on a clean main in this container (they run as root).
  • Ruff, BasedPyright (backend and tests) and Vulture are clean.
  • Not run against a real torrent client.

🤖 Generated with Claude Code

https://claude.ai/code/session_0184DMrDNkAEmGzPntj3ziqB


Generated by Claude Code

… at the worker count

A worker stayed with each torrent until the client finished downloading it, so
only MAX_CONCURRENT_DOWNLOADS torrents (3 by default) were ever in flight, however
many MAM unsatisfied slots were free.

- Once a hash-keyed torrent client (Deluge, qBittorrent, Transmission, rTorrent)
  has had a torrent for 30 seconds (or it stalls), its worker is freed. The task
  stays downloading, with the client's progress (checked every 30 seconds), and
  goes back to the workers to import when the client has finished it; it fails
  if the torrent leaves the client.
- BookQueue.detach frees the worker and re-saves the retry data (with the hash),
  so a retry after a restart finds the torrent in the client.
- Coming back to import isn't a new MAM snatch: the handoff is recorded once per
  task in the client, and such tasks don't count as pending snatches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0184DMrDNkAEmGzPntj3ziqB
@mikaeltarquin
mikaeltarquin merged commit 5d57fdc into main Oct 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants