feat: torrents download in the background, so MAM grabs aren't capped at the worker count

CHANGELOG.md

@@ -16,6 +16,14 @@ Releases of this fork. Images: `ghcr.io/mikaeltarquin/shelfmark:<version>` and `
 
 ### Changed
 
+- **Torrents download in the background, so more go at once.** A worker used to stay
+  with each torrent until the client had downloaded it, so only **Max Concurrent
+  Downloads** torrents (3 by default) were ever in flight, however many MyAnonamouse
+  slots were free. Now, once Deluge, qBittorrent, Transmission or rTorrent has had a
+  torrent for 30 seconds, its worker moves on to the next download; the torrent stays in
+  **Downloads** with the client's progress, and a worker comes back to pick up the files
+  when it's finished. MAM grabs are now limited by the free unsatisfied slots (and the
+  slots kept free), not by the workers.
 - **The MyAnonamouse panel's Buffer is now Room to your ratio**: how much more can be
   downloaded before the ratio drops below **Keep Ratio At Least**, counting Shelfmark's
   active MAM downloads. The plain buffer is in its tooltip.
@@ -32,10 +40,9 @@ Releases of this fork. Images: `ghcr.io/mikaeltarquin/shelfmark:<version>` and `
   lifts, with a countdown in the top bar.
 - qBittorrent's MyAnonamouse torrents with no working tracker are no longer left out of
   the slot timing.
-- **A torrent with no progress for 5 minutes waits in the torrent client instead of being
-  cancelled.** It was left downloading there with nothing to pick up the files, and shown
-  as Cancelled. Now it waits on **Activity → Queued** ("Waiting in Deluge: no progress…")
-  and carries on once the client shows it moving or finished.
+- **A torrent with no progress for 5 minutes is no longer cancelled.** It was left
+  downloading in the torrent client with nothing to pick up the files, and shown as
+  Cancelled. Now it stays in Downloads, saying how long it has made no progress.
 - **A retry finds its torrent in the client by its hash**, without downloading the
   .torrent again, so it works while MyAnonamouse has frozen downloads. The hash is kept
   from when the torrent was handed to the client.

docs/troubleshooting.md

@@ -37,15 +37,17 @@ IP registration (mousehole, VPN) and the downloads go once a re-announce succeed
 until the time shown. See
 [Unsatisfied limit](myanonamouse-enrichment.md#unsatisfied-limit).
 
-## "Waiting in Deluge: no progress for 5 minutes"
-
-Not an error: the torrent is in your torrent client but hasn't moved for 5 minutes (no
-peers yet, queued in the client, or the tracker refusing it for now). It goes on
-downloading there, and Shelfmark picks it up again once the client shows it moving or
-finished (checked every 2 minutes), so it doesn't hold up other downloads meanwhile. If
-the torrent is removed from the client, the download fails saying so. Cancel it from
-Activity to stop waiting. Like other waiting downloads it doesn't survive a restart: use
-Retry, which finds the torrent in the client.
+## "Downloading in Deluge" or "No progress in Deluge for … minutes"
+
+Not an error. Once the torrent client has had a torrent for 30 seconds, Shelfmark leaves
+it downloading there and moves on to the next download, so more torrents download at
+once than there are download workers. The row in **Downloads** shows the client's
+progress (checked every 30 seconds), and the files are picked up when it's finished.
+"No progress" means it hasn't moved for 5 minutes or more: no peers yet, queued in the
+client, or the tracker refusing it for now. If the torrent is removed from the client,
+the download fails saying so. Cancel it from Activity to stop watching it (the torrent
+stays in the client). After a restart it shows as interrupted: Retry finds the torrent
+in the client by its hash.
 
 ## A queued book doesn't download
 

shelfmark/core/models.py

@@ -110,6 +110,7 @@ class DownloadTask:
     # The info hash the torrent client holds the download by, once handed to one that keeps
     # it (Deluge, qBittorrent, Transmission, rTorrent): a stalled one waits there for peers.
     torrent_client_hash: str | None = None
+    handoff_recorded: bool = False  # Counted as a snatch already (see BookQueue)
     retry_ratio_limit: float | None = None  # Optional post-download seeding ratio
     retry_seeding_time_limit_minutes: int | None = None  # Optional post-download seeding time limit
     retry_source_context: dict[str, Any] = field(

shelfmark/core/queue.py

@@ -165,7 +165,10 @@ def _apply_status_locked(
             and previous_status in _NOT_STARTED
             and status not in _NOT_STARTED
             and status != QueueStatus.CANCELLED
+            # Back only to pick up files its torrent client finished: no new snatch.
+            and not (task.torrent_client_hash and task.handoff_recorded)
         ):
+            task.handoff_recorded = True
             now = time.time()
             self._handoffs[book_id] = (now, task)
             cutoff = now - _HANDOFF_MEMORY_SECONDS
@@ -207,6 +210,9 @@ def forget_handoff(self, task_id: str) -> None:
         """Drop a task's handoff: it never reached the indexer (refused at grab)."""
         with self._lock:
             self._handoffs.pop(task_id, None)
+            task = self._task_data.get(task_id)
+            if task is not None:
+                task.handoff_recorded = False
 
     def hold(self, task_id: str, message: str) -> bool:
         """Put a picked-up task back as queued without queueing it, to wait for room.
@@ -225,6 +231,29 @@ def hold(self, task_id: str, message: str) -> bool:
             self._update_status(task_id, QueueStatus.QUEUED)
             return True
 
+    def detach(self, task_id: str, message: str) -> bool:
+        """Free a downloading task's worker while its download goes on elsewhere.
+
+        For a torrent the client is downloading: it stays downloading, with `message`,
+        until `enqueue_existing` brings it back to pick up the files (or it is
+        cancelled). Its retry data is saved again, so a retry after a restart finds
+        the torrent in the client.
+        """
+        with self._lock:
+            task = self._task_data.get(task_id)
+            if task is None or self._status.get(task_id) != QueueStatus.DOWNLOADING:
+                return False
+            self._active_downloads.pop(task_id, None)
+            self._cancel_flags.pop(task_id, None)
+            task.status_message = message
+            hook = self._queue_hook
+        if hook is not None:
+            try:
+                hook(task_id, task)
+            except _QUEUE_HOOK_ERRORS as exc:
+                logger.warning("Queue hook failed while detaching task %s: %s", task_id, exc)
+        return True
+
     def update_status(self, book_id: str, status: QueueStatus) -> None:
         """Update status of a book in the queue."""
         with self._lock: