fix: don't grab past MAM's unsatisfied limit when its count lags; respect its download freeze

CHANGELOG.md

@@ -2,6 +2,21 @@
 
 Releases of this fork. Images: `ghcr.io/mikaeltarquin/shelfmark:<version>` and `:latest`.
 
+## Unreleased
+
+### Fixed
+
+- **MyAnonamouse downloads no longer run past the unsatisfied limit when MAM's count
+  lags.** MAM only counts a torrent once its tracker hears from the client, so while it
+  rejected the client's announces (an unregistered IP, say) Shelfmark went on grabbing.
+  Now each grab also counts the MAM torrents in the torrent client, and MAM downloads wait
+  while MAM rejects the client's announces or can't be read.
+- **MAM's day-long download freeze is respected.** Read from a refused download (or
+  entered under **Download pause** in the account panel), it holds MAM downloads until it
+  lifts, with a countdown in the top bar.
+- qBittorrent's MyAnonamouse torrents with no working tracker are no longer left out of
+  the slot timing.
+
 ## v1.2.0
 
 ### Added

docs/myanonamouse-enrichment.md

@@ -90,6 +90,12 @@ MyAnonamouse limits how many torrents can be unsatisfied (not yet seeded 72 hour
 
 A MAM download that's short only of unsatisfied slots doesn't have to be held back or fail. The hold-back prompt offers **Queue and wait for a slot**. A queued MAM download whose turn comes while the account is full, counting the slots kept free (or whose grab MAM refuses while at the limit), goes back to the queue as **Waiting for an unsatisfied slot on MyAnonamouse**. Waiting downloads are released, longest waiting first, once a slot is free (checked every 2 minutes), in their order on **Activity → Queued**. A torrent already in your torrent client (a retry after it was handed over) doesn't wait for a slot: it already has one. Shelfmark's own snatches that MAM's count doesn't show yet count too, since MAM's figure lags; once MAM's count has caught up they aren't counted a second time. Cancel it from Activity to drop it. Waiting downloads don't survive a restart; they show as interrupted, with Retry.
 
+MAM's count only moves when its tracker hears from your torrent client, so Shelfmark doesn't trust it alone. Before each grab it also counts the MyAnonamouse torrents in your torrent client (Deluge, qBittorrent or Transmission) still downloading or short of 72 hours' seeding, and goes by whichever count is higher. And MAM downloads wait (saying why on **Activity → Queued**) while:
+
+- **MAM is rejecting the client's announces** ("Unrecognized host/PassKey", say, when the client's IP isn't registered: mousehole failing, a VPN changing address). MAM counts none of the new torrents meanwhile, so its figure is short of the truth. Downloads go again once a re-announce succeeds; force one from the client to hurry it.
+- **MAM can't be read** (an expired or locked session ID, a network error). Unlike the check when you queue, this one waits rather than going ahead.
+- **MAM has frozen downloads.** A grab past the unsatisfied limit gets every download refused for about a day ("Attempted to Download Past Unsatisfied limit … Time Till Download Allowed: 1d 02:04:02"). Shelfmark reads that from a refused download, or you can enter it under **Download pause** in the account panel, with **Clear** if it lifts early. The top-bar button counts down to when it lifts, in red. It's kept across restarts.
+
 ### When slots free up
 
 The account panel (and the top-bar button and its tooltip) also estimates when unsatisfied
@@ -105,3 +111,5 @@ after 72 hours minus the time it has already seeded.
 - Torrents still downloading haven't started their 72 hours. Torrents MAM counts but that
   are no longer in your client can't be timed, and don't free a slot by seeding.
 - Only counts and times are shown, no torrent names, so every user can see it.
+- Torrents whose announces MAM rejected weren't seeding as far as MAM is concerned, so
+  those slots free later than shown.

docs/troubleshooting.md

@@ -28,6 +28,15 @@ It's listed under **Activity → Queued → Up next**, with the countdown to the
 the MyAnonamouse button. See
 [Unsatisfied limit](myanonamouse-enrichment.md#unsatisfied-limit).
 
+## "MyAnonamouse is rejecting …'s announces" or "MyAnonamouse has paused downloads"
+
+MAM downloads are on hold, whatever the free slots. **Rejected announces**: MAM doesn't
+recognise the torrent client's IP or passkey, so it isn't counting new torrents; fix the
+IP registration (mousehole, VPN) and the downloads go once a re-announce succeeds.
+**Paused downloads**: a grab went past the unsatisfied limit and MAM refuses downloads
+until the time shown. See
+[Unsatisfied limit](myanonamouse-enrichment.md#unsatisfied-limit).
+
 ## A queued book doesn't download
 
 **Activity → Queued** says what each saved pick is waiting for (unsatisfied slots, buffer,

shelfmark/core/mam_routes.py

@@ -14,7 +14,7 @@
 from shelfmark.core.auth_modes import load_active_auth_mode
 from shelfmark.core.logger import setup_logger
 from shelfmark.core.route_guards import admin_only
-from shelfmark.release_sources.prowlarr import mam_account, mam_autobuy, mam_points
+from shelfmark.release_sources.prowlarr import mam_account, mam_autobuy, mam_freeze, mam_points
 
 if TYPE_CHECKING:
     from collections.abc import Callable
@@ -80,6 +80,27 @@ def api_mam_unsat_timing() -> Response:
         # Counts and times only, no torrent names: every user downloads on the account.
         return jsonify(mam_account.unsat_timing())
 
+    @app.route("/api/mam/freeze", methods=["POST"])
+    @login_required
+    @admin_only
+    def api_mam_freeze_set() -> Response | tuple[Response, int]:
+        # MAM's "Time Till Download Allowed", as the site shows it ("1d 02:04:02").
+        data = request.get_json(silent=True) or {}
+        seconds = mam_freeze.parse_duration(str(data.get("remaining") or ""))
+        if not seconds:
+            return jsonify({"error": "Enter the time left as MAM shows it, e.g. 1d 02:04:02"}), 400
+        mam_freeze.set_freeze(seconds, source="manual")
+        logger.info("MAM download freeze entered by hand: %s s", seconds)
+        return jsonify(mam_account.unsat_timing())
+
+    @app.route("/api/mam/freeze", methods=["DELETE"])
+    @login_required
+    @admin_only
+    def api_mam_freeze_clear() -> Response:
+        mam_freeze.clear()
+        logger.info("MAM download freeze cleared by hand")
+        return jsonify(mam_account.unsat_timing())
+
     @app.route("/api/mam/buffer-check", methods=["POST"])
     @login_required
     def api_mam_buffer_check() -> Response | tuple[Response, int]:

shelfmark/download/clients/__init__.py

@@ -182,6 +182,9 @@ class TrackerTorrent:
     name: str
     seeding_seconds: int  # Time spent seeding since it completed
     complete: bool  # Finished downloading (the seeding clock runs only after)
+    # The tracker's reply when its last announce failed ("Unrecognized host/PassKey"):
+    # MAM isn't counting or crediting the torrent meanwhile.
+    tracker_error: str | None = None
 
 
 class DownloadClient(ABC):

shelfmark/download/clients/deluge.py

@@ -71,6 +71,14 @@ def _get_error_message(error: object) -> tuple[str, int | None]:
     return str(error), None
 
 
+def _tracker_error(tracker_status: object) -> str | None:
+    """The tracker's reason from Deluge's "Error: <reason>" status, if the announce failed."""
+    text = str(tracker_status or "").strip()
+    if not text.lower().startswith("error"):
+        return None
+    return text.partition(":")[2].strip() or text
+
+
 def _raise_runtime_error(message: str) -> NoReturn:
     raise RuntimeError(message)
 
@@ -415,7 +423,7 @@ def list_tracker_torrents(self, tracker: str) -> list[TrackerTorrent] | None:
         statuses = self._rpc_call(
             "core.get_torrents_status",
             {},
-            ["name", "tracker_host", "seeding_time", "is_finished"],
+            ["name", "tracker_host", "seeding_time", "is_finished", "tracker_status"],
         )
         if not isinstance(statuses, dict):
             return []
@@ -425,6 +433,7 @@ def list_tracker_torrents(self, tracker: str) -> list[TrackerTorrent] | None:
                 name=str(status.get("name") or ""),
                 seeding_seconds=coerce_optional_int(status.get("seeding_time")) or 0,
                 complete=bool(status.get("is_finished")),
+                tracker_error=_tracker_error(status.get("tracker_status")),
             )
             for status in statuses.values()
             if isinstance(status, dict) and needle in str(status.get("tracker_host") or "").lower()

shelfmark/download/clients/qbittorrent.py

@@ -44,6 +44,12 @@
 _HASH_LENGTH_40 = 40
 _HASH_LENGTH_ED2K = 32
 _HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
+_TRACKER_NOT_WORKING = 4  # qBittorrent's tracker status
+# Paused or stopped: no announces, so no tracker is "working" and none has failed either.
+_IDLE_STATES = frozenset({"pausedUP", "pausedDL", "stoppedUP", "stoppedDL"})
+# (torrent hash, tracker) -> whether the torrent is on it; never changes, so asked once.
+_on_tracker: dict[tuple[str, str], bool] = {}
+_ON_TRACKER_MAX = 20_000
 _HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
 _METADATA_DOWNLOAD_STATES = {"forcedMetaDL", "metaDL"}
 # How long add_download waits for magnet metadata before falling back to the info
@@ -296,15 +302,56 @@ def list_tracker_torrents(self, tracker: str) -> list[TrackerTorrent] | None:
         if error:
             raise RuntimeError(error)
         needle = tracker.lower()
-        return [
-            TrackerTorrent(
-                name=str(getattr(torrent, "name", "") or ""),
-                seeding_seconds=coerce_optional_int(getattr(torrent, "seeding_time", 0)) or 0,
-                complete=(coerce_optional_float(getattr(torrent, "progress", 0)) or 0) >= 1.0,
+        result: list[TrackerTorrent] = []
+        for torrent in torrents:
+            tracker_error: str | None = None
+            working = str(getattr(torrent, "tracker", "") or "")
+            if working:
+                if needle not in working.lower():
+                    continue
+            else:
+                # `tracker` is the first *working* tracker: empty when every announce
+                # fails, just when it matters, so ask which trackers the torrent has.
+                torrent_hash = str(getattr(torrent, "hash", "") or "")
+                key = (torrent_hash, needle)
+                known = _on_tracker.get(key)
+                if known is False:
+                    continue
+                idle = str(getattr(torrent, "state", "") or "") in _IDLE_STATES
+                if not (known and idle):  # An idle one has no announce to have failed
+                    on_it, tracker_error = self._failing_tracker(torrent_hash, needle)
+                    if len(_on_tracker) > _ON_TRACKER_MAX:
+                        _on_tracker.clear()
+                    _on_tracker[key] = on_it
+                    if not on_it:
+                        continue
+            result.append(
+                TrackerTorrent(
+                    name=str(getattr(torrent, "name", "") or ""),
+                    seeding_seconds=coerce_optional_int(getattr(torrent, "seeding_time", 0)) or 0,
+                    complete=(coerce_optional_float(getattr(torrent, "progress", 0)) or 0) >= 1.0,
+                    tracker_error=tracker_error,
+                )
             )
-            for torrent in torrents
-            if needle in str(getattr(torrent, "tracker", "") or "").lower()
-        ]
+        return result
+
+    def _failing_tracker(self, torrent_hash: str, needle: str) -> tuple[bool, str | None]:
+        """Whether a torrent with no working tracker is on `needle`'s, and its message."""
+        if not torrent_hash:
+            return False, None
+        self._ensure_authenticated()
+        response = self._client._session.get(
+            f"{self._base_url}/api/v2/torrents/trackers",
+            params={"hash": torrent_hash},
+            timeout=10,
+        )
+        response.raise_for_status()
+        for row in response.json():
+            if isinstance(row, dict) and needle in str(row.get("url") or "").lower():
+                # Status 4 is "not working"; the others are still contacting it.
+                failed = coerce_optional_int(row.get("status")) == _TRACKER_NOT_WORKING
+                return True, (str(row.get("msg") or "") or "announce failed") if failed else None
+        return False, None
 
     def _get_torrent_info(self, download_id: str) -> tuple[SimpleNamespace | None, str | None]:
         """Get one torrent by its current qBittorrent hash."""

shelfmark/download/clients/torrent_utils.py

@@ -29,6 +29,9 @@
 _BTIH_DIGEST_LENGTH = 32
 _BTIH_HASH_LENGTH_40 = 40
 _BTIH_HASH_LENGTH_32 = 32
+_HTTP_ERROR_MIN = 400
+# A refusal page is small; anything bigger isn't one.
+_REFUSAL_MAX_BYTES = 512 * 1024
 _TORRENT_FETCH_ERRORS = (
     requests.exceptions.RequestException,
     OSError,
@@ -225,6 +228,19 @@ def clear_torrent_fetch_cache() -> None:
         _torrent_fetch_cache.clear()
 
 
+def _note_tracker_refusal(body: bytes | None) -> None:
+    """Show a reply that isn't a torrent to MAM's freeze detection.
+
+    MAM refuses a grab past the unsatisfied limit with a page saying so, and then
+    refuses every grab for a day: held downloads must wait that out.
+    """
+    if not isinstance(body, bytes) or not body or len(body) > _REFUSAL_MAX_BYTES:
+        return
+    from shelfmark.release_sources.prowlarr import mam_freeze
+
+    mam_freeze.note_refusal(body.decode("utf-8", errors="ignore"))
+
+
 def _fetch_torrent_info(url: str) -> TorrentInfo:
     """Fetch a .torrent URL and parse out the info_hash and raw torrent data.
 
@@ -297,6 +313,8 @@ def resolve_url(current: str, location: str) -> str:
             logger.debug("Following redirect to: %s...", _safe_url(redirect_url))
             current_url = redirect_url
 
+        if isinstance(resp.status_code, int) and resp.status_code >= _HTTP_ERROR_MIN:
+            _note_tracker_refusal(resp.content)
         resp.raise_for_status()
         torrent_data = resp.content
 
@@ -318,6 +336,7 @@ def resolve_url(current: str, location: str) -> str:
             logger.debug("Extracted hash from torrent file: %s", info_hash)
         else:
             logger.warning("Could not extract hash from torrent file")
+            _note_tracker_refusal(torrent_data)
         return TorrentInfo(info_hash=info_hash, torrent_data=torrent_data, is_magnet=False)
     except _TORRENT_FETCH_ERRORS as e:
         # Exception messages can repeat the source or redirect URL, including its

shelfmark/download/clients/transmission.py

@@ -7,7 +7,7 @@
 
 import importlib
 from contextlib import contextmanager, suppress
-from typing import TYPE_CHECKING, Protocol, TypeGuard
+from typing import TYPE_CHECKING, Any, Protocol, TypeGuard
 
 from shelfmark.core.config import config
 from shelfmark.core.logger import setup_logger
@@ -139,6 +139,18 @@ def _apply_transmission_ssl_verify(client: object, url: str) -> None:
         logger.debug("Unable to apply Transmission TLS verify setting: %s", e)
 
 
+def _tracker_error(torrent: Any, needle: str) -> str | None:
+    """The tracker's reason when the last announce to it failed."""
+    for stats in getattr(torrent, "tracker_stats", None) or []:
+        if needle not in str(getattr(stats, "announce", "") or "").lower():
+            continue
+        if getattr(stats, "has_announced", False) and not getattr(
+            stats, "last_announce_succeeded", True
+        ):
+            return str(getattr(stats, "last_announce_result", "") or "") or "announce failed"
+    return None
+
+
 @register_client("torrent")
 class TransmissionClient(DownloadClient):
     """Transmission download client using transmission-rpc library."""
@@ -367,7 +379,7 @@ def get_status(self, download_id: str) -> DownloadStatus:
     def list_tracker_torrents(self, tracker: str) -> list[TrackerTorrent] | None:
         """Every torrent on a tracker, with Transmission's seeding time."""
         torrents = self._client.get_torrents(
-            arguments=["name", "secondsSeeding", "percentDone", "trackers"]
+            arguments=["name", "secondsSeeding", "percentDone", "trackers", "trackerStats"]
         )
         needle = tracker.lower()
         result: list[TrackerTorrent] = []
@@ -380,6 +392,7 @@ def list_tracker_torrents(self, tracker: str) -> list[TrackerTorrent] | None:
                     name=str(torrent.name or ""),
                     seeding_seconds=int(torrent.seconds_seeding or 0),
                     complete=float(torrent.percent_done or 0) >= 1.0,
+                    tracker_error=_tracker_error(torrent, needle),
                 )
             )
         return result