feat(indexer): send an indexer's grabs to a pinned download client (#3080)

changelog.d/3080-per-indexer-download-client.md

@@ -0,0 +1,2 @@
+### Added
+- **Per-indexer download client** (#3080): each indexer now has an optional **Download client** setting in Settings → Indexers. Leave it on Default and nothing changes. Pick a client and every grab from that indexer goes to it: scheduled searches, the search page, held releases in Pending and Retry download. It overrides the client's priority and category routing, so a private tracker's torrents can live on their own qBittorrent. If that client is disabled, missing, the wrong protocol or not answering, the grab fails with a message naming the indexer (and the client, for an admin) and is never sent to another client. A scheduled search skips only that indexer's releases and still grabs from your others, and a held release from that indexer stays in Pending until its client is back. A download client that an indexer is pinned to cannot be deleted (the request is refused with the indexer names), and the indexer row shows a warning while its client is disabled, missing or the wrong protocol. Prowlarr syncs keep the setting.

cmd/bindery/main.go

@@ -666,6 +666,7 @@ func main() {
 		WithAliases(authorAliasRepo).
 		WithQualityProfiles(qualityProfileRepo).
 		WithEditions(editionRepo).
+		WithDownloadClients(dlClientRepo).
 		WithSearchResults(searchResults).
 		WithSearchDebugLog(searchDebugLog)
 	if clients, err := dlClientRepo.List(ctxBoot); err == nil {

docs/API.md

@@ -538,6 +538,72 @@ value is rejected with 400. `GET /indexer` and `GET /indexer/{id}` also return
 `dailyQueriesUsed` on capped indexers, which is a display figure summed from the
 stored hourly buckets and lags the live tally by up to one flush interval.
 
+#### Per-indexer download client pin
+
+`downloadClientId` on an indexer pins every grab from it to one download client
+(#3080). `null` and omitted on create mean no pin, which keeps the usual routing.
+Omitted on `PUT` keeps the stored pin, and `null` clears it. The client must exist
+and its protocol must match the indexer's (`torznab` is torrent, `newznab` is
+usenet), otherwise `POST` and `PUT` answer `400` with the reason. A client that is
+disabled is refused for a new or changed pin, but an unrelated edit of an
+indexer already pinned to a client that was disabled later still saves.
+
+A pinned grab ignores the client's priority and media type routing. If the pinned
+client is disabled, missing or of the wrong protocol, the grab fails and no other
+client is tried: `POST /queue/grab`, `POST /queue/{id}/retry` and
+`POST /pending/{id}/grab` answer `400` with `code: "indexer_client_unavailable"`.
+The message names the indexer, and the client only for an admin caller. The bulk
+retry returns the same text per item in its results map instead of a status. A
+client that is enabled but does not answer is the usual `502`, with the message
+extended to say the indexer is pinned and no other client was tried. The field is
+only returned by the admin only indexer endpoints.
+
+
+Which indexer a grab belongs to is decided once, from the `indexerId` you send
+and the release URL's host, before the URL is signed. Signing never overrides
+the decision. The rules, in order:
+
+1. An `indexerId` that exists and is pinned always wins. Its pin applies,
+   whatever the URL's host is and whoever could sign the URL.
+2. An `indexerId` that exists and is not pinned is taken at face value when its
+   host matches the release URL's host, even if a pinned indexer shares that
+   host. When its host does not match the URL's host and some indexer that does
+   match is pinned, the grab is refused. Otherwise it stands.
+3. An `indexerId` that matches no indexer (or `null`, as on a retry of a row whose
+   indexer was deleted) is treated as no id at all.
+4. With no usable id, the indexers whose host matches the URL decide. None: the
+   grab is routed as if unpinned. Exactly one: that is the release's indexer, and
+   its pin, if any, applies (this finds a recreated indexer, and keyless
+   indexers). Several: if any is pinned the grab is refused, because it cannot be
+   told which one it came from; if none is pinned nothing changes.
+
+A refusal is the `400` with `code: "indexer_client_unavailable"` and nothing is
+sent. The message says the release could not be tied to one indexer (or that the
+named indexer does not match the release's host), and lists the pinned indexers
+for an admin only. The row's `indexerId` is the decided indexer. The retry
+and the pending grab go through the same decision.
+
+The host comparison ignores case, one trailing dot and the port, in any
+combination: `prowlarr` and `prowlarr:9696` are the same host. Ignoring a
+differing port is deliberate, because a spurious refusal is safer than letting a
+private release reach the public client. These do not match, so an id-less grab
+through them is routed as if unpinned unless you send `indexerId`: aliases of one
+machine (`localhost`, `127.0.0.1` and a DNS name), `www.` and other subdomains,
+scheme-less indexer URLs, and punycode against Unicode spellings. On a shared
+Prowlarr or Jackett host, one pinned indexer makes id-less API grabs fail for
+every other indexer on that host; the web app and the scheduler always send the
+indexer and are unaffected.
+
+On save, a client with the wrong protocol is refused only when the request sets
+the pin or changes the indexer `type`. Enabling, disabling or reprioritising an
+indexer whose pinned client was changed to the other protocol later still saves;
+its grabs fail at grab time.
+
+`DELETE /downloadclient/{id}` answers `409` while any indexer is pinned to the
+client, with `error` and an `indexers` array of their names. Disabling the client
+with `PUT` is not refused. A Prowlarr sync, and any other update from outside,
+keeps the pin.
+
 #### Per-indexer seed limits
 
 Three optional overrides are applied to a torrent grabbed from the indexer:

docs/Troubleshooting-Wiki.md

@@ -249,6 +249,54 @@ A Queue row in `failed` never produced a file: the grab did not reach the downlo
 
 To retry in bulk, tick the rows and use **Retry selected**, or use **Retry all failed** above the list, which covers both stages: an import failure has its import re-armed, a failed download has its release re-sent. Both ask for confirmation when a re-send is in the batch, because that hands work back to your download client.
 
+### A grab fails with "indexer ... is pinned to download client ..."
+
+The indexer has a **Download client** set in Settings → Indexers, and that client
+cannot take the grab. The message says which of three things is wrong: the client
+is **disabled** (enable it in Settings → Download Clients), it **no longer
+exists** (choose another client on the indexer), or it is the **wrong protocol**
+(a torrent client on a usenet indexer, usually because the client's type was
+changed after the pin). If the client is enabled but the message says the
+release could not be sent, the client is not answering: check that it is running
+and reachable from Bindery.
+
+The message names the indexer, and the client too when you are an admin; other
+users see the indexer only.
+
+Bindery will not send the release to a different client. That is deliberate: the
+pin exists so a private tracker's torrents never reach your public client. To
+send the indexer's grabs through the normal routing again, set its **Download
+client** back to *Default*. A grab refused before anything was sent creates no
+Queue row: from the search page the error appears on the page, and a scheduled
+search records it in History (once a day per book and indexer, and as a
+notification naming the indexer). Just search or grab again once the cause is
+fixed. Only a grab that reached an enabled client and failed there leaves a
+failed Queue row with **Retry download**.
+
+Limits worth knowing:
+
+* A scheduled search skips only the failing indexer's releases. Releases from
+  other indexers for the same book are still grabbed, and nothing from the
+  pinned indexer goes to another client.
+* Which indexer a grab belongs to is decided from the `indexerId` sent and the
+  release URL's host: a pinned `indexerId` always wins; an unpinned one is taken
+  at face value if its host matches the URL (and refused if it contradicts the
+  URL's host while a pinned indexer owns that host); an unknown or missing id is
+  treated as no id, and then the indexers on the URL's host decide (none:
+  unpinned, one: that indexer, several with one pinned: refused). Grabs from the
+  Bindery UI and the scheduler always send the indexer.
+* An id-less API grab can only be tied to a pinned indexer by host. The match
+  ignores case, a trailing dot and the port, so `prowlarr` against
+  `prowlarr:9696` works. It does not match aliases of one machine (`localhost`,
+  `127.0.0.1` and a DNS name), `www.` or other subdomains, scheme-less indexer
+  URLs or punycode spellings; such a grab is routed as if unpinned. Send
+  `indexerId` to be sure the pin applies.
+* On a shared Prowlarr or Jackett host, one pinned indexer makes id-less API
+  grabs fail for every other indexer on that host, because such a grab cannot be
+  told apart. Send `indexerId` (the UI and the scheduler already do).
+* A pending (held) release from a pinned indexer whose client is down stays in
+  Pending and is tried again on the next sweep. It is not discarded.
+
 ### The Queue looks empty, or shorter than it should be
 
 Look for a banner above the list saying *Could not reach `<client>`. This list may be incomplete*. Bindery gives each download client a short deadline when it renders the Queue, and a client that misses it is dropped from that render, so its rows are missing until the next poll succeeds. The rows and the downloads themselves are fine. Check the client is up and reachable at the host and port in **Settings → Download clients** (its **Test** button is the quickest answer), and remember a client that is up but very slow to answer will do this intermittently.

docs/User-Guide-Wiki.md

@@ -424,6 +424,36 @@ book set to both formats is still held for its sibling and released with it. Aud
 the indexer, and why the category (default `books`) must already exist in the
 client. Bindery fetches the .torrent/NZB itself and hands it over.
 
+**Pinning an indexer to a download client.** Settings → Indexers has a
+**Download client** dropdown on every indexer. Leave it on *Default* and
+nothing changes: Bindery picks the client by protocol, priority and media type
+as before. Pick a client and every grab from that indexer goes to that client,
+whether it came from the scheduled search, the search page, the Pending tab, or
+Retry download on the Queue. This is how you keep a private tracker's torrents
+on a separate qBittorrent (its own VPN, port or seeding policy) from your public
+one. The list shows only enabled clients of the indexer's protocol, and saving a
+pin to a client of the other protocol is refused.
+
+* **A pin wins over everything else.** It overrides the client's priority and
+  its media type or category routing: an audiobook from a pinned indexer goes to
+  the pinned client even if another client has an audiobook category.
+* **There is no fallback.** If the pinned client is disabled, missing or of the
+  wrong protocol, or does not answer, the grab fails with a message naming the
+  indexer (and the client too, if you are an admin), and nothing is sent to any
+  other client. A scheduled search skips that indexer's releases, still grabs
+  from your other indexers, and records the failure in History once a day per
+  book and indexer. The indexer's row in Settings shows an amber warning while
+  its pinned client is disabled, missing or the wrong protocol. Enable the
+  client or pick another one and the next grab goes through. A grab with no
+  indexer id (an API caller) is tied to its indexer by the release URL's host;
+  see the troubleshooting page for the limits.
+* **A pinned client cannot be deleted.** Deleting a download client that an
+  indexer is pinned to is refused (HTTP 409) with the indexer names. Choose
+  another client for those indexers, or set them back to *Default*, first.
+  Disabling the client is allowed.
+* **A Prowlarr sync keeps the pin.** It is local configuration like priority and
+  the seed limits.
+
 **Seeding limits.** A torrent indexer can carry its own seeding rules in
 Settings → Indexers, so a private tracker's minimum is met without changing the
 client's global rules for everything else. **Seed ratio** stops seeding at that

internal/api/download_clients.go

@@ -403,6 +403,14 @@ func (h *DownloadClientHandler) Delete(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 	if err := h.clients.Delete(r.Context(), id); err != nil {
+		// An indexer pinned to this client (#3080): refuse, and name them,
+		// because deleting it would quietly send their grabs to the default
+		// client, which is what the pin exists to prevent.
+		var pinned *db.ClientPinnedError
+		if errors.As(err, &pinned) {
+			writeJSON(w, http.StatusConflict, map[string]any{"error": pinned.Error(), "indexers": pinned.Indexers})
+			return
+		}
 		writeServerError(w, r, err)
 		return
 	}

internal/api/grab_pin.go

@@ -0,0 +1,125 @@
+package api
+
+import (
+	"strings"
+
+	"github.com/vavallee/bindery/internal/indexer/newznab"
+	"github.com/vavallee/bindery/internal/models"
+)
+
+// grabPinKind is the outcome of resolveGrabPin.
+type grabPinKind int
+
+const (
+	// pinNoInfo: nothing identifies an indexer and nothing on the host calls for
+	// action. The grab is unpinned and routed as it always was. Signing may still
+	// attribute an indexer afterwards, for the seed limits.
+	pinNoInfo grabPinKind = iota
+	// pinUseIndexer: the grab is attributed to grabPinDecision.IndexerID, and that
+	// indexer's pin, or lack of one, decides the client. Nothing downstream may
+	// replace it.
+	pinUseIndexer
+	// pinRefuse: the grab cannot safely be routed and is refused.
+	pinRefuse
+)
+
+// Reasons for a refusal.
+const (
+	pinRefuseAmbiguous     = "ambiguous"
+	pinRefuseContradiction = "contradiction"
+)
+
+// grabPinDecision is the one answer to "which indexer does this grab belong to,
+// and may it be sent".
+type grabPinDecision struct {
+	Kind      grabPinKind
+	IndexerID int64    // pinUseIndexer
+	Reason    string   // pinRefuse
+	Pinned    []string // pinRefuse: the pinned indexers that made it unsafe
+}
+
+// resolveGrabPin decides, from the caller's indexer id (nil when none), the
+// release URL and the full indexer list, which indexer a grab belongs to for the
+// purpose of a download client pin (#3080). It does no signing and no IO. The
+// rules, in order:
+//
+//	a. A supplied id that exists and is pinned wins, whatever the host.
+//	b. A supplied id that exists and is unpinned is trusted when its host
+//	   matches the release URL's host. When it does not match (a contradiction)
+//	   and any pinned indexer does match the URL's host, the grab is refused.
+//	   Otherwise the supplied indexer stands.
+//	c. A supplied id that matches no indexer is treated as no id.
+//	d. With no usable id, the indexers whose host matches the URL decide: none
+//	   gives no information; exactly one is the release's indexer; several are
+//	   refused when any of them is pinned (the release cannot be told apart, and
+//	   signing's pick is a guess) and otherwise give no information.
+//
+// Host matching is newznab.DownloadHostMatchesIndexer: case, one trailing dot
+// and the port are ignored.
+func resolveGrabPin(callerID *int64, releaseURL string, idxs []models.Indexer) grabPinDecision {
+	matches := func(idx models.Indexer) bool {
+		return newznab.DownloadHostMatchesIndexer(releaseURL, idx.URL)
+	}
+	var matching, pinnedOnHost []models.Indexer
+	var supplied *models.Indexer
+	for i := range idxs {
+		if callerID != nil && idxs[i].ID == *callerID {
+			supplied = &idxs[i]
+		}
+		if matches(idxs[i]) {
+			matching = append(matching, idxs[i])
+			if idxs[i].DownloadClientID != nil {
+				pinnedOnHost = append(pinnedOnHost, idxs[i])
+			}
+		}
+	}
+	names := func(list []models.Indexer) []string {
+		out := make([]string, 0, len(list))
+		for _, idx := range list {
+			out = append(out, idx.Name)
+		}
+		return out
+	}
+
+	if supplied != nil {
+		if supplied.DownloadClientID != nil || matches(*supplied) || len(pinnedOnHost) == 0 {
+			return grabPinDecision{Kind: pinUseIndexer, IndexerID: supplied.ID}
+		}
+		return grabPinDecision{Kind: pinRefuse, Reason: pinRefuseContradiction, Pinned: names(pinnedOnHost)}
+	}
+
+	switch {
+	case len(matching) == 0:
+		return grabPinDecision{Kind: pinNoInfo}
+	case len(matching) == 1:
+		return grabPinDecision{Kind: pinUseIndexer, IndexerID: matching[0].ID}
+	case len(pinnedOnHost) > 0:
+		return grabPinDecision{Kind: pinRefuse, Reason: pinRefuseAmbiguous, Pinned: names(pinnedOnHost)}
+	default:
+		return grabPinDecision{Kind: pinNoInfo}
+	}
+}
+
+// ambiguousPinError refuses a grab that cannot safely be routed because of a
+// pinned indexer (#3080). Sending it to the default client could put a private
+// tracker's release on the public client, so it is not sent.
+type ambiguousPinError struct {
+	reason string
+	pinned []string
+}
+
+func (e *ambiguousPinError) Error() string { return e.Message(true) }
+
+// Message names the pinned indexers only for an admin.
+func (e *ambiguousPinError) Message(includeNames bool) string {
+	names := ""
+	if includeNames && len(e.pinned) > 0 {
+		names = " (" + strings.Join(e.pinned, ", ") + ")"
+	}
+	if e.reason == pinRefuseContradiction {
+		return "the indexer named on this grab does not match the release's host, and an indexer on that host is pinned to a download client" + names +
+			". It was not sent to any client. Grab it from the search results so the right indexer is known."
+	}
+	return "this release could not be tied to one indexer, and an indexer on its host is pinned to a download client" + names +
+		". It was not sent to any client. Grab it from the search results so the indexer is known."
+}