feat: adopt unmatched library books from the Import page (#2547)

README.md

@@ -117,6 +117,7 @@ Clean-room Go rewrite, modern React UI, MIT-licensed, actively developed.
 - Dual-format books — each title holds an ebook *and* an audiobook in independent slots, with separate search, grab, and import pipelines, and the audiobook side moves multi-part `.m4b` / `.mp3` folders as one unit.
 - Series support with position tracking and search by series title, edition tracking (format / ISBN / publisher / page count), Calendar view of upcoming releases, and multiple library roots.
 - Library scan with four-tier matching: ASIN → title + author → series name + position → fuzzy title. Honours librarian sort-suffix form (`Title, The`) and series-annotated filenames (`[Mistborn, Book 1]`).
+- **Library adoption**: whatever the scan could not match waits on the Import page as one row per book (an audiobook folder or disc set is one row), with suggested matches (one click Confirm only for a strong match), one "Add author" decision per missing author, and Undo. Files are registered where they are, never moved.
 - Author aliases (`RR Haywood` / `R.R. Haywood` / `R R Haywood` merge into one canonical row), and metadata re-bind to correct a wrong match without delete-and-re-add. On a book’s **File → Re-bind** dialog, search configured metadata providers and select an OpenLibrary work or Hardcover book, or enter an exact provider ID and press Enter. Results already in the library are marked and link to the existing book; records belonging to another library book cannot be selected. Each result’s **Links** menu opens its upstream page when available; OpenLibrary editions can be viewed but require a work ID for re-binding. Re-binding updates metadata and series without moving files; author mismatches require confirmation.
 - Explicit author-catalogue reconciliation with a selectable preview: remove chosen stale metadata-only Wanted rows after changing provider or metadata profile while always protecting imported books and every row with a tracked file.
 

changelog.d/library-adoption.md

@@ -0,0 +1,5 @@
+### Added
+- **Adopt the books a library scan could not match** (#2547): they now wait on the Import page under **In your library**, which is where `/import` opens. Each row is a book rather than a file, so a 193 track audiobook is one row, a disc set is one row named after its folder, and `Dune.epub` beside `Dune.mobi` is one row. A close title by the same author is one click to confirm, a weaker one is offered as a possible match to check first. Books by an author who is not in your library yet are one row with a single Add author. Any row opens in place to pick a book, search your library, or search metadata, and metadata providers are only asked when you press Search. Adopting registers the files where they are: nothing is moved and nothing is searched for, a book added from metadata comes in unmonitored in the format you adopted, and Undo reverses it exactly. Ignored books stay hidden across scans until you unignore them. Admins see how many books need a decision on the Import nav entry.
+
+### Changed
+- **The unmatched files table in Settings and the bulk folder scan in Settings > Import moved to the Import page.** Settings now shows how many books need a decision with a link to review them, and the folder scan is **Import > From a folder**. The old list stopped at 1000 files; the new one has no such cap for ordinary libraries (it lists up to 20,000 books per scan and says when there are more). The `library.lastScan` summary keeps its counters, adds `unmatched_units`, `ignored_units` and `units_truncated`, and returns `unmatched_files` as an empty list for one release.

cmd/bindery/adoption_routes_test.go

@@ -0,0 +1,112 @@
+package main
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"sort"
+	"strings"
+	"testing"
+
+	"github.com/go-chi/chi/v5"
+
+	"github.com/vavallee/bindery/internal/auth"
+)
+
+// stubAdoptionHandler records which adoption route ran.
+type stubAdoptionHandler struct{ called []string }
+
+func (h *stubAdoptionHandler) record(name string, w http.ResponseWriter) {
+	h.called = append(h.called, name)
+	w.WriteHeader(http.StatusNoContent)
+}
+
+func (h *stubAdoptionHandler) List(w http.ResponseWriter, _ *http.Request)    { h.record("list", w) }
+func (h *stubAdoptionHandler) Summary(w http.ResponseWriter, _ *http.Request) { h.record("summary", w) }
+func (h *stubAdoptionHandler) Adopt(w http.ResponseWriter, _ *http.Request)   { h.record("adopt", w) }
+func (h *stubAdoptionHandler) Undo(w http.ResponseWriter, _ *http.Request)    { h.record("undo", w) }
+func (h *stubAdoptionHandler) Ignore(w http.ResponseWriter, _ *http.Request)  { h.record("ignore", w) }
+func (h *stubAdoptionHandler) Unignore(w http.ResponseWriter, _ *http.Request) {
+	h.record("unignore", w)
+}
+func (h *stubAdoptionHandler) IgnoreMany(w http.ResponseWriter, _ *http.Request) {
+	h.record("ignore-many", w)
+}
+
+// adoptionRoutes is every route registerAdoptionRoutes mounts, with a concrete
+// path for each and the handler it must reach.
+var adoptionRoutes = []struct {
+	method, pattern, path, called string
+}{
+	{http.MethodGet, "/library/unmatched", "/library/unmatched", "list"},
+	{http.MethodGet, "/library/unmatched/summary", "/library/unmatched/summary", "summary"},
+	{http.MethodPost, "/library/unmatched/ignore", "/library/unmatched/ignore", "ignore-many"},
+	{http.MethodPost, "/library/unmatched/{id}/adopt", "/library/unmatched/7/adopt", "adopt"},
+	{http.MethodPost, "/library/unmatched/{id}/undo", "/library/unmatched/7/undo", "undo"},
+	{http.MethodPost, "/library/unmatched/{id}/ignore", "/library/unmatched/7/ignore", "ignore"},
+	{http.MethodPost, "/library/unmatched/{id}/unignore", "/library/unmatched/7/unignore", "unignore"},
+}
+
+// TestAdoptionRoutesEnumerated walks the mounted router so a route added to
+// registerAdoptionRoutes without a row in adoptionRoutes (and so without the
+// admin checks below) fails here.
+func TestAdoptionRoutesEnumerated(t *testing.T) {
+	router := chi.NewRouter()
+	registerAdoptionRoutes(router, &stubAdoptionHandler{})
+	var mounted, want []string
+	if err := chi.Walk(router, func(method, route string, _ http.Handler, _ ...func(http.Handler) http.Handler) error {
+		mounted = append(mounted, method+" "+route)
+		return nil
+	}); err != nil {
+		t.Fatal(err)
+	}
+	for _, r := range adoptionRoutes {
+		want = append(want, r.method+" "+r.pattern)
+	}
+	sort.Strings(mounted)
+	sort.Strings(want)
+	if strings.Join(mounted, "\n") != strings.Join(want, "\n") {
+		t.Fatalf("mounted adoption routes:\n%s\nwant:\n%s", strings.Join(mounted, "\n"), strings.Join(want, "\n"))
+	}
+}
+
+// TestAdoptionRoutesRequireAdmin: the rows carry absolute library paths and
+// adopt writes authors, books and book files, so a user role, or no role at
+// all, is refused before any handler runs.
+func TestAdoptionRoutesRequireAdmin(t *testing.T) {
+	for _, role := range []string{"user", ""} {
+		for _, rt := range adoptionRoutes {
+			t.Run(role+" "+rt.method+" "+rt.path, func(t *testing.T) {
+				h := &stubAdoptionHandler{}
+				router := chi.NewRouter()
+				registerAdoptionRoutes(router, h)
+				req := httptest.NewRequest(rt.method, rt.path, nil)
+				if role != "" {
+					req = req.WithContext(auth.WithUserRole(req.Context(), role))
+				}
+				rec := httptest.NewRecorder()
+				router.ServeHTTP(rec, req)
+				if rec.Code != http.StatusForbidden || len(h.called) != 0 {
+					t.Fatalf("status = %d, called %v; want 403 and no handler", rec.Code, h.called)
+				}
+			})
+		}
+	}
+}
+
+// TestAdoptionRoutesAllowAdmin is the symmetry case.
+func TestAdoptionRoutesAllowAdmin(t *testing.T) {
+	for _, rt := range adoptionRoutes {
+		t.Run(rt.method+" "+rt.path, func(t *testing.T) {
+			h := &stubAdoptionHandler{}
+			router := chi.NewRouter()
+			registerAdoptionRoutes(router, h)
+			req := httptest.NewRequest(rt.method, rt.path, nil)
+			req = req.WithContext(auth.WithUserRole(req.Context(), "admin"))
+			rec := httptest.NewRecorder()
+			router.ServeHTTP(rec, req)
+			if rec.Code != http.StatusNoContent || len(h.called) != 1 || h.called[0] != rt.called {
+				t.Fatalf("status = %d, called %v; want 204 and %s", rec.Code, h.called, rt.called)
+			}
+		})
+	}
+}

cmd/bindery/main.go

@@ -701,6 +701,18 @@ func main() {
 	}
 
 	libraryHandler := api.NewLibraryHandler(importScanner).WithSettings(settingsRepo)
+	// Library adoption: the scan stores the books it could not match, and the
+	// Import page's "In your library" view acts on them. Library roots, not
+	// importRoots: adoption only registers files already in the library.
+	unmatchedUnitRepo := db.NewUnmatchedUnitRepo(database)
+	importScanner.WithUnmatchedUnits(unmatchedUnitRepo)
+	adoptionHandler := api.NewAdoptionHandler(unmatchedUnitRepo, bookRepo, authorRepo, authorHandler, libraryRoots, importScanner, settingsRepo)
+	// Any claim present at startup belongs to a process that is gone.
+	if n, err := adoptionHandler.RecoverStaleClaims(ctxBoot, 0); err != nil {
+		slog.Warn("library adoption: could not recover abandoned claims", "error", err)
+	} else if n > 0 {
+		slog.Info("library adoption: recovered abandoned claims", "count", n)
+	}
 	fileHandler := api.NewFileHandler(bookRepo, cfg.LibraryDir, cfg.AudiobookDir).
 		WithRootFolders(rootFolderRepo)
 	historyHandler := api.NewHistoryHandler(historyRepo, blocklistRepo, bookRepo)
@@ -1138,6 +1150,8 @@ func main() {
 		// admin only (#2361); see registerLibraryScanStatusRoute.
 		r.Post("/library/scan", libraryHandler.Scan)
 		registerLibraryScanStatusRoute(r, libraryHandler)
+		// Library adoption, admin only (see registerAdoptionRoutes).
+		registerAdoptionRoutes(r, adoptionHandler)
 
 		// Refresh metadata for ALL authors (background job, #863). Per-selection
 		// bulk refresh lives at /author/bulk; this is the "populate everything"

cmd/bindery/sensitive_routes.go

@@ -201,6 +201,35 @@ func registerStorageRoutes(r chi.Router, h storageRouteHandler) {
 	})
 }
 
+// adoptionRouteHandler is the surface registerAdoptionRoutes needs.
+type adoptionRouteHandler interface {
+	List(http.ResponseWriter, *http.Request)
+	Summary(http.ResponseWriter, *http.Request)
+	Adopt(http.ResponseWriter, *http.Request)
+	Undo(http.ResponseWriter, *http.Request)
+	Ignore(http.ResponseWriter, *http.Request)
+	Unignore(http.ResponseWriter, *http.Request)
+	IgnoreMany(http.ResponseWriter, *http.Request)
+}
+
+// registerAdoptionRoutes mounts /library/unmatched, library adoption, admin
+// only. The rows carry absolute library paths (the same disclosure
+// registerLibraryScanStatusRoute gates), adopt creates authors and books and
+// writes book_files, and the summary behind the nav badge counts them. No
+// route takes a path: every row is addressed by id.
+func registerAdoptionRoutes(r chi.Router, h adoptionRouteHandler) {
+	r.Group(func(r chi.Router) {
+		r.Use(auth.RequireAdmin)
+		r.Get("/library/unmatched", h.List)
+		r.Get("/library/unmatched/summary", h.Summary)
+		r.Post("/library/unmatched/ignore", h.IgnoreMany)
+		r.Post("/library/unmatched/{id}/adopt", h.Adopt)
+		r.Post("/library/unmatched/{id}/undo", h.Undo)
+		r.Post("/library/unmatched/{id}/ignore", h.Ignore)
+		r.Post("/library/unmatched/{id}/unignore", h.Unignore)
+	})
+}
+
 // libraryScanStatusRouteHandler is the surface registerLibraryScanStatusRoute
 // needs.
 type libraryScanStatusRouteHandler interface {

docs/API.md

@@ -374,6 +374,13 @@ POST   /api/v1/backup/{filename}/restore          stage a backup for the next re
 GET    /api/v1/system/status                      version, uptime, build info
 POST   /api/v1/library/scan                       start a library scan in the background (202)
 GET    /api/v1/library/scan/status                summary of the last library scan, paths included (admin)
+GET    /api/v1/library/unmatched                  books the scan could not match, one row per book (admin)
+GET    /api/v1/library/unmatched/summary          pending, ignored and adopted counts plus scan status (admin)
+POST   /api/v1/library/unmatched/{id}/adopt       register the row's files in place against a book (admin)
+POST   /api/v1/library/unmatched/{id}/undo        reverse an adoption exactly (admin)
+POST   /api/v1/library/unmatched/{id}/ignore      set a pending row aside (admin)
+POST   /api/v1/library/unmatched/{id}/unignore    return an ignored row to pending (admin)
+POST   /api/v1/library/unmatched/ignore           ignore pending rows by {"ids":[..]} or {"authorFolder":"..."} (admin)
 PUT    /api/v1/system/loglevel                    runtime log-level switch (debug/info/warn/error)
 GET    /api/v1/images?url=<encoded>               proxied + cached cover image (30-day TTL)
 ```
@@ -382,7 +389,68 @@ GET    /api/v1/images?url=<encoded>               proxied + cached cover image (
 scan: the counts, the library roots it walked and the path of every unmatched
 file. That is server filesystem layout, so the route is admin only in the same
 way as `/system/storage`, and a non admin gets `403` (#2361). A `404` means no
-scan has run yet.
+scan has run yet. The unmatched files themselves moved to
+`/library/unmatched`: the summary now carries `unmatched_units`,
+`ignored_units` and `units_truncated`, and `unmatched_files` is always an
+empty list, kept for one release so an older cached web bundle still parses.
+
+#### Library adoption
+
+`GET /api/v1/library/unmatched` lists the books a library scan could not
+match. A row is a book: an audiobook folder, a disc set, or same named ebook
+files are one row. Query parameters, all optional:
+
+| Parameter | Values |
+|---|---|
+| `state` | `pending` (default), `ignored`, `adopted` |
+| `reason` | `author_not_in_library`, `no_candidate_books`, `no_title_match`, `no_title_parsed` |
+| `authorFolder` | first folder under the library root |
+| `format` | `ebook`, `audiobook` |
+| `search` | words matched against title, author and path |
+| `sort`, `dir` | `score` (default), `title`, `folder`, `files`, `size`, `seen`; `asc` or `desc` |
+| `limit`, `offset` | page, at most 250 rows |
+| `facets` | `1` adds grouped counts by reason, format and top author folders |
+
+The response is `{items, total, facets?, summary, scan}`. Each item carries
+`id`, `kind`, `format`, `fileCount`, `sizeBytes`, `relPath`, `rootPath`,
+`authorFolder`, `parsedTitle`, `parsedAuthor`, `reason`, up to three
+`candidates` (`{book, score}`, a title similarity from 0 to 1), `state`, the
+adopted `book` if any, `bookCreated`, `authorCreated` and the first 20
+`members` file names. No provider is called to build it.
+
+`POST /api/v1/library/unmatched/{id}/adopt` takes either `{"bookId": 12}` for
+a book already in the library or `{"foreignBookId": "...", "foreignAuthorId":
+"...", "authorName": "..."}` for a metadata result. An optional `"format"`
+must equal the row's format; audio files cannot be adopted as an ebook. No
+field is a path. The files are registered in place (an audiobook folder as its
+folder, a disc set disc folder by disc folder), nothing is moved and no search
+starts. A book added from metadata is created unmonitored with its media type
+set to the files' format. Every side effect is recorded on the row before the
+next, so an adoption interrupted by a crash is reversed at the next start.
+Answers:
+
+| Status | Meaning |
+|---|---|
+| `200` | the updated row |
+| `400` | neither or both of `bookId` and `foreignBookId`, or a format that is not the files' format |
+| `404` | no such row, or the book is gone |
+| `409` | the row is not pending (the body names its `state`), or a file already belongs to a book |
+| `422` | a file is gone, is not a regular file, or resolves outside the library folders |
+| `502`, `503` | the metadata provider failed or the primary provider is down |
+
+`POST .../undo` removes exactly the book file entries the adoption made, each
+only while it still belongs to the book it was registered to, and a book or
+author it created when nothing else holds them (books by the author count
+whether excluded or not) and the book has not been used since (monitored,
+edited, linked to a series, searched or downloaded). A kept book is reported in
+the response's `message`. The row returns to pending only once every step
+succeeded; a `503` means a step failed (a busy database, say), the adoption
+is still on record and is completed automatically within a few minutes, and
+a `409` means another request took the row before the undo finished. An adopt
+that fails and cannot reverse what it did also answers `503` and is cleaned up
+the same way. Adopt, undo, ignore and unignore each claim the row with a compare and
+swap, so a double click or two admins acting at once get one `200` and one
+`409`.
 
 #### Webhook payload
 

docs/Troubleshooting-Wiki.md

@@ -313,8 +313,8 @@ co-author or a joint pen name. The log shows lines like:
 DEBUG calibre import: alias record skipped error="alias \"Isaac Asimov\" already points at author 1125 (refusing to reassign to 105)" name="Isaac Asimov"
 ```
 
-The scan then reports those same names under **Unmatched files** with "Parsed
-author isn't in your library", because the name only exists as an alias.
+The scan then lists those books on **Import → In your library** as by an
+author who is not in your library, because the name only exists as an alias.
 
 What happened: for a book credited to several people, the import used to record
 every co-author as an *alias* of the first credited author ([#1684](https://github.com/vavallee/bindery/issues/1684)).
@@ -374,6 +374,29 @@ Two things this does not do:
 
 If you reshape your library regularly, **Rename files** on the book or author page is the supported way to do it: it moves the file *and* repoints the same tracking row at the new location, so there is never a second row to clean up.
 
+## A scan leaves files unmatched
+
+Go to **Import → In your library**. Every book the last scan could not match
+is listed there, one row per book, with a sentence saying what the scan found
+and what to do: add the missing author and scan again, confirm a suggested
+book, or choose one. Adopting registers the files where they are and can be
+undone. The full walkthrough is in the user guide under [Adopting files
+already in your library](User-Guide-Wiki.md#adopting-files-already-in-your-library).
+
+A few things that look wrong but are not:
+
+- **A file you expected is not listed.** Symlinks are never listed, and a file
+  whose folder resolves outside your library folders is skipped. Replace the
+  link with the file, or add the real folder as a root.
+- **The list did not change after the volume came back.** A scan that finds no
+  files at all changes nothing on the list, so an unmounted library does not
+  erase your decisions. Scan again once it is mounted.
+- **A row you ignored is gone for good.** It is on the **Ignored** list, and
+  **Unignore** brings it back.
+
+If you file a bug about a row, include the reason code that shows when you
+hover its sentence.
+
 ## Collecting logs for a bug report
 
 `Settings → Logs` is the whole log store, so you don't need shell access to the container (rootless images give you nowhere to `cat` a file anyway).