feat: enrich bare imports + remember the unresolvable

.env.example

@@ -8,6 +8,10 @@ COLOPHON_ADMIN_GROUP=admin
 # Read-only DB snapshot is taken via `podman exec <container> mariadb ... <db>`:
 COLOPHON_DB_CONTAINER=grimmory-db
 COLOPHON_DB_NAME=grimmory
+# Public URL of the grimmory UI. Set this to put a deep-link to each book's page
+# (route /book/<id>, Authelia-gated) in the daily digest's manual-review list.
+# Unset = the digest still lists the books, just without links.
+# COLOPHON_BOOKSTORE_URL=https://books.example.com
 
 # --- Metadata provider (Hardcover) ---
 COLOPHON_HARDCOVER_KEY=
@@ -23,6 +27,11 @@ ANTHROPIC_API_KEY=
 # A book is also re-queried automatically once its title/author changes.
 # COLOPHON_RESOLVE_RETRY_DAYS=0
 
+# `enrich`: mark a bare (no-hcid) watch-import "stuck" after N failed sweeps — drop
+# it from the sweep (stops the churn) and surface it once in the daily digest.
+# Default 6 (~3h at the deployed 30-min cadence).
+# COLOPHON_ENRICH_STUCK_AFTER=6
+
 # --- State (defaults are repo-relative; set absolute paths for a service) ---
 # COLOPHON_DB=/var/lib/colophon/colophon.db
 # COLOPHON_REPORTS=/var/lib/colophon/reports

CHANGELOG.md

@@ -7,6 +7,17 @@ All notable changes to this project are documented here. The format is based on
 ## [Unreleased]
 
 ### Added
+- **`enrich` — seed bare imports + remember the unresolvable.** Watch-imports arrive with
+  no provider id and the server never auto-enriches them, so a frequent sweep submits a
+  `REPLACE_MISSING` refresh for the no-hcid books. New `colophon/enrich.py` + the
+  `colophon enrich` subcommand (dry-run unless `--apply`, precondition-gated). The sweep now
+  has **memory** (new `store.enrich_state` table): a book that never seeds is marked *stuck*
+  after `COLOPHON_ENRICH_STUCK_AFTER` failed sweeps (default 6), dropped from the sweep — so
+  it stops being re-poked every cycle — and surfaced **once** in the `maintain` digest with a
+  UI deep-link (`COLOPHON_BOOKSTORE_URL`, route `/book/<id>`) for the human to delete or keep
+  (silence = keep). `maintain` marks a book reported only after a successful email on an
+  `--apply` run. Replaces the memory-less external enrich loop that re-refreshed the same
+  unmatchable books indefinitely.
 - **EPUB inspection for low-confidence mis-seeds** — when `resolve` lands below the
   confidence gate (or finds no match), it now inspects the book's own EPUB and
   re-adjudicates with two extra signals: the OPF `dc:identifier` ISBN (a real ISBN that

README.md

@@ -29,6 +29,10 @@ lock pins the edition, so a healed book *stays* healed (set-once, no oscillation
 
 - **Backfill** — books with a correct provider id but a broken/missing ISBN → set the
   canonical ISBN, lock, refresh.
+- **Enrich** — bare imports that arrive with *no* provider id → submit a missing-only
+  refresh so they seed. Books that can never match (no/odd ISBN, self-published editions
+  the provider lacks) are remembered after a few failed sweeps, dropped from the sweep, and
+  surfaced **once** in the daily summary with a link — delete or keep, your call.
 - **Resolve** — mis-identified books (title disagrees with the matched work). Searches
   the metadata provider for candidates and asks an LLM to pick the *same work* — but only
   from the retrieved set, validated against it (the model can never invent an id), and
@@ -83,6 +87,8 @@ Copy `.env.example` to `.env` and fill it in (or export the variables):
 | `COLOPHON_DB`, `COLOPHON_REPORTS` | where to write the changelog + reports |
 | `COLOPHON_RESOLVE_RETRY_DAYS` | re-query a cached-unresolvable mis-seed after N days (default `0` = never) |
 | `COLOPHON_BOOKS_ROOT` | host path of the library root; set it to let `resolve` inspect a below-threshold book's own EPUB (OPF ISBN + colophon). Unset ⇒ feature off |
+| `COLOPHON_BOOKSTORE_URL` | public UI base; set it to deep-link each stuck book in the digest (route `/book/<id>`). Unset ⇒ listed without links |
+| `COLOPHON_ENRICH_STUCK_AFTER` | mark a bare import stuck after N failed `enrich` sweeps (default `6`) |
 | `SMTP_*`, `OVERSIGHT_TO` | oversight + `maintain --email` summaries |
 
 ## Usage
@@ -91,6 +97,7 @@ Copy `.env.example` to `.env` and fill it in (or export the variables):
 python -m colophon.cli precheck                 # assert the files-never-touched preconditions
 python -m colophon.cli backfill                 # survey + propose (dry-run)
 python -m colophon.cli backfill --apply         # heal broken ISBNs
+python -m colophon.cli enrich --apply           # seed bare no-id imports; remember the unresolvable
 python -m colophon.cli resolve --apply          # LLM-resolve mis-identified books (>= 0.9 conf)
 python -m colophon.cli resolve --force          # re-query the cached-unresolvable mis-seeds
 python -m colophon.cli series-audit             # series-number report (read-only)

colophon/cli.py

@@ -125,6 +125,24 @@ def cmd_backfill(args, g, store):
     return 0
 
 
+def cmd_enrich(args, g, store):
+    """Seed bare watch-imports (no Hardcover id) via a REPLACE_MISSING refresh, with
+    memory: a book that never seeds is marked stuck after N failed sweeps — dropped
+    from the sweep (the churn stops) and surfaced once in the daily digest. Dry-run
+    unless --apply."""
+    from . import enrich as E
+    stuck_after = args.stuck_after if args.stuck_after is not None else E.STUCK_AFTER
+    res = E.run_enrich(g, store, apply=args.apply, stuck_after=stuck_after)
+    mode = "APPLIED" if args.apply else "DRY-RUN (no write)"
+    print(f"[{mode}] enrich — {len(res['unseeded'])} un-seeded · "
+          f"{len(res['active'])} {'refreshed' if res['submitted'] else 'to refresh'} · "
+          f"{len(res['stuck'])} stuck (excluded; stuck_after={res['stuck_after']})")
+    if res["stuck"]:
+        print("  stuck — need manual review (surfaced in the daily digest): "
+              + " ".join(str(b) for b in res["stuck"]))
+    return 0
+
+
 def cmd_audit(args, g, store):
     res = run_audit(limit=args.limit)
     report = render_report(res)
@@ -225,6 +243,11 @@ def cmd_maintain(args, g, store):
                             "Check `journalctl -u colophon.service` on the host.\n")
             ok, detail = oversight.send_email(subject, mail)
             print(f"email: {detail}")
+            # Surface each stuck book exactly once — only after it has actually been
+            # emailed, and only on a real (apply) run so dry-run testing doesn't
+            # consume the one-shot report. Silence thereafter = the human keeps it.
+            if ok and args.apply and res and res.get("stuck"):
+                store.enrich_mark_reported([s["book_id"] for s in res["stuck"]])
     return 0 if (res and res["ok"]) else 1
 
 
@@ -262,6 +285,10 @@ def main(argv=None):
     bf = sub.add_parser("backfill", help="survey the library + propose heals (dry-run unless --apply)")
     bf.add_argument("--limit", type=int, default=20)
     bf.add_argument("--apply", action="store_true")
+    en = sub.add_parser("enrich", help="seed bare watch-imports (no hcid) + remember the unresolvable (dry-run unless --apply)")
+    en.add_argument("--apply", action="store_true", help="actually submit the refresh (default: dry-run)")
+    en.add_argument("--stuck-after", type=int, default=None,
+                    help="mark a book stuck after N failed sweeps (default: COLOPHON_ENRICH_STUCK_AFTER or 6)")
     au = sub.add_parser("audit", help="read-only library audit + report (writes nothing)")
     au.add_argument("--limit", type=int, default=None)
     rs = sub.add_parser("resolve", help="Haiku-resolve mis-seeds (propose-only unless --apply)")
@@ -292,8 +319,8 @@ def main(argv=None):
 
     g, store = Grimmory(), Store()
     fn = {"precheck": cmd_precheck, "heal": cmd_heal, "log": cmd_log,
-          "revert": cmd_revert, "backfill": cmd_backfill, "audit": cmd_audit,
-          "resolve": cmd_resolve, "series-audit": cmd_series_audit,
+          "revert": cmd_revert, "backfill": cmd_backfill, "enrich": cmd_enrich,
+          "audit": cmd_audit, "resolve": cmd_resolve, "series-audit": cmd_series_audit,
           "oversight": cmd_oversight, "maintain": cmd_maintain,
           "verify": cmd_verify}[args.cmd]
     try:

colophon/enrich.py

@@ -0,0 +1,54 @@
+"""Phase 1a' — seed bare watch-imports, and remember the ones that never match.
+
+Fresh grabs land via the library *watch* with embedded metadata only; grimmory
+does NOT auto-enrich watch-imports, so a book with no Hardcover id never joins its
+series or drops from the missing list until something nudges it. This submits a
+Hardcover-first REPLACE_MISSING refresh for the un-seeded books — light and
+rate-limit-friendly (it never touches the already-seeded library).
+
+The problem this module adds memory for: a book with no usable ISBN (or a
+self-pub/KDP edition Hardcover does not carry) can NEVER seed, so a memory-less
+sweep re-poked the same books every 30 minutes forever (tens of metadata refreshes
+a day on days with no new books). Here each sweep records its observations; after
+`stuck_after` failed sweeps a book is marked stuck — dropped from the sweep so the
+churn stops, and surfaced ONCE in the daily digest for the human to delete or keep.
+
+Dry-run by default; --apply submits the refresh. The refresh is precondition-gated
+(book files are never touched) just like every other colophon write.
+"""
+import os
+
+from . import grimmory
+from .heal import assert_preconditions
+
+# Mark an un-seeded book stuck after this many failed sweeps. Default 6 ≈ 3h at the
+# deployed 30-min cadence — long enough to ride out a Hardcover 429/outage, short
+# enough to surface same-day. Tunable via the environment.
+STUCK_AFTER = int(os.environ.get("COLOPHON_ENRICH_STUCK_AFTER", "6"))
+
+# Books that still lack a Hardcover id (the bare watch-imports), excluding
+# soft-deleted ones. Locks are irrelevant — an un-seeded book has no identity to
+# protect.
+_UNSEEDED = (
+    "SELECT bm.book_id FROM book_metadata bm JOIN book b ON b.id=bm.book_id "
+    "WHERE (bm.hardcover_book_id IS NULL OR bm.hardcover_book_id='') "
+    "AND (b.deleted IS NULL OR b.deleted=0) ORDER BY bm.book_id;"
+)
+
+
+def unseeded_ids():
+    return [int(x) for x in grimmory._db(_UNSEEDED).split()]
+
+
+def run_enrich(g, store, apply=False, stuck_after=STUCK_AFTER):
+    """Sweep once: record observations, drop the stuck, refresh the rest."""
+    unseeded = unseeded_ids()
+    stuck = set(store.enrich_observe(unseeded, stuck_after))
+    active = [b for b in unseeded if b not in stuck]
+    submitted = False
+    if apply and active:
+        assert_preconditions(g)
+        g.refresh(active, refresh_covers=True, replace_mode="REPLACE_MISSING")
+        submitted = True
+    return {"apply": apply, "unseeded": unseeded, "active": active,
+            "stuck": sorted(stuck), "submitted": submitted, "stuck_after": stuck_after}

colophon/grimmory.py

@@ -23,6 +23,10 @@
 # to enable EPUB inspection in resolve; book_file paths are relative to it. Unset =
 # the feature stays off (the resolver simply never inspects files).
 BOOKS_ROOT = os.environ.get("COLOPHON_BOOKS_ROOT")
+# Public base URL of the grimmory UI (e.g. https://books.example.com). Set this to
+# put a deep-link to each book's page in the daily digest's manual-review list.
+# Unset = the digest lists the books without links. The UI route is /book/<id>.
+BOOKSTORE_URL = os.environ.get("COLOPHON_BOOKSTORE_URL")
 
 
 class GrimmoryError(Exception):
@@ -160,6 +164,38 @@ def book_ids_by_filename(file_name):
     return [int(x) for x in out.split() if x.strip().isdigit()]
 
 
+def book_url(book_id):
+    """Deep-link to the book's page in the grimmory UI (Authelia-gated), or None if
+    COLOPHON_BOOKSTORE_URL is unset. The digest links here; deletion happens via the
+    UI's own button — no destructive link is ever put in an email."""
+    if not BOOKSTORE_URL:
+        return None
+    return f"{BOOKSTORE_URL.rstrip('/')}/book/{int(book_id)}"
+
+
+def briefs(book_ids):
+    """{book_id: {title, isbn, authors}} for the given ids — read-only, for the
+    digest's manual-review list. Empty in, empty out."""
+    ids = [int(b) for b in book_ids]
+    if not ids:
+        return {}
+    csv = ",".join(str(b) for b in ids)
+    out = _db(
+        "SELECT bm.book_id, IFNULL(bm.title,''), IFNULL(bm.isbn_13,''), "
+        "IFNULL((SELECT GROUP_CONCAT(a.name ORDER BY m.sort_order SEPARATOR ', ') "
+        "  FROM book_metadata_author_mapping m JOIN author a ON a.id=m.author_id "
+        "  WHERE m.book_id=bm.book_id),'') "
+        f"FROM book_metadata bm WHERE bm.book_id IN ({csv});"
+    )
+    res = {}
+    for line in out.splitlines():
+        col = line.split("\t")
+        if len(col) < 4:
+            continue
+        res[int(col[0])] = {"title": col[1], "isbn": col[2], "authors": col[3]}
+    return res
+
+
 def snapshot(book_id):
     """Current metadata for a book as a dict, or None if it doesn't exist."""
     bid = int(book_id)

colophon/maintain.py

@@ -12,7 +12,7 @@
 """
 import time
 
-from . import backfill
+from . import backfill, grimmory
 from .heal import assert_preconditions
 from .resolver import run_resolve
 
@@ -42,9 +42,31 @@ def run_maintain(g, store, limit=20, min_conf=0.9, apply=False, force=False):
     except Exception as e:  # noqa: BLE001
         res["ok"] = False
         res["errors"].append(f"resolve: {str(e)[:200]}")
+    # Informational only — surfacing the enrich stuck-list must never fail the run.
+    res["stuck"] = []
+    try:
+        res["stuck"] = _gather_stuck(store)
+    except Exception as e:  # noqa: BLE001
+        res["stuck_error"] = str(e)[:200]
     return res
 
 
+def _gather_stuck(store):
+    """The actionable manual-review list: un-seeded books the enrich sweep gave up
+    on and has not yet surfaced. Decorated with title/author/isbn + a UI deep-link."""
+    rows = store.enrich_stuck_unreported()
+    if not rows:
+        return []
+    briefs = grimmory.briefs([r["book_id"] for r in rows])
+    out = []
+    for r in rows:
+        b = briefs.get(r["book_id"], {})
+        out.append({"book_id": r["book_id"], "title": b.get("title") or "(unknown)",
+                    "authors": b.get("authors") or "", "isbn": b.get("isbn") or "",
+                    "fail_count": r["fail_count"], "url": grimmory.book_url(r["book_id"])})
+    return out
+
+
 def verdict(res):
     if res["aborted"]:
         return "ABORTED"
@@ -110,5 +132,18 @@ def render_summary(res):
     else:
         L.append("Resolve: did not run")
 
+    stuck = res.get("stuck") or []
+    if stuck:
+        L.append("")
+        L.append(f"Unresolvable — manual review ({len(stuck)}): bare imports Hardcover can't "
+                 "match (no/odd ISBN, self-pub).")
+        L.append("  Delete or keep each via its link; listed here ONCE, then silence = keep.")
+        for s in stuck:
+            who = f" — {s['authors']}" if s["authors"] else ""
+            isbn = f" · isbn {s['isbn']}" if s["isbn"] else " · no isbn"
+            L.append(f"  ? book {s['book_id']} {s['title']!r}{who}{isbn} (failed {s['fail_count']}x)")
+            if s.get("url"):
+                L.append(f"      {s['url']}")
+
     L += ["", "Full reports under reports/ on the host."]
     return "\n".join(L) + "\n"

colophon/store.py

@@ -54,6 +54,24 @@ def __init__(self, path=DB_PATH):
                     attempts INTEGER NOT NULL DEFAULT 1
                 )"""
             )
+            # Un-seeded books (no Hardcover id — the bare watch-imports) that the
+            # enrich sweep keeps failing to match. `fail_count` is the number of
+            # sweeps the book has been seen still un-seeded; at the stuck threshold
+            # it is marked `stuck` (dropped from the 30-min sweep so it stops being
+            # re-poked) and surfaced ONCE in the daily digest (`reported`) for the
+            # human to delete or keep. A book that seeds or is deleted drops out of
+            # the un-seeded set and is pruned from this table.
+            c.execute(
+                """CREATE TABLE IF NOT EXISTS enrich_state(
+                    book_id INTEGER PRIMARY KEY,
+                    fail_count INTEGER NOT NULL DEFAULT 0,
+                    first_seen TEXT NOT NULL,
+                    last_seen TEXT NOT NULL,
+                    stuck INTEGER NOT NULL DEFAULT 0,
+                    stuck_since TEXT,
+                    reported INTEGER NOT NULL DEFAULT 0
+                )"""
+            )
 
     def epoch(self):
         """Re-audit epoch. Settled (locked) books are only re-opened when this is
@@ -130,6 +148,53 @@ def skip_clear(self, book_id=None):
                 return c.execute("DELETE FROM resolve_skip").rowcount
             return c.execute("DELETE FROM resolve_skip WHERE book_id=?", (int(book_id),)).rowcount
 
+    # --- enrich memory (don't keep re-poking the unseedable; surface them once) ---
+
+    def enrich_observe(self, unseeded_ids, stuck_after):
+        """Record one enrich sweep. `unseeded_ids` is the full current set of books
+        that still lack a Hardcover id. Each is upserted with fail_count+1; any
+        previously-tracked id NOT in the set has seeded or been deleted, so it is
+        pruned. A book crosses to `stuck` at fail_count >= stuck_after. Returns the
+        set of currently-stuck book_ids."""
+        now = time.strftime("%Y-%m-%dT%H:%M:%S")
+        ids = {int(b) for b in unseeded_ids}
+        with self._conn() as c:
+            tracked = {r["book_id"] for r in c.execute("SELECT book_id FROM enrich_state")}
+            gone = tracked - ids
+            if gone:
+                c.executemany("DELETE FROM enrich_state WHERE book_id=?", [(b,) for b in gone])
+            for b in ids:
+                c.execute(
+                    "INSERT INTO enrich_state(book_id,fail_count,first_seen,last_seen) "
+                    "VALUES(?,1,?,?) ON CONFLICT(book_id) DO UPDATE SET "
+                    "fail_count=fail_count+1, last_seen=excluded.last_seen",
+                    (b, now, now),
+                )
+            c.execute("UPDATE enrich_state SET stuck=1, stuck_since=? "
+                      "WHERE stuck=0 AND fail_count>=?", (now, int(stuck_after)))
+            return [r["book_id"] for r in
+                    c.execute("SELECT book_id FROM enrich_state WHERE stuck=1")]
+
+    def enrich_stuck_ids(self):
+        """All currently-stuck book_ids (excluded from the sweep)."""
+        with self._conn() as c:
+            return [r["book_id"] for r in
+                    c.execute("SELECT book_id FROM enrich_state WHERE stuck=1")]
+
+    def enrich_stuck_unreported(self):
+        """Stuck books not yet surfaced in a digest (the actionable list)."""
+        with self._conn() as c:
+            return [dict(r) for r in c.execute(
+                "SELECT * FROM enrich_state WHERE stuck=1 AND reported=0 ORDER BY book_id")]
+
+    def enrich_mark_reported(self, book_ids):
+        """Mark stuck books as surfaced — they will not appear in a later digest
+        (silence = keep). Returns rows touched."""
+        with self._conn() as c:
+            return c.executemany(
+                "UPDATE enrich_state SET reported=1 WHERE book_id=?",
+                [(int(b),) for b in book_ids]).rowcount
+
     @staticmethod
     def loads(row, field):
         v = row.get(field)