feat(metadata): Nasjonalbiblioteket as an opt-in primary provider (#2979)

[francisrath]

Summary

Adds Nasjonalbiblioteket (the National Library of Norway) as an opt-in primary metadata provider. NB holds Norwegian publications by legal deposit under their original Norwegian titles, where other providers mostly hold the English translations, so a Norwegian library's files match the author's catalogue instead of falling back to weak fuzzy matches.

Closes #2979.

• New internal/metadata/nb package implementing metadata.Provider, plus GetAuthorWorks / GetAuthorWorksSnapshot.
• nb accepted for metadata.primary_provider; nb: foreign IDs route to it.
• Opt-in only: unlike every other provider, NB is not registered as an enricher when something else is primary, so installs that don't select it send NB no traffic.
• Settings → Metadata Profiles selector entry with help text; provider labels and a link to the record's nb.no page for nb: books.

Review follow-ups (from #2979)

#	Request	Change
1	Fragment style	changelog.d/2979-… now uses "Title (#2979). Text."
2	429 / 5xx as provider errors	get wraps providererr.ErrRateLimited for 429 and providererr.ErrUnavailable for 5xx; a 4xx stays a plain error.
3	Skip per-work GetEditions for nb	Skipped, and the work's own editions are used as the evidence instead of leaving it indeterminate as for DNB, since NB works arrive with every edition attached. A work with no editions attached stays indeterminate. If you'd rather match DNB exactly, it's a one-line change.
4	docs/third-party-data.md section	The delivered records are CC0 while the search API Bindery reads states no licence of its own; publisher-written summaries; covers excluded; Sikt authority file under NLOD 2.0, with its attribution line for redistribution.
5	Authority found, search empty → incomplete	Reported as a partial catalogue, never complete and empty.
6	Docs: leaving NB as primary	User guide and architecture notes say NB-linked authors stop syncing until relinked.

The optional gated live test is not included. Its assertions would have to name real books, and I'd rather keep real titles out of the repo; happy to add one asserting structure only (works found, series numbered) if you'd like it.

More from the same requests

Added after review, with no extra requests to NB or Sikt:

Data	Source	Goes into
Narrator	narrator credits (nrt) on audiobook editions	Book.Narrator
Running time	the audiobook record's extent (1 lydfil (11 t, 16 min), 21:34:00, (3 t, 7 min))	Edition/Book.DurationSeconds
Genres	subject genres, minus format terms and Nynorsk twins	Book.Genres
Name variants	the authority record's other name forms (MARC 400)	Author.AlternateNames, kept as aliases only when LatinAliasBinds accepts them

What the NB API supports (investigated, not assumed)

Question	Finding
Endpoint	GET https://api.nb.no/catalog/v1/items; OpenAPI at /catalog/v1/items/api-docs. The deprecated services/search/v2 is unused.
Auth	None for search. The OAuth flow on digitalpreservation.no is for the DPS submission API only.
Search	ISBN via q=isbn:<n> (any media type, so audiobook ISBNs resolve); author search via the api_nameauthor index; an author's works via filter=namecreators:"Last, First" (the author index misses many records, and cre credits count as authorship); title via free q with searchType=FIELD_RESTRICTED_SEARCH (the default also searches OCR'd full text).
Author credits	Only with expand=metadata: people[] with MARC relator roles and a Norwegian authority-file ID.
Author by ID	Not supported by NB search, so the name heading comes from the authority file's public read API (authority.bibsys.no), and catalogue records are kept only when they credit that ID as author, which drops namesakes and translator/narrator credits.
IDs	sesamId on every record (nb:<sesamId>); URNs exist only for digitised items. Authors: nb:author:<authority id>.
Works	NB has no work record. Editions are folded into works by author and title; translations join their original through the cataloguer's original/uniform title, and a volume catalogued under its series' name (<series> : <volume>) takes its own title from the part name.
Series	The name is in the search JSON (mixed with publisher imprint series); the number is in each record's MODS. Only the author's own series is taken: the MODS entry linked to their authority record.
Name index gaps	NB leaves some records out of every name index although they credit the author correctly. A search on each of the author's series names recovers them (one extra request per series).
Covers	NB's image service refuses in-copyright books; the images some records link to are licensed to library catalogues only. No CoverProvider.
Paging	Max page size 100; author catalogues are capped at 20 pages (the creator index counts every translation) and reported partial beyond that.
Rate limits	None published, none in response headers.

Behaviour notes

• Upstream failures (search, authority lookup, a failed series recall) are errors or a partial catalogue, never a complete empty one: the aggregator takes an empty answer from the primary as fact (CSV, Readarr and Goodreads importers skip the #2271 SafeToBind guard and hardcode metadata_provider=openlibrary, so a provider timeout binds authors to the wrong provider #2332).
• ErrProviderNotConfigured: NB needs no credentials, so a client from New() is always usable; the sentinel is returned for a zero-value client only.
• With NB as primary, a non-Norwegian author present in NB binds to NB and gets the Norwegian translations; authors NB doesn't hold fall back to the other providers. NB is a choice for Norwegian-language libraries, as DNB is for German ones, and the help text says so.

How it was verified

• Unit tests use fixtures in the exact shape of NB, authority-file and MODS responses, with an invented author, titles and ISBNs. They cover: author search; ISBN lookup (audiobook); a translation folded into its original; a volume titled by its series; series and number from MODS (imprint series excluded, an unlinked entry of a known series accepted, MODS failure non-fatal); recovery of volumes missing from the name index; a catalogue split by a spacing slip; capped, empty and failed-recall catalogues reported partial; 429/5xx mapping; reconciliation using a work's own editions; not-configured path; malformed IDs rejected before any request.
• Live against the real API, through the aggregator with NB as primary: Norwegian author search resolves to the NB record; catalogues list books under their original titles; an audiobook ISBN resolves to its work with the audiobook edition attached; two long series resolve to one series row each with every position in order, including volumes NB's name index misses; an English-language author absent from NB falls back to OpenLibrary.

Checklist

• Every commit carries a Signed-off-by that matches its author (git commit -s)
• Changelog fragment added as changelog.d/2979-metadata-nasjonalbiblioteket.md
• Tests added or updated
• docs/DEPLOYMENT.md updated if env vars, config, or upgrade path changed (no change needed: no new env var or config)
• Wiki pages under docs/ updated (User Guide, architecture, third-party data)
• No new dependency

Test plan

• make check steps: gofmt, Go version check, build, vet, golangci-lint v2.11.4, govulncheck, make test, web typecheck/lint/build/test, licence drift
• Race detector on the changed packages (internal/metadata/..., internal/models, cmd/..., reconciliation tests in internal/api); the full internal/api and internal/db race shards run in CI

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.62810% with 39 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/metadata/nb/client.go	92.59%	11 Missing and 9 partials ⚠️
internal/metadata/nb/works.go	96.00%	4 Missing and 4 partials ⚠️
internal/metadata/nb/series.go	96.25%	4 Missing and 3 partials ⚠️
internal/metadata/nb/types.go	98.00%	1 Missing ⚠️
web/src/components/AddToLibraryModal.tsx	0.00%	0 Missing and 1 partial ⚠️
web/src/components/AuthorMetadataLinkModal.tsx	0.00%	1 Missing ⚠️
web/src/util/authorMetadata.ts	0.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[vavallee]

Thanks @francisrath, this came back really clean. All six asks are in, the fragment reads right, and the 429/5xx mapping plugs straight into discovery's backoff. Fixtures with invented data are a nice touch too.

Three small things and it's good to go:

1. Edition evidence on a partial catalogue. I like using the work's own editions instead of going indeterminate like DNB. But when the snapshot is partial (over the page cap, or a series recall failed) a work can be missing editions, and right now a short edition still makes it a page count candidate. Could you only use them when snapshot.Complete is true and leave it indeterminate otherwise? Adding that case to your test would be great.
2. Authority record gone. In GetAuthorWorksSnapshot, a 404 or deleted authority record returns an empty catalogue marked complete. Sikt merges duplicate records now and then, and reconciliation would then offer to remove all of that author's wanted books. Please report it as partial, same as your empty search fix.
3. Docs. Please add the CC0 source to the Nasjonalbiblioteket section in third-party-data.md: https://doc.aja.bs.no/hente/bibliografiske-data/mlnb.html

Tiny nit: the GetEditions comment says one record, but it returns the whole work.

No need for the live test, the fixtures cover it. Once these land I'll merge it for v1.41.0, right after the v1.40.1 patch ships. Really nice work!

[francisrath]

Thanks @vavallee, all three are in, plus the nit:

1. Edition evidence on a partial catalogue (fa89d8bd): a work's own editions are used only when snapshot.Complete; on a partial catalogue NB works stay indeterminate. The reconciliation test now runs both a complete and a partial snapshot, and the partial one makes no candidates.
2. Authority record gone (67a7d432): a 404 or a deleted authority record now returns a partial catalogue instead of a complete empty one, covered by a test for each case.
3. Docs (cd8d1e44): the CC0 source is linked in the Nasjonalbiblioteket section of third-party-data.md.

Nit: the GetEditions comment now describes the whole work it returns.

make test, lint, govulncheck and the race detector on the changed packages pass locally. Marking this ready for review.

[francisrath]

@vavallee one more addition, sorry for growing the PR at this stage. While checking the provider against real catalogues we found more useful data in the responses it already fetches, so this needs no extra requests to NB or Sikt:

• Narrator on audiobooks, from the narrator credit on the audiobook editions.
• Running time on audiobooks, parsed from the record's extent (1 lydfil (11 t, 16 min), 21:34:00, or a CD set's (3 t, 7 min)). Records that state none get none.
• Genres, from the cataloguer's subject genres, minus format terms (Lydbøker) and the Nynorsk twin NB lists beside each Bokmål term.
• Author name variants from the authority record, as AlternateNames. The add-author flow keeps only the ones LatinAliasBinds accepts as another spelling of the same name; for one author that was exactly the form without ø, out of 19 variants, so release names without Norwegian letters still match.

All of it stays inside internal/metadata/nb (one commit, a4ddca73), with tests on invented fixtures. make test, lint, govulncheck and the race detector pass locally. If you'd rather review it as a follow-up PR, say so and I'll move it out so this one can merge as it was.

[francisrath]

Two more fixes from testing against a wider set of Norwegian authors:

0d280415: works from the creator index. NB leaves many records that credit an author out of its nameauthor index, and catalogues authorship as the generic cre about as often as aut. For some authors the provider saw only a small fraction of their catalogue. Works now come from namecreators, and cre/creator count as author credits. The authority-ID check still drops translator and narrator credits. The page cap goes from 5 to 20, because the creator index counts every translation.

1321c9b5: series.

• Merging two names: NB sometimes catalogues one series under two names, both linked to the author, which split it in two. Names that one work's records link with the same number are now merged under the name most books carry.
• Language order: since works now include their translations, the series probe could land on a translation's record and name the series in that language. Series now come from Norwegian records first, then Danish and Swedish ones.
• Numbers: a numbered entry is preferred over an unnumbered one.
• Fallback: some series are linked only on a translation's record, so those are used as a last resort. That sometimes gives a Norwegian book a series with a foreign name, but the books are still grouped and numbered, which beats no series. The commit message spells this out.

Validated against 28 Norwegian authors of crime, literary and children's fiction (not committed; it calls the live API):

	Before	After
Errors / partial catalogues	0 / 0	0 / 0
Books with a series	286	310
Books that lost a series compared with a strict own-language-only variant	n/a	0
Books given a foreign-named series by the fallback	n/a	9

New tests cover the merge, the probe order and the fallback ordering.

[francisrath]

Re-requesting review: the earlier requested changes are addressed in the commits since (summaries in the comments above). The non-API race shard on the last run was cancelled at the 30-minute job limit, not by a test failure; could you re-run it? I can't trigger it from the fork.

[vavallee]

Thanks @francisrath, all three asks are in and I checked each one by reverting the fix: both new tests go red without it, so they're doing their job. The language hunk is gone, and it merges cleanly with current main, including the settings and tenancy changes that landed since. Build, vet, the Go tests and the frontend tests are all green on the merged tree.

I also poked at the cre change against the live API, since that's where I'd expect editors or translators to slip in. They don't. Editors come through as edt, red or utg and translators as trl, so cre really is the author. Nice catch on the index gap.

One small docs thing before merge: the CC0 source you linked is Biblioteksentralen's page for the metadata delivery over OAI-PMH and SRU, but Bindery reads the catalog search API, which doesn't publish any terms. Could you narrow that bullet so it says the delivery records are CC0 and the search API states no licence of its own? Same for the "(CC0 metadata)" note at the top of client.go. A line saying the NLOD attribution matters when someone redistributes the data, not for a self hosted install, would help too.

On the race shard: your #2983 is on main now, so merging main into the branch picks up the faster test DB and that shard should finish well inside the limit.

Not blocking: a heavily translated author can now cost a few hundred requests per refresh, so I'll keep an eye on that once it's out. Great work on this, happy to merge it for v1.41.0 once the docs tweak is in.

[francisrath]

Thanks! Both done:

• Docs (3c3e4b18):
  • docs/third-party-data.md now says the CC0 statement covers NB's metadata delivery over OAI-PMH and SRU, and that the search API Bindery reads states no licence of its own.
  • The client.go package comment says the same.
  • The NLOD attribution line is scoped to redistribution, with a note that a self-hosted install showing the names to its own users doesn't redistribute them.
• Main merged in (89c66616), which picks up test(db): migrate OpenMemory's schema once per process and copy it #2983's test DB for the race shard.

I also updated the PR description where it had gone stale: works now come from the creator index, the page cap is 20, and the licence wording matches.

On request volume, agreed. Most of it on a heavily translated author is the creator-index pages plus the per-work MODS probes for series. If it shows up in practice, the probes are the first thing I'd trim.