From 0f3c84391133228b2e2b7f529f0b418317c528cd Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:08:06 -0300 Subject: [PATCH 01/13] feat(metadata): BnF SRU client (#3144) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A read-only client for the Bibliothèque nationale de France catalogue général through its public SRU 1.2 service, asking for UNIMARC (unimarcXchange). Dublin Core was the other candidate: it folds the statement of responsibility into the title, gives the role as text in at least three spellings, and carries no authority links, so it cannot tell the author from the translator reliably. - Books are works folded from edition records: editions sharing a title, or a translation's original title (454, a bracketed original, or the uniform title), group together per author. The work is named after its French edition and its ID is the ARK of the earliest entered French record, so it is stable across new printings. - Authors are BnF person authority records (bnf:author:). Only relator 070, or the 2026 text role "Auteur du texte", is the author; translators, illustrators and adapted-from authors never are. Records BnF has not linked to an authority yet are linked by exact name when one person has that heading. - Only printed text records are read (bib.doctype "a" in the query, and the record's own type checked again), so music and sound recordings never get through. No covers, no Dewey or subject authorities. - An author's catalogue is never reported complete: records created since BnF's June 2026 model change are not linked to their author yet, and are found by a bounded name query. Politeness: one request a second, two in flight, for the whole process, through a pacer in the HTTP transport so retries are paced too. The shared provider loop honours Retry-After and holds the provider on a refusal; a 429 or a local pacing refusal surfaces as ErrRateLimited, a 5xx or network failure as ErrUnavailable. 20 s per request, 4 MiB per body, at most 10 pages for a catalogue and 2 for the name query, 2 name lookups per search. User text is reduced to one quoted CQL term with the special characters removed rather than escaped. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/metadata/bnf/client.go | 649 ++++++++++++++++++ internal/metadata/bnf/client_test.go | 399 +++++++++++ internal/metadata/bnf/cql.go | 51 ++ internal/metadata/bnf/link_test.go | 115 ++++ internal/metadata/bnf/main_test.go | 18 + internal/metadata/bnf/older_records_test.go | 69 ++ internal/metadata/bnf/pacer.go | 126 ++++ internal/metadata/bnf/ratelimit_test.go | 316 +++++++++ internal/metadata/bnf/records.go | 550 +++++++++++++++ internal/metadata/bnf/records_test.go | 270 ++++++++ internal/metadata/bnf/testdata/SOURCE.md | 13 + .../bnf/testdata/author_works_carroll.xml | 562 +++++++++++++++ .../bnf/testdata/authority_carroll.xml | 157 +++++ .../bnf/testdata/authority_search_carroll.xml | 248 +++++++ internal/metadata/bnf/testdata/diagnostic.xml | 16 + internal/metadata/bnf/testdata/empty.xml | 1 + .../metadata/bnf/testdata/isbn_miroir.xml | 148 ++++ .../bnf/testdata/older_records_alice.xml | 371 ++++++++++ .../metadata/bnf/testdata/recent_carroll.xml | 123 ++++ .../metadata/bnf/testdata/record_miroir.xml | 148 ++++ .../metadata/bnf/testdata/search_alice.xml | 330 +++++++++ .../metadata/bnf/testdata/search_outsider.xml | 385 +++++++++++ internal/metadata/bnf/types.go | 136 ++++ internal/metadata/bnf/works.go | 340 +++++++++ 24 files changed, 5541 insertions(+) create mode 100644 internal/metadata/bnf/client.go create mode 100644 internal/metadata/bnf/client_test.go create mode 100644 internal/metadata/bnf/cql.go create mode 100644 internal/metadata/bnf/link_test.go create mode 100644 internal/metadata/bnf/main_test.go create mode 100644 internal/metadata/bnf/older_records_test.go create mode 100644 internal/metadata/bnf/pacer.go create mode 100644 internal/metadata/bnf/ratelimit_test.go create mode 100644 internal/metadata/bnf/records.go create mode 100644 internal/metadata/bnf/records_test.go create mode 100644 internal/metadata/bnf/testdata/SOURCE.md create mode 100644 internal/metadata/bnf/testdata/author_works_carroll.xml create mode 100644 internal/metadata/bnf/testdata/authority_carroll.xml create mode 100644 internal/metadata/bnf/testdata/authority_search_carroll.xml create mode 100644 internal/metadata/bnf/testdata/diagnostic.xml create mode 100644 internal/metadata/bnf/testdata/empty.xml create mode 100644 internal/metadata/bnf/testdata/isbn_miroir.xml create mode 100644 internal/metadata/bnf/testdata/older_records_alice.xml create mode 100644 internal/metadata/bnf/testdata/recent_carroll.xml create mode 100644 internal/metadata/bnf/testdata/record_miroir.xml create mode 100644 internal/metadata/bnf/testdata/search_alice.xml create mode 100644 internal/metadata/bnf/testdata/search_outsider.xml create mode 100644 internal/metadata/bnf/types.go create mode 100644 internal/metadata/bnf/works.go diff --git a/internal/metadata/bnf/client.go b/internal/metadata/bnf/client.go new file mode 100644 index 000000000..5986eee99 --- /dev/null +++ b/internal/metadata/bnf/client.go @@ -0,0 +1,649 @@ +// Package bnf provides a read-only client for the Bibliothèque nationale de +// France (BnF) catalogue général, through its public SRU 1.2 service. No API +// key is required. +// +// Role: opt-in primary. Legal deposit puts nearly every book published in +// France in the BnF catalogue, under its French edition title, where +// OpenLibrary and Hardcover tend to carry a translated book under its +// original English title (#3144). It is wired only when selected as the +// primary provider, so an install that did not choose it never calls BnF. +// +// Endpoint: https://catalogue.bnf.fr/api/SRU (operations explain and +// searchRetrieve). Records are requested as UNIMARC in marcxchange +// (recordSchema=unimarcXchange): unlike BnF's Dublin Core, it gives the +// author's role as a relator code, the authority record behind each name, the +// original title of a translation and the title without its statement of +// responsibility. BnF changed its cataloguing model in June 2026 ("Noemi") +// and is migrating records to the end of 2026; records converted so far carry +// the role as text and no authority link, and parsing accepts both forms. +// Every field is optional: a missing or unknown one is left empty. +// +// Licence: BnF's bibliographic records are under the Licence Ouverte / Open +// Licence 2.0 (Etalab), which asks for credit to the source and the date. See +// docs/third-party-data.md. Bindery fetches no cover image from BnF, Gallica +// or data.bnf.fr, and never queries the Dewey classification authority +// records. +package bnf + +import ( + "bufio" + "context" + "encoding/xml" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "strconv" + "strings" + "time" + + "github.com/vavallee/bindery/internal/httpsec" + "github.com/vavallee/bindery/internal/isbnutil" + "github.com/vavallee/bindery/internal/metadata" + "github.com/vavallee/bindery/internal/metadata/providererr" + "github.com/vavallee/bindery/internal/metadata/providerhttp" + "github.com/vavallee/bindery/internal/models" + "github.com/vavallee/bindery/internal/textutil" +) + +const ( + sruBase = "https://catalogue.bnf.fr/api/SRU" + providerName = "bnf" + idPrefix = "bnf:" + authorPrefix = "bnf:author:" + recordSchema = "unimarcXchange" + + // searchPageSize is the records asked for by a search: one request, as + // the aggregator gives each provider a few seconds per search. + searchPageSize = 30 + // authorSearchSize is the person records asked for by an author search. + authorSearchSize = 20 + // maxNameLookups bounds the authority lookups linkUnlinked adds to one + // search, at one request each. + maxNameLookups = 2 + // worksPageSize and maxWorksPages bound an author's catalogue walk at + // 1000 records in at most 10 requests. BnF's Explain allows up to 1000 + // records a request but publishes no limit on how often; 100 keeps a + // page well under maxResponseBytes (a UNIMARC record is 2 to 8 KiB). + worksPageSize = 100 + maxWorksPages = 10 + // recentPageSize and maxRecentPages bound the second, name based query + // for records BnF has not yet linked to the author (see + // GetAuthorWorksSnapshot). + recentPageSize = 100 + maxRecentPages = 2 + // noemiCutover is when BnF started cataloguing in its new model. Records + // created since may lack the authority link author2bib follows. + noemiCutover = "20260601" + + // maxResponseBytes caps the body handed to the XML decoder. A body cut + // off here fails to decode and is reported as an error, never read as a + // short result. + maxResponseBytes = 4 << 20 + // requestTimeout bounds one request, pacing wait included. + requestTimeout = 20 * time.Second +) + +// defaultGate holds every BnF request in the process while BnF is refusing +// (HTTP 429 or 503 with Retry-After), shared like defaultPacer. +var defaultGate = providerhttp.NewGate() + +// Client implements metadata.Provider for the BnF catalogue. +type Client struct { + http *http.Client + // gate holds requests while BnF is refusing. Nil (a test Client) holds + // each call on its own. + gate *providerhttp.Gate + // base is the SRU endpoint; empty means sruBase. Tests point it at a + // fake server. + base string +} + +// New creates a BnF client. Every client shares one request pacer and one +// refusal gate, so the limits in pacer.go hold for the whole process. +func New() *Client { + return &Client{ + http: &http.Client{ + Timeout: requestTimeout, + Transport: &pacedTransport{base: httpsec.DefaultProxyTransport(), pacer: defaultPacer}, + }, + gate: defaultGate, + } +} + +func (c *Client) Name() string { return providerName } + +func (c *Client) ready() error { + if c == nil || c.http == nil { + return metadata.ErrProviderNotConfigured + } + return nil +} + +// SearchAuthors finds people in the BnF authority file whose authorised +// name heading matches every word of query. Each is keyed by authority +// record, so two people sharing a name stay apart. A person who matches only +// through a variant (a pseudonym, a transliteration) is left out, so +// "carroll lewis" offers Lewis Carroll and not the singer Johnny Carroll, +// whose variant is "Carroll, Johnny Lewis". +func (c *Client) SearchAuthors(ctx context.Context, query string) ([]models.Author, error) { + if err := c.ready(); err != nil { + return nil, err + } + term := cqlTerm(query) + if term == "" { + return nil, nil + } + words := strings.Fields(strings.Trim(term, `"`)) + auths, err := c.searchAuthorities(ctx, term, authorSearchSize) + if err != nil { + return nil, fmt.Errorf("bnf search authors: %w", err) + } + authors := []models.Author{} + for _, a := range auths { + if !nameMatches(a.heading, words) { + continue + } + author := a.heading.toAuthor() + author.AlternateNames = a.variants + authors = append(authors, author) + } + return authors, nil +} + +// SearchBooks searches every field of the book records (title, names, +// publisher) and folds the matching editions into works. A query that is +// nothing but an ISBN is answered by GetBookByISBN. +func (c *Client) SearchBooks(ctx context.Context, query string) ([]models.Book, error) { + if err := c.ready(); err != nil { + return nil, err + } + query = strings.TrimSpace(query) + bare := query + if len(bare) > 5 && strings.EqualFold(bare[:5], "isbn:") { + bare = strings.TrimSpace(bare[5:]) + } + if isbn13, isbn10 := isbnutil.Extract(bare); bare != "" && isbnutil.Normalize(bare) == firstNonEmpty(isbn13, isbn10) { + b, err := c.GetBookByISBN(ctx, bare) + if err != nil || b == nil { + return nil, err + } + return []models.Book{*b}, nil + } + term := cqlTerm(query) + if term == "" { + return nil, nil + } + page, err := c.searchRetrieve(ctx, "bib.anywhere all "+term+bookFilter, 1, searchPageSize) + if err != nil { + return nil, fmt.Errorf("bnf search books: %w", err) + } + return groupWorks(c.linkUnlinked(ctx, parseRecords(page)), ""), nil +} + +// linkUnlinked gives author credits that carry no authority link one, by +// looking their exact name up in the authority file. BnF's records converted +// to its 2026 model, and its sparse publisher supplied records, name the +// author without a link, and a book whose author Bindery cannot identify +// cannot be added. A name is linked only when exactly one person record has +// that heading; a name shared by two people stays unlinked. Credits whose +// name is linked elsewhere in the same results are linked from there first +// (linkByName), and at most maxNameLookups names are looked up, so a search +// costs at most that many extra requests. Best effort: a failed lookup +// leaves the credit as it was. +func (c *Client) linkUnlinked(ctx context.Context, eds []edition) []edition { + eds = linkByName(eds) + resolved := make(map[string]string) + var names []credit + for _, ed := range eds { + for _, cr := range ed.credits { + if !cr.author || cr.authorityID != "" { + continue + } + key := textutil.FoldForSearch(cr.sortName()) + if _, ok := resolved[key]; ok { + continue + } + resolved[key] = "" + names = append(names, cr) + } + } + if len(names) == 0 { + return eds + } + if len(names) > maxNameLookups { + names = names[:maxNameLookups] + } + for _, name := range names { + term := cqlTerm(name.surname + " " + name.forename) + if term == "" { + continue + } + auths, err := c.searchAuthorities(ctx, term, 5) + if err != nil { + continue + } + id := "" + for _, a := range auths { + if !sameName(a.heading, name) { + continue + } + if id != "" { + id = "" // two people with this heading: ambiguous + break + } + id = a.id + } + resolved[textutil.FoldForSearch(name.sortName())] = id + } + out := make([]edition, len(eds)) + for i, ed := range eds { + out[i] = ed + copied := false + for j, cr := range ed.credits { + if !cr.author || cr.authorityID != "" { + continue + } + if id := resolved[textutil.FoldForSearch(cr.sortName())]; id != "" { + if !copied { + out[i].credits = append([]credit(nil), ed.credits...) + copied = true + } + out[i].credits[j].authorityID = id + } + } + } + return out +} + +// searchAuthorities searches person authority records by name. Only the +// "pep" (person) type is ever asked for: Bindery has no use for, and never +// queries, the Dewey or subject authority files. +func (c *Client) searchAuthorities(ctx context.Context, term string, maxRecords int) ([]authority, error) { + page, err := c.searchRetrieve(ctx, "aut.accesspoint all "+term+` and aut.type any "pep"`, 1, maxRecords) + if err != nil { + return nil, err + } + var out []authority + for _, r := range page.Records { + if a, ok := parseAuthority(r.Data.Record); ok { + out = append(out, a) + } + } + return out, nil +} + +// GetAuthor resolves "bnf:author:" through the BnF +// person authority file. (nil, nil) when there is no such person record. +func (c *Client) GetAuthor(ctx context.Context, foreignID string) (*models.Author, error) { + if err := c.ready(); err != nil { + return nil, err + } + id, err := authorityIDFromForeignID(foreignID) + if err != nil { + return nil, err + } + auth, found, err := c.authority(ctx, id) + if err != nil { + return nil, fmt.Errorf("bnf get author %s: %w", foreignID, err) + } + if !found { + return nil, nil + } + a := auth.heading.toAuthor() + // Bindery keeps only the variants that are another spelling of the same + // name as aliases (LatinAliasBinds), so a pseudonym's real name is not + // adopted as one. + a.AlternateNames = auth.variants + return &a, nil +} + +// GetAuthorWorks returns the author's catalogue, folded into works. +func (c *Client) GetAuthorWorks(ctx context.Context, authorForeignID string) ([]models.Book, error) { + books, _, err := c.GetAuthorWorksSnapshot(ctx, authorForeignID) + return books, err +} + +// GetAuthorWorksSnapshot is GetAuthorWorks plus whether the catalogue is +// complete, and it never is. The catalogue is the book records linked to the +// author's authority record (bib.author2bib), but records BnF created in its +// new model since June 2026 are not linked yet: they are found by a second, +// name based query limited to records created since then, and a name is not +// proof. So catalogue reconciliation must not read a missing work as +// removed, and the snapshot says so. The walk is bounded by maxWorksPages +// and maxRecentPages. +// +// A failed lookup is an error, never an empty catalogue, and an authority +// record that no longer exists returns nothing rather than an empty complete +// catalogue (BnF merges duplicate records). +func (c *Client) GetAuthorWorksSnapshot(ctx context.Context, authorForeignID string) ([]models.Book, bool, error) { + if err := c.ready(); err != nil { + return nil, false, err + } + id, err := authorityIDFromForeignID(authorForeignID) + if err != nil { + return nil, false, err + } + auth, found, err := c.authority(ctx, id) + if err != nil { + return nil, false, fmt.Errorf("bnf get author works %s: %w", authorForeignID, err) + } + if !found { + return nil, false, nil + } + linked, err := c.searchAll(ctx, `bib.author2bib any "`+id+`"`+bookFilter, worksPageSize, maxWorksPages) + if err != nil { + return nil, false, fmt.Errorf("bnf get author works %s: %w", authorForeignID, err) + } + eds := parseRecords(linked...) + + // Records not yet linked to the authority record: the same name, created + // since the cutover. Best effort: a failure here leaves the linked + // catalogue, which is already marked incomplete. + if term := cqlTerm(auth.heading.surname + " " + auth.heading.forename); term != "" { + q := "bib.author all " + term + bookFilter + ` and bib.creationdate >= "` + noemiCutover + `"` + if recent, err := c.searchAll(ctx, q, recentPageSize, maxRecentPages); err == nil { + eds = append(eds, adoptUnlinked(parseRecords(recent...), auth.heading)...) + } + } + return groupWorks(dedupeEditions(eds), id), false, nil +} + +// adoptUnlinked keeps the editions crediting, without an authority link, an +// author whose name is exactly the authority heading's, and links that credit +// to it. Editions already linked to someone are left to author2bib. +func adoptUnlinked(eds []edition, heading credit) []edition { + var out []edition + for _, ed := range eds { + match := -1 + for i, cr := range ed.credits { + if cr.author && cr.authorityID == "" && sameName(cr, heading) { + match = i + break + } + } + if match < 0 { + continue + } + ed.credits = append([]credit(nil), ed.credits...) + ed.credits[match].authorityID = heading.authorityID + out = append(out, ed) + } + return out +} + +func dedupeEditions(eds []edition) []edition { + seen := make(map[string]bool, len(eds)) + out := eds[:0:0] + for _, ed := range eds { + if seen[ed.ark] { + continue + } + seen[ed.ark] = true + out = append(out, ed) + } + return out +} + +// GetBook fetches the record "bnf:" and returns the work it belongs +// to: the record plus the other editions of the same author and title, found +// with one more search. When that search fails or the record has no linked +// author, the record alone is returned. +func (c *Client) GetBook(ctx context.Context, foreignID string) (*models.Book, error) { + if err := c.ready(); err != nil { + return nil, err + } + ark, err := arkFromForeignID(foreignID) + if err != nil { + return nil, err + } + page, err := c.searchRetrieve(ctx, `bib.persistentid all "ark:/12148/`+ark+`"`, 1, 2) + if err != nil { + return nil, fmt.Errorf("bnf get book %s: %w", foreignID, err) + } + eds := c.linkUnlinked(ctx, parseRecords(page)) + if len(eds) == 0 { + return nil, nil + } + ed := eds[0] + if b := c.workOf(ctx, ed); b != nil { + b.ForeignID = foreignID + return b, nil + } + books := groupWorks(eds[:1], "") + if len(books) == 0 { + return nil, nil + } + // Keep the requested ID: BnF redirects a merged record's ARK, and + // callers look the book up by the ID they asked for. + books[0].ForeignID = foreignID + return &books[0], nil +} + +// workOf finds the other editions of ed's work: the records linked to the +// same author whose title matches. Best effort; nil when it cannot. +func (c *Client) workOf(ctx context.Context, ed edition) *models.Book { + a := primaryAuthor(ed, "") + if a == nil || a.authorityID == "" { + return nil + } + term := cqlTerm(ed.title) + if term == "" { + return nil + } + q := `bib.author2bib any "` + a.authorityID + `" and bib.title all ` + term + bookFilter + page, err := c.searchRetrieve(ctx, q, 1, worksPageSize) + if err != nil { + return nil + } + books := groupWorks(dedupeEditions(append([]edition{ed}, parseRecords(page)...)), a.authorityID) + want := idPrefix + ed.ark + for i := range books { + for _, e := range books[i].Editions { + if e.ForeignID == want { + return &books[i] + } + } + } + return nil +} + +// GetEditions returns the editions of the work bookForeignID belongs to, as +// GetBook assembles it. +func (c *Client) GetEditions(ctx context.Context, bookForeignID string) ([]models.Edition, error) { + b, err := c.GetBook(ctx, bookForeignID) + if err != nil || b == nil { + return nil, err + } + return b.Editions, nil +} + +// GetBookByISBN looks a book up by ISBN-10, ISBN-13 or EAN (bib.fuzzyIsbn +// matches any of them). +func (c *Client) GetBookByISBN(ctx context.Context, isbn string) (*models.Book, error) { + if err := c.ready(); err != nil { + return nil, err + } + isbn13, isbn10 := isbnutil.Extract(isbn) + want := firstNonEmpty(isbn13, isbn10) + if want == "" { + return nil, nil + } + page, err := c.searchRetrieve(ctx, `bib.fuzzyIsbn all "`+want+`"`+bookFilter, 1, 5) + if err != nil { + return nil, fmt.Errorf("bnf get book by ISBN: %w", err) + } + books := groupWorks(c.linkUnlinked(ctx, parseRecords(page)), "") + if len(books) == 0 { + return nil, nil + } + return &books[0], nil +} + +// authority fetches a person authority record. found is false when BnF has +// no such person record. +func (c *Client) authority(ctx context.Context, id string) (authority, bool, error) { + page, err := c.searchRetrieve(ctx, `aut.recordid all "`+id+`" and aut.type any "pep"`, 1, 2) + if err != nil { + return authority{}, false, err + } + for _, r := range page.Records { + if a, ok := parseAuthority(r.Data.Record); ok && a.id == id { + return a, true, nil + } + } + return authority{}, false, nil +} + +// searchAll pages through a query, at most maxPages pages of pageSize. It +// stops at the end of the results, at an empty page, or at the page cap, +// whichever comes first, so a server whose cursor never ends costs at most +// maxPages requests and pageSize*maxPages records. +func (c *Client) searchAll(ctx context.Context, cql string, pageSize, maxPages int) ([]*sruResponse, error) { + var pages []*sruResponse + start, seen := 1, 0 + for p := 0; p < maxPages; p++ { + page, err := c.searchRetrieve(ctx, cql, start, pageSize) + if err != nil { + return nil, err + } + pages = append(pages, page) + n := len(page.Records) + seen += n + total := page.total() + if n == 0 || (total >= 0 && start+n > total) || seen >= pageSize*maxPages { + break + } + start += n + } + return pages, nil +} + +// searchRetrieve runs one SRU searchRetrieve request. A response with zero +// records is an empty result; an SRU diagnostic is an error. +func (c *Client) searchRetrieve(ctx context.Context, cql string, start, maxRecords int) (*sruResponse, error) { + params := url.Values{ + "version": {"1.2"}, + "operation": {"searchRetrieve"}, + "query": {cql}, + "recordSchema": {recordSchema}, + "maximumRecords": {strconv.Itoa(maxRecords)}, + "startRecord": {strconv.Itoa(start)}, + } + base := c.base + if base == "" { + base = sruBase + } + var out sruResponse + if err := c.get(ctx, base+"?"+params.Encode(), &out); err != nil { + return nil, err + } + if len(out.Diagnostics) > 0 { + d := out.Diagnostics[0] + return nil, fmt.Errorf("bnf SRU diagnostic %s: %s %s", d.URI, d.Message, d.Details) + } + if len(out.Records) > maxRecords { + out.Records = out.Records[:maxRecords] + } + return &out, nil +} + +// get GETs endpoint and decodes the size-limited body into out. +// +// The shared provider request loop retries a 429, 502, 503 or 504 with +// Retry-After (capped) or exponential backoff with jitter, holds every BnF +// request while one is refused, and never retries another 4xx. What outlives +// the retries is marked for the aggregator and scheduled discovery: a 429 or +// a local pacing refusal as providererr.ErrRateLimited, a 5xx or a network +// failure as providererr.ErrUnavailable. +func (c *Client) get(ctx context.Context, endpoint string, out any) error { + resp, err := providerhttp.Do(ctx, c.http, c.gate, providerhttp.Request{ + Provider: providerName, + URL: endpoint, + Header: http.Header{"Accept": {"application/xml, text/xml"}}, + }) + if err != nil { + return classify(err) + } + defer resp.Body.Close() + if err := decodeXML(io.LimitReader(resp.Body, maxResponseBytes), out); err != nil { + return fmt.Errorf("decode SRU response: %w", err) + } + return nil +} + +func classify(err error) error { + var statusErr *providerhttp.StatusError + var heldErr *providerhttp.HeldError + switch { + case errors.Is(err, errPaced): + return fmt.Errorf("%w: %w", providererr.ErrRateLimited, err) + case errors.As(err, &statusErr), errors.As(err, &heldErr), errors.Is(err, context.Canceled): + return err + default: + return fmt.Errorf("%w: %w", providererr.ErrUnavailable, err) + } +} + +// decodeXML decodes r, skipping a UTF-8 byte order mark (BnF's Explain +// response starts with one). +func decodeXML(r io.Reader, out any) error { + br := bufio.NewReader(r) + if b, err := br.Peek(3); err == nil && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF { + _, _ = br.Discard(3) + } + return xml.NewDecoder(br).Decode(out) +} + +func parseRecords(pages ...*sruResponse) []edition { + var out []edition + for _, p := range pages { + if p == nil { + continue + } + for _, r := range p.Records { + if ed, ok := parseRecord(r.Data.Record); ok { + out = append(out, ed) + } + } + } + return out +} + +func authorityIDFromForeignID(foreignID string) (string, error) { + trimmed := strings.TrimSpace(foreignID) + id := strings.TrimPrefix(trimmed, authorPrefix) + if id == trimmed || !authorityIDRe.MatchString(id) { + return "", errors.New("bnf: not an author id: " + strconv.Quote(foreignID)) + } + return id, nil +} + +func arkFromForeignID(foreignID string) (string, error) { + trimmed := strings.TrimSpace(foreignID) + ark := strings.TrimPrefix(trimmed, idPrefix) + if ark == trimmed || !arkNameRe.MatchString(ark) { + return "", errors.New("bnf: not a book id: " + strconv.Quote(foreignID)) + } + return ark, nil +} + +// nameMatches reports whether every query word appears in the credit's name, +// ignoring case and diacritics. +func nameMatches(c credit, words []string) bool { + folded := textutil.FoldForSearch(c.sortName() + " " + c.displayName()) + for _, w := range words { + if !strings.Contains(folded, textutil.FoldForSearch(w)) { + return false + } + } + return true +} + +func firstNonEmpty(a, b string) string { + if a != "" { + return a + } + return b +} diff --git a/internal/metadata/bnf/client_test.go b/internal/metadata/bnf/client_test.go new file mode 100644 index 000000000..8fc3e5e9e --- /dev/null +++ b/internal/metadata/bnf/client_test.go @@ -0,0 +1,399 @@ +package bnf + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "strings" + "sync" + "testing" + + "github.com/vavallee/bindery/internal/metadata" +) + +// The testdata responses are real BnF SRU answers retrieved on 2026-10-11 +// (see testdata/SOURCE.md), with third party summaries replaced. + +func fixture(t *testing.T, name string) []byte { + t.Helper() + b, err := os.ReadFile(filepath.Join("testdata", name)) + if err != nil { + t.Fatal(err) + } + return b +} + +// fixtureServer answers each SRU query with the fixture route picks, and +// records the queries it was sent. +type fixtureServer struct { + *httptest.Server + mu sync.Mutex + queries []string +} + +func newFixtureServer(t *testing.T, route func(v url.Values) string) *fixtureServer { + t.Helper() + fs := &fixtureServer{} + fs.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query().Get("query") + fs.mu.Lock() + fs.queries = append(fs.queries, q) + fs.mu.Unlock() + name := route(r.URL.Query()) + if name == "" { + name = "empty.xml" + } + w.Header().Set("Content-Type", "text/xml;charset=UTF-8") + _, _ = w.Write(fixture(t, name)) + })) + t.Cleanup(fs.Close) + return fs +} + +func (fs *fixtureServer) sent() []string { + fs.mu.Lock() + defer fs.mu.Unlock() + return append([]string(nil), fs.queries...) +} + +func testClient(base string) *Client { + return &Client{http: &http.Client{}, base: base} +} + +// carrollRoute answers like BnF would for the Lewis Carroll fixtures. Only +// the first page of a result has records. +func carrollRoute(v url.Values) string { + q := strings.ToLower(v.Get("query")) + if s := v.Get("startRecord"); s != "" && s != "1" { + return "empty.xml" + } + switch { + case strings.Contains(q, "aut.accesspoint"): + if strings.Contains(q, "carroll") { + return "authority_search_carroll.xml" + } + return "empty.xml" + case strings.Contains(q, "aut.recordid"): + if strings.Contains(q, `"11885918"`) { + return "authority_carroll.xml" + } + return "empty.xml" + case strings.Contains(q, "bib.creationdate"): + return "recent_carroll.xml" + case strings.Contains(q, "bib.author2bib") && strings.Contains(q, "bib.title"): + return "isbn_miroir.xml" + case strings.Contains(q, "bib.author2bib"): + return "author_works_carroll.xml" + case strings.Contains(q, "bib.fuzzyisbn"): + return "isbn_miroir.xml" + case strings.Contains(q, "bib.persistentid"): + return "record_miroir.xml" + case strings.Contains(q, "outsider"): + return "search_outsider.xml" + } + return "search_alice.xml" +} + +func TestSearchBooks_FrenchEditionTitleTranslatorIsNotAuthor(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + books, err := testClient(fs.URL).SearchBooks(context.Background(), "alice pays merveilles carroll") + if err != nil { + t.Fatal(err) + } + if len(books) == 0 { + t.Fatal("no books") + } + got := books[0] + if got.Title != "Alice au pays des merveilles" { + t.Errorf("title = %q, want the French edition title", got.Title) + } + if got.Language != "fre" { + t.Errorf("language = %q, want fre", got.Language) + } + if got.Author == nil || got.Author.Name != "Lewis Carroll" || got.Author.ForeignID != "bnf:author:11885918" { + t.Errorf("author = %+v, want Lewis Carroll linked to bnf:author:11885918", got.Author) + } + if got.ImageURL != "" { + t.Errorf("ImageURL = %q, want none: BnF covers are never used", got.ImageURL) + } + if !strings.HasPrefix(got.ForeignID, "bnf:cb") { + t.Errorf("ForeignID = %q, want the record's ARK", got.ForeignID) + } + for _, b := range books { + if b.Author != nil && strings.Contains(b.Author.Name, "Bué") { + t.Errorf("translator became the author of %q", b.Title) + } + } + q := fs.sent()[0] + if !strings.HasPrefix(q, `bib.anywhere all "alice pays merveilles carroll"`) || !strings.HasSuffix(q, ` and bib.doctype any "a"`) { + t.Errorf("query = %q, want an anywhere search restricted to printed text", q) + } +} + +// The 2026 converted record (no authority links, roles as text) joins the +// work of the linked records with the same title instead of becoming a +// second, authorless book. +func TestSearchBooks_UnlinkedRecordJoinsItsWork(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + books, err := testClient(fs.URL).SearchBooks(context.Background(), "alice") + if err != nil { + t.Fatal(err) + } + var alice []string + for _, b := range books { + if b.Title == "Alice au pays des merveilles" { + alice = append(alice, b.ForeignID) + if b.Author == nil || b.Author.ForeignID != "bnf:author:11885918" { + t.Errorf("%s author = %+v, want the linked Lewis Carroll", b.ForeignID, b.Author) + } + } + } + if len(alice) != 1 { + t.Errorf("works titled Alice au pays des merveilles = %v, want one", alice) + } +} + +func TestSearchBooks_OriginalTitleAndElision(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + books, err := testClient(fs.URL).SearchBooks(context.Background(), "L'Outsider") + if err != nil { + t.Fatal(err) + } + if len(books) != 1 { + t.Fatalf("books = %d, want the editions folded into one work", len(books)) + } + b := books[0] + if b.Title != "L'outsider" || b.OriginalTitle != "The outsider" { + t.Errorf("title = %q original = %q, want L'outsider / The outsider", b.Title, b.OriginalTitle) + } + if b.Author == nil || b.Author.Name != "Stephen King" { + t.Errorf("author = %+v, want Stephen King (Jean Esch is the translator)", b.Author) + } + if len(b.Editions) != 3 { + t.Errorf("editions = %d, want 3", len(b.Editions)) + } + // The earliest entered French record represents the work. + if b.ForeignID != "bnf:cb456640410" { + t.Errorf("ForeignID = %q, want bnf:cb456640410 (entered 2019-02-06)", b.ForeignID) + } + if b.Description == "" { + t.Error("publisher summary dropped, want it kept as the description") + } + if q := fs.sent()[0]; !strings.Contains(q, `"L'Outsider"`) { + t.Errorf("query = %q, want the elided article kept", q) + } +} + +func TestGetBookByISBN(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + b, err := testClient(fs.URL).GetBookByISBN(context.Background(), "2-07-062897-3") + if err != nil { + t.Fatal(err) + } + if b == nil || b.Title != "Ce qu'Alice trouva de l'autre côté du miroir" { + t.Fatalf("book = %+v", b) + } + ed := b.Editions[0] + if ed.ISBN13 == nil || *ed.ISBN13 != "9782070628971" || ed.Publisher != "Gallimard-Jeunesse" || ed.NumPages == nil || *ed.NumPages != 185 { + t.Errorf("edition = %+v", ed) + } + if b.ReleaseDate == nil || b.ReleaseDate.Year() != 2009 { + t.Errorf("release date = %v, want 2009", b.ReleaseDate) + } + // Electre's summary is a third party's text, not BnF data. + if b.Description != "" { + t.Errorf("description = %q, want the Electre summary left out", b.Description) + } + if q := fs.sent()[0]; q != `bib.fuzzyIsbn all "2070628973" and bib.doctype any "a"` { + t.Errorf("query = %q", q) + } +} + +func TestSearchBooks_ISBNQueryUsesISBNLookup(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + books, err := testClient(fs.URL).SearchBooks(context.Background(), "isbn:9782070628971") + if err != nil || len(books) != 1 { + t.Fatalf("books = %d err = %v", len(books), err) + } + if q := fs.sent()[0]; !strings.HasPrefix(q, "bib.fuzzyIsbn") { + t.Errorf("query = %q, want an ISBN lookup", q) + } +} + +func TestSearchAuthors_AuthorityHeadingsOnly(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + authors, err := testClient(fs.URL).SearchAuthors(context.Background(), "carroll lewis") + if err != nil { + t.Fatal(err) + } + // Johnny Carroll's record matches the query only through his variant + // "Carroll, Johnny Lewis", so he is not offered. + if len(authors) != 1 || authors[0].ForeignID != "bnf:author:11885918" || authors[0].Name != "Lewis Carroll" || authors[0].SortName != "Carroll, Lewis" { + t.Fatalf("authors = %+v, want only Lewis Carroll", authors) + } + if q := fs.sent()[0]; q != `aut.accesspoint all "carroll lewis" and aut.type any "pep"` { + t.Errorf("query = %q, want a person authority search", q) + } +} + +func TestGetAuthor(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + c := testClient(fs.URL) + a, err := c.GetAuthor(context.Background(), "bnf:author:11885918") + if err != nil || a == nil { + t.Fatalf("author = %+v err = %v", a, err) + } + if a.Name != "Lewis Carroll" || a.ForeignID != "bnf:author:11885918" || a.MetadataProvider != "bnf" { + t.Errorf("author = %+v", a) + } + found := false + for _, n := range a.AlternateNames { + if n == "Charles Lutwidge Dodgson" { + found = true + } + } + if !found { + t.Errorf("alternate names = %v, want the authority record's variants", a.AlternateNames) + } + if q := fs.sent()[0]; q != `aut.recordid all "11885918" and aut.type any "pep"` { + t.Errorf("query = %q, want a person authority lookup", q) + } + + missing, err := c.GetAuthor(context.Background(), "bnf:author:99999999") + if err != nil || missing != nil { + t.Errorf("missing author = %+v, %v; want nil, nil", missing, err) + } + for _, bad := range []string{"bnf:author:1\" or aut.type any \"dew", "nb:author:11885918", "bnf:author:"} { + if _, err := c.GetAuthor(context.Background(), bad); err == nil { + t.Errorf("GetAuthor(%q) accepted a malformed id", bad) + } + } +} + +func TestGetAuthorWorksSnapshot(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + books, complete, err := testClient(fs.URL).GetAuthorWorksSnapshot(context.Background(), "bnf:author:11885918") + if err != nil { + t.Fatal(err) + } + if complete { + t.Error("complete = true; BnF's unlinked 2026 records mean the catalogue is never complete") + } + titles := map[string]bool{} + for _, b := range books { + titles[b.Title] = true + if b.Author == nil || b.Author.ForeignID != "bnf:author:11885918" { + t.Errorf("%q author = %+v", b.Title, b.Author) + } + } + // Magazine issues (461 $x) are not books; Disney's "d'après Lewis + // Carroll" credits him with relator 100, not as author. + for _, unwanted := range []string{"Mille et une histoires", "J'aime lire"} { + if titles[unwanted] { + t.Errorf("%q kept: an issue of a periodical is not a book", unwanted) + } + } + if !titles["Ce qu'Alice trouva de l'autre côté du miroir"] { + t.Errorf("titles = %v, want the authored translation", titles) + } + // The unlinked 2026 record found by name joins the catalogue. + if !titles["Alice au pays des merveilles"] { + t.Errorf("titles = %v, want the 2026 Hatier edition adopted by name", titles) + } + sent := fs.sent() + // The fixture claims 704 records but the fake server's second page is + // empty, which ends the walk. + if len(sent) != 4 { + t.Errorf("requests = %v, want authority, two author2bib pages and the recent name query", sent) + } +} + +func TestGetAuthorWorksSnapshot_MissingAuthorityIsNotAnEmptyCatalogue(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + books, complete, err := testClient(fs.URL).GetAuthorWorksSnapshot(context.Background(), "bnf:author:99999999") + if err != nil || books != nil || complete { + t.Errorf("= %v, %v, %v; want nil, false, nil", books, complete, err) + } + if len(fs.sent()) != 1 { + t.Errorf("requests = %v, want only the authority lookup", fs.sent()) + } +} + +func TestGetBook(t *testing.T) { + fs := newFixtureServer(t, carrollRoute) + c := testClient(fs.URL) + b, err := c.GetBook(context.Background(), "bnf:cb434747912") + if err != nil || b == nil { + t.Fatalf("book = %+v err = %v", b, err) + } + if b.ForeignID != "bnf:cb434747912" || b.Title != "Ce qu'Alice trouva de l'autre côté du miroir" { + t.Errorf("book = %+v", b) + } + sent := fs.sent() + if len(sent) == 0 || sent[0] != `bib.persistentid all "ark:/12148/cb434747912"` { + t.Errorf("queries = %v", sent) + } + for _, bad := range []string{"bnf:cb4347479\" or \"x", "bnf:", "dnb:123", "bnf:../x"} { + if _, err := c.GetBook(context.Background(), bad); err == nil { + t.Errorf("GetBook(%q) accepted a malformed id", bad) + } + } +} + +func TestEmptyResultIsNotAnError(t *testing.T) { + fs := newFixtureServer(t, func(url.Values) string { return "empty.xml" }) + c := testClient(fs.URL) + books, err := c.SearchBooks(context.Background(), "zzqxbinderynothing") + if err != nil || len(books) != 0 { + t.Errorf("SearchBooks = %v, %v; want empty, nil", books, err) + } + b, err := c.GetBookByISBN(context.Background(), "9782070628971") + if err != nil || b != nil { + t.Errorf("GetBookByISBN = %v, %v; want nil, nil", b, err) + } +} + +func TestDiagnosticIsAnError(t *testing.T) { + fs := newFixtureServer(t, func(url.Values) string { return "diagnostic.xml" }) + _, err := testClient(fs.URL).SearchBooks(context.Background(), "x") + if err == nil || !strings.Contains(err.Error(), "Index non supporté") { + t.Errorf("err = %v, want the SRU diagnostic", err) + } +} + +func TestZeroValueClientIsNotConfigured(t *testing.T) { + var c Client + if _, err := c.SearchBooks(context.Background(), "x"); !errors.Is(err, metadata.ErrProviderNotConfigured) { + t.Errorf("err = %v", err) + } +} + +// A malformed or truncated body is a decode error, never a panic or a +// silently short result. +func TestGarbageBodyIsAnError(t *testing.T) { + for _, body := range []string{"", "x"} { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(body)) + })) + _, err := testClient(srv.URL).SearchBooks(context.Background(), "x") + srv.Close() + if err == nil { + t.Errorf("body %q: err = nil, want a decode error", body) + } + } +} + +func TestByteOrderMarkIsSkipped(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write(append([]byte{0xEF, 0xBB, 0xBF}, fixture(t, "isbn_miroir.xml")...)) + })) + defer srv.Close() + b, err := testClient(srv.URL).GetBookByISBN(context.Background(), "9782070628971") + if err != nil || b == nil { + t.Errorf("= %v, %v; want the record despite the BOM", b, err) + } +} diff --git a/internal/metadata/bnf/cql.go b/internal/metadata/bnf/cql.go new file mode 100644 index 000000000..bfef5d200 --- /dev/null +++ b/internal/metadata/bnf/cql.go @@ -0,0 +1,51 @@ +package bnf + +import ( + "strings" + "unicode" +) + +// maxTermRunes bounds a search term. A title longer than this is cut, which +// only makes an "all" search a little looser. +const maxTermRunes = 200 + +// cqlTerm turns user text into one quoted CQL term, or "" when nothing +// searchable is left. +// +// The characters CQL gives a meaning to inside or around a quoted term are +// replaced with a space rather than backslash escaped: the double quote and +// backslash (which end or escape the term), the masking characters *, ? and +// ^, and parentheses, comparison and slash characters for good measure. +// Replacing is safer than escaping because BnF's handling of \" could not be +// confirmed: sent `x\" or bib.title any \"*`, the server echoed the query +// back as `bib.title all "x" or bib.title any "*"`. Searching with "all" or +// "any" matches words, so a dropped punctuation mark costs nothing, and a CQL +// keyword such as "or" inside the quotes is just a word. The apostrophe of +// an elided article ("L'outsider") is kept: it is not special in CQL and BnF +// matches it. +func cqlTerm(s string) string { + var b strings.Builder + n := 0 + for _, r := range s { + if n >= maxTermRunes { + break + } + switch { + case strings.ContainsRune(`"\*?^()<>=/`, r), unicode.IsControl(r): + r = ' ' + } + b.WriteRune(r) + n++ + } + term := strings.Join(strings.Fields(b.String()), " ") + if term == "" { + return "" + } + return `"` + term + `"` +} + +// bookFilter restricts a bibliographic search to printed text (doctype "a"). +// It keeps out sound recordings and printed music, which include the third +// party music records BnF's terms carve out of the open licence, as well as +// maps, images and objects. parseRecord checks each record again. +const bookFilter = ` and bib.doctype any "a"` diff --git a/internal/metadata/bnf/link_test.go b/internal/metadata/bnf/link_test.go new file mode 100644 index 000000000..0673ad56a --- /dev/null +++ b/internal/metadata/bnf/link_test.go @@ -0,0 +1,115 @@ +package bnf + +import ( + "context" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" +) + +func TestLinkByName(t *testing.T) { + unlinked := credit{surname: "Carroll", forename: "Lewis", author: true} + martinA := credit{authorityID: "10000001", surname: "Martin", forename: "Paul", author: true} + martinB := credit{authorityID: "10000002", surname: "Martin", forename: "Paul", author: true} + martin := credit{surname: "Martin", forename: "Paul", author: true} + in := []edition{ + edWith("cb10000001a", "Alice", "", "fre", "", carroll), + edWith("cb10000002b", "Alice", "", "fre", "", unlinked), + edWith("cb10000003c", "Un", "", "fre", "", martinA), + edWith("cb10000004d", "Deux", "", "fre", "", martinB), + edWith("cb10000005f", "Trois", "", "fre", "", martin), + } + out := linkByName(in) + if got := out[1].credits[0].authorityID; got != "11885918" { + t.Errorf("unlinked Lewis Carroll = %q, want the link another edition carries", got) + } + if got := out[4].credits[0].authorityID; got != "" { + t.Errorf("unlinked Paul Martin = %q, want none: two people have that name", got) + } + if in[1].credits[0].authorityID != "" { + t.Error("linkByName modified its input") + } +} + +// A page of unlinked authors costs at most maxNameLookups authority lookups, +// so one search stays within the aggregator's per provider budget. +func TestLinkUnlinked_BoundedLookups(t *testing.T) { + var lookups atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.Contains(r.URL.Query().Get("query"), "aut.accesspoint") { + lookups.Add(1) + _, _ = w.Write([]byte(emptySRU)) + return + } + var b strings.Builder + for i, name := range []string{"Un", "Deux", "Trois", "Quatre", "Cinq"} { + b.WriteString(` a 22 450 ` + + `Livre ` + name + `` + + `Auteur` + name + `Jean` + + ``) + } + _, _ = w.Write([]byte(`5` + b.String() + ``)) + })) + defer srv.Close() + books, err := testClient(srv.URL).SearchBooks(context.Background(), "livre") + if err != nil || len(books) != 5 { + t.Fatalf("books = %d err = %v", len(books), err) + } + if got := lookups.Load(); got != maxNameLookups { + t.Errorf("authority lookups = %d, want the cap of %d", got, maxNameLookups) + } +} + +// An unlinked author is linked through the authority file only when exactly +// one person record has that heading, and a name another result already +// links costs no lookup. +func TestLinkUnlinked(t *testing.T) { + person := func(id, forename string) string { + return ` cx a22 45 ` + + `Martin` + forename + `` + } + book := func(ark, sub3 string) string { + return ` a 22 450 ` + + `Un livre` + + `` + sub3 + `MartinPaulAuteur du texte` + + `` + } + for _, tc := range []struct { + name string + books string + authorities string + want string + lookups int32 + }{ + {"one person has the heading", book("20000001", ""), person("10000001", "Paul") + person("10000002", "Pauline"), "bnf:author:10000001", 1}, + {"two people share the heading", book("20000001", ""), person("10000001", "Paul") + person("10000002", "Paul"), "", 1}, + {"linked on another edition", book("20000001", "") + book("20000002", `10000003`), "", "bnf:author:10000003", 0}, + } { + var lookups atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + records := tc.books + if strings.Contains(r.URL.Query().Get("query"), "aut.accesspoint") { + lookups.Add(1) + records = tc.authorities + } + _, _ = w.Write([]byte(`2` + records + ``)) + })) + books, err := testClient(srv.URL).SearchBooks(context.Background(), "un livre") + srv.Close() + if err != nil || len(books) != 1 { + t.Fatalf("%s: books = %+v err = %v", tc.name, books, err) + } + got := "" + if books[0].Author != nil { + got = books[0].Author.ForeignID + } + if got != tc.want { + t.Errorf("%s: author = %q, want %q", tc.name, got, tc.want) + } + if lookups.Load() != tc.lookups { + t.Errorf("%s: authority lookups = %d, want %d", tc.name, lookups.Load(), tc.lookups) + } + } +} diff --git a/internal/metadata/bnf/main_test.go b/internal/metadata/bnf/main_test.go new file mode 100644 index 000000000..03ccbe047 --- /dev/null +++ b/internal/metadata/bnf/main_test.go @@ -0,0 +1,18 @@ +package bnf + +import ( + "os" + "testing" + "time" + + "github.com/vavallee/bindery/internal/metadata/providerhttp" +) + +// TestMain shortens the shared retry backoff so the tests that exercise a +// refusal or an outage do not spend real seconds waiting it out. +func TestMain(m *testing.M) { + restore := providerhttp.SetBaseDelay(time.Millisecond) + code := m.Run() + restore() + os.Exit(code) +} diff --git a/internal/metadata/bnf/older_records_test.go b/internal/metadata/bnf/older_records_test.go new file mode 100644 index 000000000..a131738f5 --- /dev/null +++ b/internal/metadata/bnf/older_records_test.go @@ -0,0 +1,69 @@ +package bnf + +import ( + "context" + "net/url" + "testing" +) + +// Records BnF converted from its older catalogues hold the whole title +// statement in 200 $a, give the original title in brackets or only as a +// uniform title (500), and credit adaptations with relator 100. +func TestOlderRecords(t *testing.T) { + fs := newFixtureServer(t, func(url.Values) string { return "older_records_alice.xml" }) + page, err := testClient(fs.URL).searchRetrieve(context.Background(), "x", 1, 5) + if err != nil { + t.Fatal(err) + } + eds := parseRecords(page) + if len(eds) != 3 { + t.Fatalf("editions = %d, want 3", len(eds)) + } + byARK := map[string]edition{} + for _, ed := range eds { + byARK[ed.ark] = ed + } + + statement := byARK["cb32941384z"] + if statement.title != "Alice au pays des merveilles" { + t.Errorf("title = %q, want the statement of responsibility cut off", statement.title) + } + if statement.originalTitle != "Alice's adventures in wonderland" { + t.Errorf("original = %q, want the bracketed original", statement.originalTitle) + } + if a := primaryAuthor(statement, ""); a == nil || a.displayName() != "Lewis Carroll" { + t.Errorf("author = %+v", a) + } + + adaptation := byARK["cb31911075d"] + if adaptation.originalTitle != "Alice's adventures in Wonderland" { + t.Errorf("original = %q, want the uniform title", adaptation.originalTitle) + } + if a := primaryAuthor(adaptation, ""); a != nil { + t.Errorf("author = %+v, want none: relator 100 is the author adapted from", a) + } + + // Relator 330 is a dubious attribution, not the author. + if a := primaryAuthor(byARK["cb34614049g"], ""); a != nil { + t.Errorf("author = %+v, want none for relator 330", a) + } +} + +func TestSplitTitleStatement(t *testing.T) { + cases := []struct{ in, title, original string }{ + {"L'outsider", "L'outsider", ""}, + {`Alice au pays des merveilles ["Alice's adventures in wonderland"]. Traduction de Henriette Rouillard`, "Alice au pays des merveilles", "Alice's adventures in wonderland"}, + {"Alice au pays des merveilles. Traduit par Françoise Durane", "Alice au pays des merveilles", ""}, + {"Alice au pays des merveilles , par Lewis Carroll. Adaptation Gisèle Vallerey", "Alice au pays des merveilles", ""}, + {"Alice au Pays des Merveilles , d'après Lewis Carroll. Par Walt Disney", "Alice au Pays des Merveilles", ""}, + {"Le Petit prince / Antoine de Saint-Exupéry", "Le Petit prince", ""}, + {"Vingt mille lieues sous les mers", "Vingt mille lieues sous les mers", ""}, + {"M. Ripois et la Némésis", "M. Ripois et la Némésis", ""}, + } + for _, tc := range cases { + title, original := splitTitleStatement(tc.in) + if title != tc.title || original != tc.original { + t.Errorf("splitTitleStatement(%q) = %q, %q; want %q, %q", tc.in, title, original, tc.title, tc.original) + } + } +} diff --git a/internal/metadata/bnf/pacer.go b/internal/metadata/bnf/pacer.go new file mode 100644 index 000000000..d977b5a5a --- /dev/null +++ b/internal/metadata/bnf/pacer.go @@ -0,0 +1,126 @@ +package bnf + +import ( + "context" + "errors" + "io" + "net/http" + "sync" + "time" +) + +// BnF publishes no rate limit for its SRU service, and every Bindery install +// that picks this provider is an independent client of it. So the pace is +// set conservatively here rather than discovered from refusals: +// +// - requestInterval: at most one request starts per second, with no burst. +// A search is one request and an author's catalogue a dozen, so a person +// using the app never waits long, while a background refresh of a whole +// library stays at the pace of a patient human clicking through the +// catalogue. +// - maxConcurrent: at most two requests in flight. Two, not one, so a +// search typed while an author refresh walks its pages is not stuck +// behind the whole walk; the start spacing above still applies to both. +// +// One pacer is shared by every client New returns, so the limits hold for +// the process however many clients are built. Retries made by the shared +// provider request loop go through it too, because it sits in the HTTP +// transport. +const ( + requestInterval = time.Second + maxConcurrent = 2 +) + +// errPaced reports that waiting for a request slot would have run past the +// caller's deadline. Failing at once tells the caller BnF was rate limited +// (locally) rather than letting it time out and look like an outage, as the +// Hardcover pacer does. +var errPaced = errors.New("bnf: waiting for the request pacing would pass the caller's deadline") + +var defaultPacer = newPacer(requestInterval, maxConcurrent) + +// pacer spaces request starts by interval and caps concurrent requests. +type pacer struct { + interval time.Duration + slots chan struct{} + + mu sync.Mutex + next time.Time // the earliest start of the next request +} + +func newPacer(interval time.Duration, concurrent int) *pacer { + if concurrent < 1 { + concurrent = 1 + } + return &pacer{interval: interval, slots: make(chan struct{}, concurrent)} +} + +// acquire waits for a concurrency slot and then for this request's start +// time. It returns the func that frees the slot. +func (p *pacer) acquire(ctx context.Context) (func(), error) { + select { + case p.slots <- struct{}{}: + case <-ctx.Done(): + return nil, ctx.Err() + } + var once sync.Once + release := func() { once.Do(func() { <-p.slots }) } + + p.mu.Lock() + start := time.Now() + if p.next.After(start) { + start = p.next + } + if deadline, ok := ctx.Deadline(); ok && start.After(deadline) { + p.mu.Unlock() + release() + return nil, errPaced + } + p.next = start.Add(p.interval) + p.mu.Unlock() + + if wait := time.Until(start); wait > 0 { + t := time.NewTimer(wait) + defer t.Stop() + select { + case <-t.C: + case <-ctx.Done(): + release() + return nil, ctx.Err() + } + } + return release, nil +} + +// pacedTransport runs every request through a pacer. The concurrency slot is +// held until the response body is closed, not just until the headers arrive, +// so a slow download counts as in flight. +type pacedTransport struct { + base http.RoundTripper + pacer *pacer +} + +func (t *pacedTransport) RoundTrip(req *http.Request) (*http.Response, error) { + release, err := t.pacer.acquire(req.Context()) + if err != nil { + return nil, err + } + resp, err := t.base.RoundTrip(req) + if err != nil { + release() + return nil, err + } + resp.Body = &releasingBody{ReadCloser: resp.Body, release: release} + return resp, nil +} + +type releasingBody struct { + io.ReadCloser + release func() +} + +func (b *releasingBody) Close() error { + err := b.ReadCloser.Close() + b.release() + return err +} diff --git a/internal/metadata/bnf/ratelimit_test.go b/internal/metadata/bnf/ratelimit_test.go new file mode 100644 index 000000000..728b90365 --- /dev/null +++ b/internal/metadata/bnf/ratelimit_test.go @@ -0,0 +1,316 @@ +package bnf + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/vavallee/bindery/internal/metadata/providererr" + "github.com/vavallee/bindery/internal/useragent" +) + +func pacedClient(base string, p *pacer, timeout time.Duration) *Client { + return &Client{ + http: &http.Client{Timeout: timeout, Transport: &pacedTransport{base: http.DefaultTransport, pacer: p}}, + base: base, + } +} + +const emptySRU = `0` + +func TestNewClientsShareOnePacerAndGate(t *testing.T) { + a, b := New(), New() + pa, ok := a.http.Transport.(*pacedTransport) + if !ok { + t.Fatalf("transport = %T, want the pacer in front of every request", a.http.Transport) + } + if pb := b.http.Transport.(*pacedTransport); pa.pacer != pb.pacer || pa.pacer != defaultPacer { + t.Error("clients have their own pacers; the limit must hold for the process") + } + if a.gate != b.gate { + t.Error("clients have their own refusal gates") + } + if defaultPacer.interval < time.Second || cap(defaultPacer.slots) > 2 { + t.Errorf("default pacing = %v, %d concurrent; want at most one start a second and two in flight", + defaultPacer.interval, cap(defaultPacer.slots)) + } + if a.http.Timeout <= 0 { + t.Error("no per request timeout") + } +} + +func TestPacer_SpacesRequestStarts(t *testing.T) { + var mu sync.Mutex + var starts []time.Time + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + starts = append(starts, time.Now()) + mu.Unlock() + if ua := r.Header.Get("User-Agent"); ua != useragent.Get() || !strings.Contains(strings.ToLower(ua), "bindery") { + t.Errorf("User-Agent = %q, want Bindery's", ua) + } + _, _ = w.Write([]byte(emptySRU)) + })) + defer srv.Close() + + const interval = 80 * time.Millisecond + c := pacedClient(srv.URL, newPacer(interval, 2), 5*time.Second) + var wg sync.WaitGroup + for i := 0; i < 4; i++ { + wg.Add(1) + go func() { + defer wg.Done() + if _, err := c.SearchBooks(context.Background(), "x"); err != nil { + t.Error(err) + } + }() + } + wg.Wait() + if len(starts) != 4 { + t.Fatalf("requests = %d", len(starts)) + } + for i := 1; i < len(starts); i++ { + // Server side arrival can jitter slightly; allow 15 ms. + if gap := starts[i].Sub(starts[i-1]); gap < interval-15*time.Millisecond { + t.Errorf("gap %d = %v, want at least %v between request starts", i, gap, interval) + } + } +} + +func TestPacer_CapsConcurrentRequests(t *testing.T) { + var inFlight, peak atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + n := inFlight.Add(1) + for { + p := peak.Load() + if n <= p || peak.CompareAndSwap(p, n) { + break + } + } + time.Sleep(40 * time.Millisecond) + inFlight.Add(-1) + _, _ = w.Write([]byte(emptySRU)) + })) + defer srv.Close() + + c := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second) + var wg sync.WaitGroup + for i := 0; i < 6; i++ { + wg.Add(1) + go func() { + defer wg.Done() + _, _ = c.SearchBooks(context.Background(), "x") + }() + } + wg.Wait() + if got := peak.Load(); got > 2 { + t.Errorf("peak concurrent requests = %d, want at most 2", got) + } +} + +// A request that could only start after its caller's deadline is refused at +// once as rate limited, instead of sleeping into a timeout. +func TestPacer_RefusesPastTheDeadlineAsRateLimited(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(emptySRU)) + })) + defer srv.Close() + c := pacedClient(srv.URL, newPacer(time.Hour, 2), 5*time.Second) + if _, err := c.SearchBooks(context.Background(), "first"); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond) + defer cancel() + began := time.Now() + _, err := c.SearchBooks(ctx, "second") + if !errors.Is(err, providererr.ErrRateLimited) { + t.Errorf("err = %v, want ErrRateLimited", err) + } + if time.Since(began) > 150*time.Millisecond { + t.Errorf("took %v, want an immediate refusal", time.Since(began)) + } +} + +func TestRetryAfterIsHonoured(t *testing.T) { + var calls atomic.Int32 + var mu sync.Mutex + var times []time.Time + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + times = append(times, time.Now()) + mu.Unlock() + if calls.Add(1) == 1 { + w.Header().Set("Retry-After", "1") + w.WriteHeader(http.StatusTooManyRequests) + return + } + _, _ = w.Write([]byte(emptySRU)) + })) + defer srv.Close() + c := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second) + if _, err := c.SearchBooks(context.Background(), "x"); err != nil { + t.Fatalf("err = %v, want the retry after the wait to succeed", err) + } + if len(times) != 2 { + t.Fatalf("requests = %d, want 2", len(times)) + } + if gap := times[1].Sub(times[0]); gap < 950*time.Millisecond { + t.Errorf("retried after %v, want Retry-After's 1 s honoured", gap) + } +} + +func TestPersistent429IsRateLimited(t *testing.T) { + var calls atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + w.WriteHeader(http.StatusTooManyRequests) + })) + defer srv.Close() + _, err := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second).SearchBooks(context.Background(), "x") + if !errors.Is(err, providererr.ErrRateLimited) { + t.Errorf("err = %v, want ErrRateLimited", err) + } + if got := calls.Load(); got != 4 { + t.Errorf("requests = %d, want the first and three retries", got) + } +} + +func TestServerErrorsAreUnavailable(t *testing.T) { + for _, code := range []int{http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable} { + var calls atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + w.WriteHeader(code) + })) + _, err := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second).SearchBooks(context.Background(), "x") + srv.Close() + if !errors.Is(err, providererr.ErrUnavailable) || errors.Is(err, providererr.ErrRateLimited) { + t.Errorf("HTTP %d: err = %v, want ErrUnavailable", code, err) + } + if code == http.StatusInternalServerError && calls.Load() != 1 { + t.Errorf("HTTP 500 sent %d requests, want no retry", calls.Load()) + } + } +} + +func TestOther4xxIsNotRetriedOrMarked(t *testing.T) { + var calls atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + w.WriteHeader(http.StatusBadRequest) + })) + defer srv.Close() + _, err := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second).SearchBooks(context.Background(), "x") + if err == nil || errors.Is(err, providererr.ErrUnavailable) || errors.Is(err, providererr.ErrRateLimited) { + t.Errorf("err = %v, want a plain error", err) + } + if calls.Load() != 1 { + t.Errorf("requests = %d, want 1", calls.Load()) + } +} + +func TestNetworkFailureIsUnavailable(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + url := srv.URL + srv.Close() + _, err := pacedClient(url, newPacer(0, 2), 2*time.Second).SearchBooks(context.Background(), "x") + if !errors.Is(err, providererr.ErrUnavailable) { + t.Errorf("err = %v, want ErrUnavailable", err) + } +} + +func TestHungServerHitsTheTimeout(t *testing.T) { + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-release: + } + })) + defer srv.Close() + defer close(release) + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + began := time.Now() + _, err := pacedClient(srv.URL, newPacer(0, 2), 150*time.Millisecond).SearchBooks(ctx, "x") + if err == nil { + t.Fatal("err = nil, want a timeout") + } + if !errors.Is(err, providererr.ErrUnavailable) { + t.Errorf("err = %v, want it marked unavailable", err) + } + if took := time.Since(began); took > 3*time.Second { + t.Errorf("took %v, want the per request timeout to end each attempt", took) + } +} + +// A body past maxResponseBytes is cut off and fails to decode. The padding is +// an XML comment, so without the cap the document would decode as a valid +// empty result. +func TestOversizedBodyIsCutOff(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(`0`)) + })) + defer srv.Close() + _, err := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second).SearchBooks(context.Background(), "x") + if err == nil || !strings.Contains(err.Error(), "decode") { + t.Errorf("err = %v, want the cut off body to fail to decode", err) + } +} + +// A server whose cursor never ends costs at most maxWorksPages requests for +// the linked catalogue and maxRecentPages for the name query. +func TestPageCapStopsARunawayCursor(t *testing.T) { + var author2bib, recent atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query().Get("query") + switch { + case strings.Contains(q, "aut.recordid"): + _, _ = w.Write(fixture(t, "authority_carroll.xml")) + return + case strings.Contains(q, "bib.author2bib"): + author2bib.Add(1) + case strings.Contains(q, "bib.creationdate"): + recent.Add(1) + } + start, _ := strconv.Atoi(r.URL.Query().Get("startRecord")) + // One new book record per page, and a total that never ends. + fmt.Fprintf(w, `99999999`+ + ` nam 22 450 `+ + `Livre %d`+ + `11885918CarrollLewis070`+ + ``, start, start) + })) + defer srv.Close() + books, complete, err := pacedClient(srv.URL, newPacer(0, 2), 5*time.Second).GetAuthorWorksSnapshot(context.Background(), "bnf:author:11885918") + if err != nil { + t.Fatal(err) + } + if complete { + t.Error("complete = true for a capped walk") + } + if got := author2bib.Load(); got != maxWorksPages { + t.Errorf("author2bib requests = %d, want the cap of %d", got, maxWorksPages) + } + if got := recent.Load(); got != maxRecentPages { + t.Errorf("recent requests = %d, want the cap of %d", got, maxRecentPages) + } + if len(books) != maxWorksPages { + t.Errorf("books = %d, want one per page read", len(books)) + } +} diff --git a/internal/metadata/bnf/records.go b/internal/metadata/bnf/records.go new file mode 100644 index 000000000..780d65b68 --- /dev/null +++ b/internal/metadata/bnf/records.go @@ -0,0 +1,550 @@ +package bnf + +import ( + "regexp" + "strconv" + "strings" + "time" + "unicode" + + "github.com/vavallee/bindery/internal/isbnutil" + "github.com/vavallee/bindery/internal/models" +) + +// arkNameRe matches the name part of a BnF catalogue ARK: "cb", eight or nine +// digits and a NOID check character. It is validated before it is put in a +// query and before it becomes a foreign ID. +var arkNameRe = regexp.MustCompile(`^cb[0-9]{8,9}[0-9bcdfghjkmnpqrstvwxz]$`) + +// arkInTextRe finds a catalogue ARK in a record's id attribute or its 003 +// control field ("http://catalogue.bnf.fr/ark:/12148/cb48780493c"). +var arkInTextRe = regexp.MustCompile(`ark:/12148/(cb[0-9a-z]+)`) + +// authorityIDRe matches a BnF authority record number, the value of $3 in a +// UNIMARC 7XX field and of aut.recordid. +var authorityIDRe = regexp.MustCompile(`^[0-9]{8}$`) + +// edition is what Bindery reads from one bibliographic record. BnF catalogues +// manifestations, so one record is one edition; works are assembled from +// them in works.go. +type edition struct { + ark string // "cb48780493c" + title string + originalTitle string // 454 $t, the title of the work this translates + language string // normalised ISO 639-2/B + publisher string + date *time.Time + entered string // 100 $a date entered on file, YYYYMMDD + isbn13 []string + isbn10 []string + pages int + ebook bool + description string + credits []credit +} + +// credit is one name in a UNIMARC 700, 701 or 702 field. +type credit struct { + authorityID string // BnF authority record number, "" when the record has no link + surname string + forename string + author bool +} + +// sortName is the catalogue form, "Carroll, Lewis". +func (c credit) sortName() string { + if c.forename == "" { + return c.surname + } + return c.surname + ", " + c.forename +} + +// displayName is "Lewis Carroll". A forename ending in an elided particle +// ("Jean d'") joins the surname without a space: "Jean d'Ormesson". +func (c credit) displayName() string { + if c.forename == "" { + return c.surname + } + if strings.HasSuffix(c.forename, "'") || strings.HasSuffix(c.forename, "\u2019") { + return c.forename + c.surname + } + return c.forename + " " + c.surname +} + +func (c credit) toAuthor() models.Author { + return models.Author{ + ForeignID: authorPrefix + c.authorityID, + Name: c.displayName(), + SortName: c.sortName(), + MetadataProvider: providerName, + } +} + +// parseRecord turns a bibliographic record into an edition. ok is false for +// anything that is not a printed or electronic text monograph Bindery can +// use: authority records, sound recordings and printed music (the CGU's +// third party music records among them), issues of a periodical, and records +// without an ARK or a title. Every field is optional; a missing one is left +// empty. +func parseRecord(r marcRecord) (edition, bool) { + if !isBookRecord(r) { + return edition{}, false + } + ark := recordARK(r) + if ark == "" { + return edition{}, false + } + title := recordTitle(r) + if title == "" { + return edition{}, false + } + ed := edition{ + ark: ark, + title: title, + originalTitle: originalTitle(r, title), + language: models.NormalizeLanguageCode(r.first("101", "a")), + description: recordDescription(r), + credits: recordCredits(r), + } + if f := r.first("100", "a"); len(f) >= 8 && allDigits(f[:8]) { + ed.entered = f[:8] + } + ed.publisher, ed.date = publication(r) + if ed.date == nil { + // 100 $a positions 9 to 12 hold the first publication date. + if f := r.first("100", "a"); len(f) >= 13 { + ed.date = parseYear(f[9:13]) + } + } + ed.isbn13, ed.isbn10 = recordISBNs(r) + ed.pages = pageCount(r.first("215", "a")) + ed.ebook = isElectronic(r) + return ed, true +} + +// isBookRecord keeps text monographs. The SRU query already asks for +// bib.doctype "a" (printed text); this is the second check, on the record +// itself, because BnF's record format is changing (Noemi, 2026) and a +// response must not be trusted to honour the filter. +// +// - Authority records are never books. +// - Leader position 6 is the type of record: "a" is language material. A +// blank is tolerated because records converted from BnF's new model in +// 2026 leave the leader positions empty; anything else (music, sound +// recordings, images, maps) is dropped. +// - An RDA content type (181 $c), when present, must include "txt". +// - A record linked to a periodical by ISSN (461 $x) is an issue of a +// magazine, not a book. +func isBookRecord(r marcRecord) bool { + if r.Type != "" && !strings.EqualFold(r.Type, "Bibliographic") { + return false + } + if len(r.Leader) > 6 { + switch r.Leader[6] { + case 'a', ' ': + default: + return false + } + } + if content := r.all("181", "c"); len(content) > 0 { + text := false + for _, c := range content { + if strings.EqualFold(c, "txt") { + text = true + } + } + if !text { + return false + } + } + for _, df := range r.fields("461") { + if df.first("x") != "" { + return false + } + } + return true +} + +// recordARK returns the ARK name ("cb48780493c") from the record's id +// attribute or its 003 control field, or "" when neither holds a valid one. +func recordARK(r marcRecord) string { + for _, s := range []string{r.ID, r.controlField("003")} { + if m := arkInTextRe.FindStringSubmatch(s); m != nil && arkNameRe.MatchString(m[1]) { + return m[1] + } + } + return "" +} + +// recordTitle is the title proper (200 $a). A volume catalogued as a part of +// a larger work carries that work's name in $a and its own title in $i, so +// the part title wins: otherwise every volume of a multi-part work would be +// named after the whole. +func recordTitle(r marcRecord) string { + fs := r.fields("200") + if len(fs) == 0 { + return "" + } + if part := cleanTitle(fs[0].first("i")); part != "" { + return part + } + title, _ := splitTitleStatement(fs[0].first("a")) + return title +} + +// originalTitle is the title of the work a translation renders: 454 $t, else +// a bracketed original in an older record's title statement, else the +// uniform title (500 $a) when it differs from the record's own title. BnF's +// 2026 records give a French uniform title for a French edition, which is +// not an original title, so that case is left out. +func originalTitle(r marcRecord, title string) string { + if t := cleanTitle(r.first("454", "t")); t != "" { + return t + } + if fs := r.fields("200"); len(fs) > 0 { + if _, orig := splitTitleStatement(fs[0].first("a")); orig != "" { + return orig + } + } + if t := cleanTitle(r.first("500", "a")); t != "" && titleKey(t) != titleKey(title) { + return t + } + return "" +} + +// statementCutRe finds where a title statement stops being the title, in +// records BnF converted from its older catalogues, whose 200 $a holds the +// whole statement: "Alice au pays des merveilles ["Alice's adventures in +// wonderland"]. Traduction de Henriette Rouillard", "Alice au pays des +// merveilles , par Lewis Carroll. Adaptation Gisèle Vallerey". It matches a +// bracket, a " / " or " , " separator, or a full stop followed by a word that +// opens a statement of responsibility. +var statementCutRe = regexp.MustCompile(`(?i)\s\[|\s/\s|\s,\s|\.\s+(?:traduit|traduction|trad\.|adapt|illustr|préf|texte|version|par\s|d'après|introd|notes?\s|avec\s|dessins|images)`) + +// bracketedOriginalRe reads a bracketed original title: ["Alice in Wonderland"]. +var bracketedOriginalRe = regexp.MustCompile(`\[\s*"([^"\]]+)"\s*\]`) + +// splitTitleStatement returns the title proper of a 200 $a and, when the +// statement carries one in brackets, the original title. A modern record's +// $a is only the title and comes back unchanged. +func splitTitleStatement(s string) (title, original string) { + s = cleanTitle(s) + if m := bracketedOriginalRe.FindStringSubmatch(s); m != nil { + original = cleanTitle(m[1]) + } + if loc := statementCutRe.FindStringIndex(s); loc != nil && loc[0] > 0 { + s = cleanTitle(s[:loc[0]]) + } + return s, original +} + +// cleanTitle strips the non-sorting markers some UNIMARC exports put around a +// leading article (U+0098 and U+009C, C1 control characters like every other +// control character dropped here), collapses whitespace and trims trailing +// ISBD punctuation. An elided article ("L'outsider") is part of the title and +// is kept as is. +func cleanTitle(s string) string { + s = strings.Map(func(r rune) rune { + if unicode.IsControl(r) { + return -1 + } + return r + }, s) + s = strings.Join(strings.Fields(s), " ") + return strings.TrimSpace(strings.TrimRight(s, " /:;,.=")) +} + +// recordDescription returns the record's summary (330 $a) when it is the +// cataloguer's own or the publisher's, and "" otherwise. A summary whose +// source ($2) names a third party, such as the Electre database, is that +// party's text, not BnF data, and is left out. +func recordDescription(r marcRecord) string { + for _, df := range r.fields("330") { + source := strings.ToLower(strings.TrimSpace(df.first("2"))) + if source != "" && source != "éditeur" && source != "editeur" { + continue + } + if text := strings.Join(strings.Fields(df.first("a")), " "); text != "" { + return text + } + } + return "" +} + +// publication reads the publisher and year from 214 (the current field, +// indicator 2 "0" for publication; "3" is the printer) or the older 210. +func publication(r marcRecord) (string, *time.Time) { + for _, df := range r.fields("214") { + if strings.TrimSpace(df.Ind2) != "0" { + continue + } + if pub, date := cleanPublisher(df.first("c")), parseYear(df.first("d")); pub != "" || date != nil { + return pub, date + } + } + for _, df := range r.fields("210") { + if pub, date := cleanPublisher(df.first("c")), parseYear(df.first("d")); pub != "" || date != nil { + return pub, date + } + } + return "", nil +} + +func cleanPublisher(s string) string { + s = strings.Join(strings.Fields(s), " ") + return strings.TrimSpace(strings.Trim(s, " []:;,")) +} + +// recordISBNs collects ISBNs from 010 $a and the EAN in 073 $a, which for a +// book is its ISBN-13. +func recordISBNs(r marcRecord) (isbn13, isbn10 []string) { + seen := make(map[string]bool) + add := func(raw string) { + i13, i10 := isbnutil.Extract(raw) + if i13 != "" && !seen[i13] { + seen[i13] = true + isbn13 = append(isbn13, i13) + } + if i10 != "" && !seen[i10] { + seen[i10] = true + isbn10 = append(isbn10, i10) + } + } + for _, v := range r.all("010", "a") { + add(v) + } + for _, v := range r.all("073", "a") { + if strings.HasPrefix(v, "978") || strings.HasPrefix(v, "979") { + add(v) + } + } + return isbn13, isbn10 +} + +var pagesRe = regexp.MustCompile(`(\d+)\s*p\b`) + +// pageCount reads "185 p.", "1 vol. (569 p.)" or "1 volume 160 p" from the +// extent (215 $a). 0 when none is stated. +func pageCount(extent string) int { + m := pagesRe.FindStringSubmatch(extent) + if m == nil { + return 0 + } + n, err := strconv.Atoi(m[1]) + if err != nil || n <= 0 || n > 20000 { + return 0 + } + return n +} + +// isElectronic reports an e-book record: BnF describes one as "Texte +// électronique" (200 $b) or "ressource dématérialisée" (215 $a). +func isElectronic(r marcRecord) bool { + for _, v := range r.all("200", "b") { + if strings.Contains(strings.ToLower(v), "électronique") { + return true + } + } + return strings.Contains(strings.ToLower(r.first("215", "a")), "dématérialisée") +} + +// recordCredits lists the people in 700 (main responsibility), 701 and 702. +// Corporate bodies (71X) are not people and are not read. +func recordCredits(r marcRecord) []credit { + var out []credit + for _, tag := range []string{"700", "701", "702"} { + for _, df := range r.fields(tag) { + surname := cleanName(df.first("a")) + if surname == "" { + continue + } + c := credit{ + surname: surname, + forename: cleanName(df.first("b")), + author: isAuthorCredit(tag, df), + } + if id := df.first("3"); authorityIDRe.MatchString(id) { + c.authorityID = id + } + out = append(out, c) + } + } + return out +} + +// cleanName trims a name part and drops a role BnF's Dublin Core export +// appends to it ("Lewis (Auteur du texte)", "Lewis. Traducteur"), should the +// same text reach a UNIMARC subfield. +func cleanName(s string) string { + s = strings.Join(strings.Fields(s), " ") + if i := strings.Index(s, " ("); i > 0 && strings.HasSuffix(s, ")") && isRoleText(s[i+2:len(s)-1]) { + s = s[:i] + } + if i := strings.LastIndex(s, ". "); i > 0 && isRoleText(s[i+2:]) { + s = s[:i] + } + return strings.TrimSpace(strings.TrimRight(s, " ,;:")) +} + +// authorRelator is the UNIMARC relator code for an author. Others seen on +// BnF records: 730 translator, 440 illustrator, 340 editor, 100 the author a +// work is adapted from ("d'après"). +const authorRelator = "070" + +// isAuthorCredit reports whether a 7XX field credits the person as author of +// the text. A relator code ($4) decides when there is one. Records converted +// from BnF's 2026 model carry the role as text in $c instead ("Auteur du +// texte", "Traducteur"). With neither, only 700, the main responsibility +// field, counts as the author. +func isAuthorCredit(tag string, df dataField) bool { + if codes := df.all("4"); len(codes) > 0 { + for _, code := range codes { + if code == authorRelator { + return true + } + } + return false + } + if roles := df.all("c"); len(roles) > 0 { + for _, role := range roles { + if isAuthorRole(role) { + return true + } + } + if anyRoleText(roles) { + return false + } + } + return tag == "700" +} + +// authorRoles are the role labels meaning author of the text, lower case. +var authorRoles = map[string]bool{ + "auteur du texte": true, "autrice du texte": true, "auteur": true, "autrice": true, +} + +// otherRoles are labels BnF uses for people who are not the author. The list +// only has to recognise a $c as a role rather than a name addition; any role +// not in authorRoles is not an author. +var otherRoles = map[string]bool{ + "traducteur": true, "traductrice": true, "illustrateur": true, "illustratrice": true, + "préfacier": true, "préfacière": true, "postfacier": true, "éditeur scientifique": true, + "éditrice scientifique": true, "adaptateur": true, "adaptatrice": true, "narrateur": true, + "narratrice": true, "commentateur": true, "commentatrice": true, "auteur adapté": true, + "autrice adaptée": true, "collaborateur": true, "collaboratrice": true, +} + +func roleParts(role string) []string { + var parts []string + for _, p := range strings.Split(strings.ToLower(role), "/") { + if p = strings.TrimSpace(strings.Trim(p, " .()")); p != "" { + parts = append(parts, p) + } + } + return parts +} + +func isAuthorRole(role string) bool { + for _, p := range roleParts(role) { + if authorRoles[p] { + return true + } + } + return false +} + +func isRoleText(s string) bool { + for _, p := range roleParts(s) { + if authorRoles[p] || otherRoles[p] { + return true + } + } + return false +} + +func anyRoleText(roles []string) bool { + for _, r := range roles { + if isRoleText(r) { + return true + } + } + return false +} + +// parseYear reads the first plausible four-digit year in s: "2019", +// "DL 2019", "cop. 2010", "[2025]". +func parseYear(s string) *time.Time { + run := 0 + for i, r := range s { + if r < '0' || r > '9' { + run = 0 + continue + } + run++ + if run == 4 { + year, _ := strconv.Atoi(s[i-3 : i+1]) + if year < 1400 || year > 2100 { + run = 0 + continue + } + t := time.Date(year, 1, 1, 0, 0, 0, 0, time.UTC) + return &t + } + } + return nil +} + +func allDigits(s string) bool { + for _, r := range s { + if r < '0' || r > '9' { + return false + } + } + return s != "" +} + +// authority is what Bindery reads from a BnF person authority record. +type authority struct { + id string + heading credit + variants []string +} + +// parseAuthority reads a person authority record. ok is false for anything +// else: a record that is not an authority, or one without a 200 heading. +// Only personal names are read; Bindery never asks for the Dewey or subject +// authority files. +func parseAuthority(r marcRecord) (authority, bool) { + if !strings.EqualFold(r.Type, "Authority") { + return authority{}, false + } + id := "" + if m := arkInTextRe.FindStringSubmatch(r.ID); m != nil && len(m[1]) == 11 { + id = m[1][2:10] + } + if f := r.controlField("001"); id == "" && strings.HasPrefix(f, "FRBNF") && len(f) >= 13 { + id = f[5:13] + } + if !authorityIDRe.MatchString(id) { + return authority{}, false + } + h := r.fields("200") + if len(h) == 0 { + return authority{}, false + } + heading := credit{authorityID: id, surname: cleanName(h[0].first("a")), forename: cleanName(h[0].first("b")), author: true} + if heading.surname == "" { + return authority{}, false + } + a := authority{id: id, heading: heading} + seen := map[string]bool{heading.displayName(): true} + for _, df := range r.fields("400") { + v := credit{surname: cleanName(df.first("a")), forename: cleanName(df.first("b"))} + if v.surname == "" || seen[v.displayName()] { + continue + } + seen[v.displayName()] = true + a.variants = append(a.variants, v.displayName()) + } + return a, true +} diff --git a/internal/metadata/bnf/records_test.go b/internal/metadata/bnf/records_test.go new file mode 100644 index 000000000..26d8373b9 --- /dev/null +++ b/internal/metadata/bnf/records_test.go @@ -0,0 +1,270 @@ +package bnf + +import ( + "encoding/xml" + "strings" + "testing" +) + +// The records in this file are invented, in the shape of BnF's UNIMARC. + +func record(t *testing.T, leader, body string) marcRecord { + t.Helper() + doc := `` + + `` + leader + `` + body + `` + var r marcRecord + if err := xml.Unmarshal([]byte(doc), &r); err != nil { + t.Fatal(err) + } + return r +} + +func df(tag string, subfields ...string) string { + var b strings.Builder + b.WriteString(``) + for i := 0; i+1 < len(subfields); i += 2 { + b.WriteString(`` + subfields[i+1] + ``) + } + b.WriteString(``) + return b.String() +} + +const bookLeader = " nam 22 450 " + +func TestCredits_Roles(t *testing.T) { + cases := []struct { + name string + field string + author bool + }{ + {"relator 070 is the author", df("700", "a", "King", "b", "Stephen", "4", "070"), true}, + {"relator 730 is the translator", df("702", "a", "Esch", "b", "Jean", "4", "730"), false}, + {"relator 100 is the author adapted from", df("702", "a", "Carroll", "b", "Lewis", "4", "100"), false}, + {"2026 text role: author", df("702", "a", "Carroll", "b", "Lewis", "c", "Auteur du texte"), true}, + {"2026 text role: both genders", df("702", "a", "King", "b", "Stephen", "c", "Auteur du texte / Autrice du texte"), true}, + {"2026 text role: translator", df("702", "a", "Bué", "b", "Henri", "c", "Traducteur"), false}, + {"main entry with no role", df("700", "a", "Hugo", "b", "Victor"), true}, + {"added entry with no role", df("702", "a", "Tenniel", "b", "John"), false}, + {"name addition is not a role", df("700", "a", "Dumas", "b", "Alexandre", "c", "fils"), true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ed, ok := parseRecord(record(t, bookLeader, df("200", "a", "Titre")+tc.field)) + if !ok || len(ed.credits) != 1 { + t.Fatalf("parse = %+v, %v", ed, ok) + } + if ed.credits[0].author != tc.author { + t.Errorf("author = %v, want %v", ed.credits[0].author, tc.author) + } + }) + } +} + +func TestCredits_NamesWithRolesAndElision(t *testing.T) { + ed, ok := parseRecord(record(t, bookLeader, df("200", "a", "Titre")+ + df("700", "a", "Carroll", "b", "Lewis (Auteur du texte)", "3", "11885918")+ + df("702", "a", "Bué", "b", "Henri. Traducteur")+ + df("701", "a", "Ormesson", "b", "Jean d'", "4", "070"))) + if !ok || len(ed.credits) != 3 { + t.Fatalf("parse = %+v, %v", ed, ok) + } + if got := ed.credits[0].displayName(); got != "Lewis Carroll" { + t.Errorf("display = %q, want the role dropped", got) + } + // Credits are read 700, 701, 702 in turn. + if got := ed.credits[2].sortName(); got != "Bué, Henri" { + t.Errorf("sort = %q, want the role dropped", got) + } + if got := ed.credits[1].displayName(); got != "Jean d'Ormesson" { + t.Errorf("display = %q, want the elided particle joined", got) + } + if got := ed.credits[1].sortName(); got != "Ormesson, Jean d'" { + t.Errorf("sort = %q", got) + } +} + +// Only text monographs are books. The CGU carve out the third party music +// records, so sound recordings and printed music must never get through. +func TestParseRecord_DropsWhatIsNotABook(t *testing.T) { + title := df("200", "a", "Titre") + cases := []struct { + name string + rec marcRecord + keep bool + }{ + {"printed text", record(t, bookLeader, title), true}, + {"2026 record with a blank leader", record(t, " a 22 450 ", title+df("181", "c", "txt", "2", "rdacontent")), true}, + {"musical sound recording", record(t, " njm 22 450 ", title), false}, + {"printed music", record(t, " ncm 22 450 ", title), false}, + {"spoken word content type", record(t, " a 22 450 ", title+df("181", "c", "spw", "2", "rdacontent")), false}, + {"periodical issue", record(t, bookLeader, title+df("461", "t", "J'aime lire", "x", "0399-4600", "v", "398")), false}, + {"no title", record(t, bookLeader, df("010", "a", "978-2-07-062897-1")), false}, + } + for _, tc := range cases { + if _, ok := parseRecord(tc.rec); ok != tc.keep { + t.Errorf("%s: kept = %v, want %v", tc.name, ok, tc.keep) + } + } + auth := record(t, bookLeader, title) + auth.Type = "Authority" + if _, ok := parseRecord(auth); ok { + t.Error("an authority record was read as a book") + } + noARK := record(t, bookLeader, title) + noARK.ID = "ark:/12148/../../x" + if _, ok := parseRecord(noARK); ok { + t.Error("a record without a valid ARK was kept") + } +} + +func TestParseRecord_Fields(t *testing.T) { + ed, ok := parseRecord(record(t, bookLeader, + df("100", "a", "20190206d2019 m y0frey50 ba")+ + df("101", "a", "FRE ", "c", "eng")+ + df("200", "a", "\u0098Les \u009cRougon-Macquart", "h", "1", "i", "La fortune des Rougon")+ + `Impr. CPI`+ + df("214", "a", "Paris", "c", "Albin Michel", "d", "DL 2019")+ + df("215", "a", "1 vol. (576 p.)")+ + df("330", "a", "Résumé de l'éditeur.", "2", "éditeur")+ + df("454", "t", "The outsider")+ + df("856", "u", "1239559", "b", "Première de couverture"))) + if !ok { + t.Fatal("not parsed") + } + if ed.title != "La fortune des Rougon" { + t.Errorf("title = %q, want the part title", ed.title) + } + if got := cleanTitle("\u0098Les \u009cmisérables /"); got != "Les misérables" { + t.Errorf("cleanTitle = %q, want the non-sorting markers and ISBD punctuation gone", got) + } + if ed.language != "fre" || ed.publisher != "Albin Michel" || ed.pages != 576 || ed.date == nil || ed.date.Year() != 2019 { + t.Errorf("edition = %+v", ed) + } + if ed.originalTitle != "The outsider" || ed.entered != "20190206" { + t.Errorf("original = %q entered = %q", ed.originalTitle, ed.entered) + } + if ed.description != "Résumé de l'éditeur." { + t.Errorf("description = %q", ed.description) + } + b := buildWork([]edition{ed}, "") + if b.ImageURL != "" || b.Editions[0].ImageURL != "" { + t.Error("a cover URL was taken from the record; BnF covers are never used") + } +} + +func TestRecordDescription_ThirdPartySummaryIsDropped(t *testing.T) { + for source, want := range map[string]string{"Electre": "", "Dilicom": "", "éditeur": "texte", "": "texte"} { + fields := []string{"a", "texte"} + if source != "" { + fields = append(fields, "2", source) + } + ed, _ := parseRecord(record(t, bookLeader, df("200", "a", "Titre")+df("330", fields...))) + if ed.description != want { + t.Errorf("source %q: description = %q, want %q", source, ed.description, want) + } + } +} + +func edWith(ark, title, orig, lang, entered string, credits ...credit) edition { + return edition{ark: ark, title: title, originalTitle: orig, language: lang, entered: entered, credits: credits} +} + +var ( + carroll = credit{authorityID: "11885918", surname: "Carroll", forename: "Lewis", author: true} + bue = credit{authorityID: "12345678", surname: "Bué", forename: "Henri"} +) + +func TestGroupWorks_TranslationsOfOneOriginalAreOneWork(t *testing.T) { + books := groupWorks([]edition{ + edWith("cb10000001a", "Alice au pays des merveilles", "Alice's adventures in Wonderland", "fre", "19900101", carroll), + edWith("cb10000002b", "Aventures d'Alice au pays des merveilles", "Alice's adventures in Wonderland", "fre", "20000101", carroll), + edWith("cb10000003c", "Alice au pays des merveilles", "", "fre", "20260601", carroll), + edWith("cb10000004d", "L’outsider", "", "fre", "20190101", carroll), + edWith("cb10000005f", "L'outsider", "", "fre", "20200101", carroll), + }, "") + if len(books) != 2 { + t.Fatalf("works = %d, want 2", len(books)) + } + if books[0].Title != "Alice au pays des merveilles" || len(books[0].Editions) != 3 || books[0].ForeignID != "bnf:cb10000001a" { + t.Errorf("first work = %q %s with %d editions", books[0].Title, books[0].ForeignID, len(books[0].Editions)) + } + if books[0].OriginalTitle != "Alice's adventures in Wonderland" { + t.Errorf("original title = %q", books[0].OriginalTitle) + } + if len(books[1].Editions) != 2 { + t.Errorf("L'outsider editions = %d, want the two apostrophes grouped", len(books[1].Editions)) + } +} + +func TestGroupWorks_TranslatorIsNeverTheAuthor(t *testing.T) { + unlinked := credit{surname: "Carroll", forename: "Lewis", author: true} + books := groupWorks([]edition{edWith("cb10000001a", "Alice au pays des merveilles", "", "fre", "", unlinked, bue)}, "") + if len(books) != 1 { + t.Fatalf("works = %d", len(books)) + } + if a := books[0].Author; a != nil { + t.Errorf("author = %+v, want none: the only linked name is the translator's", a) + } +} + +func TestGroupWorks_AuthorFilter(t *testing.T) { + adaptation := credit{authorityID: "11885918", surname: "Carroll", forename: "Lewis"} // relator 100 + books := groupWorks([]edition{ + edWith("cb10000001a", "Alice", "", "fre", "", carroll), + edWith("cb10000002b", "Alice (Disney)", "", "fre", "", adaptation), + edWith("cb10000003c", "Le Hobbit", "", "fre", "", bue), + }, "11885918") + if len(books) != 1 || books[0].Title != "Alice" { + t.Errorf("books = %+v, want only the record crediting him as author", books) + } +} + +func TestParseAuthority(t *testing.T) { + r := record(t, " cx a22 45 ", + df("200", "a", "Carroll", "b", "Lewis", "f", "1832-1898")+ + df("400", "a", "Dodgson", "b", "Charles Lutwidge")+ + df("400", "a", "Carroll", "b", "Lewis")) + r.Type = "Authority" + r.ID = "ark:/12148/cb118859183" + a, ok := parseAuthority(r) + if !ok || a.id != "11885918" || a.heading.displayName() != "Lewis Carroll" { + t.Fatalf("authority = %+v, %v", a, ok) + } + if len(a.variants) != 1 || a.variants[0] != "Charles Lutwidge Dodgson" { + t.Errorf("variants = %v, want the heading itself left out", a.variants) + } + r.Type = "Bibliographic" + if _, ok := parseAuthority(r); ok { + t.Error("a bibliographic record was read as an authority") + } +} + +func TestCQLTerm(t *testing.T) { + cases := map[string]string{ + `alice pays merveilles`: `"alice pays merveilles"`, + `x" or bib.title any "*`: `"x or bib.title any"`, + `a\" or "b`: `"a or b"`, + `wild* card? ^anchor`: `"wild card anchor"`, + `(Le petit prince) = x < y > z / w`: `"Le petit prince x y z w"`, + `L'Outsider`: `"L'Outsider"`, + `Pride and Prejudice not Zombies or prox`: `"Pride and Prejudice not Zombies or prox"`, + "tab\tand\nnewline": `"tab and newline"`, + `"*?^\"`: ``, + ` `: ``, + } + for in, want := range cases { + if got := cqlTerm(in); got != want { + t.Errorf("cqlTerm(%q) = %q, want %q", in, got, want) + } + } + long := strings.Repeat("a", 500) + if got := cqlTerm(long); len(got) != maxTermRunes+2 { + t.Errorf("long term length = %d, want capped at %d", len(got), maxTermRunes+2) + } + // Whatever the input, the result holds exactly two quotes: one term. + for in := range cases { + if got := cqlTerm(in); got != "" && strings.Count(got, `"`) != 2 { + t.Errorf("cqlTerm(%q) = %q is not a single quoted term", in, got) + } + } +} diff --git a/internal/metadata/bnf/testdata/SOURCE.md b/internal/metadata/bnf/testdata/SOURCE.md new file mode 100644 index 000000000..e505760db --- /dev/null +++ b/internal/metadata/bnf/testdata/SOURCE.md @@ -0,0 +1,13 @@ +# Source of these fixtures + +The `.xml` files here are responses from the BnF SRU service +(https://catalogue.bnf.fr/api/SRU), retrieved on 2026-10-11, in the +`unimarcXchange` record schema. + +Contains data from the Bibliothèque nationale de France under the Licence +Ouverte / Open Licence 2.0, retrieved 2026-10-11. + +Changes made for Bindery's tests: the text of every summary (UNIMARC field +330 $a) is replaced with a placeholder, because those summaries are the +publisher's or a third party's (Electre) copy and not BnF data under the open +licence. Nothing else is altered. BnF does not endorse Bindery. diff --git a/internal/metadata/bnf/testdata/author_works_carroll.xml b/internal/metadata/bnf/testdata/author_works_carroll.xml new file mode 100644 index 000000000..79c6cc850 --- /dev/null +++ b/internal/metadata/bnf/testdata/author_works_carroll.xml @@ -0,0 +1,562 @@ + +1.2 + +1.2 +bib.author2bib any "11885918" and bib.doctype any "a" + +704 + + +marcxchange +xml + + + nam 22 3 450 +FRBNF439253770000006 +http://catalogue.bnf.fr/ark:/12148/cb43925377g + +ALS +1688285 + + +20130701d2010 m y0frey50 ba + + +fre + + +FR + + +||||z 00||| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Mille et une histoires +117 +Avril 2010 +Texte imprimé + + +2010 + + +1 n° + + +Numéro de : Mille et une histoires = ISSN 1297-0662, 117 + + +37014172 +Mille et une histoires +1297-0662 +117 + + +FR +FR-751131015 +20130701 +AFNOR +FRBNF439253770000006 +intermrc + + +ark:/12148/cb43925377g +1 + +20141215 +20141215 +5.392673 + + + +marcxchange +xml + + + nam 22 3 450 +FRBNF439253660000002 +http://catalogue.bnf.fr/ark:/12148/cb43925366t + +ALS +1688038 + + +20130701d2010 m y0frey50 ba + + +fre + + +FR + + +||||z 00||| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +J'aime lire +398 +Mars 2010 +Texte imprimé + + +2010 + + +1 n° + + +Numéro de : J'aime lire = ISSN 0399-4600, 398 + + +34382070 +J'aime lire +0399-4600 +398 + + +FR +FR-751131015 +20130701 +AFNOR +FRBNF439253660000002 +intermrc + + +ark:/12148/cb43925366t +2 + +20141215 +20141215 +5.3923087 + + + +marcxchange +xml + + + nam 22 450 +FRBNF37159437000000X +http://catalogue.bnf.fr/ark:/12148/cb371594375 + +2-23-000200-7 +rel. +52 F + + +FR +09310664 + + +OPL +015010534 + + +9782230002009 +ALS + + +19930316d1993 a y0frey50 ba + + +fre + + +FR + + +a z 00|a| + + +r + + +01 +i +xxxe +b +xb2e + + +02 +txt +sti +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice au pays des merveilles +Texte imprimé +Walt Disney + + +[Paris] +Disney Hachette éd. +1993 +54-Maxéville +Impr. J. Lamour + + +79 p. +ill. en coul., couv. ill. en coul. +24 cm + + +Disney classique + + +D'après Lewis Carroll + + +34231436 +Disney classiques (Imprimé) +0338-6430 +1992 + + +823 +Cadre de classement de la Bibliographie nationale française + + +11885918 +ISNI000000012137136X +Carroll +Lewis +1832-1898 +100 + + +11868138 +ISNI0000000121496525 +Walt Disney company +070 + + +FR +FR-751131015 +19930316 +AFNOR +FRBNF37159437000000X +intermrc + + +ark:/12148/cb371594375 +3 + +19930316 +20140630 +5.3916383 + + + +marcxchange +xml + + + nam 22 450 +FRBNF434747910000007 +http://catalogue.bnf.fr/ark:/12148/cb434747912 + +978-2-07-062897-1 +Poche +5.90 EUR + + +ALS +1655223 + + +9782070628971 + + +20090707d2009 a y0frey50 ba + + +fre +eng + + +FR + + +||||z 00|y| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Ce qu'Alice trouva de l'autre côté du miroir +Texte imprimé +Lewis Carroll +traduit de l'anglais par Jacques Papy +illustrations de John Tenniel + + +Nouv. présentation + + +Paris +Gallimard-Jeunesse +2009 + + +185 p. +18 x 13 cm + + +Folio junior +Nicolas Balutet +0153-0593 +151 + + +Titre de couv. : De l'autre côté du miroir + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +Electre + + +34226165 +Collection Folio junior +0153-0593 +151 + + +De l'autre côté du miroir + + +11885918 +ISNI000000012137136X +Carroll +Lewis +070 + + +11918569 +ISNI0000000119322075 +Papy +Jacques +730 + + +11926193 +ISNI0000000121302807 +Tenniel +John +440 + + +FR +FR-751131015 +20090707 +AFNOR +FRBNF434747910000007 +intermrc + + +120251 +Première de couverture + + +ark:/12148/cb434747912 +4 + +20140820 +20151221 +5.3904915 + + + +marcxchange +xml + + + nam 22 450 +FRBNF434785490000000 +http://catalogue.bnf.fr/ark:/12148/cb43478549p + +978-2-01-463918-6 +Br. +3.90 EUR + + +ALS +1725225 + + +9782014639186 + + +20110118d2011 a y0frey50 ba + + +fre + + +FR + + +||||z 00|y| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice au pays des merveilles +Texte imprimé +d'après Lewis Carroll +Walt Disney company + + +Paris +Hachette jeunesse-Disney +2011 + + +1 vol. +ill. en coul. + + +Le monde enchanté +1140-180X + + +Trad. de l'anglais + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +Electre + + +34227510 +Le Monde enchanté de Walt Disney +0224-800X + + +11885918 +ISNI000000012137136X +Carroll +Lewis +100 + + +11868138 +ISNI0000000121496525 +Walt Disney company +070 + + +FR +FR-751131015 +20110118 +AFNOR +FRBNF434785490000000 +intermrc + + +118350 +Première de couverture + + +ark:/12148/cb43478549p +5 + +20140820 +20151221 +5.3904915 + + + +6 + diff --git a/internal/metadata/bnf/testdata/authority_carroll.xml b/internal/metadata/bnf/testdata/authority_carroll.xml new file mode 100644 index 000000000..ffe6dea41 --- /dev/null +++ b/internal/metadata/bnf/testdata/authority_carroll.xml @@ -0,0 +1,157 @@ + +1.2 + +1.2 +aut.recordid all "11885918" + +1 + + +marcxchange +xml + + + cx a22 45 +FRBNF118859180 +http://catalogue.bnf.fr/ark:/12148/cb118859183 +20170301 + +000000012137136X +ISNI +20170818 + + +OPL +00251318X +AMA + + +OPL +00035641X +APP + + +OPP +14856271 +20061010 + + +OPP +10889028 +20120518 + + +OPP +14672331 +20120518 + + +OPP +16450529 +20131204 + + +OPP +16782838 +20160804 + + +19840425afrey50 ba0 + + +eng + + +GB + + + 18320127 18980114 + + +a +i + + +010 + + +b + + +2 + + +ba0yba0y +fre + 0 +Carroll +Lewis +1832-1898 + + +Écrivain. - Photographe. - Professeur de mathématiques + + +Daresbury (Royaume-Uni) +Guildford (Royaume-Uni) + + +ba0yba0y +fre + +Dodgson +Charles Lutwidge +1832-1898 + + +e +ba0yba0y +fre + +Holiday +Henry +1832-1898 + + +ba0yba0y +freara + +Kārūl +Lūwīs +1832-1898 + + +16741446 +Saylor +Maurice +1957-.... +[The hunting of the snark] + + +FR +FR-751131015 +20170301 + + +Euclid and his modern rivals / by Charles L. Dodgson, 1879 +Alice au pays des merveilles / Lewis Carroll ; texte français de Michel J. Arnaud, [1932] +Alīs fī bilād al-ʿaǧāʾib / taʾlīf Lūwīs Kārūl ; rusūm Ǧūn Tinyil, cop. 2015 + + +BLNA, 1993-09 : Carroll, Lewis, 1832-1898 et Dodgson, C. L. (Charles Lutwidge), 1832-1898 +GDEL + + +BN Cat. gén. : Dodgson (Charles Lutwidge) + + +ark:/12148/cb118859183 +1 + +19850622 +20170818 +7.180302 + + + + diff --git a/internal/metadata/bnf/testdata/authority_search_carroll.xml b/internal/metadata/bnf/testdata/authority_search_carroll.xml new file mode 100644 index 000000000..e15846dcb --- /dev/null +++ b/internal/metadata/bnf/testdata/authority_search_carroll.xml @@ -0,0 +1,248 @@ + +1.2 + +1.2 +aut.accesspoint all "carroll lewis" and aut.type any "pep" + +2 + + +marcxchange +xml + + + cx a22 45 +FRBNF139769989 +http://catalogue.bnf.fr/ark:/12148/cb139769982 +20240729 + +0000000059338445 +VIAF +20130731 + + +19950602afrey50 ba0 + + +US + + + 19371023 19950218 + + +i + + +1 + + +b + + +2 + + +ba0yba0y +fre + 0 +Carroll +Johnny +1937-1995 + + +Chanteur et guitariste de rockabilly + + +Cleburne (Texas, Etats-Unis) +Godley (Texas, Etats-Unis) + + +ba0yba0y +fre + +Carroll +Johnny Lewis + + +xxxk +Membre de : +17818314 +ba0yba0y +fre +20 +Johnny Carroll and his hot rocks + + +FR +FR-751131015 +20240729 + + +Rock baby, rock it / Johnny Carroll. In : The very best of Sun rockabilly. Charly CPCD82742 [Musique exécutée : audio] + + +LC authorities - https://lccn.loc.gov/no98007589 (2024-07-29) +Jukebox magazine n °92, mai 1995 + + +https://lccn.loc.gov/no98007589 +2024-07-29 + + +ark:/12148/cb139769982 +1 + +19950602 +20240729 +8.962906 + + + +marcxchange +xml + + + cx a22 45 +FRBNF118859180 +http://catalogue.bnf.fr/ark:/12148/cb118859183 +20170301 + +000000012137136X +ISNI +20170818 + + +OPL +00251318X +AMA + + +OPL +00035641X +APP + + +OPP +14856271 +20061010 + + +OPP +10889028 +20120518 + + +OPP +14672331 +20120518 + + +OPP +16450529 +20131204 + + +OPP +16782838 +20160804 + + +19840425afrey50 ba0 + + +eng + + +GB + + + 18320127 18980114 + + +a +i + + +010 + + +b + + +2 + + +ba0yba0y +fre + 0 +Carroll +Lewis +1832-1898 + + +Écrivain. - Photographe. - Professeur de mathématiques + + +Daresbury (Royaume-Uni) +Guildford (Royaume-Uni) + + +ba0yba0y +fre + +Dodgson +Charles Lutwidge +1832-1898 + + +e +ba0yba0y +fre + +Holiday +Henry +1832-1898 + + +ba0yba0y +freara + +Kārūl +Lūwīs +1832-1898 + + +16741446 +Saylor +Maurice +1957-.... +[The hunting of the snark] + + +FR +FR-751131015 +20170301 + + +Euclid and his modern rivals / by Charles L. Dodgson, 1879 +Alice au pays des merveilles / Lewis Carroll ; texte français de Michel J. Arnaud, [1932] +Alīs fī bilād al-ʿaǧāʾib / taʾlīf Lūwīs Kārūl ; rusūm Ǧūn Tinyil, cop. 2015 + + +BLNA, 1993-09 : Carroll, Lewis, 1832-1898 et Dodgson, C. L. (Charles Lutwidge), 1832-1898 +GDEL + + +BN Cat. gén. : Dodgson (Charles Lutwidge) + + +ark:/12148/cb118859183 +2 + +19850622 +20170818 +4.5725117 + + + + diff --git a/internal/metadata/bnf/testdata/diagnostic.xml b/internal/metadata/bnf/testdata/diagnostic.xml new file mode 100644 index 000000000..82030cc2a --- /dev/null +++ b/internal/metadata/bnf/testdata/diagnostic.xml @@ -0,0 +1,16 @@ + + + 1.2 + + 1.2 + bib.nonsense all "x" + + 0 + + + info:srw/diagnostic/1/16 + bib.nonsense + Index non supporté + + + diff --git a/internal/metadata/bnf/testdata/empty.xml b/internal/metadata/bnf/testdata/empty.xml new file mode 100644 index 000000000..21a7b26dd --- /dev/null +++ b/internal/metadata/bnf/testdata/empty.xml @@ -0,0 +1 @@ +1.21.2bib.title all "zzqxbinderynothing" and bib.doctype any "a"0 \ No newline at end of file diff --git a/internal/metadata/bnf/testdata/isbn_miroir.xml b/internal/metadata/bnf/testdata/isbn_miroir.xml new file mode 100644 index 000000000..521397bb4 --- /dev/null +++ b/internal/metadata/bnf/testdata/isbn_miroir.xml @@ -0,0 +1,148 @@ + +1.2 + +1.2 +bib.fuzzyIsbn all "2070628973" and bib.doctype any "a" + +1 + + +marcxchange +xml + + + nam 22 450 +FRBNF434747910000007 +http://catalogue.bnf.fr/ark:/12148/cb434747912 + +978-2-07-062897-1 +Poche +5.90 EUR + + +ALS +1655223 + + +9782070628971 + + +20090707d2009 a y0frey50 ba + + +fre +eng + + +FR + + +||||z 00|y| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Ce qu'Alice trouva de l'autre côté du miroir +Texte imprimé +Lewis Carroll +traduit de l'anglais par Jacques Papy +illustrations de John Tenniel + + +Nouv. présentation + + +Paris +Gallimard-Jeunesse +2009 + + +185 p. +18 x 13 cm + + +Folio junior +Nicolas Balutet +0153-0593 +151 + + +Titre de couv. : De l'autre côté du miroir + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +Electre + + +34226165 +Collection Folio junior +0153-0593 +151 + + +De l'autre côté du miroir + + +11885918 +ISNI000000012137136X +Carroll +Lewis +070 + + +11918569 +ISNI0000000119322075 +Papy +Jacques +730 + + +11926193 +ISNI0000000121302807 +Tenniel +John +440 + + +FR +FR-751131015 +20090707 +AFNOR +FRBNF434747910000007 +intermrc + + +120251 +Première de couverture + + +ark:/12148/cb434747912 +1 + +20140820 +20151221 +7.152142 + + + + diff --git a/internal/metadata/bnf/testdata/older_records_alice.xml b/internal/metadata/bnf/testdata/older_records_alice.xml new file mode 100644 index 000000000..0c1352f1c --- /dev/null +++ b/internal/metadata/bnf/testdata/older_records_alice.xml @@ -0,0 +1,371 @@ + +1.2 + +1.2 +bib.persistentid all "ark:/12148/cb31911075d" or bib.persistentid all "ark:/12148/cb34614049g" or bib.persistentid all "ark:/12148/cb32941384z" + +3 + + +marcxchange +xml + + + nam 22 3n 450 +FRBNF329413840000007 +http://catalogue.bnf.fr/ark:/12148/cb32941384z + +CRI +LX000403860207P + + +19970701d1964 m y0frey50 ba + + +fre + + +FR + + +||||z 00||| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice au pays des merveilles ["Alice's adventures in wonderland"]. Traduction de Henriette Rouillard... +Texte imprimé +Lewis Carroll + + +[Paris,] +Delagrave (Évreux, impr. Labadie) +1964 + + +In-16 (18 cm), 158 p., ill., couv. ill. en coul. [D. L. 10239-64] + + +Bibliothèque Juventa + + +12011248 +Alice's adventures in Wonderland +français +lien automatique + + +11885918 +ISNI000000012137136X +Carroll +Lewis +070 + + +12057970 +ISNI0000000077123317 +Rouillard +Henriette +730 + + +FR +FR-751131015 +19970701 +AFNOR +FRBNF329413840000007 +intermrc + + +FR-751131010:32941384001001 +16-Y2-27689 +759999999 +Tolbiac - Rez de Jardin - Littérature et art - Magasin +O + + +ark:/12148/cb32941384z +1 + +19970701 +20180607 +7.5445237 + + + +marcxchange +xml + + + cam 22 3 450 +FRBNF319110750000009 +http://catalogue.bnf.fr/ark:/12148/cb31911075d + +CRI +SU022904260001S + + +19970701d1955 a y0frey50 ba + + +fre + + +FR + + +a z 00|a| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice au Pays des Merveilles +Texte imprimé +Lewis Carroll +illustrations d'Alice Huertas + + +Paris +éditions Bias +DL 1955 + + +Paris +Impr. Moderne du Lion + + +46 p. n. ch. +ill. en n. et en coul., cart. ill. en coul. +30 cm + + +Collection Belles lectures + + +Achevé d'imprimer : 1956 + + +32954523 +Collection Belles lectures +1955 + + +12011248 +Alice's adventures in Wonderland +français +Adaptation + + +11885918 +ISNI000000012137136X +Carroll +Lewis +1832-1898 +100 + + +17100175 +ISNI0000000423310271 +Huertas +Alice +19..?-19..? +440 + + +FR +FR-751131015 +19970701 +AFNOR +FRBNF319110750000009 +intermrc + + +FR-751131010:31911075001001 +4-Y2-4510 +759999999 +Tolbiac - Rez de Jardin - Littérature et art - Magasin +O + + +ark:/12148/cb31911075d +2 + +19970701 +20250604 +7.544245 + + + +marcxchange +xml + + + nam 22 450 +FRBNF34614049000000X +http://catalogue.bnf.fr/ark:/12148/cb34614049g + +Br. +3 F + + +FR +07906680 + + +OPL +000723841 + + +19790313d1977 a y0frey50 ba + + +fre +eng + + +BE + + +a z 00|a| + + +r + + +01 +i +xxxe +b +xb2e + + +02 +txt +sti +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice aux pays des merveilles +Texte imprimé +illustrations, José Luis Macias S. + + +[Chevron] +[Paris] +Hemma +1977 +impr. en Belgique + + +[11] p. +ill. en coul., couv. ill. en coul. +29 cm + + +Titre de couv. : "Alice au pays des merveilles" + + +D'après Lewis Carroll + + +12011248 +Alice's adventures in Wonderland +français +lien automatique + + +Alice au pays des merveilles + + +83 +Cadre de classement de la Bibliographie nationale française + + +11885918 +ISNI000000012137136X +Carroll +Lewis +330 + + +12123400 +ISNI0000000079803754 +Macias Sampedro +José Luis +440 + + +FR +FR-751131015 +19790313 +AFNOR +FRBNF34614049000000X +intermrc + + +FR-751131010:34614049001001 +EL 4-Y PIECE-176 +759999999 +Tolbiac - Rez de Jardin - Littérature et art - Magasin +O + + +ark:/12148/cb34614049g +3 + +19860717 +20210623 +7.543034 + + + + diff --git a/internal/metadata/bnf/testdata/recent_carroll.xml b/internal/metadata/bnf/testdata/recent_carroll.xml new file mode 100644 index 000000000..42a07265a --- /dev/null +++ b/internal/metadata/bnf/testdata/recent_carroll.xml @@ -0,0 +1,123 @@ + +1.2 + +1.2 +bib.author all "carroll lewis" and bib.doctype any "a" and bib.creationdate >= "20260601" + +1 + + +marcxchange +xml + + + a 22 450 +FRBNF487804930000008 +http://catalogue.bnf.fr/ark:/12148/cb48780493c + +978-2-401-12020-4 +3,50 EUR +br + + +9782401120204 + + +20260601u2026 m y0frey50 ba + + +fre + + +FR + + +||||z 00||| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice au pays des merveilles + + +Paris +Hatier +2026 + + +00- +Impr. GRAFO + + +1 volume 160 p +18 cm + + +Oeuvres & Thèmes + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +éditeur + + +87199254 +Alice au pays des merveilles +français + + +Carroll +Lewis +Auteur du texte + + +Bué +Henri +Traducteur + + +FR +FR-751131015 +20260601 +AFNOR +FRBNF487804930000008 +intermrc + + +FR +FR-751131015 + + +1239559 +Première de couverture + + +ark:/12148/cb48780493c +1 + +20260601 +20260601 +11.497916 + + + + diff --git a/internal/metadata/bnf/testdata/record_miroir.xml b/internal/metadata/bnf/testdata/record_miroir.xml new file mode 100644 index 000000000..0751b2b9a --- /dev/null +++ b/internal/metadata/bnf/testdata/record_miroir.xml @@ -0,0 +1,148 @@ + +1.2 + +1.2 +bib.persistentid all "ark:/12148/cb434747912" + +1 + + +marcxchange +xml + + + nam 22 450 +FRBNF434747910000007 +http://catalogue.bnf.fr/ark:/12148/cb434747912 + +978-2-07-062897-1 +Poche +5.90 EUR + + +ALS +1655223 + + +9782070628971 + + +20090707d2009 a y0frey50 ba + + +fre +eng + + +FR + + +||||z 00|y| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Ce qu'Alice trouva de l'autre côté du miroir +Texte imprimé +Lewis Carroll +traduit de l'anglais par Jacques Papy +illustrations de John Tenniel + + +Nouv. présentation + + +Paris +Gallimard-Jeunesse +2009 + + +185 p. +18 x 13 cm + + +Folio junior +Nicolas Balutet +0153-0593 +151 + + +Titre de couv. : De l'autre côté du miroir + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +Electre + + +34226165 +Collection Folio junior +0153-0593 +151 + + +De l'autre côté du miroir + + +11885918 +ISNI000000012137136X +Carroll +Lewis +070 + + +11918569 +ISNI0000000119322075 +Papy +Jacques +730 + + +11926193 +ISNI0000000121302807 +Tenniel +John +440 + + +FR +FR-751131015 +20090707 +AFNOR +FRBNF434747910000007 +intermrc + + +120251 +Première de couverture + + +ark:/12148/cb434747912 +1 + +20140820 +20151221 +7.543979 + + + + diff --git a/internal/metadata/bnf/testdata/search_alice.xml b/internal/metadata/bnf/testdata/search_alice.xml new file mode 100644 index 000000000..567a8ad4d --- /dev/null +++ b/internal/metadata/bnf/testdata/search_alice.xml @@ -0,0 +1,330 @@ + +1.2 + +1.2 +bib.title all "alice pays merveilles" and bib.author all "carroll" and bib.doctype any "a" + +454 + + +marcxchange +xml + + + a 22 450 +FRBNF487804930000008 +http://catalogue.bnf.fr/ark:/12148/cb48780493c + +978-2-401-12020-4 +3,50 EUR +br + + +9782401120204 + + +20260601u2026 m y0frey50 ba + + +fre + + +FR + + +||||z 00||| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Alice au pays des merveilles + + +Paris +Hatier +2026 + + +00- +Impr. GRAFO + + +1 volume 160 p +18 cm + + +Oeuvres & Thèmes + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +éditeur + + +87199254 +Alice au pays des merveilles +français + + +Carroll +Lewis +Auteur du texte + + +Bué +Henri +Traducteur + + +FR +FR-751131015 +20260601 +AFNOR +FRBNF487804930000008 +intermrc + + +FR +FR-751131015 + + +1239559 +Première de couverture + + +ark:/12148/cb48780493c +1 + +20260601 +20260601 +20.234125 + + + +marcxchange +xml + + + nam 22 3n 450 +FRBNF486557600000002 +http://catalogue.bnf.fr/ark:/12148/cb48655760n + +978-2-322-66883-0 + + +43999020413939782322668830 + + +20251127d2025 m y0frey50 ba + + +fre + + +DE + + +||||z 00||| + + +s +r + + +dru|||||||||| + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +b + + +02 +c +rdamedia + + +Alice au pays des merveilles +Texte électronique +Lewis Carroll +édité par Edmond Bronstein + + +[Forbach ] +BoD - Books on Demand +2025 + + +1 ressource dématérialisée + + +9.11 Mo1 + + +Métadonnées fournies par l'éditeur + + +Pagination restituée par l'éditeur : 138 pages + + +Carroll +Lewis +070 + + +Bronstein +Edmond +340 + + +20251127 +4399902041393 +9782322668830 +20251007 +onix/3.0 + + +1140279 +Première de couverture + + +FR-759999999:48655760001001 +LNUM-61586 +759999999 +Document numérisé +N +2025/11/27 + + +ark:/12148/cb48655760n +2 + +20251127 +20251127 +20.179876 + + + +marcxchange +xml + + + nam 22 450 +FRBNF477075420000005 +http://catalogue.bnf.fr/ark:/12148/cb47707542n + +978-2-322-56196-4 +br. +14,90 EUR + + +9782322561964 + + +20250415d2025 m y0frey50 ba + + +fre + + +FR + + +||||z 00||| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +Aventures d'Alice au pays des merveilles +Texte imprimé + + +Forbach +Books on Demand +2025 + + +1 vol. (153 p.) +21 cm + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +éditeur + + +Carroll +Lewis +Auteur du texte + + +FR +FR-751131015 +20250415 +AFNOR +FRBNF477075420000005 +intermrc + + +1046685 +Première de couverture + + +ark:/12148/cb47707542n +3 + +20250415 +20250415 +18.922945 + + + +4 + diff --git a/internal/metadata/bnf/testdata/search_outsider.xml b/internal/metadata/bnf/testdata/search_outsider.xml new file mode 100644 index 000000000..d80c3abff --- /dev/null +++ b/internal/metadata/bnf/testdata/search_outsider.xml @@ -0,0 +1,385 @@ + +1.2 + +1.2 +bib.title all "L'Outsider" and bib.author all "king stephen" and bib.doctype any "a" + +4 + + +marcxchange +xml + + + cam 22 450 +FRBNF456692120000004 +http://catalogue.bnf.fr/ark:/12148/cb456692125 + +978-2-298-14847-3 +br. +24,90 EUR + + +FR +01921675 + + +9782298148473 + + +20190213d2019 m y0frey50 ba + + +fre +eng + + +FR + + +||||z 00|a| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +L'outsider +Texte imprimé +roman +Stephen King +traduit de l'anglais (États-Unis) par Jean Esch + + +Paris +Éditions France loisirs +DL 2019 + + +18-Saint-Amand-Montrond +Impr. CPI Bussière + + +1 vol. (569 p.) +24 cm + + +The outsider + + +803 +Cadre de classement de la Bibliographie nationale française + + +11909772 +ISNI0000000121446296 +King +Stephen +1947-.... +070 + + +12058042 +ISNI0000000121319051 +Esch +Jean +1962-.... +730 + + +FR +FR-751131015 +20190213 +AFNOR +FRBNF456692120000004 +intermrc + + +FR-751131010:45669212001001 +2019-33520 +759999999 +Tolbiac - Rez de Jardin - Littérature et art - Magasin +O + + +ark:/12148/cb456692125 +1 + +20190213 +20190426 +19.170538 + + + +marcxchange +xml + + + cam 22 450 +FRBNF456707390000008 +http://catalogue.bnf.fr/ark:/12148/cb45670739m + +978-2-286-16248-1 +br. +24,90 EUR + + +FR +01921676 + + +9782286162481 + + +20190214d2019 m y0frey50 ba + + +fre +eng + + +FR + + +||||z 00|a| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +L'outsider +Texte imprimé +roman +Stephen King +traduit de l'anglais (États-Unis) par Jean Esch + + +[Paris] +le Grand livre du mois +DL 2019 + + +18-Saint-Amand-Montrond +Impr. CPI Bussière + + +1 vol. (569 p.) +24 cm + + +The outsider + + +803 +Cadre de classement de la Bibliographie nationale française + + +11909772 +ISNI0000000121446296 +King +Stephen +1947-.... +070 + + +12058042 +ISNI0000000121319051 +Esch +Jean +1962-.... +730 + + +FR +FR-751131015 +20190214 +AFNOR +FRBNF456707390000008 +intermrc + + +FR-751131010:45670739001001 +2019-34862 +759999999 +Tolbiac - Rez de Jardin - Littérature et art - Magasin +O + + +ark:/12148/cb45670739m +2 + +20190214 +20190426 +19.170538 + + + +marcxchange +xml + + + cam 22 450 +FRBNF456640410000002 +http://catalogue.bnf.fr/ark:/12148/cb456640410 + +978-2-226-43589-7 +br. +24,90 EUR + + +FR +01921674 + + +9782226435897 + + +20190206d2019 m y0frey50 ba + + +fre +eng + + +FR + + +||||z 00|a| + + +r + + +01 +i +xxxe + + +02 +txt +rdacontent + + +01 +n + + +02 +n +rdamedia + + +L'outsider +Texte imprimé +roman +Stephen King +traduit de l'anglais (États-Unis) par Jean Esch + + +Paris +Albin Michel +DL 2019 + + +18-Saint-Amand-Montrond +Impr. CPI Bussière + + +1 vol. (576 p.) +24 cm + + +Texte de quatrième de couverture remplacé dans ce fixture : le résumé d’origine est une notice de tiers. +éditeur + + +The outsider + + +803 +Cadre de classement de la Bibliographie nationale française + + +11909772 +ISNI0000000121446296 +King +Stephen +1947-.... +070 + + +12058042 +ISNI0000000121319051 +Esch +Jean +1962-.... +730 + + +FR +FR-751131015 +20190206 +AFNOR +FRBNF456640410000002 +intermrc + + +358844 +Première de couverture + + +FR-751131010:45664041001001 +2019-26949 +759999999 +Tolbiac - Rez de Jardin - Littérature et art - Magasin +O + + +ark:/12148/cb456640410 +3 + +20190206 +20190426 +19.168356 + + + +4 + diff --git a/internal/metadata/bnf/types.go b/internal/metadata/bnf/types.go new file mode 100644 index 000000000..3263c1a7e --- /dev/null +++ b/internal/metadata/bnf/types.go @@ -0,0 +1,136 @@ +package bnf + +import ( + "encoding/xml" + "strconv" + "strings" +) + +// sruResponse is an SRU 1.2 searchRetrieveResponse. Element names are matched +// on their local name, so the srw: prefix BnF uses does not matter, and every +// field is optional: a response missing one decodes to its zero value. +type sruResponse struct { + XMLName xml.Name `xml:"searchRetrieveResponse"` + NumberOfRecords string `xml:"numberOfRecords"` + Records []sruRecord `xml:"records>record"` + Diagnostics []sruDiagnostic `xml:"diagnostics>diagnostic"` +} + +// total is numberOfRecords, or -1 when the response does not state a usable +// count. +func (r *sruResponse) total() int { + n, err := strconv.Atoi(strings.TrimSpace(r.NumberOfRecords)) + if err != nil || n < 0 { + return -1 + } + return n +} + +type sruRecord struct { + Data struct { + Record marcRecord `xml:"record"` + } `xml:"recordData"` +} + +// sruDiagnostic is an SRU error. BnF sends one with HTTP 200, for example +// "Index non supporté" for an unknown index. +type sruDiagnostic struct { + URI string `xml:"uri"` + Details string `xml:"details"` + Message string `xml:"message"` +} + +// marcRecord is one marcxchange record in UNIMARC, the schema Bindery asks +// for (recordSchema=unimarcXchange). Type is "Bibliographic" or "Authority"; +// ID is the record's ARK ("ark:/12148/cb..."). +type marcRecord struct { + ID string `xml:"id,attr"` + Type string `xml:"type,attr"` + Format string `xml:"format,attr"` + Leader string `xml:"leader"` + ControlFields []controlField `xml:"controlfield"` + DataFields []dataField `xml:"datafield"` +} + +type controlField struct { + Tag string `xml:"tag,attr"` + Value string `xml:",chardata"` +} + +type dataField struct { + Tag string `xml:"tag,attr"` + Ind1 string `xml:"ind1,attr"` + Ind2 string `xml:"ind2,attr"` + Subfields []subfield `xml:"subfield"` +} + +type subfield struct { + Code string `xml:"code,attr"` + Value string `xml:",chardata"` +} + +func (r marcRecord) controlField(tag string) string { + for _, cf := range r.ControlFields { + if cf.Tag == tag { + return strings.TrimSpace(cf.Value) + } + } + return "" +} + +func (r marcRecord) fields(tag string) []dataField { + var out []dataField + for _, df := range r.DataFields { + if df.Tag == tag { + out = append(out, df) + } + } + return out +} + +// first returns the first $code of the first field tagged tag that has one. +func (r marcRecord) first(tag, code string) string { + for _, df := range r.DataFields { + if df.Tag != tag { + continue + } + if v := df.first(code); v != "" { + return v + } + } + return "" +} + +// all returns every $code value across every field tagged tag. +func (r marcRecord) all(tag, code string) []string { + var out []string + for _, df := range r.DataFields { + if df.Tag == tag { + out = append(out, df.all(code)...) + } + } + return out +} + +func (df dataField) first(code string) string { + for _, sf := range df.Subfields { + if sf.Code == code { + if v := strings.TrimSpace(sf.Value); v != "" { + return v + } + } + } + return "" +} + +func (df dataField) all(code string) []string { + var out []string + for _, sf := range df.Subfields { + if sf.Code == code { + if v := strings.TrimSpace(sf.Value); v != "" { + out = append(out, v) + } + } + } + return out +} diff --git a/internal/metadata/bnf/works.go b/internal/metadata/bnf/works.go new file mode 100644 index 000000000..2dfd0094b --- /dev/null +++ b/internal/metadata/bnf/works.go @@ -0,0 +1,340 @@ +package bnf + +import ( + "sort" + "strings" + + "github.com/vavallee/bindery/internal/indexer" + "github.com/vavallee/bindery/internal/models" + "github.com/vavallee/bindery/internal/textutil" +) + +// frenchLanguage is the code the representative edition is preferred in. +// This provider exists for French libraries, so a work is named after its +// French edition when it has one. +const frenchLanguage = "fre" + +// groupWorks folds edition records into one book per work, in first +// appearance order so BnF's relevance ranking survives. +// +// When authorID is set, only records crediting that authority record as +// author are kept: author2bib also returns the records where the person is +// translator, illustrator or the author a work is adapted from. +// +// Two editions belong to one work when they share a title, or when one's +// title is the other's original title (454 $t). The second rule is what puts +// "Alice au pays des merveilles" and "Aventures d'Alice au pays des +// merveilles" together: both translate "Alice's adventures in Wonderland". +// Titles are compared per author, so two authors' books of the same name +// stay apart. Callers link unlinked credits first (Client.linkUnlinked), so +// an edition BnF has not linked to its author still joins the work. +func groupWorks(eds []edition, authorID string) []models.Book { + type entry struct { + ed edition + author *credit + } + var kept []entry + for _, ed := range eds { + a := primaryAuthor(ed, authorID) + if authorID != "" && a == nil { + continue + } + kept = append(kept, entry{ed: ed, author: a}) + } + + uf := newUnionFind() + nodes := make([]string, len(kept)) + for i, e := range kept { + ak := authorKey(e.author) + nodes[i] = ak + "|" + titleKey(e.ed.title) + uf.add(nodes[i]) + if e.ed.originalTitle != "" { + orig := ak + "|" + titleKey(e.ed.originalTitle) + uf.add(orig) + uf.union(nodes[i], orig) + } + } + + var order []string + groups := make(map[string][]edition) + for i, e := range kept { + root := uf.find(nodes[i]) + if _, ok := groups[root]; !ok { + order = append(order, root) + } + groups[root] = append(groups[root], e.ed) + } + books := make([]models.Book, 0, len(order)) + for _, root := range order { + books = append(books, buildWork(groups[root], authorID)) + } + return books +} + +// buildWork turns one work's editions into a book. +// +// The representative edition supplies the foreign ID. It is the French +// edition entered on file first, so the ID stays put when BnF catalogues a +// new printing; the title is the one most French editions carry, so one +// record's cataloguing cannot rename the book. +func buildWork(group []edition, authorID string) models.Book { + rep := group[0] + for _, ed := range group[1:] { + if betterRepresentative(ed, rep) { + rep = ed + } + } + b := models.Book{ + ForeignID: idPrefix + rep.ark, + Title: commonTitle(group, rep), + OriginalTitle: commonOriginalTitle(group), + Description: rep.description, + Language: rep.language, + MetadataProvider: providerName, + Monitored: true, + Status: models.BookStatusWanted, + Genres: []string{}, + } + b.SortTitle = b.Title + if a := workAuthor(group, rep, authorID); a != nil { + author := a.toAuthor() + b.Author = &author + } + seenCredit := make(map[string]bool) + for _, ed := range group { + if b.Description == "" { + b.Description = ed.description + } + if ed.date != nil && ed.language == rep.language && (b.ReleaseDate == nil || ed.date.Before(*b.ReleaseDate)) { + b.ReleaseDate = ed.date + } + for _, c := range ed.credits { + if c.author && c.authorityID != "" && !seenCredit[c.authorityID] { + seenCredit[c.authorityID] = true + b.CreditedAuthorForeignIDs = append(b.CreditedAuthorForeignIDs, authorPrefix+c.authorityID) + } + } + b.Editions = append(b.Editions, toEdition(ed)) + b.ProviderISBNs = append(b.ProviderISBNs, ed.isbn13...) + b.ProviderISBNs = append(b.ProviderISBNs, ed.isbn10...) + } + return b +} + +// betterRepresentative reports whether a should replace b as the work's +// representative: French first, then the earlier entry date, then the ARK, +// so the choice is the same on every fetch of the same records. +func betterRepresentative(a, b edition) bool { + if fa, fb := a.language == frenchLanguage, b.language == frenchLanguage; fa != fb { + return fa + } + if a.entered != b.entered { + if a.entered == "" || b.entered == "" { + return a.entered != "" + } + return a.entered < b.entered + } + return a.ark < b.ark +} + +// commonTitle is the title most editions in the representative's language +// carry. Ties go to the representative's own title, then to sort order. +func commonTitle(group []edition, rep edition) string { + counts := make(map[string]int) + for _, ed := range group { + if ed.language == rep.language { + counts[ed.title]++ + } + } + return mostCommon(counts, rep.title) +} + +func commonOriginalTitle(group []edition) string { + counts := make(map[string]int) + for _, ed := range group { + if ed.originalTitle != "" { + counts[ed.originalTitle]++ + } + } + return mostCommon(counts, "") +} + +func mostCommon(counts map[string]int, fallback string) string { + keys := make([]string, 0, len(counts)) + for k := range counts { + keys = append(keys, k) + } + sort.Strings(keys) + best := fallback + for _, k := range keys { + if counts[k] > counts[best] { + best = k + } + } + return best +} + +func toEdition(ed edition) models.Edition { + e := models.Edition{ + ForeignID: idPrefix + ed.ark, + Title: ed.title, + Publisher: ed.publisher, + PublishDate: ed.date, + Language: ed.language, + IsEbook: ed.ebook, + } + if len(ed.isbn13) > 0 { + v := ed.isbn13[0] + e.ISBN13 = &v + } + if len(ed.isbn10) > 0 { + v := ed.isbn10[0] + e.ISBN10 = &v + } + if ed.pages > 0 { + pages := ed.pages + e.NumPages = &pages + } + if ed.ebook { + e.Format = models.MediaTypeEbook + } + return e +} + +// primaryAuthor is the record's author credit: the one linked to authorID +// when given, otherwise the first author credit, preferring a linked one. +func primaryAuthor(ed edition, authorID string) *credit { + var first *credit + for i := range ed.credits { + c := &ed.credits[i] + if !c.author { + continue + } + if authorID != "" { + if c.authorityID == authorID { + return c + } + continue + } + if c.authorityID != "" { + return c + } + if first == nil { + first = c + } + } + return first +} + +// workAuthor is the author the book is filed under. Records converted to +// BnF's 2026 model name the author without an authority link; when the +// representative is one of those, a linked credit for the same name on +// another edition of the work stands in, so the book still carries an author +// Bindery can look up. With no linked credit anywhere the book has no +// author: a name alone is not an identity Bindery can bind. +func workAuthor(group []edition, rep edition, authorID string) *credit { + a := primaryAuthor(rep, authorID) + if a != nil && a.authorityID != "" { + return a + } + for _, ed := range group { + for i := range ed.credits { + c := &ed.credits[i] + if !c.author || c.authorityID == "" { + continue + } + if a == nil || sameName(*a, *c) { + return c + } + } + } + return nil +} + +// linkByName gives an unlinked author credit the authority link a credit of +// the exact same name carries elsewhere in the same result set, so the +// editions BnF converted to its 2026 model (which arrive without links) join +// their work instead of forming a second one. A name shared by two linked +// people links nothing. The input is not modified. +func linkByName(eds []edition) []edition { + ids := make(map[string]string) + for _, ed := range eds { + for _, c := range ed.credits { + if !c.author || c.authorityID == "" { + continue + } + key := textutil.FoldForSearch(c.sortName()) + if prev, ok := ids[key]; ok && prev != c.authorityID { + ids[key] = "" // ambiguous + continue + } + ids[key] = c.authorityID + } + } + out := make([]edition, len(eds)) + for i, ed := range eds { + out[i] = ed + changed := false + for j, c := range ed.credits { + if !c.author || c.authorityID != "" { + continue + } + if id := ids[textutil.FoldForSearch(c.sortName())]; id != "" { + if !changed { + out[i].credits = append([]credit(nil), ed.credits...) + changed = true + } + out[i].credits[j].authorityID = id + } + } + } + return out +} + +func sameName(a, b credit) bool { + return textutil.FoldForSearch(a.sortName()) == textutil.FoldForSearch(b.sortName()) +} + +func authorKey(c *credit) string { + if c == nil { + return "" + } + if c.authorityID != "" { + return c.authorityID + } + return textutil.FoldForSearch(c.sortName()) +} + +// titleKey is the grouping key for a title. Spaces are dropped so a +// cataloguing slip that splits a word stays with its work, and a typographic +// apostrophe is the same as a straight one ("L’outsider", "L'outsider"). +func titleKey(title string) string { + title = strings.ReplaceAll(title, "\u2019", "'") + return strings.ReplaceAll(indexer.CanonicalDedupKey(title), " ", "") +} + +// unionFind joins title keys that name the same work. +type unionFind struct{ parent map[string]string } + +func newUnionFind() *unionFind { return &unionFind{parent: make(map[string]string)} } + +func (u *unionFind) add(k string) { + if _, ok := u.parent[k]; !ok { + u.parent[k] = k + } +} + +func (u *unionFind) find(k string) string { + for u.parent[k] != k { + u.parent[k] = u.parent[u.parent[k]] + k = u.parent[k] + } + return k +} + +func (u *unionFind) union(a, b string) { + ra, rb := u.find(a), u.find(b) + if ra != rb { + u.parent[rb] = ra + } +} From f915649c4a0378ef1b15c2cd74c1be0eca4264b0 Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:08:15 -0300 Subject: [PATCH 02/13] feat(metadata): BnF as an opt-in primary provider (#3144) metadata.primary_provider accepts "bnf". Like Nasjonalbiblioteket it is wired only when chosen, never as an enricher, so an install that did not choose it sends BnF nothing. The "bnf:" prefix is known to the foreign ID classifiers. Before, it fell through to openlibrary: with OpenLibrary primary a bnf: author would have been routed to the OpenLibrary client, and SafeToBind would have taken a BnF match for the primary's after an OpenLibrary failure. Catalogue reconciliation skips the per work editions lookup for bnf works, as for dnb and nb: each one is a full GetBook, at one request a second. Calibre identifiers carry the bare ARK. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- cmd/bindery/main.go | 8 ++ cmd/bindery/metadata_providers_test.go | 2 + internal/api/catalogue_reconciliation.go | 17 +-- internal/api/catalogue_reconciliation_test.go | 118 ++++++++++-------- internal/api/settings_handler.go | 4 +- internal/api/settings_handler_test.go | 3 + internal/calibre/identifiers.go | 2 + internal/calibre/metadata_test.go | 1 + .../aggregator_foreign_id_classifier_test.go | 30 +++++ internal/models/author.go | 2 + internal/models/author_test.go | 2 + internal/models/book.go | 2 + internal/models/book_identity_test.go | 1 + 13 files changed, 128 insertions(+), 64 deletions(-) diff --git a/cmd/bindery/main.go b/cmd/bindery/main.go index 2f85a05b2..75ce226a3 100644 --- a/cmd/bindery/main.go +++ b/cmd/bindery/main.go @@ -38,6 +38,7 @@ import ( "github.com/vavallee/bindery/internal/jobs" "github.com/vavallee/bindery/internal/logbuf" "github.com/vavallee/bindery/internal/metadata" + "github.com/vavallee/bindery/internal/metadata/bnf" "github.com/vavallee/bindery/internal/metadata/dnb" "github.com/vavallee/bindery/internal/metadata/googlebooks" "github.com/vavallee/bindery/internal/metadata/hardcover" @@ -310,6 +311,8 @@ func main() { // where OpenLibrary coverage is too thin for German-language books. // - "nb" is the same for Norwegian ones: original titles where // OpenLibrary has the English translation. + // - "bnf" is the same for French editions, translations included: the + // French title an indexer names a release after (#3144). // - "hardcover" trades breadth for a cleaner, editorially curated // catalogue: no translation editions masquerading as separate works, no // omnibus bundles, no non-book merchandise rows (#2040). @@ -327,6 +330,11 @@ func main() { // so an install that did not choose it sends NB no traffic. NB // publishes no rate limits for a fleet of independent installs. primaryProvider = nb.New() + case "bnf": + // Opt-in only, like NB: never an enricher, so an install that did not + // choose it sends BnF no traffic. BnF publishes no rate limits; the + // client paces itself at one request a second (internal/metadata/bnf). + primaryProvider = bnf.New() case "hardcover": primaryProvider = hcClient default: diff --git a/cmd/bindery/metadata_providers_test.go b/cmd/bindery/metadata_providers_test.go index a0f1c8c48..914619b92 100644 --- a/cmd/bindery/metadata_providers_test.go +++ b/cmd/bindery/metadata_providers_test.go @@ -17,6 +17,8 @@ func TestResolveMetadataPrimaryProvider(t *testing.T) { {"unset defaults to openlibrary even with a token", "", true, "openlibrary"}, {"explicit openlibrary", "openlibrary", false, "openlibrary"}, {"explicit dnb", "dnb", false, "dnb"}, + {"explicit nb", "nb", false, "nb"}, + {"explicit bnf", "bnf", false, "bnf"}, {"hardcover with a token", "hardcover", true, "hardcover"}, {"hardcover without a token falls back", "hardcover", false, "openlibrary"}, {"unknown value falls back", "goodreads", true, "openlibrary"}, diff --git a/internal/api/catalogue_reconciliation.go b/internal/api/catalogue_reconciliation.go index f6f17ee92..bd93c14b9 100644 --- a/internal/api/catalogue_reconciliation.go +++ b/internal/api/catalogue_reconciliation.go @@ -275,13 +275,16 @@ func (h *AuthorHandler) buildCatalogueReconciliation(ctx context.Context, author if provider == "dnb" { return } - // NB works come from the author catalogue with every edition - // already attached, and its per-work editions lookup is a full - // GetBook, so the work's own editions are the evidence. None - // attached is not evidence of no ISBN or too few pages, and on a - // partial catalogue a work may be missing editions, so then - // nothing is judged on them. - if provider == "nb" { + // NB and BnF works come from the author catalogue with every + // edition already attached, and their per-work editions lookup + // is a full GetBook, so the work's own editions are the + // evidence. None attached is not evidence of no ISBN or too few + // pages, and on a partial catalogue a work may be missing + // editions, so then nothing is judged on them. BnF's catalogue + // is always partial while its 2026 record migration runs, so its + // works stay indeterminate here rather than costing a BnF + // request each. + if provider == "nb" || provider == "bnf" { if snapshot.Complete && len(work.Editions) > 0 { mu.Lock() editions[work.ForeignID] = editionEvidence{editions: work.Editions, known: true} diff --git a/internal/api/catalogue_reconciliation_test.go b/internal/api/catalogue_reconciliation_test.go index 7d7351d88..26a68e4a5 100644 --- a/internal/api/catalogue_reconciliation_test.go +++ b/internal/api/catalogue_reconciliation_test.go @@ -847,64 +847,72 @@ func TestBuildCatalogueReconciliation_UsesConservativeEditionAndIdentityEvidence } } -// NB works arrive from the author catalogue with every edition attached, so -// the per-work editions lookup (a full GetBook each) is skipped and the -// work's own editions are the evidence. A work that somehow has none stays -// indeterminate: no editions is not proof of no ISBN or too few pages. On a -// partial catalogue (page cap, failed series recall) a work may be missing -// editions, so none of them is judged on that evidence. +// NB and BnF works arrive from the author catalogue with every edition +// attached, so the per-work editions lookup (a full GetBook each) is skipped +// and the work's own editions are the evidence. A work that somehow has none +// stays indeterminate: no editions is not proof of no ISBN or too few pages. +// On a partial catalogue (page cap, failed series recall, BnF's unlinked 2026 +// records) a work may be missing editions, so none of them is judged on that +// evidence. func TestBuildCatalogueReconciliation_NBUsesTheWorksOwnEditions(t *testing.T) { - for _, complete := range []bool{true, false} { - pages100, pages300 := 100, 300 - isbn := "9788200000011" - lookupErr := errors.New("per-work editions lookup must not run for nb works") - provider := &editionResultsProvider{ - partialSnapshotProvider: partialSnapshotProvider{ - stubMetaProvider: stubMetaProvider{name: "nb", works: []models.Book{ - {ForeignID: "nb:long", Title: "Long Work", Language: "eng", MetadataProvider: "nb", - Editions: []models.Edition{{NumPages: &pages300, ISBN13: &isbn}}}, - {ForeignID: "nb:short", Title: "Short Work", Language: "eng", MetadataProvider: "nb", - Editions: []models.Edition{{NumPages: &pages100, ISBN13: &isbn}}}, - {ForeignID: "nb:bare", Title: "Bare Work", Language: "eng", MetadataProvider: "nb"}, - }}, - complete: complete, - }, - errors: map[string]error{"nb:long": lookupErr, "nb:short": lookupErr, "nb:bare": lookupErr}, - } - f := newReconciliationFixture(t, provider) - profile, err := f.profiles.GetByID(context.Background(), models.DefaultMetadataProfileID) - if err != nil || profile == nil { - t.Fatalf("load profile: profile=%+v err=%v", profile, err) - } - profile.MinPages = 200 - if err := f.profiles.Update(context.Background(), profile); err != nil { - t.Fatal(err) - } - long := f.createBook(t, "nb:long", "Long Work", "nb", models.BookStatusWanted) - short := f.createBook(t, "nb:short", "Short Work", "nb", models.BookStatusWanted) - bare := f.createBook(t, "nb:bare", "Bare Work", "nb", models.BookStatusWanted) - f.author.ForeignID = "nb:author:10000001" // an NB author: the snapshot routes by prefix + for _, tc := range []struct{ provider, authorID string }{ + {"nb", "nb:author:10000001"}, + {"bnf", "bnf:author:11885918"}, + } { + for _, complete := range []bool{true, false} { + name := tc.provider + id := func(s string) string { return name + ":" + s } + pages100, pages300 := 100, 300 + isbn := "9788200000011" + lookupErr := errors.New("per-work editions lookup must not run for " + name + " works") + provider := &editionResultsProvider{ + partialSnapshotProvider: partialSnapshotProvider{ + stubMetaProvider: stubMetaProvider{name: name, works: []models.Book{ + {ForeignID: id("long"), Title: "Long Work", Language: "eng", MetadataProvider: name, + Editions: []models.Edition{{NumPages: &pages300, ISBN13: &isbn}}}, + {ForeignID: id("short"), Title: "Short Work", Language: "eng", MetadataProvider: name, + Editions: []models.Edition{{NumPages: &pages100, ISBN13: &isbn}}}, + {ForeignID: id("bare"), Title: "Bare Work", Language: "eng", MetadataProvider: name}, + }}, + complete: complete, + }, + errors: map[string]error{id("long"): lookupErr, id("short"): lookupErr, id("bare"): lookupErr}, + } + f := newReconciliationFixture(t, provider) + profile, err := f.profiles.GetByID(context.Background(), models.DefaultMetadataProfileID) + if err != nil || profile == nil { + t.Fatalf("load profile: profile=%+v err=%v", profile, err) + } + profile.MinPages = 200 + if err := f.profiles.Update(context.Background(), profile); err != nil { + t.Fatal(err) + } + long := f.createBook(t, id("long"), "Long Work", name, models.BookStatusWanted) + short := f.createBook(t, id("short"), "Short Work", name, models.BookStatusWanted) + bare := f.createBook(t, id("bare"), "Bare Work", name, models.BookStatusWanted) + f.author.ForeignID = tc.authorID // the snapshot routes by prefix - ctx := auth.WithUserRole(auth.WithUserID(context.Background(), 7), "user") - got, err := f.handler.buildCatalogueReconciliation(ctx, f.author) - if err != nil { - t.Fatal(err) - } - assertIndeterminateRow(t, got, bare, reconcileIndeterminateReasonEditionUnavailable) - if !complete { - if len(got.Candidates) != 0 { - t.Errorf("partial catalogue: candidates = %+v, want none judged on its editions", got.Candidates) + ctx := auth.WithUserRole(auth.WithUserID(context.Background(), 7), "user") + got, err := f.handler.buildCatalogueReconciliation(ctx, f.author) + if err != nil { + t.Fatal(err) } - assertIndeterminateRow(t, got, short, reconcileIndeterminateReasonEditionUnavailable) - assertIndeterminateRow(t, got, long, reconcileIndeterminateReasonEditionUnavailable) - continue - } - if len(got.Candidates) != 1 || got.Candidates[0].BookID != short.ID || got.Candidates[0].Reason != reconcileReasonPages { - t.Errorf("candidates = %+v, want only the short work, for pages", got.Candidates) - } - for _, row := range got.IndeterminateRows { - if row.BookID == long.ID || row.BookID == short.ID { - t.Errorf("%q is indeterminate (%s): its own editions were not used", row.Title, row.Reason) + assertIndeterminateRow(t, got, bare, reconcileIndeterminateReasonEditionUnavailable) + if !complete { + if len(got.Candidates) != 0 { + t.Errorf("%s partial catalogue: candidates = %+v, want none judged on its editions", name, got.Candidates) + } + assertIndeterminateRow(t, got, short, reconcileIndeterminateReasonEditionUnavailable) + assertIndeterminateRow(t, got, long, reconcileIndeterminateReasonEditionUnavailable) + continue + } + if len(got.Candidates) != 1 || got.Candidates[0].BookID != short.ID || got.Candidates[0].Reason != reconcileReasonPages { + t.Errorf("%s candidates = %+v, want only the short work, for pages", name, got.Candidates) + } + for _, row := range got.IndeterminateRows { + if row.BookID == long.ID || row.BookID == short.ID { + t.Errorf("%s %q is indeterminate (%s): its own editions were not used", name, row.Title, row.Reason) + } } } } diff --git a/internal/api/settings_handler.go b/internal/api/settings_handler.go index 96546ead8..73527c6f9 100644 --- a/internal/api/settings_handler.go +++ b/internal/api/settings_handler.go @@ -72,7 +72,7 @@ const SettingDefaultAudiobookRootFolderID = "library.defaultAudiobookRootFolderI // SettingMetadataPrimaryProvider is the KV key that selects the primary // metadata provider used for author/book search and lookup. Valid values are -// "openlibrary" (default), "dnb", "nb", and "hardcover". Empty or unset falls back to +// "openlibrary" (default), "dnb", "nb", "bnf", and "hardcover". Empty or unset falls back to // "openlibrary" for backwards compatibility. // // "hardcover" requires a Hardcover API token (SettingHardcoverAPIToken) — @@ -85,7 +85,7 @@ const SettingMetadataPrimaryProvider = "metadata.primary_provider" // MetadataPrimaryProviders lists the accepted values of // SettingMetadataPrimaryProvider, in the order the UI presents them. The first // entry is the default used when the setting is empty or unset. -var MetadataPrimaryProviders = []string{"openlibrary", "dnb", "nb", "hardcover"} +var MetadataPrimaryProviders = []string{"openlibrary", "dnb", "nb", "bnf", "hardcover"} // IsMetadataPrimaryProviderValid reports whether value names a provider that // may be promoted to primary. The empty string is valid and means "default". diff --git a/internal/api/settings_handler_test.go b/internal/api/settings_handler_test.go index c1e90bb01..7b0b9beec 100644 --- a/internal/api/settings_handler_test.go +++ b/internal/api/settings_handler_test.go @@ -566,6 +566,9 @@ func TestSettings_MetadataPrimaryProvider(t *testing.T) { {"", true}, {"openlibrary", true}, {"dnb", true}, + {"nb", true}, + {"bnf", true}, + {"BNF", false}, {"hardcover", true}, {"googlebooks", false}, {"unknown", false}, diff --git a/internal/calibre/identifiers.go b/internal/calibre/identifiers.go index 6fd59dc76..b0587fb75 100644 --- a/internal/calibre/identifiers.go +++ b/internal/calibre/identifiers.go @@ -49,6 +49,8 @@ func addProviderIdentifier(out map[string]string, provider, foreignID string) { val = trimIdentifierPrefix(val, "gb:") case "dnb": val = trimIdentifierPrefix(val, "dnb:") + case "bnf": + val = trimIdentifierPrefix(val, "bnf:") case "hardcover": val = trimIdentifierPrefix(val, "hc:") case "openlibrary": diff --git a/internal/calibre/metadata_test.go b/internal/calibre/metadata_test.go index 6c577648e..67dc4070b 100644 --- a/internal/calibre/metadata_test.go +++ b/internal/calibre/metadata_test.go @@ -111,6 +111,7 @@ func TestIdentifiersForBook_NormalizesProviderIdentifiers(t *testing.T) { {"googlebooks", "googlebooks", "gb:zyTCAlFPjgYC", "google", "zyTCAlFPjgYC"}, {"google", "google", "gb:zyTCAlFPjgYC", "google", "zyTCAlFPjgYC"}, {"dnb", "dnb", "dnb:123456789", "dnb", "123456789"}, + {"bnf", "bnf", "bnf:cb48780493c", "bnf", "cb48780493c"}, {"fallback", "Other.Provider", " provider:value ", "other_provider", "provider:value"}, } for _, tt := range tests { diff --git a/internal/metadata/aggregator_foreign_id_classifier_test.go b/internal/metadata/aggregator_foreign_id_classifier_test.go index 329c038e9..0036b6c8f 100644 --- a/internal/metadata/aggregator_foreign_id_classifier_test.go +++ b/internal/metadata/aggregator_foreign_id_classifier_test.go @@ -16,6 +16,8 @@ func TestProviderNameForForeignIDMatchesModels(t *testing.T) { "hc:brandon": "hardcover", "gb:zyTCAlFPjgYC": "googlebooks", "dnb:118540238": "dnb", + "nb:author:1": "nb", + "bnf:cb48780493c": "bnf", "OL26320A": "openlibrary", "": "openlibrary", } @@ -81,3 +83,31 @@ func TestSafeToBindUnchangedForProviderIDs(t *testing.T) { } } } + +// TestBnFForeignIDsRouteOnlyToBnF pins the "bnf:" prefix (#3144). Before the +// prefix was known it classified as openlibrary, so with OpenLibrary primary +// a bnf: author would have been handed to the OpenLibrary client, and +// SafeToBind would have treated a BnF match as the primary's after an +// OpenLibrary failure. +func TestBnFForeignIDsRouteOnlyToBnF(t *testing.T) { + ol := &mockProvider{name: "openlibrary"} + if got := newTestAggregator(ol).providerForForeignID("bnf:author:11885918"); got != nil { + t.Errorf("bnf: id routed to %q with OpenLibrary primary, want nil", providerName(got)) + } + bnf := &mockProvider{name: "bnf"} + if got := providerName(newTestAggregator(bnf, ol).providerForForeignID("bnf:cb48780493c")); got != "bnf" { + t.Errorf("bnf: id routed to %q with BnF primary, want bnf", got) + } + + olFailed := SearchOutcome{Primary: "openlibrary", PrimaryFailed: true, FailedProviders: []string{"openlibrary"}} + if olFailed.SafeToBind("bnf:author:11885918") { + t.Error("SafeToBind(bnf:) = true after an OpenLibrary primary failure, want false") + } + bnfFailed := SearchOutcome{Primary: "bnf", PrimaryFailed: true, FailedProviders: []string{"bnf"}} + if !bnfFailed.SafeToBind("bnf:author:11885918") { + t.Error("SafeToBind(bnf:) = false with BnF primary, want true: the match came from the primary") + } + if bnfFailed.SafeToBind("OL26320A") { + t.Error("SafeToBind(OL id) = true after a BnF primary failure, want false") + } +} diff --git a/internal/models/author.go b/internal/models/author.go index 4cf557ed5..98c1d99d0 100644 --- a/internal/models/author.go +++ b/internal/models/author.go @@ -293,6 +293,8 @@ func AuthorProviderFromForeignID(foreignID string) string { return "dnb" case strings.HasPrefix(foreignID, "nb:"): return "nb" + case strings.HasPrefix(foreignID, "bnf:"): + return "bnf" case strings.HasPrefix(foreignID, "calibre:"): return "calibre" case strings.HasPrefix(foreignID, "abs:"): diff --git a/internal/models/author_test.go b/internal/models/author_test.go index eb9c15171..1c0518c4a 100644 --- a/internal/models/author_test.go +++ b/internal/models/author_test.go @@ -48,6 +48,8 @@ func TestAuthorProviderFromForeignID(t *testing.T) { {id: "hc:emilia-jae", want: "hardcover"}, {id: "dnb:123456789", want: "dnb"}, {id: "nb:author:10000001", want: "nb"}, + {id: "bnf:author:11885918", want: "bnf"}, + {id: "BNF:author:11885918", want: "bnf"}, {id: "gb:volume", want: "googlebooks"}, {id: "calibre:author:1", want: "calibre"}, {id: "abs:author:lib:author", want: "audiobookshelf"}, diff --git a/internal/models/book.go b/internal/models/book.go index 368a8c38b..fbb0f6bab 100644 --- a/internal/models/book.go +++ b/internal/models/book.go @@ -327,6 +327,8 @@ func BookProviderFromForeignID(foreignID string) string { return "dnb" case strings.HasPrefix(foreignID, "nb:"): return "nb" + case strings.HasPrefix(foreignID, "bnf:"): + return "bnf" case strings.HasPrefix(foreignID, "calibre:"): return "calibre" case strings.HasPrefix(foreignID, "abs:"): diff --git a/internal/models/book_identity_test.go b/internal/models/book_identity_test.go index e79482623..c76aeb2b5 100644 --- a/internal/models/book_identity_test.go +++ b/internal/models/book_identity_test.go @@ -15,6 +15,7 @@ func TestBookProviderFromForeignID(t *testing.T) { "gb:xyz": "googlebooks", "dnb:123": "dnb", "nb:a0000000000000000000000000000001": "nb", + "bnf:cb48780493c": "bnf", "": "openlibrary", " hc:spaced ": "hardcover", "something-random": "openlibrary", From 68a168a6e9ad8bc9137d5dbae95fb9a212f9044f Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:08:22 -0300 Subject: [PATCH 03/13] feat(web): BnF in the primary provider setting, with its credit (#3144) The primary provider select offers BnF. The Licence Ouverte asks for credit to the source and the retrieval date, so the setting's description names it, and choosing BnF shows the credit line and that BnF does not endorse Bindery. BnF book IDs link to their catalogue record by ARK in the Links menu; authors have no link, as their ARK needs a check character Bindery does not store. Only en.json gets the new strings. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- web/src/components/AddToLibraryModal.tsx | 2 +- .../components/AuthorMetadataLinkModal.tsx | 1 + web/src/i18n/locales/en.json | 4 +++- .../MetadataTab.primaryProvider.test.tsx | 19 +++++++++++++++++++ web/src/pages/settings/MetadataTab.tsx | 10 +++++++++- web/src/util/authorMetadata.ts | 1 + web/src/util/metadataSource.test.ts | 13 +++++++++++++ web/src/util/metadataSource.ts | 13 ++++++++++++- 8 files changed, 59 insertions(+), 4 deletions(-) diff --git a/web/src/components/AddToLibraryModal.tsx b/web/src/components/AddToLibraryModal.tsx index 33eb739d8..550d5c238 100644 --- a/web/src/components/AddToLibraryModal.tsx +++ b/web/src/components/AddToLibraryModal.tsx @@ -93,7 +93,7 @@ export default function AddToLibraryModal({ onClose, onAdded, initialQuery, mode const value = (s.value || '').trim().toLowerCase() // Mirror MetadataPrimaryProviders on the backend; anything else means // no explicit choice, so no notice. - if (value === 'openlibrary' || value === 'dnb' || value === 'nb' || value === 'hardcover') setPrimaryProvider(value) + if (value === 'openlibrary' || value === 'dnb' || value === 'nb' || value === 'bnf' || value === 'hardcover') setPrimaryProvider(value) }) .catch(() => { /* unset; no provider notice needed */ }) }, [isRequester]) diff --git a/web/src/components/AuthorMetadataLinkModal.tsx b/web/src/components/AuthorMetadataLinkModal.tsx index a6142755c..60b7133c8 100644 --- a/web/src/components/AuthorMetadataLinkModal.tsx +++ b/web/src/components/AuthorMetadataLinkModal.tsx @@ -16,6 +16,7 @@ function providerLabel(author: Author): string { if (provider === 'openlibrary' || author.foreignAuthorId.startsWith('OL')) return 'OpenLibrary' if (provider === 'dnb' || author.foreignAuthorId.startsWith('dnb:')) return 'DNB' if (provider === 'nb' || author.foreignAuthorId.startsWith('nb:')) return 'Nasjonalbiblioteket' + if (provider === 'bnf' || author.foreignAuthorId.startsWith('bnf:')) return 'BnF' if (provider) return provider return 'Metadata' } diff --git a/web/src/i18n/locales/en.json b/web/src/i18n/locales/en.json index 758490c8c..d5d35cb2e 100644 --- a/web/src/i18n/locales/en.json +++ b/web/src/i18n/locales/en.json @@ -1515,10 +1515,12 @@ "telemetryDetail": "Sends install_id (random UUID), version, OS, architecture, deploy method, feature-usage counts, and coarse error counters (ERROR/WARN totals over the last 24h plus the 5 most frequent fixed error-message strings — never log details, titles, paths, or URLs) to getbindery.dev once per day. Disable at any time.", "metadataProvider": "Metadata Provider", "metadataProviderLabel": "Primary metadata provider", - "metadataProviderHint": "Selects the source used for author and book search, and decides what an author's catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket, which is only queried when it is the primary.", + "metadataProviderHint": "Selects the source used for author and book search, and decides what an author's catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. BnF (Bibliothèque nationale de France) is recommended for French-language libraries: legal deposit puts nearly every book published in France in its catalogue, under the French edition title, translator included, where OpenLibrary often has only the original English title. Contains data from the Bibliothèque nationale de France under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed; BnF does not endorse Bindery. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket and BnF, which are only queried when chosen as the primary.", "metadataProviderOpenlibrary": "OpenLibrary (default)", "metadataProviderDnb": "DNB — Deutsche Nationalbibliothek (German/DACH)", "metadataProviderNb": "Nasjonalbiblioteket — National Library of Norway (Norwegian)", + "metadataProviderBnf": "BnF, Bibliothèque nationale de France (French)", + "metadataProviderBnfCredit": "Contains data from the Bibliothèque nationale de France (BnF) under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed. BnF does not endorse Bindery. Only BnF's catalogue records are used: no cover images are fetched, and BnF is contacted only while it is the primary provider.", "metadataProviderHardcover": "Hardcover (curated catalogue)", "metadataProviderHardcoverNoToken": "Hardcover — requires an API token (Settings → API Keys)", "metadataProviderRestartHint": "Takes effect after the next Bindery restart — the provider is wired at startup, so nothing changes until then. Authors already in your library stay linked to the provider they were added from; the new primary applies to authors you add afterwards. To move an existing author, use \"Link metadata\" on their page.", diff --git a/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx b/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx index b868e8fb2..5714c56bd 100644 --- a/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx +++ b/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx @@ -111,4 +111,23 @@ describe('Primary metadata provider selector', () => { expect(hint!.textContent).toMatch(/restart/i) expect(hint!.textContent).toMatch(/already in your library/i) }) + + // #3144: BnF is opt in, and its Licence Ouverte asks for credit to the + // source and the retrieval date where the provider is chosen. + it('offers BnF, persists it, and credits the BnF when it is chosen', async () => { + render() + const select = await primaryProviderSelect() + expect(screen.queryByTestId('bnf-credit')).toBeNull() + expect(select.parentElement!.textContent).toMatch(/Licence Ouverte/) + + fireEvent.change(select, { target: { value: 'bnf' } }) + await waitFor(() => { + expect(api.setSetting).toHaveBeenCalledWith('metadata.primary_provider', 'bnf') + }) + const credit = await screen.findByTestId('bnf-credit') + expect(credit.textContent).toMatch(/Contains data from the Bibliothèque nationale de France/) + expect(credit.textContent).toMatch(/Licence Ouverte \/ Open Licence 2\.0/) + expect(credit.textContent).toMatch(/retrieved when/) + expect(credit.textContent).toMatch(/does not endorse/) + }) }) diff --git a/web/src/pages/settings/MetadataTab.tsx b/web/src/pages/settings/MetadataTab.tsx index 4f6c69f2a..2a35623fb 100644 --- a/web/src/pages/settings/MetadataTab.tsx +++ b/web/src/pages/settings/MetadataTab.tsx @@ -89,7 +89,7 @@ export default function MetadataTab() { {t('settings.general.metadataProviderLabel', 'Primary metadata provider')}

- {t('settings.general.metadataProviderHint', 'Selects the source used for author and book search, and decides what an author\'s catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket, which is only queried when it is the primary.')} + {t('settings.general.metadataProviderHint', 'Selects the source used for author and book search, and decides what an author\'s catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. BnF (Bibliothèque nationale de France) is recommended for French-language libraries: legal deposit puts nearly every book published in France in its catalogue, under the French edition title, translator included, where OpenLibrary often has only the original English title. Contains data from the Bibliothèque nationale de France under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed; BnF does not endorse Bindery. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket and BnF, which are only queried when chosen as the primary.')}

+ {/* The Licence Ouverte asks for credit to the source and the date of + retrieval wherever the data is shown with its provider (#3144). */} + {settings['metadata.primary_provider'] === 'bnf' && ( +

+ {t('settings.general.metadataProviderBnfCredit', 'Contains data from the Bibliothèque nationale de France (BnF) under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed. BnF does not endorse Bindery. Only BnF\'s catalogue records are used: no cover images are fetched, and BnF is contacted only while it is the primary provider.')} +

+ )} {/* The generic restartRequired string used to sit here, but it did not say what a user flipping this selector actually needs to know: the provider is resolved once at boot (cmd/bindery/main.go), and even diff --git a/web/src/util/authorMetadata.ts b/web/src/util/authorMetadata.ts index fbe00a018..15a2feb78 100644 --- a/web/src/util/authorMetadata.ts +++ b/web/src/util/authorMetadata.ts @@ -24,6 +24,7 @@ export function authorProviderKey(author?: Pick { expect(metadataSourceLink('nb:../x', 'book')).toBeNull() }) + it('links BnF book records to their catalogue ARK but not authority-file authors', () => { + expect(metadataSourceLink('bnf:cb48780493c', 'book')).toEqual({ + url: 'https://catalogue.bnf.fr/ark:/12148/cb48780493c', + label: 'BnF', + }) + expect(metadataSourceLink('bnf:author:11885918', 'author')).toBeNull() + expect(metadataSourceLink('bnf:author:11885918', 'book')).toBeNull() + expect(metadataSourceLink('bnf:../x', 'book')).toBeNull() + expect(metadataSourceLink('bnf:cb48780493c', 'author')).toBeNull() + }) + it('returns null for local or malformed provider ids', () => { expect(metadataSourceLink('abs:abc', 'book')).toBeNull() expect(metadataSourceLink('calibre:7', 'book')).toBeNull() @@ -86,6 +97,7 @@ describe('providerDisplayName', () => { expect(providerDisplayName('googlebooks')).toBe('Google Books') expect(providerDisplayName('dnb')).toBe('DNB') expect(providerDisplayName('nb')).toBe('Nasjonalbiblioteket') + expect(providerDisplayName('bnf')).toBe('BnF') expect(providerDisplayName('calibre')).toBe('Calibre') expect(providerDisplayName('audiobookshelf')).toBe('Audiobookshelf') }) @@ -112,6 +124,7 @@ describe('providerFromBookForeignId', () => { expect(providerFromBookForeignId('HC:123')).toBe('hardcover') expect(providerFromBookForeignId('dnb:123')).toBe('dnb') expect(providerFromBookForeignId('nb:a0000000000000000000000000000001')).toBe('nb') + expect(providerFromBookForeignId('bnf:cb48780493c')).toBe('bnf') expect(providerFromBookForeignId('calibre:7')).toBe('calibre') expect(providerFromBookForeignId('abs:lib:item')).toBe('audiobookshelf') }) diff --git a/web/src/util/metadataSource.ts b/web/src/util/metadataSource.ts index 5fce9b15c..f3667f9e4 100644 --- a/web/src/util/metadataSource.ts +++ b/web/src/util/metadataSource.ts @@ -46,8 +46,17 @@ export function metadataSourceLink( return { url: `https://www.nb.no/items/${record}`, label: 'Nasjonalbiblioteket' } } + // Book IDs are the record's ARK, whose public page is the BnF catalogue + // record. Author IDs are authority record numbers; their ARK needs a + // check character Bindery does not store, so authors get no link. + if (kind === 'book' && lowerID.startsWith('bnf:')) { + const ark = lowerID.slice(4) + if (!/^cb[0-9]{8,9}[0-9bcdfghjkmnpqrstvwxz]$/.test(ark)) return null + return { url: `https://catalogue.bnf.fr/ark:/12148/${ark}`, label: 'BnF' } + } + // No reliable public URL for these providers. - if (lowerID.startsWith('hc:') || lowerID.startsWith('dnb:') || lowerID.startsWith('nb:') || lowerID.startsWith('abs:') || lowerID.startsWith('calibre:')) { + if (lowerID.startsWith('hc:') || lowerID.startsWith('dnb:') || lowerID.startsWith('nb:') || lowerID.startsWith('bnf:') || lowerID.startsWith('abs:') || lowerID.startsWith('calibre:')) { return null } @@ -72,6 +81,7 @@ const PROVIDER_NAMES: Record = { googlebooks: 'Google Books', dnb: 'DNB', nb: 'Nasjonalbiblioteket', + bnf: 'BnF', calibre: 'Calibre', audiobookshelf: 'Audiobookshelf', } @@ -92,6 +102,7 @@ export function providerFromBookForeignId(foreignId: string | undefined | null): if (id.startsWith('hc:')) return 'hardcover' if (id.startsWith('dnb:')) return 'dnb' if (id.startsWith('nb:')) return 'nb' + if (id.startsWith('bnf:')) return 'bnf' if (id.startsWith('calibre:')) return 'calibre' if (id.startsWith('abs:')) return 'audiobookshelf' return 'openlibrary' From eb196699a77b98dec3f1f21af0132de3b5ad50ec Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:08:30 -0300 Subject: [PATCH 04/13] docs: BnF provider, its licence terms and the changelog fragment (#3144) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/third-party-data.md gains a Bibliothèque nationale de France section: the Licence Ouverte 2.0 and its credit line, where the retrieval date lives, no endorsement, the CGU carve outs (music records, Dewey authorities) and how Bindery keeps clear of them, no covers, summaries not claimed as open data, names as personal data, and the pacing. README, ARCHITECTURE and the User Guide list BnF as an opt-in primary. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- README.md | 1 + changelog.d/3144-metadata-bnf.md | 2 ++ docs/ARCHITECTURE.md | 11 +++--- docs/User-Guide-Wiki.md | 24 ++++++++++--- docs/third-party-data.md | 61 ++++++++++++++++++++++++++++++++ 5 files changed, 89 insertions(+), 10 deletions(-) create mode 100644 changelog.d/3144-metadata-bnf.md diff --git a/README.md b/README.md index c82801903..b5b142a49 100644 --- a/README.md +++ b/README.md @@ -155,6 +155,7 @@ Repeated provider searches use a bounded five-minute cache, and identical in-fli | [Hardcover.app](https://hardcover.app) | API token (free) | Search enrichment + community ratings, series, wishlist; can be promoted to **primary** for a curated, translation-free catalogue. Token required for **all** queries — without it Hardcover is silently skipped ([troubleshooting](docs/Troubleshooting-Wiki.md#a-book-is-on-hardcoverapp-but-doesnt-show-up-in-the-add-to-library-search)) | | [DNB](https://www.dnb.de/) | None (public SRU) | German-language descriptions, language, year, publisher; can be promoted to **primary** | | [Nasjonalbiblioteket](https://www.nb.no/) | None (public API) | Norwegian catalogues under their original titles (legal deposit), with series and series numbers; opt-in **primary** only, never queried otherwise | +| [BnF](https://catalogue.bnf.fr/) | None (public SRU) | French editions under their French titles (legal deposit), with translator, publisher, year and ISBN; opt-in **primary** only, never queried otherwise; no covers. Data under the Licence Ouverte 2.0 ([credit and terms](docs/third-party-data.md#bibliothèque-nationale-de-france)) | | [Audnex](https://api.audnex.us) | None | Audiobook narrator, duration, cover by ASIN | | [Audible](https://audible.com) | None | Supplemental audiobook author lookup — pulls ASINs OL/Hardcover miss | diff --git a/changelog.d/3144-metadata-bnf.md b/changelog.d/3144-metadata-bnf.md new file mode 100644 index 000000000..fafd48fd8 --- /dev/null +++ b/changelog.d/3144-metadata-bnf.md @@ -0,0 +1,2 @@ +### Added +- **Bibliothèque nationale de France metadata provider** (#3144). The BnF can be chosen as the primary metadata provider in Settings, Metadata Profiles. Legal deposit puts nearly every book published in France in its catalogue under the French edition title, so a French library whose indexers name releases after the French edition ("Alice au pays des merveilles") gets that title instead of the original English one other providers list. Books carry their language, publisher, year, ISBNs and page count, and the original title of a translation; translators and illustrators are never taken for the author. It reads only BnF's printed book records, under the Licence Ouverte 2.0, credited in Settings, and fetches no covers. It is opt in: an install that does not select it never contacts the BnF, and when selected it sends at most one request a second. Thanks to mnivoliez for the request. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 69becebb4..2e82b19f6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -23,10 +23,11 @@ OpenLibrary Google Books, Hardcover.app, Audnex, Audible Exactly one provider is *primary* — it defines what an author's catalogue is. `metadata.primary_provider` selects OpenLibrary (default), DNB, Nasjonalbiblioteket -(`nb`), or Hardcover (token required); every provider that isn't primary is wired -as an enricher, except Nasjonalbiblioteket, which is wired only when it is primary. -Switching the primary away from it therefore leaves `nb:` authors with no provider -to sync from until they are relinked. +(`nb`), the Bibliothèque nationale de France (`bnf`), or Hardcover (token required); +every provider that isn't primary is wired as an enricher, except Nasjonalbiblioteket +and BnF, which are wired only when primary. Switching the primary away from either +therefore leaves its `nb:` or `bnf:` authors with no provider to sync from until +they are relinked. ## Components @@ -51,7 +52,7 @@ The `internal/` tree is organised by domain, not by layer: | `db` | Connection pooling, transaction helpers, repository interfaces, and the embedded schema migrations under `db/migrations/` (`NNN_description.sql`, applied idempotently at boot). | | `migrate` | Bulk-import of authors and related records from a `readarr.db` or a Goodreads CSV export. | | `models` | Domain types (Author, Book, Edition, Series, Indexer, etc.) shared across handlers, repos, and pipelines. | -| `metadata` | OpenLibrary, Google Books, Hardcover, DNB, Nasjonalbiblioteket, Audnex, Audible: fetchers and unifying interfaces. The HTTP providers share one request loop (`metadata/providerhttp`): a 429, 502, 503, 504 or transport error is retried up to 3 times, `Retry-After` is honoured up to 30 seconds with jittered backoff otherwise, and a refusal holds every request that client makes, failing one at once when the hold would outlast its deadline, reported as rate limited after a 429 and as unavailable after a 5xx. A Google Books daily quota refusal (`dailyLimitExceeded`, `quotaExceeded`) is not retried. The CSV, Readarr and Goodreads importers wait out a primary's hold and ask again (up to 3 times per lookup, and at most 10 minutes of waiting per run) instead of failing the row. A lookup still refused after that counts toward their primary outage breaker, so a primary that keeps refusing stops the import after 3 such lookups and the remaining rows fail with a "try again later" reason. The Add Book ISBN and ASIN lookups run under one 20 second deadline. Hardcover paces itself adaptively instead, and treats a 200 whose GraphQL errors report a rate limit as a refusal ([#2369](https://github.com/vavallee/bindery/issues/2369), [#2791](https://github.com/vavallee/bindery/issues/2791)). | +| `metadata` | OpenLibrary, Google Books, Hardcover, DNB, Nasjonalbiblioteket, BnF, Audnex, Audible: fetchers and unifying interfaces. The HTTP providers share one request loop (`metadata/providerhttp`): a 429, 502, 503, 504 or transport error is retried up to 3 times, `Retry-After` is honoured up to 30 seconds with jittered backoff otherwise, and a refusal holds every request that client makes, failing one at once when the hold would outlast its deadline, reported as rate limited after a 429 and as unavailable after a 5xx. A Google Books daily quota refusal (`dailyLimitExceeded`, `quotaExceeded`) is not retried. The CSV, Readarr and Goodreads importers wait out a primary's hold and ask again (up to 3 times per lookup, and at most 10 minutes of waiting per run) instead of failing the row. A lookup still refused after that counts toward their primary outage breaker, so a primary that keeps refusing stops the import after 3 such lookups and the remaining rows fail with a "try again later" reason. The Add Book ISBN and ASIN lookups run under one 20 second deadline. BnF also paces itself at a fixed one request a second, two in flight, for the whole process, because BnF publishes no limit (`metadata/bnf/pacer.go`). Hardcover paces itself adaptively instead, and treats a 200 whose GraphQL errors report a rate limit as a refusal ([#2369](https://github.com/vavallee/bindery/issues/2369), [#2791](https://github.com/vavallee/bindery/issues/2791)). | | `indexer` | Newznab/Torznab clients, query builder, four-tier fallback, per-indexer query deduplication, result deduplication, ranking. | | `decision` | Quality profiles, language filter, custom formats, delay profiles, blocklist consultation. | | `downloader` | SABnzbd, NZBGet, qBittorrent, Transmission, Deluge, rTorrent clients (queue/history polling, submission, deletion). | diff --git a/docs/User-Guide-Wiki.md b/docs/User-Guide-Wiki.md index 3020a728c..07e4040c5 100644 --- a/docs/User-Guide-Wiki.md +++ b/docs/User-Guide-Wiki.md @@ -784,9 +784,23 @@ like everything else on that tab that names server paths. (Settings → Metadata Profiles → Library Defaults) offers OpenLibrary, **DNB** (German National Library), **Nasjonalbiblioteket** (National Library of Norway: Norwegian books under their original titles, only used when chosen - as primary), and **Hardcover**. Switching the primary away from - Nasjonalbiblioteket switches it off entirely, so authors linked to it stop - syncing until you relink each one with "Link metadata" on their page. + as primary), **BnF** (Bibliothèque nationale de France: French editions under + their French titles, translations included, only used when chosen as primary), + and **Hardcover**. Switching the primary away from Nasjonalbiblioteket or BnF + switches it off entirely, so authors linked to it stop syncing until you + relink each one with "Link metadata" on their page. +- **BnF** suits a library of French editions whose indexers name releases + after the French title ("Alice au pays des merveilles", not "Alice's + Adventures in Wonderland"). Its data is under the Licence Ouverte / Open + Licence 2.0, credited in Settings where you choose it; Bindery fetches no + covers from BnF, so covers come from the other providers. BnF is moving its + catalogue to a new record model until the end of 2026, and records created + since June 2026 are not yet linked to their author, so Bindery finds those by + the author's name and treats an author's BnF catalogue as possibly + incomplete: catalogue reconciliation never offers to remove a book because + BnF did not list it. BnF publishes no rate limit, so Bindery sends it at most + one request a second; refreshing an author with a long catalogue can take + around fifteen seconds. - **Hardcover** is an enricher by default — it improves search results, ratings, and series data, and powers import lists and the Discover wishlist row. **Without an API token (Settings → API Keys) Hardcover is silently @@ -870,8 +884,8 @@ bound to with a copy button, and lists any other provider ids the same book is known by. That is the thing to check before deciding a book needs re-binding, and the id is what to quote in a bug report. Hover or activate **Links** while confirming a book in the Add to library dialog or in the book header to open -trustworthy upstream pages for OpenLibrary, Google Books, Hardcover, DNB and -Nasjonalbiblioteket records. +trustworthy upstream pages for OpenLibrary, Google Books, Hardcover, DNB, +Nasjonalbiblioteket and BnF records. Calibre and Audiobookshelf ids remain visible only under **Metadata source** because they do not map to stable public pages. diff --git a/docs/third-party-data.md b/docs/third-party-data.md index 9e8574a36..6573afff4 100644 --- a/docs/third-party-data.md +++ b/docs/third-party-data.md @@ -110,6 +110,67 @@ API) and (the Norwegian authority file). Reviewed A self-hosted install that shows the names to its own users does not redistribute them. +## Bibliothèque nationale de France + +Sources: the BnF catalogue's general terms of use, sections 2 and 4 +(), the Licence Ouverte / Open Licence +2.0 text (), and +the SRU service description (). +Reviewed 2026-10-11. Only contacted when BnF is the primary provider: an +install that does not choose it sends BnF no request at all, not even at +startup. + +- **The catalogue records are under the Licence Ouverte 2.0 (Etalab).** It + allows reuse, including commercial reuse, modification and redistribution, + with no copyleft, and is compatible with CC BY. Its condition is credit: + name the source (at least "Bibliothèque nationale de France") and the date + the data was last updated. Bindery shows this credit in Settings, Metadata + Profiles, where BnF is chosen as the primary provider: + + > Contains data from the Bibliothèque nationale de France under the Licence + > Ouverte / Open Licence 2.0, retrieved . + + The date is when Bindery fetched the record. Each book and author keeps it: + `created_at` when it was added, and `last_metadata_refresh_at` when it was + last refreshed from its provider. A book's ID is the record's ARK + (`bnf:cb48780493c` is ), so + the source record of every BnF book can be found, and the Links menu opens + it. A self-hosted install that shows the data to its own users does not + redistribute it. Anyone who publishes the data, for example as a dataset or + on a public site, should keep the credit line with the retrieval date. +- **No endorsement, nothing misattributed.** The licence forbids suggesting + that BnF endorses the reuse, and BnF does not endorse Bindery. Values Bindery + derives itself (how editions are grouped into works, which edition names the + work, ratings or series from other providers) are Bindery's, not BnF's. +- **Only book records are used.** The terms carve out the third party records + for music audio recordings, and the Dewey classification authority records, + from the open licence. Bindery searches only bibliographic records of + document type "printed text" (`bib.doctype any "a"`), drops any record whose + own type says it is a sound recording, music, an image or a map, and asks the + authority file only for persons (`aut.type any "pep"`). It never queries the + Dewey or subject authorities and does not read subject or classification + fields. +- **Covers are excluded.** Bindery fetches no image from BnF, data.bnf.fr or + Gallica, and ignores the cover links some records carry: Gallica has its own + terms, images reused on Wikimedia Commons carry each image's own licence, and + no terms were found for BnF's cover image service. BnF books get covers only + from the other providers. +- **Summaries are not claimed as open data.** A record's summary is often the + publisher's text, or a commercial database's (Electre), not BnF's + cataloguing, so the open licence may not reach it. Bindery stores a summary + only as the book description, the way it does for the other providers, and + only when the record gives no source or names the publisher; a summary + attributed to any other third party is skipped. +- **Names are personal data.** Authors and translators are people, and their + names and dates come from BnF's authority file. BnF gives the people named + in its catalogue and data.bnf.fr rights of access and correction, exercised + with BnF. Bindery stores the names in the install's own database, like every + other provider's, and sends them nowhere. +- **No published rate limit.** BnF publishes none, so Bindery paces itself: at + most one request a second and two in flight for the whole process, a + `Retry-After` is honoured, and a refusal holds every BnF request. See + `internal/metadata/bnf/pacer.go`. + ## Audible `internal/metadata/audible` calls an unpublished Amazon endpoint, and Amazon's From c54a5b2c735c4c2a508fd9c247aea6d71698cd58 Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:38:58 -0300 Subject: [PATCH 05/13] fix(metadata): BnF drops serials and cuts English responsibility statements (#3144) bib.doctype "a" includes periodicals, and a serial title record carries no 461 link, so a search for "Le Monde diplomatique" read ten serial records as books. Leader position 7 "s" (serial) and "i" (integrating resource) are now rejected; monographs, collections, component parts and the blank level of a 2026 record are kept. Title statements such as "Sylvie and Bruno, by Lewis Carroll" kept their statement of responsibility, and `L. Carrol ["sic"]. Alice au pays des merveilles` became the title "L. Carrol" with the original title "sic". A ", by" or ", par" with no space before the comma is now cut only when what follows names one of the record's people or is a plain cue ("the author of"), so "Ruby, by Moonlight" keeps its comma. A leading name flagged [sic] is dropped when it is a misspelling of one of the record's people, and a bracketed sic is never an original title. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/metadata/bnf/older_records_test.go | 49 +++++++- internal/metadata/bnf/records.go | 129 ++++++++++++++++++-- internal/metadata/bnf/records_test.go | 7 ++ 3 files changed, 172 insertions(+), 13 deletions(-) diff --git a/internal/metadata/bnf/older_records_test.go b/internal/metadata/bnf/older_records_test.go index a131738f5..9de096bf3 100644 --- a/internal/metadata/bnf/older_records_test.go +++ b/internal/metadata/bnf/older_records_test.go @@ -61,9 +61,56 @@ func TestSplitTitleStatement(t *testing.T) { {"M. Ripois et la Némésis", "M. Ripois et la Némésis", ""}, } for _, tc := range cases { - title, original := splitTitleStatement(tc.in) + title, original := splitTitleStatement(tc.in, nil) if title != tc.title || original != tc.original { t.Errorf("splitTitleStatement(%q) = %q, %q; want %q, %q", tc.in, title, original, tc.title, tc.original) } } } + +// English and French ", by" / ", par" statements of responsibility are cut +// only when what follows names one of the record's people or is a plain +// responsibility cue, so a title that merely contains ", by" keeps it. The +// first four inputs are 200 $a values from real BnF records. +func TestSplitTitleStatement_ResponsibilityAfterComma(t *testing.T) { + carroll := []string{"Carroll"} + cases := []struct { + name, in string + names []string + title string + original string + }{ + {"by the record's author", "Sylvie and Bruno, by Lewis Carroll", carroll, "Sylvie and Bruno", ""}, + {"by, then an illustrator statement", "Phantasmagoria and other poems, by Lewis Carroll. With illustrations by Arthur B. Frost", carroll, "Phantasmagoria and other poems", ""}, + {"question marks and spacing kept", "Rhyme ? and reason ?, by Lewis Carroll", carroll, "Rhyme ? and reason ?", ""}, + {"leading name flagged sic", `L. Carrol ["sic"]. Alice au pays des merveilles . Illustrations de J. Sornani...`, carroll, "Alice au pays des merveilles", ""}, + {"par and the record's author", "Le Rouge et le Noir, par Stendhal", []string{"Stendhal"}, "Le Rouge et le Noir", ""}, + {"case insensitive", "Sylvie and Bruno, BY LEWIS CARROLL", carroll, "Sylvie and Bruno", ""}, + {"responsibility cue without a name", "Sylvie and Bruno concluded, by the author of Alice in Wonderland", nil, "Sylvie and Bruno concluded", ""}, + {"title containing , by", "Ruby, by Moonlight", carroll, "Ruby, by Moonlight", ""}, + {"a name inside a word is not the name", "Ruby, by Moonlight", []string{"Moon"}, "Ruby, by Moonlight", ""}, + {"no people on the record", "Sylvie and Bruno, by Lewis Carroll", nil, "Sylvie and Bruno, by Lewis Carroll", ""}, + {"sic is never an original title", `Ce que j'ai vu ["sic"]. Souvenirs`, []string{"Dupont"}, "Ce que j'ai vu", ""}, + {"sic alongside a real original", `Alice au pays des merveilles [sic] ["Alice's adventures in wonderland"]`, carroll, "Alice au pays des merveilles", "Alice's adventures in wonderland"}, + } + for _, tc := range cases { + title, original := splitTitleStatement(tc.in, tc.names) + if title != tc.title || original != tc.original { + t.Errorf("%s: splitTitleStatement(%q) = %q, %q; want %q, %q", tc.name, tc.in, title, original, tc.title, tc.original) + } + } +} + +// The record the review found: its title became "L. Carrol" and its original +// title "sic". +func TestParseRecord_SicStatement(t *testing.T) { + ed, ok := parseRecord(record(t, bookLeader, + df("200", "a", `L. Carrol ["sic"]. Alice au pays des merveilles . Illustrations de J. Sornani...`, "b", "Texte imprimé")+ + df("700", "3", "11885918", "a", "Carroll", "b", "Lewis", "4", "070"))) + if !ok { + t.Fatal("not parsed") + } + if ed.title != "Alice au pays des merveilles" || ed.originalTitle != "" { + t.Errorf("title = %q original = %q, want Alice au pays des merveilles and no original", ed.title, ed.originalTitle) + } +} diff --git a/internal/metadata/bnf/records.go b/internal/metadata/bnf/records.go index 780d65b68..fd57d243a 100644 --- a/internal/metadata/bnf/records.go +++ b/internal/metadata/bnf/records.go @@ -9,6 +9,7 @@ import ( "github.com/vavallee/bindery/internal/isbnutil" "github.com/vavallee/bindery/internal/models" + "github.com/vavallee/bindery/internal/textutil" ) // arkNameRe matches the name part of a BnF catalogue ARK: "cb", eight or nine @@ -94,17 +95,22 @@ func parseRecord(r marcRecord) (edition, bool) { if ark == "" { return edition{}, false } - title := recordTitle(r) + credits := recordCredits(r) + surnames := make([]string, 0, len(credits)) + for _, c := range credits { + surnames = append(surnames, c.surname) + } + title := recordTitle(r, surnames) if title == "" { return edition{}, false } ed := edition{ ark: ark, title: title, - originalTitle: originalTitle(r, title), + originalTitle: originalTitle(r, title, surnames), language: models.NormalizeLanguageCode(r.first("101", "a")), description: recordDescription(r), - credits: recordCredits(r), + credits: credits, } if f := r.first("100", "a"); len(f) >= 8 && allDigits(f[:8]) { ed.entered = f[:8] @@ -132,6 +138,12 @@ func parseRecord(r marcRecord) (edition, bool) { // blank is tolerated because records converted from BnF's new model in // 2026 leave the leader positions empty; anything else (music, sound // recordings, images, maps) is dropped. +// - Leader position 7 is the bibliographic level. A serial ("s") or an +// integrating resource ("i", a loose leaf or a website) is not a book, +// and bib.doctype "a" does include periodicals: a search for "Le Monde +// diplomatique" returned ten serial title records. Monographs ("m"), +// collections ("c"), component parts ("a") and the blank of a 2026 +// record are kept. // - An RDA content type (181 $c), when present, must include "txt". // - A record linked to a periodical by ISSN (461 $x) is an issue of a // magazine, not a book. @@ -146,6 +158,12 @@ func isBookRecord(r marcRecord) bool { return false } } + if len(r.Leader) > 7 { + switch r.Leader[7] { + case 's', 'i': + return false + } + } if content := r.all("181", "c"); len(content) > 0 { text := false for _, c := range content { @@ -179,8 +197,9 @@ func recordARK(r marcRecord) string { // recordTitle is the title proper (200 $a). A volume catalogued as a part of // a larger work carries that work's name in $a and its own title in $i, so // the part title wins: otherwise every volume of a multi-part work would be -// named after the whole. -func recordTitle(r marcRecord) string { +// named after the whole. surnames are the record's people (7XX $a), which +// tell a statement of responsibility from a title (splitTitleStatement). +func recordTitle(r marcRecord, surnames []string) string { fs := r.fields("200") if len(fs) == 0 { return "" @@ -188,7 +207,7 @@ func recordTitle(r marcRecord) string { if part := cleanTitle(fs[0].first("i")); part != "" { return part } - title, _ := splitTitleStatement(fs[0].first("a")) + title, _ := splitTitleStatement(fs[0].first("a"), surnames) return title } @@ -197,12 +216,12 @@ func recordTitle(r marcRecord) string { // uniform title (500 $a) when it differs from the record's own title. BnF's // 2026 records give a French uniform title for a French edition, which is // not an original title, so that case is left out. -func originalTitle(r marcRecord, title string) string { +func originalTitle(r marcRecord, title string, surnames []string) string { if t := cleanTitle(r.first("454", "t")); t != "" { return t } if fs := r.fields("200"); len(fs) > 0 { - if _, orig := splitTitleStatement(fs[0].first("a")); orig != "" { + if _, orig := splitTitleStatement(fs[0].first("a"), surnames); orig != "" { return orig } } @@ -221,23 +240,109 @@ func originalTitle(r marcRecord, title string) string { // opens a statement of responsibility. var statementCutRe = regexp.MustCompile(`(?i)\s\[|\s/\s|\s,\s|\.\s+(?:traduit|traduction|trad\.|adapt|illustr|préf|texte|version|par\s|d'après|introd|notes?\s|avec\s|dessins|images)`) +// byStatementRe finds a ", by" or ", par" with no space before the comma: +// "Sylvie and Bruno, by Lewis Carroll". Unlike " , " this is also how a +// title can read ("Ruby, by Moonlight"), so splitTitleStatement cuts there +// only when what follows names one of the record's people or is a plain +// responsibility cue (respCueRe). +var byStatementRe = regexp.MustCompile(`(?i),\s+(?:by|par)\s+`) + +// respCueRe is a statement of responsibility that names nobody: "by the +// author of Alice in Wonderland", "par l'auteur de ...". +var respCueRe = regexp.MustCompile(`(?i)^(?:the\s+author\s+of\b|l['’]auteur\s+d)`) + // bracketedOriginalRe reads a bracketed original title: ["Alice in Wonderland"]. var bracketedOriginalRe = regexp.MustCompile(`\[\s*"([^"\]]+)"\s*\]`) +// leadingSicRe reads a title statement that opens with a name the +// cataloguer transcribed as printed and flagged: `L. Carrol ["sic"]. Alice au +// pays des merveilles`. Group 1 is the name, group 2 the rest. +var leadingSicRe = regexp.MustCompile(`(?i)^(.{1,60}?)\s*\[\s*"?\s*sic\s*"?\s*\]\s*\.\s+(\S.*)$`) + // splitTitleStatement returns the title proper of a 200 $a and, when the // statement carries one in brackets, the original title. A modern record's // $a is only the title and comes back unchanged. -func splitTitleStatement(s string) (title, original string) { +// +// surnames are the record's people. A ", by " is cut only when the +// name is one of theirs, and a leading name flagged [sic] is dropped only +// when it is a misspelling of one of theirs, so a title is never cut on +// punctuation alone. A bracketed "sic" is the cataloguer's marker, never an +// original title. +func splitTitleStatement(s string, surnames []string) (title, original string) { s = cleanTitle(s) - if m := bracketedOriginalRe.FindStringSubmatch(s); m != nil { - original = cleanTitle(m[1]) + if m := leadingSicRe.FindStringSubmatch(s); m != nil && namesSomeone(m[1], surnames, true) { + s = m[2] + } + for _, m := range bracketedOriginalRe.FindAllStringSubmatch(s, -1) { + if o := cleanTitle(m[1]); o != "" && !strings.EqualFold(o, "sic") { + original = o + break + } } + cut := -1 if loc := statementCutRe.FindStringIndex(s); loc != nil && loc[0] > 0 { - s = cleanTitle(s[:loc[0]]) + cut = loc[0] + } + for _, loc := range byStatementRe.FindAllStringIndex(s, -1) { + if loc[0] == 0 || (cut >= 0 && loc[0] >= cut) { + continue + } + if follow := s[loc[1]:]; respCueRe.MatchString(follow) || namesSomeone(follow, surnames, false) { + cut = loc[0] + break + } + } + if cut > 0 { + s = cleanTitle(s[:cut]) } return s, original } +// namesSomeone reports whether text names one of surnames: every word of a +// surname appears as a whole word of text, ignoring case and accents. With +// misspelt set, a word of four letters or more that a surname word starts +// with, or that starts with one, also counts: the names a cataloguer flags +// [sic] are misspellings ("L. Carrol" for Carroll). +func namesSomeone(text string, surnames []string, misspelt bool) bool { + words := foldWords(text) + for _, surname := range surnames { + sw := foldWords(surname) + if len(sw) == 0 { + continue + } + all := true + for _, w := range sw { + if !hasWord(words, w, misspelt) { + all = false + break + } + } + if all { + return true + } + } + return false +} + +func hasWord(words []string, w string, misspelt bool) bool { + for _, t := range words { + if t == w { + return true + } + if misspelt && len(t) >= 4 && len(w) >= 4 && (strings.HasPrefix(w, t) || strings.HasPrefix(t, w)) { + return true + } + } + return false +} + +// foldWords splits s into case and accent folded words. +func foldWords(s string) []string { + return strings.FieldsFunc(textutil.FoldForSearch(s), func(r rune) bool { + return !unicode.IsLetter(r) && !unicode.IsNumber(r) + }) +} + // cleanTitle strips the non-sorting markers some UNIMARC exports put around a // leading article (U+0098 and U+009C, C1 control characters like every other // control character dropped here), collapses whitespace and trims trailing diff --git a/internal/metadata/bnf/records_test.go b/internal/metadata/bnf/records_test.go index 26d8373b9..ce31e31ef 100644 --- a/internal/metadata/bnf/records_test.go +++ b/internal/metadata/bnf/records_test.go @@ -98,6 +98,13 @@ func TestParseRecord_DropsWhatIsNotABook(t *testing.T) { {"printed music", record(t, " ncm 22 450 ", title), false}, {"spoken word content type", record(t, " a 22 450 ", title+df("181", "c", "spw", "2", "rdacontent")), false}, {"periodical issue", record(t, bookLeader, title+df("461", "t", "J'aime lire", "x", "0399-4600", "v", "398")), false}, + // A serial's own title record (leader position 7 "s") carries no 461 + // link, and doctype "a" includes periodicals: a search for "Le Monde + // diplomatique" came back as ten such records, all read as books. + {"serial title record", record(t, " cas 22 450 ", df("011", "a", "0026-9395")+df("200", "a", "Le Monde diplomatique")+df("210", "a", "Paris", "c", "Le Monde", "d", "1954-")), false}, + {"integrating resource", record(t, " nai 22 450 ", df("200", "a", "Encyclopédie en ligne")), false}, + {"collection of monographs", record(t, " nac 22 450 ", title), true}, + {"component part", record(t, " naa 22 450 ", title), true}, {"no title", record(t, bookLeader, df("010", "a", "978-2-07-062897-1")), false}, } for _, tc := range cases { From 48bdd44d78f5ca87830ca043a5a476295d36f508 Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:41:00 -0300 Subject: [PATCH 06/13] fix(metadata): BnF adopts an unlinked record only when no namesake exists (#3144) The catalogue walk's name query adopted every record crediting, without an authority link, an author named exactly like the heading. Two people named "Martin, Jean" are two authority records, so the other one's new books joined this author's catalogue. Adoption now makes one authority lookup, only when there is something to adopt, and adopts nothing unless this person is the only one with that heading. Both that check and the search time linking now read one page of 20 and treat a lookup that matched more records than it read as inconclusive: the second namesake could be on the page not read. Before, linking read 5 records and could miss it. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/metadata/bnf/client.go | 85 +++++++++++++++++++--------- internal/metadata/bnf/client_test.go | 7 ++- internal/metadata/bnf/link_test.go | 78 +++++++++++++++++++++++++ 3 files changed, 139 insertions(+), 31 deletions(-) diff --git a/internal/metadata/bnf/client.go b/internal/metadata/bnf/client.go index 5986eee99..5f6767a7e 100644 --- a/internal/metadata/bnf/client.go +++ b/internal/metadata/bnf/client.go @@ -136,7 +136,7 @@ func (c *Client) SearchAuthors(ctx context.Context, query string) ([]models.Auth return nil, nil } words := strings.Fields(strings.Trim(term, `"`)) - auths, err := c.searchAuthorities(ctx, term, authorSearchSize) + auths, _, err := c.searchAuthorities(ctx, term, authorSearchSize) if err != nil { return nil, fmt.Errorf("bnf search authors: %w", err) } @@ -216,26 +216,11 @@ func (c *Client) linkUnlinked(ctx context.Context, eds []edition) []edition { names = names[:maxNameLookups] } for _, name := range names { - term := cqlTerm(name.surname + " " + name.forename) - if term == "" { - continue - } - auths, err := c.searchAuthorities(ctx, term, 5) - if err != nil { - continue + ids, err := c.namesakes(ctx, name) + if err != nil || len(ids) != 1 { + continue // no one, or two people with this heading: ambiguous } - id := "" - for _, a := range auths { - if !sameName(a.heading, name) { - continue - } - if id != "" { - id = "" // two people with this heading: ambiguous - break - } - id = a.id - } - resolved[textutil.FoldForSearch(name.sortName())] = id + resolved[textutil.FoldForSearch(name.sortName())] = ids[0] } out := make([]edition, len(eds)) for i, ed := range eds { @@ -257,21 +242,52 @@ func (c *Client) linkUnlinked(ctx context.Context, eds []edition) []edition { return out } -// searchAuthorities searches person authority records by name. Only the -// "pep" (person) type is ever asked for: Bindery has no use for, and never -// queries, the Dewey or subject authority files. -func (c *Client) searchAuthorities(ctx context.Context, term string, maxRecords int) ([]authority, error) { - page, err := c.searchRetrieve(ctx, "aut.accesspoint all "+term+` and aut.type any "pep"`, 1, maxRecords) +// errTooManyNamesakes reports an authority lookup whose matches did not fit +// in one page, so a second person with the same heading may be among those +// not read. +var errTooManyNamesakes = errors.New("bnf: more people match this name than one lookup reads") + +// namesakes returns the authority record numbers of every person whose +// heading is exactly name's, from one lookup. When the lookup matched more +// records than it read, a namesake could be among the rest, so it reports +// errTooManyNamesakes rather than an answer that may be short. +func (c *Client) namesakes(ctx context.Context, name credit) ([]string, error) { + term := cqlTerm(name.surname + " " + name.forename) + if term == "" { + return nil, nil + } + auths, total, err := c.searchAuthorities(ctx, term, authorSearchSize) if err != nil { return nil, err } + if total < 0 || total > authorSearchSize { + return nil, errTooManyNamesakes + } + var ids []string + for _, a := range auths { + if sameName(a.heading, name) { + ids = append(ids, a.id) + } + } + return ids, nil +} + +// searchAuthorities searches person authority records by name, and returns +// the number of records the search matched (-1 when BnF does not say). Only +// the "pep" (person) type is ever asked for: Bindery has no use for, and +// never queries, the Dewey or subject authority files. +func (c *Client) searchAuthorities(ctx context.Context, term string, maxRecords int) ([]authority, int, error) { + page, err := c.searchRetrieve(ctx, "aut.accesspoint all "+term+` and aut.type any "pep"`, 1, maxRecords) + if err != nil { + return nil, 0, err + } var out []authority for _, r := range page.Records { if a, ok := parseAuthority(r.Data.Record); ok { out = append(out, a) } } - return out, nil + return out, page.total(), nil } // GetAuthor resolves "bnf:author:" through the BnF @@ -344,15 +360,28 @@ func (c *Client) GetAuthorWorksSnapshot(ctx context.Context, authorForeignID str if term := cqlTerm(auth.heading.surname + " " + auth.heading.forename); term != "" { q := "bib.author all " + term + bookFilter + ` and bib.creationdate >= "` + noemiCutover + `"` if recent, err := c.searchAll(ctx, q, recentPageSize, maxRecentPages); err == nil { - eds = append(eds, adoptUnlinked(parseRecords(recent...), auth.heading)...) + if adopted := adoptUnlinked(parseRecords(recent...), auth.heading); len(adopted) > 0 && c.soleHolder(ctx, auth.heading) { + eds = append(eds, adopted...) + } } } return groupWorks(dedupeEditions(eds), id), false, nil } +// soleHolder reports whether heading's person is the only one in the +// authority file with that exact heading. A name is not an identity: when a +// namesake exists (two people called "Martin, Jean"), an unlinked credit +// could be either, so nothing is adopted. One lookup, made only when there +// is something to adopt; a failed or inconclusive lookup adopts nothing. +func (c *Client) soleHolder(ctx context.Context, heading credit) bool { + ids, err := c.namesakes(ctx, heading) + return err == nil && len(ids) == 1 && ids[0] == heading.authorityID +} + // adoptUnlinked keeps the editions crediting, without an authority link, an // author whose name is exactly the authority heading's, and links that credit -// to it. Editions already linked to someone are left to author2bib. +// to it. Editions already linked to someone are left to author2bib. The +// caller checks that no namesake shares the heading (soleHolder). func adoptUnlinked(eds []edition, heading credit) []edition { var out []edition for _, ed := range eds { diff --git a/internal/metadata/bnf/client_test.go b/internal/metadata/bnf/client_test.go index 8fc3e5e9e..583ad83c5 100644 --- a/internal/metadata/bnf/client_test.go +++ b/internal/metadata/bnf/client_test.go @@ -306,9 +306,10 @@ func TestGetAuthorWorksSnapshot(t *testing.T) { } sent := fs.sent() // The fixture claims 704 records but the fake server's second page is - // empty, which ends the walk. - if len(sent) != 4 { - t.Errorf("requests = %v, want authority, two author2bib pages and the recent name query", sent) + // empty, which ends the walk. The recent record is adopted only after one + // lookup shows no namesake shares the heading. + if len(sent) != 5 || !strings.HasPrefix(sent[4], `aut.accesspoint all "Carroll Lewis"`) { + t.Errorf("requests = %v, want authority, two author2bib pages, the recent name query and the namesake check", sent) } } diff --git a/internal/metadata/bnf/link_test.go b/internal/metadata/bnf/link_test.go index 0673ad56a..5fa2be01b 100644 --- a/internal/metadata/bnf/link_test.go +++ b/internal/metadata/bnf/link_test.go @@ -113,3 +113,81 @@ func TestLinkUnlinked(t *testing.T) { } } } + +// The catalogue walk's name query finds records created since the Noemi +// cutover that BnF has not linked to anyone. Such a record is adopted into +// the author's catalogue only when no other person in the authority file has +// the same heading, and only through the credit with that exact name. +func TestAdoptUnlinked_NamesakeCheck(t *testing.T) { + person := func(id, forename string) string { + return ` cx a22 45 ` + + `Martin` + forename + `` + } + recent := func(ark, surname, forename string) string { + return ` a 22 450 ` + + `20260702u2026 m y0frey50 ba` + + `Livre de ` + forename + `` + + `` + surname + `` + forename + `Auteur du texte` + + `MartinJeanTraducteur` + + `` + } + wrap := func(records string) string { + return `9` + records + `` + } + for _, tc := range []struct { + name string + authorities string + recent string + want []string + lookups int32 + }{ + {"only this person has the heading", person("11111111", "Jean") + person("33333333", "Jeanne"), recent("20000001", "Martin", "Jean"), []string{"Livre de Jean"}, 1}, + {"two people share the heading", person("11111111", "Jean") + person("22222222", "Jean"), recent("20000001", "Martin", "Jean"), nil, 1}, + {"the namesake alone is found", person("22222222", "Jean"), recent("20000001", "Martin", "Jean"), nil, 1}, + // Jean Martin is only the translator here and the author is someone + // else, so nothing is adopted and no lookup is spent. + {"another person's book", person("11111111", "Jean"), recent("20000002", "Durand", "Paul"), nil, 0}, + // More people match the name than one lookup reads: a namesake may + // be on the next page, so nothing is adopted. + {"more matches than one lookup reads", person("11111111", "Jean") + ``, recent("20000001", "Martin", "Jean"), nil, 1}, + } { + var lookups atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query().Get("query") + if r.URL.Query().Get("startRecord") != "1" { + _, _ = w.Write([]byte(emptySRU)) + return + } + switch { + case strings.Contains(q, "aut.recordid"): + _, _ = w.Write([]byte(wrap(person("11111111", "Jean")))) + case strings.Contains(q, "aut.accesspoint"): + lookups.Add(1) + body := wrap(tc.authorities) + if strings.Contains(tc.authorities, "") { + body = strings.Replace(body, "9<", "57<", 1) + } + _, _ = w.Write([]byte(body)) + case strings.Contains(q, "bib.creationdate"): + _, _ = w.Write([]byte(wrap(tc.recent))) + default: + _, _ = w.Write([]byte(emptySRU)) + } + })) + books, _, err := testClient(srv.URL).GetAuthorWorksSnapshot(context.Background(), "bnf:author:11111111") + srv.Close() + if err != nil { + t.Fatalf("%s: %v", tc.name, err) + } + var got []string + for _, b := range books { + got = append(got, b.Title) + } + if strings.Join(got, "|") != strings.Join(tc.want, "|") { + t.Errorf("%s: books = %v, want %v", tc.name, got, tc.want) + } + if lookups.Load() != tc.lookups { + t.Errorf("%s: authority lookups = %d, want %d", tc.name, lookups.Load(), tc.lookups) + } + } +} From beec36202470314b50cee91c45f2468a5985e1bf Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:43:33 -0300 Subject: [PATCH 07/13] test(metadata): cover BnF guards that mutations survived (#3144) Each test below fails with its guard disabled on a compiling tree: - the shared refusal gate and pacer are compared with the package defaults, so a New() without the gate (two nil gates are equal) fails - the French edition represents a work even when another language's edition was entered earlier and comes first - GetBook adds the work's other editions (workOf) - cqlTerm replaces control characters strings.Fields does not split on (NUL, SOH, DEL, the C1 markers) - the pacer's concurrency slot is held until the response body is closed Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/metadata/bnf/client_test.go | 47 +++++++++++++++++++++++++ internal/metadata/bnf/ratelimit_test.go | 41 +++++++++++++++++++-- internal/metadata/bnf/records_test.go | 32 +++++++++++++++++ 3 files changed, 117 insertions(+), 3 deletions(-) diff --git a/internal/metadata/bnf/client_test.go b/internal/metadata/bnf/client_test.go index 583ad83c5..3668358f9 100644 --- a/internal/metadata/bnf/client_test.go +++ b/internal/metadata/bnf/client_test.go @@ -345,6 +345,53 @@ func TestGetBook(t *testing.T) { } } +// GetBook returns the whole work: the requested record plus the other +// editions found by author and title, under the ID that was asked for. +func TestGetBook_AddsTheWorksOtherEditions(t *testing.T) { + rec := func(ark, entered, isbn string) string { + return ` nam 22 450 ` + + `` + isbn + `` + + `` + entered + `d2000 m y0frey50 ba` + + `fre` + + `Alice au pays des merveilles` + + `11885918CarrollLewis070` + + `` + } + wrap := func(records string) string { + return `2` + records + `` + } + fs := &fixtureServer{} + fs.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query().Get("query") + fs.mu.Lock() + fs.queries = append(fs.queries, q) + fs.mu.Unlock() + switch { + case strings.Contains(q, "bib.persistentid"): + _, _ = w.Write([]byte(wrap(rec("cb20000002b", "20050101", "978-2-07-000001-0")))) + case strings.Contains(q, "bib.author2bib"): + _, _ = w.Write([]byte(wrap(rec("cb20000001c", "19900101", "978-2-07-000002-7") + rec("cb20000002b", "20050101", "978-2-07-000001-0")))) + default: + _, _ = w.Write([]byte(emptySRU)) + } + })) + defer fs.Close() + b, err := testClient(fs.URL).GetBook(context.Background(), "bnf:cb20000002b") + if err != nil || b == nil { + t.Fatalf("book = %+v err = %v", b, err) + } + if b.ForeignID != "bnf:cb20000002b" { + t.Errorf("ForeignID = %q, want the requested record's", b.ForeignID) + } + if len(b.Editions) != 2 { + t.Errorf("editions = %d, want the requested record and the other edition of the work", len(b.Editions)) + } + sent := fs.sent() + if len(sent) != 2 || sent[1] != `bib.author2bib any "11885918" and bib.title all "Alice au pays des merveilles" and bib.doctype any "a"` { + t.Errorf("queries = %v, want the record, then its work by author and title", sent) + } +} + func TestEmptyResultIsNotAnError(t *testing.T) { fs := newFixtureServer(t, func(url.Values) string { return "empty.xml" }) c := testClient(fs.URL) diff --git a/internal/metadata/bnf/ratelimit_test.go b/internal/metadata/bnf/ratelimit_test.go index 728b90365..2da1075cf 100644 --- a/internal/metadata/bnf/ratelimit_test.go +++ b/internal/metadata/bnf/ratelimit_test.go @@ -32,11 +32,13 @@ func TestNewClientsShareOnePacerAndGate(t *testing.T) { if !ok { t.Fatalf("transport = %T, want the pacer in front of every request", a.http.Transport) } - if pb := b.http.Transport.(*pacedTransport); pa.pacer != pb.pacer || pa.pacer != defaultPacer { + // Compared with the package defaults, not only with each other: two nil + // gates are equal, and a nil gate holds nothing while BnF refuses (#3100). + if pb := b.http.Transport.(*pacedTransport); pa.pacer == nil || pa.pacer != defaultPacer || pb.pacer != defaultPacer { t.Error("clients have their own pacers; the limit must hold for the process") } - if a.gate != b.gate { - t.Error("clients have their own refusal gates") + if a.gate == nil || a.gate != defaultGate || b.gate != defaultGate { + t.Errorf("gates = %p, %p, want both the shared refusal gate %p", a.gate, b.gate, defaultGate) } if defaultPacer.interval < time.Second || cap(defaultPacer.slots) > 2 { t.Errorf("default pacing = %v, %d concurrent; want at most one start a second and two in flight", @@ -116,6 +118,39 @@ func TestPacer_CapsConcurrentRequests(t *testing.T) { } } +// The concurrency slot is held until the response body is closed, not just +// until the headers arrive: a slow download still counts as in flight. +func TestPacer_SlotHeldUntilBodyClosed(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(emptySRU)) + })) + defer srv.Close() + tr := &pacedTransport{base: http.DefaultTransport, pacer: newPacer(0, 1)} + get := func(ctx context.Context) (*http.Response, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, srv.URL, nil) + if err != nil { + t.Fatal(err) + } + return tr.RoundTrip(req) + } + first, err := get(context.Background()) + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(context.Background(), 150*time.Millisecond) + defer cancel() + if resp, err := get(ctx); err == nil { + _ = resp.Body.Close() + t.Error("second request started while the first body was still open; want it held for the slot") + } + _ = first.Body.Close() + second, err := get(context.Background()) + if err != nil { + t.Fatalf("after the first body closed: %v", err) + } + _ = second.Body.Close() +} + // A request that could only start after its caller's deadline is refused at // once as rate limited, instead of sleeping into a timeout. func TestPacer_RefusesPastTheDeadlineAsRateLimited(t *testing.T) { diff --git a/internal/metadata/bnf/records_test.go b/internal/metadata/bnf/records_test.go index ce31e31ef..ec45b227b 100644 --- a/internal/metadata/bnf/records_test.go +++ b/internal/metadata/bnf/records_test.go @@ -203,6 +203,22 @@ func TestGroupWorks_TranslationsOfOneOriginalAreOneWork(t *testing.T) { } } +// The French edition represents the work even when an edition in another +// language was entered on file earlier and comes first in the results. +func TestGroupWorks_FrenchEditionRepresentsTheWork(t *testing.T) { + books := groupWorks([]edition{ + edWith("cb10000001a", "Alice's adventures in Wonderland", "", "eng", "19800101", carroll), + edWith("cb10000002b", "Alice au pays des merveilles", "Alice's adventures in Wonderland", "fre", "20000101", carroll), + edWith("cb10000003c", "Alicia en el país de las maravillas", "Alice's adventures in Wonderland", "spa", "19700101", carroll), + }, "") + if len(books) != 1 { + t.Fatalf("works = %d, want 1", len(books)) + } + if b := books[0]; b.ForeignID != "bnf:cb10000002b" || b.Title != "Alice au pays des merveilles" || b.Language != "fre" { + t.Errorf("work = %s %q %s, want the French edition bnf:cb10000002b", b.ForeignID, b.Title, b.Language) + } +} + func TestGroupWorks_TranslatorIsNeverTheAuthor(t *testing.T) { unlinked := credit{surname: "Carroll", forename: "Lewis", author: true} books := groupWorks([]edition{edWith("cb10000001a", "Alice au pays des merveilles", "", "fre", "", unlinked, bue)}, "") @@ -264,6 +280,22 @@ func TestCQLTerm(t *testing.T) { t.Errorf("cqlTerm(%q) = %q, want %q", in, got, want) } } + // Control characters that strings.Fields does not split on (NUL, SOH, + // DEL, the C1 non-sorting markers) are only removed by the replacement + // itself. Vertical tab, form feed, NEL and the line separator are + // whitespace to strings.Fields as well; they are here so a change to + // either step is seen. + controls := map[string]string{ + "nul\x00byte": `"nul byte"`, + "soh\x01del\x7fend": `"soh del end"`, + "\u0098Les \u009cmots": `"Les mots"`, + "vt\vff\fnel\u0085ls
x": `"vt ff nel ls x"`, + } + for in, want := range controls { + if got := cqlTerm(in); got != want { + t.Errorf("cqlTerm(%q) = %q, want %q", in, got, want) + } + } long := strings.Repeat("a", 500) if got := cqlTerm(long); len(got) != maxTermRunes+2 { t.Errorf("long term length = %d, want capped at %d", len(got), maxTermRunes+2) From d7a897b752a9edaf6593b559445809e9515bbbbe Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:43:33 -0300 Subject: [PATCH 08/13] test(cmd): wire the primary metadata provider through a tested function (#3144) The provider switch sat inline in main(), so dropping its "bnf" case left every test green while a BnF setting booted OpenLibrary. It moves to primaryMetadataProvider, unchanged, with a test that each accepted value builds that provider. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- cmd/bindery/main.go | 22 +------------------- cmd/bindery/metadata_providers.go | 28 ++++++++++++++++++++++++++ cmd/bindery/metadata_providers_test.go | 27 ++++++++++++++++++++++++- 3 files changed, 55 insertions(+), 22 deletions(-) diff --git a/cmd/bindery/main.go b/cmd/bindery/main.go index 75ce226a3..f23d7ee1b 100644 --- a/cmd/bindery/main.go +++ b/cmd/bindery/main.go @@ -38,11 +38,9 @@ import ( "github.com/vavallee/bindery/internal/jobs" "github.com/vavallee/bindery/internal/logbuf" "github.com/vavallee/bindery/internal/metadata" - "github.com/vavallee/bindery/internal/metadata/bnf" "github.com/vavallee/bindery/internal/metadata/dnb" "github.com/vavallee/bindery/internal/metadata/googlebooks" "github.com/vavallee/bindery/internal/metadata/hardcover" - "github.com/vavallee/bindery/internal/metadata/nb" "github.com/vavallee/bindery/internal/metadata/openlibrary" "github.com/vavallee/bindery/internal/metrics" "github.com/vavallee/bindery/internal/models" @@ -321,25 +319,7 @@ func main() { configuredPrimary = s.Value } primaryName := resolveMetadataPrimaryProvider(configuredPrimary, api.GetHardcoverAPIToken(ctxBoot, settingsRepo) != "") - var primaryProvider metadata.Provider - switch primaryName { - case "dnb": - primaryProvider = dnbClient - case "nb": - // Opt-in only: unlike the others, NB is never added as an enricher, - // so an install that did not choose it sends NB no traffic. NB - // publishes no rate limits for a fleet of independent installs. - primaryProvider = nb.New() - case "bnf": - // Opt-in only, like NB: never an enricher, so an install that did not - // choose it sends BnF no traffic. BnF publishes no rate limits; the - // client paces itself at one request a second (internal/metadata/bnf). - primaryProvider = bnf.New() - case "hardcover": - primaryProvider = hcClient - default: - primaryProvider = olClient - } + primaryProvider := primaryMetadataProvider(primaryName, olClient, dnbClient, hcClient) slog.Info("metadata primary provider", "provider", primaryName) var enrichers []metadata.Provider diff --git a/cmd/bindery/metadata_providers.go b/cmd/bindery/metadata_providers.go index c5ce6e293..9e3310f62 100644 --- a/cmd/bindery/metadata_providers.go +++ b/cmd/bindery/metadata_providers.go @@ -4,8 +4,36 @@ import ( "log/slog" "github.com/vavallee/bindery/internal/api" + "github.com/vavallee/bindery/internal/metadata" + "github.com/vavallee/bindery/internal/metadata/bnf" + "github.com/vavallee/bindery/internal/metadata/nb" ) +// primaryMetadataProvider is the provider wired as primary for name, a value +// resolveMetadataPrimaryProvider returned. The clients that are also +// enrichers are passed in so primary and enricher share one instance; NB and +// BnF are built here because they are never enrichers. +func primaryMetadataProvider(name string, openLibrary, dnbClient, hardcover metadata.Provider) metadata.Provider { + switch name { + case "dnb": + return dnbClient + case "nb": + // Opt-in only: unlike the others, NB is never added as an enricher, + // so an install that did not choose it sends NB no traffic. NB + // publishes no rate limits for a fleet of independent installs. + return nb.New() + case "bnf": + // Opt-in only, like NB: never an enricher, so an install that did not + // choose it sends BnF no traffic. BnF publishes no rate limits; the + // client paces itself at one request a second (internal/metadata/bnf). + return bnf.New() + case "hardcover": + return hardcover + default: + return openLibrary + } +} + // metadataPrimaryProviderDefault is the provider used when // metadata.primary_provider is empty, unknown, or unusable. const metadataPrimaryProviderDefault = "openlibrary" diff --git a/cmd/bindery/metadata_providers_test.go b/cmd/bindery/metadata_providers_test.go index 914619b92..25839500b 100644 --- a/cmd/bindery/metadata_providers_test.go +++ b/cmd/bindery/metadata_providers_test.go @@ -1,6 +1,31 @@ package main -import "testing" +import ( + "testing" + + "github.com/vavallee/bindery/internal/metadata/bnf" + "github.com/vavallee/bindery/internal/metadata/dnb" + "github.com/vavallee/bindery/internal/metadata/hardcover" + "github.com/vavallee/bindery/internal/metadata/openlibrary" +) + +// TestPrimaryMetadataProvider covers the wiring after resolution: each +// accepted value builds that provider, so a value the settings accept never +// silently boots OpenLibrary instead. +func TestPrimaryMetadataProvider(t *testing.T) { + ol, dnbClient, hc := openlibrary.New(), dnb.New(), hardcover.New() + for _, name := range []string{"openlibrary", "dnb", "nb", "bnf", "hardcover"} { + if got := primaryMetadataProvider(name, ol, dnbClient, hc); got == nil || got.Name() != name { + t.Errorf("primaryMetadataProvider(%q) = %v, want the %s provider", name, got, name) + } + } + if _, ok := primaryMetadataProvider("bnf", ol, dnbClient, hc).(*bnf.Client); !ok { + t.Error("bnf is not wired to the BnF client") + } + if got := primaryMetadataProvider("dnb", ol, dnbClient, hc); got != dnbClient { + t.Error("dnb primary is a second DNB client, want the enricher's instance") + } +} // TestResolveMetadataPrimaryProvider covers the boot-time provider selection: // openlibrary stays the default for every unset / unusable value, dnb and From 014167c13126a7ecde0e9c0e4edbbfe3eba039e7 Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:44:19 -0300 Subject: [PATCH 09/13] fix(metadata): compose BnF query terms to NFC (#3144) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit cqlTerm passed decomposed input through, so "Les Misérables" was sent as e plus a combining accent and counted the accent against the 200 rune cap. It is composed first now. golang.org/x/text is already a direct dependency; go.mod and go.sum are unchanged. Checked live with one request each: BnF folds both forms itself (both queries were echoed as "Les Miserables" and matched 638 records), so this changes no results today. It keeps the query in the form BnF's records use and the cap counting letters. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/metadata/bnf/cql.go | 7 +++++++ internal/metadata/bnf/records_test.go | 11 +++++++++++ 2 files changed, 18 insertions(+) diff --git a/internal/metadata/bnf/cql.go b/internal/metadata/bnf/cql.go index bfef5d200..dd1117f46 100644 --- a/internal/metadata/bnf/cql.go +++ b/internal/metadata/bnf/cql.go @@ -3,6 +3,8 @@ package bnf import ( "strings" "unicode" + + "golang.org/x/text/unicode/norm" ) // maxTermRunes bounds a search term. A title longer than this is cut, which @@ -23,7 +25,12 @@ const maxTermRunes = 200 // keyword such as "or" inside the quotes is just a word. The apostrophe of // an elided article ("L'outsider") is kept: it is not special in CQL and BnF // matches it. +// +// The text is composed (NFC) first: a title typed or pasted on macOS can +// arrive decomposed, "e" plus a combining accent, while BnF's records are +// composed, and the rune cap below should count letters, not marks. func cqlTerm(s string) string { + s = norm.NFC.String(s) var b strings.Builder n := 0 for _, r := range s { diff --git a/internal/metadata/bnf/records_test.go b/internal/metadata/bnf/records_test.go index ec45b227b..c5c25e89d 100644 --- a/internal/metadata/bnf/records_test.go +++ b/internal/metadata/bnf/records_test.go @@ -296,6 +296,17 @@ func TestCQLTerm(t *testing.T) { t.Errorf("cqlTerm(%q) = %q, want %q", in, got, want) } } + // Decomposed input (what macOS hands back for a typed or pasted title) + // becomes the composed query, the form BnF's records use. + for in, want := range map[string]string{ + "Les Misérables": "\"Les Misérables\"", + "Œuvres complètes": "\"Œuvres complètes\"", + "Céline L'étranger": "\"Céline L'étranger\"", + } { + if got := cqlTerm(in); got != want { + t.Errorf("cqlTerm(%q) = %q (%d bytes), want the NFC %q (%d bytes)", in, got, len(got), want, len(want)) + } + } long := strings.Repeat("a", 500) if got := cqlTerm(long); len(got) != maxTermRunes+2 { t.Errorf("long term length = %d, want capped at %d", len(got), maxTermRunes+2) From 708bd943261c6ae263db968f83dff3c36fcaf374 Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:45:50 -0300 Subject: [PATCH 10/13] fix(metadata): bound BnF XML nesting and fold accents in work grouping (#3144) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A 4 MiB body of nested elements grew encoding/xml's element stack to about 256 MiB before failing. Decoding now goes through a token reader that refuses nesting past 64 levels (a real SRU response nests about eight), and the test measures the allocation. The byte order mark skip is removed: encoding/xml already skips a leading BOM, and the BOM test passes without it. Work grouping keyed titles on CanonicalDedupKey alone, which keeps accents and ligatures, so "Les Misérables" and "les miserables", or "Œuvres" and "Oeuvres", became two works. The key now also goes through textutil.FoldForSearch, the fold the search keys use. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/metadata/bnf/client.go | 39 ++++++++++++++++++++++----- internal/metadata/bnf/client_test.go | 26 ++++++++++++++++++ internal/metadata/bnf/records_test.go | 21 +++++++++++++++ internal/metadata/bnf/works.go | 9 ++++--- 4 files changed, 85 insertions(+), 10 deletions(-) diff --git a/internal/metadata/bnf/client.go b/internal/metadata/bnf/client.go index 5f6767a7e..79be62abe 100644 --- a/internal/metadata/bnf/client.go +++ b/internal/metadata/bnf/client.go @@ -26,7 +26,6 @@ package bnf import ( - "bufio" "context" "encoding/xml" "errors" @@ -81,6 +80,10 @@ const ( // off here fails to decode and is reported as an error, never read as a // short result. maxResponseBytes = 4 << 20 + // maxXMLDepth bounds element nesting in a response. A real SRU response + // nests about eight deep; without a bound, a 4 MiB body of nested + // elements grows the decoder's element stack to about 260 MiB. + maxXMLDepth = 64 // requestTimeout bounds one request, pacing wait included. requestTimeout = 20 * time.Second ) @@ -615,14 +618,36 @@ func classify(err error) error { } } -// decodeXML decodes r, skipping a UTF-8 byte order mark (BnF's Explain -// response starts with one). +// errTooDeep reports a response nested deeper than maxXMLDepth. +var errTooDeep = errors.New("bnf: response elements nest deeper than any SRU response") + +// decodeXML decodes r with element nesting bounded by maxXMLDepth, so a +// hostile or broken body is refused before the decoder's element stack grows +// with it. (encoding/xml skips a leading byte order mark itself.) func decodeXML(r io.Reader, out any) error { - br := bufio.NewReader(r) - if b, err := br.Peek(3); err == nil && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF { - _, _ = br.Discard(3) + return xml.NewTokenDecoder(&depthLimiter{d: xml.NewDecoder(r), max: maxXMLDepth}).Decode(out) +} + +// depthLimiter passes tokens through and fails once elements nest deeper +// than max. +type depthLimiter struct { + d *xml.Decoder + depth int + max int +} + +func (l *depthLimiter) Token() (xml.Token, error) { + tok, err := l.d.Token() + switch tok.(type) { + case xml.StartElement: + l.depth++ + if l.depth > l.max { + return nil, errTooDeep + } + case xml.EndElement: + l.depth-- } - return xml.NewDecoder(br).Decode(out) + return tok, err } func parseRecords(pages ...*sruResponse) []edition { diff --git a/internal/metadata/bnf/client_test.go b/internal/metadata/bnf/client_test.go index 3668358f9..5346e0a7d 100644 --- a/internal/metadata/bnf/client_test.go +++ b/internal/metadata/bnf/client_test.go @@ -8,6 +8,7 @@ import ( "net/url" "os" "path/filepath" + "runtime" "strings" "sync" "testing" @@ -435,6 +436,31 @@ func TestGarbageBodyIsAnError(t *testing.T) { } } +// A body of nested elements is refused once it passes maxXMLDepth, before +// the decoder's element stack grows: a 4 MiB nest of cost about 260 MiB +// of allocation without the limit. +func TestDeeplyNestedBodyIsRefused(t *testing.T) { + body := "" + strings.Repeat("", (maxResponseBytes-64)/3) + var before, after runtime.MemStats + runtime.GC() + runtime.ReadMemStats(&before) + var out sruResponse + err := decodeXML(strings.NewReader(body), &out) + runtime.ReadMemStats(&after) + if err == nil || !strings.Contains(err.Error(), "nest") { + t.Errorf("err = %v, want the nesting refused", err) + } + if alloc := after.TotalAlloc - before.TotalAlloc; alloc > 32<<20 { + t.Errorf("decoding allocated %d MiB, want the nest refused before it costs more than 32 MiB", alloc>>20) + } + // Real responses nest about eight deep and decode as before. + if err := decodeXML(strings.NewReader(string(fixture(t, "search_alice.xml"))), &out); err != nil || len(out.Records) == 0 { + t.Errorf("real response: %d records, err = %v", len(out.Records), err) + } +} + +// encoding/xml skips a leading byte order mark itself (BnF's Explain +// response starts with one), so decodeXML has no code of its own for it. func TestByteOrderMarkIsSkipped(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write(append([]byte{0xEF, 0xBB, 0xBF}, fixture(t, "isbn_miroir.xml")...)) diff --git a/internal/metadata/bnf/records_test.go b/internal/metadata/bnf/records_test.go index c5c25e89d..a65557e41 100644 --- a/internal/metadata/bnf/records_test.go +++ b/internal/metadata/bnf/records_test.go @@ -203,6 +203,27 @@ func TestGroupWorks_TranslationsOfOneOriginalAreOneWork(t *testing.T) { } } +// Accents, ligatures and case are cataloguing variation, not different +// works: older converted records often drop them. +func TestGroupWorks_TitleKeyFoldsAccentsAndLigatures(t *testing.T) { + books := groupWorks([]edition{ + edWith("cb10000001b", "Les Misérables", "", "fre", "19900101", carroll), + edWith("cb10000002c", "les miserables", "", "fre", "19950101", carroll), + edWith("cb10000003d", "Œuvres complètes", "", "fre", "19900101", carroll), + edWith("cb10000004f", "Oeuvres completes", "", "fre", "19950101", carroll), + edWith("cb10000005g", "Les Misérables", "", "fre", "19900101", bue), + }, "") + if len(books) != 3 { + for _, b := range books { + t.Logf("work %s %q with %d editions", b.ForeignID, b.Title, len(b.Editions)) + } + t.Fatalf("works = %d, want 3: one per title per author", len(books)) + } + if len(books[0].Editions) != 2 || len(books[1].Editions) != 2 { + t.Errorf("editions = %d, %d, want the spelling variants grouped", len(books[0].Editions), len(books[1].Editions)) + } +} + // The French edition represents the work even when an edition in another // language was entered on file earlier and comes first in the results. func TestGroupWorks_FrenchEditionRepresentsTheWork(t *testing.T) { diff --git a/internal/metadata/bnf/works.go b/internal/metadata/bnf/works.go index 2dfd0094b..0ae6ccdf6 100644 --- a/internal/metadata/bnf/works.go +++ b/internal/metadata/bnf/works.go @@ -306,11 +306,14 @@ func authorKey(c *credit) string { } // titleKey is the grouping key for a title. Spaces are dropped so a -// cataloguing slip that splits a word stays with its work, and a typographic -// apostrophe is the same as a straight one ("L’outsider", "L'outsider"). +// cataloguing slip that splits a word stays with its work, accents, +// ligatures and case are folded with the search key's fold (older converted +// records drop them: "Les Misérables" and "les miserables", "Œuvres" and +// "Oeuvres" are one work), and a typographic apostrophe is the same as a +// straight one ("L’outsider", "L'outsider"). func titleKey(title string) string { title = strings.ReplaceAll(title, "\u2019", "'") - return strings.ReplaceAll(indexer.CanonicalDedupKey(title), " ", "") + return strings.ReplaceAll(textutil.FoldForSearch(indexer.CanonicalDedupKey(title)), " ", "") } // unionFind joins title keys that name the same work. From 891e08f559d0e9762e9ad211ec957846472254eb Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:48:21 -0300 Subject: [PATCH 11/13] fix(api): judge BnF works on their own editions during author refresh (#3144) With MinPages or SkipMissingISBN set, author refresh called GetEditions for every new work. For BnF that is a full GetBook: the ARK lookup, up to two name lookups and the work search, at one request a second, so a refresh of Lewis Carroll (about 49 works) cost 100 to 200 requests and around three minutes. BnF works arrive from the author catalogue with every edition attached, the same records the lookup would fetch again, so the prefetch uses them as catalogue reconciliation already does. A work with none attached still goes to the lookup. The filters reach the same verdicts; the test checks both filters and that no lookup is made. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- internal/api/authors.go | 13 +++++++ internal/api/authors_test.go | 74 ++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) diff --git a/internal/api/authors.go b/internal/api/authors.go index 0f09e66e4..348511954 100644 --- a/internal/api/authors.go +++ b/internal/api/authors.go @@ -2333,6 +2333,19 @@ func (h *AuthorHandler) runCatalogueSync(ctx context.Context, author *models.Aut editionsByForeignID = make(map[string][]models.Edition, len(prefetch)) var mu sync.Mutex concurrency.RunBounded(ctx, prefetch, authorAutoSearchConcurrency, func(ctx context.Context, b models.Book) { + // BnF works come from the author catalogue with every edition + // already attached, and BnF's per work editions lookup is a full + // GetBook: up to four requests at one a second, so a large + // author cost minutes. The attached editions are the same + // records, so they are the evidence (as in catalogue + // reconciliation). A work with none attached still goes to the + // lookup. + if bookProvider(&b) == "bnf" && len(b.Editions) > 0 { + mu.Lock() + editionsByForeignID[b.ForeignID] = b.Editions + mu.Unlock() + return + } editions, err := h.meta.GetEditions(ctx, b.ForeignID) if err != nil { slog.Debug("edition lookup failed while checking MinPages/SkipMissingISBN; not enforcing for this work", diff --git a/internal/api/authors_test.go b/internal/api/authors_test.go index b346548e1..fc9ed738f 100644 --- a/internal/api/authors_test.go +++ b/internal/api/authors_test.go @@ -7942,6 +7942,80 @@ func TestFetchAuthorBooks_MinPagesKeptWhenZero(t *testing.T) { } } +// TestFetchAuthorBooks_BnFWorksJudgedOnTheirOwnEditions: BnF's author +// catalogue arrives with every edition attached, and its per work editions +// lookup is a full GetBook (up to four paced requests, one a second). So +// MinPages and SkipMissingISBN judge a BnF work on the editions it carries, +// with no GetEditions call, and reach the same verdicts. +func TestFetchAuthorBooks_BnFWorksJudgedOnTheirOwnEditions(t *testing.T) { + database, err := db.OpenMemory() + if err != nil { + t.Fatal(err) + } + defer database.Close() + + authorRepo := db.NewAuthorRepo(database) + bookRepo := db.NewBookRepo(database) + profileRepo := db.NewMetadataProfileRepo(database) + settingsRepo := db.NewSettingsRepo(database) + ctx := context.Background() + + profile, err := profileRepo.GetByID(ctx, models.DefaultMetadataProfileID) + if err != nil || profile == nil { + t.Fatalf("GetByID(default profile) failed: %v", err) + } + profile.MinPages = 200 + profile.SkipMissingISBN = true + profile.AllowedLanguages = "fre" + if err := profileRepo.Update(ctx, profile); err != nil { + t.Fatal(err) + } + author := &models.Author{ + ForeignID: "bnf:author:11885918", Name: "Lewis Carroll", SortName: "Carroll, Lewis", + MetadataProvider: "bnf", Monitored: false, + } + if err := authorRepo.Create(ctx, author); err != nil { + t.Fatal(err) + } + work := func(ark, title string, pages int, isbn string) models.Book { + ed := models.Edition{ForeignID: "bnf:" + ark, Title: title, NumPages: intPtr(pages)} + if isbn != "" { + ed.ISBN13 = strPtr(isbn) + } + return models.Book{ForeignID: "bnf:" + ark, Title: title, SortTitle: title, Language: "fre", + MediaType: models.MediaTypeEbook, Status: models.BookStatusWanted, MetadataProvider: "bnf", + Editions: []models.Edition{ed}} + } + works := []models.Book{ + work("cb10000001b", "Court", 50, "9782070000017"), + work("cb10000002c", "Long", 350, "9782070000024"), + work("cb10000003d", "Sans ISBN", 300, ""), + } + provider := &stubMetaProvider{name: "bnf", works: works} + agg := metadata.NewAggregator(provider) + h := NewAuthorHandler(authorRepo, nil, bookRepo, nil, agg, settingsRepo, profileRepo, nil) + h.FetchAuthorBooks(author, false, models.MediaTypeEbook) + + got, err := bookRepo.ListByAuthor(ctx, author.ID) + if err != nil { + t.Fatal(err) + } + var titles []string + for _, b := range got { + titles = append(titles, b.Title) + } + if len(titles) != 1 || titles[0] != "Long" { + t.Errorf("created %v, want only Long: Court is under the page floor and Sans ISBN has no ISBN", titles) + } + if calls := provider.editionCalls; len(calls) != 0 { + t.Errorf("GetEditions called for %v, want none: BnF works carry their editions", calls) + } + summary := h.syncSummaries.get(author.ID) + if summary == nil || summary.SkippedMinPages != 1 || summary.SkippedMissingISBN != 1 { + t.Errorf("summary = %+v, want one skipped for pages and one for the missing ISBN", summary) + } +} + // TestFetchAuthorBooks_EditionLookupSkipsFreeFilterRejects verifies that the // edition prefetch only runs for works surviving the free (in-memory) // filters ahead of it — a junk-title work and a wrong-language work should From d062e976a04b229c6d7340a16c421bd3062f2620 Mon Sep 17 00:00:00 2001 From: vavallee Date: Sun, 11 Oct 2026 14:54:14 -0300 Subject: [PATCH 12/13] feat(web): show the BnF credit with its retrieval date (#3144) The Licence Ouverte / Open Licence 2.0 that BnF's records are under asks for the source and the date, and no date was shown anywhere. A BnF book's Metadata source row and a BnF author's header now read "Data from the BnF, Licence Ouverte / Open Licence 2.0, retrieved ". The date is the row's own: lastMetadataRefreshAt, else createdAt, both already in the API response; no column or migration. The Settings credit now says where the date is shown. Also covered by tests now: the BnF label in the link metadata dialog, authorProviderKey for bnf: ids, and bnf as a primary the add dialog honours. Each fails with its line removed. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9 Signed-off-by: vavallee --- web/src/api/authors.ts | 4 ++++ web/src/api/books.ts | 3 +++ web/src/components/AddToLibraryModal.test.tsx | 19 +++++++++++++++ .../AuthorMetadataLinkModal.test.tsx | 19 +++++++++++++++ web/src/i18n/locales/en.json | 5 ++-- web/src/pages/AuthorDetailPage.test.tsx | 22 +++++++++++++++++ web/src/pages/AuthorDetailPage.tsx | 14 +++++++++-- web/src/pages/BookDetailPage.test.tsx | 24 +++++++++++++++++++ web/src/pages/BookDetailPage.tsx | 10 +++++++- .../MetadataTab.primaryProvider.test.tsx | 2 +- web/src/pages/settings/MetadataTab.tsx | 4 ++-- web/src/util/authorMetadata.test.ts | 10 +++++++- web/src/util/metadataSource.test.ts | 15 ++++++++++++ web/src/util/metadataSource.ts | 18 ++++++++++++++ 14 files changed, 160 insertions(+), 9 deletions(-) diff --git a/web/src/api/authors.ts b/web/src/api/authors.ts index c70a21b72..679b1607b 100644 --- a/web/src/api/authors.ts +++ b/web/src/api/authors.ts @@ -18,6 +18,10 @@ export interface Author { averageRating: number monitored: boolean metadataProvider?: string + createdAt?: string + // When this author's metadata was last refreshed from its provider; null + // until the first refresh. Shown with the BnF credit (#3144). + lastMetadataRefreshAt?: string | null monitorMode?: AuthorMonitorMode monitorLatestCount?: number // Policy for works discovered after the initial sync (#1348): 'all' follows diff --git a/web/src/api/books.ts b/web/src/api/books.ts index 570553e5c..6a8aabd62 100644 --- a/web/src/api/books.ts +++ b/web/src/api/books.ts @@ -37,6 +37,9 @@ export interface Book { imageUrl: string releaseDate?: string createdAt?: string + // When this book's metadata was last fetched from its provider (a rebind); + // null until then. Shown with the BnF credit (#3144). + lastMetadataRefreshAt?: string | null genres: string[] monitored: boolean status: string diff --git a/web/src/components/AddToLibraryModal.test.tsx b/web/src/components/AddToLibraryModal.test.tsx index 64d252565..a81ee711d 100644 --- a/web/src/components/AddToLibraryModal.test.tsx +++ b/web/src/components/AddToLibraryModal.test.tsx @@ -763,6 +763,25 @@ describe('AddToLibraryModal - selection and confirm steps', () => { expect(notice.textContent).toContain('Hardcover') }) + // #3144: BnF is an accepted primary, so a result from elsewhere is flagged + // and a BnF result is not. + it('honours BnF as the configured primary provider', async () => { + vi.mocked(api.getSetting).mockImplementation(async (key: string) => { + if (key === 'metadata.primary_provider') return { key, value: 'bnf' } + throw new Error('HTTP 404') + }) + vi.mocked(api.searchAuthors).mockResolvedValue([ + author({ foreignAuthorId: 'OL3101279A', authorName: 'Matt Dinniman', metadataProvider: 'openlibrary' }), + author({ foreignAuthorId: 'bnf:author:11885918', authorName: 'Lewis Carroll', metadataProvider: 'bnf' }), + ]) + vi.mocked(api.searchBooks).mockResolvedValue([]) + render() + typeAndSearch('carroll') + await waitFor(() => expect(screen.getByText('Lewis Carroll')).toBeInTheDocument()) + expect(await screen.findByText('from OpenLibrary')).toBeInTheDocument() + expect(screen.getAllByText(/^from /)).toHaveLength(1) + }) + it('does not flag a result from the configured primary provider', async () => { vi.mocked(api.getSetting).mockImplementation(async (key: string) => { if (key === 'metadata.primary_provider') return { key, value: 'openlibrary' } diff --git a/web/src/components/AuthorMetadataLinkModal.test.tsx b/web/src/components/AuthorMetadataLinkModal.test.tsx index f3bf85036..70ddefb33 100644 --- a/web/src/components/AuthorMetadataLinkModal.test.tsx +++ b/web/src/components/AuthorMetadataLinkModal.test.tsx @@ -174,6 +174,25 @@ describe('AuthorMetadataLinkModal', () => { expect(screen.queryByRole('link', { name: /View on/ })).toBeNull() }) + // #3144: a BnF candidate is labelled by its foreign-ID prefix, not shown as + // the raw provider key or "Metadata". + it('labels a BnF candidate', async () => { + vi.mocked(api.searchAuthorLinkCandidates).mockResolvedValue([ + author({ id: 0, foreignAuthorId: 'bnf:author:11885918', authorName: 'Lewis Carroll', metadataProvider: '' }), + ]) + + render( + , + ) + + await screen.findByRole('button', { name: 'Link' }) + expect(screen.getByText('BnF')).toBeInTheDocument() + }) + // #2688: a record the author was linked to before is offered again rather // than hidden, so relinking is not a one way door. It has to stay linkable, // and it has to be distinguishable from a record that was never used. diff --git a/web/src/i18n/locales/en.json b/web/src/i18n/locales/en.json index d5d35cb2e..2fd0c77d3 100644 --- a/web/src/i18n/locales/en.json +++ b/web/src/i18n/locales/en.json @@ -5,6 +5,7 @@ "retry": "Retry", "retrying": "Retrying…", "viewOnSource": "View on {{source}} ↗", + "bnfSourceCredit": "Data from the BnF, Licence Ouverte / Open Licence 2.0, retrieved {{date}}", "links": "Links", "gridView": "Grid view", "tableView": "Table view", @@ -1515,12 +1516,12 @@ "telemetryDetail": "Sends install_id (random UUID), version, OS, architecture, deploy method, feature-usage counts, and coarse error counters (ERROR/WARN totals over the last 24h plus the 5 most frequent fixed error-message strings — never log details, titles, paths, or URLs) to getbindery.dev once per day. Disable at any time.", "metadataProvider": "Metadata Provider", "metadataProviderLabel": "Primary metadata provider", - "metadataProviderHint": "Selects the source used for author and book search, and decides what an author's catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. BnF (Bibliothèque nationale de France) is recommended for French-language libraries: legal deposit puts nearly every book published in France in its catalogue, under the French edition title, translator included, where OpenLibrary often has only the original English title. Contains data from the Bibliothèque nationale de France under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed; BnF does not endorse Bindery. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket and BnF, which are only queried when chosen as the primary.", + "metadataProviderHint": "Selects the source used for author and book search, and decides what an author's catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. BnF (Bibliothèque nationale de France) is recommended for French-language libraries: legal deposit puts nearly every book published in France in its catalogue, under the French edition title, translator included, where OpenLibrary often has only the original English title. Contains data from the Bibliothèque nationale de France under the Licence Ouverte / Open Licence 2.0, with the date each record was retrieved shown on its book and author pages; BnF does not endorse Bindery. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket and BnF, which are only queried when chosen as the primary.", "metadataProviderOpenlibrary": "OpenLibrary (default)", "metadataProviderDnb": "DNB — Deutsche Nationalbibliothek (German/DACH)", "metadataProviderNb": "Nasjonalbiblioteket — National Library of Norway (Norwegian)", "metadataProviderBnf": "BnF, Bibliothèque nationale de France (French)", - "metadataProviderBnfCredit": "Contains data from the Bibliothèque nationale de France (BnF) under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed. BnF does not endorse Bindery. Only BnF's catalogue records are used: no cover images are fetched, and BnF is contacted only while it is the primary provider.", + "metadataProviderBnfCredit": "Contains data from the Bibliothèque nationale de France (BnF) under the Licence Ouverte / Open Licence 2.0, with the date each record was retrieved shown on its book and author pages. BnF does not endorse Bindery. Only BnF's catalogue records are used: no cover images are fetched, and BnF is contacted only while it is the primary provider.", "metadataProviderHardcover": "Hardcover (curated catalogue)", "metadataProviderHardcoverNoToken": "Hardcover — requires an API token (Settings → API Keys)", "metadataProviderRestartHint": "Takes effect after the next Bindery restart — the provider is wired at startup, so nothing changes until then. Authors already in your library stay linked to the provider they were added from; the new primary applies to authors you add afterwards. To move an existing author, use \"Link metadata\" on their page.", diff --git a/web/src/pages/AuthorDetailPage.test.tsx b/web/src/pages/AuthorDetailPage.test.tsx index d0e2f68bf..19806846c 100644 --- a/web/src/pages/AuthorDetailPage.test.tsx +++ b/web/src/pages/AuthorDetailPage.test.tsx @@ -307,6 +307,28 @@ describe('AuthorDetailPage', () => { expect(api.searchAuthorWanted).not.toHaveBeenCalled() }) + // #3144: BnF's Licence Ouverte asks for the source and the date. + it('credits the BnF with the date of the last metadata refresh for a BnF author', async () => { + renderAuthorDetailPage([], 'grid', { + foreignAuthorId: 'bnf:author:11885918', + authorName: 'Lewis Carroll', + sortName: 'Carroll, Lewis', + metadataProvider: 'bnf', + createdAt: '2026-10-01T08:00:00Z', + lastMetadataRefreshAt: '2026-10-10T07:00:00Z', + }) + + const credit = await screen.findByTestId('bnf-source-credit') + expect(credit.textContent).toContain('BnF') + expect(credit.textContent).toContain('2026-10-10') + }) + + it('shows no BnF credit for an author from another provider', async () => { + renderAuthorDetailPage([], 'grid', { createdAt: '2026-10-01T08:00:00Z' }) + await screen.findByRole('heading', { level: 2 }) + expect(screen.queryByTestId('bnf-source-credit')).toBeNull() + }) + it('shows link metadata for unlinked authors', async () => { renderAuthorDetailPage([], 'grid', { foreignAuthorId: 'abs:author:library:emilia-jae', diff --git a/web/src/pages/AuthorDetailPage.tsx b/web/src/pages/AuthorDetailPage.tsx index b954ccb3e..368328939 100644 --- a/web/src/pages/AuthorDetailPage.tsx +++ b/web/src/pages/AuthorDetailPage.tsx @@ -12,8 +12,8 @@ import RenameFilesModal from '../components/RenameFilesModal' import BulkActionBar from '../components/BulkActionBar' import { useView } from '../components/useView' import MarkdownDescription from '../components/MarkdownDescription' -import { canLinkAuthorMetadata } from '../util/authorMetadata' -import { metadataSourceLink } from '../util/metadataSource' +import { authorProviderKey, canLinkAuthorMetadata } from '../util/authorMetadata' +import { metadataRetrievedDate, metadataSourceLink } from '../util/metadataSource' import { isAutoGrabRefusal } from '../util/autoGrabRefusal' import { canonicalLanguage, languageName } from '../util/language' import { btn, btnSize } from '../components/buttons' @@ -1011,6 +1011,16 @@ export default function AuthorDetailPage() { ) : null })()} + {/* BnF's Licence Ouverte asks for the source and the date + wherever its data is shown (#3144). */} + {(() => { + const date = authorProviderKey(author) === 'bnf' ? metadataRetrievedDate(author) : null + return date ? ( + + {t('common.bnfSourceCredit', { date, defaultValue: 'Data from the BnF, Licence Ouverte / Open Licence 2.0, retrieved {{date}}' })} + + ) : null + })()} {/* Four fixed cells, replacing a run-on sentence. The audiobook count diff --git a/web/src/pages/BookDetailPage.test.tsx b/web/src/pages/BookDetailPage.test.tsx index 09dcfbc1c..3d23fc88a 100644 --- a/web/src/pages/BookDetailPage.test.tsx +++ b/web/src/pages/BookDetailPage.test.tsx @@ -1385,6 +1385,30 @@ describe('BookDetailPage metadata source (#1707)', () => { expect(within(list).getByText('hc:12345')).toBeInTheDocument() }) + // #3144: BnF's Licence Ouverte asks for the source and the date. + it('credits the BnF with the date its record was retrieved', async () => { + vi.mocked(api.getBook).mockResolvedValue( + makeBook({ + foreignBookId: 'bnf:cb48780493c', metadataProvider: 'bnf', + createdAt: '2026-10-02T09:00:00Z', lastMetadataRefreshAt: '2026-10-09T18:30:00Z', + }), + ) + renderBookDetailPage() + + const credit = await screen.findByTestId('bnf-source-credit') + expect(credit.textContent).toBe('Data from the BnF, Licence Ouverte / Open Licence 2.0, retrieved 2026-10-09') + }) + + it('shows no BnF credit on a book from another provider', async () => { + vi.mocked(api.getBook).mockResolvedValue( + makeBook({ foreignBookId: 'OL27448W', metadataProvider: 'openlibrary', createdAt: '2026-10-02T09:00:00Z' }), + ) + renderBookDetailPage() + + await screen.findByTestId('metadata-source-list') + expect(screen.queryByTestId('bnf-source-credit')).toBeNull() + }) + it('falls back to the foreign-id prefix when the provider column is empty', async () => { vi.mocked(api.getBook).mockResolvedValue( makeBook({ foreignBookId: 'dnb:1234567890', metadataProvider: '' }), diff --git a/web/src/pages/BookDetailPage.tsx b/web/src/pages/BookDetailPage.tsx index c9ce22103..549551f60 100644 --- a/web/src/pages/BookDetailPage.tsx +++ b/web/src/pages/BookDetailPage.tsx @@ -21,7 +21,7 @@ import ClipboardManualFallback from '../components/ClipboardManualFallback' import { useClipboardCopy } from '../components/useClipboardCopy' import { isAutoGrabRefusal } from '../util/autoGrabRefusal' import { safeHref } from '../util/safeHref' -import { metadataSourceLink, providerDisplayName, providerFromBookForeignId } from '../util/metadataSource' +import { metadataRetrievedDate, metadataSourceLink, providerDisplayName, providerFromBookForeignId } from '../util/metadataSource' import FixMatchModal from '../components/FixMatchModal' import EditBookModal from '../components/EditBookModal' import { formatBytes } from '../util/format' @@ -796,6 +796,7 @@ function BookDetailPageInner() { return rows })() const sourceLinks = identityRows.flatMap(row => row.link ? [row.link] : []) + const bnfRetrieved = metadataRetrievedDate(book) // Display truth is the file inventory, never the declared media type. const rows = fileRows(book) @@ -1385,6 +1386,13 @@ function BookDetailPageInner() { {t('common.viewOnSource', { source: row.link.label, defaultValue: 'View on {{source}} ↗' })} )} + {/* BnF's Licence Ouverte asks for the source and the date + wherever its data is shown (#3144). */} + {row.primary && row.provider === 'bnf' && bnfRetrieved && ( + + {t('common.bnfSourceCredit', { date: bnfRetrieved, defaultValue: 'Data from the BnF, Licence Ouverte / Open Licence 2.0, retrieved {{date}}' })} + + )} ))} diff --git a/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx b/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx index 5714c56bd..7037d66b8 100644 --- a/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx +++ b/web/src/pages/settings/MetadataTab.primaryProvider.test.tsx @@ -127,7 +127,7 @@ describe('Primary metadata provider selector', () => { const credit = await screen.findByTestId('bnf-credit') expect(credit.textContent).toMatch(/Contains data from the Bibliothèque nationale de France/) expect(credit.textContent).toMatch(/Licence Ouverte \/ Open Licence 2\.0/) - expect(credit.textContent).toMatch(/retrieved when/) + expect(credit.textContent).toMatch(/date each record was retrieved/) expect(credit.textContent).toMatch(/does not endorse/) }) }) diff --git a/web/src/pages/settings/MetadataTab.tsx b/web/src/pages/settings/MetadataTab.tsx index 2a35623fb..7886cf1ec 100644 --- a/web/src/pages/settings/MetadataTab.tsx +++ b/web/src/pages/settings/MetadataTab.tsx @@ -89,7 +89,7 @@ export default function MetadataTab() { {t('settings.general.metadataProviderLabel', 'Primary metadata provider')}

- {t('settings.general.metadataProviderHint', 'Selects the source used for author and book search, and decides what an author\'s catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. BnF (Bibliothèque nationale de France) is recommended for French-language libraries: legal deposit puts nearly every book published in France in its catalogue, under the French edition title, translator included, where OpenLibrary often has only the original English title. Contains data from the Bibliothèque nationale de France under the Licence Ouverte / Open Licence 2.0, retrieved when each book or author is added or refreshed; BnF does not endorse Bindery. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket and BnF, which are only queried when chosen as the primary.')} + {t('settings.general.metadataProviderHint', 'Selects the source used for author and book search, and decides what an author\'s catalogue looks like. DNB (Deutsche Nationalbibliothek) is recommended for German, Austrian, and Swiss catalogues — it covers German-language publications since 1913 where OpenLibrary coverage is thin. Nasjonalbiblioteket (the National Library of Norway) is recommended for Norwegian catalogues — it holds every Norwegian publication by legal deposit, under its original Norwegian title where OpenLibrary often has only the English translation. BnF (Bibliothèque nationale de France) is recommended for French-language libraries: legal deposit puts nearly every book published in France in its catalogue, under the French edition title, translator included, where OpenLibrary often has only the original English title. Contains data from the Bibliothèque nationale de France under the Licence Ouverte / Open Licence 2.0, with the date each record was retrieved shown on its book and author pages; BnF does not endorse Bindery. Hardcover is a curated catalogue: fewer translation editions, omnibus bundles, and non-book entries than OpenLibrary, at the cost of a thinner long tail. OpenLibrary remains the default. Whichever you pick, the other providers are still used as enrichers, except Nasjonalbiblioteket and BnF, which are only queried when chosen as the primary.')}