fix(importer): reject a download whose file is in a language the profile does not allow (#2998)

[vavallee]

Summary

A release whose name does not say its language passes the grab time language filter. At import, #1933's applyEmbeddedLanguage then read the EPUB's dc:language and relabelled the book to match, so an English only library filled with Swedish and Dutch editions that showed as imported and were never searched again (65 of 4,500 books in the report). Now, when the book's allowed languages are an explicit list and the downloaded EPUB declares a specific language outside it, the release is rejected the same way a disallowed format is: blocklisted, download blocked with a message naming both languages, book left Wanted.

Closes #2998.

Implementation notes

Design decision: on by default for explicit lists, no new setting

The issue suggested a setting so people who like today's relabelling keep it. Those people are the ones whose profile allows any language, and for them nothing changes: an empty or any profile resolves to "no restriction" and #1933's relabelling runs exactly as before. Enforcement fires only where the user has already written down a list of allowed languages, and that same list already hard filters release names at grab time. The import check is that filter applied to better evidence (the file instead of its name), so a file is never held to a stricter rule than its name would have been. A toggle would only let someone keep a restriction at search time while waiving it at import, which is the bug. No setting, no migration, no web change.

Where the allowed list comes from

importAllowedLanguages mirrors Scheduler.searchAndGrabFormat and IndexerHandler.SearchBook exactly:

Situation	Enforced list
Author's metadata profile has allowed_languages (default "Standard" profile is eng)	that list, via ParseAllowedLanguages
Profile is any or empty, search.preferredLanguage = en	eng (the only value indexer.FilterByLanguage acts on)
Profile is any or empty, preferred language unset or anything else	none, relabelling as today
Profile cannot be loaded	falls through like the scheduler does; never rejects on a load error

What counts as a disallowed file

• Every dc:language of every EPUB the import loop would place counts (EpubMetadata.Languages; Language stays the first for existing callers). An EPUB is allowed when any declared language is allowed, disallowed when it declares a specific language and none is allowed, unknown otherwise. The release is rejected only when some EPUB is disallowed and none is allowed, the shape of the format gate. A release mixing allowed and disallowed EPUBs imports the allowed ones; the ebook loop skips the others (languageCheck.skip), exactly where it skips a disallowed format. Under a restricted profile, relabelling picks the first allowed declared language.
• Compared with models.IsLanguageAllowed, which normalises both sides, so en, en-US, en_GB, EN, eng all match eng.
• Only a definite language can be rejected: und, mul, zxx, mis, qaa..qtz, missing, and anything that is not a 3 letter code after normalisation (unknown, x-default) import as today.
• iso639TwoLetterToB now covers all 183 ISO 639-1 codes (it held 25, so uk/he/sk passed as "no language"), languageNames has an English name for every code produced, and cmn/yue fold onto chi.
• nb/nob/nn/nno now fold onto nor in NormalizeLanguageCode via iso639NorwegianToB, matching francisrath's feat(metadata): Nasjonalbiblioteket as an opt-in primary provider (#2979) #2995 hunk line for line (credited as co author). The editor offers only nor, so a Norwegian profile used to reject Bokmål tagged books in the catalogue filter too; folding there fixes every caller at once. LanguageCodeVariants gains the individual codes for provider side filtering.
• Language lock: if the user locked the book's language, that language is added to the allowed set for this book only. The user said "this book is in X", which beats the author wide profile; it does not widen to other languages (a German file for a book locked to Swedish under an eng profile is still rejected).

How rejection reuses the format enforcement path (#1782)

Wired through WithLanguageEnforcement(metadataProfileRepo, blocklistRepo) next to WithFormatEnforcement in cmd/bindery/main.go; unwired scanners (every existing test, any other caller) behave as before. The gate sits right after the format and multi book guards, before anything is placed, and calls the same helpers in the same order: recordUnmatchedImportPath (so Match to book can force it in), blocklistRejectedRelease, failImport(StateImportBlocked, reason), which writes the importFailed history row, sets the download status and sends the downloadFailed notification. Message example: file declares Swedish (swe), but the metadata profile "Standard" allows only English (eng). Not imported, and the release was blocklisted so the next search picks another. To keep this file anyway, use Match to book in the Queue or manual import. models.LanguageName supplies the names, with a test that every code the alias tables produce has one.

Manual import, scans, existing books

ImportFromPath and ImportFilesFromPath (manual import, Reassign, Match to book, batch import; every caller is a user action in internal/api) mark the context as manual. A manual import with a disallowed language logs a warning and imports, and relabelling records the real language as before. The format check treats formatHint as its override instead; I did not change that here. Library scans and adoption never run tryImportInternal, and nothing touches already imported books. External import mode returns before files are inspected, so it is unchanged (documented).

Checklist

• Tests added or updated
• Doc-update gate cleared: docs/User-Guide-Wiki.md (Import section), docs/Troubleshooting-Wiki.md (A grabbed book never imports), godoc on new symbols
• Wiki pages updated if user-facing behaviour changed (the in repo wiki docs are updated; the GitHub wiki mirror, if any, is not)

Test plan

Fail before: the new importer tests run against origin/main (with a compile shim for WithLanguageEnforcement) fail RejectsDisallowedLanguage, PreferredLanguageFallback and LockedLanguageDoesNotWidenToOthers with download status = "imported", want "importBlocked"; the new nb/nob/nn/nno cases in TestNormalizeLanguageCode fail on main ("nob" = "nob", want "nor"). All pass on this branch.

• go test ./internal/importer/ -run TestLanguageEnforcement: English only profile + Swedish EPUB is blocked, blocklisted, book stays Wanted and eng, no file placed, no correction event, one importFailed row, downloadFailed sent; allowed languages in 12 spellings import (incl. en_GB, de-AT vs ger, nob vs nor); undeclared/und/mul/zxx/unknown import; any and empty profiles keep relabelling; preferred language fallback; manual single and multi file import not rejected; locked language allowed and not widened; unwired scanner unchanged
• Review reproductions, failing before the second commit: fr,en EPUB blocked (importBlocked ... file declares French), und,en-GB relabelled the book und, A sv + B en rejected, uk/he/sk/fa-IR/is all imported under eng. All pass now, plus all-disallowed multi EPUB rejection naming every language
• go test ./internal/models/ (normalisation, LanguageName, name coverage, all 183 ISO 639-1 codes)
• go test ./cmd/... ./internal/... (full suite)
• gofmt -l, go vet ./cmd/... ./internal/..., golangci-lint run v2.11.4: 0 issues
• make smoke (not run)

Follow-ups (not in this PR)

• Audiobooks: language from ID3 TLAN / MP4 language atom is not checked yet.
• With an any language profile, applyEmbeddedLanguage still relabels a book to und or mul when the EPUB says so; it should skip non specific codes too.
• Match to book passes an empty formatHint, so a format rejected (Quality profile format enforcement exists only at grab time — the importer never checks the actual downloaded file against the allowed-formats list #1782) release cannot be forced in through it; the manual marker added here could be its override as well.

🤖 Generated with Claude Code

https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9

[codecov]

Codecov Report

❌ Patch coverage is 87.40741% with 17 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/importer/language_enforcement.go	84.61%	7 Missing and 7 partials ⚠️
internal/models/language.go	83.33%	1 Missing and 1 partial ⚠️
internal/importer/scanner.go	96.29%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Good fix. The enforcement logic is correctly scoped (EPUB-only, opt-in via WithLanguageEnforcement, falls through on load errors), the relabelling behaviour is preserved for unrestricted profiles, and the test coverage is thorough — the 12-spelling roundtrip cases, mixed-release partial-import, locked-language widening check, and the reproduction list all read as genuine regression guards.

Two items worth a look before merge:

1. Shared blocklist field between the two enforcement methods — WithFormatEnforcement and WithLanguageEnforcement both write to s.blocklist (scanner.go). In main.go both calls receive the same blocklistRepo instance so there is no bug today, but the coupling is invisible at the call site. If a future caller constructs a scanner with only one of the two wired (which is a normal pattern — tests do exactly this), the second With… call would silently overwrite the first method's blocklist. Because blocklistRejectedRelease guards on s.blocklist == nil, an accidentally-nil write from the language path would also disable blocklisting for format-rejected releases with no error. Consider either promoting blocklist to a separate WithBlocklist setter that both enforcement methods share, or documenting the coupling prominently on the field.

2. Double-normalize in languageList — language_enforcement.go:460 calls NormalizeLanguageCode(code) and then passes the result n to LanguageName(n), which calls NormalizeLanguageCode again internally. Both inputs (lc.declared and profileAllowed) are already normalised three-letter codes, so all three calls are idempotent and the output is correct. But the pattern muddies the input contract of languageList. languageList(codes []string) could just call models.LanguageName(code) for the name part and models.NormalizeLanguageCode(code) for the display code, with a godoc note that codes are normalised before rendering.

Neither is a blocking defect and the smoke run being skipped is noted in the checklist. The rest of the diff — the ISO 639-1 completion, the Norwegian fold, the EpubMetadata.Languages slice, the definiteLanguage private-use range guard — all look correct.

— 🤖 Bindery triage bot (automated). Reply to correct me; a human will see it.