Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ Clean-room Go rewrite, modern React UI, MIT-licensed, actively developed.
- Library scan matching in five passes: exact ASIN, then a backwards `Author - Title` filename whose folder author the catalogue already knows, then the filename's title, then the book folder's title, and finally series name plus position. Every title pass is one fuzzy comparison that has to agree on the author as well. Honours librarian sort-suffix form (`Title, The`) and series-annotated filenames (`[Mistborn, Book 1]`).
- **Library adoption**: whatever the scan could not match waits on the Import page as one row per book (an audiobook folder or disc set is one row), with suggested matches (one click Confirm only for a strong match), one "Add author" decision per missing author, and Undo. Files are registered where they are, never moved.
- Author aliases (`RR Haywood` / `R.R. Haywood` / `R R Haywood` merge into one canonical row), and metadata re-bind to correct a wrong match without delete-and-re-add. On a book’s **File → Re-bind** dialog, search configured metadata providers and select an OpenLibrary work or Hardcover book, or enter an exact provider ID and press Enter. Results already in the library are marked and link to the existing book; records belonging to another library book cannot be selected. Each result’s **Links** menu opens its upstream page when available; OpenLibrary editions can be viewed but require a work ID for re-binding. Re-binding updates metadata and series without moving files; author mismatches require confirmation.
- Explicit author-catalogue reconciliation with a selectable preview: remove chosen stale metadata-only Wanted rows after changing provider or metadata profile while always protecting imported books and every row with a tracked file.
- Explicit author-catalogue reconciliation with a selectable preview: remove chosen stale metadata-only Wanted rows after changing provider or metadata profile, review rows kept because evidence is incomplete, and always protect imported books and every row with a tracked file.
- Manual metadata editing with field locks: edit a book's title, description, genres, language or release date and the edit survives every refresh. **Fix match** reassigns a file that was attached to the wrong book. See [docs/Metadata-Editing-Wiki.md](docs/Metadata-Editing-Wiki.md).

**Search & downloads**
Expand Down
2 changes: 2 additions & 0 deletions changelog.d/2892-reconciliation-indeterminate-rows.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
### Added
- **Reconcile catalogue now lists the rows it kept for lack of evidence** (#2892): rows kept because provider or profile evidence was incomplete used to show up only as a count. Each one is now listed with its title, provider and the reason, such as language could not be confirmed, so you can review them yourself. They still can never be removed automatically. Thanks kevinatlee.
10 changes: 8 additions & 2 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,14 @@ The field is absent when the providers match or no primary is configured.

Catalogue reconciliation is deliberately separate from refresh. The GET route
queries the current primary provider without using its cached author catalogue
and returns `candidates`, a reason-count summary, protection counts, and
`providerComplete`. A partial provider result never treats absence as a reason
and returns `candidates`, `indeterminateRows`, a reason-count summary,
protection counts, and `providerComplete`. Each `indeterminateRows` entry has a
`bookId`, `title`, `metadataProvider`, and display-only `reason`. These rows are
informational: they are kept because evidence is incomplete and are never
deletion candidates. Stable indeterminate reasons are `language_unknown`,
`language_evidence_lookup_failed`, `edition_evidence_unavailable`,
`partial_catalogue`, and `unmatched_cross_provider`. A partial provider result
never treats absence as a reason
to remove a row. A complete result can make an absent same-provider row a
candidate, but an unmatched row from another provider is kept as indeterminate:
provider migration alone is not deletion evidence. Explicit profile rejections
Expand Down
5 changes: 4 additions & 1 deletion docs/User-Guide-Wiki.md
Original file line number Diff line number Diff line change
Expand Up @@ -834,7 +834,10 @@ an absent row from that same provider may be actionable. A row from a different
provider is kept as indeterminate when Bindery cannot correlate it to the
current catalogue; changing providers alone is not evidence that a work is
obsolete. A correlated work that the metadata profile explicitly rejects can
still be removed for that rejection reason.
still be removed for that rejection reason. The preview lists each
indeterminate row separately with its provider and the incomplete-evidence
reason so it can be reviewed manually; these informational rows have no
selection control and cannot be sent for removal.

**Duplicate titles.** The same book often reaches the catalogue twice under
slightly different titles — "The Martian" and "Martian", "Dune" and "Dune
Expand Down
125 changes: 87 additions & 38 deletions internal/api/catalogue_reconciliation.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,20 @@ const (
reconcileReasonCatalogueFilter = "catalogue_filter"
)

// CatalogueReconciliationIndeterminateReason identifies why a local row was
// kept because the available provider or profile evidence was not conclusive.
// These reasons are informational only and never make a row eligible for
// deletion.
type CatalogueReconciliationIndeterminateReason string

const (
reconcileIndeterminateReasonLanguageUnknown CatalogueReconciliationIndeterminateReason = "language_unknown"
reconcileIndeterminateReasonLanguageLookupFailed CatalogueReconciliationIndeterminateReason = "language_evidence_lookup_failed"
reconcileIndeterminateReasonEditionUnavailable CatalogueReconciliationIndeterminateReason = "edition_evidence_unavailable"
reconcileIndeterminateReasonPartialCatalogue CatalogueReconciliationIndeterminateReason = "partial_catalogue"
reconcileIndeterminateReasonUnmatchedCrossProvider CatalogueReconciliationIndeterminateReason = "unmatched_cross_provider"
)

// CatalogueReconciliationCandidate is a local, metadata-only Wanted row that
// the current primary-provider catalogue and metadata profile no longer
// accept. It contains no file operation: reconciliation deletes database rows
Expand All @@ -42,6 +56,17 @@ type CatalogueReconciliationCandidate struct {
Reason string `json:"reason"`
}

// CatalogueReconciliationIndeterminateRow is a local row that reconciliation
// kept because provider or profile evidence was incomplete. It is deliberately
// separate from CatalogueReconciliationCandidate so it cannot be mistaken for
// a safe deletion target.
type CatalogueReconciliationIndeterminateRow struct {
BookID int64 `json:"bookId"`
Title string `json:"title"`
MetadataProvider string `json:"metadataProvider"`
Reason CatalogueReconciliationIndeterminateReason `json:"reason"`
}

type CatalogueReconciliationSummary struct {
Total int `json:"total"`
Candidates int `json:"candidates"`
Expand All @@ -66,15 +91,16 @@ type CatalogueReconciliationApplySummary struct {
// only a partial author catalogue; in that case absence is never used as a
// deletion reason.
type CatalogueReconciliation struct {
AuthorID int64 `json:"authorId"`
AuthorName string `json:"authorName"`
Provider string `json:"provider"`
ProviderComplete bool `json:"providerComplete"`
ProfileName string `json:"profileName"`
Warning string `json:"warning,omitempty"`
Candidates []CatalogueReconciliationCandidate `json:"candidates"`
Summary CatalogueReconciliationSummary `json:"summary"`
Applied *CatalogueReconciliationApplySummary `json:"applied,omitempty"`
AuthorID int64 `json:"authorId"`
AuthorName string `json:"authorName"`
Provider string `json:"provider"`
ProviderComplete bool `json:"providerComplete"`
ProfileName string `json:"profileName"`
Warning string `json:"warning,omitempty"`
Candidates []CatalogueReconciliationCandidate `json:"candidates"`
IndeterminateRows []CatalogueReconciliationIndeterminateRow `json:"indeterminateRows"`
Summary CatalogueReconciliationSummary `json:"summary"`
Applied *CatalogueReconciliationApplySummary `json:"applied,omitempty"`
}

type applyCatalogueReconciliationRequest struct {
Expand Down Expand Up @@ -261,24 +287,24 @@ func (h *AuthorHandler) buildCatalogueReconciliation(ctx context.Context, author
}

acceptedIDs := make(map[string]struct{}, len(works))
indeterminateIDs := make(map[string]struct{})
indeterminateIDs := make(map[string]CatalogueReconciliationIndeterminateReason)
rejectedIDs := make(map[string]string, len(works))
acceptedTitles := indexer.NewTitleIndex[struct{}]()
indeterminateTitles := indexer.NewTitleIndex[struct{}]()
indeterminateTitles := indexer.NewTitleIndex[CatalogueReconciliationIndeterminateReason]()
rejectedTitles := make(map[string]string)
normalizedAuthor := strings.ToLower(strings.TrimSpace(author.Name))
for _, work := range works {
reason, indeterminate := reconciliationRejectReason(work, normalizedAuthor, profile, editions[work.ForeignID], languageEvidence, languageEvidenceFailed)
reason, indeterminateReason := reconciliationRejectReason(work, normalizedAuthor, profile, editions[work.ForeignID], languageEvidence, languageEvidenceFailed)
if reason == "" {
if strings.TrimSpace(work.ForeignID) != "" {
acceptedIDs[work.ForeignID] = struct{}{}
}
acceptedTitles.Add(work.Title, struct{}{})
if indeterminate {
if indeterminateReason != "" {
if strings.TrimSpace(work.ForeignID) != "" {
indeterminateIDs[work.ForeignID] = struct{}{}
indeterminateIDs[work.ForeignID] = indeterminateReason
}
indeterminateTitles.Add(work.Title, struct{}{})
indeterminateTitles.Add(work.Title, indeterminateReason)
}
continue
}
Expand All @@ -299,12 +325,13 @@ func (h *AuthorHandler) buildCatalogueReconciliation(ctx context.Context, author
return CatalogueReconciliation{}, fmt.Errorf("list identifiers for author %d: %w", author.ID, err)
}
result := CatalogueReconciliation{
AuthorID: author.ID,
AuthorName: author.Name,
Provider: snapshot.Provider,
ProviderComplete: snapshot.Complete,
ProfileName: profile.name,
Candidates: []CatalogueReconciliationCandidate{},
AuthorID: author.ID,
AuthorName: author.Name,
Provider: snapshot.Provider,
ProviderComplete: snapshot.Complete,
ProfileName: profile.name,
Candidates: []CatalogueReconciliationCandidate{},
IndeterminateRows: []CatalogueReconciliationIndeterminateRow{},
Summary: CatalogueReconciliationSummary{
Total: len(localBooks),
Reasons: map[string]int{},
Expand Down Expand Up @@ -354,15 +381,15 @@ func (h *AuthorHandler) buildCatalogueReconciliation(ctx context.Context, author
}
if anyReconciliationIDAccepted(ids, acceptedIDs) {
result.Summary.Kept++
if anyReconciliationIDAccepted(ids, indeterminateIDs) {
result.Summary.Indeterminate++
if indeterminateReason := reconciliationIndeterminateReasonForIDs(ids, indeterminateIDs); indeterminateReason != "" {
result.IndeterminateRows = append(result.IndeterminateRows, catalogueReconciliationIndeterminateRow(book, indeterminateReason))
}
continue
}
if _, accepted := acceptedTitles.Lookup(book.Title); accepted {
result.Summary.Kept++
if _, indeterminate := indeterminateTitles.Lookup(book.Title); indeterminate {
result.Summary.Indeterminate++
if indeterminateReason, indeterminate := indeterminateTitles.Lookup(book.Title); indeterminate {
result.IndeterminateRows = append(result.IndeterminateRows, catalogueReconciliationIndeterminateRow(book, indeterminateReason))
}
continue
}
Expand All @@ -376,12 +403,12 @@ func (h *AuthorHandler) buildCatalogueReconciliation(ctx context.Context, author
if reason == "" {
if !snapshot.Complete {
result.Summary.Kept++
result.Summary.Indeterminate++
result.IndeterminateRows = append(result.IndeterminateRows, catalogueReconciliationIndeterminateRow(book, reconcileIndeterminateReasonPartialCatalogue))
continue
}
if bookProvider(book) != snapshot.Provider {
result.Summary.Kept++
result.Summary.Indeterminate++
result.IndeterminateRows = append(result.IndeterminateRows, catalogueReconciliationIndeterminateRow(book, reconcileIndeterminateReasonUnmatchedCrossProvider))
continue
}
reason = reconcileReasonNotInCatalogue
Expand All @@ -399,45 +426,58 @@ func (h *AuthorHandler) buildCatalogueReconciliation(ctx context.Context, author
sort.Slice(result.Candidates, func(i, j int) bool {
return strings.ToLower(result.Candidates[i].Title) < strings.ToLower(result.Candidates[j].Title)
})
sort.Slice(result.IndeterminateRows, func(i, j int) bool {
return strings.ToLower(result.IndeterminateRows[i].Title) < strings.ToLower(result.IndeterminateRows[j].Title)
})
result.Summary.Candidates = len(result.Candidates)
result.Summary.Indeterminate = len(result.IndeterminateRows)
return result, nil
}

func reconciliationRejectReason(work models.Book, normalizedAuthor string, profile reconciliationProfile, evidence editionEvidence, languageEvidence map[string]metadata.AuthorWorkLanguageEvidence, languageEvidenceFailed bool) (string, bool) {
func reconciliationRejectReason(work models.Book, normalizedAuthor string, profile reconciliationProfile, evidence editionEvidence, languageEvidence map[string]metadata.AuthorWorkLanguageEvidence, languageEvidenceFailed bool) (string, CatalogueReconciliationIndeterminateReason) {
normalizedTitle := strings.ToLower(strings.TrimSpace(work.Title))
if normalizedTitle == "" || normalizedTitle == normalizedAuthor || work.IsCompilation || metadata.IsUnambiguousBundleTitle(work.Title) {
return reconcileReasonCatalogueFilter, false
return reconcileReasonCatalogueFilter, ""
}
if len(profile.allowedLangs) > 0 {
if languageEvidenceFailed {
return "", true
return "", reconcileIndeterminateReasonLanguageLookupFailed
}
languageAllowed, indeterminate := authorWorkPassesLanguageFilter(&work, profile.allowedLangs, profile.unknownFail, languageEvidence)
if indeterminate {
return "", true
return "", reconcileIndeterminateReasonLanguageUnknown
}
if !languageAllowed {
return reconcileReasonLanguage, false
return reconcileReasonLanguage, ""
}
}
if profile.skipPartBooks && isPartBookTitle(work.Title) {
return reconcileReasonPartBook, false
return reconcileReasonPartBook, ""
}
if profile.skipMissingDate && work.ReleaseDate == nil {
return reconcileReasonMissingDate, false
return reconcileReasonMissingDate, ""
}
if profile.minPages > 0 || profile.skipMissingISBN {
if !evidence.known {
return "", true
return "", reconcileIndeterminateReasonEditionUnavailable
}
if profile.skipMissingISBN && !anyEditionHasISBN(evidence.editions) {
return reconcileReasonISBN, false
return reconcileReasonISBN, ""
}
if profile.minPages > 0 && !passesMinPagesFilter(evidence.editions, profile.minPages) {
return reconcileReasonPages, false
return reconcileReasonPages, ""
}
}
return "", false
return "", ""
}

func catalogueReconciliationIndeterminateRow(book *models.Book, reason CatalogueReconciliationIndeterminateReason) CatalogueReconciliationIndeterminateRow {
return CatalogueReconciliationIndeterminateRow{
BookID: book.ID,
Title: book.Title,
MetadataProvider: bookProvider(book),
Reason: reason,
}
}

func anyReconciliationIDAccepted(ids []string, accepted map[string]struct{}) bool {
Expand All @@ -449,6 +489,15 @@ func anyReconciliationIDAccepted(ids []string, accepted map[string]struct{}) boo
return false
}

func reconciliationIndeterminateReasonForIDs(ids []string, reasons map[string]CatalogueReconciliationIndeterminateReason) CatalogueReconciliationIndeterminateReason {
for _, id := range ids {
if reason := reasons[id]; reason != "" {
return reason
}
}
return ""
}

func reconciliationReasonForIDs(ids []string, rejected map[string]string) string {
for _, id := range ids {
if reason := rejected[id]; reason != "" {
Expand Down
Loading
Loading