catalog-forge

Entity resolution and attribute extraction for heterogeneous product catalogs.

Merges product listings from several feeds into one deduplicated catalog with a product → variant → offer hierarchy and structured attributes. Runs on a single machine, reproducibly, with no cloud dependency and no API keys.

pip install catalog-forge
forge run --demo          # 10,069 listings end to end in ~25 s

Measured, not asserted: every number below comes from make bench, make ablation or make scale, including the one that says the performance target was missed.

Numbers, not adjectives

Pairwise precision, recall and F1 on the test split of each public corpus. The classifier is trained on the corpus's own train split, the threshold is tuned on valid, and neither is ever fitted on test. Regenerate with make bench.

dataset	records	precision	recall	F1	AP	threshold
amazon-google	4,589	0.639	0.697	0.667	0.642	0.480
abt-buy	2,173	0.706	0.723	0.715	0.758	0.230
dblp-scholar	66,879	0.928	0.931	0.930	0.981	0.430
wdc-computers	4,298	0.750	0.840	0.792	0.850	0.300
wdc-cameras	3,193	0.760	0.857	0.806	0.854	0.310
wdc-watches	3,700	0.833	0.880	0.856	0.922	0.240
wdc-shoes	2,773	0.711	0.849	0.774	0.797	0.440

Against the two libraries you would otherwise reach for, on the same splits with the same metric code:

dataset	system	kind	precision	recall	F1	note
amazon-google	title Jaccard	trivial	0.393	0.637	0.486	threshold 0.44 tuned on valid
amazon-google	Splink 4	unsupervised	0.102	1.000	0.185	own blocking rules; recall bounded by them, not by the model
amazon-google	Dedupe	supervised	0.304	0.432	0.357	same train split (600/600 examples)
amazon-google	catalog-forge	supervised	0.639	0.697	0.667	threshold 0.48 tuned on valid
abt-buy	title Jaccard	trivial	0.541	0.354	0.428	threshold 0.50 tuned on valid
abt-buy	Splink 4	unsupervised	0.108	1.000	0.194	own blocking rules; recall bounded by them, not by the model
abt-buy	Dedupe	supervised	0.268	0.437	0.332	same train split (600/600 examples)
abt-buy	catalog-forge	supervised	0.706	0.723	0.715	threshold 0.23 tuned on valid
dblp-scholar	title Jaccard	trivial	0.780	0.821	0.800	threshold 0.61 tuned on valid
dblp-scholar	Splink 4	unsupervised	0.736	0.775	0.755	own blocking rules; recall bounded by them, not by the model
dblp-scholar	Dedupe	supervised	0.919	0.837	0.876	same train split (600/600 examples, predicates on a 10,440-record sample)
dblp-scholar	catalog-forge	supervised	0.928	0.931	0.930	threshold 0.43 tuned on valid
wdc-computers	title Jaccard	trivial	0.378	0.700	0.491	threshold 0.29 tuned on valid
wdc-computers	Splink 4	unsupervised	0.390	0.337	0.361	own blocking rules; recall bounded by them, not by the model
wdc-computers	Dedupe	supervised	0.415	0.557	0.476	same train split (600/600 examples)
wdc-computers	catalog-forge	supervised	0.750	0.840	0.792	threshold 0.3 tuned on valid
wdc-cameras	title Jaccard	trivial	0.427	0.700	0.530	threshold 0.27 tuned on valid
wdc-cameras	Splink 4	unsupervised	0.446	0.430	0.438	own blocking rules; recall bounded by them, not by the model
wdc-cameras	Dedupe	supervised	0.445	0.597	0.510	same train split (600/600 examples)
wdc-cameras	catalog-forge	supervised	0.760	0.857	0.806	threshold 0.31 tuned on valid
wdc-watches	title Jaccard	trivial	0.379	0.717	0.496	threshold 0.33 tuned on valid
wdc-watches	Splink 4	unsupervised	0.214	0.287	0.245	own blocking rules; recall bounded by them, not by the model
wdc-watches	Dedupe	supervised	0.333	0.343	0.338	same train split (600/600 examples)
wdc-watches	catalog-forge	supervised	0.833	0.880	0.856	threshold 0.24 tuned on valid
wdc-shoes	title Jaccard	trivial	0.356	0.863	0.504	threshold 0.23 tuned on valid
wdc-shoes	Splink 4	unsupervised	0.320	0.391	0.352	own blocking rules; recall bounded by them, not by the model
wdc-shoes	Dedupe	supervised	0.339	0.930	0.497	same train split (600/600 examples)
wdc-shoes	catalog-forge	supervised	0.711	0.849	0.774	threshold 0.44 tuned on valid

Three things about that table, because a comparison is only useful if you know what it is comparing.

Splink is unsupervised and it shows. It never sees a label. On DBLP-Scholar, structured bibliographic data of exactly the kind Fellegi-Sunter was designed for, it reaches 0.755 — respectable. On Amazon-Google and Abt-Buy its F1-optimal threshold on the validation split turns out to be ~0, i.e. "accept every pair my blocking rules produced": precision 0.10 at recall 1.0. That is not Splink being bad, it is Splink being asked to separate short marketing titles without a single label, and it is the right reason to reach for a supervised matcher on product data.

Dedupe is the fair fight, and it is closest where it belongs. On DBLP-Scholar it reaches 0.876 against our 0.930 — a real gap but a narrow one, on the domain it was built for. On the four WDC product categories the gap widens to 20–50 points, which is what the product-specific features (identifier validation, unit normalization, attribute roles) are actually buying.

Where we lose. Fine-tuned transformer matchers (Ditto, HierGAT and descendants) beat this on the text-heavy corpora — published Ditto numbers are around F1 0.89 on Abt-Buy against our 0.715. That gap is real and it is the price of a feature-based model that trains in seconds on a laptop and explains every decision it makes. If you have a GPU and want the last few points, use a cross-encoder as the judge stage; the interface is one method.

Blocking

Pair completeness is the hard ceiling on recall for everything downstream, and reduction ratio is what makes the run finish. Reporting one without the other is meaningless — you can have PC 1.0 by comparing everything.

dataset	records	candidate pairs	pair completeness	reduction ratio
amazon-google	4,589	97,554	0.869	0.991
abt-buy	2,173	44,509	0.908	0.981
dblp-scholar	66,879	2,037,932	0.994	0.999
wdc-computers	4,298	133,089	0.725	0.986
wdc-cameras	3,193	78,228	0.759	0.985
wdc-watches	3,700	146,425	0.763	0.979
wdc-shoes	2,773	69,490	0.712	0.982

What each stage is actually worth

Each row removes exactly one thing from the pipeline and re-runs the same evaluation with the same seed, so the delta is attributable. A stage whose delta is inside the noise should be deleted rather than defended.

The catalog used here is deliberately harsher than the default: 19% of listings carry a GTIN and 15% an MPN, which is the regime real feeds are in. With identifier coverage anywhere near complete the exact-identifier block answers almost everything and every other row reads as "this contributes nothing" — a property of the fixture, not of the pipeline.

configuration	pair completeness	precision	recall	F1	ΔF1	cluster B3 F1
full pipeline	0.910	0.518	0.815	0.633	0.000	0.770
no normalization	0.891	0.491	0.801	0.609	-0.024	0.750
no vocabularies	0.896	0.499	0.799	0.614	-0.019	0.765
blocking: MinHash only	0.716	0.555	0.630	0.590	-0.043	0.764
blocking: identifiers only	0.061	0.924	0.060	0.114	-0.520	0.610
no rule layer	0.910	0.518	0.815	0.633	0.000	0.770
no classifier	0.910	0.940	0.052	0.098	-0.535	0.604
no judge stage	0.910	0.520	0.818	0.636	0.002	0.773
no attribute roles	0.910	0.518	0.815	0.633	0.000	0.770
no cohesion check	0.910	0.518	0.815	0.633	0.000	0.668

Four rows are worth reading carefully, because three of them look like a stage earning nothing.

The rule layer costs 0.000 F1. It is not there for accuracy — the classifier reaches the same F1 without it. It is there for speed: it resolves 83% of pairs at 13 µs against the classifier's 36 µs, and it is the only layer whose decisions come with a reason a human can read. Delete it and the pipeline is just as accurate and roughly twice as slow, with unexplainable output.

The judge stage costs −0.002 F1, which is noise. On this fixture it sees thirty pairs out of six hundred thousand, because isotonic calibration pushes almost everything out of the uncertainty band. It earns its place on corpora where the band is wider, and on this one it does not. That is what the row is for.

Attribute roles cost 0.000 F1 — at variant level, which is all this table measures. Their entire job is the variant→product step, which the pairwise metric cannot see. That number lives in the section above.

The cohesion check costs 0.000 F1 and 0.102 B-cubed F1. Exactly the shape it was designed for: welding two clusters together barely moves a pairwise metric and wrecks a per-record one. If you only tracked F1 you would delete it.

The two rows that dominate are the classifier (−0.535) and blocking breadth (−0.520 with identifiers alone, whose pair completeness collapses to 0.061 at 15% MPN coverage). Normalization is worth +0.024 here, and +0.033 / +0.046 / +0.032 on three further test seeds — small, consistent, and larger than it looks because most of its value shows up in blocking recall rather than in matching.

Two minutes

git clone https://github.com/alex-hahn/catalog-forge && cd catalog-forge
make install
make demo

make demo runs the bundled 10,069-listing sample catalog through the whole pipeline — normalize, categorise, extract, block, match, cluster, build the hierarchy, pick golden records — and writes artifacts/demo/catalog.csv. No downloads, no API keys, about 25 seconds on a laptop.

Then look at what it did:

forge info --artifacts artifacts/demo      # stage timings and counts
forge review --artifacts artifacts/demo    # what a human should check first

Everything it wrote is Parquet, so duckdb or polars can read it directly:

-- which fields did the sources disagree on, and how was it decided?
SELECT field, rule, count(*) AS conflicts
FROM 'artifacts/demo/conflicts.parquet'
GROUP BY 1, 2 ORDER BY 3 DESC;

The part that is not generic record linkage

Entity resolution libraries exist. Almost all of them treat a product like a generic record, and then the catalog is wrong in a way the metrics do not show:

a blue and a red T-shirt are the same product, different variants;
a 128 GB and a 256 GB iPhone 15 are the same product, different variants;
an iPhone 15 and an iPhone 15 Pro are different products;
a 0.33 L and a 1.5 L Coke are different products — while a 6-pack and a 24-pack of the same bottle are variants of one.

Collapse variants into the product and you lose the price per colour. Leave them unlinked and you lose the product page. Both are visible to the end user.

The distinction cannot be a global list of "variant attributes", because the same attribute flips role between categories:

attribute	on a phone	on an SSD
`storage_gb`	variant-defining	product-defining

attribute	on a soft drink	on a paint tin
`volume_ml`	product-defining	variant-defining

So the roles are declared per category in configs/attribute_roles.yaml, resolved along the category path from most to least specific, and an undeclared attribute is unknown — which is explicitly not a conflict. Guessing that an unknown attribute is variant-defining would split every product carrying a merchant-specific field.

electronics.phones:
  match: ["mobile phones", "smartphones"]
  attributes:
    storage_gb: variant_defining   # 128 vs 256 GB iPhone 15 = one product
    series:     product_defining   # iPhone 15 vs 15 Pro = two products

electronics.storage:
  match: ["ssd", "hard drives", "memory cards"]
  attributes:
    storage_gb: product_defining   # a 1 TB SSD is not a 2 TB SSD

Given true variant clusters, the variant → product step reaches B-cubed P 0.87 / R 0.81 / F1 0.84 on the synthetic catalog. A precision-first policy is one flag away and reaches P 0.98 / R 0.66. Both numbers are in docs/adr/0004 and in the tests, which read as a specification of the domain.

Architecture

flowchart TB
    subgraph ingest["1 · Ingest"]
        A1[Google Merchant<br/>XML · TSV]
        A2[Shopify<br/>CSV · JSON]
        A3[Amazon flat file]
        A4[Scraper JSON/L]
        A1 & A2 & A3 & A4 --> B[Canonical listing<br/>+ field-level provenance]
    end

    B --> C

    subgraph norm["2 · Normalize"]
        C[Registered transforms] --> C1[units → canonical<br/>500ml = 0.5L = 16.9 fl oz]
        C --> C2[colours → controlled vocab<br/>anthrazit = charcoal = dark grey]
        C --> C3[sizes, scoped by category<br/>38 dress ≠ 38 shoe]
        C --> C4[brands: aliases + guarded fuzzy]
        C --> C5[language detection]
    end

    C1 & C2 & C3 & C4 & C5 --> D

    subgraph enrich["3 · Enrich"]
        D[Taxonomy mapping<br/>hierarchical, abstains] --> E[Attribute extraction cascade<br/>identifiers → patterns → learned → LLM]
    end

    E --> F

    subgraph block["4 · Block"]
        F[exact GTIN/MPN] --> J[union]
        F2[MinHash-LSH on titles] --> J
        F3[embedding ANN] --> J
        F4[sorted neighborhood] --> J
    end

    J --> G

    subgraph match["5 · Match"]
        G[rules<br/>~83% of pairs, free] --> H[gradient boosting<br/>calibrated probability]
        H --> I[judge<br/>uncertainty band only]
    end

    I --> K

    subgraph cluster["6 · Resolve"]
        K[connected components] --> L[cohesion check<br/>→ correlation clustering]
        L --> M[variant → product hierarchy<br/>via attribute roles]
        M --> N[golden record<br/>declarative survivorship]
    end

    N --> O[Catalog + conflict report]
    L --> P[Review queue<br/>ranked by information gain]
    P -.labels.-> H

Each stage writes a Parquet artifact and reads the previous one's, so any stage can be re-run on its own and every intermediate is inspectable with anything that reads Parquet. That is worth more than an in-memory pipeline that is faster on the happy path and opaque when it is not.

Where the time goes

On the bundled 10,069-listing sample, one process on a laptop:

stage	seconds	throughput
normalize	1.2	8,700 listings/s
taxonomy	0.04	261,000 listings/s
extract	5.5	1,800 listings/s
blocking	3.1	3,200 listings/s
matching	13.5	50,600 pairs/s
clustering	0.5	22,300 listings/s
hierarchy	0.3	40,000 listings/s
golden records	0.6	6,100 clusters/s
total	24.6

Scale: the target is missed, and here is by how much

The design goal was a million listings in under thirty minutes. It is not met. make scale measures a real run rather than extrapolating one, and the measurement says why:

listings	candidate pairs	pairs/listing	wall clock
10,069	682,354	67.8	25 s
136,802	21,825,574	159.5	20 min 40 s

Candidate pairs grow as listings^1.33, not linearly — a denser catalog gives every listing more plausible neighbours. Extrapolating that exponent to a million listings gives roughly 306 million candidate pairs and, at the measured 27k pairs/s for matching plus 79k/s for clustering, about 4.5 hours. The 30-minute figure was wrong.

Two things can be said in mitigation, and one of them is honest engineering rather than an excuse.

Tighter blocking hits the time budget and costs recall. Dropping sorted neighborhood and the title-prefix key and raising the LSH threshold to bands=16, rows=8 cuts candidates from 83 to 5.8 per listing — a 14× reduction that would bring a million listings inside the half hour. Measured on the same synthetic catalog, pair completeness falls from 0.964 to 0.753. That is a real operating point, it is one config block away, and 21 points of blocking recall is what it costs. Whether that trade is right depends on the catalog, so the pipeline does not make it for you.

The bottleneck is one loop, and it is not fixed. Matching spends its time in per-pair Python feature extraction at 27k pairs/s. The features are arithmetic over pre-tokenised columns and would vectorise; nothing about the design prevents it. It has not been done, so the number above is what the code actually does rather than what it could.

The matching cascade pays for itself

Three stages, each an order of magnitude more expensive, each seeing only what the previous could not decide. On the sample:

stage	pairs resolved	share	µs/pair
rules	567,221	83.1%	13.4
classifier	115,103	16.9%	35.7
judge	30	0.004%	110.3

The rule layer is not a heuristic fallback — it is the layer that handles the cases where a model would be worse. Two listings with the same checksum-valid GTIN are the same variant, and no amount of learned title similarity should be allowed to overrule that.

The extraction cascade too

stage	values found	share	µs/listing
identifiers (check digits)	1	0.0%	7.1
patterns + gazetteers	18,299	67.0%	91.3
learned token classifier	9,012	33.0%	214.4
LLM (structured, cached)	0	0.0%	0.6

The identifier stage finds almost nothing here, and that is the fixture's fault rather than the stage's: the generator puts GTINs in the gtin column, where the rule matcher reads them directly, not in the title text this cascade mines. On feeds that paste EAN: 4006381333931 into the description it is the first stage to fire and the only one whose output is checksum-guaranteed. It stays first because it costs 7 µs to ask.

The LLM stage is off by default and reports zero visibly rather than being absent — see ADR 9. Every number in this README is produced without it.

Using it

Configuration is YAML, not code

run:
  seed: 42                      # every stochastic component takes it
  artifacts_dir: artifacts

sources:
  - source_id: gmc:main
    type: google_merchant
    path: feeds/merchant.xml
    priority: 90                # survivorship tie-breaks
    trust: 1.0                  # folded into every field's confidence, once
  - source_id: scraper:comp
    type: scraper
    path: feeds/scraped.jsonl
    priority: 30
    trust: 0.7

normalize:
  pipeline: [unicode_nfkc, strip_html, collapse_whitespace, squeeze_punctuation,
             detect_language, normalize_brand, normalize_color, normalize_size,
             normalize_units]
  brand_fuzzy_threshold: 0.88

blocking:
  strategies:
    - {strategy: exact_identifier}
    - {strategy: composite_key}
    - {strategy: minhash_lsh, bands: 32, rows: 4}
    - {strategy: sorted_neighborhood, window: 8}
  max_block_size: 100           # one popular brand must not restore O(n²)

forge transforms lists every registered normalizer with the fields it touches. A typo in the pipeline list fails at startup with the valid names, not three stages later as a recall drop.

CLI

forge ingest      read every configured source into the canonical schema
forge normalize   run the transform pipeline
forge match       block and match
forge run         all of the above, plus cluster, hierarchy and golden records
forge review      the highest-value pairs to look at, or record a decision
forge export      the deduplicated catalog (parquet | csv | jsonl)
forge train       train the pair classifier on generated ground truth
forge bench       run | download | ablation | scale
forge taxonomy    show | download (the full 5,595-node Google taxonomy)

Incremental

New listings resolve against the existing catalog without recomputing it: blocking structures are persisted, and only the clusters a match touches are re-clustered.

from catalog_forge.incremental import CatalogState, PersistentIndex, resolve_incremental

state = CatalogState.load("artifacts/demo")
index = PersistentIndex.load("artifacts/demo")
result = resolve_incremental(new_listings, catalog, state, index, cascade)

Adding the last 20% of the synthetic catalog incrementally reaches B-cubed F1 0.966 against 0.968 for a full rerun, with 99.8% of existing cluster ids unchanged. Ids survive growth, merges and splits by explicit rules — see ADR 8.

Review and active learning

The queue is ordered by expected information gain, not by score: entropy of the calibrated probability × blast radius of the decision × novelty against what is already queued. Without the novelty term the queue fills with fifty near-identical pairs and spends a reviewer's afternoon on one cluster.

forge review --limit 20
forge review --accept LISTING_A,LISTING_B --reviewer alex

Labels land in an append-only JSONL store and fold back into training, upweighted — a human label comes from exactly the region the model is worst in, and one row of it against a hundred thousand automatic ones is noise.

What this does not do

The point of this section is that the numbers above should be believable, and they are not unless this section is honest.

It is beaten by fine-tuned transformers on text-heavy matching. Ditto and its descendants are roughly 15 F1 points ahead on Abt-Buy. Feature-based matching buys explainability, seconds-long training and CPU inference; it does not buy state of the art.

It does not do a million listings in half an hour. Measured: 137k listings in 20 minutes, with candidate pairs growing as listings^1.33, which projects to about 4.5 hours at a million. A tighter blocking configuration fits the budget and costs 21 points of pair completeness. Both numbers are in the performance section above rather than buried here.

Blocking pair completeness is measured on synthetic and public corpora, not on your catalog. If your titles are systematically different — marketplace listings in a language the vocabularies do not cover, say — the first thing to measure is PC on your own data, not F1.

The product level is weaker than the variant level. Variant clustering reaches B-cubed F1 ~0.98 on synthetic data; grouping variants into products reaches ~0.84 with true variants as input, and lower end to end. Product grouping depends on a model anchor mined from titles, and titles are where the noise is.

The default embeddings do not cross languages. The hashing backend is character n-grams: it handles typos and rewording, not Kopfhörer ↔ headphones. Install the embeddings extra for a multilingual encoder; the default stays offline and deterministic.

The learned extraction stage is a logistic regression, not a NER model. It is trained by distant supervision on fields the feeds already declared. It is better than the patterns and worse than a real fine-tuned tagger. The stage protocol takes three methods if you have one.

No streaming. Batch and incremental-batch only. A change-data-capture integration would need a different persistence story than Parquet artifacts.

The cluster journal grows without bound. Long-lived catalogs will need compaction. Not implemented.

Vocabularies cover en/de and lean European. French, Spanish, Italian and Dutch are present but thinner. They are YAML — extending them needs no code.

Reproducing everything here

make install
make bench-data      # ~15 MB from third-party hosts, gitignored
make bench           # matching + baselines, rewrites the tables above
make ablation        # rewrites the ablation table
make scale           # the performance table

Every run is seeded. Two runs with the same config produce the same output, and the config's digest is written into run.json next to the results — comparing two artifact directories produced by different configs is the most common way to publish a wrong benchmark number.

Corpora are not vendored: they are other people's data under their own licences. forge bench download fetches Amazon-Google, Abt-Buy and DBLP-Scholar from the DeepMatcher release, and the four WDC LSPC 2017 product categories (computers, cameras, watches, shoes) from their Hugging Face mirror. WDC ships four training sizes; these numbers use medium, and the corpus's own test split.

The baselines are the versions that were current when the table was generated: Splink 4 and Dedupe 3. Both take work to run fairly, and both are configured in eval/baselines.py with a comment explaining each choice — see run_splink on why a second EM pass is not optional, and run_dedupe on why pre-labelled pairs have to reach it through prepare_training.

Documentation

docs/adr/ — the decisions that were not obvious, and why
CONTRIBUTING.md — how to add a vocabulary entry, a transform, a feature or a blocking strategy
Module docstrings carry the reasoning; configs/ carries the knowledge

License

MIT. See LICENSE.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
configs		configs
data/sample		data/sample
docs/adr		docs/adr
src/catalog_forge		src/catalog_forge
tests		tests
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml

Navigation Menu

Repository files navigation

catalog-forge

Numbers, not adjectives

Blocking

What each stage is actually worth

Two minutes

The part that is not generic record linkage

Architecture

Where the time goes

Scale: the target is missed, and here is by how much

The matching cascade pays for itself

The extraction cascade too

Using it

Configuration is YAML, not code

CLI

Incremental

Review and active learning

What this does not do

Reproducing everything here

Documentation

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

catalog-forge

Numbers, not adjectives

Blocking

What each stage is actually worth

Two minutes

The part that is not generic record linkage

Architecture

Where the time goes

Scale: the target is missed, and here is by how much

The matching cascade pays for itself

The extraction cascade too

Using it

Configuration is YAML, not code

CLI

Incremental

Review and active learning

What this does not do

Reproducing everything here

Documentation

License

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages