Multi-Stage Recommendation Ranking Pipeline

End-to-end offline recommendation system — retrieval → ranking → diversity reranking — built on Amazon Electronics (1.64M users, 15.5M interactions). Pure PyTorch, no external recsys libraries.

What This Demonstrates

• Multi-stage recommender architecture: retrieval, ranking, and reranking as independently evaluable, composable stages
• Retrieval: ItemCF (cosine-IUF similarity) + Two-Tower (in-batch contrastive learning) with multi-channel weighted fusion
• Ranking: DeepFM and DIN built from scratch in pure PyTorch; evaluated on the full 1.64M-user validation split with no subsampling (~164M scored user-item pairs)
• Diversity reranking: MMR using DIN item embeddings as the similarity signal; two-phase evaluation isolating the effect of candidate pool composition
• Metric-driven experimentation: per-stage ablations, activity-stratified analysis, lambda sweep behavior characterized across candidate distributions
• Engineering depth: 135× evaluation speedup via profiling and numpy vectorization

Key Results

Stage	Metric	Baseline	This system	Δ	Eval setting
Recall fusion	Recall@200	0.0421 (ItemCF alone)	0.0435	+3.3%	Full validation split
Ranking	NDCG@10	0.3981 (DeepFM)	0.4421 (DIN)	+11.1%	Controlled (1+99)
Ranking	HR@10	0.5915 (DeepFM)	0.6369 (DIN)	+7.7%	Controlled (1+99)
End-to-end: DIN vs recall-only	HR@10	0.0133	0.0228	+71%	End-to-end (real recall)
End-to-end: MMR reranking	ILS@10	0.096 (λ=1.0)	0.017 (λ=0.7)	−82%	End-to-end (real recall)

Note on end-to-end metrics: Phase B HR@10 is capped near 4.35% (= Recall@200 = 0.0435) because the ground-truth item is in the recall pool for only 4.35% of users. Gains are measured across pipeline configs. MMR at λ=0.7 retains 81% of pure-DIN HR@10 (0.0186 / 0.0228) at 82% lower intra-list similarity. Phase A (Stage 3) shows HR@10 up to 0.62 — those are controlled experiments where the positive was placed in every user's candidate set by construction, not end-to-end pipeline results.

Architecture

graph LR
  A["12.2M Train<br/>Interactions"] --> B[Recall]
  B --> B1["ItemCF<br/>R@200=0.0421"]
  B --> B2["Two-Tower<br/>R@200=0.0273"]
  B1 --> C["Multi-Channel Merge<br/>alpha=0.9<br/>R@200=0.0435"]
  B2 --> C
  C --> D["DIN Ranking<br/>GAUC=0.836<br/>NDCG@10=0.442"]
  D --> E["MMR Reranking<br/>lambda=0.7<br/>ILS -82%"]
  E --> F[Final Top-10]

Loading

Each stage produces cached parquet outputs and is independently re-evaluable. The full end-to-end evaluation on 200K users runs in ~14 seconds (after one-time DIN scoring).

Dataset

Amazon Reviews 2023 Electronics — 5-core (≥ 5 interactions per user and item)

• 1,641,026 users · 367,052 items · 15,473,536 interactions (train 12,191,484 / val 1,641,026 / test 1,641,026)
• Temporal leave-last-1-out split: val = second-latest interaction, test = latest

python scripts/download_data.py --source official_5core
python scripts/prepare_data.py --config config/default.yaml

Stage 1: Candidate Retrieval

Two recall channels trained independently and fused via per-user min-max normalized weighted-score blending. Both produce 200 candidates per user; the merge pipeline sweeps the ItemCF weight (α) from 0.0 to 1.0. Evaluated on the full 1,641,026-user validation split.

Method	Recall@50	Recall@200
Two-Tower (N=10 samples/user)	0.0114	0.0273
ItemCF (cosine-IUF)	0.0251	0.0421
Merged (α=0.9)	0.0255	0.0435

Note: the standalone Two-Tower Recall@200 above (0.0273, from experiments/summary/recall_two_tower.csv) does not match the α=0.0 degenerate point of the fusion sweep (0.0292, from experiments/summary/merge_alpha_sweep.csv), even though both nominally isolate the pure Two-Tower channel and Recall@50 agrees exactly between the two (0.0114). The cause could not be determined from the code in this pass — candidates include the two files being generated from different training runs/checkpoints of the Two-Tower model, or tie-breaking in the merge's score union at α=0.0 (see src/pipeline/merge_recall.py::load_and_merge_candidates). Flagged rather than reconciled.

Engineering insights:

• ItemCF substantially outperforms the pure-ID Two-Tower (Recall@200: 0.0421 vs 0.0273) in this sparse implicit-feedback setting. Pure-ID embeddings underfit when interaction density is low — collaborative filtering is the right tool here.
• Two-Tower still adds complementary signal: the merged system (α=0.9) yields +3.3% over ItemCF alone. The optimal 90/10 split is consistent with Two-Tower contributing items at the margin rather than at the top, though this repo does not measure candidate overlap directly — see Limitations.
• Two-Tower max_samples_per_user sweep (N=1 → 10) showed monotonic improvement; N=10 yielded an 84% Recall@200 lift over N=1 (0.0273 vs 0.0148).
• Cold-item recall is 0 for both channels — both retrieve only items seen during training. A feature-based item tower (metadata + text embeddings) is the clear next step.

Reproducing Stage 1

python scripts/prepare_data.py --config config/default.yaml
python scripts/run_recall.py --config config/default.yaml --method itemcf \
  --eval-split val --save-candidates experiments/candidates/itemcf_val.parquet
python scripts/run_two_tower.py --config config/default.yaml \
  --eval-split val --max-samples-per-user 10 \
  --save-candidates experiments/candidates/two_tower_val.parquet
python scripts/run_merge_recall.py --config config/default.yaml --sweep-alpha
python scripts/plot_alpha_sweep.py

Stage 2: Neural Ranking

Ranking models are trained on an independently-sampled candidate set (1 positive + sampled negatives per user), not on Stage 1's retrieved candidates. The stages are wired together at evaluation time in Stage 3 Phase B, which scores the real fused top-200 pool. This train/serve candidate mismatch is a known limitation (see Limitations). Two models built from scratch in pure PyTorch — no DeepCTR, no Torch-RecSys:

• DeepFM — mean pooling over user history (baseline)
• DIN (Deep Interest Network) — target-aware attention pooling over user history

Scale: 30.3M train rows (≤5 most-recent positives/user, avg 3.7/user, each with 1 positive + 4 sampled negatives) · 163.9M validation rows (1,638,682 users × 100 candidates each — 2,344 of the original 1,641,026 validation users are excluded because their ground-truth item isn't in the training vocabulary; 1 positive + 99 sampled negatives per remaining user). No validation subsampling.

Features

Side	Key fields
User	interaction history (max 50 items), avg rating, avg price (log-scaled), preferred category, activity segment
Item	category_leaf (1,021 values), category_l2, store (top-1,000 + OTHER bucket for long-tail sellers), price (log+imputed), avg rating, title length

Results

Model	AUC	GAUC	NDCG@10	HR@10
DeepFM	0.8320	0.8330	0.3981	0.5915
DIN	0.8324	0.8361	0.4421	0.6369

Per-Activity-Segment GAUC

Segment	DeepFM	DIN	Δ
Low-activity (≤3, effectively exactly 3)	0.8341	0.8109	−0.0232
Medium-activity	0.8359	0.8386	+0.0027
High-activity	0.8255	0.8609	+0.0354

Engineering insights:

• DIN's top-of-list improvement is disproportionately large: NDCG@10 +11.1% and HR@10 +7.7% vs overall GAUC +0.3pp. Target-aware attention helps most at the very top of the ranking — the slots that matter in a production feed — rather than improving average pairwise ordering uniformly.
• DIN benefits high-activity users; regresses on low-activity users. With only ~3 history items, attention overfits to spurious patterns that mean pooling smooths over (−2.3pp GAUC). DIN's aggregate gain comes entirely from medium and high-activity segments.
• Architectural implication for production: Gate between DeepFM (mean pooling) for sparse users and DIN (attention) for engaged users. The crossover behavior measured here would directly motivate that design choice in a real deployment.
• Attention learns L2-level category clusters, not leaf-level intent: Across 6 case-study users, 0 had the top-attended history item from the candidate's leaf category (1,021 labels). Attention concentrated at L2 granularity (e.g., "Power Accessories" for a "Surge Protectors" candidate) — soft category-cluster matching, not the strict field alignment described in the original DIN paper.

Detailed attention analysis: experiments/summary/din_attention_report.md

Reproducing Stage 2

python scripts/build_item_features.py --config config/default.yaml
python scripts/build_user_features.py --config config/default.yaml
python scripts/build_encoded_features.py --config config/default.yaml
python scripts/build_ranking_samples.py --config config/default.yaml
python scripts/train_deepfm.py --config config/default.yaml
python scripts/train_din.py --config config/default.yaml
python scripts/extract_din_attention.py --config config/default.yaml
python scripts/plot_ranking_comparison.py

Stage 3: Diversity-Aware Reranking

MMR (Maximal Marginal Relevance) reranks DIN's top-50 candidates into a diversified top-10. The diversity signal is cosine similarity of DIN's own item embeddings — the same representation space as the ranking signal, so the diversity measure is semantically coherent with relevance.

λ=1.0 reproduces pure DIN ranking; λ=0.0 maximizes diversity.

Two evaluation phases on a 200,000-user random sample of the validation split (sampled without replacement, seed=42 — see scripts/run_end_to_end_eval.py::_sample_users):

• Phase A — 1 positive + 99 random negatives per user (controlled, isolates the λ trade-off curve)
• Phase B — Real recall candidates from Stage 1 (end-to-end, tests MMR on semantically clustered candidate pools)

Phase A: Lambda Sweep on Random-Negative Validation

λ	NDCG@10	HR@10	Coverage@10	ILS@10
0.0	0.3096	0.3759	5.33	−0.072
0.3	0.3824	0.5478	5.03	−0.061
0.5	0.4151	0.6064	4.90	−0.038
0.7	0.4277	0.6224	4.85	−0.014
1.0	0.4321	0.6261	4.85	+0.008

The Pareto curve is flat because random negatives are already diverse — DIN's top-10 already spans ~4.85 L2 categories before reranking. λ=0.7 costs only −1% NDCG@10 while shifting ILS from +0.008 to −0.014 (a measurable diversity improvement at negligible relevance cost).

Phase B: End-to-End Pipeline on Real Recall Candidates

Stage	HR@10	NDCG@10	Coverage@10	ILS@10
Recall only (merge top-10)	0.0133	0.0080	4.89	0.011
+ DIN ranking	0.0228	0.0130	2.82	0.096
+ DIN + MMR (λ=0.7)	0.0186	0.0108	2.94	0.017

Phase B full lambda sweep on the end-to-end pipeline:

λ	HR@10	NDCG@10	Coverage@10	ILS@10
1.0	0.0228	0.0130	2.82	+0.0961
0.7	0.0186	0.0108	2.94	+0.0172
0.5	0.0162	0.0098	3.07	−0.0222
0.3	0.0137	0.0087	3.34	−0.0529
0.0	0.0095	0.0070	3.87	−0.0674

λ=1.0 reproduces the pure-DIN row; as λ decreases, HR@10 declines monotonically, Coverage rises, and ILS turns negative at λ=0.5. λ=0.7 retains 81% of pure-DIN HR@10 (0.0186 / 0.0228) at substantially lower intra-list similarity. Ground-truth in recall pool: 4.35% (matches Recall@200 = 0.0435 ✓).

Per-segment results (HR@10):

Segment	Recall only	+ DIN	DIN gain	+ MMR (λ=0.7)	MMR cost
Low-activity	0.0173	0.0266	+54%	0.0212	−20%
Medium-activity	0.0136	0.0237	+74%	0.0194	−18%
High-activity	0.0078	0.0164	+110%	0.0135	−17%

Absolute HR@10 is inversely related to activity because high-activity users have broader, harder-to-predict interests. DIN's relative gain still rises with activity (+54% low / +74% medium / +110% high), consistent with the Stage 2 segment analysis.

Lambda Behavior Across Candidate Distributions

The same 5-λ sweep run on both Phase A and Phase B. λ=0.7 is selected by the Phase B rule (_ILS_SWEET_SPOT_THRESHOLD in scripts/run_end_to_end_eval.py): the highest λ whose ILS@10 stays under 0.03. Phase A is not an independent selection of the same operating point — under that same rule, every λ in the Phase A sweep satisfies ILS ≤ 0.03 (even λ=1.0, at +0.0076), so the rule would select λ=1.0 in Phase A, not 0.7, since random-negative candidate pools are already diverse. (Phase A's own script, run_rerank_sweep.py, uses a different coverage-midpoint criterion and picks λ=0.0 — see experiments/summary/rerank_mmr_report.md.) What Phase A does show is that λ=0.7 costs only −1% NDCG@10 on a clean candidate distribution, i.e. the operating point does not degrade when the pool is easy. ILS Gap = Phase B ILS − Phase A ILS.

λ	Phase A NDCG@10	Phase B HR@10	Phase A ILS	Phase B ILS	ILS Gap
1.0	0.4321	0.0228	+0.0076	+0.0961	+0.0885
0.7	0.4277	0.0186	−0.0138	+0.0172	+0.0310
0.5	0.4151	0.0162	−0.0377	−0.0222	+0.0155
0.3	0.3824	0.0137	−0.0611	−0.0529	+0.0082
0.0	0.3096	0.0095	−0.0723	−0.0674	+0.0049

Per-segment residual ILS at λ=0.7:

Segment	ILS at λ=1.0	ILS at λ=0.7
Low-activity	0.092	0.012
Medium-activity	0.097	0.017
High-activity	0.098	0.023

Engineering insights:

• MMR's value scales with upstream semantic homogeneity. On random negatives (Phase A), MMR's ILS impact was 0.022 — DIN's top-10 was already diverse. On real recall candidates (Phase B), the same algorithm reduced ILS by 82% (0.096 → 0.017). The variable is isolated cleanly: collaborative-filtering recall produces semantically clustered candidates that embedding-based MMR is well-suited to diversify.
• Embedding-based MMR operates within categories, not across them. Coverage@10 moved only +4% (2.82 → 2.94) while ILS dropped 82%. MMR selects items semantically distant within the same L2 category — sub-category differentiation that coarse category labels cannot capture.
• DIN creates the homogeneity that MMR then partially undoes. Ranking raises ILS@10 from 0.011 to 0.096 (≈9×) and cuts category Coverage@10 from 4.89 to 2.82 (−42%). MMR recovers most of the similarity (0.096 → 0.017) but only +4% of coverage (2.82 → 2.94), and final ILS still sits above the recall-only baseline (0.017 vs 0.011). This is consistent with the point above: MMR differentiates within an L2 category, so cross-category diversity lost at the ranking stage is not recoverable at rerank.
• Engineering: The evaluation loop initially had quadratic complexity — per-user boolean indexing on a 40M-row DataFrame generated 8 trillion comparisons for 200K users. Refactored to a globally pre-sorted array with O(1) numpy slice lookups via precomputed user offsets: 135× speedup (30+ min → 13.6s).

Reproducing Stage 3

python scripts/save_din_scores.py --config config/default.yaml
python scripts/save_item_embeddings.py --config config/default.yaml
python scripts/run_rerank_sweep.py --config config/default.yaml
python scripts/run_end_to_end_eval.py --config config/default.yaml --lambdas "0.0,0.3,0.5,0.7,1.0"
python scripts/plot_rerank_pareto.py
python scripts/plot_pipeline_comparison.py

Stage 4: Serving Demo

A FastAPI serving layer that turns the offline pipeline into a local, production-style HTTP service. This is a V1 demo using precomputed DIN scores from Stage 3 — real-time DIN inference (TorchScript/ONNX export) is documented as the V2 path and left for follow-up.

What this stage demonstrates:

• API design with Pydantic-validated request/response schemas
• Lifespan-managed artifact loading (load-once, serve-many pattern)
• Sampled artifact strategy for tractable local demo (5K users, < 100 MB)
• Latency benchmarking in both core and HTTP modes to isolate framework overhead

Request flow:

graph LR
  A[Client] --> B["FastAPI /recommend"]
  B --> C["ArtifactStore<br/>candidates lookup"]
  C --> D["Recommender<br/>top-N by DIN score"]
  D --> E["MMR<br/>(lambda=0.7)"]
  E --> F[JSON Response]
  F --> A

Loading

Endpoints

Method	Path	Purpose
GET	/health	Service status, artifacts loaded
GET	/recommend	Top-K reranked recommendations for a user
GET	/debug/user/{user_id}	Inspect user's history, recall pool, GT presence
GET	/sample_users?n=10	Random valid user IDs for exploration

Example request and response

# Get 5 sample user IDs, then query one
curl "http://localhost:8000/sample_users?n=5"
curl "http://localhost:8000/recommend?user_id=<id>&k=5&lambda_mmr=0.7"

{
  "user_id": "...",
  "k": 5,
  "lambda_mmr": 0.7,
  "candidate_limit": 50,
  "latency_ms": 1.4,
  "recommendations": [
    {"rank": 1, "item_id": "B0...", "din_score": 0.89,
     "mmr_score": 0.62, "category_l2": "Cables", "category_leaf": "USB Cables"},
    {"rank": 2, "item_id": "B0...", "din_score": 0.85,
     "mmr_score": 0.54, "category_l2": "Power Accessories", "category_leaf": "Surge Protectors"}
  ],
  "warning": null
}

Latency benchmark

Measured on Apple M4 Max 64GB, 500 requests (50 warmup excluded), k=10, lambda_mmr=0.7, candidate_limit=50.

Mode	Mean	p50	p95	p99
Core (in-process)	0.10ms	0.10ms	0.11ms	0.13ms
HTTP (loopback)	1.32ms	1.30ms	1.44ms	1.61ms

Engineering insights:

• The same load-once-serve-many pattern that gave Stage 3's 135× speedup applies here: candidates are grouped by user at startup, so each request is an O(1) dict lookup + numpy slice rather than a DataFrame scan.
• The core-vs-HTTP gap is the FastAPI + Pydantic + JSON overhead — useful to know when budgeting latency in production where a more compact protocol (e.g., gRPC) would recover most of that gap.
• This V1 demo deliberately avoids real-time DIN inference. V2 path: export DIN to TorchScript with INT8 quantization, expect +10–30ms per request, host on a separate model-server process.

Stage 4 setup

# Install serving dependencies
pip install fastapi 'uvicorn[standard]' pydantic requests

# Export sampled artifacts (one-time, ~30s)
python scripts/export_serving_artifacts.py \
    --num-users 5000 \
    --output-dir experiments/serving_artifacts \
    --config config/default.yaml

# Start the server
uvicorn serving.app:app --host 0.0.0.0 --port 8000

# In another terminal, run the benchmark
python serving/benchmark.py --mode core --num-requests 500
python serving/benchmark.py --mode http --num-requests 500

Production Considerations

This is an offline research benchmark. A production deployment would additionally require:

Component	What's needed
Recall serving	FastAPI endpoint, precomputed candidate index, ANN search (Faiss / ScaNN)
User-history cache	Redis or equivalent for bounded per-request retrieval
Model serving	TorchScript / ONNX export with INT8 quantization for DIN inference
Latency benchmark	End-to-end p99 measurement (recall + rank + rerank)
Online evaluation	A/B test on CTR, session diversity, and return rate

The λ=0.7 MMR operating point was selected from offline diversity-relevance trade-offs. The actual production decision requires A/B testing — diversity interventions often reduce short-term CTR while improving long-term retention and trust metrics, which offline metrics cannot distinguish.

Limitations

• Cold-item recall is unsolved. Both ItemCF and the pure-ID Two-Tower retrieve only items present in the training vocabulary; 2,344 validation users (0.14% of 1,641,026) have a ground-truth item outside the vocabulary and structurally cannot get a hit — Recall@200 = 0 for those cases. A feature-based item tower using item metadata and pre-trained text encoders would address this.
• Evaluation is offline only. Recall@K, NDCG@K, HR@K, ILS, Coverage — no online signals. Diversity interventions especially require A/B testing since short-term CTR can decline while long-term retention improves.
• Negatives are implicit, not impression-based. Without exposure logs, negatives are sampled uniformly from the item vocabulary (excluding each user's train history). Exposed-but-unclicked negatives carry stronger signal and would be used in a production training pipeline.
• Ranking is trained on a different candidate distribution than it serves. Training negatives are sampled uniformly from the item vocabulary; at end-to-end evaluation the model scores collaborative-filtering candidates, which are far harder and semantically clustered. Training on mined recall candidates would close this gap.
• Retrieval complementarity is inferred, not measured. The +3.3% fusion gain is evidence that the two channels differ, but there is no candidate-overlap analysis (unique hits per channel, coverage decomposition) to confirm how they differ. A per-channel unique-hit breakdown is the natural next step.
• No popularity baseline. Recall@200 = 4.35% (Stage 1 fusion) is reported without a MostPopular reference point, so the absolute quality of the retrieval stage cannot be judged — only the relative gain from fusion.
• Serving is a local demo, not a deployed service. It uses precomputed DIN scores and sampled artifacts; production deployment would require real-time model serving, distributed caching, ANN recall, observability, and CI/CD.

Project Status

• ✅ Stage 1: Candidate Retrieval — ItemCF + Two-Tower fused, Recall@200 = 0.0435 (+3.3% over ItemCF alone)
• ✅ Stage 2: Neural Ranking — DeepFM + DIN, GAUC = 0.8361, NDCG@10 = 0.4421 (+11.1% over DeepFM)
• ✅ Stage 3: Diversity-Aware Reranking — MMR end-to-end pipeline, HR@10 +71% over recall-only, ILS −82%; λ=0.7 selected by the Phase B ILS threshold, behavior characterized on both candidate distributions
• ✅ Stage 4: Serving Demo — FastAPI service with Pydantic schemas, 5K-user sampled artifacts, latency-benchmarked in core and HTTP modes; precomputed DIN scores (V2 with real-time inference documented as next step)