feat: Phase 10 two-stage retrieval and re-rank (ADR-010)

.github/AGENTS.md

@@ -32,3 +32,8 @@ GitHub Actions here must verify the native foundation without side effects.
   MovieLens, do not regenerate `results/serving_latency.*` in CI, and
   do not bind a public port. The load-test script is a measured local
   run, not a CI job.
+  Phase 10 re-rank tests (`python/tests/test_rerank.py`, picked up by
+  unittest discovery) use synthetic graphs and must not download
+  MovieLens, run `scripts/run_rerank_eval.py`, or regenerate
+  `results/rerank_eval.*` or `results/serving_latency.*`. Do not add
+  LightGBM to the CI image.

AGENTS.md

@@ -14,7 +14,8 @@ Phase 1 native foundation is verified. ADR-001 (MovieLens 100K), ADR-002
 (matrix factorization), ADR-005 (uniform sampling without replacement),
 ADR-006 (PyG SAGEConv on native `graph_sampler` neighborhoods),
 ADR-007 (multi-seed MF vs GraphSAGE leaderboard), ADR-008 (demo CLI
-serving), and ADR-009 (HTTP serving on precomputed embeddings) are
+serving), ADR-009 (HTTP serving on precomputed embeddings), and
+ADR-010 (two-stage retrieval and re-ranking) are
 accepted; MovieLens 1M is deferred. The verified slice includes the C++ CSR graph, seeded
 neighbor sampler, MovieLens 100K `u.data` parser, `graph_sampler`
 bindings, a naive Python reference sampler with native parity tests,
@@ -23,7 +24,7 @@ ADR-003 leave-one-out prep, MovieLens 100K download and on-disk
 a native-backed mini-batch neighborhood helper, GraphSAGE training on
 that harness (PyG `SAGEConv`, native neighborhoods), and CI.
 
-The current verified tip is Phase 9 / ADR-009. Phase 2 is delivered for
+The current verified tip is Phase 10 / ADR-010. Phase 2 is delivered for
 MovieLens 100K: download + on-disk `processed/` prep, and timing charts
 (native `graph_sampler` versus the Python reference sampler, with stored
 JSON/SVG under `results/`; synthetic graph by default; optional
@@ -43,9 +44,16 @@ PairScorer-compatible checkpoint, score candidate movie IDs, print top-K
 (`python/sagerec_serve.py`, `scripts/demo_recommend.py`). Phase 9 is
 delivered for the ADR-009 HTTP API: offline embedding export, FastAPI
 `GET /recommend` and `/healthz` over a NumPy top-K index, and measured
-latency under `results/serving_latency.*`. Tiny synthetic
+latency under `results/serving_latency.*`. Phase 10 is delivered for
+ADR-010 two-stage retrieval and re-ranking: stage 1 is the existing
+embedding top-N, stage 2 is a small NumPy MLP (retrieval score, train-only
+popularity, genre affinity, and recency) trained on the ADR-003 validation
+labels inside that pool. Offline comparison is
+`results/rerank_eval.json`. `GET /recommend` accepts optional `rerank`
+(default false) and the load test records p50/p95/p99 for both modes
+(`serving-latency-v2`). Tiny synthetic
 GraphSAGE metrics remain protocol smoke and must stay separate from the
-stored 100K numbers. Do not add MovieLens 1M, GCN, or batch-export
+stored 100K numbers. Do not add MovieLens 1M, GCN, LightGBM, or batch-export
 paths. Do not invent metrics or hand-edit timings.
 
 ## Read order
@@ -82,6 +90,10 @@ Accepted and recorded in `docs/decisions.md`:
 - ADR-009: HTTP serving on precomputed embeddings (FastAPI
   `GET /recommend` and `/healthz`). NumPy exact top-K. Not batch export.
   FAISS is not required.
+- ADR-010: Two-stage retrieval and re-ranking. Stage 1 is ADR-009 top-N
+  (default 200). Stage 2 is a NumPy MLP on train-only features, optional
+  on `GET /recommend` (`rerank=false` by default). Not LightGBM, not
+  per-request sampling or training, and not batch export.
 
 Do not add node2vec unless a superseding ADR accepts it.
 

Dockerfile

@@ -1,15 +1,16 @@
-# SageRec HTTP serving image (ADR-009).
+# SageRec HTTP serving image (ADR-009, optional ADR-010 re-rank).
 # Serves a precomputed sagerec_serving_artifact. It does not train,
 # compile graph_sampler, or download MovieLens. Mount the artifact at
-# /models/embeddings.npz (or set SAGEREC_SERVING_ARTIFACT).
+# /models/embeddings.npz (or set SAGEREC_SERVING_ARTIFACT). Optional
+# re-ranking needs SAGEREC_RERANK_ARTIFACT and rerank=true on the request.
 FROM python:3.12-slim
 
 WORKDIR /app
 
 COPY python/requirements-serve.txt /tmp/requirements-serve.txt
 RUN pip install --no-cache-dir -r /tmp/requirements-serve.txt
 
-COPY python/sagerec_serving.py python/sagerec_api.py /app/
+COPY python/sagerec_serving.py python/sagerec_rerank.py python/sagerec_api.py /app/
 
 ENV PYTHONPATH=/app \
     SAGEREC_SERVING_ARTIFACT=/models/embeddings.npz

Makefile

@@ -1,6 +1,7 @@
-# SageRec Phase 9 HTTP serving (ADR-009).
+# SageRec HTTP serving (ADR-009) and optional re-rank (ADR-010).
 # Export needs PYTHONPATH to include the Release build (graph_sampler) and python/.
 # The API process only needs python/ plus a .npz artifact.
+# Optional: RERANK=path/to/rerank.npz (requests still need rerank=true).
 
 PYTHON ?= python3
 export PYTHONPATH := $(CURDIR)/build:$(CURDIR)/python:$(PYTHONPATH)
@@ -21,7 +22,8 @@ export-embeddings:
 	  $(if $(TITLES),--titles $(TITLES),)
 
 serve:
-	$(PYTHON) scripts/serve_api.py --artifact $(ARTIFACT) --host $(HOST) --port $(PORT)
+	$(PYTHON) scripts/serve_api.py --artifact $(ARTIFACT) --host $(HOST) --port $(PORT) \
+	  $(if $(RERANK),--rerank $(RERANK),)
 
 loadtest:
 	$(PYTHON) scripts/loadtest_serving.py
@@ -35,5 +37,6 @@ docker-build:
 docker-run:
 	docker run --rm -p $(PORT):8000 \
 	  -v $(abspath $(ARTIFACT)):/models/embeddings.npz:ro \
+	  $(if $(RERANK),-v $(abspath $(RERANK)):/models/rerank.npz:ro -e SAGEREC_RERANK_ARTIFACT=/models/rerank.npz,) \
 	  -e SAGEREC_SERVING_ARTIFACT=/models/embeddings.npz \
 	  $(IMAGE)