Epic 1/server foundation - #4
Merged
Merged
Conversation
…iance, logs, sync) Bring the remaining EPIC-1 server sub-tasks to dev-complete and start the server side of the EPIC-3 sync protocol. All wired into the FastAPI app and covered by tests (43 server + 7 dashboard passing). - 1.B.4 Index build-job registry + status + completion webhook (services/jobs.py, services/webhook.py, tests/test_index_jobs.py; FR-36) - 1.C.5 TLS 1.3 enforcement two ways: uvicorn TLS-1.3-min entrypoint (run.py, ZETIX_SSL_*) and an nginx gateway (server/gateway/, TLSv1.3-only + dev-cert.sh). Private keys stay out of the repo (.gitignore; NFR-S3) - 1.C.6 GDPR/NDPA PII-free posture + data-handling disclosure endpoint (compliance.py, tests/test_compliance.py; NFR-S6) - 1.D.3 Privacy-preserving search-log store — non-reversible metadata only, never vectors/images (services/search_log.py, dashboard view, tests/test_search_logs.py; ADR-0002/D-7, NFR-S1/S2/S6) - 3.B.1 Catalog sync endpoints GET /sync/manifest|snapshot|delta, server side of the on-device sync protocol; ships index vectors only, never query vectors (routers/sync.py, services/sync.py, packages/contracts, test_sync.py) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion + quantisation Advance the M1 eval tooling and start the EPIC-2 on-device model pipeline. All tests pass (4 models tests, incl. a real ONNX convert+quantise). - 1.E.1 Scale generator: deterministic 10k-product labelled catalog + query set across 8 categories (eval/generate.py, `make gen-dataset`). Verified the full /admin/catalog -> /search harness runs at 10k. - 1.E.3/1.E.4 Real-embedding retrieval benchmark on real pixels with augmented (query != catalog) images (eval/image_eval.py, `make image-eval`). Runtime- verified the OpenCLIP path: Top-1 93.3% / Top-3 100% on 60 products. - 2.C.1/2.C.2 Model conversion (CLIP/MobileNet -> ONNX) and INT8 quantisation scripts + tests (conversion/, quantisation/). M1 still needs a real labelled 10k product-photo set (1.E.1 real data) to assert the >80% Top-3 bar on genuine products (1.E.4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Docs had drifted (last updated 2026-05-29) behind the committed server + models work. Reflect current reality: - EPIC-1: 1.B.4 / 1.C.5 / 1.C.6 / 1.D.3 -> dev-complete (green); 1.E.1 -> in process (scale generator + real-image benchmark exist) - EPIC-3: 3.B.1 -> dev-complete (server-side sync endpoints) - EPIC-2: 2.C.1 / 2.C.2 -> in process (conversion + quantisation scripts) - FR-36 / NFR-S3 / NFR-S6 -> green; refreshed summary counts + narrative M1 remains NOT met: still needs a real labelled 10k product-photo set and the >80% Top-3 bar on real products (1.E.1 real data / 1.E.4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….1/1.E.4) Add eval/retail_eval.py: the honest M1 measurement on real product photos. Streams nyris/stanford-online-products-v1 from the HF Hub (no full download), groups photos by product, indexes one view and queries with a *different* view through the real OpenCLIP + in-memory vector store, and reports Top-1/3/5 + per-category. `--require-bar` (make retail-eval BAR=1) fails unless Top-3 > 80%. The sample is balanced across the 12 SOP categories (shuffle + per-category cap), since SOP streams in category order. Wired in: `make retail-eval [N=.. BAR=1]`, README section, and a new `eval` extra (datasets) in server/pyproject.toml. Early real signal (stock ViT-B/32 zero-shot, no fine-tuning): * 40 products, diverse categories: Top-3 97.5% * 300 products, all one category: Top-3 75.3% (below the 80% bar) i.e. retrieval degrades under same-category density (the PRD niche-category risk) — the full 10k gate likely needs a fine-tuned model (see D-6/OQ-6). The definitive run is `make retail-eval N=10000 BAR=1` (a long job — streams most of the ~120k-image set to cover all categories). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ase 1
The real product-photo benchmark (make retail-eval) gives the first honest M1
signal: stock OpenCLIP ViT-B/32 zero-shot = Top-3 97.5% on a small diverse
sample but 75.3% on 300 same-category items — below the >80% bar. Retrieval
degrades under same-category density (PRD niche-category risk); clearing M1 at
10k scale likely needs a fine-tuned model (D-6/OQ-6).
- TASKS.md / FEATURE_STATUS.md: 1.E note records the measured result; M1 not met
- GAP_ANALYSIS.md: Phase 1 table was fully stale ("0% complete", every row Not
started) — reconciled to reflect the dev-complete server pipeline + the eval
bar as the one remaining gap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The full M1 gate (--n 10000) streams most of the ~120k-image SOP set and embeds ~20k photos — several minutes to hours on CPU. Emit [collect]/[embed] progress to stderr so the run is observably alive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cal adapters Records the fine-tuning decision: one shared, fine-tuned CLIP base (trained on a broader sourced retail dataset, disjoint from the M1 eval set) plus lightweight per-vertical LoRA-style adapters for niche categories that lag. Chosen for the on-device size budget (NFR-P6, one <25MB base) and multi-store composability over shipping N per-vertical models. Marks D-6/OQ-6 resolved in TASKS.md + GAP_ANALYSIS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The full-scale gate (make retail-eval N=10000 --require-bar) is in: stock OpenCLIP ViT-B/32 zero-shot on real product photos scores Top-1 45.6% / Top-3 55.5% / Top-5 59.7% across all 12 categories — far below the 80% bar (exit 1). Weakest: coffee maker 39.2%, kettle 43.2%, bicycle 43.5%; strongest: table 68.6%, chair 66.3%. Accuracy collapses under same-category density at scale, confirming fine-tuning is required (ADR-0003). Supersedes the earlier small-sample figures across FEATURE_STATUS / TASKS / GAP_ANALYSIS / ADR-0003. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0003) Add models/finetune/: the shared-base fine-tuning pipeline to lift retail retrieval past the M1 bar (stock zero-shot baseline: Top-3 55.5%). - abo.py: ingest Amazon Berkeley Objects (CC BY 4.0; Amazon-sourced, so disjoint from the eBay-sourced SOP eval set). Joins listings metadata (item_id, product_type, main/other_image_id) with the image id->path map and emits a positive-pairs manifest with a product-disjoint train/val split. - dataset.py: AboPairDataset yields two views of a product for contrastive pairs. - train.py: freezes the base, injects LoRA (peft) on the visual transformer's linear layers, trains a CLIP-style symmetric InfoNCE loss on MPS, merges the adapter back in, and saves a plain OpenCLIP checkpoint (no peft at inference). - Embedder: OpenCLIPEmbeddingService loads ZETIX_CLIP_CHECKPOINT over the stock weights when set (default unchanged). - Makefile: ft-data / ft-manifest / ft-train / ft-eval. New [finetune] extra (peft, imagehash). data/ + ckpt/ git-ignored (3GB archives, 600MB checkpoints). Validated end-to-end on synthetic data: LoRA wraps 36 visual linears (0.58% trainable), trains + merges on MPS, checkpoint round-trips through the server embedder (512-dim, L2-normalised). Server tests still green (43). Real ABO training + ft-eval pending the 3GB download. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Save a non-destructive merged OpenCLIP checkpoint after each epoch (merge_adapter/unmerge_adapter) as insurance for long MPS runs and so an early epoch can be evaluated without waiting for the full run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parse the train.py log into a one-screen snapshot — progress bar, %, ETA, it/s, loss sparkline, and saved checkpoints. `make ft-watch` for a snapshot; `make ft-watch-live` (or `watch -n 3 make ft-watch`) for a real-time view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…line Add models/finetune/dashboard.py (+ dashboard.html): a small FastAPI app that renders the whole run live in the browser — training progress bar, ETA, a real-time loss chart (vanilla canvas, no CDN), saved checkpoints, then the M1 eval phase and the final Top-1/3/5 + per-category table with the lift vs the 55.5% baseline. Auto-refreshes every 2s; read-only (only parses logs). - logparse.py: shared parser (parse_train/parse_eval) used by both the browser dashboard and the terminal watch, so they agree on the numbers. Fixes a progress bug — train.py's step counter is already global, so the % no longer double-counts past 100. - watch.py: refactored onto the shared parser. - Makefile: `make ft-dashboard` (PORT/LOG overridable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…plit + held-out eval The ABO fine-tune barely moved the held-out SOP benchmark (Top-3 55.5%→56.2%), with gains concentrated in ABO's strong categories (furniture) — pointing at a train/eval domain gap. To test that directly: - sop.py: cache Stanford Online Products locally (non-streaming download for reliability) grouped by product; emit an ABO-schema manifest with a category-stratified, product-disjoint train/test split (11,318 products → 7,540 train / 3,778 test, all 12 categories). - retail_eval.py: add --local-manifest/--split to evaluate a held-out split from local images (leak-free), so we can compare stock vs fine-tuned on the SAME held-out products. - Makefile: ft-sop-data / ft-sop-train / ft-sop-eval. - ADR-0003 + FEATURE_STATUS: record the attempt-1 result and the diagnostic plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The held-out SOP baseline scored Top-3 81.4% at a 3,778-product gallery vs 55.5% at 10k — suggesting retrieval accuracy is dominated by catalog size, not the model. Add --distractor-split to retail_eval: index extra products as catalog-only (never queried), enlarging the gallery while holding the query set fixed, so gallery size can be varied as a single controlled variable. Report now prints queries + gallery size (+distractors). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add --diagnostic mode to the dashboard: renders the 2x2 model×catalog-size Top-3 matrix (stock vs fine-tuned, small vs large gallery) with lift arrows and PASS/FAIL, the SOP training loss curve, a one-line verdict, and the per-category table for the worst-case (fine-tuned @ large gallery). Reads the four SOP diagnostic eval logs. Replaces the stale single-run ABO view. Result shown: stock 81.4/75.3, fine-tuned 92.6/88.6 — in-domain fine-tuning clears 80% at both catalog sizes (+11.2 / +13.3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cached held-out eval used adjacent view-pairs (easier) than the streaming harness's random pairs, inflating absolute numbers. --shuffle-views picks the catalog/query views at random (seeded) from each product's photos, matching the streaming difficulty while staying leak-free — so the fine-tuned model can be measured on a test that is both HARD and disjoint from training. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repoint the diagnostic matrix to the leak-free @11k gallery with easy vs hard (random, realistic) view-pairs. Hard column is the honest, streaming-equivalent measurement: stock 56.9% (≈ the original 55.5% baseline, validating the test) → fine-tuned 78.8% (+21.9). Verdict leads with the honest number. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…esolved) Attempt 2 (in-domain SOP fine-tune, leak-free hard-pair held-out eval): stock 56.9% -> fine-tuned 78.8% (+21.9), vs +0.7 for out-of-domain ABO. Confirms the domain-gap hypothesis: it's a data problem, not a method problem. Lands 1.2 pts under the 80% bar from a quick 5-epoch diagnostic with clear headroom. Path to M1 is now in-domain training data at scale. ADR-0003 result log + FEATURE_STATUS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rgets The shipped model must handle many marketplaces (eBay/Amazon/AliExpress/… and local Jumia/Konga), so training data must span domains — single-domain does not transfer. combine.py merges per-source manifests into one domain-labeled, domain-BALANCED corpus (per-source cap so ABO's ~124k doesn't drown SOP's 7.5k), rewriting image paths to absolute so dataset.py resolves across source dirs. Built: 12,000 ABO + 7,540 SOP = 19,540 balanced train products across 2 domains. Makefile: ft-combine / ft-combine-train / ft-jumia-data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ion test) Robots-respecting (bot-identified UA, rate-limited), sitemap-driven fetcher that builds a small Jumia (jumia.com.ng) eval set — an unseen Nigerian marketplace — to measure cross-domain generalization of the domain-general model. Eval-only. Gallery extraction is pattern-agnostic: groups all jumia.is image URLs by CDN directory and takes the largest group (the product's own gallery), handling both seller (/cms/external/pet/<SKU>/) and Jumia (/product/<id>/N.jpg) hosting, and filtering out recommendation-carousel images. Retries transient network errors. Note: many Jumia PDPs expose only 1 static image (rest lazy-load via JS), so per-product yield is ~1/3; enough for a directional local signal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets the per-domain eval hold gallery size comparable across domains (ABO's 13.8k val vs SOP's 3.8k test) for a fair stock-vs-combined comparison. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Attempt 3 — shared base on SOP(eBay)+ABO(Amazon), balanced, rank-16. Per-domain held-out (hard pairs, ~11k): stock 56.9/22.7 -> combined 76.0/35.0. One model beats stock on BOTH domains with no catastrophic forgetting (76.0 on SOP vs the SOP-specialist's 78.8) — the multi-marketplace shared-base approach is validated. Caveats: ABO's 360-spin views make it an adversarially hard domain (not representative of real phone-photo search); hard domains are the case for per-domain LoRA adapters. Local (Nigerian) data remains a data-access task (Jumia scraping impractical; use Konga feed / real-machine collection). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.