[MarcusElwin]

Summary

This PR adds the design and the first working pipeline for crawl2action. It covers crawling e-commerce catalogs and labeling them with Jev; post-training and serving are scaffolded but not yet run.

• Design: docs/PLAN.md covers sources, task families, the decision layer (the System One contract), the post-training recipe (Tinker primary, Modal fallback), C2A-Bench, serving on Modal and the milestones.
• Skeleton (M0):
  • UCP-aligned schemas (prices in integer minor units);
  • a c2a CLI;
  • scoring checks and ranking metrics (the single source of truth for rewards, evals and serving guards);
  • calibration metrics;
  • training record formats, rewards and advantage estimators (Dr.GRPO/GRPO, Rank-GRPO-style, DAPO dynamic sampling);
  • Tinker SFT/RL/OPD loops, tested against a fake backend;
  • a dev FastAPI gateway;
  • CI running ruff and pytest.
• Compliant crawling (M1):
  • Default-deny store registry: c2a registry approve|deny --by records who approved a store and when.
  • CrawlSession is the single gate for every store fetch: terms approval, robots.txt, same domain, rate limit, and retries honouring Retry-After.
  • Ingestion paths:
    • UCP catalog (spec 2026-01-01);
    • Shopify /products.json (currency fallback, short/repeated page guards);
    • Firecrawl v2 (map → URL triage → Jev page-type gate → JSON extract, with a credit budget).
  • Discovery: platform fingerprinting, and search-based candidates that are written for review and never auto-added.
  • Incremental c2a crawl with a manifest per store.
  • Open-dataset importers: ESCI → human pair@1 labels, Amazon-M2 sessions, Amazon Reviews 2023.
• Jev labeling:
  • A System One client for hosted Jev.
  • Versioned question sets: page_v1, product_v1, and pair_v1 (exact / substitute / complement / irrelevant).
  • Confidence gating, with escalation to the teacher and then human review.
  • An on-disk response cache and request budget.
  • Exports to C2A-Bench gold and Kev-format decider training data. Jev-only labels are excluded by default until TypeSafe's terms are checked.
• Pilot tooling:
  • .env loading;
  • c2a doctor [--live] (preflight with fix commands);
  • c2a pilot (crawl → product labels → pair labels → report);
  • c2a label review (terminal review → human gold labels);
  • c2a report;
  • the runbook docs/PILOT.md.

How to test locally

Full steps are in docs/PILOT.md. Short version:

git checkout claude/magical-davinci-7u43rl
uv sync --extra dev
uv run pytest -q                      # 95 offline tests
cp .env.example .env                  # add TYPESAFE_API_KEY (FIRECRAWL_API_KEY only for non-Shopify stores)
uv run c2a registry approve allbirds --by "<you>" --note "ToS reviewed ..."   # only after reviewing the store's terms
uv run c2a discover fingerprint --store allbirds --write
uv run c2a doctor --live
uv run c2a pilot --store allbirds --max-products 50 --max-jev-requests 200
uv run c2a label review --limit 20
uv run c2a report

Not verified live

All tests run offline with mocked HTTP. No real calls have been made to stores, Jev, Firecrawl or Tinker; the development sandbox could not reach them. Please confirm on the first local run:

• Jev's real response shape matches decide/systemone.py. c2a doctor --live prints the raw body if it doesn't.
• The Firecrawl /v2/search response shape, used only by discover expand.
• The Shopify currency fallback (registry → /meta.json → country default) for multi-currency stores.
• Seed store platforms, which are best guesses until fingerprint confirms them.

Open items

• TypeSafe's terms on training our decider from Jev outputs.
• Jev and Firecrawl pricing at 1–2M products.
• A terms review for each store before approval. All seed stores are unreviewed.
• Which models the Tinker account can train.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KfA2SBD7z2s34NMaDPRf9f

---

Generated by Claude Code