crawl2action: E2E plan, skeleton, compliant crawling and Jev labeling pilot

.env.example

@@ -0,0 +1,10 @@
+# Copy to .env (never commit). In production these live in Modal Secrets.
+FIRECRAWL_API_KEY=
+TEACHER_API_KEY=
+TEACHER_BASE_URL=
+TINKER_API_KEY=
+TYPESAFE_API_KEY=
+HF_TOKEN=
+WANDB_API_KEY=
+# AC2 (later)
+AC2_API_KEY=

.github/workflows/ci.yml

@@ -0,0 +1,19 @@
+name: CI
+on:
+  push:
+  pull_request:
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: uv sync --extra dev
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run pytest -q

.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+.venv/
+.env
+.pytest_cache/
+.ruff_cache/
+dist/
+build/
+*.egg-info/
+data/
+runs/

.python-version

@@ -0,0 +1 @@
+3.11

CLAUDE.md

@@ -0,0 +1,20 @@
+# crawl2action — contributor notes
+
+- Design: `docs/PLAN.md` (milestones M0–M5). Keep it in sync when decisions change.
+- Setup: `uv sync --extra dev`. Checks (run before every push, same as CI):
+  `uv run ruff check . && uv run ruff format --check . && uv run pytest -q`
+- Optional integrations (tinker, modal, firecrawl, vllm) are extras. Import them lazily inside
+  functions so the base install and tests never need them.
+- Stubs raise `c2a.NotYetImplemented(what, milestone)`; CLI stubs exit with code 2.
+- Crawling is default-deny: a store needs `enabled: true` AND `tos_status: approved`. Only a human
+  sets these (`c2a registry approve --by`); automated code uses `Registry.update_descriptive`.
+- Every crawl fetch from a store goes through `compliance.CrawlSession` (ToS + robots + domain +
+  rate limit). The only exception is discovery fingerprinting, which reads public metadata
+  (`/.well-known/ucp`, plus `/products.json?limit=1` and the homepage if robots.txt allows them).
+- Money is integer minor units + ISO currency.
+- Graders are the single source of truth for quality: RL rewards (`train/rewards.py`), evals and
+  serving guards all call them.
+- Decisions use the System One contract (`decide/systemone.py`): hosted Jev labels data, and our
+  post-trained decider serves the same API. Bump a question set's version whenever a question changes.
+- Decider training exports leave out Jev-only labels unless TypeSafe's terms allow training on them.
+- Never train on files containing `c2a.bench.CANARY`.

README.md

@@ -1,2 +1,73 @@
 # crawl2action
-E2E pipeline to crawl and post-train LLM for vertical domain (e-commerce)
+
+End-to-end pipeline for e-commerce AI:
+
+**crawl** (Firecrawl, Shopify, UCP, open datasets) → **datasets** → **post-train** (Tinker; Modal fallback) → **benchmark** (public + C2A-Bench) → **serve** (Modal, behind vLLM Semantic Router).
+
+It post-trains models for pre- and post-checkout recommendations, product copy, localization, image briefs and calibrated decisions. See [`docs/PLAN.md`](docs/PLAN.md) for the full design.
+
+## Status: M0 (skeleton)
+Working and tested:
+- schemas (UCP-aligned)
+- store registry (default-deny)
+- crawl compliance (ToS + robots.txt + rate limit)
+- Shopify `products.json` parser
+- ranking metrics
+- recommendation graders
+- `decide` interface and calibration metrics
+- training data formats
+- RL rewards and advantage estimators (Dr.GRPO/GRPO, Rank-GRPO-style, DAPO dynamic sampling)
+- Tinker SFT/RL/OPD loops (tested with a fake backend)
+- the dev gateway
+
+Everything else is a stub that names the milestone it lands in.
+
+## Quickstart
+```bash
+uv sync --extra dev
+uv run pytest -q
+uv run c2a registry validate
+uv run c2a decide --evidence "bought running shoes; candidate: running socks" \
+  --labels exact,substitute,complement,irrelevant
+uv run c2a train sft --config configs/train/sft_student.yaml --dry-run
+```
+
+To run real post-training on Tinker, run `uv sync --extra train`, set `TINKER_API_KEY`, add a dataset at the path in the run config, then:
+```bash
+uv run c2a train sft --config configs/train/sft_student.yaml
+uv run c2a train rl  --config configs/train/rl_student_rank_grpo.yaml
+uv run c2a train opd --config configs/train/opd_reranker.yaml
+```
+
+## Crawling
+**First real run:** follow [`docs/PILOT.md`](docs/PILOT.md) (`c2a doctor` → `c2a pilot` → `c2a label review` → `c2a report`).
+
+Crawling is default-deny: a store is crawled only after a person has reviewed its terms and approved it.
+```bash
+uv run c2a registry approve allbirds --by "<your name>" --note "ToS reviewed"
+uv run c2a discover fingerprint --store allbirds --write   # ucp / shopify / firecrawl
+uv run c2a crawl --dry-run                                  # what runs, and why others are skipped
+export FIRECRAWL_API_KEY=... TYPESAFE_API_KEY=...
+uv run c2a crawl --store allbirds --max-products 20 --label
+uv run c2a discover expand --country SE --category apparel  # candidates for review
+uv run c2a data import esci --path examples.parquet --path products.parquet --out data/open/esci
+```
+Parquet imports need `uv sync --extra data`.
+
+## Labeling with Jev
+Scraped products are classified and labeled by hosted **Jev** (TypeSafe System One) through `c2a label`. Uncertain labels are escalated to the teacher and then to human review.
+```bash
+export TYPESAFE_API_KEY=...
+uv run c2a label run --input data/products.jsonl --qset product_v1 --max-requests 100
+uv run c2a label export --run-dir data/labels/product_v1 --kind gold --out data/gold/product.jsonl
+```
+Add `--dry-run` to preview a request without calling the API, or `--backend keyword` to run offline.
+
+## Layout
+- `src/c2a/sources`: registry, compliance, Shopify/UCP/Firecrawl ingesters
+- `src/c2a/graders`, `src/c2a/decide`: shared by evaluation, RL rewards and serving guards (`decide` uses the System One contract)
+- `src/c2a/labeling`: Jev question sets, confidence gating, escalation, cache, exports
+- `src/c2a/train`: data formats, rewards, `tinker/` loops, `fallback/` (Modal), `ac2/`
+- `src/c2a/bench`, `src/c2a/eval`: C2A-Bench and the benchmark harness
+- `src/c2a/serve`: gateway, vLLM and image Modal apps
+- `configs/`: defaults, training run configs, semantic-router draft

configs/default.yaml

@@ -0,0 +1,21 @@
+# crawl2action defaults. Override any key with env vars, e.g. C2A__MODELS__STUDENT=Qwen/Qwen3.5-27B
+paths:
+  data_dir: data
+  runs_dir: runs
+budgets:
+  firecrawl_credits_per_run: 5000
+  teacher_usd_per_run: 100
+  default_qps_per_domain: 1.0
+models:
+  student: Qwen/Qwen3.8-27B            # post-trained + served; confirm availability on Tinker (M3)
+  student_alt: thinkingmachines/Inkling-Small   # bake-off challenger
+  proxy_student: Qwen/Qwen3.5-4B       # fast iteration before scaling to 27B
+  teacher: kimi-k3                     # API-only data generator / rubric writer (alt: glm-5.3, qwen3.8-max)
+  decider: llm-semantic-router/Decision-1.0-Lux-9B   # starting checkpoint; we post-train it
+  image: Qwen/Qwen-Image-2.1
+  embedding: Qwen/Qwen3-Embedding-0.6B
+locales: [sv-SE, en-GB, es-ES, en-US, en-CA, fr-CA]
+labeler:                               # hosted Jev (TypeSafe System One); key: TYPESAFE_API_KEY
+  provider: typesafe
+  base_url: https://api.typesafe.ai
+  model: jev-latest

configs/labeling.yaml

@@ -0,0 +1,9 @@
+# Confidence gates for Jev labels. Tune against the human-verified gold set (M2).
+thresholds:
+  min_confidence: 0.6     # choice/score answer confidence
+  min_top_prob: 0.7       # choice/score top probability
+  noul_min: 0.85          # yes/no: max(p, 1-p)
+  per_question:
+    restricted: {noul_min: 0.95}      # compliance-relevant: escalate more
+    contains_pii: {noul_min: 0.95}
+    page_type: {min_top_prob: 0.8}    # gates Firecrawl /extract spend

configs/router/semantic-router.yaml

@@ -0,0 +1,24 @@
+# DRAFT vLLM Semantic Router policy for crawl2action (schema to be aligned with the
+# router version we deploy in M4; see https://github.com/vllm-project/semantic-router).
+model_pool:
+  - name: c2a-reranker      # OPD-distilled fast model (high-QPS /recommend)
+    endpoint: ${C2A_RERANKER_URL}
+  - name: c2a-student       # post-trained Qwen3.8-27B (+ task LoRAs)
+    endpoint: ${C2A_STUDENT_URL}
+  - name: c2a-decider       # post-trained decider (calibrated label probabilities)
+    endpoint: ${C2A_DECIDER_URL}
+  - name: c2a-image         # Qwen-Image-2.1 pipeline
+    endpoint: ${C2A_IMAGE_URL}
+signals:
+  - {name: task, type: keyword, source: path}          # /recommend, /generate/text, ...
+  - {name: language, type: language_detection}
+  - {name: complexity, type: classifier}
+  - {name: pii, type: pii_detection}
+  - {name: jailbreak, type: jailbreak_detection}
+decisions:
+  - {when: "jailbreak or pii.high", action: block}
+  - {when: "task == recommend and complexity == low", route: c2a-reranker}
+  - {when: "task == recommend", route: c2a-student, reasoning: auto}
+  - {when: "task == decide", route: c2a-decider}
+  - {when: "task == generate_image", route: c2a-image}
+  - {when: "true", route: c2a-student, reasoning: "complexity == high"}

configs/train/opd_reranker.yaml

@@ -0,0 +1,12 @@
+# On-policy distillation: RL'd student (teacher) -> fast reranker. Same tokenizer family required.
+name: opd-reranker
+stage: opd
+platform: tinker
+base_model: Qwen/Qwen3.5-4B
+teacher_model: Qwen/Qwen3.8-27B       # replace with the RL'd checkpoint path from Tinker
+dataset: data/train/opd_rec.jsonl
+kl_coef: 1.0
+group_size: 4
+batch_size: 16
+learning_rate: 5.0e-5
+steps: 50

configs/train/rl_student_rank_grpo.yaml

@@ -0,0 +1,14 @@
+# RL on recommendation prompts with Rank-GRPO-style advantages + DAPO dynamic sampling.
+name: rl-student-rec-proxy
+stage: rl
+platform: tinker
+base_model: Qwen/Qwen3.5-4B
+dataset: data/train/rl_rec.jsonl
+reward: rec
+advantage: rank_grpo
+drop_zero_variance_groups: true
+group_size: 8
+batch_size: 16
+learning_rate: 4.0e-5
+max_tokens: 512
+steps: 20

configs/train/sft_decider.yaml

@@ -0,0 +1,10 @@
+# Decider post-training step 1: SFT on ESCI + C2A gold labels (then RL with reward: decide).
+name: sft-decider
+stage: sft
+platform: tinker
+base_model: llm-semantic-router/Decision-1.0-Lux-9B   # if not trainable on Tinker, use a same-family Qwen base
+dataset: data/train/sft_decider.jsonl
+lora_rank: 32
+learning_rate: 1.0e-4
+batch_size: 64
+steps: 50

configs/train/sft_student.yaml

@@ -0,0 +1,11 @@
+# Cold-start SFT on grader-filtered teacher outputs. Start on the proxy model, then 27B.
+name: sft-student-proxy
+stage: sft
+platform: tinker
+base_model: Qwen/Qwen3.5-4B
+dataset: data/train/sft_student.jsonl
+lora_rank: 32
+learning_rate: 1.0e-4
+batch_size: 32
+steps: 20
+save_every: 10

docs/PILOT.md

@@ -0,0 +1,75 @@
+# Pilot runbook: scrape and label real stores
+
+The goal of this pilot is to crawl 3–5 Shopify stores and label their products and product pairs with Jev. It also produces the first human-reviewed gold labels. Everything runs on your machine. This takes about 30 minutes plus review time.
+
+## 1. Setup
+```bash
+git clone git@github.com:umaitech/crawl2action.git && cd crawl2action
+git checkout claude/magical-davinci-7u43rl
+uv sync --extra dev
+cp .env.example .env        # then fill in the keys below
+```
+`.env` needs:
+- `TYPESAFE_API_KEY`: required (Jev labeling).
+- `FIRECRAWL_API_KEY`: only needed if you add a non-Shopify store.
+
+The `.env` file is gitignored and is loaded automatically by every `c2a` command.
+
+## 2. Pick and approve stores
+Suggested pilot stores. They are believed to run on Shopify, which means no Firecrawl credits are needed for products; `doctor` and `fingerprint` will confirm this:
+
+| id | store | country |
+|---|---|---|
+| `allbirds` | Allbirds | US |
+| `kotn` | Kotn | CA |
+| `tentree` | Tentree | CA |
+| `djerfavenue` | Djerf Avenue | SE |
+| `gymshark` | Gymshark | UK |
+
+Review each store's terms of service and robots.txt yourself. Approve only stores whose terms allow automated collection of public product data for your purpose. Approval is recorded with your name:
+```bash
+uv run c2a registry approve allbirds --by "<your name>" --note "ToS reviewed <date>: ..."
+uv run c2a discover fingerprint --store allbirds --write   # sets platform, method, currency
+```
+Commit `src/c2a/sources/registry.yaml` afterwards, so the approval and its audit trail are shared.
+
+## 3. Preflight
+```bash
+uv run c2a doctor --live
+```
+Every line should show ✅. `--live` makes one real Jev call to confirm the key works and that our parser matches Jev's actual response format. If you see "Jev response format ❌", send the output to Claude: it includes the raw response body.
+
+## 4. Run the pilot
+```bash
+uv run c2a pilot --store allbirds --store kotn --store tentree --max-products 100 --max-jev-requests 400
+```
+The pilot runs these steps:
+1. It crawls each store politely: 1 request per second, following robots.txt and retrying on 429s.
+2. It labels new or changed products with `product_v1`. That is one Jev request per product.
+3. It builds candidate pairs inside each store (3 similar products plus 1 from another category, per anchor) and labels them with `pair_v1`.
+4. It writes `data/reports/pilot.md`.
+
+Re-running only pays for new or changed products and uncached requests.
+
+**Cost check:** Jev requests ≈ products + pairs. With 3 stores × 100 products and `--pairs-k 3`, that is about 300 + 1,200. The run stops at `--max-jev-requests`, so start small.
+
+## 5. Review uncertain labels
+```bash
+uv run c2a label review --run-dir data/labels/product_v1 --limit 30
+uv run c2a label review --run-dir data/labels/pair_v1 --limit 30
+```
+For each item, pick a number, `s` to skip or `q` to quit. Every answer is saved immediately.
+
+## 6. Look at the results
+```bash
+uv run c2a report
+uv run c2a label export --run-dir data/labels/product_v1 --kind gold --out data/gold/product_v1.jsonl
+uv run c2a label export --run-dir data/labels/pair_v1 --qset pair_v1 --kind gold --out data/gold/pair_v1.jsonl
+```
+
+## Send back to Claude
+- the output of `c2a doctor --live`;
+- `data/reports/pilot.md`;
+- any `data/raw/<store>/manifest.json` with errors.
+
+These say whether the real APIs and stores behave as the code expects, and how the confidence thresholds in `configs/labeling.yaml` should be tuned.