Review: Discogs mirror site (#34)

.claude/skills/clone-website/SKILL.md

@@ -204,7 +204,7 @@ Then run the site once locally to produce `instance/<site>.db`, copy it to
 ```bash
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41028:40000-40028 webharbor:dev
+  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
 
 # your new site is on port 41000 + its index
 curl -so /dev/null -w "%{http_code}\n" http://localhost:41000NN/
@@ -228,7 +228,7 @@ After Phase 1, you should have:
 - `sites/<your_site>/static/` with real CSS/JS/icons (and images under HF assets)
 - `sites/<your_site>/instance_seed/<site>.db` with seeded data
 - Site registered in `websyn_start.sh`, `control_server.py`, `Dockerfile`
-- All 29 sites still return 200 on the alt-port container
+- All 30 sites still return 200 on the alt-port container
 - Byte-identical reset passes
 
 ## Next step

.claude/skills/evolve-env/SKILL.md

@@ -65,15 +65,15 @@ through `BASE_DIR = os.path.dirname(os.path.abspath(__file__))`.
 After any DB-affecting change:
 
 ```bash
-# 1. Stop test container (don't touch user's working container on :40000-40028)
+# 1. Stop test container (don't touch user's working container on :40000-40029)
 docker stop wh-test 2>/dev/null || true
 
 # 2. Rebuild
 ./scripts/build.sh webharbor:dev
 
 # 3. Run on alt ports
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41028:40000-40028 webharbor:dev
+  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
 
 # 4. Reset your site and confirm byte-identity
 curl -X POST http://localhost:8201/reset/<your_site>
@@ -149,7 +149,7 @@ After Phase 3:
 - Every task in `tasks.jsonl` is hand-verified to work end-to-end
 - The mirror has no obvious leaks, broken forms, or empty pages
 - Byte-identical reset passes
-- All 29 sites still return 200
+- All 30 sites still return 200
 
 ## Next step
 

.claude/skills/harden-env/SKILL.md

@@ -130,7 +130,7 @@ Hardening changes seed data. After any DB change:
 ```bash
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41028:40000-40028 webharbor:dev
+  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
 curl -X POST http://localhost:8201/reset/<your_site>
 docker exec wh-test md5sum \
   /opt/WebSyn/<your_site>/instance/<your_site>.db \

.claude/skills/review-env/SKILL.md

@@ -33,18 +33,18 @@ gh pr checkout <pr-number>
 ./scripts/fetch_assets.sh             # pull the pinned HF revision
 ./scripts/build.sh webharbor:dev
 docker run -d --rm --name wh-review \
-  -p 8201:8101 -p 41000-41028:40000-40028 webharbor:dev
+  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
 ```
 
-Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 29 sites (40000-40028).
+Confirm the new/changed site is on the expected port (40000 + index). Note: the image now runs 30 sites (40000-40029).
 
 ### Step 2: The mechanical checks (5 minutes)
 
 Run the same Pre-PR checks the contributor was supposed to run.
 
 ```bash
-# 1. all 29 sites return 200
-for p in $(seq 41000 41028); do
+# 1. all 30 sites return 200
+for p in $(seq 41000 41029); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 
@@ -231,7 +231,7 @@ Leave a structured comment on the PR:
 ## Review: <site_name>
 
 ### Mechanical checks: PASS / FAIL
-- [x] All 29 sites return 200
+- [x] All 30 sites return 200
 - [x] Control plane healthy
 - [x] Byte-identical reset (md5 match)
 - [x] Parallel reset <10s

.claude/skills/seed-database/SKILL.md

@@ -169,10 +169,10 @@ gh pr create
 ./scripts/check_assets.sh                       # every site has instance_seed/
 ./scripts/build.sh webharbor:dev                # docker build succeeds
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41028:40000-40028 webharbor:dev
+  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
 
-# all 29 sites return 200
-for p in $(seq 41000 41028); do
+# all 30 sites return 200
+for p in $(seq 41000 41029); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

AGENTS.md

@@ -4,7 +4,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on
 
 ## What it is
 
-29 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
+30 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
 
 Two repos:
 - **code** (this one) — Flask apps, control plane, scripts.
@@ -48,17 +48,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
 # fresh clone
 ./scripts/fetch_assets.sh                     # pulls assets from HF
 ./scripts/build.sh                            # docker build -t webharbor:dev .
-docker run -d -p 8101:8101 -p 40000-40028:40000-40028 webharbor:dev
+docker run -d -p 8101:8101 -p 40000-40029:40000-40029 webharbor:dev
 ```
 
 Or use the published image directly:
 
 ```bash
-docker run -d -p 8101:8101 -p 40000-40028:40000-40028 \
+docker run -d -p 8101:8101 -p 40000-40029:40000-40029 \
   battalion7244/webharbor:latest
 ```
 
-Sites are on `40000`-`40028` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
+Sites are on `40000`-`40029` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
 
 | Method | Path                | Purpose                                   |
 |--------|---------------------|-------------------------------------------|
@@ -136,13 +136,13 @@ python3 -m py_compile sites/<site>/app.py
 
 # 3. run on alt ports (don't collide with anything you already have running)
 docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41028:40000-40028 webharbor:dev
+  -p 8201:8101 -p 41000-41029:40000-40029 webharbor:dev
 
 # 4. control plane healthy, all sites alive
 curl -s http://localhost:8201/health | python3 -m json.tool | head
 
 # 5. every site renders 200
-for p in $(seq 41000 41028); do
+for p in $(seq 41000 41029); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 

CLAUDE.md

@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl
 
 ## Existing containers
 
-If a container is already running on `:8101` / `:40000-40028`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41028`).
+If a container is already running on `:8101` / `:40000-40029`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41029`).

CONTRIBUTING.md

@@ -24,7 +24,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
 ./scripts/fetch_assets.sh                       # pull current assets
 ./scripts/new_site.py mywebsite                 # OR edit an existing site
 ./scripts/build.sh && docker run -d --rm \
-  -p 8101:8101 -p 40000-40028:40000-40028 webharbor:dev
+  -p 8101:8101 -p 40000-40029:40000-40029 webharbor:dev
 # iterate locally...
 
 ./scripts/extract_assets.sh ../webharbor-static-pr/   # split assets out

Dockerfile

@@ -92,6 +92,6 @@ os.makedirs('instance_seed', exist_ok=True); \
 shutil.copy2('instance/rotten_tomatoes.db', 'instance_seed/rotten_tomatoes.db'); \
 print('Rotten Tomatoes seed DB generated at build time.')" && rm -rf /opt/WebSyn/rotten_tomatoes/instance
 
-EXPOSE 8101 40000-40028
+EXPOSE 8101 40000-40029
 
 CMD ["/opt/websyn_start.sh"]

README.md

@@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
 - **Deep features unlocked** — carts, checkouts, accounts, all fully testable
 - **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
 - **RL-ready** — sub-second database resets between rollouts
-- **Community-driven** — 29 sites today, scaling to 100+ together
+- **Community-driven** — 30 sites today, scaling to 100+ together
 
 ## 🚀 Quickstart
 
 One command to run all web environments:
 
 ```bash
-docker run -p 8101:8101 -p 40000-40028:40000-40028 battalion7244/webharbor:latest
+docker run -p 8101:8101 -p 40000-40029:40000-40029 battalion7244/webharbor:latest
 ```
 
-Then point your agent at `http://localhost:40000` through `http://localhost:40028` to explore 29 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, and NVIDIA`.
+Then point your agent at `http://localhost:40000` through `http://localhost:40029` to explore 30 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, Healthline, Kaggle, NVIDIA, and Discogs`.
 
 For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:
 
@@ -63,27 +63,28 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor
 ./scripts/build.sh                                 # docker build -t webharbor:dev .
 ```
 
-### Local NVIDIA review candidate
+### Local NVIDIA and Discogs review candidates
 
-This branch registers **29 sites**, the 29 entries listed above; NVIDIA is the last
-entry, registry index 28, container port 40028 (local review host port 48028). The
-published-image quickstart above is not a claim that this review candidate has been
-published or accepted.
+This branch registers **30 sites**, the 30 entries listed above. NVIDIA remains at
+registry index 28 and Discogs is appended at index 29. The published-image
+quickstart above is not a claim that either review candidate has been published or
+accepted.
 
 | Site | Registry position | Container port | Local review host port |
 | --- | --- | --- | --- |
 | NVIDIA | 28 | 40028 | 48028 |
+| Discogs | 29 | 40029 | 48029 |
 
 `websyn_start.sh`, `control_server.py`, the `Dockerfile` `EXPOSE` line and every
-site's `tasks.jsonl` `web` URL agree on 29 sites and `40000-40028`;
+site's `tasks.jsonl` `web` URL agree on 30 sites and `40000-40029`;
 `scripts/check_site_registry.py` (run by `scripts/check_assets.sh`) fails when they
 drift.
 
 After preparing the candidate assets and building `webharbor:dev`, the local
 review deployment uses:
 
 ```bash
-docker run -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48028:40000-40028 webharbor:dev
+docker run -p 127.0.0.1:48080:8101 -p 127.0.0.1:48000-48029:40000-40029 webharbor:dev
 ```
 
 NVIDIA inherits the site contribution from @KaKituken
@@ -105,14 +106,15 @@ added the first reviewed NVIDIA bundle. Those PRs are merged, the pinned commit
 on `main`, and the earlier "PR not merged" and "archive rejected by the
 validator" blockers are cleared:
 
-- the pinned revision carries 31 `*.tar.gz` (one per registered site plus
-  `bandcamp.tar.gz` and `drugs_com.tar.gz`, which `fetch_assets.sh` ignores for
-  sites this checkout does not register);
+- the pinned revision carries 31 `*.tar.gz` (one per upstream NVIDIA candidate
+  site plus `bandcamp.tar.gz` and `drugs_com.tar.gz`); Discogs remains in open HF
+  PR #76 and therefore requires its immutable revision override until that PR is
+  merged;
 - `nvidia.tar.gz` at that revision has 37 file members and no directory members,
   so `scripts/validate_asset_archive.py nvidia.tar.gz nvidia` prints
   `[fetch] validated 37 managed members for nvidia` and exits 0;
-- `./scripts/fetch_assets.sh` at this pin extracts all 29 registered sites
-  (`[fetch] done — 29 site(s) extracted into sites/`).
+- `./scripts/fetch_assets.sh` at this pin extracts the 29 upstream sites. It does
+  not yet prepare Discogs from a clean checkout.
 
 | Artifact | Members | Bytes | SHA-256 |
 | --- | --- | --- | --- |
@@ -204,7 +206,7 @@ itself cannot be edited from this repository.
 
 ## 🤝 Contribute
 
-We have built 29 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
+We have built 30 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
 
 There are two ways to join the author list:
 

agent_demo/README.md

@@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1   # or your Azure / vLLM endpoi
 
 ## Run a task
 
-WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40028:40000-40028 battalion7244/webharbor:latest`).
+WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40029:40000-40029 battalion7244/webharbor:latest`).
 
 Run a single task from a site's `tasks.jsonl`:
 

control_server.py

@@ -29,8 +29,8 @@
     'cambridge_dictionary', 'coursera', 'espn', 'merriam_webster',
     'ikea', 'phys_org', 'target', 'ted',
     'osu', 'rotten_tomatoes', 'compass', 'walmart_careers',
-    'fedex', 'webmd_doctor', 'healthline', 'kaggle',
-    'nvidia',
+    'fedex', 'webmd_doctor', 'healthline', 'kaggle', 'nvidia',
+    'discogs',
 ]
 BASE_PORT = 40000
 WEBSYN_DIR = '/opt/WebSyn'

sites/discogs/_health.py

@@ -0,0 +1,3 @@
+"""Per-site health probe (optional, called by control_server)."""
+def health():
+    return {"ok": True, "site": "discogs"}