aiming-lab · Django-Jiang · May 13, 2026 · May 13, 2026 · May 13, 2026 · May 13, 2026
diff --git a/.assets-revision b/.assets-revision
@@ -2,7 +2,7 @@
 # (instance_seed/*.db, static/images/, static/external_cache/).
 #
 # fetch_assets.sh uses the `hf download` CLI. The pin below
-# is a git revision (branch name like `main`, a tag, or a specific commit
-# sha). Override at runtime with the ASSETS_REVISION env var.
+# is an immutable commit SHA. ASSETS_REVISION may assert this exact value;
+# fetch_assets.sh rejects every differing override to prevent mislabeled trees.
 repo: ChilleD/WebHarbor
-revision: 65c479f894763f64c6073e0d180ebf542d1d2c02
+revision: 18e64e4d230794f990199f3327432d26db36866f
diff --git a/.dockerignore b/.dockerignore
@@ -1,5 +1,7 @@
 # HF download metadata produced by `hf download` — not part of the image.
 sites/.cache/
+sites/.assets-state.json
+sites/**/*.asset-backup
 **/.cache/huggingface/
 
 # Don't ship — websyn_start.sh recreates instance/ from instance_seed/ at boot.
@@ -12,8 +14,10 @@ sites/*/scraped_data/
 sites/*/scripts_dev/
 sites/*/tests/
 sites/*/verify/tests/
+sites/*/verify/test*.py
 agent_demo/runs/
 .venv/
+**/.venv/
 
 # Don't ship — bytecode / venvs.
 sites/*/__pycache__/
@@ -26,6 +30,13 @@ sites/*/*.pyc
 # extracted content. Shipping both doubles the COPY layer.
 sites/*.tar.gz
 
+# Never admit Git-ignored local configuration or secret-key material into a layer.
+**/.env
+**/.env.*
+**/secrets.json
+**/*.pem
+**/*.key
+
 # Build / CI workspace not part of the image.
 build/
 .assets/

diff --git a/.gitignore b/.gitignore
@@ -18,6 +18,7 @@ sites/*/venv/
 
 # HF download metadata produced by `hf download`.
 sites/.cache/
+sites/.assets-state.json
 **/.cache/huggingface/
 
 # HF dataset auto-artifacts that arrive together with the assets but belong

diff --git a/AGENTS.md b/AGENTS.md
@@ -4,7 +4,7 @@ A coding agent (Claude Code, Cursor, Aider, Codex, ...) is reading this. Read on
 
 ## What it is
 
-23 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
+25 Flask mirror websites (Amazon, GitHub, BBC News, ...) packaged into one Docker image, plus a control plane on `:8101` for resetting per-site state. Used as a deterministic offline environment for web-agent benchmarks. ~3 GB image.
 
 Two repos:
 - **code** (this one) — Flask apps, control plane, scripts.
@@ -27,14 +27,16 @@ sites/<site>/
 ├── static/{css,js,icons}/       small UI, in git
 ├── static/images/               heavy, in HF dataset
 ├── static/external_cache/       optional, in HF dataset
-└── instance_seed/<site>.db      seed DB, in HF dataset
+└── instance_seed/<database>.db  exactly one valid SQLite seed (new sites should use <site>.db)
 
 control_server.py                :8101 control plane
 site_runner.py                   per-site supervisor (setsid + killpg)
 websyn_start.sh                  container entrypoint
 Dockerfile
+requirements.lock                exact Docker dependency versions and wheel SHA-256 values
 .assetpaths                      paths managed via HF
 .assets-revision                 pins HF dataset repo + revision
+assets-manifest.json              tracked SHA-256/size map for all HF archives plus extracted-tree digest
 scripts/{fetch,extract,check}_assets.sh
 scripts/build.sh
 scripts/new_site.py
@@ -48,21 +50,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
 # fresh clone
 ./scripts/fetch_assets.sh                     # pulls assets from HF
 ./scripts/build.sh                            # docker build -t webharbor:dev .
-docker run -d -p 8101:8101 -p 40000-40023:40000-40023 webharbor:dev
+export WEBSYN_CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"
+docker run -d -e WEBSYN_CONTROL_TOKEN -p 8101:8101 -p 40000-40024:40000-40024 webharbor:dev
 ```
 
-Or use the published image directly:
+A published image may lag the repository. Do not treat it as evidence for the checked-out source revision; build and test the local image when reviewing a PR.
 
-```bash
-docker run -d -p 8101:8101 -p 40000-40023:40000-40023 \
-  battalion7244/webharbor:latest
-```
-
-Sites are on `40000`-`40023` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
+Sites are on `40000`-`40024` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
 
 | Method | Path                | Purpose                                   |
 |--------|---------------------|-------------------------------------------|
-| GET    | `/health`           | per-site PID + alive                      |
+| GET    | `/health`           | authenticated per-site PID + alive        |
 | POST   | `/reset/<site>`     | wipe `instance/`, restore from seed, respawn |
 | POST   | `/reset-all`        | parallel reset of every site              |
 | POST   | `/restart/<site>`   | respawn process; **DB not reset**         |
@@ -135,23 +133,25 @@ python3 -m py_compile sites/<site>/app.py
 ./scripts/build.sh webharbor:dev
 
 # 3. run on alt ports (don't collide with anything you already have running)
-docker run -d --rm --name wh-test \
-  -p 8201:8101 -p 41000-41023:40000-40023 webharbor:dev
+export WEBSYN_CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"
+docker run -d --rm --name wh-test -e WEBSYN_CONTROL_TOKEN \
+  -p 8201:8101 -p 41000-41024:40000-40024 webharbor:dev
 
-# 4. control plane healthy, all sites alive
-curl -s http://localhost:8201/health | python3 -m json.tool | head
+# 4. authenticated control plane healthy, all sites alive
+curl -s -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" http://localhost:8201/health | python3 -m json.tool | head
 
 # 5. every site renders 200
-for p in $(seq 41000 41023); do
+for p in $(seq 41000 41024); do
   curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
 done
 
 # 6. byte-identical reset (the strict invariant)
-curl -X POST http://localhost:8201/reset/<your_site>
-docker exec wh-test md5sum \
-  /opt/WebSyn/<your_site>/instance/<your_site>.db \
-  /opt/WebSyn/<your_site>/instance_seed/<your_site>.db
-# the two md5s MUST match — if not, see "Idempotent seeding"
+curl -X POST -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" http://localhost:8201/reset/<your_site>
+DB_NAME=$(docker exec wh-test sh -ec 'set -- /opt/WebSyn/<your_site>/instance_seed/*.db; test "$#" -eq 1; basename "$1"')
+docker exec wh-test sha256sum \
+  "/opt/WebSyn/<your_site>/instance/$DB_NAME" \
+  "/opt/WebSyn/<your_site>/instance_seed/$DB_NAME"
+# the two SHA-256 values MUST match — if not, see "Idempotent seeding"
 
 # 7. teardown
 docker stop wh-test
@@ -209,7 +209,7 @@ Per-row gates aren't enough — even a no-op `db.session.commit()` bumps SQLite
 
 ### Runtime data lives in `instance_seed/*.db`, not JSON
 
-HTTP handlers must read from SQLAlchemy, not from `scraped_data/*.json`. If you have intermediate scrape JSON, fold it into `instance_seed/<site>.db` at build time via `seed_data.py`. The `scraped_data/` dir is gitignored + dockerignored — never shipped.
+HTTP handlers must read from SQLAlchemy, not from `scraped_data/*.json`. If you have intermediate scrape JSON, fold it into the site's single `instance_seed/*.db` at build time via `seed_data.py`. The `scraped_data/` dir is gitignored + dockerignored — never shipped.
 
 ### Sites are isolated
 

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl
 
 ## Existing containers
 
-If a container is already running on `:8101` / `:40000-40023`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41023`).
+If a container is already running on `:8101` / `:40000-40024`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41024`).
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -19,20 +19,24 @@ Workflows A and B below are the **contributor's** job. The **Reviewer role** sec
 ## TL;DR
 
 ```bash
-# fork github.com/webharbor/webharbor + huggingface.co/datasets/ChilleD/WebHarbor
+# fork github.com/aiming-lab/WebHarbor + huggingface.co/datasets/ChilleD/WebHarbor
 git clone https://github.com/<you>/webharbor && cd webharbor
 ./scripts/fetch_assets.sh                       # pull current assets
 ./scripts/new_site.py mywebsite                 # OR edit an existing site
-./scripts/build.sh && docker run -d --rm \
-  -p 8101:8101 -p 40000-40023:40000-40023 webharbor:dev
+./scripts/build.sh
+export WEBSYN_CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"
+docker run -d --rm -e WEBSYN_CONTROL_TOKEN \
+  -p 8101:8101 -p 40000-40024:40000-40024 webharbor:dev
 # iterate locally...
 
 ./scripts/extract_assets.sh ../webharbor-static-pr/   # split assets out
 cd ../webharbor-static-pr
 hf upload-large-folder <your-fork>/WebHarbor . --repo-type dataset
 # open PR on HF first → grab the merge sha
 cd ../webharbor
-echo "revision: <hf-merge-sha>" > .assets-revision
+sed -i "s/^revision:.*/revision: <hf-merge-sha>/" .assets-revision
+./scripts/fetch_assets.sh --refresh-manifest  # full fetch; rewrites tracked archive/tree digests
+git add .assets-revision assets-manifest.json
 git commit -am "feat(mywebsite): add site + bump assets to <sha>"
 gh pr create
 ```
@@ -64,7 +68,7 @@ mywebsite/
 ├── requirements.txt    ← only Flask by default
 ├── templates/index.html
 ├── static/{css,js,icons,images,external_cache}/
-├── instance_seed/      ← drop your seed DB here as <name>.db
+├── instance_seed/      ← exactly one valid SQLite DB; new sites should use <name>.db
 ├── instance/           ← gitignored, recreated at boot
 └── scraped_data/       ← gitignored, build-time only
 ```
@@ -77,8 +81,8 @@ A typical seed flow:
 
 1. Define SQLAlchemy models in `app.py` (User, Product, Article, ...)
 2. Write a `seed_data.py` that materializes a dataset into the DB. Make the function **idempotent** — `if Foo.query.count() > 0: return` at the top.
-3. Run once locally to produce `instance/<name>.db`.
-4. Copy it to `instance_seed/<name>.db`. **This is your seed.**
+3. Run once locally to produce the application-defined database filename (new sites should use `instance/<name>.db`).
+4. Copy that single database to `instance_seed/` with the same filename. **This is your seed.**
 
 ### 4. Functional checklist
 
@@ -94,18 +98,20 @@ If your site has multiple categories / pages / topics, make sure the seed DB has
 
 ```bash
 ./scripts/build.sh
-docker run -d --rm --name wh-test \
+export WEBSYN_CONTROL_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"
+docker run -d --rm --name wh-test -e WEBSYN_CONTROL_TOKEN \
   -p 8101:8101 -p 40000-400NN:40000-400NN webharbor:dev
 
 # the new site should be on port 40000+i
 curl -so /dev/null -w "%{http_code}\n" http://localhost:400NN/
-curl -X POST http://localhost:8101/reset/mywebsite
-
-# make sure /reset/mywebsite keeps the DB byte-identical to the seed
-docker exec wh-test md5sum \
-  /opt/WebSyn/mywebsite/instance/<name>.db \
-  /opt/WebSyn/mywebsite/instance_seed/<name>.db
-# both md5s MUST match — see "Idempotent seeding" below
+curl -X POST -H "Authorization: Bearer $WEBSYN_CONTROL_TOKEN" http://localhost:8101/reset/mywebsite
+
+# make sure /reset/mywebsite keeps the one application-defined DB byte-identical to the seed
+DB_NAME=$(docker exec wh-test sh -ec 'set -- /opt/WebSyn/mywebsite/instance_seed/*.db; test "$#" -eq 1; basename "$1"')
+docker exec wh-test sha256sum \
+  "/opt/WebSyn/mywebsite/instance/$DB_NAME" \
+  "/opt/WebSyn/mywebsite/instance_seed/$DB_NAME"
+# both SHA-256 values MUST match — see "Idempotent seeding" below
 ```
 
 ### 6. Write the tasks (`tasks.jsonl`)
@@ -140,9 +146,10 @@ hf upload mywebsite.tar.gz <your-fork>/WebHarbor mywebsite.tar.gz --repo-type da
 # After it's merged, copy the merge commit sha.
 
 cd ../webharbor
-# bump the pin
+# bump the pin and bind every downloaded archive plus the extracted tree
 sed -i "s/^revision:.*/revision: <hf-merge-sha>/" .assets-revision
-git add .
+./scripts/fetch_assets.sh --refresh-manifest
+git add .assets-revision assets-manifest.json .
 git commit -m "feat(mywebsite): add new site
 
 Adds Flask app, templates, and seed DB for <real-site-name>.
@@ -172,7 +179,7 @@ hf upload amazon.tar.gz <your-fork>/WebHarbor amazon.tar.gz --repo-type dataset
 # (single-file upload keeps the PR scoped to one site)
 ```
 
-Open the HF PR; once merged, bump `.assets-revision` in this repo and open the GitHub PR. CI on the GitHub PR will fail-closed if the pinned revision isn't reachable.
+Open the HF PR; once merged, bump `.assets-revision`, run `./scripts/fetch_assets.sh --refresh-manifest`, commit `assets-manifest.json`, and open the GitHub PR. CI on the GitHub PR will fail-closed if the pinned revision isn't reachable.
 
 ## Reviewer role — validate the site and grade the tasks
 
@@ -182,7 +189,7 @@ The reviewer picks up a contributor's PR (site + `tasks.jsonl` with the basic ke
 
 Build the image from the branch and run it on alt ports (see AGENTS.md "Pre-PR checks" for the exact commands). Then:
 
-1. **Mechanical** — every site returns 200; `/health` all alive; `POST /reset/<site>` wipes runtime writes and restores the DB **byte-identical** to the seed (`md5(instance) == md5(instance_seed)`); `reset-all` completes in ~1s.
+1. **Mechanical** — every site returns 200; `/health` all alive; `POST /reset/<site>` wipes runtime writes and restores the DB **byte-identical** to the seed (`sha256(instance) == sha256(instance_seed)`); verify reset-all completion from its structured per-site response.
 2. **Functional** — drive the site's routes (auth, search, list/detail, any stateful action) and confirm each renders correct, non-empty content. The contributor's tasks must be genuinely completable on these pages.
 3. **Task feasibility** — for each task in `tasks.jsonl`, confirm it is **solvable by navigating the site** and is **not trivially answerable from an LLM's prior knowledge**. Drive a few tasks end-to-end (manually or with `agent_demo/agent.py`). Reject — and send back to the contributor — tasks that:
    - can be answered without ever opening the site (e.g. a common dictionary definition),
@@ -258,10 +265,10 @@ Per-row gates are not enough: the bare act of opening a SQLAlchemy session and c
 If you have *multiple* seed phases (`seed_database`, `seed_benchmark_users`, `seed_extras`), gate **each** of them. After a fresh seed, re-running the boot path should be a no-op. Test with:
 
 ```bash
-docker exec wh-test md5sum /opt/WebSyn/<site>/instance{,_seed}/<site>.db
+docker exec wh-test sh -ec 'set -- /opt/WebSyn/<site>/instance_seed/*.db; test "$#" -eq 1; db=$(basename "$1"); sha256sum "/opt/WebSyn/<site>/instance/$db" "/opt/WebSyn/<site>/instance_seed/$db"'
 # must match
 docker restart wh-test && sleep 5
-docker exec wh-test md5sum /opt/WebSyn/<site>/instance{,_seed}/<site>.db
+docker exec wh-test sh -ec 'set -- /opt/WebSyn/<site>/instance_seed/*.db; test "$#" -eq 1; db=$(basename "$1"); sha256sum "/opt/WebSyn/<site>/instance/$db" "/opt/WebSyn/<site>/instance_seed/$db"'
 # must STILL match
 ```
 

diff --git a/Dockerfile b/Dockerfile
@@ -1,34 +1,28 @@
 # WebHarbor — slim, self-contained image.
-# 24 Flask mirror sites + control plane on :8101.
+# 25 Flask mirror sites + control plane on :8101.
 
-FROM python:3.12-slim-bookworm
+FROM python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254
 
 ENV PYTHONUNBUFFERED=1 \
     PIP_DISABLE_PIP_VERSION_CHECK=1 \
     PIP_NO_CACHE_DIR=1 \
     LANG=C.UTF-8
 
-RUN pip3 install --no-cache-dir \
-    Flask==3.1.0 \
-    Flask-SQLAlchemy==3.1.1 \
-    Flask-Login==0.6.3 \
-    Flask-WTF==1.2.2 \
-    Flask-Bcrypt==1.0.1 \
-    bcrypt==5.0.0 \
-    Werkzeug==3.1.3 \
-    Jinja2==3.1.4 \
-    SQLAlchemy==2.0.36 \
-    WTForms==3.2.1 \
-    email-validator==2.2.0 \
-    Pillow==11.0.0
+COPY requirements.lock /opt/requirements.lock
+RUN pip3 install --no-cache-dir --require-hashes -r /opt/requirements.lock
 
 WORKDIR /opt/WebSyn
 
 # Sites tree. Build context must contain the heavy assets (instance_seed/,
 # static/images/, static/external_cache/) — either commit them locally or
 # run scripts/fetch_assets.sh to pull them from Hugging Face first.
 COPY sites/ /opt/WebSyn/
+COPY .assets-revision /opt/.assets-revision
+COPY assets-manifest.json /opt/assets-manifest.json
 COPY scripts/check_asset_inventory.py /opt/check_asset_inventory.py
+COPY scripts/check_seed_databases.py /opt/check_seed_databases.py
+COPY scripts/asset_state.py /opt/asset_state.py
+RUN python3 /opt/asset_state.py verify /opt/WebSyn /opt/.assets-revision /opt/assets-manifest.json
 
 # IKEA's seed is reproducibly materialized from the tracked source catalog so code-only content fixes do not require an asset-repository write. Product images still come from the pinned asset bundle.
 RUN cd /opt/WebSyn/ikea && PYTHONHASHSEED=0 python seed_data.py && rm -rf instance
@@ -50,6 +44,11 @@ RUN python3 /opt/check_asset_inventory.py /opt/WebSyn/walmart_careers && \
 RUN cd /opt/WebSyn/walmart_careers && rm -rf instance instance_seed && \
     PYTHONHASHSEED=0 python seed_data.py && rm -rf instance
 
+# Drugs.com uses database-backed inline pill renderings and rebuilds its versioned deterministic SQLite seed from tracked source data without network access.
+RUN python3 /opt/check_asset_inventory.py /opt/WebSyn/drugs_com
+RUN cd /opt/WebSyn/drugs_com && rm -rf instance instance_seed && \
+    PYTHONHASHSEED=0 python seed_data.py && rm -rf instance
+
 COPY websyn_start.sh    /opt/websyn_start.sh
 COPY control_server.py  /opt/control_server.py
 COPY site_runner.py     /opt/site_runner.py
@@ -72,6 +71,9 @@ os.makedirs('instance_seed', exist_ok=True); \
 shutil.copy2('instance/rotten_tomatoes.db', 'instance_seed/rotten_tomatoes.db'); \
 print('Rotten Tomatoes seed DB generated at build time.')" && rm -rf /opt/WebSyn/rotten_tomatoes/instance
 
-EXPOSE 8101 40000-40023
+# Final fail-closed package gate after every tracked seed generator and migration.
+RUN python3 /opt/check_seed_databases.py /opt/WebSyn
+
+EXPOSE 8101 40000-40024
 
 CMD ["/opt/websyn_start.sh"]
Original file line number	Diff line number	Diff line change
Expand Up		@@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl

		## Existing containers

		If a container is already running on `:8101` / `:40000-40023`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41023`).
		If a container is already running on `:8101` / `:40000-40024`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41024`).