Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,17 @@ Inside the image, sites live at `/opt/WebSyn/<site>/`. The path predates the ren
# fresh clone
./scripts/fetch_assets.sh # pulls assets from HF
./scripts/build.sh # docker build -t webharbor:dev .
docker run -d -p 8101:8101 -p 40000-40025:40000-40025 webharbor:dev
docker run -d -p 8101:8101 -p 40000-40026:40000-40026 webharbor:dev
```

Or use the published image directly:

```bash
docker run -d -p 8101:8101 -p 40000-40025:40000-40025 \
docker run -d -p 8101:8101 -p 40000-40026:40000-40026 \
battalion7244/webharbor:latest
```

Sites are on `40000`-`40025` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:
Sites are on `40000`-`40026` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane:

| Method | Path | Purpose |
|--------|---------------------|-------------------------------------------|
Expand Down Expand Up @@ -136,13 +136,13 @@ python3 -m py_compile sites/<site>/app.py

# 3. run on alt ports (don't collide with anything you already have running)
docker run -d --rm --name wh-test \
-p 8201:8101 -p 41000-41025:40000-40025 webharbor:dev
-p 8201:8101 -p 41000-41026:40000-40026 webharbor:dev

# 4. control plane healthy, all sites alive
curl -s http://localhost:8201/health | python3 -m json.tool | head

# 5. every site renders 200
for p in $(seq 41000 41025); do
for p in $(seq 41000 41026); do
curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/
done

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl

## Existing containers

If a container is already running on `:8101` / `:40000-40025`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41025`).
If a container is already running on `:8101` / `:40000-40026`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41026`).
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ git clone https://github.com/<you>/webharbor && cd webharbor
./scripts/fetch_assets.sh # pull current assets
./scripts/new_site.py mywebsite # OR edit an existing site
./scripts/build.sh && docker run -d --rm \
-p 8101:8101 -p 40000-40025:40000-40025 webharbor:dev
-p 8101:8101 -p 40000-40026:40000-40026 webharbor:dev
# iterate locally...

./scripts/extract_assets.sh ../webharbor-static-pr/ # split assets out
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# WebHarbor — slim, self-contained image.
# 26 Flask mirror sites + control plane on :8101.
# 27 Flask mirror sites + control plane on :8101.

FROM python:3.12-slim-bookworm

Expand Down Expand Up @@ -86,6 +86,6 @@ os.makedirs('instance_seed', exist_ok=True); \
shutil.copy2('instance/rotten_tomatoes.db', 'instance_seed/rotten_tomatoes.db'); \
print('Rotten Tomatoes seed DB generated at build time.')" && rm -rf /opt/WebSyn/rotten_tomatoes/instance

EXPOSE 8101 40000-40025
EXPOSE 8101 40000-40026

CMD ["/opt/websyn_start.sh"]
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
- **Deep features unlocked** — carts, checkouts, accounts, all fully testable
- **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
- **RL-ready** — sub-second database resets between rollouts
- **Community-driven** — 26 sites today, scaling to 100+ together
- **Community-driven** — 27 sites today, scaling to 100+ together

## 🚀 Quickstart

One command to run all web environments:

```bash
docker run -p 8101:8101 -p 40000-40025:40000-40025 battalion7244/webharbor:latest
docker run -p 8101:8101 -p 40000-40026:40000-40026 battalion7244/webharbor:latest
```

Then point your agent at `http://localhost:40000` through `http://localhost:40025` to explore 26 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, and WebMD Doctor`.
Then point your agent at `http://localhost:40000` through `http://localhost:40025` to explore 27 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, and 9GAG`.

For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:

Expand Down
2 changes: 1 addition & 1 deletion agent_demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1 # or your Azure / vLLM endpoi

## Run a task

WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40025:40000-40025 battalion7244/webharbor:latest`).
WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40026:40000-40026 battalion7244/webharbor:latest`).

Run a single task from a site's `tasks.jsonl`:

Expand Down
1 change: 1 addition & 0 deletions control_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target',
'ted', 'osu', 'rotten_tomatoes', 'compass', 'walmart_careers', 'fedex',
'webmd_doctor',
'9gag',
]
BASE_PORT = 40000
WEBSYN_DIR = '/opt/WebSyn'
Expand Down
62 changes: 62 additions & 0 deletions review-reports/9GAG-FINAL-AUDIT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# 9GAG contributor-side final audit

## Scope

- Mirror: https://9gag.com/
- Local site: `http://localhost:40024/` (host verification port `41024`)
- Branch: `add-9gag-mirror`
- Base: upstream `main` at `3600493`
- Docker image: `sha256:0cf5649d5cb058fa05d1887a3559f78b327e84e5a78320f2aa6f795a2ab31dc6`

## Results

- All 25 registered WebHarbor sites started successfully.
- `/reset-all` returned every site as ready.
- All 20 9GAG tasks passed from the homepage through visible Playwright controls, with a fresh browser context and site reset for each task.
- 27 representative responsive checks passed across 1440, 390, and 320 px.
- No broken images, stretched images, horizontal document overflow, or out-of-bounds controls were detected.
- Five hardened search families returned 9–10 visible results; targets appeared at positions 2, 2, 4, 5, and 7.
- None of the task-specific detail facts appeared on search-result cards.
- Runtime and seed database MD5 after the final task run and `/reset-all`: `44768940829bf4a8bec07d7bdf072d97`.

## Seed inventory

- Posts: 80
- Users: 4
- Comments: 4
- Saved-post rows: 16
- Vote rows: 12
- Genuine captured post images: 38 unique files
- Additional captured 9GAG brand/interest references: 27 files

All four benchmark accounts use `@test.com` addresses and password `TestPass123!`. Both `seed_database()` and `seed_benchmark_users()` gate the entire function; repeated application left database bytes unchanged.

## Asset package prepared for Hugging Face

- Archive: `9gag.tar.gz`
- Managed members: 74
- SHA-256: `6ba9ca464d23bcae76316efff234548f75e428a7570f1d08f004c38a1123841c`
- The archive was validated locally and was not uploaded.

## Screenshots

### Desktop — 1440 px

![9GAG desktop homepage](assets/9gag-homepage-1440.png)

### Mobile — 390 px

![9GAG mobile homepage](assets/9gag-homepage-390.png)

### Narrow mobile — 320 px

![9GAG narrow mobile homepage](assets/9gag-homepage-320.png)

## Human integration steps

1. Upload the prepared `9gag.tar.gz` to the contributor's Hugging Face dataset fork and open an assets PR against `ChilleD/WebHarbor`.
2. After the HF PR merges, update `.assets-revision` to the merged asset commit.
3. Rebuild and rerun the 25-site health, reset-all, task, and MD5 checks.
4. Push `add-9gag-mirror` and open the GitHub PR, linking the merged HF asset commit and this report.

No Hugging Face or GitHub PR was opened during preparation.
Binary file added review-reports/assets/9gag-homepage-1440.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added review-reports/assets/9gag-homepage-320.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added review-reports/assets/9gag-homepage-390.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
24 changes: 24 additions & 0 deletions sites/9gag/ASSET_SOURCES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# 9GAG asset provenance

All media in `static/images/` was captured from the rendered public 9GAG homepage on 2026-09-09 with Playwright. The browser exposed the resolved CDN URLs after hydration; only then were the image bytes downloaded. No placeholders, generated images, private data, cookies, or browser-profile material are included.

## Post media

- `posts/aD2BwPZ-451efa65.jpg` — https://img-9gag-fun.9cache.com/photo/aD2BwPZ_460s.jpg
- `posts/aVvGONd-41d662a3.jpg` — https://miscmedia-9gag-fun.9cache.com/images/thumbnail-facebook/71134346_1788942069.8925_rYNuRA_700b.jpg
- `posts/aO86yA2-87d68ca6.jpg` — https://img-9gag-fun.9cache.com/photo/aO86yA2_460s.jpg
- `posts/aryPpR7-3497fabc.jpg` — https://img-9gag-fun.9cache.com/photo/aryPpR7_460s.jpg
- `posts/amoDNzv-6cbe8b39.jpg` — https://img-9gag-fun.9cache.com/photo/amoDNzv_700b.jpg
- `posts/aQzYqZq-9c4605da.jpg` — https://img-9gag-fun.9cache.com/photo/aQzYqZq_460s.jpg
- `posts/aLnqKLv-bea354c8.jpg` — https://img-9gag-fun.9cache.com/photo/aLnqKLv_700b.jpg
- `posts/adB3Nj9-f56e7b44.jpg` — https://img-9gag-fun.9cache.com/photo/adB3Nj9_460s.jpg
- `posts/aYQzNgV-f6067981.jpg` — https://img-9gag-fun.9cache.com/photo/aYQzNgV_460s.jpg
- `posts/aQzYqK8-ec9e7408.jpg` — https://img-9gag-fun.9cache.com/photo/aQzYqK8_460s.jpg

## Brand and interest references

The files under `static/images/reference/` are the 9GAG logo and interest thumbnails resolved from `miscmedia-9gag-fun.9cache.com` by the same rendered page. Their original URLs and dimensions are retained in the ignored reconnaissance record `scraped_data/capture.json`.

An additional 28 unique feed images under `static/images/posts/` were captured through Chromium from the public `/v1/group-posts/group/default/type/{hot,trending,fresh}?c=10` responses. Their source post IDs remain in every filename, and the resolved source URLs are retained in the ignored reconnaissance record `scraped_data/feed-posts.json`.

The 80-row seeded catalog rotates across 38 genuine 9GAG post images while keeping titles, descriptions, authors, tags, and task facts distinct. This avoids blank, generated, or placeholder imagery and limits repetition across feed pages.
3 changes: 3 additions & 0 deletions sites/9gag/_health.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
"""Per-site health probe (optional, called by control_server)."""
def health():
return {"ok": True, "site": "9gag"}
Loading