Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# WebHarbor — slim, self-contained image.
# 24 Flask mirror sites + control plane on :8101.
# 25 Flask mirror sites + control plane on :8101.

FROM python:3.12-slim-bookworm

Expand Down Expand Up @@ -72,6 +72,6 @@ os.makedirs('instance_seed', exist_ok=True); \
shutil.copy2('instance/rotten_tomatoes.db', 'instance_seed/rotten_tomatoes.db'); \
print('Rotten Tomatoes seed DB generated at build time.')" && rm -rf /opt/WebSyn/rotten_tomatoes/instance

EXPOSE 8101 40000-40023
EXPOSE 8101 40000-40024

CMD ["/opt/websyn_start.sh"]
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod
- **Deep features unlocked** — carts, checkouts, accounts, all fully testable
- **Evolving** — harder tasks drive richer mirrors; the environment grows with agents
- **RL-ready** — sub-second database resets between rollouts
- **Community-driven** — 24 sites today, scaling to 100+ together
- **Community-driven** — 25 sites today, scaling to 100+ together

## 🚀 Quickstart

One command to run all web environments:

```bash
docker run -p 8101:8101 -p 40000-40023:40000-40023 battalion7244/webharbor:latest
docker run -p 8101:8101 -p 40000-40024:40000-40024 battalion7244/webharbor:latest
```

Then point your agent at `http://localhost:40000` through `http://localhost:40023` to explore 24 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, and Walmart Careers`.
Then point your agent at `http://localhost:40000` through `http://localhost:40024` to explore 25 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, and Bandcamp`.

For sub-second reset between rollouts, expose the control plane and call `/reset/<site>`:

Expand All @@ -65,7 +65,7 @@ git clone https://github.com/aiming-lab/WebHarbor && cd WebHarbor

## 🤝 Contribute

We have built 23 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.
We have built 25 high-quality mirrors covering the [WebVoyager](https://github.com/MinorJerry/WebVoyager) benchmark. The next goal is **100+ sites**, covering everything in [Online-Mind2Web](https://huggingface.co/datasets/osunlp/Online-Mind2Web). We are inviting the community to build this together.

There are two ways to join the author list:

Expand Down
2 changes: 1 addition & 1 deletion agent_demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1 # or your Azure / vLLM endpoi

## Run a task

WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40023:40000-40023 battalion7244/webharbor:latest`).
WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40024:40000-40024 battalion7244/webharbor:latest`).

Run a single task from a site's `tasks.jsonl`:

Expand Down
3 changes: 2 additions & 1 deletion control_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@
'allrecipes', 'amazon', 'apple', 'arxiv', 'bbc_news', 'booking',
'github', 'google_flights', 'google_map', 'google_search',
'huggingface', 'wolfram_alpha', 'cambridge_dictionary',
'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target', 'ted', 'osu', 'rotten_tomatoes', 'compass', 'walmart_careers',
'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target', 'ted', 'osu',
'rotten_tomatoes', 'compass', 'walmart_careers', 'bandcamp',
]
BASE_PORT = 40000
WEBSYN_DIR = '/opt/WebSyn'
Expand Down
35 changes: 35 additions & 0 deletions sites/bandcamp/_health.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
"""Bandcamp mirror health check."""
from healthcheck import random_user


def run(p):
p.assert_get("home", "/", must_contain="bandcamp")
p.assert_get("discover", "/discover", must_contain="Featured release")
p.assert_get("search", "/search?q=tidal", must_contain="Tidal Memory")

user = random_user()
p.assert_post(
"register",
"/register",
{
"display_name": user["name"],
"username": user["username"],
"email": user["email"],
"city": "Seattle",
"password": user["password"],
"confirm_password": user["password"],
"favorite_format": "digital",
},
accept_status=(200, 302, 303),
)
p.get("/logout")
p.assert_post(
"login",
"/login",
{
"email": user["email"],
"password": user["password"],
},
accept_status=(200, 302, 303),
)
p.assert_get("account", "/account", must_contain=user["first_name"])
Loading