Add UC Berkeley mirror site (port 40015)

[richard-peng-xia]

TL;DR

Adds a fully functional berkeley.edu mirror site to WebHarbor at port 40015, with 30 benchmark tasks covering programs, news, events, faculty, research centers, and admissions.

Motivation

UC Berkeley is the #1 public research university in the US and one of the most-visited university portals. It covers a domain — university information browsing (academics, research, campus life) — not represented in the existing 15 sites, and offers rich multi-step navigation tasks across programs, faculty, events, and news that are well-suited for web-agent benchmarks.

Design

Flask application (sites/berkeley/app.py)

Eight SQLAlchemy models: User, College, Department, Program, NewsArticle, Event, ResearchCenter, Faculty, Bookmark. All seeded idempotently (each seed function gates on a populated DB to preserve the byte-identical reset invariant).

Route coverage mirrors the real site's navigation:

• Homepage with featured news, upcoming events, and quick stats
• News listing + article detail with category/search filters
• Academics overview listing all 14 colleges/schools
• Programs listing + detail with degree type and college filters
• Events listing + detail with category and date filters
• Research centers listing + detail pages
• Departments listing (grouped by college) + detail with faculty roster
• Faculty listing + profile pages with department filter
• Admissions overview with undergraduate/graduate tabs
• About page with real Berkeley statistics
• Unified search across programs, news, events, and faculty
• Auth: login, register, logout, account (bookmarks)

Seed database

14 UC Berkeley colleges/schools (real names), 83 degree programs (BA/BS/MA/MS/PhD/MBA/JD/MD/MEng), 121 news articles (2023–2025, 7 categories), 64 events (upcoming + past, 7 categories), 25 research centers (BAIR, QB3, MSRI, …), 82 faculty (Jennifer Doudna, Stuart Russell, Saul Perlmutter, …), and 4 benchmark users (///, password: ). Seed DB generated at image build time via .

Templates

23 Jinja2 templates styled with Berkeley Blue () and California Gold (), modeled on real berkeley.edu layout: responsive nav with five top-level sections (About / Admissions / Academics / Research / Campus Life), card-grid listings, detail pages with sidebars, and paginated results (20 items/page).

Benchmark tasks (tasks.jsonl)

30 tasks (IDs through ) covering: program search by degree type, news browsing by category, event filtering by date/type, faculty research lookup, research center exploration, admissions requirements, department navigation, and 5+ multi-step reasoning tasks.

Verification

Check	Result
	✅
All 11 main routes → HTTP 200 (werkzeug test client)	✅
Seed: 14 colleges, 83 programs, 121 articles, 64 events, 82 faculty, 25 centers, 4 users	✅
Seed idempotent (second run produces no duplicate rows)	✅
[build] missing assets, fetching from HF...
[fetch] huggingface.co/datasets/ChilleD/WebHarbor @ main -> sites/
path=/home/pxia/WebHarbor/sites/.cache/tarballs
[fetch] extracting allrecipes
[fetch] extracting amazon
[fetch] extracting apple
[fetch] extracting arxiv
[fetch] extracting bbc_news
[fetch] extracting booking
[fetch] extracting cambridge_dictionary
[fetch] extracting coursera
[fetch] extracting espn
[fetch] extracting github
[fetch] extracting google_flights
[fetch] extracting google_map
[fetch] extracting google_search
[fetch] extracting huggingface
[fetch] extracting wolfram_alpha
[fetch] done — 15 site(s) extracted into sites/
[check] all sites have instance_seed/ (15 sites lack at least one optional asset dir — that's OK)
[build] docker build -t webharbor:dev .	requires Docker daemon — not run locally
+ md5sum match	requires Docker daemon — not run locally

HuggingFace assets

Berkeley has no scraped image assets — all data is code-generated from . The is therefore built directly inside the Docker image via a step added to the Dockerfile, eliminating the need for a HuggingFace tarball for this site. No bump is required.

Registration

Site registered in all three required locations:

• — index 15, port 40015

🤖 Generated with Claude Code

[YuanDaoze]

Review

Tested on a fresh worktree of this branch with the 15 existing sites' assets hard-linked from upstream/main (this PR ships none of its own), built as webharbor:rev11, run on alt ports :8211 / :42100-42115 to avoid colliding with my own dev container.

What works ✓

Mechanical

• Three-place registration in sync (websyn_start.sh SITES, control_server.py SITES, Dockerfile EXPOSE 40000-40015).
• curl http://:8211/health shows all 16 sites alive after boot. berkeley on :42115 returns 200.
• Byte-identical reset holds: md5(instance/berkeley.db) == md5(instance_seed/berkeley.db) both before and after POST /reset/berkeley. The function-level gate if College.query.first(): return in seed() is doing its job.
• tasks.jsonl has 30 tasks, all five required fields (web_name / id / ques / web / upstream_url) on every line, port matches 40015.

Functional depth

• /academics (14 colleges with dean / founded / undergrad+grad counts), /research, /news, /events, /faculty, /admissions, /about, plus per-program / per-department / per-faculty / per-research-center detail routes.
• /search aggregates across programs + news + faculty + events + research centers — 27/52/29 results for "computer science" / "engineering" / "PhD" sampled.
• Login (alice@berkeley.edu / test1234) succeeds and /account renders. Bookmark + register routes wired with CSRF.

Task quality

• 30 tasks span 8 content types, with a reasonable difficulty gradient: single-page lookup (--0, --8) → multi-step aggregation (--24, --27) → cross-page consistency (--29).
• Spot-checked 5 random tasks, all target URLs return 200 and the target facts are present (e.g. --10 BAIR director Pieter Abbeel, founded 2013, at /research/bair).
• No obvious count-leak or single-item-catalog leaks found.

Blocking ✗

The PR description says HuggingFace upload is intentionally skipped because berkeley has no scraped images and the seed DB is generated in-Docker. Fair design choice in principle, but the implementation has two concrete bugs:

1. Build fails on a fresh clone.

RUN cd /opt/WebSyn/berkeley && \
    python3 -c "from app import app" && \
    cp instance/berkeley.db instance_seed/berkeley.db

sites/berkeley/instance_seed/ is not committed (only static/css/.gitkeep and static/js/.gitkeep are), so the cp step errors with:

cp: cannot create regular file 'instance_seed/berkeley.db': No such file or directory

I had to mkdir -p sites/berkeley/instance_seed manually before the build would succeed. Easy fix: git add sites/berkeley/instance_seed/.gitkeep.

2. The image isn't reproducible.

User.created_at = db.Column(..., default=datetime.utcnow) and seed_data.py doesn't pass an explicit created_at for the four benchmark users — every build stamps different timestamps into the DB. Within a single image the reset invariant still holds (reset re-copies the same instance_seed/berkeley.db), but md5(instance_seed/berkeley.db) will differ between two builds of the same source tree, which contradicts the WebHarbor reproducibility goal.

Two ways to fix:

• (cheapest) hardcode created_at in seed_data.py: User(..., created_at=datetime(2025,1,1)). Same for NewsArticle.published_date defaults if any rows lack an explicit value.
• (cleaner, matches the rest of the project) generate berkeley.db once locally, ship it via the HF dataset like the other 15 sites, drop the in-Docker generation. Even without images, a berkeley.tar.gz containing only instance_seed/berkeley.db would be a few hundred KB.

Should-fix (non-blocking)

3. Zero images. Home / news / faculty / research all use solid-color cards with Unicode glyphs. berkeley.edu in reality has news photos, faculty headshots, and campus images. static/images/ doesn't exist; static/css/ and static/js/ are empty directories (all CSS is inlined in base.html). Visual fidelity feels wireframe-like rather than mirror-like. Look at cambridge_dictionary — ~50 screenshot/placeholder PNGs in HF, ~7MB total — for a model that's cheap but lifts visual fidelity considerably.

4. sites/berkeley/_health.py is orphan code. It imports from healthcheck import random_user, but no healthcheck module exists in the repo and no other site has a _health.py. The file is dead weight (and import-time-broken if anything ever picks it up).

5. PR base is 3c408d8 init, behind upstream/main. Several PRs have merged since (e.g. merriam_webster, coursera). A rebase + force-push will pick those up and avoid hidden conflicts at merge.

6. Login email field is type="text" instead of type="email".

<input type="text" name="email" id="email" ...>

Functionally fine but skips browser-side email validation, and agents using a generic input[type=email] selector won't match it. Minor.

Bottom line

Code style, UX, and task design are solid. Two small fixes (commit instance_seed/.gitkeep + hardcode user created_at) make this mergeable as-is; or a slightly larger refactor (move seed DB to HF dataset) brings it in line with the other 15 sites.

[evanz37]

I'm claiming PR #11 for reviewer takeover (Track B). I'll preserve @richard-peng-xia's original contribution unchanged, carry it on a separate reviewer-owned branch (merged with current main and re-slotted to the next free port), audit and fix the environment / tasks / verifiers per CONTRIBUTING's Reviewer role, and submit a linked Draft Review: PR with reproducible evidence. I will not merge either PR.