diff --git a/AGENTS.md b/AGENTS.md index d2feea97..bbff1781 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,17 +48,17 @@ Inside the image, sites live at `/opt/WebSyn//`. The path predates the ren # fresh clone ./scripts/fetch_assets.sh # pulls assets from HF ./scripts/build.sh # docker build -t webharbor:dev . -docker run -d -p 8101:8101 -p 40000-40025:40000-40025 webharbor:dev +docker run -d -p 8101:8101 -p 40000-40026:40000-40026 webharbor:dev ``` Or use the published image directly: ```bash -docker run -d -p 8101:8101 -p 40000-40025:40000-40025 \ +docker run -d -p 8101:8101 -p 40000-40026:40000-40026 \ battalion7244/webharbor:latest ``` -Sites are on `40000`-`40025` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane: +Sites are on `40000`-`40026` in the order declared by `SITES=( ... )` in `websyn_start.sh`. Control plane: | Method | Path | Purpose | |--------|---------------------|-------------------------------------------| @@ -136,13 +136,13 @@ python3 -m py_compile sites//app.py # 3. run on alt ports (don't collide with anything you already have running) docker run -d --rm --name wh-test \ - -p 8201:8101 -p 41000-41025:40000-40025 webharbor:dev + -p 8201:8101 -p 41000-41026:40000-40026 webharbor:dev # 4. control plane healthy, all sites alive curl -s http://localhost:8201/health | python3 -m json.tool | head # 5. every site renders 200 -for p in $(seq 41000 41025); do +for p in $(seq 41000 41026); do curl -so /dev/null -w "$p:%{http_code}\n" http://localhost:$p/ done diff --git a/CLAUDE.md b/CLAUDE.md index ae29df2f..67932de3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,4 +16,4 @@ The full agent guide is loaded above via `@AGENTS.md`. The notes below apply onl ## Existing containers -If a container is already running on `:8101` / `:40000-40025`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41025`). +If a container is already running on `:8101` / `:40000-40026`, treat it as the user's working environment — don't `docker stop` or `docker rm` it without explicit confirmation. Spin up your test container under a different name on alt ports (`:8201`, `:41000-41026`). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5a452f85..8285cd88 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -24,7 +24,7 @@ git clone https://github.com//webharbor && cd webharbor ./scripts/fetch_assets.sh # pull current assets ./scripts/new_site.py mywebsite # OR edit an existing site ./scripts/build.sh && docker run -d --rm \ - -p 8101:8101 -p 40000-40025:40000-40025 webharbor:dev + -p 8101:8101 -p 40000-40026:40000-40026 webharbor:dev # iterate locally... ./scripts/extract_assets.sh ../webharbor-static-pr/ # split assets out diff --git a/Dockerfile b/Dockerfile index 68194582..583bc557 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,5 @@ # WebHarbor — slim, self-contained image. -# 26 Flask mirror sites + control plane on :8101. +# 27 Flask mirror sites + control plane on :8101. FROM python:3.12-slim-bookworm @@ -86,6 +86,6 @@ os.makedirs('instance_seed', exist_ok=True); \ shutil.copy2('instance/rotten_tomatoes.db', 'instance_seed/rotten_tomatoes.db'); \ print('Rotten Tomatoes seed DB generated at build time.')" && rm -rf /opt/WebSyn/rotten_tomatoes/instance -EXPOSE 8101 40000-40025 +EXPOSE 8101 40000-40026 CMD ["/opt/websyn_start.sh"] diff --git a/README.md b/README.md index 7830b870..93c8d0a4 100644 --- a/README.md +++ b/README.md @@ -36,17 +36,17 @@ WebHarbor takes a different approach. We leverage coding agent (e.g., Claude Cod - **Deep features unlocked** — carts, checkouts, accounts, all fully testable - **Evolving** — harder tasks drive richer mirrors; the environment grows with agents - **RL-ready** — sub-second database resets between rollouts -- **Community-driven** — 26 sites today, scaling to 100+ together +- **Community-driven** — 27 sites today, scaling to 100+ together ## 🚀 Quickstart One command to run all web environments: ```bash -docker run -p 8101:8101 -p 40000-40025:40000-40025 battalion7244/webharbor:latest +docker run -p 8101:8101 -p 40000-40026:40000-40026 battalion7244/webharbor:latest ``` -Then point your agent at `http://localhost:40000` through `http://localhost:40025` to explore 26 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, and WebMD Doctor`. +Then point your agent at `http://localhost:40000` through `http://localhost:40025` to explore 27 local mirrors of WebVoyager sites: `Allrecipes, Amazon, Apple, ArXiv, BBC News, Booking, GitHub, Google Flights, Google Maps, Google Search, Hugging Face, Wolfram Alpha, Cambridge Dictionary, Coursera, ESPN, Merriam-Webster, IKEA, Phys.org, Target, TED, Ohio State University, Rotten Tomatoes, Compass, Walmart Careers, FedEx, WebMD Doctor, and 9GAG`. For sub-second reset between rollouts, expose the control plane and call `/reset/`: diff --git a/agent_demo/README.md b/agent_demo/README.md index 1d6ec7ad..598634b3 100644 --- a/agent_demo/README.md +++ b/agent_demo/README.md @@ -19,7 +19,7 @@ export OPENAI_BASE_URL=https://api.openai.com/v1 # or your Azure / vLLM endpoi ## Run a task -WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40025:40000-40025 battalion7244/webharbor:latest`). +WebHarbor must already be running locally (`docker run -p 8101:8101 -p 40000-40026:40000-40026 battalion7244/webharbor:latest`). Run a single task from a site's `tasks.jsonl`: diff --git a/control_server.py b/control_server.py index fb3fa968..fdf53407 100644 --- a/control_server.py +++ b/control_server.py @@ -29,6 +29,7 @@ 'coursera', 'espn', 'merriam_webster', 'ikea', 'phys_org', 'target', 'ted', 'osu', 'rotten_tomatoes', 'compass', 'walmart_careers', 'fedex', 'webmd_doctor', + '9gag', ] BASE_PORT = 40000 WEBSYN_DIR = '/opt/WebSyn' diff --git a/review-reports/9GAG-FINAL-AUDIT.md b/review-reports/9GAG-FINAL-AUDIT.md new file mode 100644 index 00000000..4b0f3d13 --- /dev/null +++ b/review-reports/9GAG-FINAL-AUDIT.md @@ -0,0 +1,62 @@ +# 9GAG contributor-side final audit + +## Scope + +- Mirror: https://9gag.com/ +- Local site: `http://localhost:40024/` (host verification port `41024`) +- Branch: `add-9gag-mirror` +- Base: upstream `main` at `3600493` +- Docker image: `sha256:0cf5649d5cb058fa05d1887a3559f78b327e84e5a78320f2aa6f795a2ab31dc6` + +## Results + +- All 25 registered WebHarbor sites started successfully. +- `/reset-all` returned every site as ready. +- All 20 9GAG tasks passed from the homepage through visible Playwright controls, with a fresh browser context and site reset for each task. +- 27 representative responsive checks passed across 1440, 390, and 320 px. +- No broken images, stretched images, horizontal document overflow, or out-of-bounds controls were detected. +- Five hardened search families returned 9–10 visible results; targets appeared at positions 2, 2, 4, 5, and 7. +- None of the task-specific detail facts appeared on search-result cards. +- Runtime and seed database MD5 after the final task run and `/reset-all`: `44768940829bf4a8bec07d7bdf072d97`. + +## Seed inventory + +- Posts: 80 +- Users: 4 +- Comments: 4 +- Saved-post rows: 16 +- Vote rows: 12 +- Genuine captured post images: 38 unique files +- Additional captured 9GAG brand/interest references: 27 files + +All four benchmark accounts use `@test.com` addresses and password `TestPass123!`. Both `seed_database()` and `seed_benchmark_users()` gate the entire function; repeated application left database bytes unchanged. + +## Asset package prepared for Hugging Face + +- Archive: `9gag.tar.gz` +- Managed members: 74 +- SHA-256: `6ba9ca464d23bcae76316efff234548f75e428a7570f1d08f004c38a1123841c` +- The archive was validated locally and was not uploaded. + +## Screenshots + +### Desktop — 1440 px + +![9GAG desktop homepage](assets/9gag-homepage-1440.png) + +### Mobile — 390 px + +![9GAG mobile homepage](assets/9gag-homepage-390.png) + +### Narrow mobile — 320 px + +![9GAG narrow mobile homepage](assets/9gag-homepage-320.png) + +## Human integration steps + +1. Upload the prepared `9gag.tar.gz` to the contributor's Hugging Face dataset fork and open an assets PR against `ChilleD/WebHarbor`. +2. After the HF PR merges, update `.assets-revision` to the merged asset commit. +3. Rebuild and rerun the 25-site health, reset-all, task, and MD5 checks. +4. Push `add-9gag-mirror` and open the GitHub PR, linking the merged HF asset commit and this report. + +No Hugging Face or GitHub PR was opened during preparation. diff --git a/review-reports/assets/9gag-homepage-1440.png b/review-reports/assets/9gag-homepage-1440.png new file mode 100644 index 00000000..8ca5077e Binary files /dev/null and b/review-reports/assets/9gag-homepage-1440.png differ diff --git a/review-reports/assets/9gag-homepage-320.png b/review-reports/assets/9gag-homepage-320.png new file mode 100644 index 00000000..4725f596 Binary files /dev/null and b/review-reports/assets/9gag-homepage-320.png differ diff --git a/review-reports/assets/9gag-homepage-390.png b/review-reports/assets/9gag-homepage-390.png new file mode 100644 index 00000000..004721d3 Binary files /dev/null and b/review-reports/assets/9gag-homepage-390.png differ diff --git a/sites/9gag/ASSET_SOURCES.md b/sites/9gag/ASSET_SOURCES.md new file mode 100644 index 00000000..175341f4 --- /dev/null +++ b/sites/9gag/ASSET_SOURCES.md @@ -0,0 +1,24 @@ +# 9GAG asset provenance + +All media in `static/images/` was captured from the rendered public 9GAG homepage on 2026-09-09 with Playwright. The browser exposed the resolved CDN URLs after hydration; only then were the image bytes downloaded. No placeholders, generated images, private data, cookies, or browser-profile material are included. + +## Post media + +- `posts/aD2BwPZ-451efa65.jpg` — https://img-9gag-fun.9cache.com/photo/aD2BwPZ_460s.jpg +- `posts/aVvGONd-41d662a3.jpg` — https://miscmedia-9gag-fun.9cache.com/images/thumbnail-facebook/71134346_1788942069.8925_rYNuRA_700b.jpg +- `posts/aO86yA2-87d68ca6.jpg` — https://img-9gag-fun.9cache.com/photo/aO86yA2_460s.jpg +- `posts/aryPpR7-3497fabc.jpg` — https://img-9gag-fun.9cache.com/photo/aryPpR7_460s.jpg +- `posts/amoDNzv-6cbe8b39.jpg` — https://img-9gag-fun.9cache.com/photo/amoDNzv_700b.jpg +- `posts/aQzYqZq-9c4605da.jpg` — https://img-9gag-fun.9cache.com/photo/aQzYqZq_460s.jpg +- `posts/aLnqKLv-bea354c8.jpg` — https://img-9gag-fun.9cache.com/photo/aLnqKLv_700b.jpg +- `posts/adB3Nj9-f56e7b44.jpg` — https://img-9gag-fun.9cache.com/photo/adB3Nj9_460s.jpg +- `posts/aYQzNgV-f6067981.jpg` — https://img-9gag-fun.9cache.com/photo/aYQzNgV_460s.jpg +- `posts/aQzYqK8-ec9e7408.jpg` — https://img-9gag-fun.9cache.com/photo/aQzYqK8_460s.jpg + +## Brand and interest references + +The files under `static/images/reference/` are the 9GAG logo and interest thumbnails resolved from `miscmedia-9gag-fun.9cache.com` by the same rendered page. Their original URLs and dimensions are retained in the ignored reconnaissance record `scraped_data/capture.json`. + +An additional 28 unique feed images under `static/images/posts/` were captured through Chromium from the public `/v1/group-posts/group/default/type/{hot,trending,fresh}?c=10` responses. Their source post IDs remain in every filename, and the resolved source URLs are retained in the ignored reconnaissance record `scraped_data/feed-posts.json`. + +The 80-row seeded catalog rotates across 38 genuine 9GAG post images while keeping titles, descriptions, authors, tags, and task facts distinct. This avoids blank, generated, or placeholder imagery and limits repetition across feed pages. diff --git a/sites/9gag/_health.py b/sites/9gag/_health.py new file mode 100644 index 00000000..13420bb9 --- /dev/null +++ b/sites/9gag/_health.py @@ -0,0 +1,3 @@ +"""Per-site health probe (optional, called by control_server).""" +def health(): + return {"ok": True, "site": "9gag"} diff --git a/sites/9gag/app.py b/sites/9gag/app.py new file mode 100644 index 00000000..80eaf2b4 --- /dev/null +++ b/sites/9gag/app.py @@ -0,0 +1,507 @@ +"""A deterministic, task-complete local mirror of 9GAG.""" + +from __future__ import annotations + +import os +import re +import secrets + +from flask import Flask, abort, flash, redirect, render_template, request, url_for +from flask_login import LoginManager, UserMixin, current_user, login_required, login_user, logout_user +from flask_sqlalchemy import SQLAlchemy +from flask_wtf.csrf import CSRFProtect +from sqlalchemy import UniqueConstraint +from werkzeug.security import check_password_hash, generate_password_hash + +BASE_DIR = os.path.dirname(os.path.abspath(__file__)) +INSTANCE_DIR = os.path.join(BASE_DIR, "instance") +os.makedirs(INSTANCE_DIR, exist_ok=True) + +# Single source of truth for every closed vocabulary the forms offer: the templates render these +# and the POST handlers validate against them, so the form and the server cannot drift apart. +POST_INTERESTS = ("humor", "memes", "gaming", "animals", "science", "wholesome", + "sports", "food", "music", "politics", "random") +REPORT_REASONS = ("Spam", "Harassment", "Misinformation", "Graphic content") +PREVIEW_IMAGES = ("posts/aVvGONd-41d662a3.jpg", "posts/adB3Nj9-f56e7b44.jpg", + "posts/aYQzNgV-f6067981.jpg", "posts/aQzYqK8-ec9e7408.jpg") + +app = Flask(__name__, instance_path=INSTANCE_DIR) +app.config.update( + # Not a tracked literal: a fixed key lets anyone mint a session cookie for a seeded account. + SECRET_KEY=os.environ.get("NINEGAG_SECRET_KEY") or secrets.token_hex(32), + SQLALCHEMY_DATABASE_URI=f"sqlite:///{os.path.join(INSTANCE_DIR, '9gag.db')}", + SQLALCHEMY_TRACK_MODIFICATIONS=False, + MAX_CONTENT_LENGTH=256 * 1024, + WTF_CSRF_TIME_LIMIT=7200, +) +db = SQLAlchemy(app) +csrf = CSRFProtect(app) +login_manager = LoginManager(app) +login_manager.login_view = "login" + + +class User(UserMixin, db.Model): + id = db.Column(db.Integer, primary_key=True) + username = db.Column(db.String(80), unique=True, nullable=False) + email = db.Column(db.String(160), unique=True, nullable=False) + display_name = db.Column(db.String(120), nullable=False) + password_hash = db.Column(db.String(255), nullable=False) + bio = db.Column(db.String(500), default="") + location = db.Column(db.String(120), default="") + joined_at = db.Column(db.String(30), default="September 2026") + + def set_password(self, password): + self.password_hash = generate_password_hash(password) + + def check_password(self, password): + return check_password_hash(self.password_hash, password) + + +class Post(db.Model): + id = db.Column(db.Integer, primary_key=True) + source_id = db.Column(db.String(20), unique=True, nullable=False) + slug = db.Column(db.String(180), unique=True, nullable=False) + title = db.Column(db.String(300), nullable=False) + description = db.Column(db.Text, nullable=False) + image = db.Column(db.String(255), nullable=False) + section = db.Column(db.String(80), nullable=False) + post_type = db.Column(db.String(30), default="Photo") + tags = db.Column(db.String(400), default="") + author_name = db.Column(db.String(80), nullable=False) + up_votes = db.Column(db.Integer, default=0) + down_votes = db.Column(db.Integer, default=0) + comment_count = db.Column(db.Integer, default=0) + created_rank = db.Column(db.Integer, nullable=False) + featured = db.Column(db.Boolean, default=False) + + @property + def tag_list(self): + return [tag.strip() for tag in self.tags.split("|") if tag.strip()] + + +class Comment(db.Model): + id = db.Column(db.Integer, primary_key=True) + post_id = db.Column(db.Integer, db.ForeignKey("post.id"), nullable=False) + user_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False) + body = db.Column(db.String(1000), nullable=False) + created_at = db.Column(db.String(40), nullable=False) + post = db.relationship("Post", backref="comments") + user = db.relationship("User") + + +class Vote(db.Model): + id = db.Column(db.Integer, primary_key=True) + user_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False) + post_id = db.Column(db.Integer, db.ForeignKey("post.id"), nullable=False) + value = db.Column(db.Integer, nullable=False) + __table_args__ = (UniqueConstraint("user_id", "post_id", name="uq_vote_user_post"),) + + +class SavedPost(db.Model): + id = db.Column(db.Integer, primary_key=True) + user_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False) + post_id = db.Column(db.Integer, db.ForeignKey("post.id"), nullable=False) + post = db.relationship("Post") + __table_args__ = (UniqueConstraint("user_id", "post_id", name="uq_saved_user_post"),) + + +class HiddenPost(db.Model): + id = db.Column(db.Integer, primary_key=True) + user_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False) + post_id = db.Column(db.Integer, db.ForeignKey("post.id"), nullable=False) + __table_args__ = (UniqueConstraint("user_id", "post_id", name="uq_hidden_user_post"),) + + +class Report(db.Model): + id = db.Column(db.Integer, primary_key=True) + user_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False) + post_id = db.Column(db.Integer, db.ForeignKey("post.id"), nullable=False) + reason = db.Column(db.String(120), nullable=False) + + +@login_manager.user_loader +def load_user(user_id): + return db.session.get(User, int(user_id)) + + +def slugify(value): + return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")[:160] or "post" + + +def bounded_text(value, limit, field): + """SQLite does not enforce VARCHAR length: reject over-long input instead of storing it.""" + text = str(value or "").strip() + if len(text) > limit: + abort(400, f"{field} must be at most {limit} characters") + return text + + +def next_local_source_id(): + """MAX(numeric suffix) + 1, so a removed row can never make the next insert collide.""" + used = [int(match.group(1)) + for row in Post.query.filter(Post.source_id.like("local-%")).all() + if (match := re.fullmatch(r"local-(\d+)", row.source_id or ""))] + return f"local-{max(used, default=Post.query.count()) + 1}" + + +def scored_posts(query, posts): + stop = {"the", "a", "an", "in", "on", "at", "to", "for", "of", "and", "or", "is", "it", "by", "with"} + tokens = [token for token in re.split(r"\W+", query.lower()) if len(token) > 1 and token not in stop] + if not tokens: + return list(posts) + ranked = [] + for post in posts: + haystack = f"{post.title} {post.description} {post.section} {post.tags} {post.author_name}".lower() + score = sum(3 if token in post.title.lower() else 1 for token in tokens if token in haystack) + if score: + ranked.append((post, score)) + return [post for post, _ in sorted(ranked, key=lambda item: (-item[1], -item[0].up_votes, item[0].id))] + + +def visible_posts(): + query = Post.query + if current_user.is_authenticated: + hidden_ids = [row.post_id for row in HiddenPost.query.filter_by(user_id=current_user.id)] + if hidden_ids: + query = query.filter(~Post.id.in_(hidden_ids)) + return query + + +def render_feed(title="Hot", posts=None, active="home", query=""): + posts = posts if posts is not None else visible_posts().order_by(Post.featured.desc(), Post.up_votes.desc()).all() + page_number = max(request.args.get("page", 1, type=int), 1) + total = len(posts) + page_size = 10 + posts = posts[(page_number - 1) * page_size:page_number * page_size] + saved_ids, votes = set(), {} + if current_user.is_authenticated: + saved_ids = {row.post_id for row in SavedPost.query.filter_by(user_id=current_user.id)} + votes = {row.post_id: row.value for row in Vote.query.filter_by(user_id=current_user.id)} + return render_template("feed.html", title=title, posts=posts, active=active, query=query, + saved_ids=saved_ids, votes=votes, page_number=page_number, + has_previous=page_number > 1, has_next=page_number * page_size < total) + + +@app.route("/") +@app.route("/home") +def index(): + return render_feed() + + +@app.route("/top") +def top(): + return render_feed("Top", visible_posts().order_by(Post.up_votes.desc()).all(), "top") + + +@app.route("/trending") +def trending(): + posts = sorted(visible_posts().all(), key=lambda post: post.up_votes + post.comment_count * 4, reverse=True) + return render_feed("Trending", posts, "trending") + + +@app.route("/fresh") +def fresh(): + return render_feed("Fresh", visible_posts().order_by(Post.created_rank.desc()).all(), "fresh") + + +@app.route("/news") +def news(): + posts = visible_posts().filter(Post.section.in_(["news", "politics", "science"])).order_by(Post.created_rank.desc()).all() + return render_feed("News", posts, "news") + + +@app.route("/interest/
") +def interest(section): + posts = visible_posts().filter(db.func.lower(Post.section) == section.lower()).order_by(Post.up_votes.desc()).all() + return render_feed(section.replace("-", " ").title(), posts, section) + + +@app.route("/tag/") +def tag(tag): + needle = tag.replace("-", " ").lower() + posts = visible_posts().filter(db.func.lower(Post.tags).contains(needle)).all() + return render_feed(f"#{needle}", posts, "", needle) + + +@app.route("/search") +def search(): + query = request.args.get("q", "").strip() + return render_feed("Search", scored_posts(query, visible_posts().all()), "", query) + + +@app.route("/shuffle") +def shuffle(): + return redirect(url_for("post_detail", slug=Post.query.order_by(db.func.random()).first_or_404().slug)) + + +@app.route("/gag/") +def post_detail(slug): + post = Post.query.filter_by(slug=slug).first_or_404() + saved = current_user.is_authenticated and SavedPost.query.filter_by(user_id=current_user.id, post_id=post.id).first() is not None + vote = Vote.query.filter_by(user_id=current_user.id, post_id=post.id).first() if current_user.is_authenticated else None + related = Post.query.filter(Post.section == post.section, Post.id != post.id).order_by(Post.up_votes.desc()).limit(5).all() + return render_template("post_detail.html", post=post, saved=saved, vote=vote, related=related) + + +@app.post("/gag//vote") +@login_required +def vote(slug): + post = Post.query.filter_by(slug=slug).first_or_404() + raw_value = request.form.get("value", "") + if raw_value not in {"1", "-1"}: + abort(400, "vote value must be 1 or -1") + value = int(raw_value) + existing = Vote.query.filter_by(user_id=current_user.id, post_id=post.id).first() + if existing and existing.value == value: + db.session.delete(existing) + post.up_votes -= int(value == 1) + post.down_votes -= int(value == -1) + elif existing: + post.up_votes += 1 if value == 1 else -1 + post.down_votes += 1 if value == -1 else -1 + existing.value = value + else: + db.session.add(Vote(user_id=current_user.id, post_id=post.id, value=value)) + post.up_votes += int(value == 1) + post.down_votes += int(value == -1) + db.session.commit() + return redirect(request.referrer or url_for("post_detail", slug=slug)) + + +@app.post("/gag//save") +@login_required +def save_post(slug): + post = Post.query.filter_by(slug=slug).first_or_404() + row = SavedPost.query.filter_by(user_id=current_user.id, post_id=post.id).first() + if row: + db.session.delete(row) + flash("Removed from saved posts.") + else: + db.session.add(SavedPost(user_id=current_user.id, post_id=post.id)) + flash("Saved to your collection.") + db.session.commit() + return redirect(request.referrer or url_for("post_detail", slug=slug)) + + +@app.post("/gag//comment") +@login_required +def add_comment(slug): + post = Post.query.filter_by(slug=slug).first_or_404() + body = bounded_text(request.form.get("body"), 1000, "comment body") + if len(body) < 2: + flash("Comment must contain at least 2 characters.", "error") + else: + db.session.add(Comment(post_id=post.id, user_id=current_user.id, body=body, created_at="just now")) + post.comment_count += 1 + db.session.commit() + flash("Comment posted.") + return redirect(url_for("post_detail", slug=slug) + "#comments") + + +@app.post("/gag//hide") +@login_required +def hide_post(slug): + post = Post.query.filter_by(slug=slug).first_or_404() + if not HiddenPost.query.filter_by(user_id=current_user.id, post_id=post.id).first(): + db.session.add(HiddenPost(user_id=current_user.id, post_id=post.id)) + db.session.commit() + flash("Post hidden from your feeds.") + return redirect(url_for("index")) + + +@app.route("/gag//report", methods=["GET", "POST"]) +@login_required +def report_post(slug): + post = Post.query.filter_by(slug=slug).first_or_404() + if request.method == "POST": + reason = request.form.get("reason", "").strip() + if reason and reason not in REPORT_REASONS: + abort(400, "unknown report reason") + if reason: + db.session.add(Report(user_id=current_user.id, post_id=post.id, reason=reason)) + db.session.commit() + flash("Thanks. Your report was submitted.") + return redirect(url_for("post_detail", slug=slug)) + flash("Choose a reason.", "error") + return render_template("report.html", post=post, reasons=REPORT_REASONS) + + +@app.route("/submit", methods=["GET", "POST"]) +@login_required +def submit(): + if request.method == "POST": + title = bounded_text(request.form.get("title"), 300, "title") + description = bounded_text(request.form.get("description"), 2000, "description") + tags = bounded_text(request.form.get("tags"), 400, "tags") + section = request.form.get("section", POST_INTERESTS[0]) + image = request.form.get("image", PREVIEW_IMAGES[0]) + if section not in POST_INTERESTS: + abort(400, "unknown interest") + if image not in PREVIEW_IMAGES: + abort(400, "unknown preview image") + if len(title) < 4: + flash("A title of at least 4 characters is required.", "error") + else: + base, slug, suffix = slugify(title), slugify(title), 2 + while Post.query.filter_by(slug=slug).first(): + slug, suffix = f"{base}-{suffix}", suffix + 1 + post = Post(source_id=next_local_source_id(), slug=slug, title=title, + description=description or "Shared with the 9GAG community.", + image=image, section=section, + post_type="Photo", tags=tags, author_name=current_user.username, + up_votes=0, down_votes=0, comment_count=0, created_rank=Post.query.count() + 100) + db.session.add(post) + db.session.commit() + flash("Your post is live.") + return redirect(url_for("post_detail", slug=post.slug)) + return render_template("submit.html", interests=POST_INTERESTS, images=PREVIEW_IMAGES) + + +@app.route("/login", methods=["GET", "POST"]) +def login(): + if request.method == "POST": + identity = request.form.get("identity", "").strip() + user = User.query.filter((User.email == identity) | (User.username == identity)).first() + if user and user.check_password(request.form.get("password", "")): + login_user(user) + return redirect(request.args.get("next") or url_for("index")) + flash("Incorrect email, username, or password.", "error") + return render_template("login.html") + + +@app.route("/register", methods=["GET", "POST"]) +def register(): + if request.method == "POST": + email = bounded_text(request.form.get("email"), 160, "email").lower() + username = bounded_text(request.form.get("username"), 80, "username") + password = request.form.get("password", "") + if "@" not in email or len(username) < 3 or len(password) < 8: + flash("Enter a valid email, username, and password of at least 8 characters.", "error") + elif User.query.filter((User.email == email) | (User.username == username)).first(): + flash("That email or username is already registered.", "error") + else: + user = User(email=email, username=username, display_name=username, password_hash="") + user.set_password(password) + db.session.add(user) + db.session.commit() + login_user(user) + return redirect(url_for("index")) + return render_template("register.html") + + +@app.post("/logout") +@login_required +def logout(): + logout_user() + return redirect(url_for("index")) + + +@app.route("/account") +@login_required +def account(): + posts = Post.query.filter_by(author_name=current_user.username).order_by(Post.created_rank.desc()).all() + return render_template("account.html", profile=current_user, posts=posts) + + +@app.route("/u/") +def profile(username): + profile_user = User.query.filter_by(username=username).first_or_404() + posts = Post.query.filter_by(author_name=username).order_by(Post.created_rank.desc()).all() + return render_template("account.html", profile=profile_user, posts=posts) + + +@app.route("/settings", methods=["GET", "POST"]) +@login_required +def settings(): + if request.method == "POST": + display_name = bounded_text(request.form.get("display_name"), 120, "display name") + bio = bounded_text(request.form.get("bio"), 500, "bio") + location = bounded_text(request.form.get("location"), 120, "location") + if not display_name: + flash("Display name is required.", "error") + else: + current_user.display_name = display_name + current_user.bio = bio + current_user.location = location + db.session.commit() + flash("Profile updated.") + return render_template("settings.html") + + +@app.route("/saved") +@login_required +def saved(): + rows = SavedPost.query.filter_by(user_id=current_user.id).order_by(SavedPost.id.desc()).all() + return render_feed("Saved", [row.post for row in rows], "saved") + + +@app.route("/notifications") +@login_required +def notifications(): + return render_template("notifications.html") + + +@app.route("/about") +def about(): + return render_template("static_page.html", heading="About 9GAG", body="9GAG is a community-powered entertainment platform where people discover and discuss funny, surprising, and timely posts.") + + +@app.route("/rules") +def rules(): + return render_template("static_page.html", heading="Community rules", body="Be respectful, post original context, use accurate interests and tags, and report content that breaks community standards.") + + +@app.route("/apps") +def apps(): + return render_template("apps.html") + + +@app.route("/help") +def help_page(): + return render_template("help.html") + + +@app.route("/terms") +def terms(): + return render_template("terms.html") + + +@app.errorhandler(400) +def bad_request(error): + return render_template("error.html", code=400, heading="That request could not be processed", + body=getattr(error, "description", "") or "Check the form and try again."), 400 + + +@app.errorhandler(404) +def not_found(_error): + return render_template("error.html", code=404, heading="This page is not on 9GAG", + body="The post may have been removed, or the link may be wrong."), 404 + + +@app.errorhandler(413) +def payload_too_large(_error): + return render_template("error.html", code=413, heading="That upload is too large", + body="A 9GAG post is text plus one preview image."), 413 + + +@app.errorhandler(500) +def server_error(_error): # pragma: no cover - defensive + db.session.rollback() + return render_template("error.html", code=500, heading="Something went wrong", + body="Try again in a moment."), 500 + + +@app.route("/_health") +def health(): + return {"ok": True, "site": "9gag", "posts": Post.query.count()} + + +with app.app_context(): + db.create_all() + from seed_data import seed_benchmark_users, seed_database + seed_database(db, User, Post, Comment, SavedPost, Vote) + seed_benchmark_users(db, User, Post, Comment, SavedPost, Vote) + + +if __name__ == "__main__": + app.run(host="0.0.0.0", port=int(os.environ.get("PORT", 5000)), debug=False) diff --git a/sites/9gag/requirements.txt b/sites/9gag/requirements.txt new file mode 100644 index 00000000..69e7394c --- /dev/null +++ b/sites/9gag/requirements.txt @@ -0,0 +1,4 @@ +Flask +Flask-Login +Flask-SQLAlchemy +Flask-WTF diff --git a/sites/9gag/seed_data.py b/sites/9gag/seed_data.py new file mode 100644 index 00000000..38d8015f --- /dev/null +++ b/sites/9gag/seed_data.py @@ -0,0 +1,117 @@ +"""Deterministic seed data for the 9GAG mirror.""" + +CAPTURED = [ + ("aD2BwPZ", "Boys and girls solve the same problem differently", "A split-panel joke contrasts two wildly different approaches to the same everyday problem.", "posts/aD2BwPZ-451efa65.jpg", "humor", "boys and girls|funny", "womb69raider"), + ("aVvGONd", "Meet the fan who went viral during the football game", "A worried fan in the stadium became an unexpected reaction-image favorite.", "posts/aVvGONd-41d662a3.jpg", "sports", "football|reaction|9gag news", "gamedayframe"), + ("aO86yA2", "Life is good?", "An oddly placed LG sign turns an ordinary street scene into accidental comedy.", "posts/aO86yA2-87d68ca6.jpg", "wtf", "LG|unexpected|street", "bradleypits11"), + ("aryPpR7", "A happy dude meets his new exchange-student friend", "A cheerful travel vlog follows two new friends exploring the city together.", "posts/aryPpR7-3497fabc.jpg", "wholesome", "friends|travel|vlog", "cityhellos"), + ("amoDNzv", "Chess 1 — tech billionaire 0", "A chessboard joke shows why confidence is not the same thing as strategy.", "posts/amoDNzv-6cbe8b39.jpg", "humor", "chess|technology|funny", "knightshift"), + ("aQzYqZq", "A mysterious press-conference moment from 2009", "Archive footage prompts a long discussion about media, context, and unanswered questions.", "posts/aQzYqZq-9c4605da.jpg", "news", "archive|media|history", "waitforme"), + ("aLnqKLv", "Trust me, I have not bookmarked any of these sites", "A browser-bookmark confession that every procrastinator will recognize.", "posts/aLnqKLv-bea354c8.jpg", "random", "browser|bookmarks|internet", "tabhoarder"), + ("adB3Nj9", "How helpful are you? Yes", "A determined cat insists on helping with a household repair, whether invited or not.", "posts/adB3Nj9-f56e7b44.jpg", "animals", "cat|helping|pets", "realjarmando"), + ("aYQzNgV", "The self-driving cab became a cybercop", "A satirical sketch imagines an automated taxi making one very unexpected detour.", "posts/aYQzNgV-f6067981.jpg", "science", "technology|cars|satire", "sodablob"), + ("aQzYqK8", "A remarkably polite diplomatic rebuttal", "A defense-panel exchange demonstrates how calm delivery can sharpen a pointed response.", "posts/aQzYqK8-ec9e7408.jpg", "politics", "diplomacy|philippines|conference", "freebie3"), +] + +EXTRA_IMAGES = [ + "posts/axyp2eK-a40174af.jpg", "posts/adB36rd-63e493a8.jpg", "posts/a4PM9AZ-99be8d3d.jpg", + "posts/a9y3MD1-aac2d674.jpg", "posts/aqyvZ6L-cb566fb0.jpg", "posts/aXP8E72-b35cc332.jpg", + "posts/ayNypPp-29b03ce9.jpg", "posts/aXP8EQP-fb62f010.jpg", "posts/a9y3M80-383eb696.jpg", + "posts/aW4V0rZ-e6ecd2ab.jpg", "posts/agmR0zg-12384a06.jpg", "posts/aRBxgq7-4df2c5ea.jpg", + "posts/aByDQeZ-5b7e3094.jpg", "posts/aE0eG4e-99ba422d.jpg", "posts/a87rbN1-f9e54d20.jpg", + "posts/a2vQPXd-73b4bdec.jpg", "posts/a6ZWdgA-c2528fb7.jpg", "posts/aGyPVQZ-85dd3bcf.jpg", + "posts/aByDQAN-58fb68ce.jpg", "posts/azx6LWB-1ea1705b.jpg", "posts/a87rbRV-75ef0018.jpg", + "posts/aAynV8Z-f0fe61f7.jpg", "posts/aD2BwKO-9ef53069.jpg", "posts/aVvGQp2-63e493a8.jpg", + "posts/an7AXmE-d74017c9.jpg", "posts/aGyPVmG-0bf5e5f6.jpg", "posts/a1m4ZyD-ff8359c0.jpg", + "posts/aNDyo7w-71005b4e.jpg", +] + +CURATED = [ + ("Solar-powered camping setup survives a rainy weekend", "The builder used a 120-watt folding panel and stored the controller inside a waterproof lunch box.", "science", "camping|solar|diy", "trailpixel"), + ("Tiny lighthouse office with the best ocean view", "The circular room is 4.2 meters wide and the desk was built from reclaimed oak.", "awesome", "lighthouse|office|ocean", "northcoastcam"), + ("Rescue cat learns the sound of the treat drawer", "Miso arrived at Harbor Paws in February and learned the routine after just eleven days.", "animals", "cat|rescue|wholesome", "pawsandpixels"), + ("Mechanical keyboard made entirely from transparent parts", "The board uses silent tactile switches and a hand-polished acrylic case.", "gaming", "keyboard|pc|design", "switchcraft"), + ("Grandmother finishes her first marathon at seventy-two", "Her family met her at kilometer 38 with handmade orange flags.", "sports", "running|marathon|wholesome", "milemarker"), + ("A baker recreates a city skyline in sourdough", "The edible skyline took three attempts and nearly fourteen hours to shape.", "food", "bread|baking|art", "crumbarchitect"), + ("The museum guard who quietly sketches every visitor", "After five years, the guard has filled nineteen pocket notebooks with portraits.", "art", "drawing|museum|people", "softpencil"), + ("Neighborhood builds a miniature library for night-shift workers", "The blue cabinet is restocked every Thursday at 6 a.m. by local volunteers.", "wholesome", "books|community|library", "earlybirdreads"), + ("An engineer explains why this bridge hums in the wind", "The note is produced by evenly spaced railings resonating near 440 hertz.", "science", "engineering|bridge|sound", "civilnerd"), + ("Dog refuses to leave the kayak after the trip ends", "Pepper waited for a second lap around the lake and ignored every promised snack.", "animals", "dog|kayak|lake", "paddlingpup"), + ("Hand-painted arcade cabinet celebrates classic space games", "The cabinet runs twelve restored titles and uses original 1980s-style controls.", "gaming", "arcade|retro|space", "quartermaster"), + ("Street musician turns a rain delay into a concert", "Commuters joined the chorus under platform seven while the storm passed.", "music", "concert|rain|train", "platformbeats"), + ("A fox naps on the same garden wall every afternoon", "Residents named the visitor Copper and keep a respectful distance from the wall.", "animals", "fox|garden|wildlife", "windowwatcher"), + ("Prototype bicycle folds small enough for a train locker", "Its designer says the steel frame weighs 10.8 kilograms without accessories.", "motorvehicles", "bicycle|design|commute", "foldandroll"), + ("Students launch a weather balloon with a tiny rubber duck", "The camera reached 27 kilometers before landing safely in a cornfield.", "science", "balloon|school|weather", "skyclassroom"), + ("The office plant gets an employee badge", "Kevin the pothos has perfect attendance and a surprisingly active calendar.", "humor", "office|plant|work", "meetingescape"), + ("Restored train carriage becomes a community cinema", "Volunteers kept the original green seats and installed a retractable screen.", "movies", "cinema|train|restoration", "reeljourney"), + ("Chef makes a five-course dinner using one camp stove", "The final course was a skillet peach cake shared beside the river.", "food", "cooking|camping|challenge", "oneburner"), + ("A ten-second drawing trick that changes every cartoon face", "Moving the eyebrows two millimeters completely changes the character's mood.", "comic", "drawing|tutorial|faces", "inkminute"), + ("Local team celebrates its first championship in 34 years", "The deciding goal came with 18 seconds left in the final period.", "sports", "championship|team|celebration", "finalwhistle"), + ("Fog wraps the old harbor lighthouse at sunrise", "The keeper photographed the lantern room before the morning ferry arrived.", "awesome", "lighthouse|harbor|photography", "beaconwatch"), + ("Miniature lighthouse built for a seaside bookshop", "The model stands beside a reading desk made from pine and brass.", "art", "lighthouse|books|craft", "tinyharbor"), + ("Remote lighthouse receives a compact radio desk", "A volunteer radio operator tested the new equipment during a winter storm.", "science", "lighthouse|office|radio", "signalkeeper"), + ("Solar lantern comparison for a week of camping", "Six compact lanterns were tested for charge time and nighttime brightness.", "science", "solar|camping|gear", "campbench"), + ("Rainy campsite kitchen stays dry under one clever tarp", "The angled ridgeline moved runoff away from the stove and food boxes.", "awesome", "camping|rain|diy", "drycamp"), + ("Foldable solar mat powers a photographer's trail camera", "The lightweight kit was carried between three mountain camps.", "science", "solar|camera|camping", "trailcurrent"), + ("Rescue dog recognizes the volunteer who found him", "Basil greeted the volunteer at the shelter's spring open house.", "animals", "dog|rescue|shelter", "secondchance"), + ("Shelter cat chooses the quietest reading chair", "The senior tabby now naps beside visitors during weekend adoption events.", "animals", "cat|shelter|books", "quietpaws"), + ("Foster kitten discovers the automatic treat puzzle", "Juniper solved the final compartment after several patient attempts.", "animals", "cat|rescue|treats", "fosterframes"), + ("Suspension bridge whistles only during winter gusts", "Engineers traced the sound to a narrow seam beneath the eastern walkway.", "science", "bridge|wind|sound", "spanlistener"), + ("Footbridge becomes a percussion instrument in heavy rain", "Different deck panels produce distinct notes as drops strike the surface.", "music", "bridge|sound|rain", "river rhythm"), + ("City bridge lights react to nearby music", "Microphones beneath the deck map bass notes to slow blue pulses.", "science", "bridge|music|engineering", "nightspan"), + ("Community library adds a shelf for night-bus drivers", "Volunteers stock travel books and warm drinks beside the depot.", "wholesome", "library|community|night", "lastbusbooks"), + ("Tiny red library appears outside the fire station", "Neighbors exchange mysteries, cookbooks, and children's stories each weekend.", "books", "library|community|books", "cornerreader"), + ("Hospital staff open a midnight reading cart", "The rolling library visits three floors during the overnight shift.", "wholesome", "library|night shift|volunteers", "roundsreader"), +] + + +def seed_database(db, User, Post, Comment, SavedPost, Vote): + if Post.query.count() > 0: + return + rows = CAPTURED + CURATED + [ + (f"Community remix {index}: {title.lower()}", f"A different community member adds context: {description}", section, f"{tags}|community", f"user{index:02d}") + for index, (title, description, section, tags, _author) in enumerate(CURATED, start=1) + ] + images = [row[3] for row in CAPTURED] + EXTRA_IMAGES + for index, row in enumerate(rows): + if len(row) == 7: + source_id, title, description, image, section, tags, author = row + else: + title, description, section, tags, author = row + source_id = f"seed{index:03d}" + image = images[index % len(images)] + slug = "-".join("".join(ch.lower() if ch.isalnum() else " " for ch in title).split())[:150] + db.session.add(Post(source_id=source_id, slug=f"{slug}-{index + 1}", title=title, + description=description, image=image, section=section, post_type="Photo", + tags=tags, author_name=author, up_votes=430 + ((index * 173) % 8400), + down_votes=5 + ((index * 7) % 90), comment_count=12 + ((index * 19) % 430), + created_rank=index + 1, featured=index in {2, 11, 24})) + db.session.commit() + + +def seed_benchmark_users(db, User, Post, Comment, SavedPost, Vote): + if User.query.filter_by(email="alice.j@test.com").first(): + return + people = [ + ("alice_j", "alice.j@test.com", "Alice Johnson", "Seattle, WA", "Collector of wholesome posts and clever design."), + ("bob_c", "bob.c@test.com", "Bob Chen", "Austin, TX", "Science, games, and weekend cooking."), + ("carol_d", "carol.d@test.com", "Carol Davis", "Boston, MA", "Here for animals and community stories."), + ("david_k", "david.k@test.com", "David Kim", "Portland, OR", "Cyclist, illustrator, and serial commenter."), + ] + users = [] + for username, email, display, location, bio in people: + user = User(username=username, email=email, display_name=display, location=location, bio=bio, password_hash="") + user.set_password("TestPass123!") + db.session.add(user) + users.append(user) + db.session.flush() + posts = Post.query.order_by(Post.id).all() + for user_index, user in enumerate(users): + for offset in range(4): + db.session.add(SavedPost(user_id=user.id, post_id=posts[(user_index * 7 + offset * 5) % len(posts)].id)) + for offset in range(3): + db.session.add(Vote(user_id=user.id, post_id=posts[(user_index * 9 + offset * 4 + 2) % len(posts)].id, value=1)) + comment_post = posts[(user_index * 11 + 6) % len(posts)] + db.session.add(Comment(post_id=comment_post.id, user_id=user.id, + body=f"Saved this one for later — {user.display_name}", created_at=f"{user_index + 2} days ago")) + db.session.commit() diff --git a/sites/9gag/static/css/.gitkeep b/sites/9gag/static/css/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/sites/9gag/static/css/site.css b/sites/9gag/static/css/site.css new file mode 100644 index 00000000..e27f42eb --- /dev/null +++ b/sites/9gag/static/css/site.css @@ -0,0 +1,21 @@ +:root{--blue:#087bf5;--ink:#111;--muted:#777;--line:#e4e4e4}*{box-sizing:border-box}body{margin:0;color:var(--ink);font-family:Arial,Helvetica,sans-serif;background:#fff}a{color:inherit;text-decoration:none}button,input,textarea,select{font:inherit}.topbar{height:49px;border-top:3px solid var(--blue);border-bottom:1px solid var(--line);display:flex;align-items:center;gap:17px;padding:0 max(20px,calc((100vw - 1280px)/2));position:sticky;top:0;background:#fff;z-index:10}.menu{font-size:23px}.brand{font-size:25px;font-weight:900}.toplinks{display:flex;gap:18px;font-weight:700}.searchbar{margin-left:auto;display:flex;border-bottom:1px solid #bbb}.searchbar input{width:170px;border:0;padding:8px}.searchbar button{border:0;background:#fff;font-size:20px}.muted-link{color:#777;font-weight:700;white-space:nowrap}.post-button,.primary{background:var(--blue);color:#fff;border:0;border-radius:22px;padding:10px 20px;font-weight:700}.layout{max-width:1300px;margin:18px auto;display:grid;grid-template-columns:240px minmax(430px,640px) 260px;gap:34px;padding:0 20px}.sidebar{position:sticky;top:68px;align-self:start;max-height:calc(100vh - 80px);overflow:auto}.signup-card{border:1px solid #ddd;border-radius:8px;text-align:center;padding:14px 16px;margin-bottom:6px}.signup-card h2{font-size:20px;margin:2px}.signup-card a{display:block;background:var(--blue);color:white;border-radius:22px;padding:10px;margin-top:16px;font-weight:700}.side-nav,.interests{display:flex;flex-direction:column}.side-nav a,.interests a{display:flex;align-items:center;gap:12px;padding:9px 12px;border-radius:3px}.side-nav a{font-weight:700}.side-nav a.active,.interests a.active{background:#eee}.sidebar h3{color:#888;font-size:14px;margin:18px 12px 7px}.interests b{width:24px;text-align:center}.account-links{border-top:1px solid #eee;margin-top:8px;padding-top:8px}.feed-heading{border-bottom:1px solid var(--line);margin-bottom:14px}.feed-heading h1{font-size:24px;margin:6px 0}.feed-heading p{color:var(--muted)}.post-card{border-bottom:1px solid var(--line);padding:4px 0 24px;margin-bottom:20px}.post-card h2,.post-card h1{font-size:22px;line-height:1.28;margin:8px 0 13px}.post-meta{font-size:13px;color:#777;display:flex;gap:6px}.post-meta a:first-child{font-weight:700;color:#333}.post-image{display:block;background:#f2f2f2;overflow:hidden}.post-image img,.detail-image{display:block;width:100%;max-height:620px;object-fit:contain;background:#f5f5f5}.stats{font-size:13px;color:#666;margin:10px 0}.actions{display:flex;align-items:center;gap:8px;flex-wrap:wrap}.actions form{margin:0}.actions button,.actions a{border:1px solid #ddd;background:#fff;border-radius:4px;padding:8px 11px;color:#555;font-weight:700;cursor:pointer}.actions button.selected{color:var(--blue);border-color:var(--blue)}.right-rail{padding-top:35px}.rail-card{border:1px solid #ddd;border-radius:7px;padding:16px;margin-bottom:16px}.rail-card h3{margin-top:0}.rail-card>a{display:block;padding:7px 0;color:#555}.related{border-bottom:1px solid #eee}.description{font-size:16px;line-height:1.55}.tag-row{display:flex;gap:7px;flex-wrap:wrap}.tag-row a{background:#eee;border-radius:3px;padding:6px 9px;font-size:13px}.comments{padding-bottom:50px}.comments textarea{width:100%;min-height:90px;padding:12px}.comment{border-top:1px solid #eee;padding:14px 0}.comment small{color:#888;margin-left:8px}.auth,.form-page,.profile{width:min(620px,calc(100% - 32px));margin:60px auto}.auth{width:min(420px,calc(100% - 32px));border:1px solid #ddd;border-radius:8px;padding:28px}.auth form,.form-page form{display:grid;gap:16px}.auth label,.form-page label{display:grid;gap:7px;font-weight:700}.auth input,.form-page input,.form-page textarea,.form-page select{border:1px solid #bbb;border-radius:5px;padding:11px}.profile{width:min(760px,calc(100% - 32px))}.profile-head{display:flex;gap:20px;align-items:flex-start;border-bottom:1px solid #ddd;padding-bottom:24px}.avatar{height:90px;width:90px;background:#eee;border-radius:50%;display:grid;place-items:center;font-size:38px;font-weight:800}.profile-head h1{margin:4px 0}.profile-head .outline{margin-left:auto}.outline{border:1px solid #aaa;border-radius:5px;padding:9px 14px}.profile-post{display:flex;justify-content:space-between;padding:14px 0;border-bottom:1px solid #eee}.profile-post span{color:#777}.notification,.empty{border:1px solid #ddd;border-radius:7px;padding:18px;margin:12px 0}.store-buttons{display:flex;gap:12px}.store-buttons span{background:#111;color:#fff;border-radius:8px;padding:13px}.flashes{position:fixed;top:62px;left:50%;transform:translateX(-50%);z-index:20}.flash{background:#edf8ee;border:1px solid #9bc89e;border-radius:5px;padding:10px 20px}.flash.error{background:#fff0f0;border-color:#e5a0a0}footer{border-top:1px solid #ddd;padding:30px;display:flex;justify-content:center;gap:22px;color:#777;font-size:13px} +@media(max-width:1000px){.right-rail{display:none}.layout{grid-template-columns:220px minmax(0,1fr);max-width:934px;justify-content:center}.toplinks{display:none}}@media(max-width:680px){.topbar{padding:0 10px;gap:11px}.menu{display:none}.brand{font-size:22px}.searchbar{flex:1}.searchbar input{width:100%}.muted-link{display:none}.post-button{padding:8px 12px}.layout{display:block;margin-top:10px;padding:0 12px}.sidebar{position:static;max-height:none}.signup-card,.sidebar h3,.interests,.account-links{display:none}.side-nav{flex-direction:row;overflow-x:auto;border-bottom:1px solid #ddd;margin-bottom:14px}.side-nav a{white-space:nowrap}.feed-heading{display:none}.post-card h2,.post-card h1{font-size:19px}.post-image img,.detail-image{max-height:520px}.actions{gap:5px}.actions button,.actions a{padding:7px 8px;font-size:13px}.auth,.form-page,.profile{margin:30px auto}.profile-head{flex-wrap:wrap}.profile-head .outline{margin-left:0}footer{flex-wrap:wrap}.detail .description{font-size:15px}}@media(max-width:340px){.topbar{gap:7px}.brand{font-size:20px}.post-button{font-size:13px}.layout{padding:0 8px}.side-nav a{padding:8px}.actions button,.actions a{font-size:12px}.profile-post{gap:8px}.profile-post span{white-space:nowrap}} +.post-image{display:flex;justify-content:center}.post-image img,.detail-image{width:auto;max-width:100%;height:auto;max-height:620px;object-fit:contain}.detail-image{margin-left:auto;margin-right:auto}.pagination{display:flex;justify-content:space-between;margin:24px 0 50px}.pagination a{border:1px solid #ccc;border-radius:5px;padding:10px 15px;font-weight:700}@media(max-width:680px){.post-image img,.detail-image{max-height:520px}} +@media(max-width:680px){.side-nav{display:grid;grid-template-columns:repeat(5,minmax(0,1fr));overflow:visible}.side-nav a{min-width:0;padding:7px 2px;gap:2px;justify-content:center;flex-direction:column;font-size:12px;text-align:center}} +.topbar{border-top:0;height:48px}.layout{margin-top:18px}.recent-heading{display:flex;justify-content:space-between}.recent-heading span{font-weight:400}.recents{border-bottom:1px solid #eee;padding-bottom:8px}.feed-tabs{height:38px;display:flex;gap:24px;border-bottom:1px solid #ccc;margin-bottom:24px}.feed-tabs a{padding:5px 16px 13px;color:#999;font-weight:700}.feed-tabs a:first-child{padding-left:16px}.feed-tabs a.active{color:#111;border-bottom:3px solid #111}.topic-row{display:flex;gap:10px;overflow:hidden;margin-bottom:14px}.topic-row a{white-space:nowrap;border:1px solid #ddd;color:#777;border-radius:18px;padding:6px 13px}.content-notice{height:41px;border-left:4px solid #ffa800;background:#fff2c7;display:flex;align-items:center;gap:22px;padding:0 16px;margin-bottom:14px;font-size:14px}.content-notice a{font-weight:700}.content-notice span{margin-left:auto;font-size:26px;color:#999}.right-rail{padding-top:0}.captured-ad{display:block;border:1px solid #222;padding:34px 32px 18px;position:relative;margin-bottom:16px}.captured-ad img{display:block;width:100%;height:176px;object-fit:cover}.captured-ad span{position:absolute;right:8px;top:7px;color:#888;font-size:9px;letter-spacing:.08em}.post-card{padding-top:0}.post-card h2,.post-card h1{font-size:20px}.post-meta:before{content:'✦';display:inline-grid;place-items:center;background:#f5a623;color:#fff;width:25px;height:25px;border-radius:4px;margin-right:2px}.post-meta{align-items:center}.stats{color:#999}.actions button,.actions a{border:0;padding-left:8px;padding-right:8px;color:#999;font-size:14px} +@media(max-width:680px){.feed-tabs{gap:0;justify-content:space-between;margin-bottom:12px}.feed-tabs a{padding:6px 5px 11px;font-size:12px}.feed-tabs a:nth-last-child(-n+2){display:none}.topic-row,.content-notice{display:none}.recent-heading,.recents{display:none}.post-meta:before{width:22px;height:22px}.topbar{height:46px}} +/* Reviewer layer (merge audit). Appended so it wins over the override layers above: an identical + rule placed earlier would be defeated by the later .feed-tabs / .stats / .actions redefinitions. + 1. Tablet tab strip: between 681px and 1000px the six 16px tabs need ~628px but the main column is + narrower, so the strip pushed the document to 857px at a 768px viewport. Tighten it and let the + strip itself scroll instead of the page. + 2. WCAG 1.4.3: the 9GAG greys render at 2.85:1 (#999), 3.55:1 (#888) and 4.48:1 (#777) on white, + and white on the brand blue at 4.07:1 - all under the 4.5:1 body-text floor (3:1 for the 26px + dismiss glyph). --muted is the darkest grey that still reads as grey; --action is an accessible + action colour, with the brand --blue kept for decoration. */ +@media(min-width:681px) and (max-width:1000px){.feed-tabs{gap:8px;overflow-x:auto}.feed-tabs a{padding-left:10px;padding-right:10px}} +:root{--muted:#767676;--action:#0a74e6} +.muted-link,.post-meta,.profile-post span,footer,.feed-heading p,.topic-row a,.sidebar h3,.comment small,.stats,.captured-ad span,.feed-tabs a,.content-notice span{color:var(--muted)} +.post-button,.primary,.signup-card a{background:var(--action)} +.actions button,.actions a{color:var(--muted)} +.actions button.selected{color:var(--action);border-color:var(--action)} diff --git a/sites/9gag/static/icons/.gitkeep b/sites/9gag/static/icons/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/sites/9gag/static/js/.gitkeep b/sites/9gag/static/js/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/sites/9gag/tasks.jsonl b/sites/9gag/tasks.jsonl new file mode 100644 index 00000000..5b0b5ab2 --- /dev/null +++ b/sites/9gag/tasks.jsonl @@ -0,0 +1,20 @@ +{"web_name":"9GAG","id":"9GAG--0","ques":"Search for posts about lighthouse offices. Open the post whose title mentions the best ocean view and report the room's width and the material used for the desk.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_0.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST run a site search (a /search?q= results page) with a lighthouse/office related query. (2) The agent MUST open the detail page (/gag/...) of the post whose title mentions the best ocean view; the width and material are only on the detail page, so a results-page-only run is a FAIL. (3) The final answer MUST state the room width as a number with a unit AND the desk material, both exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--1","ques":"Search for rescue cat posts. Open the post about a cat learning the treat-drawer sound and report the month Miso arrived at Harbor Paws and how many days the routine took to learn.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_1.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST run a site search for rescue-cat posts. (2) The agent MUST open the detail page of the post about a cat learning the treat-drawer sound (not the shelter-cat or foster-kitten posts). (3) The final answer MUST give the month the cat arrived AND the number of days the routine took, both exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--2","ques":"Search for solar camping posts and open the rainy-weekend setup. What wattage was its folding panel, and what protected the controller from water?","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_2.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST run a site search for solar camping posts. (2) The agent MUST open the detail page of the rainy-weekend solar setup (not the solar-lantern comparison or the solar-mat post). (3) The final answer MUST give the panel wattage AND the object that protected the controller, both exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--3","ques":"Browse Science & Tech and find the post explaining a bridge that hums in the wind. Report the frequency of the note and the bridge feature that produces it.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_3.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST open the Science & Tech interest feed (/interest/science; it spans more than one page). (2) The agent MUST open the detail page of the post explaining why a bridge hums in the wind (not the whistling suspension bridge, the percussion footbridge or the bridge-lights posts). (3) The final answer MUST give the frequency of the note AND the bridge feature that produces it, exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--4","ques":"Search the community library posts and open the one created for night-shift workers. What color is the cabinet, and exactly when is it restocked?","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_4.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST run a site search for community library posts. (2) The agent MUST open the detail page of the library created for night-shift workers (not the night-bus-driver shelf, the fire-station library or the hospital reading cart). (3) The final answer MUST give the cabinet colour AND the restock day and time, exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--5","ques":"Browse Sports and open the post about a grandmother finishing her first marathon. At which kilometer did her family meet her, and what color were their flags?","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_5.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST open the Sports interest feed (/interest/sports). (2) The agent MUST open the detail page of the grandmother-marathon post. (3) The final answer MUST give the kilometre where her family met her AND the colour of their flags, exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--6","ques":"Search for sourdough skyline posts. Open the post about the baker who recreated a city skyline and report both the number of attempts and the approximate shaping time.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_6.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST run a site search for the sourdough skyline post. (2) The agent MUST open the detail page of the sourdough-skyline post; its 'Community remix' copy repeats the identical description, so either detail page satisfies the task. (3) The final answer MUST give the number of attempts AND the approximate shaping time, exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--7","ques":"Browse Gaming and find the transparent mechanical keyboard. What kind of switches and what case material does it use?","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_7.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST open the Gaming interest feed (/interest/gaming). (2) The agent MUST open the detail page of the transparent mechanical keyboard post. (3) The final answer MUST give the switch type AND the case material, exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--8","ques":"Search for the rain-delay concert, open the street-musician post, and report the platform number where commuters joined the chorus.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_8.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST run a site search for the rain-delay concert. (2) The agent MUST open the detail page of the street-musician post; its 'Community remix' copy repeats the identical description, so either detail page counts. (3) The final answer MUST give the platform number exactly as written in that post's description. (4) An empty answer, a value that is not on that page, or a correct-looking answer given without opening the detail page (prior-knowledge shortcut) is a FAIL. (5) Read-only task: any vote, save, comment, hide, report, post or profile write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--9","ques":"Browse Animals & Pets and compare the original rescue-cat, kayaking-dog, and garden-fox posts (not their 'Community remix' copies, which carry different point counts). Which has the most points? Open that post and report the animal's name.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_9.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST open the Animals & Pets interest feed (/interest/animals). (2) The agent MUST compare the point counts of the three ORIGINAL posts named in the task (rescue cat, kayaking dog, garden fox); the 'Community remix' copies have different points and must not be used. (3) The agent MUST open the detail page of whichever of those three has the most points. (4) The final answer MUST name that post's animal exactly as written in its description. (5) An empty answer, the wrong animal, or no detail-page visit is a FAIL. (6) Read-only task: any write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--10","ques":"Log in with alice.j@test.com and TestPass123!, search for the original tiny lighthouse office post with the best ocean view (not its 'Community remix' copy), open it, and save it to your collection.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_10.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in on /login as the account given in the task. (2) The agent MUST reach the post through a site search and open the detail page of the ORIGINAL post titled exactly 'Tiny lighthouse office with the best ocean view' (not the 'Community remix' copy). (3) The agent MUST press Save on that post: the page must show it as saved and the after-state database must contain exactly one new saved_post row for that account and that post, with no other rows changed. (4) A success claim without the saved row, saving the wrong post, or using another account is a FAIL."} +{"web_name":"9GAG","id":"9GAG--11","ques":"Log in with carol.d@test.com and TestPass123!, browse Animals & Pets, open the original post about the dog refusing to leave a kayak (not its 'Community remix' copy), and upvote it.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_11.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open the Animals & Pets interest feed and then the detail page of the ORIGINAL kayak-dog post (not the remix copy). (3) The agent MUST upvote it: exactly one new vote row (value +1) for that account and post, the post's points increased by exactly 1, and no other writes. (4) A downvote, a vote on another post, or no persisted vote is a FAIL."} +{"web_name":"9GAG","id":"9GAG--12","ques":"Log in with bob.c@test.com and TestPass123!, find the original bridge-humming explanation in Science & Tech (not its 'Community remix' copy), and comment exactly: Resonance makes ordinary structures fascinating.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_12.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open the Science & Tech interest feed and the detail page of the ORIGINAL humming-bridge post (not the remix copy). (3) The agent MUST post a comment whose text matches the sentence given in the task (capitalisation and the final period are tolerated, wording is not): exactly one new comment row for that account and post, the post's comment count increased by 1, and no other writes. (4) A different sentence, a comment on another post, two comments, or no persisted comment is a FAIL."} +{"web_name":"9GAG","id":"9GAG--13","ques":"Log in with carol.d@test.com and TestPass123!, search for the office plant with an employee badge, open the original post (not its 'Community remix' copy), and hide it from your feeds.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_13.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST run a site search and open the detail page of the ORIGINAL office-plant post (not the remix copy). (3) The agent MUST press Hide: exactly one new hidden_post row for that account and post, and no other writes. (4) Hiding the remix, hiding nothing, or any collateral write is a FAIL."} +{"web_name":"9GAG","id":"9GAG--14","ques":"Log in with david.k@test.com and TestPass123!, open News, find the mysterious 2009 press-conference post, and report it for Misinformation.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_14.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open the News feed (it has two pages), then the detail page of the 2009 press-conference post, then its Report form. (3) The agent MUST submit the report with the reason given in the task: exactly one new report row for that account, post and reason, and no other writes. (4) A different reason, another post, or no persisted report is a FAIL."} +{"web_name":"9GAG","id":"9GAG--15","ques":"Log in with alice.j@test.com and TestPass123! and create a post titled 'Quiet victories deserve confetti' in Wholesome. Use description 'My neighbor finished night school after six years.' and tags 'community|education|wholesome', then publish it.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_15.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open the Create-a-post form (/submit) and publish; the run must end on the new post's page. (3) Exactly one new post row must exist with the exact title, description, interest and tags given in the task, authored by that account; no other rows may change. (4) A wrong interest, tags or description, two posts, or no persisted post is a FAIL."} +{"web_name":"9GAG","id":"9GAG--16","ques":"Log in with alice.j@test.com and TestPass123! and update the profile: display name 'Alice J.', bio 'Memes, trail photos, and excellent tiny libraries.', and location 'Tacoma, WA'. Save the changes.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_16.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open Settings (/settings) and save. (3) The after-state database must show that account's display name, bio and location exactly as given in the task, with every other column and every other user unchanged and no other writes. (4) A partial update, a change to another user, or unsaved changes is a FAIL."} +{"web_name":"9GAG","id":"9GAG--17","ques":"Create a new account with email river.reader@test.com, username river_reader, and password RiverRead123!. Confirm that the signed-in username appears in the header.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_17.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST open the registration form (/register) and create the account with the exact email, username and password from the task; the password must verify against the stored hash. (2) Exactly one new user row may exist; the four seeded accounts and every other table must be unchanged. (3) The final answer MUST confirm the username shown in the header after sign-in. (4) A different username, a wrong password, or no persisted account is a FAIL."} +{"web_name":"9GAG","id":"9GAG--18","ques":"Log in with alice.j@test.com and TestPass123!, open Saved, locate the original solar-powered rainy-weekend camping setup (not its 'Community remix' copy), and remove it from the saved collection.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_18.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open the Saved collection (/saved). (3) The agent MUST remove the ORIGINAL solar-powered rainy-weekend camping post: exactly one saved_post row (that account, that post) removed, none added, and no other writes. (4) Removing a different saved post, removing two, or no change is a FAIL."} +{"web_name":"9GAG","id":"9GAG--19","ques":"Log in with david.k@test.com and TestPass123!. In Science & Tech, compare the original solar camping setup, humming bridge, and weather-balloon posts (not their 'Community remix' copies) by the largest number stated in each detail description. Save the post whose stated number is greatest.","web":"http://localhost:40026/","upstream_url":"https://9gag.com/","verifier_path":"sites/9gag/verify/verify_19.py","judge_rubric":"FACT CHECKPOINTS (each MUST is a hard requirement; any unmet MUST = FAIL): (1) The agent MUST log in as the account given in the task. (2) The agent MUST open the Science & Tech interest feed and the detail pages of ALL THREE original posts named in the task, because the numbers to compare appear only in the detail descriptions. (3) The agent MUST press Save on the post whose stated number is the greatest: exactly one new saved_post row for that account and that post, and no other writes. (4) Saving another of the three, saving two, skipping a detail page, or no persisted save is a FAIL."} diff --git a/sites/9gag/templates/.gitkeep b/sites/9gag/templates/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/sites/9gag/templates/_post_card.html b/sites/9gag/templates/_post_card.html new file mode 100644 index 00000000..19abd733 --- /dev/null +++ b/sites/9gag/templates/_post_card.html @@ -0,0 +1 @@ + diff --git a/sites/9gag/templates/_sidebar.html b/sites/9gag/templates/_sidebar.html new file mode 100644 index 00000000..fed27c54 --- /dev/null +++ b/sites/9gag/templates/_sidebar.html @@ -0,0 +1,24 @@ + diff --git a/sites/9gag/templates/account.html b/sites/9gag/templates/account.html new file mode 100644 index 00000000..5a0d1b8d --- /dev/null +++ b/sites/9gag/templates/account.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}{{ profile.display_name }} - 9GAG{% endblock %}{% block body %}
{{ profile.display_name[0] }}

{{ profile.display_name }}

@{{ profile.username }}

{{ profile.bio }}

{{ profile.location }} · Joined {{ profile.joined_at }}
{% if current_user.is_authenticated and current_user.id==profile.id %}Edit profile{% endif %}

Posts

{% if posts %}{% for post in posts %}{{ post.title }} {{ post.up_votes }} points{% endfor %}{% else %}

No posts yet.

{% endif %}
{% endblock %} diff --git a/sites/9gag/templates/apps.html b/sites/9gag/templates/apps.html new file mode 100644 index 00000000..b96d2c1f --- /dev/null +++ b/sites/9gag/templates/apps.html @@ -0,0 +1 @@ +{% extends 'static_page.html' %}{% set heading='Get the 9GAG app' %}{% set body='Carry the community with you and browse interests, vote, comment, and save posts from your phone.' %}{% block page_content %}
Download on the App StoreGet it on Google Play
{% endblock %} diff --git a/sites/9gag/templates/base.html b/sites/9gag/templates/base.html new file mode 100644 index 00000000..ddcc877f --- /dev/null +++ b/sites/9gag/templates/base.html @@ -0,0 +1,3 @@ +{% block title %}9GAG - Best Funny Memes and Breaking News{% endblock %} +
☰9GAG
{% if current_user.is_authenticated %}{{ current_user.username }}{% else %}Sign Up/Log In{% endif %}✎ Post
+{% with messages=get_flashed_messages(with_categories=true) %}{% if messages %}
{% for kind,message in messages %}
{{ message }}
{% endfor %}
{% endif %}{% endwith %}{% block body %}{% endblock %} diff --git a/sites/9gag/templates/error.html b/sites/9gag/templates/error.html new file mode 100644 index 00000000..2f26c044 --- /dev/null +++ b/sites/9gag/templates/error.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}{{ heading }} - 9GAG{% endblock %}{% block body %}

{{ heading }}

{{ body }}

Back to 9GAG

{% endblock %} diff --git a/sites/9gag/templates/feed.html b/sites/9gag/templates/feed.html new file mode 100644 index 00000000..16dd3015 --- /dev/null +++ b/sites/9gag/templates/feed.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}{{ title }} - 9GAG{% endblock %}{% block body %}
{% include '_sidebar.html' %}
{% for key,label in [('humor','humor'),('gaming','gaming'),('animals','animals & pets'),('science','science & tech'),('sports','sports'),('wholesome','wholesome')] %}{{label}}{% endfor %}
Want fewer political content? ⊘   Block tags×
{% if query %}

{{ title }}

Posts related to “{{ query }}”

{% endif %}{% if posts %}{% for post in posts %}{% include '_post_card.html' %}{% endfor %}{% else %}

No posts found

Try another search or explore an interest.

{% endif %}
{% endblock %} diff --git a/sites/9gag/templates/help.html b/sites/9gag/templates/help.html new file mode 100644 index 00000000..41d4a5bc --- /dev/null +++ b/sites/9gag/templates/help.html @@ -0,0 +1 @@ +{% extends 'static_page.html' %}{% set heading='Help center' %}{% set body='Find answers about accounts, posting, voting, saved posts, and community safety.' %}{% block page_content %}

Popular topics

  • Managing your profile
  • Posting and choosing an interest
  • Reporting content
{% endblock %} diff --git a/sites/9gag/templates/index.html b/sites/9gag/templates/index.html new file mode 100644 index 00000000..0c02ac31 --- /dev/null +++ b/sites/9gag/templates/index.html @@ -0,0 +1 @@ +{% extends 'feed.html' %} diff --git a/sites/9gag/templates/login.html b/sites/9gag/templates/login.html new file mode 100644 index 00000000..3d38243f --- /dev/null +++ b/sites/9gag/templates/login.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}Log in - 9GAG{% endblock %}{% block body %}

Log in

Welcome back to the fun side.

New here? Create an account

{% endblock %} diff --git a/sites/9gag/templates/notifications.html b/sites/9gag/templates/notifications.html new file mode 100644 index 00000000..f0a9c3e9 --- /dev/null +++ b/sites/9gag/templates/notifications.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}Notifications - 9GAG{% endblock %}{% block body %}

Notifications

Your saved collection is ready

Everything you save stays available from the sidebar.

Welcome back, {{ current_user.display_name }}

Fresh posts were added to Science & Tech and Animals & Pets.

{% endblock %} diff --git a/sites/9gag/templates/post_detail.html b/sites/9gag/templates/post_detail.html new file mode 100644 index 00000000..02d11262 --- /dev/null +++ b/sites/9gag/templates/post_detail.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}{{ post.title }} - 9GAG{% endblock %}{% block body %}
{% set active=post.section %}{% include '_sidebar.html' %}
{{ post.section|title }}•by @{{ post.author_name }}

{{ post.title }}

{{ post.title }}

{{ post.description }}

{% for tag in post.tag_list %}#{{ tag }}{% endfor %}
{{ post.up_votes }} points · {{ post.down_votes }} downvotes · {{ post.comment_count }} comments
Report

Comments

{% if current_user.is_authenticated %}
{% else %}

Log in to comment.

{% endif %}{% for comment in post.comments %}
{{ comment.user.display_name }}{{ comment.created_at }}

{{ comment.body }}

{% endfor %}
{% endblock %} diff --git a/sites/9gag/templates/register.html b/sites/9gag/templates/register.html new file mode 100644 index 00000000..5aa108b9 --- /dev/null +++ b/sites/9gag/templates/register.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}Sign up - 9GAG{% endblock %}{% block body %}

Join 9GAG

Already a member? Log in

{% endblock %} diff --git a/sites/9gag/templates/report.html b/sites/9gag/templates/report.html new file mode 100644 index 00000000..151b9508 --- /dev/null +++ b/sites/9gag/templates/report.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}Report post - 9GAG{% endblock %}{% block body %}

Report post

{{ post.title }}

{% endblock %} diff --git a/sites/9gag/templates/settings.html b/sites/9gag/templates/settings.html new file mode 100644 index 00000000..fe8472bb --- /dev/null +++ b/sites/9gag/templates/settings.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}Settings - 9GAG{% endblock %}{% block body %}

Edit profile

{% endblock %} diff --git a/sites/9gag/templates/static_page.html b/sites/9gag/templates/static_page.html new file mode 100644 index 00000000..a16e64bd --- /dev/null +++ b/sites/9gag/templates/static_page.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}{{ heading }} - 9GAG{% endblock %}{% block body %}

{{ heading }}

{{ body }}

{% block page_content %}{% endblock %}
{% endblock %} diff --git a/sites/9gag/templates/submit.html b/sites/9gag/templates/submit.html new file mode 100644 index 00000000..e84e3f56 --- /dev/null +++ b/sites/9gag/templates/submit.html @@ -0,0 +1 @@ +{% extends 'base.html' %}{% block title %}Create a post - 9GAG{% endblock %}{% block body %}

Create a post

Share something the community will enjoy.

{% endblock %} diff --git a/sites/9gag/templates/terms.html b/sites/9gag/templates/terms.html new file mode 100644 index 00000000..02f04321 --- /dev/null +++ b/sites/9gag/templates/terms.html @@ -0,0 +1 @@ +{% extends 'static_page.html' %}{% set heading='Terms and privacy' %}{% set body='Use the service responsibly and respect the rights and privacy of other community members.' %} diff --git a/sites/9gag/verify/README.md b/sites/9gag/verify/README.md new file mode 100644 index 00000000..dc5dd62d --- /dev/null +++ b/sites/9gag/verify/README.md @@ -0,0 +1,105 @@ +# 9GAG deterministic grading contract + +Each row in `sites/9gag/tasks.jsonl` points to `verify_0.py` … `verify_19.py`. The wrappers share +`verify_lib.py` (same API family as `sites/merriam_webster/verify/verify_lib.py`, hardened the way +`sites/walmart_careers/verify/verify_lib.py` is). Ground truth is hardcoded inside each `verify_N.py` +and never appears in `tasks.jsonl`. Every verdict is decided deterministically; the anchored LLM helpers +(`llm_text_match`, `llm_screenshot_shows`) are kept for parity and only add `[INFO]` evidence when an +endpoint is configured — they never change PASS/FAIL, and `--no_llm True` silences them. + +## Inputs + +```bash +uv run python sites/9gag/verify/verify_0.py \ + --run_dir /abs/path/to/run \ + --initial_db /abs/path/to/initial.db \ + --after_db /abs/path/to/after.db \ + --no_llm True +``` + +If explicit snapshots are omitted the verifier looks for `/initial.db` and `/after.db`, +then falls back to `docker cp` from `$WH_CONTAINER` (default `wh-review`) at +`/opt/WebSyn/9gag/instance_seed/9gag.db` (initial) and `/opt/WebSyn/9gag/instance/9gag.db` (after). +Missing or invalid inputs fail closed (`infra_error: true`). Output is JSON `{task_id, pass, reason, evidence}`; +exit 0 = PASS, 1 = FAIL. `agent_demo/eval_judge.py --run_dir --verifier True` invokes the same script. +Boolean flags take a value (`--no_llm True`), as in the other suites. + +## Package validation (every task) + +- exact `task_id`; `terminated: true` with `termination_reason: agent_done`; at least one step; +- a non-empty `final_answer`; +- every recorded URL (`start_url`, step `url`s, `final_url`) is `http://` on a loopback host **and the same + port as `start_url`** (alt-port containers are fine; mixed origins are not); +- both screenshots referenced by every step exist and decode as non-empty PNGs. + +## Snapshot contract + +Both snapshots must have exactly the seven 9GAG tables (`user, post, comment, vote, saved_post, hidden_post, +report`) with the pinned schema hash (`SCHEMA_SHA256`) and identical schemas. The initial snapshot must carry +the frozen seed counts (80 posts, 4 users, 4 comments, 12 votes, 16 saved posts, 0 hidden, 0 reports) and the +four benchmark accounts. Across snapshots the four accounts keep `id/username/email`, and the 80 seed posts +keep every catalog column (`slug, title, description, image, section, post_type, tags, author_name, +created_rank, featured`); only the counters (`up_votes, down_votes, comment_count`) may move, and only where +a task allows it. Any drift fails closed with `snapshot_contract_invalid`. + +## Per-task contract + +| Task | Navigation gates (all required) | Answer / state check | +|---|---|---| +| 0 | `/search?q=` with a lighthouse/office/ocean token; detail of the best-ocean-view post (original **or** its remix clone — same facts) | width `4.2` (decimal-exact) + `reclaimed` + `oak`; DB unchanged | +| 1 | search (rescue/cat/treat/drawer/miso); detail of the treat-drawer cat post (original or clone) | `February` + count `11`/`eleven`; DB unchanged | +| 2 | search (solar/camping/camp/rainy/panel); detail of the rainy-weekend solar setup (original or clone) | count `120` + phrase `lunch box` (`lunchbox`/`lunch-box` ok); DB unchanged | +| 3 | `/interest/science`; detail of the humming-bridge post (original or clone) | count `440` + `railing(s)`; DB unchanged | +| 4 | search (library/community/night/shift/cabinet); detail of the night-shift library (original or clone) | `blue` + `Thursday` + clock `6 am` (`6:00 AM`, `06:00`, `6 a.m.` ok); DB unchanged | +| 5 | `/interest/sports`; detail of the grandmother-marathon post (original or clone) | count `38` + `orange`; DB unchanged | +| 6 | search (sourdough/skyline/baker/bread); detail of the sourdough post (original **or** its remix clone — same facts) | count `3`/`three` + count `14`/`fourteen`; DB unchanged | +| 7 | `/interest/gaming`; detail of the transparent keyboard (original or clone) | `silent` + `tactile` + `acrylic`; DB unchanged | +| 8 | search (rain/delay/concert/musician/street/platform/chorus); detail of the street-musician post (original **or** its remix clone — same facts) | count `7`/`seven`; DB unchanged | +| 9 | `/interest/animals`; detail of the garden-fox post (the original with the most points; frozen points re-checked against the seed) | `Copper`; DB unchanged | +| 10 | login as alice (`/login` + identity typed); search (lighthouse/office/ocean); detail of the **original** lighthouse office; `/login` before the detail | exactly one added `saved_post` (alice, post 12); nothing else changes | +| 11 | login as carol; `/interest/animals`; detail of the **original** kayak-dog post; order login → feed → detail | exactly one added `vote` (carol, post 20, +1); post 20 `up_votes` +1 only; nothing else changes | +| 12 | login as bob; `/interest/science`; detail of the **original** humming-bridge post; order login → feed → detail | exactly one added `comment` (bob, post 19) whose text equals the task sentence (case and trailing period tolerated); post 19 `comment_count` +1; nothing else changes | +| 13 | login as carol; search (office/plant/badge/employee/kevin/pothos); detail of the **original** office-plant post; login before detail | exactly one added `hidden_post` (carol, post 26); nothing else changes | +| 14 | login as david; `/news`; detail of the 2009 press-conference post; its `/report` form; order login → news → detail → report | exactly one added `report` (david, post 6, `Misinformation`); nothing else changes | +| 15 | login as alice; `/submit`; the published page `/gag/quiet-victories-deserve-confetti`; order login → submit → post | exactly one added `post`: exact title, `wholesome`, exact description, tags `community|education|wholesome`, author `alice_j`; nothing else changes | +| 16 | login as alice; `/settings`; login before settings | alice's `display_name`/`bio`/`location` equal the task values; her `id/username/email/password_hash/joined_at` unchanged; no other user or table changes | +| 17 | `/register` | exactly one added `user` with the exact email + username whose stored hash verifies the task password; seeded users untouched; answer mentions `river_reader` | +| 18 | login as alice; `/saved`; login before saved | exactly one removed `saved_post` (alice, post 11); nothing else changes | +| 19 | login as david; `/interest/science`; detail pages of **all three** compared originals (11, 19, 25); login before browse | exactly one added `saved_post` (david, post 19 — the 440 hertz post); nothing else changes | + +Precondition checks (`initial_state_requires_action`) make sure the seed does not already contain the +target state, so a stale-state no-op can never pass. Answer matchers are affirmative-only (a negated +mention such as "not orange" does not count), counts must be standalone integers or number words, and +decimals must match exactly. + +Note on the "Community remix" clones: the seed duplicates every curated post as `Community remix N: …` +with the same description. The rule is **provenance only matters where it changes the graded outcome**: + +- **Read-only fact tasks (0-8): either detail page is accepted**, because the clone repeats the original's + description verbatim, so an agent that reads the fact off the clone has answered the question correctly. + Requiring the original there would fail a correct answer on a hidden provenance rule. +- **Task 9 requires the original**, because it compares *point counts* and the clones carry different ones, + so using a clone changes the answer. +- **Stateful tasks (10-13, 18, 19) require the original**, because the clone is a genuinely different row + and saving/voting/hiding/commenting on it is a different database effect. + +Every task whose grade depends on provenance now says so in its `ques` ("not its 'Community remix' copy"), +so the requirement is visible to the agent instead of hidden in the verifier. + +## Tests + +```bash +# unit matrix (synthetic trajectories over copies of the frozen seed; no site, no docker, no LLM) +cd agent_demo && uv run python -m unittest discover -s ../sites/9gag/verify/tests -p 'test_*.py' + +# live matrix (Playwright drives every task against a running mirror; CONTRIBUTING §C evidence) +cd agent_demo && uv run python ../sites/9gag/verify/tests/live_matrix.py --base http://127.0.0.1:45001 \ + --site_dir ../sites/9gag --out runs/9gag_matrix --reset_cmd "" +``` + +The unit tests need `sites/9gag/instance_seed/9gag.db` (an HF asset; `scripts/fetch_assets.sh 9gag`) and skip +otherwise. They cover: genuine PASS, run-dir snapshot discovery, no-op, shortcut (correct answer without +navigation), wrong answers per fact, wrong task id, unterminated runs, mixed origins, corrupt screenshots, +read-only writes, schema/catalog tampering, stale-state preconditions, wrong target/account/reason/text, +duplicate actions, counter-bump consistency and collateral writes. `verify/tests/` is excluded from the +image by `.dockerignore`. diff --git a/sites/9gag/verify/tests/_support.py b/sites/9gag/verify/tests/_support.py new file mode 100644 index 00000000..cf1402ad --- /dev/null +++ b/sites/9gag/verify/tests/_support.py @@ -0,0 +1,295 @@ +"""Shared fixtures for the 9gag verifier tests. + +Snapshots are copies of the real frozen seed (``instance_seed/9gag.db``, an HF asset) +with mutations applied through sqlite, and trajectories are hand-written in the +``agent_demo/agent.py`` shape (step ``url`` = page before the action). No docker, no LLM. + +Run from the agent_demo env so ``simpleArgParser`` (and Pillow) are importable: + + cd agent_demo && uv run python -m unittest discover -s ../sites/9gag/verify/tests -p 'test_*.py' +""" +from __future__ import annotations + +import base64 +import hashlib +import json +import secrets +import shutil +import sqlite3 +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path +from typing import Any, Callable + +VERIFY_DIR = Path(__file__).resolve().parents[1] +SITE_DIR = VERIFY_DIR.parent +SEED_DB = SITE_DIR / "instance_seed" / "9gag.db" +BASE = "http://localhost:41024" +PASSWORD = "TestPass123!" +USERS = { # name -> (id, email, username) + "alice": (1, "alice.j@test.com", "alice_j"), + "bob": (2, "bob.c@test.com", "bob_c"), + "carol": (3, "carol.d@test.com", "carol_d"), + "david": (4, "david.k@test.com", "david_k"), +} +PNG_1X1 = base64.b64decode( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=" +) + +if not SEED_DB.is_file(): + raise unittest.SkipTest( + f"{SEED_DB} is missing: fetch the 9gag asset bundle first (scripts/fetch_assets.sh 9gag)" + ) + + +def werkzeug_scrypt_hash(password: str) -> str: + """Same format app.py stores (werkzeug generate_password_hash default).""" + salt = secrets.token_hex(8) + digest = hashlib.scrypt(password.encode(), salt=salt.encode(), n=32768, r=8, p=1, maxmem=132 * 32768 * 8).hex() + return f"scrypt:32768:8:1${salt}${digest}" + + +class State: + """Mutations applied on top of a copy of the frozen seed.""" + + def __init__(self) -> None: + self.ops: list[Callable[[sqlite3.Connection], None]] = [] + + # -- mutators ----------------------------------------------------------- + def add_saved(self, user_id: int, post_id: int) -> "State": + self.ops.append(lambda c: c.execute("INSERT INTO saved_post(user_id, post_id) VALUES (?, ?)", (user_id, post_id))) + return self + + def remove_saved(self, user_id: int, post_id: int) -> "State": + def op(c: sqlite3.Connection) -> None: + assert c.execute("DELETE FROM saved_post WHERE user_id=? AND post_id=?", (user_id, post_id)).rowcount == 1 + self.ops.append(op) + return self + + def add_vote(self, user_id: int, post_id: int, value: int = 1, bump: bool = True) -> "State": + def op(c: sqlite3.Connection) -> None: + c.execute("INSERT INTO vote(user_id, post_id, value) VALUES (?, ?, ?)", (user_id, post_id, value)) + if bump: + col = "up_votes" if value == 1 else "down_votes" + c.execute(f"UPDATE post SET {col} = {col} + 1 WHERE id=?", (post_id,)) + self.ops.append(op) + return self + + def add_comment(self, user_id: int, post_id: int, body: str, bump: bool = True) -> "State": + def op(c: sqlite3.Connection) -> None: + c.execute("INSERT INTO comment(post_id, user_id, body, created_at) VALUES (?, ?, ?, 'just now')", (post_id, user_id, body)) + if bump: + c.execute("UPDATE post SET comment_count = comment_count + 1 WHERE id=?", (post_id,)) + self.ops.append(op) + return self + + def add_hidden(self, user_id: int, post_id: int) -> "State": + self.ops.append(lambda c: c.execute("INSERT INTO hidden_post(user_id, post_id) VALUES (?, ?)", (user_id, post_id))) + return self + + def add_report(self, user_id: int, post_id: int, reason: str) -> "State": + self.ops.append(lambda c: c.execute("INSERT INTO report(user_id, post_id, reason) VALUES (?, ?, ?)", (user_id, post_id, reason))) + return self + + def add_post(self, title: str, description: str, section: str, tags: str, author: str, + slug: str | None = None, image: str = "posts/aVvGONd-41d662a3.jpg") -> "State": + def op(c: sqlite3.Connection) -> None: + count = c.execute("SELECT count(*) FROM post").fetchone()[0] + c.execute( + "INSERT INTO post(source_id, slug, title, description, image, section, post_type, tags, author_name, " + "up_votes, down_votes, comment_count, created_rank, featured) VALUES (?, ?, ?, ?, ?, ?, 'Photo', ?, ?, 0, 0, 0, ?, 0)", + (f"local-{count + 1}", slug or title.lower().replace(" ", "-"), title, description, image, section, tags, author, count + 100), + ) + self.ops.append(op) + return self + + def set_profile(self, user_id: int, **fields: Any) -> "State": + def op(c: sqlite3.Connection) -> None: + assignments = ", ".join(f"{k} = ?" for k in fields) + c.execute(f"UPDATE user SET {assignments} WHERE id = ?", (*fields.values(), user_id)) + self.ops.append(op) + return self + + def add_user(self, email: str, username: str, password: str, display_name: str | None = None) -> "State": + def op(c: sqlite3.Connection) -> None: + c.execute( + "INSERT INTO user(username, email, display_name, password_hash, bio, location, joined_at) " + "VALUES (?, ?, ?, ?, '', '', 'September 2026')", + (username, email, display_name or username, werkzeug_scrypt_hash(password)), + ) + self.ops.append(op) + return self + + def sql(self, statement: str) -> "State": + self.ops.append(lambda c: c.execute(statement)) + return self + + # -- persistence -------------------------------------------------------- + def write(self, path: Path) -> Path: + shutil.copy2(SEED_DB, path) + con = sqlite3.connect(path) + try: + for op in self.ops: + op(con) + con.commit() + finally: + con.close() + return path + + +def step(path: str, action: str = "click", text: str | None = None) -> dict[str, Any]: + """One trajectory step in the agent.py shape; ``path`` is relative to BASE.""" + params: dict[str, Any] = {"text": text} if text is not None else {} + url = path if path.startswith("http") else f"{BASE}{path}" + return {"url": url, "action": action, "params": params} + + +def login_steps(email: str, landing: str = "/home") -> list[dict[str, Any]]: + return [ + step("/", "click"), + step("/login", "input", email), + step("/login", "input", PASSWORD), + step("/login", "click"), + step(landing, "click"), + ] + + +def write_run(run_dir: Path, task_id: str, steps: list[dict[str, Any]], answer: str, + success: bool | None = None) -> None: + run_dir.mkdir(parents=True, exist_ok=True) + shots = run_dir / "screenshots" + shots.mkdir(exist_ok=True) + numbered = [] + for index, item in enumerate(steps): + before, after = f"step_{index:03d}.png", f"step_{index + 1:03d}.png" + (shots / before).write_bytes(PNG_1X1) + (shots / after).write_bytes(PNG_1X1) + numbered.append({"step": index, "title": "9GAG", "thought": "", **item, + "screenshot_before": before, "screenshot_after": after}) + trajectory = { + "task": "synthetic", "task_id": task_id, "start_url": f"{BASE}/", "model": "unit-test", "max_steps": 30, + "steps": numbered, "terminated": bool(answer), "termination_reason": "agent_done" if answer else "max_steps", + "final_url": numbered[-1]["url"] if numbered else f"{BASE}/", + "final_answer": answer if answer else None, + "success_self_report": bool(answer) if success is None else success, + "judge_rubric": "", "verifier_path": "", + } + (run_dir / "trajectory.json").write_text(json.dumps(trajectory, indent=2), encoding="utf-8") + + +class VerifierTestCase(unittest.TestCase): + """Base class: ``self.N`` selects verify_N.py.""" + + N = -1 + + @property + def task_id(self) -> str: + return f"9GAG--{self.N}" + + def verdict(self, steps, answer, initial=None, after=None, task_id=None, snapshots_in_run_dir=False, + trajectory_updates=None, corrupt_screenshot=False, no_llm=True): + initial = initial or State() + after = after or State() + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + run_dir = root / "run" + write_run(run_dir, task_id or self.task_id, steps, answer) + if trajectory_updates: + path = run_dir / "trajectory.json" + data = json.loads(path.read_text()) + data.update(trajectory_updates) + path.write_text(json.dumps(data, indent=2)) + if corrupt_screenshot: + next((run_dir / "screenshots").glob("*.png")).write_bytes(b"not a png") + command = [sys.executable, str(VERIFY_DIR / f"verify_{self.N}.py"), "--run_dir", str(run_dir)] + if snapshots_in_run_dir: + initial.write(run_dir / "initial.db") + after.write(run_dir / "after.db") + else: + command += ["--initial_db", str(initial.write(root / "initial.db")), + "--after_db", str(after.write(root / "after.db"))] + if no_llm: + command += ["--no_llm", "True"] + result = subprocess.run(command, capture_output=True, text=True) + self.assertTrue(result.stdout.strip(), f"verifier printed nothing; stderr={result.stderr}") + verdict = json.loads(result.stdout) + verdict["returncode"] = result.returncode + return verdict + + def assertPasses(self, verdict) -> None: + self.assertTrue(verdict["pass"], verdict["evidence"]) + self.assertEqual(verdict["returncode"], 0) + self.assertEqual(verdict["reason"], "all checks passed") + + def assertFailsOn(self, verdict, reason: str) -> None: + self.assertFalse(verdict["pass"], verdict["evidence"]) + self.assertEqual(verdict["returncode"], 1) + self.assertEqual(verdict["reason"], reason, verdict["evidence"]) + + +class ReadTaskTests(VerifierTestCase): + """Shared matrix for the read-only tasks 0-9. Subclasses set N, GENUINE_STEPS, ANSWER, + FIRST_GATE (first navigation gate name), WRONG_ANSWERS ({answer: failing check}).""" + + GENUINE_STEPS: list[dict[str, Any]] = [] + ANSWER = "" + FIRST_GATE = "" + WRONG_ANSWERS: dict[str, str] = {} + + @classmethod + def setUpClass(cls) -> None: + if cls is ReadTaskTests: + raise unittest.SkipTest("abstract") + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, self.ANSWER)) + + def test_production_recorder_shape_without_final_url_passes(self) -> None: + """``agent_demo/agent.py`` never writes ``final_url`` - its trajectory keys are task, task_id, + start_url, model, max_steps, steps, terminated, termination_reason, final_answer, + judge_rubric, verifier_path (+ success_self_report on done). These fixtures do write it, so + without this case the verifiers would only ever be proven against a shape the production + recorder does not emit, where the landing page is carried solely by the ``done`` step's url.""" + self.assertPasses(self.verdict(self.GENUINE_STEPS, self.ANSWER, trajectory_updates={"final_url": None})) + + def test_run_dir_snapshots_are_discovered(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, self.ANSWER, snapshots_in_run_dir=True)) + + def test_noop_run_fails_on_empty_answer(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_correct_answer_without_navigation_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], self.ANSWER), self.FIRST_GATE) + + def test_wrong_answers_fail(self) -> None: + for answer, reason in self.WRONG_ANSWERS.items(): + with self.subTest(answer=answer): + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, answer), reason) + + def test_other_task_trajectory_fails(self) -> None: + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, self.ANSWER, task_id="9GAG--99"), "trajectory_task_matches") + + def test_unterminated_run_fails(self) -> None: + verdict = self.verdict(self.GENUINE_STEPS, self.ANSWER, trajectory_updates={"terminated": False}) + self.assertFailsOn(verdict, "trajectory_completed") + + def test_mixed_origin_run_fails(self) -> None: + verdict = self.verdict(self.GENUINE_STEPS, self.ANSWER, trajectory_updates={"start_url": "http://127.0.0.1:41024/"}) + self.assertFailsOn(verdict, "all_urls_match_local_origin") + + def test_corrupt_screenshot_fails(self) -> None: + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, self.ANSWER, corrupt_screenshot=True), "screenshots_decode") + + def test_read_only_write_fails(self) -> None: + after = State().add_saved(USERS["bob"][0], 3) + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, self.ANSWER, after=after), "read_only_db_unchanged") + + def test_schema_change_fails_closed(self) -> None: + after = State().sql("CREATE TABLE injected(id INTEGER PRIMARY KEY)") + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, self.ANSWER, after=after), "snapshot_contract_invalid") + + def test_catalog_change_fails_closed(self) -> None: + after = State().sql("UPDATE post SET description='tampered' WHERE id=12") + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, self.ANSWER, after=after), "snapshot_contract_invalid") diff --git a/sites/9gag/verify/tests/live_matrix.py b/sites/9gag/verify/tests/live_matrix.py new file mode 100644 index 00000000..1472700e --- /dev/null +++ b/sites/9gag/verify/tests/live_matrix.py @@ -0,0 +1,505 @@ +#!/usr/bin/env python3 +"""Live grading-contract matrix for the 9gag verifiers (CONTRIBUTING.md "C. Verify the grading itself"). + +Drives every task for real in Chromium (Playwright) against a running mirror, records agent.py-shaped +trajectories (step ``url`` = page before the action, real PNG screenshots), snapshots the SQLite DB +before/after, and runs each verifier on five run dirs per task: + + genuine real UI drive, correct answer, real after-state DB -> expected PASS + noop homepage only, empty answer, clean DB -> expected FAIL + shortcut correct answer / self-reported success, no on-site navigation -> expected FAIL + wrong genuine trajectory + wrong answer (read) / wrong DB target (stateful) -> expected FAIL + mismatch genuine trajectory + clean DB (stateful tasks only) -> expected FAIL + +Usage (agent_demo env; a mirror must be running): + cd agent_demo && uv run python ../sites/9gag/verify/tests/live_matrix.py \ + --base http://127.0.0.1:45001 --site_dir ../sites/9gag --out runs/9gag_matrix \ + --reset_cmd "" +With the docker image use e.g. --reset_cmd "curl -s -X POST http://localhost:8201/reset/9gag" and +--after_db_cmd "docker cp wh-review:/opt/WebSyn/9gag/instance/9gag.db {dest}". +""" +from __future__ import annotations + +import argparse +import json +import re +import shutil +import subprocess +import sys +import time +import urllib.request +from pathlib import Path + +from playwright.sync_api import sync_playwright + +HERE = Path(__file__).resolve().parent +VERIFY_DIR = HERE.parent +SITE_DIR = VERIFY_DIR.parent +sys.path.insert(0, str(HERE)) +from _support import State # noqa: E402 + +PASSWORD = "TestPass123!" +READ_TASKS = range(0, 10) +STATEFUL_TASKS = range(10, 20) + +WRONG_ANSWERS = { + 0: "The room is 3.8 meters wide and the desk is walnut.", + 1: "Miso arrived in March and learned the routine in nine days.", + 2: "A 200-watt panel; the controller sat in a dry bag.", + 3: "About 220 hertz, produced by the cables.", + 4: "Red cabinet, restocked every Friday at 7 a.m.", + 5: "Kilometer 30 with red flags.", + 6: "Two attempts and nine hours.", + 7: "Clicky switches and an aluminium case.", + 8: "Platform three.", + 9: "The kayak dog post wins; the dog is named Pepper.", +} + + +def wrong_state(n): + """A plausible-but-wrong after-state for each stateful task (built on the seed).""" + return { + 10: State().add_saved(1, 47), # the remix clone instead of the original + 11: State().add_vote(3, 20, -1), # downvote + 12: State().add_comment(2, 19, "Resonance is neat."), + 13: State().add_hidden(3, 61), # remix clone hidden + 14: State().add_report(4, 6, "Spam"), + 15: State().add_post("Quiet victories deserve confetti", "My neighbor finished night school after six years.", + "humor", "community|education|wholesome", "alice_j", slug="quiet-victories-deserve-confetti"), + 16: State().set_profile(1, display_name="Alice J.", bio="Memes, trail photos, and excellent tiny libraries.", location="Seattle, WA"), + 17: State().add_user("river.reader@test.com", "river_reader", "WrongPass123!"), + 18: State().remove_saved(1, 16), # a different saved post removed + 19: State().add_saved(4, 11), # the solar post (120) instead of the bridge (440) + }[n] + + +# ----------------------------------------------------------------------------- recorder +class Recorder: + """Writes a run dir in the agent_demo/agent.py shape while Playwright drives the page.""" + + def __init__(self, page, run_dir, task, base): + self.page, self.run_dir, self.base = page, Path(run_dir), base + self.shots = self.run_dir / "screenshots" + self.shots.mkdir(parents=True, exist_ok=True) + self.steps, self.idx = [], 0 + self.traj = {"task": task["ques"], "task_id": task["id"], "start_url": base + "/", "model": "live_matrix", + "max_steps": 60, "steps": [], "terminated": False, "termination_reason": None, "final_answer": None, + "judge_rubric": task.get("judge_rubric", ""), "verifier_path": task.get("verifier_path", "")} + page.goto(base + "/", wait_until="networkidle") + self._shot(0) + + def _shot(self, i): + self.page.screenshot(path=str(self.shots / f"step_{i:03d}.png")) + + def act(self, action, params, fn): + url, title = self.page.url, self.page.title() + fn() + self.page.wait_for_load_state("networkidle") + self.steps.append({"step": self.idx, "url": url, "title": title, "thought": "", "action": action, "params": params, + "screenshot_before": f"step_{self.idx:03d}.png", "screenshot_after": f"step_{self.idx + 1:03d}.png", + "action_result": {"is_done": False, "success": True, "error": None, "extracted_content": ""}}) + self.idx += 1 + self._shot(self.idx) + + def click(self, selector, label=None): + self.act("click", {"index": label or selector}, lambda: self.page.click(selector)) + + def fill(self, selector, text): + self.act("input", {"index": selector, "text": text}, lambda: self.page.fill(selector, text)) + + def select(self, selector, value): + self.act("input", {"index": selector, "text": value}, lambda: self.page.select_option(selector, value)) + + def done(self, text, success=True): + self.steps.append({"step": self.idx, "url": self.page.url, "title": self.page.title(), "thought": "", "action": "done", + "params": {"text": text, "success": success}, "screenshot_before": f"step_{self.idx:03d}.png", + "screenshot_after": f"step_{self.idx + 1:03d}.png"}) + self.idx += 1 + self._shot(self.idx) + # No `final_url`: agent_demo/agent.py never writes one. The landing page is carried by the + # `done` step's url above (agent.py records state.url on the done action the same way), so the + # evidence this matrix produces is shape-identical to a real recorder run. + self.traj.update(steps=self.steps, terminated=True, termination_reason="agent_done", final_answer=text, + success_self_report=success) + (self.run_dir / "trajectory.json").write_text(json.dumps(self.traj, indent=2), encoding="utf-8") + + +# ----------------------------------------------------------------------------- UI helpers +def login(rec, email): + rec.click("header a.muted-link", "Sign Up/Log In") + rec.fill("input[name=identity]", email) + rec.fill("input[name=password]", PASSWORD) + rec.click("main.auth button.primary", "Log in") + + +def search(rec, query): + rec.fill("input[name=q]", query) + rec.click("form.searchbar button", "search") + + +def sidebar(rec, path): + rec.click(f"aside.sidebar a[href='{path}']", path) + + +def title_selector(title): + return f'article.post-card h2 a:text-is("{title}")' + + +def open_title(rec, title): + rec.click(title_selector(title), title) + + +def find_in_feed(rec, path, title): + """Click the sidebar entry, page through the feed until `title` is visible, open it.""" + sidebar(rec, path) + for _ in range(10): + if rec.page.query_selector(title_selector(title)): + open_title(rec, title) + return + rec.click("nav.pagination a:has-text('More posts')", "More posts") + raise RuntimeError(f"{title!r} not found in {path}") + + +def description(rec): + return rec.page.inner_text("p.description").strip() + + +def flash(rec): + el = rec.page.query_selector(".flash") + return el.inner_text().strip() if el else "" + + +def card_points(rec, path, titles): + """Walk a feed and return {title: points} for the given titles (as shown on the cards).""" + sidebar(rec, path) + found = {} + for _ in range(10): + for art in rec.page.query_selector_all("article.post-card"): + t = art.query_selector("h2 a").inner_text().strip() + if t in titles: + found[t] = int(art.query_selector(".stats strong").inner_text().replace(",", "")) + if len(found) == len(titles): + return found + nxt = rec.page.query_selector("nav.pagination a:has-text('More posts')") + if not nxt: + break + rec.click("nav.pagination a:has-text('More posts')", "More posts") + raise RuntimeError(f"points not found for {set(titles) - set(found)}") + + +# ----------------------------------------------------------------------------- task drivers +LIGHTHOUSE = "Tiny lighthouse office with the best ocean view" +CAT = "Rescue cat learns the sound of the treat drawer" +SOLAR = "Solar-powered camping setup survives a rainy weekend" +BRIDGE = "An engineer explains why this bridge hums in the wind" +LIBRARY = "Neighborhood builds a miniature library for night-shift workers" +MARATHON = "Grandmother finishes her first marathon at seventy-two" +SOURDOUGH = "A baker recreates a city skyline in sourdough" +KEYBOARD = "Mechanical keyboard made entirely from transparent parts" +CONCERT = "Street musician turns a rain delay into a concert" +DOG = "Dog refuses to leave the kayak after the trip ends" +FOX = "A fox naps on the same garden wall every afternoon" +PLANT = "The office plant gets an employee badge" +PRESS = "A mysterious press-conference moment from 2009" +BALLOON = "Students launch a weather balloon with a tiny rubber duck" + + +def drive_0(rec): + search(rec, "lighthouse offices") + open_title(rec, LIGHTHOUSE) + return description(rec) + + +def drive_1(rec): + search(rec, "rescue cat") + open_title(rec, CAT) + return description(rec) + + +def drive_2(rec): + search(rec, "solar camping") + open_title(rec, SOLAR) + return description(rec) + + +def drive_3(rec): + find_in_feed(rec, "/interest/science", BRIDGE) + return description(rec) + + +def drive_4(rec): + search(rec, "community library") + open_title(rec, LIBRARY) + return description(rec) + + +def drive_5(rec): + find_in_feed(rec, "/interest/sports", MARATHON) + return description(rec) + + +def drive_6(rec): + search(rec, "sourdough skyline") + open_title(rec, SOURDOUGH) + return description(rec) + + +def drive_7(rec): + find_in_feed(rec, "/interest/gaming", KEYBOARD) + return description(rec) + + +def drive_8(rec): + search(rec, "rain delay concert") + open_title(rec, CONCERT) + return description(rec) + + +def drive_9(rec): + points = card_points(rec, "/interest/animals", {CAT, DOG, FOX}) + winner = max(points, key=points.get) + find_in_feed(rec, "/interest/animals", winner) + return f"{winner} has the most points ({points[winner]} vs {sorted(points.values())}). {description(rec)}" + + +def drive_10(rec): + login(rec, "alice.j@test.com") + search(rec, "lighthouse office") + open_title(rec, LIGHTHOUSE) + rec.click(".detail .actions form[action$='/save'] button", "Save") + return f"Saved the original post; button now reads {rec.page.inner_text('.detail .actions form[action$=\"/save\"] button').strip()!r}. {flash(rec)}" + + +def drive_11(rec): + login(rec, "carol.d@test.com") + find_in_feed(rec, "/interest/animals", DOG) + rec.click(".detail .actions form button:has-text('Upvote')", "Upvote") + return f"Upvoted the original kayak-dog post. Stats now: {rec.page.inner_text('.detail .stats').strip()}" + + +def drive_12(rec): + login(rec, "bob.c@test.com") + find_in_feed(rec, "/interest/science", BRIDGE) + rec.fill("textarea[name=body]", "Resonance makes ordinary structures fascinating.") + rec.click("section.comments form button.primary", "Post comment") + return f"Comment posted on the original humming-bridge post. {flash(rec)}" + + +def drive_13(rec): + login(rec, "carol.d@test.com") + search(rec, "office plant employee badge") + open_title(rec, PLANT) + rec.click(".detail .actions form[action$='/hide'] button", "Hide") + return f"Hid the original office-plant post. {flash(rec)}" + + +def drive_14(rec): + login(rec, "david.k@test.com") + find_in_feed(rec, "/news", PRESS) + rec.click(".detail .actions a:has-text('Report')", "Report") + rec.select("select[name=reason]", "Misinformation") + rec.click("main.form-page button.primary", "Submit report") + return f"Reported the 2009 press-conference post for Misinformation. {flash(rec)}" + + +def drive_15(rec): + login(rec, "alice.j@test.com") + rec.click("header a.post-button", "Post") + rec.fill("input[name=title]", "Quiet victories deserve confetti") + rec.fill("textarea[name=description]", "My neighbor finished night school after six years.") + rec.select("select[name=section]", "wholesome") + rec.fill("input[name=tags]", "community|education|wholesome") + rec.click("main.form-page button.primary", "Publish post") + return f"Published the post; it is live at {rec.page.url}. {flash(rec)}" + + +def drive_16(rec): + login(rec, "alice.j@test.com") + sidebar(rec, "/settings") + rec.fill("input[name=display_name]", "Alice J.") + rec.fill("textarea[name=bio]", "Memes, trail photos, and excellent tiny libraries.") + rec.fill("input[name=location]", "Tacoma, WA") + rec.click("main.form-page button.primary", "Save changes") + return f"Profile updated. {flash(rec)}" + + +def drive_17(rec): + rec.click("header a.muted-link", "Sign Up/Log In") + rec.click("main.auth a[href='/register']", "Create an account") + rec.fill("input[name=email]", "river.reader@test.com") + rec.fill("input[name=username]", "river_reader") + rec.fill("input[name=password]", "RiverRead123!") + rec.click("main.auth button.primary", "Sign up") + return f"Account created; the header now shows the signed-in username {rec.page.inner_text('header a.muted-link').strip()!r}." + + +def drive_18(rec): + login(rec, "alice.j@test.com") + sidebar(rec, "/saved") + rec.click(f'article.post-card:has(h2 a:text-is("{SOLAR}")) form[action$=\'/save\'] button', "Saved (remove)") + return f"Removed the original solar-powered rainy-weekend camping setup from Saved. {flash(rec)}" + + +def drive_19(rec): + login(rec, "david.k@test.com") + numbers = {} + for title in (SOLAR, BRIDGE, BALLOON): + find_in_feed(rec, "/interest/science", title) + text = description(rec) + numbers[title] = max(int(x) for x in re.findall(r"\d+", text)) + winner = max(numbers, key=numbers.get) + find_in_feed(rec, "/interest/science", winner) + rec.click(".detail .actions form[action$='/save'] button", "Save") + return f"Largest stated numbers: {numbers}. Saved {winner!r}. {flash(rec)}" + + +DRIVERS = {n: globals()[f"drive_{n}"] for n in range(20)} + + +# ----------------------------------------------------------------------------- matrix +def run_verifier(python, n, run_dir): + cmd = [python, str(VERIFY_DIR / f"verify_{n}.py"), "--run_dir", str(run_dir), "--no_llm", "True"] + r = subprocess.run(cmd, capture_output=True, text=True) + try: + out = json.loads(r.stdout) + except json.JSONDecodeError: + out = {"pass": False, "reason": f"verifier crashed: {r.stderr[-400:]}"} + out["returncode"] = r.returncode + return out + + +def clone_run(src, dst, **updates): + if dst.exists(): + shutil.rmtree(dst) + shutil.copytree(src, dst) + path = dst / "trajectory.json" + data = json.loads(path.read_text()) + data.update(updates) + path.write_text(json.dumps(data, indent=2)) + + +def wait_ready(base, timeout=30): + deadline = time.time() + timeout + while time.time() < deadline: + try: + urllib.request.urlopen(base + "/_health", timeout=3).read() + return True + except Exception: # noqa: BLE001 + time.sleep(0.5) + return False + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--base", default="http://127.0.0.1:45001") + ap.add_argument("--site_dir", default=str(SITE_DIR)) + ap.add_argument("--out", default=str(Path.cwd() / "runs" / "9gag_matrix")) + ap.add_argument("--reset_cmd", default="", help="shell command that restores the seed DB and restarts the site") + ap.add_argument("--after_db_cmd", default="", help="shell template copying the live instance DB to {dest}") + ap.add_argument("--seed_db", default="") + ap.add_argument("--tasks", default="", help="comma-separated task numbers (default all)") + ap.add_argument("--python", default=sys.executable) + args = ap.parse_args() + + site_dir = Path(args.site_dir).resolve() + seed_db = Path(args.seed_db) if args.seed_db else site_dir / "instance_seed" / "9gag.db" + live_db = site_dir / "instance" / "9gag.db" + out = Path(args.out).resolve() + out.mkdir(parents=True, exist_ok=True) + tasks = {json.loads(l)["id"]: json.loads(l) for l in (site_dir / "tasks.jsonl").read_text().splitlines() if l.strip()} + numbers = [int(x) for x in args.tasks.split(",")] if args.tasks else list(range(20)) + + def reset(): + if args.reset_cmd: + subprocess.run(args.reset_cmd, shell=True, check=True, capture_output=True) + if not wait_ready(args.base): + raise RuntimeError("site not ready after reset") + + def snapshot_after(dest): + if args.after_db_cmd: + subprocess.run(args.after_db_cmd.format(dest=dest), shell=True, check=True) + else: + shutil.copy2(live_db, dest) + + matrix = {} + with sync_playwright() as pw: + browser = pw.chromium.launch() + + # one recorded no-op run (homepage only, empty answer) reused for every task + noop_src = out / "_noop" + if noop_src.exists(): + shutil.rmtree(noop_src) + page = browser.new_context(viewport={"width": 1440, "height": 900}).new_page() + Recorder(page, noop_src, {"id": "9GAG--0", "ques": "noop"}, args.base).done("") + page.context.close() + + for n in numbers: + task_id = f"9GAG--{n}" + task = tasks[task_id] + stateful = n in STATEFUL_TASKS + reset() + ctx = browser.new_context(viewport={"width": 1440, "height": 900}) + page = ctx.new_page() + genuine = out / f"task{n:02d}_genuine" + if genuine.exists(): + shutil.rmtree(genuine) + rec = Recorder(page, genuine, task, args.base) + try: + answer = DRIVERS[n](rec) + except Exception as exc: # noqa: BLE001 + page.screenshot(path=str(out / f"task{n:02d}_drive_error.png")) + raise RuntimeError(f"task {n} drive failed: {exc}") from exc + rec.done(answer, True) + ctx.close() + shutil.copy2(seed_db, genuine / "initial.db") + snapshot_after(genuine / "after.db") + + variants = {"genuine": genuine} + noop = out / f"task{n:02d}_noop" + clone_run(noop_src, noop, task_id=task_id, task=task["ques"], final_answer=None, success_self_report=False) + shutil.copy2(seed_db, noop / "initial.db") + shutil.copy2(seed_db, noop / "after.db") + variants["noop"] = noop + + shortcut = out / f"task{n:02d}_shortcut" + clone_run(noop_src, shortcut, task_id=task_id, task=task["ques"], final_answer=answer, success_self_report=True) + shutil.copy2(seed_db, shortcut / "initial.db") + shutil.copy2(seed_db, shortcut / "after.db") + variants["shortcut"] = shortcut + + wrong = out / f"task{n:02d}_wrong" + if stateful: + clone_run(genuine, wrong) + wrong_state(n).write(wrong / "after.db") + else: + clone_run(genuine, wrong, final_answer=WRONG_ANSWERS[n]) + variants["wrong"] = wrong + + if stateful: + mismatch = out / f"task{n:02d}_mismatch" + clone_run(genuine, mismatch) + shutil.copy2(seed_db, mismatch / "after.db") + variants["mismatch"] = mismatch + + row = {} + for name, run_dir in variants.items(): + verdict = run_verifier(args.python, n, run_dir) + expected = name == "genuine" + row[name] = {"pass": bool(verdict.get("pass")), "reason": verdict.get("reason"), "expected_pass": expected, + "ok": bool(verdict.get("pass")) == expected} + row["genuine_urls"] = [s["url"].replace(args.base, "") for s in json.loads((genuine / "trajectory.json").read_text())["steps"]] + row["genuine_answer"] = answer + matrix[task_id] = row + cells = " ".join(f"{k}={'PASS' if v['pass'] else 'FAIL'}{'' if v['ok'] else '(!!)'}" for k, v in row.items() if isinstance(v, dict)) + print(f"{task_id:9s} {cells}", flush=True) + browser.close() + + reset() + (out / "matrix.json").write_text(json.dumps(matrix, indent=2, ensure_ascii=False)) + bad = [(t, k) for t, row in matrix.items() for k, v in row.items() if isinstance(v, dict) and not v["ok"]] + print(f"\nwrote {out / 'matrix.json'}; cells wrong: {bad if bad else 'none'}") + sys.exit(1 if bad else 0) + + +if __name__ == "__main__": + main() diff --git a/sites/9gag/verify/tests/test_verify_0.py b/sites/9gag/verify/tests/test_verify_0.py new file mode 100644 index 00000000..18dfe1cb --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_0.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/tiny-lighthouse-office-with-the-best-ocean-view-12" +REMIX = "/gag/community-remix-2-tiny-lighthouse-office-with-the-best-ocean-view-47" + + +class VerifyTask0Tests(ReadTaskTests): + N = 0 + GENUINE_STEPS = [step("/"), step("/", "input", "lighthouse offices"), step("/search?q=lighthouse+offices"), step(ORIGINAL, "done")] + ANSWER = "The circular room is 4.2 meters wide and the desk was built from reclaimed oak." + FIRST_GATE = "visited_search_results" + WRONG_ANSWERS = { + "The room is 4.25 meters wide and the desk is reclaimed oak.": "answer_has_room_width_meters", + "The room is 4.2 meters wide and the desk is walnut.": "answer_has_desk_material", + "The room is 4.2 meters wide; the desk is not reclaimed oak.": "answer_has_desk_material", + } + + def test_remix_clone_detail_also_counts(self) -> None: + steps = [step("/"), step("/search?q=lighthouse"), step(REMIX, "done")] + self.assertPasses(self.verdict(steps, "4.2 m wide, reclaimed oak desk")) + + def test_search_without_related_token_fails(self) -> None: + steps = [step("/"), step("/search?q=cats"), step(ORIGINAL, "done")] + self.assertFailsOn(self.verdict(steps, self.ANSWER), "visited_search_results") + + def test_results_page_only_fails(self) -> None: + steps = [step("/"), step("/search?q=lighthouse+offices", "done")] + self.assertFailsOn(self.verdict(steps, self.ANSWER), "visited_post_detail") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_1.py b/sites/9gag/verify/tests/test_verify_1.py new file mode 100644 index 00000000..2f5df4b3 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_1.py @@ -0,0 +1,30 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/rescue-cat-learns-the-sound-of-the-treat-drawer-13" + + +class VerifyTask1Tests(ReadTaskTests): + N = 1 + GENUINE_STEPS = [step("/"), step("/", "input", "rescue cat"), step("/search?q=rescue+cat"), step(ORIGINAL, "done")] + ANSWER = "Miso arrived at Harbor Paws in February and learned the routine after eleven days." + FIRST_GATE = "visited_search_results" + WRONG_ANSWERS = { + "Miso arrived in March and learned it in eleven days.": "answer_has_arrival_month", + "Miso arrived in February and learned it in 12 days.": "answer_has_days_count", + "February; it took 110 days.": "answer_has_days_count", + } + + def test_numeric_days_pass(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, "February, 11 days")) + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_10.py b/sites/9gag/verify/tests/test_verify_10.py new file mode 100644 index 00000000..d1a2e5e8 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_10.py @@ -0,0 +1,84 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +ALICE, EMAIL, _ = USERS["alice"] +ORIGINAL = "/gag/tiny-lighthouse-office-with-the-best-ocean-view-12" +REMIX = "/gag/community-remix-2-tiny-lighthouse-office-with-the-best-ocean-view-47" +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home", "input", "lighthouse office"), + step("/search?q=lighthouse+office"), + step(ORIGINAL, "click"), + step(ORIGINAL, "done"), +] +ANSWER = "Saved 'Tiny lighthouse office with the best ocean view' to Alice's collection." + + +def genuine_after() -> State: + return State().add_saved(ALICE, 12) + + +class VerifyTask10Tests(VerifierTestCase): + N = 10 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_run_dir_snapshots_are_discovered(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after(), snapshots_in_run_dir=True)) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails_on_login_gate(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "saved_post_exact_delta") + + def test_saving_the_remix_clone_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/search?q=lighthouse+office"), step(REMIX, "click"), step(REMIX, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=State().add_saved(ALICE, 47)), "visited_original_post_detail") + + def test_wrong_account_fails(self) -> None: + steps = login_steps(USERS["bob"][1]) + [step("/search?q=lighthouse"), step(ORIGINAL, "done")] + after = State().add_saved(USERS["bob"][0], 12) + self.assertFailsOn(self.verdict(steps, ANSWER, after=after), "entered_expected_account_identity") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_vote(ALICE, 12, 1) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + def test_extra_save_fails(self) -> None: + after = genuine_after().add_saved(ALICE, 47) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "saved_post_exact_delta") + + def test_missing_search_visit_fails(self) -> None: + steps = login_steps(EMAIL) + [step(ORIGINAL, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_search_results") + + def test_login_after_save_fails_order(self) -> None: + steps = [step("/"), step("/search?q=lighthouse"), step(ORIGINAL, "click")] + login_steps(EMAIL) + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "workflow_login_before_save") + + def test_initial_snapshot_with_extra_row_fails_closed(self) -> None: + initial = State().add_saved(ALICE, 12) # 17 saved rows: not the frozen seed + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, initial=initial, after=initial), "snapshot_contract_invalid") + + def test_already_saved_precondition_fails(self) -> None: + initial = State().remove_saved(ALICE, 1).add_saved(ALICE, 12) # same counts, target already saved + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, initial=initial, after=initial), "initial_state_requires_action") + + def test_mixed_origin_run_fails(self) -> None: + verdict = self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after(), trajectory_updates={"start_url": "http://localhost:40026/"}) + self.assertFailsOn(verdict, "all_urls_match_local_origin") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_11.py b/sites/9gag/verify/tests/test_verify_11.py new file mode 100644 index 00000000..4c844225 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_11.py @@ -0,0 +1,62 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +CAROL, EMAIL, _ = USERS["carol"] +ORIGINAL = "/gag/dog-refuses-to-leave-the-kayak-after-the-trip-ends-20" +REMIX = "/gag/community-remix-10-dog-refuses-to-leave-the-kayak-after-the-trip-ends-55" +GENUINE_STEPS = login_steps(EMAIL) + [step("/home"), step("/interest/animals"), step(ORIGINAL, "click"), step(ORIGINAL, "done")] +ANSWER = "Upvoted the kayak dog post." + + +def genuine_after() -> State: + return State().add_vote(CAROL, 20, 1) + + +class VerifyTask11Tests(VerifierTestCase): + N = 11 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "vote_exact_delta") + + def test_downvote_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=State().add_vote(CAROL, 20, -1)), "vote_exact_delta") + + def test_vote_on_remix_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/interest/animals"), step(REMIX, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=State().add_vote(CAROL, 55, 1)), "visited_original_post_detail") + + def test_vote_row_without_counter_bump_fails(self) -> None: + after = State().add_vote(CAROL, 20, 1, bump=False) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "post_20_counters_exact_delta") + + def test_counter_bump_on_other_post_fails(self) -> None: + after = genuine_after().sql("UPDATE post SET up_votes = up_votes + 1 WHERE id = 21") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "post_20_counters_exact_delta") + + def test_collateral_save_fails(self) -> None: + after = genuine_after().add_saved(CAROL, 12) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + def test_search_instead_of_browse_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/search?q=kayak"), step(ORIGINAL, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_animals_interest_feed") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_12.py b/sites/9gag/verify/tests/test_verify_12.py new file mode 100644 index 00000000..8ecb84f3 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_12.py @@ -0,0 +1,67 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +BOB, EMAIL, _ = USERS["bob"] +ORIGINAL = "/gag/an-engineer-explains-why-this-bridge-hums-in-the-wind-19" +COMMENT = "Resonance makes ordinary structures fascinating." +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home"), step("/interest/science"), step("/interest/science?page=2"), + step(ORIGINAL, "input", COMMENT), step(ORIGINAL, "click"), step(ORIGINAL + "#comments", "done"), +] +ANSWER = "Comment posted on the humming bridge post." + + +def genuine_after() -> State: + return State().add_comment(BOB, 19, COMMENT) + + +class VerifyTask12Tests(VerifierTestCase): + N = 12 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "comment_exact_delta") + + def test_wrong_comment_text_fails(self) -> None: + after = State().add_comment(BOB, 19, "Resonance makes ordinary structures boring.") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "comment_exact_delta") + + def test_trailing_period_and_case_are_tolerated(self) -> None: + after = State().add_comment(BOB, 19, "resonance makes ordinary structures fascinating") + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=after)) + + def test_comment_on_remix_fails(self) -> None: + after = State().add_comment(BOB, 54, COMMENT) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "comment_exact_delta") + + def test_duplicate_comment_fails(self) -> None: + after = genuine_after().add_comment(BOB, 19, COMMENT) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "comment_exact_delta") + + def test_comment_without_counter_bump_fails(self) -> None: + after = State().add_comment(BOB, 19, COMMENT, bump=False) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "post_19_counters_exact_delta") + + def test_wrong_account_fails(self) -> None: + steps = login_steps(USERS["carol"][1]) + [step("/interest/science"), step(ORIGINAL, "done")] + after = State().add_comment(USERS["carol"][0], 19, COMMENT) + self.assertFailsOn(self.verdict(steps, ANSWER, after=after), "entered_expected_account_identity") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_13.py b/sites/9gag/verify/tests/test_verify_13.py new file mode 100644 index 00000000..d7f6e3f0 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_13.py @@ -0,0 +1,54 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +CAROL, EMAIL, _ = USERS["carol"] +ORIGINAL = "/gag/the-office-plant-gets-an-employee-badge-26" +REMIX = "/gag/community-remix-16-the-office-plant-gets-an-employee-badge-61" +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home", "input", "office plant employee badge"), step("/search?q=office+plant+employee+badge"), + step(ORIGINAL, "click"), step("/", "done"), +] +ANSWER = "Hidden the office plant post from Carol's feeds." + + +def genuine_after() -> State: + return State().add_hidden(CAROL, 26) + + +class VerifyTask13Tests(VerifierTestCase): + N = 13 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "hidden_post_exact_delta") + + def test_hiding_the_remix_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/search?q=office+plant"), step(REMIX, "click"), step("/", "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=State().add_hidden(CAROL, 61)), "visited_original_post_detail") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_saved(CAROL, 26) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + def test_missing_search_fails(self) -> None: + steps = login_steps(EMAIL) + [step(ORIGINAL, "click"), step("/", "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_search_results") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_14.py b/sites/9gag/verify/tests/test_verify_14.py new file mode 100644 index 00000000..fe47864a --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_14.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +DAVID, EMAIL, _ = USERS["david"] +DETAIL = "/gag/a-mysterious-press-conference-moment-from-2009-6" +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home"), step("/news"), step("/news?page=2"), step(DETAIL, "click"), + step(DETAIL + "/report", "input", "Misinformation"), step(DETAIL + "/report", "click"), step(DETAIL, "done"), +] +ANSWER = "Reported the 2009 press-conference post for Misinformation." + + +def genuine_after() -> State: + return State().add_report(DAVID, 6, "Misinformation") + + +class VerifyTask14Tests(VerifierTestCase): + N = 14 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "report_exact_delta") + + def test_wrong_reason_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=State().add_report(DAVID, 6, "Spam")), "report_exact_delta") + + def test_wrong_post_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=State().add_report(DAVID, 7, "Misinformation")), "report_exact_delta") + + def test_missing_news_visit_fails(self) -> None: + steps = login_steps(EMAIL) + [step(DETAIL), step(DETAIL + "/report", "click"), step(DETAIL, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_news_feed") + + def test_report_form_not_opened_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/news"), step(DETAIL, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_report_form") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_hidden(DAVID, 6) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_15.py b/sites/9gag/verify/tests/test_verify_15.py new file mode 100644 index 00000000..1fef5690 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_15.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +ALICE, EMAIL, USERNAME = USERS["alice"] +TITLE = "Quiet victories deserve confetti" +DESCRIPTION = "My neighbor finished night school after six years." +TAGS = "community|education|wholesome" +NEW = "/gag/quiet-victories-deserve-confetti" +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home"), step("/submit", "input", TITLE), step("/submit", "input", DESCRIPTION), + step("/submit", "input", TAGS), step("/submit", "click"), step(NEW, "done"), +] +ANSWER = "Published the post; it is live at /gag/quiet-victories-deserve-confetti." + + +def genuine_after() -> State: + return State().add_post(TITLE, DESCRIPTION, "wholesome", TAGS, USERNAME, slug="quiet-victories-deserve-confetti") + + +class VerifyTask15Tests(VerifierTestCase): + N = 15 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "new_post_exact_delta") + + def test_wrong_section_fails(self) -> None: + after = State().add_post(TITLE, DESCRIPTION, "humor", TAGS, USERNAME, slug="quiet-victories-deserve-confetti") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_post_exact_delta") + + def test_wrong_tags_fail(self) -> None: + after = State().add_post(TITLE, DESCRIPTION, "wholesome", "community|wholesome", USERNAME, slug="quiet-victories-deserve-confetti") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_post_exact_delta") + + def test_wrong_author_fails(self) -> None: + after = State().add_post(TITLE, DESCRIPTION, "wholesome", TAGS, "bob_c", slug="quiet-victories-deserve-confetti") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_post_exact_delta") + + def test_two_posts_fail(self) -> None: + after = genuine_after().add_post(TITLE, DESCRIPTION, "wholesome", TAGS, USERNAME, slug="quiet-victories-deserve-confetti-2") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_post_exact_delta") + + def test_published_page_not_opened_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/submit", "click"), step("/submit", "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_published_post") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_saved(ALICE, 12) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_16.py b/sites/9gag/verify/tests/test_verify_16.py new file mode 100644 index 00000000..f26a11fb --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_16.py @@ -0,0 +1,65 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +ALICE, EMAIL, _ = USERS["alice"] +BIO = "Memes, trail photos, and excellent tiny libraries." +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home"), step("/settings", "input", "Alice J."), step("/settings", "input", BIO), + step("/settings", "input", "Tacoma, WA"), step("/settings", "click"), step("/settings", "done"), +] +ANSWER = "Profile updated: display name, bio and location saved." + + +def genuine_after() -> State: + return State().set_profile(ALICE, display_name="Alice J.", bio=BIO, location="Tacoma, WA") + + +class VerifyTask16Tests(VerifierTestCase): + N = 16 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "profile_fields_updated") + + def test_partial_update_fails(self) -> None: + after = State().set_profile(ALICE, display_name="Alice J.", bio=BIO) # location untouched + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "profile_fields_updated") + + def test_wrong_user_updated_fails(self) -> None: + after = State().set_profile(USERS["bob"][0], display_name="Alice J.", bio=BIO, location="Tacoma, WA") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "profile_fields_updated") + + def test_password_change_fails(self) -> None: + after = genuine_after().set_profile(ALICE, password_hash="scrypt:32768:8:1$x$y") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "user_exact_delta") + + def test_second_user_changed_fails(self) -> None: + after = genuine_after().set_profile(USERS["bob"][0], location="Nowhere") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "user_exact_delta") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_saved(ALICE, 12) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + def test_settings_page_not_opened_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/account", "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_settings_page") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_17.py b/sites/9gag/verify/tests/test_verify_17.py new file mode 100644 index 00000000..bc178a64 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_17.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import State, VerifierTestCase, step # noqa: E402 + +EMAIL, USERNAME, PASSWORD = "river.reader@test.com", "river_reader", "RiverRead123!" +GENUINE_STEPS = [ + step("/"), step("/login"), step("/register", "input", EMAIL), step("/register", "input", USERNAME), + step("/register", "input", PASSWORD), step("/register", "click"), step("/home", "done"), +] +ANSWER = "Account created and signed in; the header shows river_reader." + + +def genuine_after() -> State: + return State().add_user(EMAIL, USERNAME, PASSWORD) + + +class VerifyTask17Tests(VerifierTestCase): + N = 17 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_register_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "new_user_exact_delta") + + def test_wrong_password_fails(self) -> None: + after = State().add_user(EMAIL, USERNAME, "WrongPass123!") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_user_password_verifies") + + def test_wrong_username_fails(self) -> None: + after = State().add_user(EMAIL, "river.reader", PASSWORD) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_user_exact_delta") + + def test_answer_without_username_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, "Done.", after=genuine_after()), "answer_confirms_header_username") + + def test_existing_user_edited_fails(self) -> None: + after = genuine_after().set_profile(1, display_name="Alice X") + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "new_user_exact_delta") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_saved(1, 12) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_18.py b/sites/9gag/verify/tests/test_verify_18.py new file mode 100644 index 00000000..acb1c33b --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_18.py @@ -0,0 +1,60 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +ALICE, EMAIL, _ = USERS["alice"] +GENUINE_STEPS = login_steps(EMAIL) + [step("/home"), step("/saved", "click"), step("/saved", "done")] +ANSWER = "Removed the solar-powered rainy-weekend camping setup from Saved." + + +def genuine_after() -> State: + return State().remove_saved(ALICE, 11) + + +class VerifyTask18Tests(VerifierTestCase): + N = 18 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "saved_post_exact_delta") + + def test_removing_another_saved_post_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=State().remove_saved(ALICE, 16)), "saved_post_exact_delta") + + def test_removing_two_posts_fails(self) -> None: + after = genuine_after().remove_saved(ALICE, 16) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "saved_post_exact_delta") + + def test_saved_page_not_opened_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/gag/solar-powered-camping-setup-survives-a-rainy-weekend-11", "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_saved_collection") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_hidden(ALICE, 11) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + def test_initial_snapshot_with_missing_row_fails_closed(self) -> None: + initial = State().remove_saved(ALICE, 11) # 15 saved rows: not the frozen seed + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, initial=initial, after=initial), "snapshot_contract_invalid") + + def test_not_saved_initially_precondition_fails(self) -> None: + initial = State().remove_saved(ALICE, 11).add_saved(ALICE, 12) # same counts, target not saved + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, initial=initial, after=initial), "initial_state_requires_action") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_19.py b/sites/9gag/verify/tests/test_verify_19.py new file mode 100644 index 00000000..68b75444 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_19.py @@ -0,0 +1,63 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import USERS, State, VerifierTestCase, login_steps, step # noqa: E402 + +DAVID, EMAIL, _ = USERS["david"] +SOLAR = "/gag/solar-powered-camping-setup-survives-a-rainy-weekend-11" +BRIDGE = "/gag/an-engineer-explains-why-this-bridge-hums-in-the-wind-19" +BALLOON = "/gag/students-launch-a-weather-balloon-with-a-tiny-rubber-duck-25" +GENUINE_STEPS = login_steps(EMAIL) + [ + step("/home"), step("/interest/science"), step(BALLOON), step("/interest/science"), step("/interest/science?page=2"), + step(SOLAR), step("/interest/science?page=2"), step(BRIDGE, "click"), step(BRIDGE, "done"), +] +ANSWER = "Compared 120 W, 440 Hz and 27 km; saved the humming bridge post (440 is the largest)." + + +def genuine_after() -> State: + return State().add_saved(DAVID, 19) + + +class VerifyTask19Tests(VerifierTestCase): + N = 19 + + def test_genuine_run_passes(self) -> None: + self.assertPasses(self.verdict(GENUINE_STEPS, ANSWER, after=genuine_after())) + + def test_noop_run_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ""), "final_answer_nonempty") + + def test_shortcut_fails(self) -> None: + self.assertFailsOn(self.verdict([step("/", "done")], ANSWER, after=genuine_after()), "visited_login_page") + + def test_state_mismatch_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER), "saved_post_exact_delta") + + def test_saving_the_wrong_post_fails(self) -> None: + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=State().add_saved(DAVID, 11)), "saved_post_exact_delta") + + def test_skipping_a_compared_detail_page_fails(self) -> None: + steps = login_steps(EMAIL) + [step("/interest/science"), step(BRIDGE), step(SOLAR), step(BRIDGE, "done")] + self.assertFailsOn(self.verdict(steps, ANSWER, after=genuine_after()), "visited_compared_detail_25") + + def test_wrong_account_fails(self) -> None: + steps = login_steps(USERS["alice"][1]) + GENUINE_STEPS[5:] + after = State().add_saved(USERS["alice"][0], 19) + self.assertFailsOn(self.verdict(steps, ANSWER, after=after), "entered_expected_account_identity") + + def test_two_saves_fail(self) -> None: + after = genuine_after().add_saved(DAVID, 11) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "saved_post_exact_delta") + + def test_collateral_write_fails(self) -> None: + after = genuine_after().add_vote(DAVID, 19, 1) + self.assertFailsOn(self.verdict(GENUINE_STEPS, ANSWER, after=after), "no_collateral_writes") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_2.py b/sites/9gag/verify/tests/test_verify_2.py new file mode 100644 index 00000000..0edad916 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_2.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/solar-powered-camping-setup-survives-a-rainy-weekend-11" + + +class VerifyTask2Tests(ReadTaskTests): + N = 2 + GENUINE_STEPS = [step("/"), step("/", "input", "solar camping"), step("/search?q=solar+camping"), step(ORIGINAL, "done")] + ANSWER = "A 120-watt folding panel; the controller was stored inside a waterproof lunch box." + FIRST_GATE = "visited_search_results" + WRONG_ANSWERS = { + "A 200-watt panel and a waterproof lunch box.": "answer_has_panel_wattage", + "A 120 W panel; the controller was in a dry bag.": "answer_has_controller_protection", + "1200 W panel, lunch box.": "answer_has_panel_wattage", + } + + def test_lunchbox_spelling_variants_pass(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, "120W, waterproof lunchbox")) + self.assertPasses(self.verdict(self.GENUINE_STEPS, "120 watts; a lunch-box")) + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_3.py b/sites/9gag/verify/tests/test_verify_3.py new file mode 100644 index 00000000..8e6d55b2 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_3.py @@ -0,0 +1,30 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/an-engineer-explains-why-this-bridge-hums-in-the-wind-19" + + +class VerifyTask3Tests(ReadTaskTests): + N = 3 + GENUINE_STEPS = [step("/"), step("/interest/science"), step("/interest/science?page=2"), step(ORIGINAL, "done")] + ANSWER = "The note is near 440 hertz and is produced by evenly spaced railings." + FIRST_GATE = "visited_science_interest_feed" + WRONG_ANSWERS = { + "About 44 Hz from the railings.": "answer_has_frequency_hertz", + "440 Hz produced by the cables.": "answer_has_resonating_feature", + } + + def test_search_instead_of_browse_fails(self) -> None: + steps = [step("/"), step("/search?q=bridge"), step(ORIGINAL, "done")] + self.assertFailsOn(self.verdict(steps, self.ANSWER), "visited_science_interest_feed") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_4.py b/sites/9gag/verify/tests/test_verify_4.py new file mode 100644 index 00000000..92390ccc --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_4.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/neighborhood-builds-a-miniature-library-for-night-shift-workers-18" + + +class VerifyTask4Tests(ReadTaskTests): + N = 4 + GENUINE_STEPS = [step("/"), step("/", "input", "community library"), step("/search?q=community+library"), step(ORIGINAL, "done")] + ANSWER = "The cabinet is blue and it is restocked every Thursday at 6 a.m. by local volunteers." + FIRST_GATE = "visited_search_results" + WRONG_ANSWERS = { + "Red cabinet, restocked Thursdays at 6 a.m.": "answer_has_cabinet_colour", + "Blue cabinet, restocked Fridays at 6 a.m.": "answer_has_restock_day", + "Blue cabinet, restocked Thursday at 6 p.m.": "answer_has_restock_time", + } + + def test_time_variants_pass(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, "blue; every Thursday at 06:00")) + self.assertPasses(self.verdict(self.GENUINE_STEPS, "Blue - Thursday 6AM")) + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_5.py b/sites/9gag/verify/tests/test_verify_5.py new file mode 100644 index 00000000..e28d8e60 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_5.py @@ -0,0 +1,27 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/grandmother-finishes-her-first-marathon-at-seventy-two-15" + + +class VerifyTask5Tests(ReadTaskTests): + N = 5 + GENUINE_STEPS = [step("/"), step("/interest/sports"), step(ORIGINAL, "done")] + ANSWER = "Her family met her at kilometer 38 with handmade orange flags." + FIRST_GATE = "visited_sports_interest_feed" + WRONG_ANSWERS = { + "At kilometer 28 with orange flags.": "answer_has_kilometre", + "At kilometer 38 with red flags.": "answer_has_flag_colour", + "Kilometer 380, orange.": "answer_has_kilometre", + } + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_6.py b/sites/9gag/verify/tests/test_verify_6.py new file mode 100644 index 00000000..c0de6336 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_6.py @@ -0,0 +1,42 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/a-baker-recreates-a-city-skyline-in-sourdough-16" +REMIX = "/gag/community-remix-6-a-baker-recreates-a-city-skyline-in-sourdough-51" + + +class VerifyTask6Tests(ReadTaskTests): + N = 6 + GENUINE_STEPS = [step("/"), step("/", "input", "sourdough skyline"), step("/search?q=sourdough+skyline"), step(ORIGINAL, "done")] + ANSWER = "The edible skyline took three attempts and nearly fourteen hours to shape." + FIRST_GATE = "visited_search_results" + WRONG_ANSWERS = { + "Two attempts and fourteen hours.": "answer_has_attempt_count", + "Three attempts and four hours.": "answer_has_shaping_hours", + } + + def test_remix_clone_is_accepted(self) -> None: + """The clone repeats the original's description verbatim, so an agent that reads the facts off + it has answered the question; failing it would be a hidden provenance rule (verify/README.md).""" + steps = [step("/"), step("/search?q=sourdough"), step(REMIX, "done")] + self.assertPasses(self.verdict(steps, self.ANSWER)) + + def test_unrelated_detail_page_still_fails(self) -> None: + """Mutation proof that widening the gate did not disable it.""" + steps = [step("/"), step("/search?q=sourdough"), + step("/gag/a-fox-naps-on-the-same-garden-wall-every-afternoon-23", "done")] + self.assertFailsOn(self.verdict(steps, self.ANSWER), "visited_post_detail") + + def test_numeric_forms_pass(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, "3 attempts, ~14 hours")) + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_7.py b/sites/9gag/verify/tests/test_verify_7.py new file mode 100644 index 00000000..ef43cfb8 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_7.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/mechanical-keyboard-made-entirely-from-transparent-parts-14" + + +class VerifyTask7Tests(ReadTaskTests): + N = 7 + GENUINE_STEPS = [step("/"), step("/interest/gaming"), step(ORIGINAL, "done")] + ANSWER = "It uses silent tactile switches and a hand-polished acrylic case." + FIRST_GATE = "visited_gaming_interest_feed" + WRONG_ANSWERS = { + "Clicky switches and an acrylic case.": "answer_has_switch_type", + "Silent tactile switches and an aluminium case.": "answer_has_case_material", + } + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_8.py b/sites/9gag/verify/tests/test_verify_8.py new file mode 100644 index 00000000..8599157b --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_8.py @@ -0,0 +1,42 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, step # noqa: E402 + +ORIGINAL = "/gag/street-musician-turns-a-rain-delay-into-a-concert-22" +REMIX = "/gag/community-remix-12-street-musician-turns-a-rain-delay-into-a-concert-57" + + +class VerifyTask8Tests(ReadTaskTests): + N = 8 + GENUINE_STEPS = [step("/"), step("/", "input", "rain delay concert"), step("/search?q=rain+delay+concert"), step(ORIGINAL, "done")] + ANSWER = "Commuters joined the chorus under platform seven." + FIRST_GATE = "visited_search_results" + WRONG_ANSWERS = { + "Platform six.": "answer_has_platform_number", + "Platform 17.": "answer_has_platform_number", + } + + def test_remix_clone_is_accepted(self) -> None: + """The clone repeats the original's description verbatim, so an agent that reads the facts off + it has answered the question; failing it would be a hidden provenance rule (verify/README.md).""" + steps = [step("/"), step("/search?q=concert"), step(REMIX, "done")] + self.assertPasses(self.verdict(steps, self.ANSWER)) + + def test_unrelated_detail_page_still_fails(self) -> None: + """Mutation proof that widening the gate did not disable it.""" + steps = [step("/"), step("/search?q=concert"), + step("/gag/a-fox-naps-on-the-same-garden-wall-every-afternoon-23", "done")] + self.assertFailsOn(self.verdict(steps, self.ANSWER), "visited_post_detail") + + def test_numeric_platform_passes(self) -> None: + self.assertPasses(self.verdict(self.GENUINE_STEPS, "Platform 7")) + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_9.py b/sites/9gag/verify/tests/test_verify_9.py new file mode 100644 index 00000000..2ddf73f7 --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_9.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) + +from _support import ReadTaskTests, State, step # noqa: E402 + +FOX = "/gag/a-fox-naps-on-the-same-garden-wall-every-afternoon-23" +REMIX_CAT = "/gag/community-remix-3-rescue-cat-learns-the-sound-of-the-treat-drawer-48" + + +class VerifyTask9Tests(ReadTaskTests): + N = 9 + GENUINE_STEPS = [step("/"), step("/interest/animals"), step(FOX, "done")] + ANSWER = "The garden-fox post has the most points (4236); the fox is named Copper." + FIRST_GATE = "visited_animals_interest_feed" + WRONG_ANSWERS = { + "The kayak dog post wins; the dog is Pepper.": "answer_has_animal_name", + "The cat post wins; the cat is Miso, not Copper.": "answer_has_animal_name", + } + + def test_remix_cat_with_more_points_is_not_the_answer(self) -> None: + steps = [step("/"), step("/interest/animals"), step(REMIX_CAT, "done")] + self.assertFailsOn(self.verdict(steps, "Miso"), "visited_winning_post_detail") + + def test_seed_points_drift_fails_closed(self) -> None: + initial = State().sql("UPDATE post SET up_votes = 9000 WHERE id = 13") + after = State().sql("UPDATE post SET up_votes = 9000 WHERE id = 13") + self.assertFailsOn(self.verdict(self.GENUINE_STEPS, self.ANSWER, initial=initial, after=after), + "seed_points_match_frozen_ground_truth") + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/tests/test_verify_lib.py b/sites/9gag/verify/tests/test_verify_lib.py new file mode 100644 index 00000000..e803ed3e --- /dev/null +++ b/sites/9gag/verify/tests/test_verify_lib.py @@ -0,0 +1,124 @@ +from __future__ import annotations + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from _support import SEED_DB, State, USERS, step, write_run # noqa: E402 +import verify_lib as lib # noqa: E402 + + +class MatcherTests(unittest.TestCase): + def test_contains_count_standalone_only(self) -> None: + self.assertTrue(lib.contains_count("kilometer 38", 38)) + self.assertTrue(lib.contains_count("thirty-eight? no: 38.", 38)) + self.assertFalse(lib.contains_count("kilometer 380", 38)) + self.assertFalse(lib.contains_count("3.8 km", 38)) + self.assertFalse(lib.contains_count("38th kilometre", 38)) + self.assertTrue(lib.contains_count("three attempts", 3)) + self.assertFalse(lib.contains_count("3.5 attempts", 3)) + self.assertFalse(lib.contains_count("13 attempts", 3)) + self.assertTrue(lib.contains_count("eleven days", 11)) + + def test_contains_decimal(self) -> None: + self.assertTrue(lib.contains_decimal("4.2 meters", "4.2")) + self.assertTrue(lib.contains_decimal("4,2 m", "4.2")) + self.assertFalse(lib.contains_decimal("4.25 meters", "4.2")) + self.assertFalse(lib.contains_decimal("14.2 meters", "4.2")) + + def test_contains_clock_time(self) -> None: + for text in ("6 a.m.", "6am", "6:00 AM", "06:00", "at 6 A.M. sharp"): + self.assertTrue(lib.contains_clock_time(text, "6 am"), text) + for text in ("6 p.m.", "18:00", "16:00", "6:30 am"): + self.assertFalse(lib.contains_clock_time(text, "6 am"), text) + + def test_contains_phrase_join_variants(self) -> None: + for text in ("a lunch box", "a lunchbox", "a lunch-box", "LUNCH BOX"): + self.assertTrue(lib.contains_phrase(text, "lunch box"), text) + self.assertFalse(lib.contains_phrase("a lunch bag and a box", "lunch box")) + + def test_negation_is_not_affirmative(self) -> None: + self.assertFalse(lib.contains_all("the flags were not orange", ("orange",))) + self.assertFalse(lib.contains_all("orange is wrong; they were red", ("orange",))) + self.assertTrue(lib.contains_all("not red, orange", ("orange",))) + self.assertTrue(lib.contains_all("The flags were orange, not red.", ("orange",))) + + def test_search_visited_uses_whole_tokens(self) -> None: + traj = {"start_url": "http://localhost:41024/", "steps": [step("/search?q=lighthouses+near+me")]} + self.assertTrue(lib.search_visited(traj, ("lighthouses",))) + self.assertFalse(lib.search_visited(traj, ("lighthouse",))) + self.assertFalse(lib.search_visited({"steps": [step("/tag/lighthouse")]}, ("lighthouse",))) + + def test_detail_visited_is_exact_path(self) -> None: + traj = {"steps": [step("/gag/some-post-1/report")]} + self.assertFalse(lib.detail_visited(traj, "some-post-1")) + traj = {"steps": [step("/gag/some-post-1/")]} + self.assertTrue(lib.detail_visited(traj, "some-post-1")) + + def test_paths_in_order(self) -> None: + traj = {"steps": [step("/login"), step("/home"), step("/settings")]} + self.assertTrue(lib.paths_in_order(traj, [("/login", {}), ("/settings", {})])) + self.assertFalse(lib.paths_in_order(traj, [("/settings", {}), ("/login", {})])) + + def test_site_url_accepts_loopback_only(self) -> None: + self.assertTrue(lib.is_site_url("http://localhost:40026/")) + self.assertTrue(lib.is_site_url("http://127.0.0.1:45001/x")) + self.assertFalse(lib.is_site_url("https://9gag.com/")) + self.assertFalse(lib.is_site_url("file:///tmp/x")) + + +class SeedContractTests(unittest.TestCase): + def test_frozen_seed_matches_pinned_contract(self) -> None: + self.assertEqual(lib.schema_sha256(SEED_DB), lib.SCHEMA_SHA256) + counts = {t: len(lib.table_rows(SEED_DB, t)) for t in lib.SEED_COUNTS} + self.assertEqual(counts, lib.SEED_COUNTS) + lib.validate_snapshot_contract(str(SEED_DB), str(SEED_DB)) + + def test_password_matches_real_werkzeug_hash(self) -> None: + for _, email, _ in USERS.values(): + row = lib.user_row(SEED_DB, email) + self.assertTrue(lib.password_matches(row["password_hash"], "TestPass123!"), email) + self.assertFalse(lib.password_matches(row["password_hash"], "testpass123!"), email) + self.assertFalse(lib.password_matches("garbage", "x")) + + def test_synthetic_user_hash_round_trips(self) -> None: + import tempfile + with tempfile.TemporaryDirectory() as d: + db = State().add_user("x@test.com", "x_user", "Secret123!").write(Path(d) / "a.db") + self.assertTrue(lib.password_matches(lib.user_row(db, "x@test.com")["password_hash"], "Secret123!")) + + def test_table_delta_reports_exact_rows(self) -> None: + import tempfile + with tempfile.TemporaryDirectory() as d: + initial = State().write(Path(d) / "i.db") + after = State().add_saved(1, 12).write(Path(d) / "a.db") + delta = lib.table_delta(initial, after, "saved_post") + self.assertEqual(len(delta["added"]), 1) + self.assertEqual(tuple(delta["added"][0][1:]), (1, 12)) + self.assertEqual(lib.changed_tables(initial, after), ["saved_post"]) + + +class PackageTests(unittest.TestCase): + def test_screenshots_decode_requires_every_referenced_png(self) -> None: + import tempfile + with tempfile.TemporaryDirectory() as d: + run = Path(d) / "run" + write_run(run, "9GAG--0", [step("/"), step("/x", "done")], "answer") + traj = lib.load_run(run) + self.assertTrue(lib.screenshots_decode(traj)[0]) + (run / "screenshots" / "step_001.png").unlink() + self.assertFalse(lib.screenshots_decode(lib.load_run(run))[0]) + + def test_no_llm_flag_disables_helpers(self) -> None: + lib.Judge("9GAG--0", no_llm=True) + self.assertEqual(lib.llm_text_match("a", "b", "q"), (False, "[skipped: --no_llm]")) + judge = lib.Judge("9GAG--0", no_llm=False) + lib.advisory_llm_answer(judge, "a", "b", "q") # no env configured -> no call, no evidence + self.assertEqual(judge.evidence, []) + + +if __name__ == "__main__": + unittest.main() diff --git a/sites/9gag/verify/verify_0.py b/sites/9gag/verify/verify_0.py new file mode 100644 index 00000000..4bd748e4 --- /dev/null +++ b/sites/9gag/verify/verify_0.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--0: search lighthouse offices -> best-ocean-view post -> room width + desk material (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_search_visited, + check_trajectory_identity, contains_all, contains_decimal, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--0" +SEARCH_TOKENS = ("lighthouse", "lighthouses", "office", "offices", "ocean") +# The original post and its "Community remix 2" clone carry the same facts; either detail page counts. +DETAIL_SLUGS = ("tiny-lighthouse-office-with-the-best-ocean-view-12", + "community-remix-2-tiny-lighthouse-office-with-the-best-ocean-view-47") +ROOM_WIDTH = "4.2" +DESK_MATERIAL = ("reclaimed", "oak") +QUESTION = "Report the room's width and the material used for the desk." + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_room_width_meters", contains_decimal(answer, ROOM_WIDTH), f"expected={ROOM_WIDTH!r}, answer={answer!r}") + judge.check("answer_has_desk_material", contains_all(answer, DESK_MATERIAL), f"expected={DESK_MATERIAL!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "4.2 meters wide; desk built from reclaimed oak", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_1.py b/sites/9gag/verify/verify_1.py new file mode 100644 index 00000000..a0623e0e --- /dev/null +++ b/sites/9gag/verify/verify_1.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--1: search rescue cat -> treat-drawer post -> arrival month + days to learn (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_search_visited, + check_trajectory_identity, contains_all, contains_count, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--1" +SEARCH_TOKENS = ("rescue", "cat", "cats", "treat", "drawer", "miso") +DETAIL_SLUGS = ("rescue-cat-learns-the-sound-of-the-treat-drawer-13", + "community-remix-3-rescue-cat-learns-the-sound-of-the-treat-drawer-48") +MONTH = "February" +DAYS = 11 +QUESTION = "Report the month Miso arrived at Harbor Paws and how many days the routine took to learn." + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_arrival_month", contains_all(answer, (MONTH,)), f"expected={MONTH!r}, answer={answer!r}") + judge.check("answer_has_days_count", contains_count(answer, DAYS), f"expected={DAYS!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "arrived in February; learned the routine after eleven days", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_10.py b/sites/9gag/verify/verify_10.py new file mode 100644 index 00000000..e7ab581e --- /dev/null +++ b/sites/9gag/verify/verify_10.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--10: Alice logs in, searches the lighthouse office post, opens the ORIGINAL and saves it (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_detail_visited, check_only_tables_changed, check_paths_in_order, check_search_visited, + check_signed_in_as, check_trajectory_identity, post_by_slug, run_verifier, saved_post_ids, table_delta, + user_id_for_email, +) + +TASK_ID = "9GAG--10" +EMAIL, USERNAME = "alice.j@test.com", "alice_j" +SEARCH_TOKENS = ("lighthouse", "lighthouses", "office", "offices", "ocean") +DETAIL_SLUG = "tiny-lighthouse-office-with-the-best-ocean-view-12" # the original, not the remix clone (id 47) + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + post = post_by_slug(initial_db, DETAIL_SLUG) + judge.check("seed_has_target_post", bool(post), f"slug={DETAIL_SLUG!r}") + post_id = int(post["id"]) if post else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUG, name="visited_original_post_detail") + check_paths_in_order(judge, traj, "workflow_login_before_save", [("/login", {}), (f"/gag/{DETAIL_SLUG}", {})]) + judge.check("initial_state_requires_action", post_id not in saved_post_ids(initial_db, user_id), + f"post_id={post_id} already saved initially? {post_id in saved_post_ids(initial_db, user_id)}") + delta = table_delta(initial_db, after_db, "saved_post") + exact = (len(delta["added"]) == 1 and not delta["removed"] and not delta["changed"] + and int(delta["added"][0][1]) == user_id and int(delta["added"][0][2]) == post_id) + judge.check("saved_post_exact_delta", exact, f"expected one added row (user_id={user_id}, post_id={post_id}); delta={delta!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("saved_post",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_11.py b/sites/9gag/verify/verify_11.py new file mode 100644 index 00000000..287bd165 --- /dev/null +++ b/sites/9gag/verify/verify_11.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--11: Carol logs in, browses Animals & Pets, opens the ORIGINAL kayak-dog post and upvotes it (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_detail_visited, check_only_tables_changed, check_paths_in_order, check_post_counter_delta, + check_signed_in_as, check_trajectory_identity, check_visited_path, post_by_slug, run_verifier, + table_delta, user_id_for_email, votes_for, +) + +TASK_ID = "9GAG--11" +EMAIL, USERNAME = "carol.d@test.com", "carol_d" +INTEREST_PATH = "/interest/animals" +DETAIL_SLUG = "dog-refuses-to-leave-the-kayak-after-the-trip-ends-20" # the original, not the remix clone (id 55) + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + post = post_by_slug(initial_db, DETAIL_SLUG) + judge.check("seed_has_target_post", bool(post), f"slug={DETAIL_SLUG!r}") + post_id = int(post["id"]) if post else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_animals_interest_feed", INTEREST_PATH) + check_detail_visited(judge, traj, DETAIL_SLUG, name="visited_original_post_detail") + check_paths_in_order(judge, traj, "workflow_login_browse_detail", + [("/login", {}), (INTEREST_PATH, {}), (f"/gag/{DETAIL_SLUG}", {})]) + judge.check("initial_state_requires_action", post_id not in votes_for(initial_db, user_id), + f"initial votes for user {user_id}: {votes_for(initial_db, user_id)!r}") + delta = table_delta(initial_db, after_db, "vote") + exact = (len(delta["added"]) == 1 and not delta["removed"] and not delta["changed"] + and int(delta["added"][0][1]) == user_id and int(delta["added"][0][2]) == post_id + and int(delta["added"][0][3]) == 1) + judge.check("vote_exact_delta", exact, f"expected one added upvote (user_id={user_id}, post_id={post_id}, value=1); delta={delta!r}") + check_post_counter_delta(judge, initial_db, after_db, post_id, up_votes=1) + check_only_tables_changed(judge, initial_db, after_db, allowed=("vote", "post")) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_12.py b/sites/9gag/verify/verify_12.py new file mode 100644 index 00000000..a56b814a --- /dev/null +++ b/sites/9gag/verify/verify_12.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--12: Bob logs in, finds the ORIGINAL humming-bridge post in Science & Tech and posts an exact comment (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_detail_visited, check_only_tables_changed, check_paths_in_order, check_post_counter_delta, + check_signed_in_as, check_trajectory_identity, check_visited_path, normalize_text, post_by_slug, + run_verifier, table_delta, user_id_for_email, +) + +TASK_ID = "9GAG--12" +EMAIL, USERNAME = "bob.c@test.com", "bob_c" +INTEREST_PATH = "/interest/science" +DETAIL_SLUG = "an-engineer-explains-why-this-bridge-hums-in-the-wind-19" # the original, not the remix clone (id 54) +COMMENT = "Resonance makes ordinary structures fascinating." + + +def _same_comment(body): + return normalize_text(body).rstrip(".") == normalize_text(COMMENT).rstrip(".") + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + post = post_by_slug(initial_db, DETAIL_SLUG) + judge.check("seed_has_target_post", bool(post), f"slug={DETAIL_SLUG!r}") + post_id = int(post["id"]) if post else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_science_interest_feed", INTEREST_PATH) + check_detail_visited(judge, traj, DETAIL_SLUG, name="visited_original_post_detail") + check_paths_in_order(judge, traj, "workflow_login_browse_detail", + [("/login", {}), (INTEREST_PATH, {}), (f"/gag/{DETAIL_SLUG}", {})]) + delta = table_delta(initial_db, after_db, "comment") + added = delta["added"] + exact = (len(added) == 1 and not delta["removed"] and not delta["changed"] + and int(added[0][1]) == post_id and int(added[0][2]) == user_id and _same_comment(added[0][3])) + judge.check("comment_exact_delta", exact, + f"expected one added comment (post_id={post_id}, user_id={user_id}, body={COMMENT!r}); delta={delta!r}") + check_post_counter_delta(judge, initial_db, after_db, post_id, comment_count=1) + check_only_tables_changed(judge, initial_db, after_db, allowed=("comment", "post")) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_13.py b/sites/9gag/verify/verify_13.py new file mode 100644 index 00000000..19672ffb --- /dev/null +++ b/sites/9gag/verify/verify_13.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--13: Carol logs in, searches the office-plant post, opens the ORIGINAL and hides it (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_detail_visited, check_only_tables_changed, check_paths_in_order, check_search_visited, + check_signed_in_as, check_trajectory_identity, hidden_post_ids, post_by_slug, run_verifier, table_delta, + user_id_for_email, +) + +TASK_ID = "9GAG--13" +EMAIL, USERNAME = "carol.d@test.com", "carol_d" +SEARCH_TOKENS = ("office", "plant", "badge", "employee", "kevin", "pothos") +DETAIL_SLUG = "the-office-plant-gets-an-employee-badge-26" # the original, not the remix clone (id 61) + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + post = post_by_slug(initial_db, DETAIL_SLUG) + judge.check("seed_has_target_post", bool(post), f"slug={DETAIL_SLUG!r}") + post_id = int(post["id"]) if post else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUG, name="visited_original_post_detail") + check_paths_in_order(judge, traj, "workflow_login_before_hide", [("/login", {}), (f"/gag/{DETAIL_SLUG}", {})]) + judge.check("initial_state_requires_action", post_id not in hidden_post_ids(initial_db, user_id), + f"initial hidden posts for user {user_id}: {sorted(hidden_post_ids(initial_db, user_id))!r}") + delta = table_delta(initial_db, after_db, "hidden_post") + exact = (len(delta["added"]) == 1 and not delta["removed"] and not delta["changed"] + and int(delta["added"][0][1]) == user_id and int(delta["added"][0][2]) == post_id) + judge.check("hidden_post_exact_delta", exact, f"expected one added row (user_id={user_id}, post_id={post_id}); delta={delta!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("hidden_post",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_14.py b/sites/9gag/verify/verify_14.py new file mode 100644 index 00000000..4ad0bd10 --- /dev/null +++ b/sites/9gag/verify/verify_14.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--14: David logs in, opens News, finds the 2009 press-conference post and reports it for Misinformation (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_detail_visited, check_only_tables_changed, check_paths_in_order, check_signed_in_as, + check_trajectory_identity, check_visited_path, normalize_text, post_by_slug, run_verifier, table_delta, + user_id_for_email, +) + +TASK_ID = "9GAG--14" +EMAIL, USERNAME = "david.k@test.com", "david_k" +NEWS_PATH = "/news" +DETAIL_SLUG = "a-mysterious-press-conference-moment-from-2009-6" +REASON = "Misinformation" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + post = post_by_slug(initial_db, DETAIL_SLUG) + judge.check("seed_has_target_post", bool(post), f"slug={DETAIL_SLUG!r}") + post_id = int(post["id"]) if post else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_news_feed", NEWS_PATH) + check_detail_visited(judge, traj, DETAIL_SLUG) + check_visited_path(judge, traj, "visited_report_form", f"/gag/{DETAIL_SLUG}/report") + check_paths_in_order(judge, traj, "workflow_login_news_detail_report", + [("/login", {}), (NEWS_PATH, {}), (f"/gag/{DETAIL_SLUG}", {}), (f"/gag/{DETAIL_SLUG}/report", {})]) + delta = table_delta(initial_db, after_db, "report") + added = delta["added"] + exact = (len(added) == 1 and not delta["removed"] and not delta["changed"] + and int(added[0][1]) == user_id and int(added[0][2]) == post_id + and normalize_text(added[0][3]) == normalize_text(REASON)) + judge.check("report_exact_delta", exact, + f"expected one added report (user_id={user_id}, post_id={post_id}, reason={REASON!r}); delta={delta!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("report",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_15.py b/sites/9gag/verify/verify_15.py new file mode 100644 index 00000000..400480c1 --- /dev/null +++ b/sites/9gag/verify/verify_15.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--15: Alice logs in and publishes a new Wholesome post with the exact title/description/tags (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_only_tables_changed, check_paths_in_order, check_signed_in_as, check_trajectory_identity, + check_visited_path, normalize_text, row_dict, run_verifier, table_delta, +) + +TASK_ID = "9GAG--15" +EMAIL, USERNAME = "alice.j@test.com", "alice_j" +TITLE = "Quiet victories deserve confetti" +SECTION = "wholesome" +DESCRIPTION = "My neighbor finished night school after six years." +TAGS = "community|education|wholesome" +NEW_SLUG = "quiet-victories-deserve-confetti" # app.slugify(TITLE); the publish redirect lands on /gag/ + + +def _tags_match(value): + return [t.strip().casefold() for t in str(value or "").split("|") if t.strip()] == TAGS.split("|") + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_submit_form", "/submit") + check_visited_path(judge, traj, "visited_published_post", f"/gag/{NEW_SLUG}") + check_paths_in_order(judge, traj, "workflow_login_submit_publish", + [("/login", {}), ("/submit", {}), (f"/gag/{NEW_SLUG}", {})]) + delta = table_delta(initial_db, after_db, "post") + added = [row_dict(after_db, "post", r) for r in delta["added"]] + exact = len(added) == 1 and not delta["removed"] and not delta["changed"] + if exact: + new = added[0] + exact = (normalize_text(new["title"]) == normalize_text(TITLE) + and normalize_text(new["section"]) == SECTION + and normalize_text(new["description"]) == normalize_text(DESCRIPTION) + and _tags_match(new["tags"]) + and str(new["author_name"]) == USERNAME + and str(new["slug"]) == NEW_SLUG) + judge.check("new_post_exact_delta", exact, + f"expected exactly one new post by {USERNAME!r}: title={TITLE!r}, section={SECTION!r}, " + f"description={DESCRIPTION!r}, tags={TAGS!r}; added={added!r}, removed={delta['removed']!r}, changed={delta['changed']!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("post",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_16.py b/sites/9gag/verify/verify_16.py new file mode 100644 index 00000000..ff329fe1 --- /dev/null +++ b/sites/9gag/verify/verify_16.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--16: Alice logs in and updates display name, bio and location in Settings (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_only_tables_changed, check_paths_in_order, check_signed_in_as, check_trajectory_identity, + check_visited_path, normalize_text, run_verifier, table_delta, user_id_for_email, user_row, +) + +TASK_ID = "9GAG--16" +EMAIL, USERNAME = "alice.j@test.com", "alice_j" +DISPLAY_NAME = "Alice J." +BIO = "Memes, trail photos, and excellent tiny libraries." +LOCATION = "Tacoma, WA" +FROZEN_COLUMNS = ("id", "username", "email", "password_hash", "joined_at") + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_settings_page", "/settings") + check_paths_in_order(judge, traj, "workflow_login_before_settings", [("/login", {}), ("/settings", {})]) + before, after = user_row(initial_db, EMAIL) or {}, user_row(after_db, EMAIL) or {} + judge.check("initial_profile_requires_update", + normalize_text(before.get("display_name")) != normalize_text(DISPLAY_NAME) + or normalize_text(before.get("location")) != normalize_text(LOCATION), + f"initial display_name={before.get('display_name')!r}, location={before.get('location')!r}") + judge.check("profile_fields_updated", + normalize_text(after.get("display_name")) == normalize_text(DISPLAY_NAME) + and normalize_text(after.get("bio")) == normalize_text(BIO) + and normalize_text(after.get("location")) == normalize_text(LOCATION), + f"expected display_name={DISPLAY_NAME!r}, bio={BIO!r}, location={LOCATION!r}; " + f"after display_name={after.get('display_name')!r}, bio={after.get('bio')!r}, location={after.get('location')!r}") + delta = table_delta(initial_db, after_db, "user") + exact = (not delta["added"] and not delta["removed"] and len(delta["changed"]) == 1 + and int(delta["changed"][0][0][0]) == user_id + and all(before.get(c) == after.get(c) for c in FROZEN_COLUMNS)) + judge.check("user_exact_delta", exact, f"expected only user {user_id}'s profile columns to change; delta={delta!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("user",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_17.py b/sites/9gag/verify/verify_17.py new file mode 100644 index 00000000..5e324df9 --- /dev/null +++ b/sites/9gag/verify/verify_17.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--17: register river_reader and confirm the signed-in username in the header (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_only_tables_changed, check_trajectory_identity, check_visited_path, contains_all, + final_answer, normalize_text, password_matches, row_dict, run_verifier, table_delta, user_row, +) + +TASK_ID = "9GAG--17" +EMAIL = "river.reader@test.com" +USERNAME = "river_reader" +PASSWORD = "RiverRead123!" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + answer = final_answer(traj) + check_visited_path(judge, traj, "visited_register_page", "/register") + judge.check("initial_state_requires_action", user_row(initial_db, EMAIL) is None, f"user {EMAIL!r} absent initially") + delta = table_delta(initial_db, after_db, "user") + added = [row_dict(after_db, "user", r) for r in delta["added"]] + exact = len(added) == 1 and not delta["removed"] and not delta["changed"] + if exact: + new = added[0] + exact = (normalize_text(new["email"]) == EMAIL and str(new["username"]) == USERNAME) + judge.check("new_user_exact_delta", exact, + f"expected exactly one new user email={EMAIL!r}, username={USERNAME!r}; added={[(a['email'], a['username']) for a in added]!r}, " + f"removed={delta['removed']!r}, changed={delta['changed']!r}") + judge.check("new_user_password_verifies", exact and password_matches(added[0]["password_hash"], PASSWORD), + f"password hash for {USERNAME!r} verifies {PASSWORD!r}") + judge.check("answer_confirms_header_username", contains_all(answer, (USERNAME,)), f"expected={USERNAME!r}, answer={answer!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("user",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_18.py b/sites/9gag/verify/verify_18.py new file mode 100644 index 00000000..eb6141a1 --- /dev/null +++ b/sites/9gag/verify/verify_18.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--18: Alice logs in, opens Saved and removes the ORIGINAL solar camping post from her collection (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_only_tables_changed, check_paths_in_order, check_signed_in_as, check_trajectory_identity, + check_visited_path, post_by_slug, run_verifier, saved_post_ids, table_delta, user_id_for_email, +) + +TASK_ID = "9GAG--18" +EMAIL, USERNAME = "alice.j@test.com", "alice_j" +DETAIL_SLUG = "solar-powered-camping-setup-survives-a-rainy-weekend-11" # seeded in Alice's saved collection + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + post = post_by_slug(initial_db, DETAIL_SLUG) + judge.check("seed_has_target_post", bool(post), f"slug={DETAIL_SLUG!r}") + post_id = int(post["id"]) if post else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_saved_collection", "/saved") + check_paths_in_order(judge, traj, "workflow_login_before_saved", [("/login", {}), ("/saved", {})]) + judge.check("initial_state_requires_action", post_id in saved_post_ids(initial_db, user_id), + f"post_id={post_id} saved initially for user {user_id}? {post_id in saved_post_ids(initial_db, user_id)}") + delta = table_delta(initial_db, after_db, "saved_post") + exact = (len(delta["removed"]) == 1 and not delta["added"] and not delta["changed"] + and int(delta["removed"][0][1]) == user_id and int(delta["removed"][0][2]) == post_id) + judge.check("saved_post_exact_delta", exact, f"expected one removed row (user_id={user_id}, post_id={post_id}); delta={delta!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("saved_post",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_19.py b/sites/9gag/verify/verify_19.py new file mode 100644 index 00000000..716fc2e7 --- /dev/null +++ b/sites/9gag/verify/verify_19.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--19: David logs in, compares three ORIGINAL Science & Tech posts by their stated numbers, saves the greatest (stateful). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + check_detail_visited, check_only_tables_changed, check_paths_in_order, check_signed_in_as, + check_trajectory_identity, check_visited_path, post_by_slug, run_verifier, saved_post_ids, table_delta, + user_id_for_email, +) + +TASK_ID = "9GAG--19" +EMAIL, USERNAME = "david.k@test.com", "david_k" +INTEREST_PATH = "/interest/science" +# The comparison values live ONLY in the detail descriptions, so every detail page must be opened. +COMPARED = { + "solar-powered-camping-setup-survives-a-rainy-weekend-11": 120, # 120-watt panel + "an-engineer-explains-why-this-bridge-hums-in-the-wind-19": 440, # 440 hertz + "students-launch-a-weather-balloon-with-a-tiny-rubber-duck-25": 27, # 27 kilometers +} +WINNER_SLUG = "an-engineer-explains-why-this-bridge-hums-in-the-wind-19" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_compared_posts", all(post_by_slug(initial_db, s) for s in COMPARED), f"slugs={list(COMPARED)!r}") + winner = post_by_slug(initial_db, WINNER_SLUG) + post_id = int(winner["id"]) if winner else -1 + user_id = user_id_for_email(initial_db, EMAIL) + check_signed_in_as(judge, traj, EMAIL, USERNAME) + check_visited_path(judge, traj, "visited_science_interest_feed", INTEREST_PATH) + for slug in COMPARED: + check_detail_visited(judge, traj, slug, name=f"visited_compared_detail_{slug.rsplit('-', 1)[1]}") + check_paths_in_order(judge, traj, "workflow_login_before_browse", [("/login", {}), (INTEREST_PATH, {})]) + judge.check("initial_state_requires_action", post_id not in saved_post_ids(initial_db, user_id), + f"post_id={post_id} already saved initially? {post_id in saved_post_ids(initial_db, user_id)}") + delta = table_delta(initial_db, after_db, "saved_post") + exact = (len(delta["added"]) == 1 and not delta["removed"] and not delta["changed"] + and int(delta["added"][0][1]) == user_id and int(delta["added"][0][2]) == post_id) + judge.check("saved_post_exact_delta", exact, + f"expected one added row (user_id={user_id}, post_id={post_id} = the greatest stated number); delta={delta!r}") + check_only_tables_changed(judge, initial_db, after_db, allowed=("saved_post",)) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_2.py b/sites/9gag/verify/verify_2.py new file mode 100644 index 00000000..5f88fd1e --- /dev/null +++ b/sites/9gag/verify/verify_2.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--2: search solar camping -> rainy-weekend setup -> panel wattage + controller protection (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_search_visited, + check_trajectory_identity, contains_count, contains_phrase, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--2" +SEARCH_TOKENS = ("solar", "camping", "camp", "rainy", "panel") +DETAIL_SLUGS = ("solar-powered-camping-setup-survives-a-rainy-weekend-11", + "community-remix-1-solar-powered-camping-setup-survives-a-rainy-weekend-46") +WATTS = 120 +PROTECTION = "lunch box" +QUESTION = "What wattage was its folding panel, and what protected the controller from water?" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_panel_wattage", contains_count(answer, WATTS), f"expected={WATTS!r}, answer={answer!r}") + judge.check("answer_has_controller_protection", contains_phrase(answer, PROTECTION), f"expected={PROTECTION!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "120-watt folding panel; controller kept in a waterproof lunch box", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_3.py b/sites/9gag/verify/verify_3.py new file mode 100644 index 00000000..094f5917 --- /dev/null +++ b/sites/9gag/verify/verify_3.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--3: browse Science & Tech -> humming-bridge post -> frequency + resonating feature (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_trajectory_identity, + check_visited_path, contains_any, contains_count, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--3" +INTEREST_PATH = "/interest/science" +DETAIL_SLUGS = ("an-engineer-explains-why-this-bridge-hums-in-the-wind-19", + "community-remix-9-an-engineer-explains-why-this-bridge-hums-in-the-wind-54") +HERTZ = 440 +FEATURE = ("railing", "railings") +QUESTION = "Report the frequency of the note and the bridge feature that produces it." + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_visited_path(judge, traj, "visited_science_interest_feed", INTEREST_PATH) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_frequency_hertz", contains_count(answer, HERTZ), f"expected={HERTZ!r}, answer={answer!r}") + judge.check("answer_has_resonating_feature", contains_any(answer, FEATURE), f"expected one of {FEATURE!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "near 440 hertz; produced by evenly spaced railings", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_4.py b/sites/9gag/verify/verify_4.py new file mode 100644 index 00000000..75750480 --- /dev/null +++ b/sites/9gag/verify/verify_4.py @@ -0,0 +1,42 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--4: search community library -> night-shift workers post -> cabinet colour + restock time (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_search_visited, + check_trajectory_identity, contains_all, contains_clock_time, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--4" +SEARCH_TOKENS = ("library", "libraries", "community", "night", "shift", "cabinet") +DETAIL_SLUGS = ("neighborhood-builds-a-miniature-library-for-night-shift-workers-18", + "community-remix-8-neighborhood-builds-a-miniature-library-for-night-shift-workers-53") +COLOUR = "blue" +DAY = "Thursday" +TIME = "6 am" +QUESTION = "What color is the cabinet, and exactly when is it restocked?" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_cabinet_colour", contains_all(answer, (COLOUR,)), f"expected={COLOUR!r}, answer={answer!r}") + judge.check("answer_has_restock_day", contains_all(answer, (DAY,)), f"expected={DAY!r}, answer={answer!r}") + judge.check("answer_has_restock_time", contains_clock_time(answer, TIME), f"expected={TIME!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "blue cabinet; restocked every Thursday at 6 a.m.", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_5.py b/sites/9gag/verify/verify_5.py new file mode 100644 index 00000000..f3989868 --- /dev/null +++ b/sites/9gag/verify/verify_5.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--5: browse Sports -> grandmother marathon post -> meeting kilometre + flag colour (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_trajectory_identity, + check_visited_path, contains_all, contains_count, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--5" +INTEREST_PATH = "/interest/sports" +DETAIL_SLUGS = ("grandmother-finishes-her-first-marathon-at-seventy-two-15", + "community-remix-5-grandmother-finishes-her-first-marathon-at-seventy-two-50") +KILOMETRE = 38 +COLOUR = "orange" +QUESTION = "At which kilometer did her family meet her, and what color were their flags?" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_visited_path(judge, traj, "visited_sports_interest_feed", INTEREST_PATH) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_kilometre", contains_count(answer, KILOMETRE), f"expected={KILOMETRE!r}, answer={answer!r}") + judge.check("answer_has_flag_colour", contains_all(answer, (COLOUR,)), f"expected={COLOUR!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "kilometer 38; handmade orange flags", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_6.py b/sites/9gag/verify/verify_6.py new file mode 100644 index 00000000..ef411153 --- /dev/null +++ b/sites/9gag/verify/verify_6.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--6: search sourdough skyline -> the baker's post -> attempts + shaping time (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_search_visited, + check_trajectory_identity, contains_count, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--6" +SEARCH_TOKENS = ("sourdough", "skyline", "baker", "bread") +# The original and its "Community remix 6" clone (id 51) carry the identical description, so either +# detail page answers the question. Requiring the original would fail a correct answer on a hidden +# provenance rule; provenance is only enforced where it changes the graded outcome (9, 10-13, 18, 19). +DETAIL_SLUGS = ("a-baker-recreates-a-city-skyline-in-sourdough-16", + "community-remix-6-a-baker-recreates-a-city-skyline-in-sourdough-51") +ATTEMPTS = 3 +HOURS = 14 +QUESTION = "Report both the number of attempts and the approximate shaping time." + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_attempt_count", contains_count(answer, ATTEMPTS), f"expected={ATTEMPTS!r}, answer={answer!r}") + judge.check("answer_has_shaping_hours", contains_count(answer, HOURS), f"expected={HOURS!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "three attempts; nearly fourteen hours to shape", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_7.py b/sites/9gag/verify/verify_7.py new file mode 100644 index 00000000..46ebcb05 --- /dev/null +++ b/sites/9gag/verify/verify_7.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--7: browse Gaming -> transparent keyboard post -> switch type + case material (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_trajectory_identity, + check_visited_path, contains_all, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--7" +INTEREST_PATH = "/interest/gaming" +DETAIL_SLUGS = ("mechanical-keyboard-made-entirely-from-transparent-parts-14", + "community-remix-4-mechanical-keyboard-made-entirely-from-transparent-parts-49") +SWITCHES = ("silent", "tactile") +CASE = "acrylic" +QUESTION = "What kind of switches and what case material does it use?" + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_visited_path(judge, traj, "visited_gaming_interest_feed", INTEREST_PATH) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_switch_type", contains_all(answer, SWITCHES), f"expected={SWITCHES!r}, answer={answer!r}") + judge.check("answer_has_case_material", contains_all(answer, (CASE,)), f"expected={CASE!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "silent tactile switches; hand-polished acrylic case", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_8.py b/sites/9gag/verify/verify_8.py new file mode 100644 index 00000000..62eb4cc1 --- /dev/null +++ b/sites/9gag/verify/verify_8.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--8: search rain-delay concert -> the street-musician post -> platform number (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_search_visited, + check_trajectory_identity, contains_count, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--8" +SEARCH_TOKENS = ("rain", "delay", "concert", "musician", "street", "platform", "chorus") +# The original and its "Community remix 12" clone (id 57) carry the identical description, so either +# detail page answers the question. Requiring the original would fail a correct answer on a hidden +# provenance rule; provenance is only enforced where it changes the graded outcome (9, 10-13, 18, 19). +DETAIL_SLUGS = ("street-musician-turns-a-rain-delay-into-a-concert-22", + "community-remix-12-street-musician-turns-a-rain-delay-into-a-concert-57") +PLATFORM = 7 +QUESTION = "Report the platform number where commuters joined the chorus." + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + judge.check("seed_has_target_posts", all(post_by_slug(initial_db, s) for s in DETAIL_SLUGS), f"slugs={DETAIL_SLUGS!r}") + answer = final_answer(traj) + check_search_visited(judge, traj, SEARCH_TOKENS) + check_detail_visited(judge, traj, DETAIL_SLUGS) + judge.check("answer_has_platform_number", contains_count(answer, PLATFORM), f"expected={PLATFORM!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "platform seven", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_9.py b/sites/9gag/verify/verify_9.py new file mode 100644 index 00000000..5258d52d --- /dev/null +++ b/sites/9gag/verify/verify_9.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python3 +"""Verify 9GAG--9: browse Animals & Pets, compare the three ORIGINAL posts' points -> open the winner -> animal name (read-only). + +Deterministic only. Ground truth is hardcoded here and never appears in tasks.jsonl. +""" +from __future__ import annotations + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) + +from verify_lib import ( # noqa: E402 + advisory_llm_answer, check_detail_visited, check_read_only, check_trajectory_identity, + check_visited_path, contains_all, final_answer, post_by_slug, run_verifier, +) + +TASK_ID = "9GAG--9" +INTEREST_PATH = "/interest/animals" +CANDIDATES = { # original posts only; the remix clones carry different point counts + "rescue-cat-learns-the-sound-of-the-treat-drawer-13": 2506, + "dog-refuses-to-leave-the-kayak-after-the-trip-ends-20": 3717, + "a-fox-naps-on-the-same-garden-wall-every-afternoon-23": 4236, +} +WINNER_SLUG = "a-fox-naps-on-the-same-garden-wall-every-afternoon-23" +ANIMAL_NAME = "Copper" +QUESTION = "Which of the three original posts has the most points? Open it and report the animal's name." + + +def run_checks(judge, traj, initial_db, after_db): + check_trajectory_identity(judge, traj, TASK_ID) + seed_points = {s: (post_by_slug(initial_db, s) or {}).get("up_votes") for s in CANDIDATES} + judge.check("seed_points_match_frozen_ground_truth", seed_points == CANDIDATES, + f"expected={CANDIDATES!r}, observed={seed_points!r}") + answer = final_answer(traj) + check_visited_path(judge, traj, "visited_animals_interest_feed", INTEREST_PATH) + check_detail_visited(judge, traj, WINNER_SLUG, name="visited_winning_post_detail") + judge.check("answer_has_animal_name", contains_all(answer, (ANIMAL_NAME,)), f"expected={ANIMAL_NAME!r}, answer={answer!r}") + check_read_only(judge, initial_db, after_db) + advisory_llm_answer(judge, answer, "the garden-fox post (most points); the fox is named Copper", QUESTION) + + +if __name__ == "__main__": + run_verifier(TASK_ID, run_checks) diff --git a/sites/9gag/verify/verify_lib.py b/sites/9gag/verify/verify_lib.py new file mode 100644 index 00000000..dcb6ba3e --- /dev/null +++ b/sites/9gag/verify/verify_lib.py @@ -0,0 +1,813 @@ +#!/usr/bin/env python3 +"""verify_lib.py — shared deterministic (+ anchored, advisory-only LLM) utilities for 9GAG task verification. + +Philosophy: DETERMINISTIC FIRST — same contract as ``sites/merriam_webster/verify/verify_lib.py``, +hardened the way ``sites/walmart_careers/verify/verify_lib.py`` is. + + 1. Package identity: task_id matches, ``terminated`` with ``agent_done``, non-empty final answer, + every recorded URL on the same loopback origin AND port as ``start_url``, every referenced + screenshot a decodable PNG. + 2. Navigation gates (anti knowledge-shortcut): the agent MUST have opened the on-site pages the + task names (``/search?q=`` results, an ``/interest/`` feed, the ``/gag/`` detail page). + A correct answer with no matching navigation is a memory-recall shortcut = FAIL. + 3. Answer check: affirmative token / number / clock matching against frozen ground truth that is + HARDCODED in each ``verify_N.py`` (never in ``tasks.jsonl``). + 4. DB after-state: initial (seed) vs after (instance) SQLite snapshots. Read-only tasks require every + table row-identical; stateful tasks require the exact allowed row delta and nothing else. + 5. LLM utilities (``llm_text_match`` / ``llm_screenshot_shows``) are kept for parity with the other + suites. They are anchored on ground truth, make one call each, and are NEVER load-bearing: + every verdict is decided with ``--no_llm True`` and the helpers only add ``[INFO]`` evidence. + +Input signature (per task): + --run_dir DIR agent trajectory dir: trajectory.json + screenshots/step_NNN.png + --initial_db PATH initial-state SQLite DB (default: /initial.db, else instance_seed from the container) + --after_db PATH after-state SQLite DB (default: /after.db, else live instance DB from the container) + --container NAME docker container to fetch DBs from (default: $WH_CONTAINER or wh-review) + --no_llm True skip the advisory LLM evidence (verdicts never depend on it) +Output: JSON {task_id, pass, reason, evidence[]} to stdout; exit 0 on PASS, 1 on FAIL. +""" +# NOTE: no `from __future__ import annotations` here — simpleArgParser reads the dataclass +# field types at runtime and needs real types, not strings. +import atexit +import base64 +import hashlib +import hmac +import ipaddress +import json +import os +import re +import sqlite3 +import subprocess +import sys +import tempfile +import unicodedata +import urllib.request +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Iterable, Sequence +from urllib.parse import parse_qs, urlparse + +SITE = "9gag" +DEFAULT_CONTAINER = os.environ.get("WH_CONTAINER", "wh-review") + +# ---------------------------------------------------------------- frozen seed contract +TABLES = ("user", "post", "comment", "vote", "saved_post", "hidden_post", "report") +SEED_COUNTS = {"user": 4, "post": 80, "comment": 4, "vote": 12, "saved_post": 16, "hidden_post": 0, "report": 0} +# sha256 over sqlite_master (type, name, tbl_name, sql) of instance_seed/9gag.db (md5 44768940829bf4a8bec07d7bdf072d97). +SCHEMA_SHA256 = "498bc17a4f2750edcdf35a8a577ea1a86fba16bc379f21ce4e012479c345422a" +POST_CATALOG_COLUMNS = ("id", "source_id", "slug", "title", "description", "image", "section", + "post_type", "tags", "author_name", "created_rank", "featured") +POST_COUNTER_COLUMNS = ("up_votes", "down_votes", "comment_count") +SEED_USERS = { # email -> (id, username); identity columns never change + "alice.j@test.com": (1, "alice_j"), + "bob.c@test.com": (2, "bob_c"), + "carol.d@test.com": (3, "carol_d"), + "david.k@test.com": (4, "david_k"), +} +INPUT_ACTIONS = {"input", "type", "fill", "input_text", "type_text"} + + +# ---------------------------------------------------------------- trajectory +def load_run(run_dir): + d = Path(run_dir) + traj = json.loads((d / "trajectory.json").read_text(encoding="utf-8")) + if not isinstance(traj, dict): + raise ValueError("trajectory.json must contain a JSON object") + traj["_run_dir"] = d + shots_dir = d / "screenshots" + traj["_shots"] = {p.name: p for p in sorted(shots_dir.glob("step_*.png"))} if shots_dir.is_dir() else {} + return traj + + +def step_urls(traj): + return [str(s.get("url", "")) for s in traj.get("steps", []) if isinstance(s, dict)] + + +def trajectory_urls(traj): + """Every browser URL the recorder wrote: start_url, each step's url (+url_before/url_after), final_url.""" + urls: list[str] = [] + if traj.get("start_url"): + urls.append(str(traj["start_url"])) + for step in traj.get("steps") or []: + if not isinstance(step, dict): + continue + for key in ("url", "url_before", "url_after"): + if step.get(key): + urls.append(str(step[key])) + if traj.get("final_url"): + urls.append(str(traj["final_url"])) + return urls + + +def navigated_to(traj, substr, times=1): + """Deterministic (merriam parity): at least `times` recorded on-site URLs contain substr.""" + return sum(1 for u in site_urls(traj) if substr in u) >= times + + +def navigated_any(traj, substrs): + return any(navigated_to(traj, s) for s in substrs) + + +def final_answer(traj): + return str(traj.get("final_answer") or "").strip() + + +def final_url(traj): + if traj.get("final_url"): + return str(traj["final_url"]) + for step in reversed(traj.get("steps") or []): + if isinstance(step, dict) and step.get("url"): + return str(step["url"]) + return "" + + +def _shot(traj, name): + if not name: + return None + p = traj["_shots"].get(Path(str(name)).name) + return p if (p and p.exists()) else None + + +def shot_after_url(traj, substr): + for s in traj.get("steps", []): + if isinstance(s, dict) and substr in str(s.get("url", "")): + p = _shot(traj, s.get("screenshot_after")) + if p: + return p + return None + + +def last_shot(traj): + for s in reversed(traj.get("steps", [])): + if not isinstance(s, dict): + continue + p = _shot(traj, s.get("screenshot_after")) or _shot(traj, s.get("screenshot_before")) + if p: + return p + shots = sorted(traj["_shots"].values()) + return shots[-1] if shots else None + + +def normalized_url_path(url): + path = urlparse(str(url or "")).path or "/" + return path.rstrip("/") or "/" + + +def is_site_url(url): + """HTTP(S) URL on a loopback host (any port: alt-port containers are legitimate).""" + parsed = urlparse(str(url or "")) + if parsed.scheme not in {"http", "https"} or not parsed.hostname: + return False + host = parsed.hostname.casefold() + if host == "localhost": + return True + try: + return ipaddress.ip_address(host).is_loopback + except ValueError: + return False + + +def site_urls(traj): + return [u for u in trajectory_urls(traj) if is_site_url(u)] + + +def navigated_to_path(traj, expected_path): + expected = normalized_url_path(expected_path) + return any(normalized_url_path(u) == expected for u in site_urls(traj)) + + +def detail_visited(traj, slug): + """Exact /gag/ visit (never /gag//report or /gag//vote).""" + return navigated_to_path(traj, f"/gag/{slug}") + + +def any_detail_visited(traj, slugs): + return any(detail_visited(traj, s) for s in slugs) + + +def _tokens(text): + return set(re.findall(r"[a-z0-9]+", normalize_text(text))) + + +def search_queries(traj): + out = [] + for u in site_urls(traj): + if normalized_url_path(u) == "/search": + for q in parse_qs(urlparse(u).query, keep_blank_values=True).get("q") or []: + out.append(q) + return out + + +def search_visited(traj, tokens_any): + """Some /search?q= visit whose query contains at least one of `tokens_any` as a whole token.""" + wanted = {normalize_text(t) for t in tokens_any} + return any(_tokens(q) & wanted for q in search_queries(traj)) + + +def input_texts(traj): + values = [] + for step in traj.get("steps") or []: + if not isinstance(step, dict) or normalize_text(step.get("action")) not in INPUT_ACTIONS: + continue + params = step.get("params") + if isinstance(params, dict) and params.get("text") is not None: + values.append(str(params["text"])) + return values + + +def entered_identity(traj, *identities): + wanted = {normalize_text(i) for i in identities} + return any(normalize_text(v) in wanted for v in input_texts(traj)) + + +def paths_in_order(traj, requirements): + """True when the (path, {param: value}) requirements appear in the recorded URLs in that order.""" + urls = site_urls(traj) + cursor = 0 + for expected_path, params in requirements: + expected = normalized_url_path(expected_path) + for index in range(cursor, len(urls)): + u = urls[index] + query = parse_qs(urlparse(u).query, keep_blank_values=True) + if normalized_url_path(u) == expected and all( + any(normalize_text(v) == normalize_text(value) for v in query.get(key) or []) + for key, value in params.items() + ): + cursor = index + 1 + break + else: + return False + return True + + +# ---------------------------------------------------------------- deterministic answer match +def normalize_text(value): + text = unicodedata.normalize("NFKC", str(value or "")) + text = text.replace("’", "'").replace("‘", "'").replace("“", '"').replace("”", '"') + return re.sub(r"\s+", " ", text).strip().casefold() + + +norm = normalize_text # merriam parity + + +def _match_is_affirmative(text, match): + before = re.split(r"[.!?;:,\n]+|\b(?:but|however|instead)\b", text[: match.start()], flags=re.I)[-1] + after = text[match.end():] + return not re.search(r"\b(?:not|no|never|without|wrong|incorrect|isn't|wasn't|isnt|wasnt|rather than)\b", before, re.I) \ + and not re.match(r"\s*(?:is|was|are|were)?\s*(?:not|wrong|incorrect)\b", after, re.I) + + +def _affirmative_search(pattern, text, flags=0): + return any(_match_is_affirmative(text, m) for m in re.finditer(pattern, text, flags)) + + +def answer_equals(final, expected): + return normalize_text(final) == normalize_text(expected) + + +def contains_all(text, tokens): + normalized = normalize_text(text) + return all(bool(t) and _affirmative_search(r"(? temp file.""" + if kind not in {"instance", "instance_seed"}: + raise ValueError(f"unsupported DB kind: {kind}") + src = f"{container}:/opt/WebSyn/{SITE}/{kind}/{SITE}.db" + fd, path = tempfile.mkstemp(prefix=f"{SITE}_{kind}_", suffix=".db") + os.close(fd) + r = subprocess.run(["docker", "cp", src, path], capture_output=True, text=True) + if r.returncode != 0: + Path(path).unlink(missing_ok=True) + raise RuntimeError(f"docker cp {src} failed: {r.stderr.strip() or r.stdout.strip()}") + atexit.register(Path(path).unlink, missing_ok=True) + return path + + +def resolve_db(arg, container, kind): + if arg: + return str(arg) if Path(arg).is_file() else None + try: + return fetch_db(container, kind) + except (OSError, RuntimeError): + return None # caller treats None as "unavailable" and fails closed + + +def table_columns(db_path, table): + return [row["name"] for row in db_query(db_path, f"PRAGMA table_info({table})")] + + +def table_rows(db_path, table): + return [tuple(row) for row in db_query(db_path, f"SELECT * FROM {table} ORDER BY id")] + + +def rows_by_id(db_path, table): + return {int(row[0]): row for row in table_rows(db_path, table)} + + +def table_delta(initial_db, after_db, table): + before, after = rows_by_id(initial_db, table), rows_by_id(after_db, table) + return { + "added": [after[i] for i in sorted(set(after) - set(before))], + "removed": [before[i] for i in sorted(set(before) - set(after))], + "changed": [(before[i], after[i]) for i in sorted(set(before) & set(after)) if before[i] != after[i]], + } + + +def changed_tables(initial_db, after_db, tables=TABLES): + return [t for t in tables if table_rows(initial_db, t) != table_rows(after_db, t)] + + +def row_dict(db_path, table, row): + return dict(zip(table_columns(db_path, table), row)) + + +def _schema_objects(db_path): + return [tuple(r) for r in db_query( + db_path, + "SELECT type, name, tbl_name, sql FROM sqlite_master " + "WHERE sql IS NOT NULL AND name NOT LIKE 'sqlite_%' ORDER BY type, name")] + + +def schema_sha256(db_path): + return hashlib.sha256(json.dumps(_schema_objects(db_path), separators=(",", ":")).encode()).hexdigest() + + +def validate_snapshot_contract(initial_db, after_db): + """Raise ValueError unless both snapshots are genuine 9GAG databases derived from the frozen seed.""" + for db_path in (initial_db, after_db): + tables = {r["name"] for r in db_query(db_path, "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'")} + if tables != set(TABLES): + raise ValueError(f"unexpected tables in {db_path}: {sorted(tables)}") + if _schema_objects(initial_db) != _schema_objects(after_db): + raise ValueError("initial and after database schemas differ") + observed_hash = schema_sha256(initial_db) + if observed_hash != SCHEMA_SHA256: + raise ValueError(f"unsupported 9GAG schema hash: {observed_hash}") + counts = {t: len(table_rows(initial_db, t)) for t in SEED_COUNTS} + if counts != SEED_COUNTS: + raise ValueError(f"initial database counts differ: expected={SEED_COUNTS}, observed={counts}") + users = {r["email"]: (int(r["id"]), r["username"]) for r in db_query(initial_db, "SELECT id, username, email FROM user")} + if users != SEED_USERS: + raise ValueError(f"initial benchmark users differ: {users}") + # seed users keep their identity columns; seed posts keep every catalog column + after_users = {r["email"]: (int(r["id"]), r["username"]) for r in db_query(after_db, "SELECT id, username, email FROM user")} + if any(after_users.get(email) != ident for email, ident in SEED_USERS.items()): + raise ValueError("benchmark user identities changed in the after snapshot") + cols = ", ".join(POST_CATALOG_COLUMNS) + before = [tuple(r) for r in db_query(initial_db, f"SELECT {cols} FROM post ORDER BY id")] + after = [tuple(r) for r in db_query(after_db, f"SELECT {cols} FROM post WHERE id <= 80 ORDER BY id")] + if before != after: + raise ValueError("seed post catalog columns changed in the after snapshot") + + +def resolve_snapshots(args, task_id): + """Return validated (initial_db, after_db) or fail closed.""" + initial_db = resolve_db(args.initial_db, args.container, "instance_seed") + after_db = resolve_db(args.after_db, args.container, "instance") + if not initial_db or not after_db: + fail_closed(task_id, "database_unavailable", + "both initial (seed) and after (instance) 9gag database snapshots are required") + try: + validate_snapshot_contract(initial_db, after_db) + except (OSError, sqlite3.Error, ValueError) as exc: + fail_closed(task_id, "snapshot_contract_invalid", str(exc)) + return str(initial_db), str(after_db) + + +def user_id_for_email(db_path, email): + rows = db_query(db_path, "SELECT id FROM user WHERE lower(email) = lower(?) ORDER BY id LIMIT 1", (email,)) + return int(rows[0]["id"]) if rows else None + + +def user_row(db_path, email): + rows = db_query(db_path, "SELECT * FROM user WHERE lower(email) = lower(?) ORDER BY id LIMIT 1", (email,)) + return dict(rows[0]) if rows else None + + +def post_row(db_path, post_id): + rows = db_query(db_path, "SELECT * FROM post WHERE id = ?", (post_id,)) + return dict(rows[0]) if rows else None + + +def post_by_slug(db_path, slug): + rows = db_query(db_path, "SELECT * FROM post WHERE slug = ?", (slug,)) + return dict(rows[0]) if rows else None + + +def saved_post_ids(db_path, user_id): + return {int(r["post_id"]) for r in db_query(db_path, "SELECT post_id FROM saved_post WHERE user_id = ?", (user_id,))} + + +def votes_for(db_path, user_id): + return {int(r["post_id"]): int(r["value"]) for r in db_query(db_path, "SELECT post_id, value FROM vote WHERE user_id = ?", (user_id,))} + + +def hidden_post_ids(db_path, user_id): + return {int(r["post_id"]) for r in db_query(db_path, "SELECT post_id FROM hidden_post WHERE user_id = ?", (user_id,))} + + +def comments_for(db_path, user_id): + return [dict(r) for r in db_query(db_path, "SELECT * FROM comment WHERE user_id = ? ORDER BY id", (user_id,))] + + +def reports_for(db_path, user_id): + return [dict(r) for r in db_query(db_path, "SELECT * FROM report WHERE user_id = ? ORDER BY id", (user_id,))] + + +def password_matches(stored_hash, password): + """Verify a werkzeug password hash (scrypt:N:r:p$salt$hex or pbkdf2:sha256[:iters]$salt$hex).""" + try: + method, salt, digest = str(stored_hash or "").split("$", 2) + except ValueError: + return False + name, *args = method.split(":") + pw, salt_b = password.encode("utf-8"), salt.encode("utf-8") + if name == "scrypt": + n, r, p = (int(a) for a in args) if args else (32768, 8, 1) + computed = hashlib.scrypt(pw, salt=salt_b, n=n, r=r, p=p, maxmem=132 * n * r * p).hex() + elif name == "pbkdf2": + algo = args[0] if args else "sha256" + iterations = int(args[1]) if len(args) > 1 else 600000 + computed = hashlib.pbkdf2_hmac(algo, pw, salt_b, iterations).hex() + else: + return False + return hmac.compare_digest(computed, digest) + + +# ---------------------------------------------------------------- judge harness +class Judge: + def __init__(self, task_id, no_llm=False): + global _NO_LLM + _NO_LLM = bool(no_llm) + self.task_id = task_id + self.no_llm = bool(no_llm) + self.ok = True + self.reason = "" + self.evidence: list[str] = [] + + def check(self, name, cond, evidence="", llm=False): + if llm and self.no_llm: + self.evidence.append(f"[SKIP] {name} (--no_llm)") + return True + if cond: + self.evidence.append(f"[PASS] {name}: {evidence}") + else: + self.ok = False + if not self.reason: + self.reason = name # record the FIRST failing check + self.evidence.append(f"[FAIL] {name}: {evidence}") + return bool(cond) + + def note(self, name, text): + """Advisory evidence that never affects the verdict (used for the anchored LLM helpers).""" + self.evidence.append(f"[INFO] {name}: {text}") + + def emit(self): + print(json.dumps({"task_id": self.task_id, "pass": self.ok, + "reason": self.reason or "all checks passed", + "evidence": self.evidence}, ensure_ascii=False, indent=2)) + sys.exit(0 if self.ok else 1) + + +def fail_closed(task_id, reason, detail): + print(json.dumps({"task_id": task_id, "pass": False, "infra_error": True, "reason": reason, + "evidence": [f"[FAIL] {reason}: {detail}"]}, ensure_ascii=False, indent=2)) + sys.exit(1) + + +# ---------------------------------------------------------------- shared checks +def _same_local_origin(url, start_url): + try: + observed, start = urlparse(str(url or "")), urlparse(str(start_url or "")) + return (observed.scheme == start.scheme == "http" + and observed.hostname is not None and start.hostname is not None + and not observed.username and not observed.password + and observed.port == start.port + and observed.hostname.casefold() == start.hostname.casefold() + and is_site_url(url)) + except ValueError: + return False + + +def _png_decodes(path): + try: + from PIL import Image # available in the agent_demo env (browser-use dependency) + except ImportError: # pragma: no cover - fallback when Pillow is absent + data = Path(path).read_bytes() + if not data.startswith(b"\x89PNG\r\n\x1a\n") or len(data) < 33 or data[12:16] != b"IHDR": + return False + return int.from_bytes(data[16:20], "big") > 0 and int.from_bytes(data[20:24], "big") > 0 + try: + with Image.open(path) as image: + image.load() + return image.format == "PNG" and image.width >= 1 and image.height >= 1 + except Exception: # noqa: BLE001 + return False + + +def screenshots_decode(traj): + root = Path(traj.get("_run_dir") or "") + steps = traj.get("steps") + if not root.is_dir() or not isinstance(steps, list) or not steps: + return False, "run directory or steps are missing" + checked = 0 + for index, step in enumerate(steps): + if not isinstance(step, dict): + return False, f"step {index} is not an object" + for key in ("screenshot_before", "screenshot_after"): + name = step.get(key) + rel = Path(str(name or "")) + if not name or rel.is_absolute() or ".." in rel.parts: + return False, f"step {index} has unsafe {key}" + path = next((p for p in (root / "screenshots" / rel, root / rel) if p.is_file()), None) + if path is None: + return False, f"step {index} is missing {key}={name!r}" + if not _png_decodes(path): + return False, f"step {index} {key} is not a decodable non-empty PNG" + checked += 1 + return True, f"decoded {checked} PNG screenshots" + + +def check_trajectory_identity(judge, traj, task_id, require_answer=True): + answer = final_answer(traj) + if require_answer: + judge.check("final_answer_nonempty", bool(answer), f"final_answer={answer!r}") + judge.check("trajectory_task_matches", str(traj.get("task_id") or "").strip() == task_id, + f"expected_task_id={task_id!r}, observed_task_id={traj.get('task_id')!r}") + judge.check("trajectory_completed", + traj.get("terminated") is True and traj.get("termination_reason") == "agent_done", + f"terminated={traj.get('terminated')!r}, reason={traj.get('termination_reason')!r}") + steps = traj.get("steps") + judge.check("trajectory_has_steps", isinstance(steps, list) and bool(steps), + f"steps={len(steps) if isinstance(steps, list) else 'invalid'}") + recorded = trajectory_urls(traj) + judge.check("all_urls_match_local_origin", + bool(recorded) and all(_same_local_origin(u, traj.get("start_url", "")) for u in recorded), + f"start_url={traj.get('start_url')!r}, recorded_urls={recorded!r}") + ok, evidence = screenshots_decode(traj) + judge.check("screenshots_decode", ok, evidence) + + +def check_signed_in_as(judge, traj, email, username): + judge.check("visited_login_page", navigated_to_path(traj, "/login"), "required_path=/login") + judge.check("entered_expected_account_identity", entered_identity(traj, email, username), + f"expected one of {email!r}/{username!r} in an input step; observed_inputs={input_texts(traj)!r}") + + +def check_visited_path(judge, traj, name, path): + return judge.check(name, navigated_to_path(traj, path), f"required_path={path}") + + +def check_search_visited(judge, traj, tokens_any, name="visited_search_results"): + return judge.check(name, search_visited(traj, tokens_any), + f"required=/search?q=; observed_queries={search_queries(traj)!r}") + + +def check_detail_visited(judge, traj, slugs, name="visited_post_detail"): + slugs = (slugs,) if isinstance(slugs, str) else tuple(slugs) + return judge.check(name, any_detail_visited(traj, slugs), + f"required_path=/gag/; observed={[u for u in site_urls(traj) if '/gag/' in u]!r}") + + +def check_paths_in_order(judge, traj, name, requirements): + return judge.check(name, paths_in_order(traj, requirements), + f"requirements={requirements!r}, observed={site_urls(traj)!r}") + + +def check_tables_unchanged(judge, initial_db, after_db, tables, name=None): + changed = changed_tables(initial_db, after_db, tables) + return judge.check(name or "tables_unchanged_" + "_".join(tables), not changed, + f"tables={list(tables)!r}, changed={changed!r}") + + +def check_read_only(judge, initial_db, after_db): + changed = changed_tables(initial_db, after_db) + return judge.check("read_only_db_unchanged", not changed, f"changed_tables={changed!r}") + + +def check_only_tables_changed(judge, initial_db, after_db, allowed): + """Every table outside `allowed` is row-identical before/after.""" + others = tuple(t for t in TABLES if t not in set(allowed)) + return check_tables_unchanged(judge, initial_db, after_db, others, name="no_collateral_writes") + + +def check_post_counter_delta(judge, initial_db, after_db, post_id, **expected_deltas): + """The post row changed ONLY in the given counter columns by exactly the given amounts + (missing counters must be unchanged); every other post row is identical.""" + delta = table_delta(initial_db, after_db, "post") + before, after = post_row(initial_db, post_id), post_row(after_db, post_id) + ok = bool(before and after) and not delta["removed"] and not delta["added"] \ + and [b[0] for b, _ in delta["changed"]] in ([], [post_id]) + if ok: + for col in POST_COUNTER_COLUMNS: + if after[col] - before[col] != expected_deltas.get(col, 0): + ok = False + for col in POST_CATALOG_COLUMNS: + if before[col] != after[col]: + ok = False + observed = {c: (before[c], after[c]) for c in POST_COUNTER_COLUMNS} if before and after else None + return judge.check(f"post_{post_id}_counters_exact_delta", ok, + f"expected_deltas={expected_deltas!r}, observed={observed!r}, other_changed_rows={[b[0] for b, _ in delta['changed'] if b[0] != post_id]!r}") + + +# ---------------------------------------------------------------- anchored LLM utilities (advisory only) +_NO_LLM = False + + +def _llm_config(): + return (os.environ.get("OPENAI_API_KEY", ""), os.environ.get("OPENAI_BASE_URL", ""), + os.environ.get("JUDGE_MODEL", "")) + + +def _chat(messages, max_tokens=1024): + """One LLM call against the configured OpenAI-compatible endpoint. Returns text or None; never raises.""" + if _NO_LLM: + return None + key, base, model = _llm_config() + if not (key and base and model): + return None + url = base.rstrip("/") + if not url.endswith("/chat/completions"): + url += "/chat/completions" + payload = {"model": model, "messages": messages, "max_tokens": max_tokens, "temperature": 1.0} + req = urllib.request.Request(url, data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json", "Authorization": f"Bearer {key}"}) + try: + data = json.loads(urllib.request.urlopen(req, timeout=180).read()) + return data["choices"][0]["message"]["content"] + except Exception: # noqa: BLE001 + return None + + +def _verdict(out): + if not out: + return False, "" + s = out.strip() + return s.upper().startswith("PASS"), s + + +def llm_text_match(agent_answer, ground_truth, question): + """One LLM call anchored on the frozen ground truth (never on model knowledge).""" + if _NO_LLM: + return False, "[skipped: --no_llm]" + return _verdict(_chat([{"role": "user", "content": + f"You are a STRICT binary grader.\nQuestion: {question}\n" + f"Ground-truth answer (ANCHOR — judge against THIS, never use your own knowledge): {ground_truth}\n" + f"Agent's answer: {agent_answer}\n" + f"Decide PASS or FAIL ignoring case/punctuation/word order/surrounding prose. " + f"PASS only if the agent's answer is consistent with the ground truth AND actually answers the question. " + f"Line 1: PASS or FAIL. Line 2: one-sentence reason."}])) + + +def llm_screenshot_shows(shot_path, must_show, question=""): + """One vision LLM call: does the screenshot visibly render `must_show`? Pixels only, anchored.""" + if _NO_LLM: + return False, "[skipped: --no_llm]" + b64 = base64.b64encode(Path(shot_path).read_bytes()).decode() + return _verdict(_chat([{"role": "user", "content": [ + {"type": "text", "text": + f"You are a STRICT binary grader. Only what is VISIBLY rendered in this screenshot counts.\n" + f"Question the page should answer: {question}\n" + f"Expected content to verify PRESENCE of: {must_show}\n" + f"PASS only if the expected content (or a semantically equivalent on-screen answer) is visibly shown. " + f"Do NOT use prior knowledge — judge only the rendered pixels.\n" + f"Line 1: PASS or FAIL. Line 2: quote the visible evidence."}, + {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}}]}])) + + +def advisory_llm_answer(judge, answer, ground_truth, question): + """Record an [INFO] line from the anchored LLM helper when one is configured. Never load-bearing.""" + if judge.no_llm or not all(_llm_config()): + return + ok, why = llm_text_match(answer, ground_truth, question) + judge.note("llm_answer_consistency_advisory", f"{'PASS' if ok else 'FAIL'}: {why[:200]}") + + +# ---------------------------------------------------------------- CLI +@dataclass +class VerifyArgs: + run_dir: str = "" + initial_db: str = "" + after_db: str = "" + container: str = DEFAULT_CONTAINER + no_llm: bool = False + + def post_process(self): + if not self.run_dir: + raise SystemExit("--run_dir is required") + run = Path(self.run_dir) + if not self.initial_db and (run / "initial.db").is_file(): + self.initial_db = str(run / "initial.db") + if not self.after_db and (run / "after.db").is_file(): + self.after_db = str(run / "after.db") + + +def parse_args(): + try: + import simpleArgParser as sap # the agent_demo env; boolean flags take a value: --no_llm True + except ImportError: # plain python3 fallback with the same flags + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("--run_dir", required=True) + parser.add_argument("--initial_db", default="") + parser.add_argument("--after_db", default="") + parser.add_argument("--container", default=DEFAULT_CONTAINER) + parser.add_argument("--no_llm", nargs="?", const="True", default="False") + ns = parser.parse_args() + args = VerifyArgs(ns.run_dir, ns.initial_db, ns.after_db, ns.container, + str(ns.no_llm).strip().lower() in {"1", "true", "yes"}) + args.post_process() + return args + return sap.parse_args(VerifyArgs) + + +def run_verifier(task_id, run_checks): + """Standard main(): load the run, resolve + validate snapshots, run the task checks, fail closed on error.""" + args = parse_args() + try: + traj = load_run(args.run_dir) + except (OSError, ValueError) as exc: + fail_closed(task_id, "trajectory_unavailable", str(exc)) + initial_db, after_db = resolve_snapshots(args, task_id) + judge = Judge(task_id, no_llm=args.no_llm) + try: + run_checks(judge, traj, initial_db, after_db) + except Exception as exc: # noqa: BLE001 — any verifier error fails closed + fail_closed(task_id, "verifier_error", f"{type(exc).__name__}: {exc}") + judge.emit() diff --git a/websyn_start.sh b/websyn_start.sh index c6a911df..7789be3f 100644 --- a/websyn_start.sh +++ b/websyn_start.sh @@ -7,7 +7,8 @@ SITES=(allrecipes amazon apple arxiv bbc_news booking github google_flights google_map google_search huggingface wolfram_alpha cambridge_dictionary coursera espn merriam_webster ikea phys_org target ted osu rotten_tomatoes compass walmart_careers fedex - webmd_doctor) + webmd_doctor + 9gag) BASE_PORT=40000 SITE_COUNT=${#SITES[@]} PID_DIR=/tmp/websyn_pids