Review: Add Craigslist mirror + task verifiers (#60) - #103
Open
TabsPhasers wants to merge 7 commits into
Open
Conversation
Reviewer deliverable for the Craigslist mirror (site by @FlowRays, PR aiming-lab#5): one deterministic verifier per task under sites/craigslist/verify/, plus verifier_path + judge_rubric in every tasks.jsonl row. No answer key in tasks.jsonl — ground truth lives only inside the verifiers. Deterministic-first: listing-open navigation gate (anti knowledge-shortcut; listings are synthetic) + SQLite DB after-state for the stateful tasks (saved_listings / saved_searches / hidden_listings / messages / new postings / user profile) + answer vs frozen ground truth, LLM only as an anchored check. Validated against the official react agent (agent_demo/agent.py): a no-op run fails all 20 verifiers; on the full 20-task run verifier and LLM judge agree 14/20, and the divergences favor the deterministic verifier (it catches DB after-state and rejects blank/incomplete answers the judge false-positives). Note: verify_0 and verify_3 are limited by a task bug — their save-targets are already in the seed's saved-listings, so the save cannot be verified; the review asks the author to fix those tasks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…expressible - seed: pre-saved chair is now the Black office chair (id 2), not the Ergonomic task chair (id 1) - task 0's save target must be absent from the initial saved list; the 2006 Accord EX (id 13) stays pre-seeded for task 13 and the Accord pin is explicit. - task 3: retarget from the pre-saved 2006 Accord EX sedan to the non-pre-saved 2011 Accord EX-L (id 15); 2006 EX becomes a near-miss distractor. - verify_0/verify_3: initial->after delta checks (absent initially, present after) replace the after-only check that could not distinguish a real save. - verify_13: KEEP updates to the new pre-seeded furniture id (2). - save-search form: category and max price become user-editable (were hidden empty fields); verify_5 now enforces category=furniture and max_price=200. - README: fix 16-vs-17 site count inconsistency.
…st form, unify saved labels
- search: drop the sqft chip from result cards. The old 'and not is_housing'
guard only suppressed it on category pages while generic query pages (where
tasks 1/4 actually search) leaked listing sqft - an answer shortcut that
lets tasks be answered without opening the posting.
- listings: add POST /listing/<id>/unhide and a 'hidden listings' panel on the
account page; hiding was previously irreversible with no UI to recover.
- post form: tag each field group with its category group and toggle visibility
with a few lines of JS (progressive enhancement; all fields remain when JS
is off). Posting a bike no longer shows bedrooms/sqft/compensation fields.
- templates: unify saved-listing vocabulary - header badge and nav links now
read 'saved' instead of the mixed 'faves'/'favorites'; object-specific
headings ('saved listings', 'saved searches') are unchanged.
…otos + generated placeholders The scraped photo pool only covers apartments/cars/jobs and pick_image dealt pool photos round-robin, so 119/183 listings had no image and others showed a different real item's photo (the 2006 Accord EX carried the 2011 EX-L's photo). - real photos are now used only when the listing title matches the photo's own filename tokens (20 exact matches, incl. the correct Accord for id 13/15). - every other listing gets a deterministic SVG placeholder generated at seed time (category label + title, hash-derived colors) - 183/183 listings have a truthful image, regenerable from seed_data.py alone. - galleries collapse to the listing's own single image; pooling other items' photos into a posting's carousel was part of the mismatch defect.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review: Add Craigslist mirror + task verifiers (#60)
Summary
Review takeover of #60 — the Craigslist mirror by @FlowRays and the task
verifiers by @DEM1TASSE — with fixes for every item of the original
REQUEST CHANGES review (posted on #5). Both original contributors stay
credited: their commits are retained untouched, with three review commits on
top. Please merge with a regular merge (not squash).
Original review resolution
verify_0/verify_3check the initial→after saved-listings delta (3a5ae40)verify_5enforces name +category_slug=furniture+max_price=200(3a5ae40)not is_housingguard only suppressed it on category pages while generic query pages (where tasks 1/4 search) leaked listing sqft as an answer shortcut (da8d380)45daffd)POST /listing/<id>/unhideand a "hidden listings" panel on the account page; hiding was previously irreversible (da8d380)data-groupsand toggle per selected category via progressive-enhancement JS; posting a bike no longer shows bedrooms/sqft/compensation (da8d380)da8d380)Review changes
sites/craigslist/seed_data.py— pre-seeded saved set {office chair 2,2006 Accord EX 13, Berkeley BART studio 21}; deterministic image assignment
(title-matched real photos + generated SVG placeholders written at seed time)
sites/craigslist/tasks.jsonl— task 3 retargeted to the 2011 Accord EX-L;task 0/3 rubrics updated (no answer keys added)
sites/craigslist/verify/— initial→after delta predicates for the savetasks (
verify_0,verify_3), distractor guard on task 3, three-fieldsave-search enforcement (
verify_5), updated keep-set (verify_13),saved_searches_forextended withcategory_slug(verify_lib)sites/craigslist/app.py,templates/— unhide route + account panel,category-aware post form, sqft chip removal, saved vocabulary
README.md— site-count consistency (17)Asset delivery status
("feat(craigslist): add craigslist asset bundle (reviewed seed for WebHarbor Add Craigslist mirror + task verifiers (site by @FlowRays, verifiers by reviewer) #60)")
mismatched image pool)
craigslist.tar.gz, 523,402 bytes, 198 members, SHA-2569ea3eeefff3bed47337531407ff75634ffbae69d680fd70ebd671c1134dda93a(HF commit
672962a).assets-revision: pending — the current pin54882a6a…predatescraigslist and cannot fetch this site. It will be repinned to the HF Add Target mirror + task verifiers (site rebuilt from real target.com data) #72
merge commit with a clean
fetch_assets.sh craigslistcheck after merge.Do not merge this PR before the HF PR is merged.
Validation
Deterministic single-site review environment (review image
webharbor-pr60-craigslist:45daffd, canonical port 40016):/_health(183 listings) / control-plane health: PASSinstance == instance_seed(edcd84fe…) after everyreset: PASS
seed_data.pyregenerates a database whose full dumpequals the packaged seed: PASS
save, task 13 wrong-item removal all FAIL; positives PASS
PASS — which is exactly how the two pre-seeded save-task false positives
surfaced), Round B on the fixed seed (20/20 PASS with real DB deltas, e.g.
saved set [2,13,21] → [1,2,13,21] for task 0), Round C after the five site
fixes (7/7 affected tasks PASS)
(main-judged; no independent LLM judge was available in this environment —
recorded as a limitation)
Known remaining issues (disclosed; outside the task contract)
(390 px is clean)
the homepage use text queries rather than the category route
not run (environment limitation, see Validation)
Scope
untouched
the HF bundle), review tooling
Blocking checklist
merged →
.assets-revisionrepinned to the merge commit → cleanfetch_assets.sh craigslistverifiedReviews #60. Site track: #5.