Skip to content

Review: Add Craigslist mirror + task verifiers (#60) - #103

Open
TabsPhasers wants to merge 7 commits into
aiming-lab:mainfrom
TabsPhasers:review/pr-60-craigslist
Open

Review: Add Craigslist mirror + task verifiers (#60)#103
TabsPhasers wants to merge 7 commits into
aiming-lab:mainfrom
TabsPhasers:review/pr-60-craigslist

Conversation

@TabsPhasers

Copy link
Copy Markdown

Review: Add Craigslist mirror + task verifiers (#60)

Summary

Review takeover of #60 — the Craigslist mirror by @FlowRays and the task
verifiers by @DEM1TASSE — with fixes for every item of the original
REQUEST CHANGES review (posted on #5). Both original contributors stay
credited: their commits are retained untouched, with three review commits on
top. Please merge with a regular merge (not squash).

Original review resolution

#5 review item Status Fix
two save-tasks whose targets are pre-seeded fixed seed now pre-saves the Black office chair (id 2) instead of the Ergonomic task chair; task 3 retargeted to the non-pre-saved 2011 Accord EX-L (id 15); the 2006 Accord EX (id 13) stays pre-seeded as task 13's removal target and doubles as a near-miss distractor; verify_0/verify_3 check the initial→after saved-listings delta (3a5ae40)
a save-search that can't set the requested category fixed the save-search form exposes a category select and max price; verify_5 enforces name + category_slug=furniture + max_price=200 (3a5ae40)
a half-fixed sqft leak fixed the sqft chip is removed from search result cards; the old not is_housing guard only suppressed it on category pages while generic query pages (where tasks 1/4 search) leaked listing sqft as an answer shortcut (da8d380)
image/content mismatches fixed real photos are used only when the listing title matches the photo's own filename (20 exact matches — the 2006 Accord EX no longer carries the 2011 EX-L's photo); all other listings get a deterministic generated SVG placeholder, so 183/183 listings have a truthful, regenerable image; galleries collapse to the listing's own image (45daffd)
an incomplete hide feature fixed added POST /listing/<id>/unhide and a "hidden listings" panel on the account page; hiding was previously irreversible (da8d380)
a non-category-aware post form fixed field groups carry data-groups and toggle per selected category via progressive-enhancement JS; posting a bike no longer shows bedrooms/sqft/compensation (da8d380)
inconsistent favorites labels fixed header badge and nav links unified to "saved"; object-specific headings ("saved listings", "saved searches") unchanged (da8d380)

Review changes

  • sites/craigslist/seed_data.py — pre-seeded saved set {office chair 2,
    2006 Accord EX 13, Berkeley BART studio 21}; deterministic image assignment
    (title-matched real photos + generated SVG placeholders written at seed time)
  • sites/craigslist/tasks.jsonl — task 3 retargeted to the 2011 Accord EX-L;
    task 0/3 rubrics updated (no answer keys added)
  • sites/craigslist/verify/ — initial→after delta predicates for the save
    tasks (verify_0, verify_3), distractor guard on task 3, three-field
    save-search enforcement (verify_5), updated keep-set (verify_13),
    saved_searches_for extended with category_slug (verify_lib)
  • sites/craigslist/app.py, templates/ — unhide route + account panel,
    category-aware post form, sqft chip removal, saved vocabulary
  • README.md — site-count consistency (17)

Asset delivery status

Validation

Deterministic single-site review environment (review image
webharbor-pr60-craigslist:45daffd, canonical port 40016):

  • build / /_health (183 listings) / control-plane health: PASS
  • reset byte-identity: instance == instance_seed (edcd84fe…) after every
    reset: PASS
  • seed determinism: seed_data.py regenerates a database whose full dump
    equals the packaged seed: PASS
  • no-op baseline: 20/20 verifiers correctly FAIL (exit 1)
  • task-specific negatives: visit-only saves (tasks 0/3), task 3 without a
    save, task 13 wrong-item removal all FAIL; positives PASS
  • full-matrix real-browser runs: Round A on the original head (20/20 verifier
    PASS — which is exactly how the two pre-seeded save-task false positives
    surfaced), Round B on the fixed seed (20/20 PASS with real DB deltas, e.g.
    saved set [2,13,21] → [1,2,13,21] for task 0), Round C after the five site
    fixes (7/7 affected tasks PASS)
  • rubric judge (main-agent) vs deterministic verifiers: 20/20 agree
    (main-judged; no independent LLM judge was available in this environment —
    recorded as a limitation)

Known remaining issues (disclosed; outside the task contract)

  • 320 px viewports show slight horizontal overflow on home/search/detail
    (390 px is clean)
  • text search can surface cross-category results; for-sale category links on
    the homepage use text queries rather than the category route
  • some filler seed rows carry scrambled category/area pairs
  • independent LLM judge and independent pure-visual subagent trajectories were
    not run (environment limitation, see Validation)

Scope

  • Original site (@FlowRays) and verifier (@DEM1TASSE) commits retained
    untouched
  • Excluded: reviewer run evidence, local static images (the image set ships in
    the HF bundle), review tooling
  • No shared framework changes

Blocking checklist

  • HF PR #72
    merged → .assets-revision repinned to the merge commit → clean
    fetch_assets.sh craigslist verified

Reviews #60. Site track: #5.

FlowRays and others added 7 commits July 2, 2026 11:08
Reviewer deliverable for the Craigslist mirror (site by @FlowRays, PR aiming-lab#5):
one deterministic verifier per task under sites/craigslist/verify/, plus
verifier_path + judge_rubric in every tasks.jsonl row. No answer key in
tasks.jsonl — ground truth lives only inside the verifiers.

Deterministic-first: listing-open navigation gate (anti knowledge-shortcut;
listings are synthetic) + SQLite DB after-state for the stateful tasks
(saved_listings / saved_searches / hidden_listings / messages / new postings /
user profile) + answer vs frozen ground truth, LLM only as an anchored check.

Validated against the official react agent (agent_demo/agent.py): a no-op run
fails all 20 verifiers; on the full 20-task run verifier and LLM judge agree
14/20, and the divergences favor the deterministic verifier (it catches DB
after-state and rejects blank/incomplete answers the judge false-positives).

Note: verify_0 and verify_3 are limited by a task bug — their save-targets are
already in the seed's saved-listings, so the save cannot be verified; the
review asks the author to fix those tasks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…expressible

- seed: pre-saved chair is now the Black office chair (id 2), not the Ergonomic
  task chair (id 1) - task 0's save target must be absent from the initial
  saved list; the 2006 Accord EX (id 13) stays pre-seeded for task 13 and the
  Accord pin is explicit.
- task 3: retarget from the pre-saved 2006 Accord EX sedan to the non-pre-saved
  2011 Accord EX-L (id 15); 2006 EX becomes a near-miss distractor.
- verify_0/verify_3: initial->after delta checks (absent initially, present
  after) replace the after-only check that could not distinguish a real save.
- verify_13: KEEP updates to the new pre-seeded furniture id (2).
- save-search form: category and max price become user-editable (were hidden
  empty fields); verify_5 now enforces category=furniture and max_price=200.
- README: fix 16-vs-17 site count inconsistency.
…st form, unify saved labels

- search: drop the sqft chip from result cards. The old 'and not is_housing'
  guard only suppressed it on category pages while generic query pages (where
  tasks 1/4 actually search) leaked listing sqft - an answer shortcut that
  lets tasks be answered without opening the posting.
- listings: add POST /listing/<id>/unhide and a 'hidden listings' panel on the
  account page; hiding was previously irreversible with no UI to recover.
- post form: tag each field group with its category group and toggle visibility
  with a few lines of JS (progressive enhancement; all fields remain when JS
  is off). Posting a bike no longer shows bedrooms/sqft/compensation fields.
- templates: unify saved-listing vocabulary - header badge and nav links now
  read 'saved' instead of the mixed 'faves'/'favorites'; object-specific
  headings ('saved listings', 'saved searches') are unchanged.
…otos + generated placeholders

The scraped photo pool only covers apartments/cars/jobs and pick_image dealt
pool photos round-robin, so 119/183 listings had no image and others showed a
different real item's photo (the 2006 Accord EX carried the 2011 EX-L's photo).

- real photos are now used only when the listing title matches the photo's own
  filename tokens (20 exact matches, incl. the correct Accord for id 13/15).
- every other listing gets a deterministic SVG placeholder generated at seed
  time (category label + title, hash-derived colors) - 183/183 listings have a
  truthful image, regenerable from seed_data.py alone.
- galleries collapse to the listing's own single image; pooling other items'
  photos into a posting's carousel was part of the mismatch defect.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants