Skip to content

docs: mark the hard set as human-accepted after checking it against the PDFs - #35

Merged
yodablocks merged 3 commits into
masterfrom
docs/hard-set-accepted
Oct 4, 2026
Merged

yodablocks merged 3 commits into
masterfrom
docs/hard-set-accepted

Conversation

@yodablocks

Copy link
Copy Markdown
Owner

What

The 15-question hard set from #30 was labelled synthetic-draft (drafted by an LLM, not reviewed). Marc has now checked every answer and page of h-003..h-011 against the PDFs, so those rows are marked synthetic-draft+human-accept.

  • Data: experiments/recall/hard_v1.jsonl. Provenance changed for h-003..h-015 (h-001 and h-002 were already accepted from the first gold set). Each row's note records the check. The file now loads with the default acceptance check (load_gold(path), 15 questions), so quarq eval --dataset accepts it. I did not run the eval: it loads two models on the GPU.
  • Docs: a dated update in the recall README, and the Experiments caveat in the top-level README reworded.

What was checked

Each answer and page was opened in Preview and compared with the passage, using the PDF viewer's title-bar page index. In some of these files that differs by one from the printed page number (h-008: viewer page 7 is printed page 6), and quarq eval requires the viewer index.

The four French rewrites (h-012..h-015) share their English originals' answers and pages and were checked through them. Their French wording was not separately reviewed.

Caveats

@yodablocks
yodablocks merged commit 574178b into master Oct 4, 2026
4 checks passed
@yodablocks
yodablocks deleted the docs/hard-set-accepted branch October 4, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant