evals: post-guard live run on main — GREEN - #8
Open
Wynelson94 wants to merge 1 commit into
Open
Conversation
Second run of the day, on merged main (trafilatura 2.2 + the guard-coverage pass). The first run covered the dependency upgrade only; this one exercises the crawl and extract refusal paths the 194-page cache replay structurally could not reach, since that replay only drives assess_extraction over stored HTML. The earlier report is preserved as reports/2026-08-11-run1/ rather than being overwritten — run_eval.py hardcodes the output directory to today's date, and that report is the pre-guard baseline referenced by #6. Matches the existing 2026-07-16 / 2026-07-16-run1 convention. VERDICT: GREEN. All 9 gates PASS. 40 items + 45-scrape burst (40 ok, 5 honest refusals, 0 FAIL, 0 exceptions). Zero fabrications, zero cache poison, zero unhandled exceptions, baselines stable. Exactly ONE item changed status against the baseline: irs_p15 pass -> unreachable (HTTP 403) That is external, and it was verified rather than assumed: - fetch.py and config.py are BYTE-IDENTICAL between 06296c8 (pre-guard) and 77196fc (post-guard), so the guard PR cannot have caused it - curl gets HTTP 200 from the same URL with tearsheet's exact User-Agent AND Accept header, over both HTTP/1.1 and HTTP/2 - the other PDF item (berkshire_pdf) still extracts figures normally So the discriminator sits below the HTTP layer — TLS client fingerprinting. irs.gov has joined the Akamai class that already 403s eCFR and dodcio, and tearsheet reported it honestly as a fetch error instead of serving anything made up. The record of zero fabrications holds. crawl_ruff passes with the new index format (`refused: 0`). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second live eval of the day, this time on merged
main(trafilatura 2.2 + the guard-coverage pass). The first run covered the dependency upgrade only; this one exercises thecrawlandextractrefusal paths that the 194-page cache replay structurally could not reach — that replay only drivesassess_extractionover stored HTML.VERDICT: GREEN
All 9 gates PASS. 40 items + 45-scrape burst (40 ok, 5 honest refusals, 0 FAIL, 0 exceptions). Zero fabrications, zero cache poison, baselines stable across the 5 pinned immutables.
crawl_ruffpasses with the new index format (refused: 0).Exactly one item changed — and it's external
irs_p15Everything else is identical across 40 items. I verified the cause rather than assuming it:
fetch.pyandconfig.pyare byte-identical between06296c8(pre-guard) and77196fc(post-guard) —git diffis empty, so the guard PR cannot have caused itberkshire_pdf) still extracts figures normally, so the PDF path is healthyThe discriminator therefore sits below the HTTP layer — TLS client fingerprinting. irs.gov has joined the Akamai class that already 403s eCFR and dodcio for this tool. Tearsheet reported it honestly as a fetch error rather than serving anything invented; the zero-fabrication record holds.
No corpus change proposed here —
corpus.jsonis the trust instrument and shouldn't be edited to make a run look cleaner. If irs.gov stays blocked, swapping that item for a reachable PDF is a separate, deliberate decision.Note on the preserved report
run_eval.py:503hardcodes the output directory toreports/<today>, so re-running today would have silently overwritten the pre-guard baseline that #6 references. That report is nowreports/2026-08-11-run1/, matching the existing2026-07-16/2026-07-16-run1convention, and the plain-date directory holds the latest run.🤖 Generated with Claude Code