feat(scrape): optional challenge-wall evidence on the blocked error path - #124
Open
emandel2630 wants to merge 1 commit into
Open
feat(scrape): optional challenge-wall evidence on the blocked error path#124emandel2630 wants to merge 1 commit into
emandel2630 wants to merge 1 commit into
Conversation
A scrape that ran a browser but never cleared the challenge answers HTTP 500
with {error, timings} and discards the wall it stopped at. `timings[].reason`
says a challenge persisted, but there is no way to see it, and "blocked by a
wall" is indistinguishable from "TRAWL broke" without reading server logs.
`blockedEvidence: true` attaches the wall to that failure: `blockedEvidence` on
ScrapeError, serialized into the 500 body next to the existing `error` and
`timings` - tier, status, reason (the same string as the matching timing), the
landing url, statusCode, the markup, and the screenshot when one was requested.
The wall from the deepest browser tier that rendered one wins.
Deliberately not the blocked-as-success shape germondai#53 moved away from. A blocked
scrape stays a failure: the status stays 500, and ScrapeResult never carries a
challenge wall as content. Only the error becomes diagnosable.
- Opt-in and terminal-path only: without the flag the sink is never built, so
reportBlocked returns before reading a page or taking an image, and a stock
request is unchanged. The image needs `screenshot` as well.
- Bounded: one wall per request, its markup capped at
BLOCKED_EVIDENCE_MAX_HTML_CHARS (512000) and flagged `htmlTruncated`. The head
of a wall carries the title, vendor markers and incident id a caller
classifies on, so this one truncates rather than dropping.
- Contained: reportBlocked catches everything and logs - a failed read leaves
the evidence off the error and never changes a tier's outcome. It holds no
page reference past its own call; only strings reach the orchestrator.
- Branches with no page to read carry timings only: Tier 1, a context or page
that never opened, a hard network failure, about:neterror, an empty document,
a saturated pool.
- capturePageScreenshot gains `settle: false` for these - a challenge wall
never reaches network idle, so the settle wait only spends the next tier's
budget. The remaining-budget bound from germondai#106 is kept.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A scrape that ran a browser but never cleared the challenge answers
500 {error, timings}and discards the wall it stopped at.timings[].reasonsays a challenge persisted, but there is no way to see the page, so "blocked by a wall" and "TRAWL broke" look the same without server logs.blockedEvidence: trueonPOST /scrapeattaches the wall to that failure asblockedEvidenceon the 500 body: tier, status, reason (the same string as the matching timing), the landing URL,statusCode, the markup, and the screenshot whenscreenshotwas also requested. The wall from the deepest browser tier that rendered one wins.This is deliberately not the blocked-as-success shape #53 moved away from. A blocked scrape stays a failure: the status stays 500 and
ScrapeResultnever carries a wall as content. Only the error becomes diagnosable.Design
reportBlockedreturns before reading a page or taking an image, and a stock request is unchanged.BLOCKED_EVIDENCE_MAX_HTML_CHARS(512000) and flaggedhtmlTruncated. Truncated rather than dropped, because the head of a wall carries the title, vendor markers and incident id a caller classifies on.reportBlockedcatches everything and logs; a failed read leaves the evidence off the error and never changes a tier's outcome. Only strings reach the orchestrator.about:neterror, an empty document, a saturated pool.capturePageScreenshotgains asettle: falseoption for the wall image (a challenge wall never reaches network idle, so the settle wait only spends the next tier's budget). The remaining-budget bound from feat(scrape): safely add optional viewport screenshots #106 is kept as the second parameter.Verification
bun run check,bun run typecheckclean;bun test385 pass / 0 fail on top ofdev(c81d628). New tests inpackages/tiers/tests/blockedEvidence.test.tscover tier 2/3 reporting, the image-only-when-asked rule, truncation, a failing capture, and the orchestrator carrying the deepest wall ontoScrapeError;apps/api/src/routes/scrape.test.tscovers the 500 body.Companion to #106/#107/#109 from the same series; independent of the MHTML PR.