Conversation
The html branch of run_extract always used HttpClient.get_html, so pyscrappy extract URL out.html --render-js wrote the un-rendered page. Use GenericScraper.fetch_html when render_js is true, keep http.get_html when false, and warn that --css-selector does not apply to .html. Co-authored-by: David <dyk1454683243-sudo@users.noreply.github.com>
Co-authored-by: David <dyk1454683243-sudo@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The implementation matches the stated behavior and has focused regression coverage.
Review effort: Lite
Findings: None
What changed in this PR
Fixes .html extraction so --render-js uses browser-rendered markup and warns when --css-selector is ignored.
Changes:
- Routes rendered HTML through
fetch_html(..., render_js=True). - Preserves plain HTTP fetching when rendering is disabled.
- Adds a warning and tests for unsupported CSS selectors on HTML output.
| File | Description |
|---|---|
src/pyscrappy/cli.py |
Updates HTML extraction behavior and warning handling. |
tests/test_cli/test_extract.py |
Covers rendered, non-rendered, and selector-warning cases. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Nice work @dyk1454683243-sudo, this is exactly the fix. Verified locally: render_js=True takes the browser path, render_js=False is unchanged, and the css-selector warning fires. Both new tests fail on the pre-fix code and pass with it, with the existing tests untouched. 600 passed, ruff clean. Good call strengthening the existing test with fetch_html.assert_not_called() rather than loosening it. Merging, thanks! |
Fixes #188
Problem
pyscrappy extract URL out.html --render-jsaccepted the flag, exited 0, and still fetched viaGenericScraper.http.get_html()— the un-rendered HTTP response. Other extensions already passrender_jsintoscrape().Fix
In the
ext == "html"branch ofrun_extract:render_js=Trueuses the scraper's browser-awarefetch_html(url, render_js=True)render_js=Falsestill useshttp.get_html(existing CLI test pin)--css-selectorwith.htmlnow emits aUserWarninginstead of being silently ignoredTests
run_extract(..., out.html, render_js=True)callsfetch_html, nothttp.get_htmlrun_extract(..., out.html, render_js=False)still callshttp.get_html--css-selectorwith.htmlwarnsLocally: