Skip to content

fix(cli): honor --render-js for .html extract output - #189

Merged
vedaant00 merged 2 commits into
mldsveda:mainfrom
dyk1454683243-sudo:cursor/fix-html-extract-render-js-33b4
Sep 21, 2026
Merged

vedaant00 merged 2 commits into
mldsveda:mainfrom
dyk1454683243-sudo:cursor/fix-html-extract-render-js-33b4

Conversation

@dyk1454683243-sudo

@dyk1454683243-sudo dyk1454683243-sudo commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Fixes #188

Problem

pyscrappy extract URL out.html --render-js accepted the flag, exited 0, and still fetched via GenericScraper.http.get_html() — the un-rendered HTTP response. Other extensions already pass render_js into scrape().

Fix

In the ext == "html" branch of run_extract:

  • render_js=True uses the scraper's browser-aware fetch_html(url, render_js=True)
  • render_js=False still uses http.get_html (existing CLI test pin)
  • --css-selector with .html now emits a UserWarning instead of being silently ignored

Tests

  • run_extract(..., out.html, render_js=True) calls fetch_html, not http.get_html
  • run_extract(..., out.html, render_js=False) still calls http.get_html
  • --css-selector with .html warns

Locally:

pytest tests/test_cli/test_extract.py -v
# 8 passed
ruff check src/pyscrappy/cli.py tests/test_cli/test_extract.py
ruff format --check src/pyscrappy/cli.py tests/test_cli/test_extract.py

dyk1454683243-sudo and others added 2 commits September 19, 2026 15:41
The html branch of run_extract always used HttpClient.get_html, so
pyscrappy extract URL out.html --render-js wrote the un-rendered page.
Use GenericScraper.fetch_html when render_js is true, keep http.get_html
when false, and warn that --css-selector does not apply to .html.

Co-authored-by: David <dyk1454683243-sudo@users.noreply.github.com>
Co-authored-by: David <dyk1454683243-sudo@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation matches the stated behavior and has focused regression coverage.

Review effort: Lite
Findings: None

What changed in this PR

Fixes .html extraction so --render-js uses browser-rendered markup and warns when --css-selector is ignored.

Changes:

  • Routes rendered HTML through fetch_html(..., render_js=True).
  • Preserves plain HTTP fetching when rendering is disabled.
  • Adds a warning and tests for unsupported CSS selectors on HTML output.
File Description
src/​pyscrappy/​cli.py Updates HTML extraction behavior and warning handling.
tests/​test_cli/​test_extract.py Covers rendered, non-rendered, and selector-warning cases.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@vedaant00

Copy link
Copy Markdown
Collaborator

Nice work @dyk1454683243-sudo, this is exactly the fix. Verified locally: render_js=True takes the browser path, render_js=False is unchanged, and the css-selector warning fires. Both new tests fail on the pre-fix code and pass with it, with the existing tests untouched. 600 passed, ruff clean. Good call strengthening the existing test with fetch_html.assert_not_called() rather than loosening it. Merging, thanks!

@vedaant00
vedaant00 merged commit d5ae5d5 into mldsveda:main Sep 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--render-js is silently ignored for .html output in pyscrappy extract

3 participants