fix: X blocks bundled headless Chromium with 403 - #18
Merged
Merged
Conversation
Playwright's bundled headless Chromium gets a 403 from x.com on every request, regardless of User-Agent. Fix by: - setting a normal desktop Chrome UA on the context (removes the HeadlessChrome tell) - preferring a locally installed real Chrome (channel: chrome) over bundled Chromium when no explicit --browser-channel/--browser-executable-path is given, since X fingerprints the bundled binary itself and blocks it outright even with a spoofed UA Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014i3FdrzgdTStsSytFiyzyV
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+155
to
+156
| if (findChromePath()) { | ||
| launchOptions.channel = 'chrome'; |
Contributor
There was a problem hiding this comment.
🔴 Chrome-free installations remain unusable
Without Google Chrome, findChromePath() leaves blocked bundled Chromium selected or mistakes system Chromium for Chrome. Every extraction then fails with a 403 or launch error.
Was this helpful? React with 👍 or 👎 to provide feedback.
- Use static import of findChromePath instead of dynamic await import - Use executablePath when Chrome found, not channel: 'chrome' (more robust) - Update hardcoded UA to macOS 14 (Sonoma) + Chrome 132 (realistic pairing) - Add comment explaining UA needs periodic updates - Add Windows support to findChromePath with common installation paths - Simplify comments to explain 'why' not 'what' Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014i3FdrzgdTStsSytFiyzyV
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Devin Review correctly flagged that the CDP and login-retry flows (x-dl cdp, and the automatic login-wall retry) launch their own headless Chrome via launchPrivateBrowser, bypassing createContextAndPage entirely. That context never got the desktop UA override, so it kept the default HeadlessChrome UA and hit the same 403 the original fix addressed, just on a different path. Move DESKTOP_CHROME_UA into private.ts and apply it in launchPrivateBrowser whenever running headless (a headed, user-visible login flow doesn't need the spoof). extractor.ts now imports the constant instead of duplicating it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014i3FdrzgdTStsSytFiyzyV
createContextAndPage was always applying DESKTOP_CHROME_UA, including in headed mode, while launchPrivateBrowser only applies it when headless. This was inconsistent: headed browsers (used for login or user testing) don't need and shouldn't need UA spoofing. Both now apply the override only in headless mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014i3FdrzgdTStsSytFiyzyV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Video extraction was failing for all tweets with "Failed to extract video URL", even for public videos with no login wall.
Root cause: X returns HTTP 403 to every request from Playwright's bundled headless Chromium, regardless of User-Agent.
page.goto()never actually loaded the tweet page, so it was never a video-detection problem — the page was blank.Confirmed via direct testing:
channel: 'chrome') + its default UA (which literally containsHeadlessChrome/...) → still blockedFix
In
src/extractor.ts:userAgenton the context (removes theHeadlessChrometell).--browser-channel/--browser-executable-pathis explicitly passed, prefer a locally installed real Chrome (reusingfindChromePath()fromprivate.ts) over bundled Chromium, falling back to bundled Chromium if Chrome isn't found.Testing
Verified against two previously-failing tweet URLs — video extracted and downloaded successfully for both:
https://x.com/mattshumer_/status/2095187868746383758https://x.com/bridgemindai/status/2094894312370692443Full test suite passes (61 pass / 9 skip).
🤖 Generated with Claude Code
https://claude.ai/code/session_014i3FdrzgdTStsSytFiyzyV