Skip to content

Old soup #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Kulungey wants to merge 7 commits into from
Closed

Old soup #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
77204e2
Update README.md
Kulungey Aug 28, 2026
d7a14f7
Update package versions in requirements.txt
Kulungey Aug 28, 2026
d00b395
Revise README for clarity and feature updates
Kulungey Aug 28, 2026
7fa453c
Update requirements to remove version constraints
Kulungey Aug 28, 2026
ea58674
Update scraper.py
Kulungey Aug 28, 2026
adbcd02
Enhance README with detailed usage and features
Kulungey Aug 28, 2026
4c095e1
Fix header formatting in README.md
Kulungey Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
284 changes: 182 additions & 102 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,170 +1,250 @@
# scraper
# scrape

Video downloader with Cloudflare bypass. Paste a URL, pick a format, get the file.
A lightweight video downloader for sites that are difficult to extract from directly.

YouTube routes directly to yt-dlp. Everything else goes through a 4-layer extraction stack: direct HTTP fetch, real Chrome with CF bypass, HTML/iframe scanning, then yt-dlp as a last resort.
Paste a URL, choose a format, and let `scrape` handle the rest. It uses `yt-dlp` where possible, a real Chrome session for Cloudflare protected sites, and browser network interception when the video URL is only exposed after the page loads.

---
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
![License](https://img.shields.io/badge/license-MIT-green)

## Features

* **YouTube and Twitter/X support** through `yt-dlp`
* **Cloudflare handling** using a real Chrome browser controlled by DrissionPage
* **Token bound CDN detection** through browser network interception
* **Multiple extraction methods** with automatic fallback
* **Automatic yt-dlp updates** when a newer version is available
* **Format conversion** through FFmpeg
* **MP4, MP3, MKV, WebM, and original format output**
* **Existing downloads are skipped**
* **Terminal progress display** with a rainbow progress bar

## How it works

`scrape` does not rely on a single extraction method. It moves through several layers and stops when one successfully finds the media.

```text
URL
|
+-- YouTube / Twitter/X
| |
| +--> yt-dlp
|
+-- Direct request
| |
| +--> curl_cffi
|
+-- Cloudflare protected page
| |
| +--> Chrome + DrissionPage
|
+-- Page inspection
| |
| +--> HTML / iframe / base64 extraction
|
+-- Browser network interception
|
+--> Capture CDN stream URL
|
+--> yt-dlp / FFmpeg
```

The browser based fallback is useful for sites where the actual video URL is generated only after JavaScript runs or after Cloudflare has completed its checks.

## Requirements

**Python 3.10 or newer**
### Python

External tools (must be on PATH):
Python 3.10 or newer is required.

| Tool | Purpose | Required |
|------|---------|----------|
| ffmpeg | HLS/DASH muxing, format conversion | Strongly recommended |
| Chrome | CF bypass and token-bound CDN intercept | Required for protected sites |
| yt-dlp | YouTube and generic fallback | Recommended |
### Python packages

Python packages:
Install the required packages with:

```bash
pip install -r requirements.txt
```
pip install DrissionPage curl_cffi yt-dlp

The main dependencies are:

| Package | Purpose |
| -------------- | ------------------------------------------------ |
| `yt-dlp` | Video extraction and downloading |
| `curl_cffi` | HTTP requests with browser like TLS fingerprints |
| `DrissionPage` | Chrome automation and browser based extraction |

### System dependencies

| Dependency | Purpose | Installation |
| ------------ | ------------------------------------- | ---------------------------------------- |
| Python 3.10+ | Runs the application | [python.org](https://python.org) |
| Chrome | Browser based extraction | Install Google Chrome |
| FFmpeg | Format conversion and post processing | `winget install ffmpeg` on Windows |
| FFmpeg | Format conversion and post processing | `brew install ffmpeg` on macOS |
| FFmpeg | Format conversion and post processing | `apt install ffmpeg` on Debian or Ubuntu |

Chrome must be installed for the DrissionPage fallback to work.

FFmpeg is optional, but recommended if you want reliable format conversion.

## Installation

Clone the repository:

```bash
git clone https://github.com/yourname/scrape.git
cd scrape
```

Install ffmpeg:
- Windows: https://ffmpeg.org/download.html, add the bin folder to PATH
- Or via winget: `winget install ffmpeg`
Install the Python dependencies:

---
```bash
pip install -r requirements.txt
```

Make sure Chrome and FFmpeg are available on your system.

## Install
You can verify FFmpeg with:

```bash
git clone https://github.com/yourusername/scraper.git
cd scraper
pip install DrissionPage curl_cffi yt-dlp
ffmpeg -version
```

No virtual environment required, but use one if you prefer.
Then start the downloader:

---
```bash
python scraper.py
```

## Usage

Run the application without arguments:

```bash
python scraper.py https://example.com/video
python scraper.py
```

Or run without arguments and paste the URL when prompted:
Paste the video URL when prompted, then choose the desired output format.

You can also provide the URL directly:

```bash
python scraper.py
python scraper.py https://www.youtube.com/watch?v=dQw4w9WgXcQ
```

You will then be asked for an output format:
Supported output formats include:

```text
mp4
mp3
mkv
webm
original
```
Output format:
1. mp4
2. mp3
3. mkv
4. webm
5. original <- keeps original container/quality
Choice [1]:

Downloaded files are placed in:

```text
videos/
```

Press Enter for mp4. Type a number or a custom extension (flac, opus, avi, etc).
The directory is created automatically when needed.

Output lands in `./videos/`.
## YouTube and 403 errors

---
YouTube can require browser authentication and additional verification when downloading certain streams.

## How it works
`scrape` attempts the following when a normal `yt-dlp` download encounters a 403:

```text
yt-dlp
|
+-- Edge cookies
|
+-- Chrome cookies
|
+-- Firefox cookies
```

If you are logged into YouTube in one of these browsers, the corresponding cookies may allow `yt-dlp` to access streams that otherwise return a 403.

**YouTube / Shorts / Live** — detected by URL, handed straight to yt-dlp with best quality up to 1080p merged to the chosen format. No browser, no scraping.
Keeping `yt-dlp` updated is also important. `scrape` checks for updates when it starts and updates the installed version when necessary.

**Everything else** runs through four layers in order:
## Cloudflare protected sites

1. Direct HTTP fetch via curl_cffi (Chrome TLS fingerprint)
2. Real Chrome via DrissionPage if step 1 hits a 403 or CF challenge
3. HTML scan for media URLs, iframe player fetch, base64 decode
4. yt-dlp generic fallback
For sites protected by Cloudflare, the downloader can launch an actual Chrome session rather than relying entirely on direct HTTP requests.

If a token-bound CDN URL is detected (pipe-signature pattern), the tool opens the player in Chrome, intercepts the live CDN request, then downloads with ffmpeg.
DrissionPage controls Chrome and allows the page to complete its JavaScript based checks normally.

---
Once the page is loaded, `scrape` can inspect the page and monitor browser network traffic for media requests.

## Output formats
This is particularly useful when a site does not expose the final video URL in its initial HTML.

When you pick mp4, mkv, or webm: ffmpeg remuxes the stream into that container.
## Token bound CDN URLs

When you pick mp3, aac, flac, opus, m4a: audio is extracted, video discarded.
Some sites generate temporary CDN URLs only after the video player starts.

When you pick original: downloaded as-is, no remux.
In these cases, looking at the page source is not enough.

Custom extensions work too: type `avi`, `mov`, `ts`, whatever ffmpeg supports.
`scrape` can monitor browser network requests and identify media URLs generated during playback. When a usable stream URL is found, it is passed to the appropriate downloader or FFmpeg processing stage.

---
These URLs may be temporary or tied to the browser session, so they are not expected to remain valid indefinitely.

## Config
## Extraction order

All tunable constants are at the top of the file:
The downloader attempts extraction in the following order:

```python
OUTPUT_DIR = "videos" # output folder
MAX_RETRIES = 3 # retry count on direct download failures
MIN_MB = 2 # files smaller than this are rejected
YTDLP_TIMEOUT = 3600 # max seconds for yt-dlp (1 hour)
FFMPEG_TIMEOUT = 3600 # max seconds for ffmpeg
STREAM_TIMEOUT = 30 # per-chunk connect/read timeout
1. Native `yt-dlp` extraction for supported platforms
2. Direct HTTP extraction through `curl_cffi`
3. Chrome based extraction through DrissionPage
4. HTML, iframe, and base64 inspection
5. Browser network interception
6. Download and post processing through `yt-dlp` or FFmpeg

This allows the simplest method to handle normal sites while keeping browser automation as a fallback for more difficult ones.

## Output

Files are saved automatically inside the `videos` directory:

```text
scrape/
├── scraper.py
├── requirements.txt
├── README.md
└── videos/
└── downloaded_video.mp4
```

---
Existing files are skipped, so running the downloader again will not unnecessarily download the same file.

## Planned
## FFmpeg

- GUI with queue, progress bar, output folder picker
- 4K / quality selector flag
- Batch mode: read URLs from a text file
- YouTube playlist support
- Resume support via Range header
- `--dry-run` flag
- Structured log file per session
- Twitter/X dedicated path (currently works via intercept)
- Instagram Reels
- Bilibili with cookie injection
FFmpeg is used for operations such as:

---
* Converting between supported formats
* Extracting audio
* Merging separate audio and video streams
* Post processing downloads

## Repo setup (first time)
Without FFmpeg, some downloads and conversions may be limited by the formats provided directly by the source.

Create a new repo on GitHub with no README, no gitignore, no license (you will add these yourself).
## Limitations

Then in your project folder:
`scrape` cannot guarantee that every site will work.

```bash
git init
git add scraper.py README.md .gitignore
git commit -m "init"
git branch -M main
git remote add origin https://github.com/yourusername/scraper.git
git push -u origin main
```
Modern video platforms can use DRM, encrypted streams, authentication, expiring tokens, browser fingerprints, or site specific APIs that change without notice.

Suggested `.gitignore`:
Cloudflare handling also depends on the site configuration and the browser being able to complete its checks normally.

```
videos/
__pycache__/
*.pyc
*.part
*.part.mp4
.env
```
When a site changes its player or delivery system, the corresponding extraction layer may need to be updated.

For future changes:
## Legal and responsible use*

```bash
git add scraper.py
git commit -m "what you changed"
git push
```
Only download content that you have permission to download and use.

---
This project is intended for personal use, testing, research, and legitimate media retrieval. Respect the terms of service, copyright, and access restrictions of the websites you use it with.

## License

MIT
This project is released under the MIT License.
2 changes: 1 addition & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
DrissionPage
curl_cffi
DrissionPage
yt-dlp
Loading