Skip to content

About

A web scraper that given a root URL scrapes the content of the page and all of its subpages.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Recursive Scraper

Scrape entire websites into markdown using crawl4ai.

Features

  • Recursive crawling from any root URL
  • Outputs clean markdown files preserving site structure
  • Skips binary files (images, PDFs, etc.)
  • Configurable page limits
  • JavaScript rendering option
  • Content filtering: prune low-quality content, filter by relevance, exclude tags

Quick Start

# Install dependencies
uv sync --dev && uv pip install -e .

# Setup playwright browsers
uv run crawl4ai-setup
uv run python -m playwright install

Usage

# Scrape a website
uv run scrape https://docs.example.com

# Limit pages and custom output
uv run scrape https://docs.example.com -n 50 -o ./output

# Enable JavaScript rendering
uv run scrape https://docs.example.com --js

Content Filtering

# Automatic pruning (removes repetitive/low-quality content)
uv run scrape https://example.com -p 0.4

# Query-based relevance filtering
uv run scrape https://example.com -q "trading funded accounts"

# Exclude navigation, footer, aside
uv run scrape https://example.com -e nav -e footer -e aside

# Focus on main content area
uv run scrape https://example.com -c "main"

# Combined approach
uv run scrape https://example.com -p 0.4 -e nav -e footer -c "main"

# Skip URLs containing specific patterns
uv run scrape https://example.com -x "archive" -x "/old/"

As a Library

import asyncio
from recursive_scraper.scraper import ScrapeConfig, scrape_site

async def main():
    config = ScrapeConfig(
        output_dir="output",
        prune_threshold=0.4,
        excluded_tags=["nav", "footer", "aside"],
    )
    await scrape_site("https://docs.example.com", config=config)

asyncio.run(main())

CLI Options

Option Description
url Base URL to scrape (required)
-o, --output-dir Output directory (default: output)
-n, --max-pages Maximum pages to crawl (default: unlimited)
--js Enable JavaScript rendering via headless browser
-p, --prune-threshold Prune low-quality content (e.g., 0.4)
-q, --relevance-query Filter content by relevance to query (BM25)
-e, --excluded-tags HTML tags to exclude (repeatable)
--excluded-selector CSS selector for elements to exclude
-c, --css-selector CSS selector to focus on
-x, --exclude-url-pattern Skip URLs containing pattern (repeatable)
--no-remove-forms Keep form elements in output
--no-remove-overlays Keep overlay elements in output

Development

See DEVELOPING.md for tooling and workflows.

About

A web scraper that given a root URL scrapes the content of the page and all of its subpages.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages