Scrape entire websites into markdown using crawl4ai.
- Recursive crawling from any root URL
- Outputs clean markdown files preserving site structure
- Skips binary files (images, PDFs, etc.)
- Configurable page limits
- JavaScript rendering option
- Content filtering: prune low-quality content, filter by relevance, exclude tags
# Install dependencies
uv sync --dev && uv pip install -e .
# Setup playwright browsers
uv run crawl4ai-setup
uv run python -m playwright install# Scrape a website
uv run scrape https://docs.example.com
# Limit pages and custom output
uv run scrape https://docs.example.com -n 50 -o ./output
# Enable JavaScript rendering
uv run scrape https://docs.example.com --js# Automatic pruning (removes repetitive/low-quality content)
uv run scrape https://example.com -p 0.4
# Query-based relevance filtering
uv run scrape https://example.com -q "trading funded accounts"
# Exclude navigation, footer, aside
uv run scrape https://example.com -e nav -e footer -e aside
# Focus on main content area
uv run scrape https://example.com -c "main"
# Combined approach
uv run scrape https://example.com -p 0.4 -e nav -e footer -c "main"
# Skip URLs containing specific patterns
uv run scrape https://example.com -x "archive" -x "/old/"import asyncio
from recursive_scraper.scraper import ScrapeConfig, scrape_site
async def main():
config = ScrapeConfig(
output_dir="output",
prune_threshold=0.4,
excluded_tags=["nav", "footer", "aside"],
)
await scrape_site("https://docs.example.com", config=config)
asyncio.run(main())| Option | Description |
|---|---|
url |
Base URL to scrape (required) |
-o, --output-dir |
Output directory (default: output) |
-n, --max-pages |
Maximum pages to crawl (default: unlimited) |
--js |
Enable JavaScript rendering via headless browser |
-p, --prune-threshold |
Prune low-quality content (e.g., 0.4) |
-q, --relevance-query |
Filter content by relevance to query (BM25) |
-e, --excluded-tags |
HTML tags to exclude (repeatable) |
--excluded-selector |
CSS selector for elements to exclude |
-c, --css-selector |
CSS selector to focus on |
-x, --exclude-url-pattern |
Skip URLs containing pattern (repeatable) |
--no-remove-forms |
Keep form elements in output |
--no-remove-overlays |
Keep overlay elements in output |
See DEVELOPING.md for tooling and workflows.