On-Page SEO Analyzer

An on-page SEO analyzer that crawls a site and turns the result into actionable audits: SEO issue reports (titles, meta descriptions, canonicals, structured data, broken links), a comprehensive Markdown audit, optional LLM-powered title/meta-description fix suggestions, and an MCP server that exposes it all to AI agents. The Crawlee + Playwright crawler underneath extracts Google-supported SEO tags, structured data (JSON-LD/microdata), and AI-indexing metadata; the reporting layer is where the value is.

The typical workflow is crawl once → run reports — see Reports & Analysis.

Docker Compose is the recommended runtime (reproducible, no host setup) — run crawler commands inside the app service. Local development without Docker also works: npm install then use the npm run scripts directly (Node.js >= 20 + npx playwright install chromium).

🚀 Features

🎯 SEO Analysis

Complete Google Meta Tags: Extract all Google-supported meta tags including robots, viewport, social media tags
Canonical & Hreflang: International SEO and duplicate content management
Link Analysis: Categorize internal vs external links with detailed attributes
Response Monitoring: HTTP status codes, headers, and redirect tracking
Bot Detection Bypass: Advanced fingerprinting and realistic browsing behavior

🌐 Advanced Crawling

Smart Incremental Crawling: Only crawl new/modified content based on sitemap comparison
Human-like Behavior: Automatic page scrolling and realistic delays (3-15 seconds)
Full Browser Simulation: 1920x1080 viewport with comprehensive browser fingerprinting
Anti-Detection: Stealth mode with automatic user-agent rotation on 403 errors
Visual Debugging: Optional visible browser mode with 60-second error pauses
Smart 403 Retry: Automatic retry with visible browser when headless mode is blocked
Advanced Filtering: Domain exclusion + path-based exclusions (e.g., /user/login)

🤖 AI-Powered Indexing

Structured Data: JSON-LD and Schema.org microdata extraction
Custom Metadata: AI-specific tags for enhanced search indexing
Content Metrics: Automatic word count, reading time, and heading structure analysis
PageMap Support: Advanced search filtering attributes for AI systems

🗺️ Site Discovery & Storage

Automatic Sitemap Discovery: Parse XML sitemaps and sitemap indexes
Path Tracking: Generate comprehensive site structure maps
Domain Filtering: Stay within target domain boundaries with advanced exclusion rules
Domain-based Storage: Organized data structure: storage/domain.com/DD-MM-YYYY/datasets/
Date-based Organization: Automatic timestamped storage for historical tracking
Real-time Storage: Save data during crawling with individual files and JSONL format
Multi-domain Merging: Combine all domain JSONL files into unified dataset with metadata

📝 Input Configuration

Target URL Options

Docker Compose CLI (Recommended): Run npm scripts inside the app service
Configuration File: Set multiple URLs in YAML configuration
Apify Input: Provide via actor input when running on Apify platform

Domain Exclusion Configuration

Exclude specific subdomains or domains from crawling:

Configuration File Method

targets:
  excludedDomains:
    - 'api.example.com' # Exclude API subdomain
    - 'cdn.example.com' # Exclude CDN subdomain
    - 'static.example.com' # Exclude static assets

Command Line Method

# Exclude specific subdomains
docker compose run --rm app npm run crawl -- https://example.com --exclude-domains "api.example.com,cdn.example.com"

# Short version
docker compose run --rm app npm run crawl -- https://example.com --exclude "api.example.com,static.example.com"

# Exclude a subdomain, visible browser, with rate limiting
docker compose run --rm app npm run crawl -- https://example.com --exclude "accounts.example.com" --headless=false --rate-limit=conservative

Path-Based Exclusions

Exclude specific URL paths from crawling:

Configuration File Method

targets:
  excludedPaths:
    - '/user/login' # Exclude login pages
    - '/admin' # Exclude admin section
    - '/api/' # Exclude API endpoints
    - '/private' # Exclude private pages

Command Line Method

# Exclude specific paths
docker compose run --rm app npm run crawl -- https://example.com --exclude-paths "/user/login,/admin,/api/"

# Combined with domain exclusions
docker compose run --rm app npm run crawl -- https://example.com --exclude-domains "api.example.com" --exclude-paths "/user/login"

🔄 Smart Incremental Crawling

Dramatically reduce crawl time by only processing new or modified content since your last crawl.

How It Works

The crawler compares the current sitemap with your previous crawl data to identify:

New URLs: Pages that didn't exist before
Modified URLs: Pages with updated lastmod dates or failed in previous crawls
Unchanged URLs: Content that hasn't changed (skipped)
Removed URLs: Pages that no longer exist

Usage

Command Line

# Enable incremental mode with specific previous crawl date
docker compose run --rm app npm run crawl -- https://example.com --incremental --incremental-date 12-07-2025

# Enable incremental mode with auto-detection (fallback to full crawl)
docker compose run --rm app npm run crawl -- https://example.com --incremental

Configuration File

crawler:
  incrementalMode: true
  incrementalConfig:
    previousCrawlDate: '12-07-2025' # DD-MM-YYYY format
    mode: 'incremental' # incremental | new-only | modified-only | all
    autoDetectPreviousCrawl: true
    maxAgeThresholdDays: 30 # Consider URLs older than 30 days as modified

Standalone Comparison

# Compare sitemaps and see what would be crawled
docker compose run --rm app npx tsx scripts/compare-sitemaps.ts --domain example.com --previous-date 12-07-2025

# Get only new URLs as JSON for scripting
docker compose run --rm app npx tsx scripts/compare-sitemaps.ts --domain example.com --mode new-only --output json

# Get incremental URL list
docker compose run --rm app npx tsx scripts/compare-sitemaps.ts --domain example.com --output list --limit 100

Incremental Modes

incremental: Crawl new + modified URLs (recommended)
new-only: Only crawl completely new URLs
modified-only: Only crawl URLs that have changed
all: Full crawl regardless of previous data

Expected Results

📊 Sitemap Comparison Summary:
├── 🆕 New URLs: 45
├── 🔄 Modified URLs: 23
├── ✅ Unchanged URLs: 892
├── 🗑️ Removed URLs: 12
└── 📋 Total current URLs: 960

🎯 Incremental crawl will process 68 URLs (7.1% of total)

Time Savings: Reduce crawl scope by 70-90% for established sites!

⏱️ Advanced Rate Limiting

Prevent overwhelming target servers with advanced rate limiting that supports multiple time windows and intelligent request distribution.

How It Works

The rate limiting system uses sliding time windows to accurately track and limit requests:

Multiple Rules: Apply several rate limits simultaneously (e.g., 100/hour + 300/3hours)
Sliding Windows: More accurate than fixed time periods
Persistent Tracking: Request history saved across crawler sessions
Smart Distribution: Automatically calculates optimal delays between requests

Usage

Docker Compose CLI (Recommended)

# Use built-in presets
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=conservative  # 100 requests/hour
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=moderate      # 200 requests/2h
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=aggressive    # 500 requests/3h
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=bulk          # 1000 requests/5h
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=tiered        # Multiple rules

# Custom format: "requests/hours"
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=200/3         # 200 requests per 3 hours
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=100/1         # 100 requests per hour
docker compose run --rm app npm run crawl -- https://example.com --rate-limit=600/5         # 600 requests per 5 hours

Configuration File

crawler:
  rateLimiting:
    enabled: true
    persistData: true # Save request history across sessions

    # Use a preset (uncomment one):
    # preset: "conservative"       # 100 requests per hour
    # preset: "moderate"           # 200 requests per 2 hours
    # preset: "aggressive"         # 500 requests per 3 hours
    # preset: "bulk"               # 1000 requests per 5 hours
    # preset: "tiered"             # Multiple rules: 120/h, 300/3h, 600/5h

    # Custom rules (overrides preset):
    rules:
      - windowHours: 1 # 1 hour window
        maxRequests: 100 # Max 100 requests per hour
        enabled: true
        description: 'Hourly limit'

      - windowHours: 3 # 3 hour window
        maxRequests: 250 # Max 250 requests per 3 hours
        enabled: true
        description: '3-hour limit'

Built-in Presets

Preset	Description	Use Case
`conservative`	100 requests/hour	Gentle crawling, small sites
`moderate`	200 requests/2 hours	Balanced performance
`aggressive`	500 requests/3 hours	Fast crawling, larger sites
`bulk`	1000 requests/5 hours	High-volume operations
`tiered`	120/h + 300/3h + 600/5h	Multiple concurrent limits

Rate Limiting Status

The crawler provides real-time rate limiting information:

📊 Rate Limit Status:
├── Last hour: 45 requests
├── Last 2 hours: 89 requests
├── Last 3 hours: 134 requests
└── Last 5 hours: 201 requests

✅ Requests allowed
⏱️ Recommended delay: 36s

When rate limited:

🚫 BLOCKED by 1h rule (100 max)
⏰ Next request allowed in: 15 minutes

Features

Sliding Time Windows: More accurate than fixed time periods
Multiple Concurrent Rules: Apply several limits simultaneously
Persistent Storage: Request history survives crawler restarts
Smart Distribution: Even request spacing to avoid bursts
Real-time Monitoring: Status updates every 10 requests
Automatic Delays: Built-in waiting when limits are reached

Optional Configuration

Max pages to crawl: Limit the number of pages (default: unlimited)
Max concurrency: Number of concurrent requests (default: 1)
Request delay: Delay between requests in milliseconds (default: 1000-3000ms)
Sitemap discovery: Auto-discover URLs from XML sitemaps (default: true)
Headless mode: Control browser visibility - false=visible, true=invisible (default: false)
Domain exclusion: Filter out specific subdomains (API, CDN, static assets)

Data Extraction Modules

Enable/disable specific data extraction:

✅ Basic data: Title, URL, timestamp
✅ Response data: HTTP status, headers
✅ Links: Internal/external link analysis
✅ SEO tags: Google meta tags
✅ Special links: Canonical, hreflang
✅ Structured data: JSON-LD, microdata
✅ AI metadata: Custom fields
✅ Content metrics: Word count, reading time
⚪ PageMap data: Advanced search attributes

Performance Options

Skip heavy extraction: Disable expensive operations for speed
Request timeout: Maximum time to wait per page (default: 60s)
Debug mode: Enable detailed logging and browser visibility

📊 Output Data

The crawler provides comprehensive data for each crawled page:

{
  "title": "Page Title",
  "url": "https://example.com/page",
  "timestamp": "2025-01-12T10:30:00.000Z",
  "response": {
    "status": 200,
    "statusText": "OK",
    "headers": {
      /* HTTP headers */
    }
  },
  "links": {
    "internal": [
      /* same-domain links */
    ],
    "external": [
      /* external links */
    ],
    "total": 45
  },
  "seo": {
    "metaTags": {
      "description": "Page description",
      "robots": "index, follow",
      "og:title": "Social media title"
    },
    "specialLinks": {
      "canonical": "https://example.com/canonical-url",
      "alternate": [
        /* language alternatives */
      ]
    }
  },
  "aiMetadata": {
    "structuredData": {
      "jsonLd": [
        /* Schema.org objects */
      ],
      "microdata": [
        /* Microdata items */
      ]
    },
    "customMetadata": {
      "wordCount": 1250,
      "readingTime": "7 min",
      "headingStructure": [
        /* H1-H6 hierarchy */
      ]
    }
  }
}

📊 Reports & Analysis

After crawling a domain, generate analysis reports from the stored dataset. All reports read from storage/datasets/<domain>/ and write to storage/reports/<domain>/<date>/ (JSON, with most also supporting --csv). Run any of them inside the app container.

Reports are generated from already-crawled data — crawl the site first (see Quick Start). The active link/href checkers additionally make live HTTP requests.

Language: every report accepts --language <code> — empty/absent = English (default), cs = Czech. Czech output is written to a separate …-cs.{md,json,csv} file so both languages can coexist. Example: … npm run seo-audit -- --domain example.com --language cs.

Crawl scope — reports describe the latest crawl, not the whole history. storage/datasets/ merges every crawl of a domain into one JSONL, so it accumulates the union of all URLs ever seen. A URL retired between crawls (now redirected, so nothing links to it and the crawler never reaches it again) keeps its last record forever. Reports therefore default to the newest crawl date and print how many stale URLs they excluded:

📄 example.com — snapshot 28-08-2026: 311 pages
   ⚠ 47 URL(s) excluded — not seen in the latest crawl, last seen 19-07-2025 … 07-08-2026
     → use --all-crawls to include every crawl ever recorded

Pass --all-crawls to any report (report:404, report:seo-issues, report-link-graph-issues, generate-title-description-fixes, seo-audit) to get the historical union instead — useful for "has this URL ever 404ed", misleading as a picture of the site today. seo-audit --date DD-MM-YYYY still pins one specific crawl. The 404 report additionally keeps its excluded findings in a stale_not_in_latest_crawl array (and a snapshot/last_seen column pair in the CSV), so a dead legacy URL is visible but never counted as a current issue without re-verification.

Incremental crawls are handled as deltas, not snapshots. A --incremental run writes only the URLs it re-fetched, so its date folder is not a picture of the site on its own — scoping a report to it would drop every page that happened not to change. Each crawl therefore records a _crawl-meta.json manifest in its date folder (mode: full | incremental), and reports anchor the snapshot on the newest full crawl, layering every incremental crawl since on top of it:

📄 example.com — snapshot 12-08-2026 … 28-08-2026 (+2 incremental crawls): 311 pages

Date folders crawled before the manifest existed carry no mode and count as full crawls, so existing datasets behave exactly as they did.

Report JSON carries its crawl scope. Every persisted report JSON begins with schema_version, generated_at and the crawl(s) it was built from, and per-domain report filenames are stamped with the crawl date rather than today's — a saved report from a months-old crawl must not look current. schema_version is 2; version 1 was the unversioned pre-snapshot shape (the 404 report was a bare ReportEntry[] array rather than an object), so any external consumer reading 404-link-report-*.json as an array needs updating to read .entries.

SEO Audit — `seo-audit.ts`

A comprehensive Markdown audit of a crawled site. Per page it checks indexability (noindex/non-200), missing <title>, overlong title (>63 chars), missing/overlong meta description (>163), missing canonical, missing Open Graph (og:title/og:description/og:image), missing twitter:card, absent JSON-LD structured data, thin content (<300 words), and orphan pages (no internal links). It also aggregates structured-data coverage, classifies page types (Homepage, Service, FAQ, Branch/Contact, …), and emits a prioritized recommendations table.

docker compose run --rm app npm run seo-audit -- --domain example.com
# Options: --date DD-MM-YYYY (pick a crawl date), --all-crawls (union of every crawl),
#          --output <file.md>
# Output:  storage/reports/seo-audit-<domain>-<date>.md

PDF Export — `report-pdf.ts`

Renders an existing audit report to a print-ready A4 PDF next to the .md, with a linked table of contents, repeating table headers and page numbers. Raw Markdown reads badly on a phone, so this is what /api/crawl attaches to the result email — the server renders it lazily at send time and caches it, so this script is only needed to preview the typography or to backfill older reports.

Rendering uses the Chromium already present in the image (no extra dependency); renders are serialised one at a time and bounded by SEO_PDF_TIMEOUT_MS (default 120 s) so a busy container cannot end up holding two browsers at once. If a render fails, the email falls back to attaching the .md.

docker compose run --rm app npm run report:pdf -- --domain example.com
# Options: --md <report.md> (explicit file), --date DD-MM-YYYY, --force (re-render)
# Output:  storage/reports/<domain>/<date>/seo-audit-<date>[-cs].pdf

SEO Issues — `report-seo-issues.ts`

Focused, machine-readable issue lists for bulk fixing. Flags meta description and title problems categorized as missing / too_short / too_long / pixel_too_long / duplicate, plus heading-structure and structured-data gaps. Length is checked both by character count and by estimated pixel width (titles ~579px, meta descriptions ~919px — closer to how Google actually truncates SERP snippets).

docker compose run --rm app npm run report:seo-issues -- --domain example.com --csv
# Options: --domain <d>, --output-dir <dir>, --csv, --all-crawls
# Output:  per-issue JSON (and CSV with --csv) in storage/reports/

LLM Title/Meta Fix Suggestions — `generate-title-description-fixes.ts`

For each page with a flagged title or meta description, generates a rewritten suggestion using an LLM, grounded in the page's own heading structure and content excerpt. Provider-agnostic via an OpenAI-compatible client — point it at OpenAI or a local model (Ollama) through env vars: LLM_PROVIDER (openai | ollama), LLM_API_KEY, LLM_MODEL, LLM_BASE_URL.

docker compose run --rm app npx tsx scripts/generate-title-description-fixes.ts --domain example.com --csv
# Options: --domain <d> (required), --output-dir <dir>, --csv, --language <cs|en>, --all-crawls
# Output:  storage/reports/<domain>/title-description-fixes-<date>.json (+ .csv)

404 Link Report — `report-404s.ts`

Lists URLs that returned HTTP 404 among the pages the crawler actually visited, grouped with their referrers (which page linked to them, the link text, and crawl date) and the discovery source (linked_from_page vs seeded_or_sitemap). Fast — it reads crawl results only and makes no new requests.

docker compose run --rm app npm run report:404 -- --domain example.com --csv
# --domain is optional (processes all crawled domains if omitted)
# Options: --all-crawls (report every 404 ever seen, not just the latest crawl's)
# Output:  storage/reports/<domain>/404-link-report-<date>.json (+ .csv with --csv)
#          JSON shape: { schema_version: 2, domain, crawl_date, baseline_crawl_date,
#                        incremental_crawl_dates, snapshot_mode, pages_analyzed, total,
#                        entries: [...], stale_not_in_latest_crawl: [...] }
#          Breaking vs. schema_version 1: the file was a bare array of entries.

Broken Link Validation — `check-broken-links.ts`

Unlike the 404 report, this actively probes every referenced URL — internal links, external links, and image src/srcset/<picture> sources — with a HEAD request (falling back to GET) and reports any 4xx/5xx or network failure, grouped by the page(s) referencing it.

docker compose run --rm app npx tsx scripts/check-broken-links.ts --domain example.com
# Options: --concurrency 10, --timeout 20000, --skip-external, --status 403, --output <file.json>
#   --status <code> narrows the report to that exact status (e.g. 403) and folds in crawled
#   pages whose own response had that status but which nothing links to.
# Output:  storage/reports/<domain>/<code>-report-<date>.json/.csv

Empty / Missing href — `check-empty-href.ts`

Scans each page's raw HTML for anchors with broken hrefs: empty (href="" → reloads the page), missing (no href → non-navigable), hash (href="#"), and javascript: pseudo-links. Requires HTML content extraction (--html-content at crawl time, or extraction.modules.htmlContent: true).

docker compose run --rm app npx tsx scripts/check-empty-href.ts --domain example.com
# Options: --include hash,javascript (include review-only categories), --output <file.json>
# Output:  storage/reports/<domain>/empty-href-<date>.json (+ .csv)

🎯 Use Cases

SEO Auditing

Analyze meta tag completeness and optimization
Identify missing canonical URLs or hreflang tags
Monitor robots directives and indexing status
Track social media optimization (Open Graph, Twitter Cards)

Content Analysis

Extract structured data for rich snippets
Analyze content metrics (word count, reading time)
Monitor heading structure and content organization
Track custom metadata for content categorization

Technical SEO

Monitor HTTP response codes and redirects
Analyze response headers for performance insights
Track site structure and internal linking
Identify crawl errors and accessibility issues

AI-Powered Search

Extract metadata for Google Cloud AI App Builder
Support advanced filtering and content boosting
Enable rich content understanding for generative AI
Provide structured data for enhanced search experiences

🐳 Docker Compose Usage

Quick Start

# 1. Create your .env from the template (required first step), then edit values as needed
cp .env.example .env

# 2. Build the crawler image
docker compose build app

# 3. Crawl a site
docker compose run --rm app npm run crawl -- https://example.com --headless=true

# Crawl with exclusions and rate limiting
docker compose run --rm app npm run crawl -- https://example.com \
  --exclude-domains "api.example.com,cdn.example.com" \
  --rate-limit=conservative

Workflow Rules

Docker Compose is the recommended runtime: docker compose run --rm app ...
Use docker compose build app after Dockerfile or dependency changes
Prefer the container for reproducible runs; running the npm run scripts on the host (Node.js >= 20) is also supported for local development

Running the MCP server

docker-compose.yml describes the application on its own — no external Docker network, no reverse proxy, no sibling repositories. A fresh clone needs nothing beyond .env.

The mcp service exposes the crawler over HTTP and the Model Context Protocol. It refuses to start without a token, which is the right default for anything listening on a socket:

# generate a token and put it in .env
echo "SEO_MCP_TOKEN=$(openssl rand -hex 24)" >> .env

docker compose up -d mcp
curl -s http://127.0.0.1:3001/health          # {"status":"ok","activeJobs":0}

The port is published on 127.0.0.1 only. clientIp() trusts X-Forwarded-For, so the server must be reachable only through a proxy that overwrites that header — otherwise per-IP rate limiting can be bypassed by forging it.

Static frontend files are served from SEO_FRONTEND_DIR (default ./storage/frontend, an empty directory — the frontend routes 404 while the API and MCP endpoints work normally). Point it at your own build output if you have one.

Deploying behind a reverse proxy — an external network, vhost labels, extra mounts and so on — is deployment-specific and does not belong in this repository. Supply those with your own compose overlay:

docker compose -f docker-compose.yml -f /path/to/your-deployment.yml up -d mcp

Relative paths inside the overlay resolve against the base file's directory, not the overlay's.

Running NPM Scripts in Docker

# Crawl
docker compose run --rm app npm run crawl -- https://example.com --headless=true

# Merge all crawled JSONL files
docker compose run --rm app npm run merge-to-jsonl

# Build TypeScript
docker compose run --rm app npm run build

# Run tests
docker compose run --rm app npm test

# Lint and format
docker compose run --rm app npm run lint
docker compose run --rm app npm run format
docker compose run --rm app npm run style

Storage Structure

After running, your local storage will contain:

./storage/
├── datasets/
│   ├── domain.com/
│   │   └── 14-07-2025/
│   │       ├── crawl-data.jsonl      # Domain-specific JSONL data
│   │       ├── page-1-timestamp.json # Individual page files
│   │       └── page-2-timestamp.json
│   └── anotherdomain.com/
│       └── 15-07-2025/
├── key_value_stores/
│   └── domain.com/             # Sitemaps and metadata
├── request_queues/
│   └── domain.com/             # Processing queues
└── all-domains-merged.jsonl    # Unified JSONL from all domains

Data Export & Merging

Merge all domain data into unified JSONL:

# Merge all JSONL files from all domains and dates
docker compose run --rm app npm run merge-to-jsonl

# Output: ./storage/all-domains-merged.jsonl

Features:

Multi-domain Support: Combines data from all crawled domains
Metadata Enrichment: Adds domain, crawl date, and source file information
Historical Data: Preserves data from different crawl dates
Standard Format: JSONL output (one JSON object per line)

Example merged record:

{
  "title": "Page Title",
  "url": "https://example.com/page",
  "seo": { "metaTags": {...} },
  "_metadata": {
    "domain": "example.com",
    "crawlDate": "19-07-2025",
    "sourceFile": "example.com/19-07-2025/crawl-data.jsonl"
  }
}

🛠️ Docker Compose Workflows

Run every project command inside the app container:

# Run with specific target URL
docker compose run --rm app npm run crawl -- https://example.com

# Exclude specific domains/subdomains
docker compose run --rm app npm run crawl -- https://example.com --exclude-domains "api.example.com,cdn.example.com"

# Control browser visibility
docker compose run --rm app npm run crawl -- https://example.com --headless=false
docker compose run --rm app npm run crawl -- https://example.com --headless=true

# Single URL mode
docker compose run --rm app npm run crawl -- https://example.com --single

# Merge all domain JSONL files
docker compose run --rm app npm run merge-to-jsonl

# Build, test, and code quality
docker compose run --rm app npm run build
docker compose run --rm app npm test
docker compose run --rm app npm run lint
docker compose run --rm app npm run format
docker compose run --rm app npm run style

Quick Start Examples

# Crawl a blog for SEO analysis (with visible browser)
docker compose run --rm app npm run crawl -- https://myblog.com --headless=false

# Analyze e-commerce site structure (excluding API and CDN)
docker compose run --rm app npm run crawl -- https://mystore.com --exclude-domains "api.mystore.com,cdn.mystore.com"

# Debug crawling with visible browser
docker compose run --rm app npm run crawl -- https://example.com --headless=false

# Stealth crawling with invisible browser
docker compose run --rm app npm run crawl -- https://example.com --headless=true

# Test local development site
docker compose run --rm app npm run crawl -- http://localhost:3000 --headless=false

# Single page analysis with visible browser
docker compose run --rm app npm run crawl -- https://example.com --single --headless=false

# Visible browser with a domain exclusion
docker compose run --rm app npm run crawl -- https://www.example.com --headless=false --exclude-domains "accounts.example.com"

Configuration Files

config/crawler.yml - Main configuration (5 pages limit for testing)
config/examples/basic.yml - Simple crawling setup (50 pages)
config/examples/advanced.yml - Full-featured analysis (unlimited)
config/examples/performance.yml - Speed-optimized setup (1000 pages)

Key Configuration Options

targets:
  excludedDomains:
    - 'api.example.com'
    - 'cdn.example.com'
    - 'static.example.com'

crawler:
  maxRequestsPerCrawl: 0 # Page limit (0 = unlimited)
  maxConcurrency: 1 # Concurrent requests
  headless: false # Show browser window (true for stealth mode)
  requestDelayMin: 1000 # Min delay between requests (ms)
  requestDelayMax: 3000 # Max delay between requests (ms)

  # Rate limiting configuration
  rateLimiting:
    enabled: false # Enable/disable rate limiting
    persistData: true # Save request history across sessions
    preset: 'moderate' # Use built-in preset
    # OR define custom rules:
    rules:
      - windowHours: 1
        maxRequests: 100
        enabled: true
        description: 'Hourly limit'

  # Browser launch arguments (configurable in YAML)
  launchArgs:
    headless: # Stealth mode arguments
      - '--no-sandbox'
      - '--disable-setuid-sandbox'
      - '--disable-dev-shm-usage'
      - '--disable-blink-features=AutomationControlled'
      - '--disable-features=VizDisplayCompositor'

    visible: # Visible browser arguments
      - '--no-sandbox'
      - '--disable-setuid-sandbox'
      - '--disable-dev-shm-usage'
      - '--disable-blink-features=AutomationControlled'
      - '--no-first-run'
      - '--no-default-browser-check'

output:
  storage:
    realTimeStorage:
      enabled: true # Save files during crawling
      saveIndividualFiles: true # Individual JSON files
      saveJsonl: true # Continuous JSONL file

Configuration Best Practices

Browser Launch Arguments

Headless Mode: Optimized for stealth crawling with minimal detection
Visible Mode: Simplified arguments for debugging and development
Customizable: Easily modify browser behavior through YAML configuration
Maintainable: No hardcoded arguments in source code

Storage Organization

Domain-based: Automatic organization by target domain
Date-based: Daily folders for historical tracking
Real-time: Data saved during crawling for immediate analysis
Multiple formats: JSON, JSONL, and CSV export options

Command Line Options

# Basic usage
docker compose run --rm app npm run crawl -- <URL>

# Advanced options
docker compose run --rm app npm run crawl -- <URL> [OPTIONS]

Options:
  --url, -u <URL>                    Target URL to crawl
  --exclude-domains <domains>        Comma-separated list of domains to exclude
  --exclude <domains>                Short version of --exclude-domains
  --exclude-paths <paths>            Comma-separated list of URL paths to exclude
  --headless=<true|false>            Set headless mode (true=invisible, false=visible)
  --single, -s                       Single URL mode - don't follow links
  --incremental                      Enable incremental crawling mode
  --incremental-date <date>          Previous crawl date (DD-MM-YYYY format)
  --rate-limit=<preset|format>       Rate limiting: preset name or "requests/hours"
  --max-requests=<N>                 Hard cap on number of pages crawled
  --concurrency=<N>                  Max parallel page loads (default 2 from crawler.yml)
  --delay-min=<ms>                   Min pause after each page (default 50; 0 is valid)
  --delay-max=<ms>                   Max pause after each page (default 200)
  --max-retries=<N>                  Crawlee maxRequestRetries (default 3 = 4 attempts)
  --block-assets                     Skip CSS/images/fonts/JS subresources
  --ignore-robots                    Disable robots.txt enforcement
  --date-folder, --date <DD-MM-YYYY> Write into a specific date folder

Low-load crawling of third-party sites:
  Use --concurrency, --delay-min/--delay-max and --block-assets rather than
  --rate-limit. Without --block-assets each page navigation is a full browser load
  of every asset the page references, so one "request" in the crawl statistics can
  be dozens of hits on the origin server. Example — a deliberately gentle crawl:

    docker compose run --rm app npm run crawl -- https://example.com \
      --headless=true --max-requests=50 --concurrency=1 \
      --delay-min=4500 --delay-max=5500 --max-retries=1 --block-assets

  Avoid --rate-limit for this: the limiter sleeps inside the request handler, which
  Crawlee bounds by requestHandlerTimeoutSecs (60s), so a throttle longer than that
  turns into a timeout plus retries — more load, not less. The inter-request delay
  sleeps in the same place, so the two bounds are validated against each other and
  against that timeout: an inverted pair (min > max, including against the crawler.yml
  value when only one side is overridden) is swapped with a warning, and a delay above
  half the handler timeout is clamped. --block-assets filters by the browser's own
  resource type (stylesheet/image/media/font/script), so JSON/XHR responses the page
  fetches are never blocked.

Rate Limiting Presets:
  conservative  - 100 requests per hour
  moderate      - 200 requests per 2 hours
  aggressive    - 500 requests per 3 hours
  bulk          - 1000 requests per 5 hours
  tiered        - Multiple rules: 120/h, 300/3h, 600/5h


Command Line Argument Parsing (lines 31-54):
    - --url or -u: Target URL
    - --single or -s: Single URL mode
    - --exclude-domains: Comma-separated domains to exclude
    - --headless=true/false: Headless mode control

Examples:
  docker compose run --rm app npm run crawl -- https://example.com
  docker compose run --rm app npm run crawl -- https://example.com --exclude-domains "api.example.com,cdn.example.com"
  docker compose run --rm app npm run crawl -- https://example.com --exclude-paths "/user/login,/admin"
  docker compose run --rm app npm run crawl -- https://example.com --headless=false
  docker compose run --rm app npm run crawl -- https://example.com --rate-limit=moderate
  docker compose run --rm app npm run crawl -- https://example.com --rate-limit=200/3
  docker compose run --rm app npm run crawl -- https://example.com --incremental --incremental-date 19-07-2025 --headless=false --rate-limit=conservative
  docker compose run --rm app npm run crawl -- https://example.com --headless=true --single --rate-limit=conservative

 Usage examples:
  - docker compose run --rm app npm run crawl -- https://example.com --headless=false
  - docker compose run --rm app npm run crawl -- https://example.com --single
  - docker compose run --rm app npm run crawl -- https://example.com --single --headless=false

Domain Requirement

The crawler requires a target URL to be provided via:

Docker Compose command: docker compose run --rm app npm run crawl -- https://example.com
Configuration file: Set in config/crawler.yml
Docker environment: Configure in container

If no URL is provided, the crawler will exit with "Domain is required" error.

🤖 MCP Server & AI Persona (Marek)

The project includes an MCP (Model Context Protocol) server that enables LLM models and AI agent workflows (like Claude Code, Cursor, or custom gateway bots) to interact with the crawling engine and access reports.

It also integrates the AI persona Marek — a senior SEO consultant.

Key MCP Features:

Tools:
- crawl: Trigger a crawl job asynchronously.
- get_report: Retrieve report status or data for a domain.
- list_reports: List all crawled domains and their audit dates.
Prompts (Templates):
- seo-consultant-marek: Exposes Marek's persona instructions (compiled from ./ai/persona/*). Supports a domain argument which appends the latest crawl report as context.
Resources (Data Sources):
- seo://reports/{domain}/latest: Serves the latest generated Markdown audit report for the specified domain.

Running the MCP Server

Via Docker Compose (Recommended): The server runs on host port 3001 (mapped to container port 3000 on the mcp service).
Locally: bash npm run mcp Requires setting SEO_MCP_TOKEN in .env for Basic Authorization (if configured). For more details on the persona's role and rules, see docs/SEO-consultant.md.

📈 Performance Tips

Start with basic extraction modules for initial testing
Use max pages limit to control scope during development
Adjust concurrency based on target server capacity
Enable debug mode for troubleshooting crawling issues
Use performance mode for large-scale crawling

🔗 Integration

The tool integrates well with:

AI agents / LLM workflows via the built-in MCP server (see above)
Google Search Console for SEO monitoring
Data analysis tools via JSON / JSONL / CSV export
Content management systems for metadata enrichment

📞 Support

For issues or feature requests:

Check the Crawlee documentation
Review the configuration options in config/
Enable debug mode for detailed logging

Built with ❤️ for SEO professionals, content creators, and developers seeking comprehensive on-page SEO analysis

Name		Name	Last commit message	Last commit date
.actor		.actor
.github/workflows		.github/workflows
.husky		.husky
ai/persona		ai/persona
config		config
docs		docs
scripts		scripts
src		src
.codex		.codex
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.lintstagedrc.json		.lintstagedrc.json
.prettierignore		.prettierignore
.prettierrc.js		.prettierrc.js
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README-TESTS.md		README-TESTS.md
README.md		README.md
docker-compose.yml		docker-compose.yml
eslint.config.js		eslint.config.js
jest.config.js		jest.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
tsconfig.test.json		tsconfig.test.json

Latest commit

History

Folders and files

Repository files navigation

On-Page SEO Analyzer

🚀 Features

🎯 SEO Analysis

🌐 Advanced Crawling

🤖 AI-Powered Indexing

🗺️ Site Discovery & Storage

📝 Input Configuration

Target URL Options

Domain Exclusion Configuration

Configuration File Method

Command Line Method

Path-Based Exclusions

Configuration File Method

Command Line Method

🔄 Smart Incremental Crawling

How It Works

Usage

Command Line

Configuration File

Standalone Comparison

Incremental Modes

Expected Results

⏱️ Advanced Rate Limiting

How It Works

Usage

Docker Compose CLI (Recommended)

Configuration File

Built-in Presets

Rate Limiting Status

Features

Optional Configuration

Data Extraction Modules

Performance Options

📊 Output Data

📊 Reports & Analysis

SEO Audit — seo-audit.ts

PDF Export — report-pdf.ts

SEO Issues — report-seo-issues.ts

LLM Title/Meta Fix Suggestions — generate-title-description-fixes.ts

404 Link Report — report-404s.ts

Broken Link Validation — check-broken-links.ts

Empty / Missing href — check-empty-href.ts

🎯 Use Cases

SEO Auditing

Content Analysis

Technical SEO

AI-Powered Search

🐳 Docker Compose Usage

Quick Start

Workflow Rules

Running the MCP server

Running NPM Scripts in Docker

Storage Structure

Data Export & Merging

🛠️ Docker Compose Workflows

Quick Start Examples

Configuration Files

Key Configuration Options

Configuration Best Practices

Browser Launch Arguments

Storage Organization

Command Line Options

Domain Requirement

🤖 MCP Server & AI Persona (Marek)

Key MCP Features:

Running the MCP Server

📈 Performance Tips

🔗 Integration

📞 Support

About

Topics

Resources

Security policy

Stars

Watchers

Forks

SEO Audit — `seo-audit.ts`

PDF Export — `report-pdf.ts`

SEO Issues — `report-seo-issues.ts`

LLM Title/Meta Fix Suggestions — `generate-title-description-fixes.ts`

404 Link Report — `report-404s.ts`

Broken Link Validation — `check-broken-links.ts`

Empty / Missing href — `check-empty-href.ts`