screamingfrog-skill

A Claude Code skill that wraps the ScreamingFrog SEO Spider — letting Claude run headless crawls, read existing crawls' Derby database directly, diff two crawls, and produce client-facing HTML audit reports.

What this is

A Markdown file plus a handful of scripts that live under ~/.claude/skills/screamingfrog/. When installed, Claude Code surfaces it as /screamingfrog and auto-triggers on phrases like "crawl this site", "compare these two crawls", "audit report for X". The actual heavy lifting is done by:

• crawl.sh — wraps screamingfrogseospider --headless --save-crawl ... with sensible defaults (project-name derivation, timestamped output, auto-PageSpeed if a key is on disk).
• query.py — copies the embedded Apache Derby database that ScreamingFrog produces (~/.ScreamingFrogSEOSpider/ProjectInstanceData/<uuid>/results_*/sql/) and runs canned or ad-hoc SQL against it. Uses only Python stdlib + the Derby jars that ship with ScreamingFrog.
• diff.py — set-difference URLs between two crawls; surface status-code transitions and title/H1 changes.
• report.py — runs all canned checks, computes a 0–100 health score, renders a standalone HTML audit (Chart.js via CDN, localStorage-backed recommendations checklist).
• sql/*.sql — canonical checks: broken links, redirect chains, missing/duplicate titles, missing H1, thin content, oversized pages, noindex-with-inlinks, orphan candidates, plus a summary scorecard.

Prerequisites

1. A licensed ScreamingFrog SEO Spider installation. Headless mode requires a paid license. Without one, crawl.sh will refuse to run and screamingfrogseospider --headless crawls cap at 500 URLs. This skill does not include a license — you must bring your own. Save it as two lines (username, key) at ~/.ScreamingFrogSEOSpider/licence.txt:

   YOUR_USERNAME
   YOUR_LICENSE_KEY
   

   The query.py / diff.py / report.py flows work on existing GUI-produced crawls regardless of license state — license is only needed for fresh headless crawls.
2. Linux or macOS with screamingfrogseospider on $PATH (and the bundled Derby jars under /usr/share/screamingfrogseospider/lib/ on Linux — the path is hardcoded in query.py and easy to adjust for other platforms).
3. Java (already a dependency of ScreamingFrog).
4. Python 3.10+ (uses dict | None syntax). No pip dependencies.
5. Claude Code. The skill loads at session start via ~/.claude/skills/screamingfrog/SKILL.md.

Installation

cd ~/.claude/skills
git clone https://github.com/<your-user>/screamingfrog-skill screamingfrog

Then start a new Claude Code session. The skill will appear in the available-skills list with description "Run, read, diff, and report on Screaming Frog SEO Spider crawls…".

Direct CLI usage

The scripts work fine outside Claude Code:

# List crawls on disk
~/.claude/skills/screamingfrog/scripts/list_crawls.py

# Crawl a new site
~/.claude/skills/screamingfrog/scripts/crawl.sh https://example.com

# Run a canned check on the most recent matching crawl
~/.claude/skills/screamingfrog/scripts/query.py --crawl example.com --check broken_links --csv

# Ad-hoc SQL
~/.claude/skills/screamingfrog/scripts/query.py --crawl example.com \
  --sql "SELECT COUNT(*) FROM URLS WHERE RESPONSE_CODE >= 400"

# Diff two crawls (uuids, domain substrings, or sql/ paths)
~/.claude/skills/screamingfrog/scripts/diff.py old-crawl new-crawl --html /tmp/diff.html

# Full audit report
~/.claude/skills/screamingfrog/scripts/report.py --crawl example.com --out ~/audit.html

Canned SQL checks

Drop a new .sql file in sql/ and it shows up in query.py --list automatically. The checks shipped here:

Check	What it surfaces
broken_links	Internal URLs returning 4xx/5xx, with referring page count
redirect_chains	Redirect hops of length ≥ 2
missing_or_dup_titles	Empty or non-unique <title> on internal HTML 200s
missing_h1	Internal HTML 200s with no <h1>
thin_content	Internal HTML 200s under 300 words
large_pages	Internal HTML 200s over 500KB transferred
noindex_with_inlinks	Pages with a noindex meta robots tag still receiving internal links
orphan_candidates	Internal HTML 200s with zero internal hyperlink inlinks
summary	Top-level counters used by report.py

ScreamingFrog Derby schema notes

Some pitfalls discovered while building this — useful if you're writing new checks:

• URLS.IS_INTERNAL is BOOLEAN, not integer. Use IS_INTERNAL = TRUE (or FALSE), not = 1.
• There is no INDEXABILITY, IS_REDIRECT, or IS_CANONICALISED column — ScreamingFrog computes those in the UI. Reconstruct from META_CONTENT_n LIKE '%noindex%' etc.
• The page size columns are PAGE_SIZE and PAGE_TRANSFER_SIZE, not SIZE_BYTES.
• ScreamingFrog holds an exclusive lock on the Derby DB while running. Always copy sql/ to a scratch dir and remove db.lck / dbex.lck before connecting.
• The results_<uuid>/ folder's UUID is not the same as the project dir's UUID. Glob results_*/sql to find it.
• SYSCS_UTIL.SYSCS_EXPORT_QUERY emits CSV with no header row. Column names are parsed in Python from the SELECT clause (handles AS aliases, dotted refs, subquery columns).
• Derby's ij console truncates header rows on wide VARCHARs and rejects FETCH FIRST 0 ROWS ONLY (minimum is 1).
• Pass -Dderby.stream.error.file=/dev/null to Java when invoking ij, or it litters CWD with derby.log.

Customizing

• Per-client crawl configs: save a .seospiderconfig file (exported from the ScreamingFrog GUI via File > Configuration > Save As…) under configs/ and pass to crawl.sh --config. Useful for sites needing JS rendering, authentication, or custom extractions.
• New checks: drop a .sql file in sql/. Conventions: column aliases via AS, internal-only filter via IS_INTERNAL = TRUE, status filter via RESPONSE_CODE = 200 AND CONTENT_TYPE LIKE '%text/html%' when you want only indexable HTML.
• Report styling: edit the inline HTML/CSS in scripts/report.py. There's no template engine — it's a Python f-string.
• Health-score weights: see WEIGHTS in scripts/report.py.

License

MIT. See LICENSE.

Acknowledgements

• ScreamingFrog SEO Spider for being the best technical SEO tool out there and for shipping a sane CLI mode.
• Apache Derby for being embeddable and SQL-standards-compliant enough to query directly.
• Chart.js for the report donut.

Disclaimer

This is an independent project. Not affiliated with or endorsed by ScreamingFrog Ltd. ScreamingFrog SEO Spider is a paid product with its own license terms — review ScreamingFrog's licensing page for what your license allows.