Runnable, tested artifacts for every production-worthy code block in SEO for Engineers by Nima Jafari. Sitemaps, robots.txt CI gates, Googlebot verification, log analysis, and more, in Python & TypeScript.
SEO crawl budget manipulation methodology as a Claude Skill — fast indexing for low-competition long-tail keywords
Turn raw server/CDN access logs into verified-crawler analytics: parse many log formats, verify Googlebot/Bingbot by IP (not user-agent), enrich with DuckDB, and analyze crawl budget, crawl waste, and crawl traps with SQL.
A framework-agnostic reference for handling faceted navigation without wrecking crawl budget: a five-tier classifier (index / canonicalize / noindex / robots-block) driven by one facet registry, with sitemap, robots.txt, and parameter normalization.
Streaming Apache/Nginx log analyzer for SEO and crawl-budget analysis on multi-GB logs. 70+ bot families incl. modern AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider), reverse-DNS verification, 24 reports, React dashboard. Tested at 16 GB / 50.9M lines.
Add a description, image, and links to the crawl-budget topic page so that developers can more easily learn about it.
To associate your repository with the crawl-budget topic, visit your repo's landing page and select "manage topics."