robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Modern robots.txt analyzer with instant analysis, security recommendations, and export capabilities. Built with Qwik and deployed on Cloudflare Pages.

⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.

Google's robots.txt parser ported to TypeScript - RFC 9309 compliant

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

robotstxt gem

Simple robots txt parser for python

A web tool for validating URLs against robots.txt rules. Built with RFC 9309 compliant parsing.

A dependency-free robots.txt checker that makes it easy for respectful AI agents, crawlers, scrapers, and other bots to follow the rules. Includes fetching, parsing, and caching based on the RFC 9309 industry standard.

Free, open-source robots.txt tester — Google's retired tester rebuilt with RFC 9309 matching. Test any URL against Googlebot, GPTBot, ClaudeBot, PerplexityBot and more, with a one-click all-bots matrix.