Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
-
Updated
May 10, 2026 - Python
Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
Customs-house auditor for AI agents. Scores 8 dimensions covering crawler policy, MCP/ACP exposure, agent-actionable Product/Offer, brand identity corroboration; generates drop-in robots.txt + .well-known manifests. MIT, a gift to the Algorand ecosystem.
Open specification for agent-readiness — 25 checks across HTTP-protocol and content-shape practices, organized into a 3-tier ladder (L1–L3). CC-BY 4.0.
Add a description, image, and links to the crawler-policy topic page so that developers can more easily learn about it.
To associate your repository with the crawler-policy topic, visit your repo's landing page and select "manage topics."