Validates whether GPTBot, ClaudeBot, PerplexityBot and other LLM crawlers are allowed by a robots.txt file.
python geo robots-txt rfc-9309 gptbot generative-engine-optimization perplexitybot claudebot crawler-policy compliance-geo llm-crawlers
-
Updated
May 10, 2026 - Python