Inject AI crawler rules into Magento 2 robots.txt (OAI-SearchBot, GPTBot, Google-Extended)
-
Updated
Jun 14, 2026 - PHP
#
oai-searchbot
Here are 3 public repositories matching this topic...
Production indexability and AI crawler monitoring CLI plus GitHub Action: noindex, canonical, robots.txt, and optional edge checks.
cli waf cloudflare robots-txt openai noindex claude canonical perplexity github-actions github-action website-monitoring technical-seo ai-crawler anthropic oai-searchbot crawler-checker crawler-monitoring claude-searchbot
-
Updated
Jul 13, 2026 - JavaScript
AI Observatory — verified AI and search crawler retrieval monitoring for business websites
gptbot ai-observatory ai-visibility ai-retrieval perplexitybot claudebot oai-searchbot ai-crawler-monitoring crawler-telemetry server-side-monitoring
-
Updated
Aug 21, 2026
Improve this page
Add a description, image, and links to the oai-searchbot topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the oai-searchbot topic, visit your repo's landing page and select "manage topics."