A dependency-free command-line tool and schedulable GitHub Action for checking production indexability state, AI crawler robots.txt policy, and optional synthetic homepage responses. It catches accidental noindex, canonical drift, response changes, and crawler-policy changes without adding a runtime dependency to the site.
The default check reports:
- homepage HTTP response
<meta name="robots">directivesX-Robots-Tagdirectives- canonical URL
- robots.txt policy for eight named crawler tokens
It reports the homepage rule for:
OAI-SearchBot,GPTBot, andOAI-AdsBotClaude-SearchBot,ClaudeBot, andClaude-UserPerplexityBotGoogle-Extended
The checker separates search discovery, user-requested retrieval, model training, and ad validation instead of treating every “AI bot” as the same thing.
Choosing a monitoring layer? Compare robots.txt monitoring approaches by what they actually watch: file changes, resolved policy, real crawler logs, general page changes, or edge enforcement.
Node.js 20 or newer is required.
Install with Homebrew on macOS or Linux:
brew install unitedideas/tap/actablesite-check
actablesite-check example.comOr run the tagged release directly from GitHub:
npx github:unitedideas/actablesite-check#v1.5.0 example.comCompare robots.txt with three synthetic AI search-crawler homepage requests:
actablesite-check example.com --edgeThis sends separately labeled OAI-SearchBot, Claude-SearchBot, and PerplexityBot user-agent requests. It reports the observed HTTP status and whether Cloudflare response headers were present.
Machine-readable output:
actablesite-check example.com --jsonPrint a conservative starter policy that allows discovery while restricting named training controls:
actablesite-check --starterRun the same check in CI without installing a package:
- name: Check production indexability
id: indexability
uses: unitedideas/actablesite-check@v1
with:
website: example.com
check-homepage-indexability: "true"
fail-on-noindex: "true"
fail-on-blocked: "false"
check-edge-responses: "true"
fail-on-restricted: "false"The action writes a job summary and exposes result, allowed-count, blocked-count, indexability-result, noindex-found, edge-result, and restricted-count outputs. Homepage indexability inspection is on by default; set fail-on-noindex to "true" when a discovered noindex should fail deployment. Set fail-on-blocked to "true" when a blocked robots token should fail the workflow. Set fail-on-restricted to "true" only with check-edge-responses: "true" when a synthetic 401, 403, 429, or other restricted response should fail the workflow.
Copy a scheduled or pull-request workflow and review when to observe policy drift versus fail a deployment.
The homepage check reads the returned HTTP response, raw HTML, and response headers. It does not run browser JavaScript, so a client-rendered canonical or meta directive is outside its evidence boundary. A missing canonical is reported as information rather than treated as failure.
The parser follows robots.txt group precedence, longest matching path, wildcard paths, end anchors, and Allow-on-tie behavior for the requested homepage path.
An ALLOWED result means no matching robots.txt restriction won for that token on /. It does not prove that a provider can pass a firewall, that a request is an authentic provider bot, or that any system will crawl, index, cite, rank, recommend, or send traffic.
An edge result is also bounded evidence. The requests come from the machine running this tool, not from provider IP ranges, and do not authenticate a provider bot or prove that Cloudflare caused the response. A restriction may come from Cloudflare, another CDN or WAF, the origin, rate limiting, or application logic.
Private and local-network targets are rejected. Homepage and robots.txt responses are limited to 1 MB and fetches time out after 10 seconds.
Use the free visual Cloudflare AI crawler checker or the broader eight-token policy checker. ActableSite also provides a public website readiness scan and a one-time repair report.
Crawler Watch monitors homepage response status, robots meta, X-Robots-Tag, canonical, the eight-token robots policy, synthetic crawler responses, sitemap.xml, and llms.txt every 15 minutes. A changed state must appear twice before it sends an email. The plan covers one public website for $9/month.
The service reads returned HTML and headers without running browser JavaScript. External crawler requests are synthetic. Neither check authenticates provider IP ranges or proves crawling, indexing, citation, ranking, or traffic.
npm test
node bin/actablesite-check.js actablesite.comMIT