Repository navigation
feat(zone): let the site's robots.txt decide on AI training crawlers - #82
Merged
Merged
Conversation
The zone's AI training crawler policy (ai_training = "disallow") answered GPTBot, ClaudeBot and PerplexityBot with 403 at the edge, while the origin answers them with 200 and the site's robots.txt allows every crawler. "Block AI bots" was already off. Set ai_training to disabled so each site states its crawler policy in its own robots.txt. Also retain the state from the zone-settings apply (run 43702045747, serial 7), which authenticates with the independent identity and matches the fresh runner's recovered copy. Part of #21. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devantler
commented
Nov 15, 2026
devantler
left a comment
Contributor
Author
There was a problem hiding this comment.
Local review at edd6ca2. CodeRabbit used its hourly allowance on #81 at 09:54 UTC. Codex and Bugbot are at their usage limits.
zone.tf: addsai_training = "disabled"and updates the comment. The live valuedisallowblocked GPTBot, ClaudeBot and PerplexityBot at the edge (403), while the origin answers them with 200. The search and assistant policies stay unset, so they keep their live values. The guard admits this as an update of the imported bot management resource, and nothing else changes.recovery/backend-source/: this is theafterpackage of run 43702045747. It authenticates with the independent identity and has the same state digest (b5b1ffd3) as the fresh runner'srecoveredcopy. The lineage is unchanged, the serial goes 6 → 7, and the configuration is unchanged.- Checks:
scripts/test.shpasses (22). The plan's changed-attribute list is the final check before applying.
Clean at edd6ca2.
🤖 Generated with Claude Code
devantler
marked this pull request as ready for review
November 15, 2026 10:07
This was referenced Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cloudflare still answers AI training crawlers (GPTBot, ClaudeBot, PerplexityBot) with 403 on devantler.tech, although the site's own robots.txt allows every crawler and the platform answers them with 200. The cause is the zone's AI training crawler policy, not "Block AI bots", which #81 found already off.
What
disabled, so each site's robots.txt decides. The other AI crawler settings (search and assistant) are unchanged: today they block nothing on this site.Result of #81, already applied: all 69 sitemap pages now match the origin byte for byte, with no email obfuscation, bot script or hidden links.
Rollback: set
ai_trainingback todisallow, then plan and apply.Part of #21
🤖 Generated with Claude Code