Skip to content

feat(zone): let the site's robots.txt decide on AI training crawlers - #82

Merged
devantler merged 1 commit into
mainfrom
claude/zone-ai-training
Nov 15, 2026
Merged

devantler merged 1 commit into
mainfrom
claude/zone-ai-training

Conversation

@devantler

Copy link
Copy Markdown
Contributor

Why

Cloudflare still answers AI training crawlers (GPTBot, ClaudeBot, PerplexityBot) with 403 on devantler.tech, although the site's own robots.txt allows every crawler and the platform answers them with 200. The cause is the zone's AI training crawler policy, not "Block AI bots", which #81 found already off.

What

Result of #81, already applied: all 69 sitemap pages now match the origin byte for byte, with no email obfuscation, bot script or hidden links.

Rollback: set ai_training back to disallow, then plan and apply.

Part of #21

🤖 Generated with Claude Code

The zone's AI training crawler policy (ai_training = "disallow") answered
GPTBot, ClaudeBot and PerplexityBot with 403 at the edge, while the origin
answers them with 200 and the site's robots.txt allows every crawler.
"Block AI bots" was already off. Set ai_training to disabled so each site
states its crawler policy in its own robots.txt.

Also retain the state from the zone-settings apply (run 43702045747,
serial 7), which authenticates with the independent identity and matches
the fresh runner's recovered copy.

Part of #21.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@devantler devantler left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local review at edd6ca2. CodeRabbit used its hourly allowance on #81 at 09:54 UTC. Codex and Bugbot are at their usage limits.

  • zone.tf: adds ai_training = "disabled" and updates the comment. The live value disallow blocked GPTBot, ClaudeBot and PerplexityBot at the edge (403), while the origin answers them with 200. The search and assistant policies stay unset, so they keep their live values. The guard admits this as an update of the imported bot management resource, and nothing else changes.
  • recovery/backend-source/: this is the after package of run 43702045747. It authenticates with the independent identity and has the same state digest (b5b1ffd3) as the fresh runner's recovered copy. The lineage is unchanged, the serial goes 6 → 7, and the configuration is unchanged.
  • Checks: scripts/test.sh passes (22). The plan's changed-attribute list is the final check before applying.

Clean at edd6ca2.

🤖 Generated with Claude Code

@devantler
devantler marked this pull request as ready for review November 15, 2026 10:07
@devantler
devantler merged commit 23a0c22 into main Nov 15, 2026
16 checks passed
@devantler
devantler deleted the claude/zone-ai-training branch November 15, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant