Your robots.txt quietly became a policy document: it now decides whether AI systems may learn from your content, search it, and fetch it for users mid-conversation. Three different questions, three different bots, and most sites have answered none of them on purpose.
Here is the decision framework, then the syntax, then the mistakes that turn a policy into an accident.
Decide by purpose, not by company
The crawler cast splits into three jobs, and each deserves its own stance.
The training trade is the strategic one, in memory-currency terms: blocking training bots protects your words from becoming model weights, and steps you back from the door where unprompted recommendations come from. Publishers with licensing leverage weigh that differently than businesses that want to be recommended.
Three sane configurations
Open (the visibility-first default): no AI-specific rules at all. Everything crawls, you are present at both doors, and your content contributes to training. Most businesses seeking AI visibility are here, deliberately or not.
Search yes, training no (the middle path):
| robots.txt lines | Effect |
|---|---|
User-agent: GPTBot + Disallow: / | No OpenAI training |
User-agent: Google-Extended + Disallow: / | No Google AI training use |
User-agent: ClaudeBot + Disallow: / | No Anthropic training |
User-agent: OAI-SearchBot + Allow: / | Stay in ChatGPT search |
User-agent: PerplexityBot + Allow: / | Stay in Perplexity |
Closed: every named AI agent disallowed. Coherent for paywalled and licensing-minded publishers, and it costs exactly what it saves: presence in the systems a growing share of buyers ask for advice.
The reality checks
robots.txt is a request, honored voluntarily. The major labs publicly commit to respecting their named crawlers' rules, and the disreputable scrapers were never going to read the file, so treat it as policy for the actors that matter, not as security.
It also only works forward: blocking GPTBot today does not un-train anything already learned. And it is per-crawler, so a blanket User-agent: * disallow written for some other purpose silently answers all three AI questions with "no", which is the most common accidental policy on the web.
On llms.txt, the proposed companion standard for AI-facing site summaries: adoption is real but limited and the debate unsettled, so file it under optional experiments, after the robots decisions, not instead of them.
Deploy, then verify
After any change: fetch /robots.txt yourself and read the rules as a crawler would, top to bottom, remembering that agent-specific blocks override the general ones. Then confirm behavior in the logs over the following weeks, the same log check as ever: the blocked agents should taper off, the allowed ones should keep visiting.
And check the whole stack agrees: a robots file that welcomes OAI-SearchBot while the CDN challenges it has a policy in writing and a different one in production.
The one-line takeaway: write your AI policy per purpose: training bots are the real trade-off, search bots and live fetchers are usually allows for anyone who wants to be found. Say it explicitly in robots.txt, make the firewall agree, and verify in the logs.