GPTBot and robots.txt: How to Handle AI Crawlers
Why robots.txt suddenly matters again
For twenty years robots.txt was plumbing. You set it once, blocked /admin/, and forgot it existed. Then AI companies started crawling the web at scale, and this dusty little text file became the main lever you have over whether your content trains models, appears in AI search answers, or neither.
The catch is that "AI bots" isn't one thing. Some crawlers collect training data. Some build search indexes that cite you. Some fetch pages live because a user asked about your site. Handling AI crawlers well comes down to one principle: allow the search and user-request agents if you want AI visibility, and decide separately — per bot, by name — whether the training crawlers get access, because those are independent permissions. Treating them all the same is the single most common mistake I see, and it's usually an expensive one.
Who's crawling you — GPTBot and friends
Here's the current cast, by user-agent token. Last updated: August 2026.
| Bot | Owner | Purpose | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training | Content stays out of future model training; ChatGPT search unaffected |
| OAI-SearchBot | OpenAI | Search | You fade out of ChatGPT's linked, cited answers |
| ChatGPT-User | OpenAI | User request | ChatGPT can't read your page even when a human explicitly asks it to |
| PerplexityBot | Perplexity | Search | You lose Perplexity citations |
| Google-Extended | Training (control token) | Content excluded from Gemini training and grounding; Google search and AI Overviews unaffected | |
| ClaudeBot | Anthropic | Training | Content stays out of Claude training |
| anthropic-ai | Anthropic | Training (legacy token) | Belt-and-braces alongside ClaudeBot; harmless to include |
| Bytespider | ByteDance | Training | Reportedly ignores robots.txt anyway; enforcement needs your CDN or firewall |
Three footnotes to that table. ChatGPT-User isn't a crawler at all — it fetches a page live when a user asks about it. Perplexity has been credibly accused of also fetching via undeclared agents, so robots.txt may not tell the whole story there. Google-Extended is a control token, not a bot you'll see in logs; regular Googlebot does the actual crawling either way. Anthropic also documents Claude-User (live fetches) and Claude-SearchBot (search), and CCBot (Common Crawl) deserves an honourable mention, since its dataset has fed many models' training runs over the years.
Example robots.txt setups
The setup most commercial sites actually want — stay visible in AI answers, opt out of model training:
# Opt out of AI training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Stay in AI search and user-requested fetches
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
The scorched-earth version, if you've decided AI gets nothing:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
And the third option is doing nothing, which allows everything. For a lot of small sites chasing visibility, that's genuinely fine.
Remember each User-agent group needs its own rules; there's no inheritance between named groups, and a bot that finds its own name listed will ignore your User-agent: * block entirely.
The AI crawler trade-off, honestly
Blocking training crawlers is a defensible stance. Publishers whose entire business is their content have real reasons not to donate it to model training. You lose little visibility by blocking GPTBot alone, since the search and user-fetch agents are separate.
Blocking the search and live-fetch bots is a different decision with different consequences. Do that and you're choosing invisibility in AI answers. No citations in ChatGPT search, no Perplexity sources, no live reads when a user pastes your URL. For most brands trying to get recommended, that's self-harm. I'd only advise it for sites whose content is the product and who'd rather be unread than uncompensated.
One more dose of honesty: robots.txt is a polite request, not a lock. Well-behaved bots honour it. Bytespider reportedly doesn't, and there will always be scrapers that never announce themselves. If you need enforcement, that happens at the CDN or firewall level, not in a text file.
Common robots.txt mistakes
Blocking CSS and JavaScript directories. An old habit from the 2000s that never died. Googlebot needs those assets to render your pages properly, and Google's index feeds AI Overviews, so this quietly damages your AI visibility through the side door. Most AI crawlers don't execute JavaScript anyway, which is its own argument for making sure your content exists in the raw HTML.
Blocking everything, then wondering why you're invisible. This one is epidemic. Someone enables a one-click "block AI bots" toggle at the CDN — Cloudflare has made this the default for new zones — and eighteen months later marketing is commissioning a study into why the brand never appears in ChatGPT. Check your CDN settings before you check anything else. The robots.txt file can say Allow all day while the firewall returns 403s.
Misunderstanding Google-Extended. Blocking it does not remove you from AI Overviews. AI Overviews are built from the ordinary search index that Googlebot crawls, so the only way out of them is out of Google search entirely, which is a much bigger decision than most people intend to make.
Blocking ChatGPT-User because it looked like scraper traffic. Those fetches are individual humans asking about your pages. That's the traffic you want.
A short real-world pattern
A B2B hardware company, the kind with a 20-year-old domain and good organic rankings, asks why AI assistants never cite them. Their robots.txt looks fine. Their server logs show every OAI-SearchBot request getting a 403 from a bot-management rule the security team enabled during a scraping scare a year earlier. Nobody had connected the two. One rule change and the crawlers were back within days. The moral: logs beat assumptions.
Before touching anything, it's worth seeing your site the way the bots see it, and you can do exactly that with the crawler simulator — it fetches your pages as each AI user-agent and shows you who's actually getting through. Crawler access is also the first gate in our scoring methodology, for exactly this reason.
This is one piece of the broader technical GEO checklist.