Why is robots.txt for AI crawlers a strategic decision, not a technical one?
Your robots.txt rules for AI crawlers now control three separate things at once: whether your content feeds model training, whether your pages appear in AI search results, and whether a generative engine — a system that answers with a generated response instead of a list of links (ChatGPT, Perplexity, Google AI Overviews, Copilot) — can fetch your page when a user asks for it. One wrong line can zero out your brand’s visibility across an entire engine, which makes this a business decision, not server-room housekeeping.
The scale is measurable. According to Cloudflare (2025), roughly 80% of AI bot crawling in the 12 months to July 2025 served model training, versus 18% for search — and training’s share grew from 72% (July 2024) to 79% (July 2025). The two most active AI crawlers are GPTBot and ClaudeBot: in July 2025 they accounted for 28.1% and 23.3% of all AI bot traffic on Cloudflare’s network, respectively (Cloudflare, 2025).
The wave of AI blocking started when OpenAI publicly documented GPTBot on August 7, 2023 (Glide, 2023). Since then, every major AI provider has published an official crawler catalog — and those catalogs, not user-agent lists copied from forums, should be the basis of your file. For the strategic context behind this shift, see our post on GEO vs SEO — what changes and our guide to GEO (generative engine optimization), the practice of creating and optimizing content so that it gets cited in AI-generated answers.
Which AI crawlers does OpenAI send, and what does each one do?
OpenAI splits robots.txt control across three independent tokens — GPTBot, OAI-SearchBot, and ChatGPT-User — so you can block model training while staying fully visible in ChatGPT search (OpenAI crawler documentation, 2025). This is the single most important fact in this topic, because most self-inflicted damage comes from treating OpenAI’s bots as one “ChatGPT bot”.
- GPTBot crawls content for training future models. Blocking it excludes your site from training material — not from search.
- OAI-SearchBot builds the ChatGPT search index. The documentation is explicit: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers” (OpenAI, 2025) — blocking it removes your site from ChatGPT search results.
- ChatGPT-User fetches a specific page when a user asks for it in a conversation. After the December 2025 documentation revision, OpenAI states: “Because these actions are initiated by a user, robots.txt rules may not apply” (OpenAI, 2025). Blocking these visits reliably requires a server-side layer, such as filtering against OpenAI’s published IP ranges.
- OAI-AdsBot — the fourth and newest bot, publicly documented since April 21, 2026 — validates the safety of ad landing pages in ChatGPT and only visits pages submitted as ads (OpenAI, 2026).
OpenAI publishes each bot’s IP ranges in separate JSON files (including openai.com/gptbot.json and chatgpt-user.json), which lets you verify visits by IP address instead of trusting the user-agent header alone (OpenAI, 2025).
How do ClaudeBot and Anthropic’s other crawlers work?
Anthropic uses the same three-way split as OpenAI: ClaudeBot collects content for model training, Claude-SearchBot indexes pages to improve Claude’s search result quality, and Claude-User fetches a page when a user asks Claude about it — and by Anthropic’s own statement, all three respect robots.txt, including the Crawl-delay directive (Anthropic support documentation, accessed July 2026).
The consequences of blocking are separable. Blocking ClaudeBot means “future materials should be excluded from our AI model training datasets”, while blocking Claude-User cuts off on-demand fetching and — in Anthropic’s own words — “may reduce your site’s visibility”. Blocking does not remove data collected earlier (Anthropic, accessed July 2026). Anthropic publishes its bots’ IP ranges at claude.com/crawling/bots.json.
Watch out for legacy tokens: “anthropic-ai” and “claude-web” do not appear in Anthropic’s current official documentation — the current agents are exclusively ClaudeBot, Claude-User, and Claude-SearchBot — yet 72% and 66% of top news sites still block them, respectively (Search Engine Land, 2026, citing BuzzStream data). Dead rules like these do no harm, but they clutter the file and create a false sense of control.
What is the difference between PerplexityBot and Perplexity-User?
PerplexityBot indexes pages to “surface and link websites in search results on Perplexity” and, per Perplexity’s declaration, “is not used to crawl content for AI foundation models” — it serves search visibility, not training (Perplexity crawler documentation, accessed July 2026). The second agent, Perplexity-User, fetches a page in response to a user’s question and — per the same documentation — “generally ignores robots.txt rules”. Perplexity publishes both bots’ IP ranges in perplexitybot.json and perplexity-user.json.
Declarations should be weighed against measurements, though. On August 4, 2025, Cloudflare documented that Perplexity was using undeclared stealth crawlers: they impersonated Chrome on macOS and rotated IP addresses and ASNs outside the officially published ranges, bypassing robots.txt blocks and WAF rules; Perplexity denied the findings (Cloudflare, 2025). The practical takeaway: treat robots.txt rules for Perplexity as a statement of intent, and enforce hard access control at the server.
What does Google-Extended actually control?
Google-Extended is not a crawler — it is a control token in robots.txt with no user agent of its own (“Google-Extended doesn’t have a separate HTTP request user agent string”); the actual fetching is done by Google’s existing agents, and the token only decides what the content may be used for (Google Search Central, 2026). Announced on September 28, 2023, it controls two uses: training Gemini models (Gemini Apps, Vertex AI) and grounding — feeding content from the Google Search index to the model at prompt time (Google, 2026).
The most common misunderstanding: blocking Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google Search Central, 2026), and it does not remove your pages from AI Overviews or AI Mode — those features run as part of regular Google Search and its index (Google Search Central, 2025). The only controls over AI Overviews are blocking Googlebot, a noindex tag, or the nosnippet / data-nosnippet / max-snippet directives — and every one of them also hits your classic search results.
What do you lose when you block each AI bot?
Each bot has a different function, so each block has a different price — the table below maps the bot catalog from official documentation (as of July 2026) to the consequence of blocking it in robots.txt.
| Bot (token) | Owner | Function | Consequence of blocking |
|---|---|---|---|
| GPTBot | OpenAI | crawls content for model training | content excluded from future model training; in practice also correlates with near-zero ChatGPT citations |
| OAI-SearchBot | OpenAI | ChatGPT search index | site disappears from ChatGPT search answers |
| ChatGPT-User | OpenAI | fetches a page on user request | no on-demand fetch; rules “may not apply” — reliable blocking only via IP/WAF |
| OAI-AdsBot | OpenAI | validates ad landing pages in ChatGPT | only affects pages submitted as ads |
| ClaudeBot | Anthropic | collects content for model training | future materials excluded from training datasets; previously collected data remains |
| Claude-SearchBot | Anthropic | indexing for Claude’s search results | reduced or no presence in Claude search results |
| Claude-User | Anthropic | page access when a user asks Claude | no on-demand fetch; per Anthropic, “may reduce your site’s visibility” |
| PerplexityBot | Perplexity | Perplexity search index (not training) | site disappears from Perplexity results and links |
| Perplexity-User | Perplexity | fetch on a user’s question | declaredly “generally ignores robots.txt” — blocking requires a server-side layer |
| Google-Extended | control token: Gemini training + grounding | content excluded from Gemini training and grounding; zero effect on Search, rankings, and AI Overviews |
Sources for functions and consequences: official documentation from OpenAI, Anthropic, Perplexity, and Google (accessed July 2026).
Which robots.txt rules match your strategy?
The right rules depend on one decision: whether visibility in AI answers matters more to you than control over training use — below are three ready-made rule sets. The cost of a wrong choice is quantifiable: in a cloro.dev study (2026) of 1,058 prominent domains, the median “citation propensity” (ChatGPT citations per one appearance in Google’s organic results) was 0.003 for domains blocking GPTBot versus 0.417 for non-blockers, and dropped to zero for domains blocking OAI-SearchBot (versus 0.394).
robots.txt for AI crawlers is not a technical file — it is a price list for visibility: every “Disallow” line has a countable cost in citations, so before you add one, check the official bot catalog to see exactly what you are switching off.
Strategy 1: maximum AI visibility (the default recommendation for brands that want to be cited). Crawlers are allowed by default, so not blocking is enough — but an explicit “Allow” documents the decision and protects against accidentally inheriting a “Disallow” from a general section:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /Strategy 2: visibility without training. You block the training bots while keeping the AI search indexes — possible because the controls are separate (OpenAI, 2025):
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /Strategy 3: block all AI bots. You add “Disallow: /” for every token in the table above — and accept that the file alone is not enough. RFC 9309, the formal IETF standard for robots.txt since September 2022, states plainly that its rules “are not a form of access authorization” (IETF RFC 9309, 2022). A hard block requires a server-side layer: a WAF plus verification of visits against published IP ranges. The market is already moving that way — since July 1, 2025, Cloudflare blocks AI crawlers by default for newly registered domains (Cloudflare, 2025).
If you are not sure which strategy fits your business model, start with measurement: an AI visibility audit shows which engines already cite your site and what each block would actually cost you. For the technical side of readiness — rendering, sitemaps, content accessibility for bots — follow our step-by-step guide on how to prepare your website for generative engines.
Which mistakes most often break robots.txt for AI crawlers?
The most expensive mistakes come from treating one provider’s bots as a single bot and from copying other sites’ files without checking the documentation. Five recurring problems:
- Blocking OAI-SearchBot “along with” GPTBot. These are two different functions: blocking GPTBot switches off training, blocking OAI-SearchBot removes your site from ChatGPT search answers (OpenAI, 2025). In the cloro.dev study (2026), 13.9% of domains blocked GPTBot but only 3.4% blocked OAI-SearchBot — most of the market already understands the difference.
- Copying outdated user-agent lists. The tokens “anthropic-ai” and “claude-web” do not exist in Anthropic’s current documentation, yet 72% and 66% of top news sites still block them (Search Engine Land, 2026).
- Treating Google-Extended as an AI Overviews kill switch. Blocking it does not remove your pages from AI Overviews or AI Mode — it only controls Gemini training and grounding (Google Search Central, 2025).
- Believing robots.txt is a technical block. For ChatGPT-User the rules “may not apply” (OpenAI, 2025), Perplexity-User “generally ignores robots.txt” (Perplexity, accessed July 2026), and the stealth crawling documented by Cloudflare (2025) shows declarations get broken. Enforcement means a WAF and IP verification.
- Replacing robots.txt with llms.txt. llms.txt — a proposed standard for a site content map for AI systems — has no confirmed adoption: per Ahrefs (2026), 97% of roughly 38,000 domains with a valid llms.txt file received zero requests for it in May 2026, and no major provider (OpenAI, Anthropic, Google) supports it. Access control is the job of robots.txt and your server, not llms.txt.
FAQ: robots.txt and AI crawlers
Will blocking GPTBot remove my site from ChatGPT?
Not directly — GPTBot handles model training, while ChatGPT search runs on OAI-SearchBot (OpenAI, 2025). In practice, blocking GPTBot correlates with near-zero citability: the median number of ChatGPT citations per Google organic appearance is 0.003 for blockers versus 0.417 for non-blockers (cloro.dev, 2026).
Will blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google Search Central, 2026). The token only controls whether your content is used for Gemini model training and grounding; it affects neither rankings nor AI Overviews.
Do AI crawlers always respect robots.txt?
No. RFC 9309 defines robots.txt as a voluntary convention — its rules “are not a form of access authorization” (IETF, 2022). User-triggered fetchers (ChatGPT-User, Perplexity-User) may skip the rules per their own documentation, and in 2025 Cloudflare documented Perplexity stealth crawling that bypassed blocks entirely (Cloudflare, 2025).
Does llms.txt replace robots.txt?
No. They are different files with different roles: robots.txt controls bot access, while llms.txt is a proposed content map for AI systems with no confirmed adoption. Per Ahrefs (2026), 97% of domains with a valid llms.txt received zero requests for the file in May 2026; OpenAI, Anthropic, and Google do not support it.
