AI Crawlers & robots.txt GPTBot, ClaudeBot & More
AI Crawlers & SEO How to Configure robots.txt for GPTBot, ClaudeBot & PerplexityBot
AI bots now account for over 20% of verified bot traffic on the web. The biggest shift: every major AI provider has split its crawlers into separate bots for training, search, and real-time user requests — and each one needs its own robots.txt rule. Here’s the complete reference.
A year ago, managing AI crawlers meant adding one or two lines to your robots.txt. In 2026, the landscape is fundamentally different. OpenAI runs three separate bots. Anthropic runs three. Google separates its AI training crawler from regular Googlebot. And the consequences of getting it wrong have gotten real — blocking the wrong bot can remove you from the fastest-growing discovery channel on the web, while allowing the wrong one can feed your entire content archive into model training without compensation.
This guide gives you the full crawler reference table, four ready-to-paste robots.txt configurations for common strategies, and a step-by-step process for deciding which bots to allow on your specific site.
Table of Contents
1. The AI Crawler Landscape in 2026
The scale of AI crawling has grown dramatically. According to Cloudflare Radar data from March 2025 through March 2026, GPTBot and ClaudeBot account for roughly 12% and 9.2% of global bot traffic, respectively — putting ClaudeBot on par with Bingbot. Together with Meta-ExternalAgent, Bytespider, and other AI crawlers, AI-related bot activity now represents a significant portion of all web traffic hitting your server.
Meanwhile, blocking rates remain relatively low on the broader web. About 10–14% of websites block at least one AI crawler, though among news publishers the rate is much higher — roughly 50% or more block GPTBot specifically. The dominant emerging strategy is surgical: block training bots while allowing search bots, rather than blanket-blocking everything.
2. Complete Crawler Reference Table
Every major AI provider now runs multiple bots with distinct purposes. Blocking one does not block the others — each requires its own robots.txt directive.
| Bot | Operator | Purpose | User-Agent | Respects robots.txt? |
|---|---|---|---|---|
| GPTBot | OpenAI | Model training data | GPTBot |
Yes |
| OAI-SearchBot | OpenAI | ChatGPT search indexing | OAI-SearchBot |
Yes |
| ChatGPT-User | OpenAI | Real-time page fetch (user request) | ChatGPT-User |
Partially* |
| ClaudeBot | Anthropic | Model training data | ClaudeBot |
Yes |
| Claude-SearchBot | Anthropic | Search indexing | Claude-SearchBot |
Yes |
| Claude-User | Anthropic | Real-time page fetch (user request) | Claude-User |
Yes |
| PerplexityBot | Perplexity | Search + training | PerplexityBot |
Documented, but history of non-compliance |
| Google-Extended | AI training (Gemini/Vertex) | Google-Extended |
Yes | |
| Applebot-Extended | Apple | Apple Intelligence training | Applebot-Extended |
Yes |
| Bytespider | ByteDance | Training + general crawling | Bytespider |
Known non-compliance |
| CCBot | Common Crawl | Open dataset (used by many AI models) | CCBot |
Yes |
| Meta-ExternalAgent | Meta | AI training (Llama models) | Meta-ExternalAgent |
Yes |
*ChatGPT-User note: OpenAI states that ChatGPT-User fetches pages at a user’s direct request and may not be governed by robots.txt the same way automated crawlers are. This is similar to a human using a browser — the fetch happens because a specific user asked for it.
The critical takeaway from this table is that blocking ClaudeBot does not block Claude-SearchBot or Claude-User. The same applies to OpenAI: blocking GPTBot does not affect OAI-SearchBot or ChatGPT-User. Each bot is independently controllable, which is exactly what enables the “block training, allow search” strategy that most sophisticated publishers are adopting.
3. The Training vs. Search Distinction
This is the most important concept in AI crawler management for 2026. Every major AI company now separates its bots by purpose:
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent) collect content to build or improve AI models. Your content becomes part of the model’s knowledge. You get no direct traffic in return — the data is used to train the model itself.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your pages so they can be cited in AI-generated answers. When a user asks ChatGPT or Claude a question and your page is relevant, these systems can cite your content and potentially send traffic back to your site.
User-action bots (ChatGPT-User, Claude-User) fetch pages in real time when a specific user asks the AI to read a URL. These are the closest equivalent to a human clicking a link.
The data underscores why this matters: 89.4% of AI crawler traffic serves training purposes, while only about 8% is search-related. The vast majority of AI bot requests hitting your server are extracting content for training, not indexing it for future citation. This asymmetry is why the “block training, allow search” approach has become the dominant strategy among publishers who want AI visibility without giving away training data.
4. Four robots.txt Strategies (Ready to Paste)
Choose the strategy that matches your goals. Generate a complete robots.txt file with the right directives using our Robots.txt Generator.
Strategy A: Allow everything (maximum AI visibility)
Best for: marketing sites, tool sites, content businesses that want maximum AI citation and traffic. This is the right approach if your content is your marketing — you want it cited everywhere.
robots.txt — Allow All AI Crawlers
# Allow all OpenAI crawlers User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Allow all Anthropic crawlers User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # Allow Perplexity User-agent: PerplexityBot Allow: / # Allow Google AI training User-agent: Google-Extended Allow: / # Allow Apple Intelligence User-agent: Applebot-Extended Allow: / # Block non-compliant bots User-agent: Bytespider Disallow: /
Strategy B: Block training, allow search (most popular in 2026)
Best for: publishers, content creators, and businesses that want their content cited in AI answers but don’t want it used for model training. This is the dominant strategy among sophisticated publishers — approximately 30% of top sites use some version of this approach.
robots.txt — Block Training, Allow Search
# Block OpenAI training, allow search + user User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Block Anthropic training, allow search + user User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # Allow Perplexity (search) User-agent: PerplexityBot Allow: / # Block Google AI training (does NOT affect Google Search) User-agent: Google-Extended Disallow: / # Block Common Crawl training data User-agent: CCBot Disallow: / # Block Meta training User-agent: Meta-ExternalAgent Disallow: / # Block non-compliant bots User-agent: Bytespider Disallow: /
Important: Blocking Google-Extended opts you out of Gemini training but does not affect your Google Search rankings. Google-Extended is entirely separate from Googlebot.
Strategy C: Block all AI crawlers
Best for: sites with proprietary content, paid/gated content, or businesses that have taken a firm position against AI training and don’t need AI search visibility.
robots.txt — Block All AI Crawlers
# Block all AI crawlers
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /
Strategy D: Selective access (allow specific sections only)
Best for: sites that want their blog or public pages indexed by AI search, but want to keep product pages, pricing, or proprietary sections out of training data.
robots.txt — Selective Section Access
# Allow AI search bots on public content only User-agent: OAI-SearchBot Allow: /blog/ Allow: /tools/ Disallow: / User-agent: Claude-SearchBot Allow: /blog/ Allow: /tools/ Disallow: / User-agent: PerplexityBot Allow: /blog/ Allow: /tools/ Disallow: / # Block all training crawlers entirely User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: /
Generate your robots.txt in seconds
ToolsVale’s Robots.txt Generator lets you configure rules for every major AI crawler and search bot, set your sitemap URL, and download the finished file — no manual syntax required.
5. How to Decide: The Decision Framework
Before writing any rules, answer four questions for each AI crawler:
1. Does this crawler’s platform send you traffic? Check your analytics for referral traffic from ChatGPT, Claude, Perplexity, or Google AI Overviews. If a platform is already driving visits, blocking its search crawler directly reduces your visibility there.
2. Is your content a competitive moat or a marketing surface? If your content is what you sell (paywalled articles, premium research, proprietary data), blocking training crawlers protects your product. If your content is how you market (blog posts, tool pages, guides), allowing crawlers increases your reach.
3. Are you comfortable with this content being in training data? Once content is ingested for model training, there’s no taking it back. If you’re unsure, the “block training, allow search” strategy gives you visibility without the training data commitment.
4. What does your legal team say? In some industries (legal, healthcare, financial services), compliance requirements may mandate restricting AI access. Under 10% of sites in these sectors currently publish any AI crawler policy.
For ToolsVale’s audience specifically: if you run a website with free tools, blog content, and public resources — Strategy A or Strategy B is almost always the right choice. Your tools and guides benefit from being cited in AI answers. There’s no moat to protect — the visibility is pure upside. Run an SEO audit to check your current crawl health before making changes.
6. What About llms.txt?
You may have heard about llms.txt — a proposed convention where websites publish a plain text file at their root directory to help AI systems understand which pages are most important. If you want to understand the format in detail, we’ve written a full explainer on what llms.txt is and how it works, and you can generate one for your own site using our LLMs.txt Generator.
Here’s the honest state of play in September 2026:
The practical picture: adoption is growing quickly but from a small base. Google’s Gary Illyes has confirmed Google doesn’t support llms.txt. AI crawlers overwhelmingly fetch HTML pages, not llms.txt files — monitoring of over 500 million AI bot events found only a few hundred requests targeting /llms.txt directly. The primary proven use case today is developer documentation (AI coding assistants like Cursor and Claude Code actively look for the file when integrating with libraries), not AI search visibility.
Our recommendation: if you can generate it in five minutes (and you can, using our LLMs.txt Generator), it costs nothing to have. But don’t skip robots.txt configuration — that’s where the actual access control happens, and it’s where the crawlers are actually looking.
7. How to Verify Your Configuration
After updating your robots.txt, verify that the rules work as intended.
Step 1: Check your robots.txt is accessible. Visit yourdomain.com/robots.txt in a browser and confirm your new rules appear correctly. Common gotchas: caching (your CDN may serve the old version), CMS overrides (some CMS platforms regenerate robots.txt), and file permissions.
Step 2: Test with Google Search Console. Use the robots.txt Tester in Google Search Console to confirm your rules are correctly blocking or allowing specific crawlers and URLs.
Step 3: Validate your sitemap. Confirm that your robots.txt includes a Sitemap: directive pointing to your XML sitemap, and that the sitemap itself is valid. Use the Sitemap Validator to check for errors, dead URLs, and accessibility issues across different crawler contexts.
Step 4: Run a site audit. Use the SEO Audit Checker to verify that your robots.txt changes haven’t accidentally blocked important pages from regular search crawlers. Also check for broken links that might prevent crawlers from reaching your content — even if your robots.txt is perfect, broken internal links can create the same visibility problem. According to broken link statistics, 23% of news websites contain at least one broken link that interrupts the crawl path.
Step 5: Monitor server logs. After a week, check your server access logs for AI crawler user-agent strings. Confirm that blocked bots are receiving 403 or not appearing, and that allowed bots are crawling successfully. If Bytespider is still hitting your server despite a robots.txt block, you’ll need server-level or WAF-level blocking — Bytespider has documented history of ignoring robots.txt directives.
8. Frequently Asked Questions
Should I block or allow GPTBot in robots.txt?
It depends on your goals. GPTBot is OpenAI’s training crawler — blocking it prevents your content from being used in model training. But blocking GPTBot does NOT remove you from ChatGPT search results — that’s controlled by OAI-SearchBot, a separate bot. Most publishers in 2026 block GPTBot (training) while allowing OAI-SearchBot (search) to maintain visibility without contributing training data.
What percentage of websites block AI crawlers?
Roughly 10–14% of websites across the general web block at least one AI crawler as of mid-2026. Among news publishers, the rate is much higher — around 50% or more block GPTBot. GPTBot is the most commonly blocked AI crawler (appearing in about 5.5% of all Disallow rules), followed by CCBot, ClaudeBot, and Google-Extended.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data to train OpenAI’s language models. OAI-SearchBot indexes pages for ChatGPT’s real-time search feature so they can appear in search-driven answers. ChatGPT-User fetches individual pages when a user specifically asks ChatGPT to read a URL. Each uses a separate user-agent string and can be controlled independently in robots.txt.
Does blocking AI crawlers hurt my Google Search SEO?
No. Blocking training crawlers like GPTBot, ClaudeBot, and Google-Extended has no effect on your Google Search rankings. These are entirely separate from Googlebot. However, blocking search-specific AI bots (OAI-SearchBot, Claude-SearchBot) means your content won’t appear when users search within those AI platforms — an increasingly significant traffic source.
What is llms.txt and should I use it?
llms.txt is a proposed convention where websites publish a file at their root directory to guide AI systems to important content. Adoption is around 10% of websites as of mid-2026, but Google has confirmed it does not support llms.txt, and AI crawlers rarely fetch it directly. The practical use case today is mainly developer documentation. If it takes five minutes to generate (using a tool like ToolsVale’s LLMs.txt Generator), it’s worth having. But robots.txt is where actual access control happens.
How do I handle Bytespider (ByteDance)?
Bytespider has documented history of ignoring robots.txt directives. Add a Disallow rule in robots.txt as a first line of defense, but for actual enforcement, you’ll need server-level blocking through your firewall, CDN, or WAF rules. Bytespider is blocked by roughly 22–35% of top websites — higher than most AI crawlers — reflecting both security and geopolitical concerns.
Summary
The AI crawler landscape in 2026 requires nuanced management, not blanket decisions. The key insight is that training and search are now separate crawlers at every major AI company — you can opt out of model training while remaining visible in AI-powered search. The most common strategy is to block training bots (GPTBot, ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent) while allowing search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-action bots (ChatGPT-User, Claude-User). Use our Robots.txt Generator to configure your rules without manual syntax, then validate with the Sitemap Validator to ensure everything is accessible as intended.