Reference Guide
The Complete List of AI Crawlers & Bots (2026)
Every AI crawler currently active on the web - their User-agent strings, what they do, which company runs them, and whether you should allow or block them in your robots.txt.
AI Answer Engines
AI Answer Engines
These crawlers fetch pages in real time to answer user queries. Blocking them removes you from AI-generated answers.
| Bot Name | Company | Purpose | Recommendation |
|---|---|---|---|
|
GPTBot
GPTBot
|
OpenAI | Powers ChatGPT browsing and search citations | ✓ Allow |
|
OAI-SearchBot
OAI-SearchBot
|
OpenAI | OpenAI search index crawler | ✓ Allow |
|
ChatGPT-User
ChatGPT-User
|
OpenAI | ChatGPT browsing plugin real-time requests | ✓ Allow |
|
ClaudeBot
ClaudeBot
|
Anthropic | Powers Claude AI answers and citations | ✓ Allow |
|
Claude-Web
Claude-Web
|
Anthropic | Older Anthropic crawler (now largely ClaudeBot) | ✓ Allow |
|
PerplexityBot
PerplexityBot
|
Perplexity | Powers Perplexity AI search answers and citations | ✓ Allow |
|
GoogleOther
GoogleOther
|
Google AI Overviews, Bard, and experimental products | ✓ Allow | |
|
Applebot-Extended
Applebot-Extended
|
Apple | Apple Intelligence and Siri knowledge base | ✓ Allow |
|
cohere-ai
cohere-ai
|
Cohere | Cohere AI language model data fetching | ✓ Allow |
AI Training Crawlers
AI Training Crawlers
These crawlers collect data to train language models. They may use your content without attribution. Consider blocking if you do not want your content used for model training.
| Bot Name | Company | Purpose | Recommendation |
|---|---|---|---|
|
CCBot
CCBot
|
Common Crawl | Supplies training data to GPT, LLaMA, and many other LLMs | ✗ Consider Blocking |
|
Bytespider
Bytespider
|
ByteDance | TikTok parent company AI training data collection | ✗ Consider Blocking |
|
DiffBot
DiffBot
|
Diffbot | Machine learning data extraction for Knowledge Graph | ✗ Consider Blocking |
|
Omgili / Webz.io
omgili
|
Webz.io | Content harvesting for AI training datasets | ✗ Consider Blocking |
|
Timpibot
Timpibot
|
Timpi | Decentralised AI search index crawler | ✓ Allow |
|
img2dataset
img2dataset
|
Community | Open-source image dataset scraper used by AI researchers | ✗ Consider Blocking |
Search Engine Crawlers
Search Engine Crawlers
Standard search engine bots. Block these only for specific pages - blocking globally removes you from search results.
| Bot Name | Company | Purpose | Recommendation |
|---|---|---|---|
|
Googlebot
Googlebot
|
Google Search indexing | ✓ Allow | |
|
Googlebot-Image
Googlebot-Image
|
Google Image Search indexing | ✓ Allow | |
|
Bingbot
Bingbot
|
Microsoft | Bing and Microsoft Copilot search index | ✓ Allow |
|
DuckDuckBot
DuckDuckBot
|
DuckDuckGo | DuckDuckGo search indexing | ✓ Allow |
|
Slurp
Slurp
|
Yahoo | Yahoo Search indexing | ✓ Allow |
|
YandexBot
YandexBot
|
Yandex | Yandex Search indexing | ✓ Allow |
|
Amazonbot
Amazonbot
|
Amazon | Alexa and Amazon AI knowledge base | ✓ Allow |
SEO & Analytics Crawlers
SEO & Analytics Crawlers
Used by SEO tools for auditing and research. Safe to allow - blocking these only prevents SEO tools from crawling your site.
| Bot Name | Company | Purpose | Recommendation |
|---|---|---|---|
|
AhrefsBot
AhrefsBot
|
Ahrefs | Ahrefs SEO tool backlink and site auditing | ✓ Allow |
|
SemrushBot
SemrushBot
|
Semrush | Semrush SEO tool site auditing | ✓ Allow |
|
MJ12bot
MJ12bot
|
Majestic | Majestic SEO link intelligence crawler | ✓ Allow |
|
DotBot
DotBot
|
Moz | Moz SEO tool link and site data | ✓ Allow |
Quick Reference
Recommended robots.txt for Most Sites
This configuration allows all AI answer engines and search crawlers while blocking the most common training data scrapers. Copy and save as robots.txt in your site root.
# Block AI training data scrapers User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: DiffBot Disallow: / User-agent: omgili Disallow: / # Allow all search engines and AI answer crawlers User-agent: * Allow: / # Block admin and internal paths from all bots User-agent: * Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /cart/ Disallow: /checkout/ Sitemap: https://yourdomain.com/sitemap.xml
FAQs
AI Crawlers: Common Questions
An AI crawler is an automated bot that fetches web pages for use by AI systems. Some fetch pages to answer user queries in real time (answer crawlers like GPTBot and ClaudeBot). Others harvest content to train language models (training crawlers like CCBot and Bytespider).
No. Blocking AI answer crawlers like GPTBot and ClaudeBot removes your site from AI-generated answers, which is increasingly where users go first. The more targeted approach is to allow answer crawlers and block training crawlers (CCBot, Bytespider, DiffBot).
Reputable crawlers - OpenAI, Anthropic, Google, Perplexity, and major search engines - honour robots.txt. Malicious scrapers do not. For legitimate AI companies, robots.txt is respected.
GPTBot is OpenAI's general crawling bot that indexes pages for ChatGPT's knowledge base. ChatGPT-User is the user-agent used when a ChatGPT user triggers a real-time browse request. They serve different functions but both can be controlled via robots.txt.
Check your server access logs for User-agent strings. In Apache or Nginx logs, filter for known bot User-agents. Google Search Console also shows Googlebot crawl activity. For a full picture, a server-side analytics tool is the most reliable approach.
CCBot is operated by Common Crawl, a non-profit that makes large web crawl datasets available publicly. These datasets are used to train many open and commercial LLMs including early versions of GPT. If you do not want your content used as training data without attribution or compensation, blocking CCBot is the most widely recommended step.
Configure Your AI Crawler Permissions
Use our free tools to generate your robots.txt and llms.txt files with the right settings for every bot on this list.
Build Your robots.txt Free →