AI Crawler User Agents — The 2026 Cheat Sheet
One scannable reference for the major AI bots — GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), Google-Extended, Applebot-Extended, PerplexityBot, CCBot, and Bytespider — with each bot's owner, purpose, and whether it actually obeys robots.txt.
The major AI bots split into three jobs — training, search-and-cite, and on-demand fetch — and each needs its own robots.txt rule. OpenAI alone runs three separate crawlers; Anthropic runs three more. Below is one scannable reference so you can decide who to allow.
Quick answer
There is no single AI bot. GPTBot and Google-Extended train models; OAI-SearchBot and Claude-SearchBot retrieve and cite; ChatGPT-User and Claude-User fetch on demand. Each is a distinct user-agent — address them individually, and know that a few (Bytespider, some stealth crawlers) ignore robots.txt.
Which bots power training vs citation?
They are different bots with different goals. Training crawlers like GPTBot, Google-Extended, and Applebot-Extended collect content to teach future models — blocking them keeps you out of the training set but not out of live answers. Search-and-cite bots like OAI-SearchBot and Claude-SearchBot fetch pages to build the answers users see right now, and they are the ones that send citations. On-demand bots (ChatGPT-User, Claude-User) fetch a single page the moment a user references it.
Why does each bot need its own rule?
Because a rule for one says nothing about the others. Allowing GPTBot does not allow OAI-SearchBot, and blocking ClaudeBot does not block Claude-User. They share a company but not a user-agent string, so you must name each one explicitly in robots.txt.
Do these crawlers actually respect robots.txt?
Mostly, but not all. The major OpenAI, Anthropic, and Google bots honor disallow rules. Bytespider (ByteDance/TikTok) and certain stealth Perplexity crawlers have been documented fetching content despite robots.txt, so treat robots.txt as a signal, not a wall.
| Bot | Company | Purpose | Obeys robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Yes |
| OAI-SearchBot | OpenAI | Search retrieval + citation | Yes |
| ChatGPT-User | OpenAI | On-demand fetch | Yes |
| ClaudeBot | Anthropic | Model training | Yes |
| Claude-SearchBot | Anthropic | Search retrieval + citation | Yes |
| Claude-User | Anthropic | On-demand fetch | Yes |
| Google-Extended | Gemini training only (no Search impact) | Yes | |
| Applebot-Extended | Apple | Apple Intelligence training | Yes |
| PerplexityBot | Perplexity | Search retrieval + citation | Mostly — stealth crawlers documented ignoring it |
| CCBot | Common Crawl | Open dataset (feeds many models) | Yes |
| Bytespider | ByteDance / TikTok | Training / scraping | Documented ignoring it |
Use this to decide who reads you. Next, see which AI crawlers to allow and the full GPTBot glossary entry.
Related questions
Which AI crawlers should I allow?
Allow the search-and-cite bots at minimum — they send citations. Training bots are a separate choice.
Read the full answer →What is GPTBot?
OpenAI's training crawler — separate from OAI-SearchBot, which handles ChatGPT Search retrieval.
Read the full answer →How do meta robots tags affect AI crawlers?
Some AI bots honor noindex and specific meta directives — but not all, and not identically.
Read the full answer →Frequently asked questions
- How many AI crawlers does OpenAI run?
- Three. GPTBot collects training data, OAI-SearchBot powers retrieval and citation for ChatGPT Search, and ChatGPT-User fetches a page on demand when a user asks about it. Each is a separate user-agent, so a robots.txt rule for one does not cover the others.
- Does Google-Extended affect my Google Search ranking?
- No. Google-Extended controls only whether your content trains Gemini and related AI products. Blocking it has no effect on Googlebot, indexing, or your position in classic Google Search results.
- Do all AI crawlers obey robots.txt?
- No. Most major bots from OpenAI, Anthropic, and Google honor robots.txt, but Bytespider and some stealth crawlers have been documented fetching content despite disallow rules. Robots.txt is a request, not an enforced firewall.