GPTBot, OAI-SearchBot and the rest: which AI crawlers affect whether you appear in AI search
Blocking GPTBot does not remove you from ChatGPT search. A plain-English guide to which AI crawlers affect AI search, and which only train models.
Ben Rosenbaum. Technical. Published .
A lot of advice lumps all “AI crawlers” together. In fact each major AI company runs several, for different purposes, and blocking the wrong one, or worrying about the wrong one, is common.
Here is what each company says about its own crawlers.
Three kinds of AI crawler
- Search crawlers index pages so the platform can find and cite them in search-style answers. These are the ones that matter for appearing in AI search.
- Training crawlers collect content that may be used to train future models. Blocking them is a choice about your content, not about your visibility in AI search.
- User fetchers visit a page because a person asked the assistant something that needed it. They are not crawling the web on their own.
The crawlers, company by company
| Company | Crawler | What it is for | Affects AI search? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfacing websites in ChatGPT search | Yes |
| OpenAI | GPTBot | Content that may be used to train models | No, independent of search |
| OpenAI | ChatGPT-User | Visiting pages for user requests | Not used for search inclusion |
| Anthropic | Claude-SearchBot | Improving search results in Claude | Yes |
| Anthropic | ClaudeBot | Content that may contribute to training | No |
| Anthropic | Claude-User | Fetching pages when a person asks Claude | Can affect user-directed search |
| Perplexity | PerplexityBot | Surfacing and linking sites in Perplexity search | Yes |
| Perplexity | Perplexity-User | Visiting pages to answer a user’s question | Not a crawler |
| Googlebot | Google Search, which includes AI Overviews and AI Mode | Yes | |
| Google-Extended (a robots.txt token) | Gemini model training and grounding in Gemini Apps and Vertex AI | Not Google Search |
OpenAI
OpenAI’s crawler documentation says OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features”. GPTBot is used for content that may be used to train its models, and OpenAI says each setting “is independent of the others”: you can allow OAI-SearchBot and disallow GPTBot. ChatGPT-User handles actions a user starts, and OpenAI says it is not used to decide search inclusion and that robots.txt rules may not apply to it.
In practice: blocking GPTBot does not remove you from ChatGPT search. Blocking OAI-SearchBot does.
Anthropic
Anthropic’s documentation describes ClaudeBot as collecting content that could contribute to training, Claude-SearchBot as indexing content to improve search results, and Claude-User as fetching pages when a person asks Claude a question. Anthropic says blocking Claude-SearchBot or Claude-User may reduce your visibility in search results.
Perplexity
Perplexity’s documentation says PerplexityBot is designed to surface and link websites in Perplexity search and is not used to crawl content for training foundation models. Perplexity-User visits pages to answer a user’s question and, because a user requested the fetch, generally ignores robots.txt.
Google AI Overviews and AI Mode are part of Google Search, so they rely on Googlebot like the rest of Search. Google’s crawler documentation describes Google-Extended as a robots.txt token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and states that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”.
Because the token also covers grounding in Gemini Apps, it is worth reading Google’s description before you decide either way.
What to check on your own site
- Read your robots.txt. Make sure OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot are not disallowed, unless you intend them to be.
- Check what your hosting or security layer does. Bot protection at your CDN or firewall can block crawlers even when robots.txt allows them.
- Check your pages load without JavaScript. If your main content only appears after scripts run, some crawlers may see an empty page.
- Decide on training crawlers separately. Allowing or blocking GPTBot, ClaudeBot or Google-Extended is a legitimate choice either way. It is not the same decision as appearing in AI search.
Ansoar’s Crawlability check runs these checks across your whole site, reports each page as Clear, Needs attention or Couldn’t be checked, and never reports a blocked training crawler as a search visibility failure. For what it is worth, ansoar.io’s own robots.txt allows every crawler listed above.