Skip to content
AEO Canon · the reference for answer-engine optimization
AEOCanon

Lesson 2 of 6

Which Crawlers to Allow

Crawlers do different jobs — search, user-fetch, and training. This lesson decides which to allow for citations.

Learning objectives

  • ▸Distinguish search, user-fetch, and training crawlers.
  • ▸Allow the citation-driving bots.
  • ▸Decide training opt-in deliberately.

The lesson

Read the full lesson →Which AI Crawlers Should You Allow?Not all AI crawlers do the same job. Search and user-fetch bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Bingbot) drive citations and should be allowed; training crawlers (GPTBot, ClaudeBot) and Google-Extended are an opt-in choice. Here's what each one does.3 min read

Key takeaways

  • ▸Allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, and Bingbot.
  • ▸Training crawlers like GPTBot and ClaudeBot are a separate choice.
  • ▸Google-Extended gates Gemini's use of your content.

Knowledge check

Knowledge check

0 / 2

  1. 1. Which bots drive citations?

  2. 2. What is Google-Extended?