Skip to content
AEO Canon · the reference for answer-engine optimization
AEOCanon

Lesson 1 of 6

Allow AI Crawlers in robots.txt

AI engines can only cite pages their crawlers may fetch. This lesson deploys a verified, permissive robots.txt.

Learning objectives

  • ▸Write an explicit allow block for AI crawlers.
  • ▸Avoid the leftover Disallow that blocks everyone.
  • ▸Verify robots.txt is live and correct.

The lesson

Read the full lesson →How to Allow AI Crawlers in robots.txtAI engines can only cite pages their crawlers are allowed to fetch. This guide gives you a verified, copy-paste robots.txt block that explicitly allows GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and Bingbot — plus the one mistake that silently blocks them all.3 min read

Key takeaways

  • ▸The most specific user-agent group wins.
  • ▸A stray site-wide Disallow silently blocks every crawler.
  • ▸robots.txt must return 200 as text/plain at the root.

Knowledge check

Knowledge check

0 / 2

  1. 1. Which robots.txt group does a crawler obey?

  2. 2. What is the most common blocking mistake?