Skip to content
AEO Canon · the reference for answer-engine optimization
AEOCanon

Lesson 3 of 6

Test What Crawlers See

Most AI crawlers do not run JavaScript, so test the raw HTML, not the rendered DOM. This lesson runs the JS-disabled and curl tests.

Learning objectives

  • ▸Run the JavaScript-disabled browser test.
  • ▸Fetch your page as a crawler with curl.
  • ▸Tell raw HTML from the rendered DOM.

The lesson

Read the full lesson →How to Check if AI Crawlers Can Read Your SiteMost AI crawlers don't run JavaScript, so the test is simple — view what's in the raw HTML, not the rendered DOM. This guide walks through the JS-disabled browser test and curl checks that show you exactly what GPTBot and PerplexityBot see.3 min read

Key takeaways

  • ▸Test the raw HTML via View Source or curl, not Inspect.
  • ▸If content vanishes with JavaScript off, crawlers miss it too.
  • ▸Fetching with a crawler user-agent shows the exact bytes it receives.

Knowledge check

Knowledge check

0 / 2

  1. 1. What should you inspect for crawlability?

  2. 2. Content shows in the browser but not in curl — why?