Crawlability
Crawlability is whether automated crawlers can reach and read your pages, the absolute prerequisite for being indexed and cited, since content a crawler can't fetch is invisible to AI.
Crawlability is whether a bot can get to your content at all. It depends on crawlers being able to find your pages (via links and sitemaps), being permitted to fetch them (robots.txt), and receiving real content in the response rather than errors or empty shells. If a crawler can't reach a page, nothing else about it matters.
It's the front of the access pillar — the price of admission to AI visibility. The common failures are blunt: disallowing crawlers in robots.txt, hiding content behind JavaScript that bots don't execute, broken links, or server errors. Fixing crawlability is usually the highest-leverage AEO work precisely because it's binary — you're either readable or you're invisible. Server-side rendering is the most reliable way to guarantee your content is there to be crawled.
Example. A beautiful, authoritative page that returns content only after JavaScript runs may be effectively uncrawlable to an AI bot — present to humans, absent to the engine. Serving it as plain HTML makes it crawlable.
Relevant pillar
Related terms
- robots.txtrobots.txt is a plain text file at the root of your domain that tells crawlers which user-agents may access which parts of your site, and is how you allow or block AI crawlers.
- IndexabilityIndexability is whether a crawled page is eligible to be stored in a search index, since a page can be crawlable yet still excluded from the index that retrieval and AI answers draw from.
- Server-Side Rendering (SSR)Server-side rendering is when a web server generates a page's full HTML for each request and sends it ready-to-read, so content is present immediately for both browsers and AI crawlers.