Token
A token is the basic unit of text an AI model processes — typically a word or word-piece — and is how model limits, costs, and context windows are measured.
A token is a chunk of text the model treats as one unit. Models don't read characters or whole words exactly; they read tokens — often a short word or a piece of a longer one. Roughly, a token averages about four characters of English, so 100 tokens is around 75 words. Tokens are the currency for everything a model does.
Tokens matter to AEO indirectly but really: context windows are sized in tokens, API costs are billed per token, and retrieval systems budget tokens when deciding how much of your content to include. Writing extractable, information-dense passages means more answer per token, which makes your content a more efficient — and more likely — inclusion in a limited window.
Example. The sentence "AEO is answer engine optimization" is about six tokens. When an engine assembles an answer, every retrieved passage spends tokens from a fixed budget, so concise, high-value writing competes better for the space.
Relevant pillar
Related terms
- TokenizationTokenization is the process of splitting text into tokens before an AI model can process it, converting human-readable language into the units the model actually operates on.
- Context WindowThe context window is the maximum amount of text an AI model can consider at once, including the question, any retrieved sources, and its own answer, measured in tokens.
- Large Language Model (LLM)A large language model is an AI system trained on vast amounts of text to predict and generate language, and is the engine that writes the answers in AI search.