Tokens
Tokens are the smallest units of text that LLMs process, roughly equivalent to words or word fragments. Most LLMs process input and output as sequences of tokens rather than whole words.
Definition
One token typically represents 1-2 words in English, though the exact split depends on the language and the model's tokenizer. Special tokens mark punctuation, formatting, and structural elements. Understanding tokens matters because model pricing, context window limits, and performance metrics are all measured in tokens, not word count.
When you paste content into an AI search engine, it converts your text to tokens before processing. Longer documents consume more of the model's available context window, potentially forcing the engine to summarize or drop less important content, including citations to your brand.
Why it matters for AI visibility
Token limits directly affect whether your entire webpage is processed by AI search engines. Long-form content that exceeds a model's budget may be truncated, meaning your brand's important information gets cut off. Knowing token economics helps you understand why concise, well-organized content has better chances of full inclusion in AI answers.
Related terms
Context Window
Context Window is the maximum amount of text an LLM can process in a single request, measured in tokens. It determines how much of a document the model can read before having to truncate or summarize.
AITransformer Architecture
Transformer Architecture is the neural network design that powers modern LLMs. It uses attention mechanisms to process all words in a document simultaneously, enabling systems to understand relationships across entire passages.
AILarge Language Model (LLM)
Large Language Models are neural networks trained on massive text datasets to predict and generate human language. They form the foundation of modern AI search and answer engines.
AIEmbeddings
Embeddings are numerical representations of text, converting words, phrases, or documents into lists of numbers that capture their meaning. AI search engines use embeddings to find relevant sources for answers.