LLMs.txt
LLMs.txt is a proposed convention: a plain markdown file served at the root of a website that gives large language models a curated summary of the site's most important content. It helps AI systems find, understand, and use the right pages at answer time instead of parsing cluttered HTML.
Definition
The file lives at /llms.txt and is written in markdown: a short description of the site followed by organized links to key pages, often with one-line summaries. Where robots.txt tells crawlers what they may not access, LLMs.txt does the opposite job: it hands AI systems a prioritized reading list. The format was proposed in 2024 by Jeremy Howard of Answer.AI and has since been adopted by many documentation-heavy sites.
Adoption is voluntary on both sides. Publishing the file costs little, but no major AI provider has committed to reading it systematically, so treat it as a low-cost hedge rather than a guaranteed channel. It pairs naturally with clean information architecture: if a site cannot produce a sensible LLMs.txt, its content hierarchy probably confuses machine readers too.
Why it matters for AI visibility
LLMs.txt is part of a broader shift toward publishing for machine readers as well as human visitors. Sites that maintain one signal which pages carry their canonical answers, which is exactly what an AI assistant needs when deciding what to cite. Even while provider support remains inconsistent, the exercise forces clarity about which pages should represent the brand in AI answers.
Related terms
LLMs-full.txt
LLMs-full.txt is a proposed extension to LLMs.txt that includes the full text of content publishers want included in LLM training datasets, allowing direct provision of training data to AI systems.
GEOAGENTS.md
AGENTS.md is an open convention: a markdown file placed in a code repository that gives AI coding agents project-specific instructions, such as how to build, test, and navigate the codebase and which conventions to follow. It works like a README written for machine collaborators instead of human ones.
GEOAI Web Crawlers
AI Web Crawlers are automated systems deployed by LLM providers and AI companies to index and retrieve web content for LLM training, inference retrieval, or answer synthesis processes.
SEORobots.txt
Robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers and other bots which pages they can crawl and which to exclude. Using simple directives, it manages crawl budget allocation, prevents indexing of duplicate or low-value content, and protects sensitive areas, while helping publishers communicate with both search engine and AI crawlers.