Robots.txt
Robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers and other bots which pages they can crawl and which to exclude. Using simple directives, it manages crawl budget allocation, prevents indexing of duplicate or low-value content, and protects sensitive areas, while helping publishers communicate with both search engine and AI crawlers.
Definition
The robots.txt file uses simple directives to allow or disallow specific crawlers from accessing certain pages or directories. Common uses include blocking duplicate content, excluding low-value pages, and protecting sensitive areas. The file does not prevent indexation of URLs if they are linked from other indexed pages; it only prevents crawlers from directly accessing those URLs. Crawlers should respect robots.txt, though some bots ignore it.
Proper robots.txt management involves identifying which pages should be crawled and which should be blocked, understanding crawl budget allocation, and testing rules for accuracy. A restrictive robots.txt that blocks important pages wastes crawl budget on excluded content. Conversely, allowing crawling of duplicate or thin content can reduce efficiency. Monitoring crawl statistics in Google Search Console helps optimize robots.txt effectiveness.
Why it matters for AI visibility
AI web crawlers like GPTBot and other LLM training crawlers respect or ignore robots.txt directives based on their implementation. If you block AI crawlers in robots.txt, your content is excluded from AI training data and from real-time retrieval by AI systems. Conversely, allowing AI crawlers to access your content increases the chance it is included in AI training and used for AI-generated answers. Strategic robots.txt management for AI crawlers is essential for AI visibility.
Related terms
Crawling and Indexing
Crawling is the process of search engine and AI bots discovering web pages by following hyperlinks and sitemaps, while indexing is the process of storing, parsing, and analyzing page content so it can be retrieved and ranked in search results or cited in AI responses. Not all crawled content is indexed; pages may be excluded due to directives, quality signals, or duplication.
GEOAI Web Crawlers
AI Web Crawlers are automated systems deployed by LLM providers and AI companies to index and retrieve web content for LLM training, inference retrieval, or answer synthesis processes.
SEOCrawl Budget
Crawl Budget is the number of URLs a search engine crawler can visit on a website within a given time period, allocated based on site authority and server capacity. Limited crawl budget requires strategic resource allocation to ensure crawlers prioritize important pages and avoid wasting resources on duplicate content, errors, or low-value pages that should not be indexed.
GEOGPTBot
GPTBot is OpenAI's web crawler that collects publicly available content used to train and improve OpenAI's models, identifiable by the GPTBot user agent in HTTP requests. OpenAI operates separate agents for other jobs: OAI-SearchBot indexes content for ChatGPT search, and ChatGPT-User fetches pages when a user asks for them.