GPTBot
GPTBot is OpenAI's web crawler that collects publicly available content used to train and improve OpenAI's models, identifiable by the GPTBot user agent in HTTP requests. OpenAI operates separate agents for other jobs: OAI-SearchBot indexes content for ChatGPT search, and ChatGPT-User fetches pages when a user asks for them.
Definition
GPTBot is the primary crawler operated by OpenAI for content discovery and indexing. The crawler accesses websites, retrieves content, and sends data to OpenAI systems for processing. Publishers can identify GPTBot visits by examining server logs for the GPTBot user agent. The crawler respects robots.txt directives, allowing publishers to block GPTBot selectively while allowing other crawlers to access content.
GPTBot's activity has become a major consideration for publishers, with many deciding whether to allow or block the crawler based on business and content strategy decisions. Blocking GPTBot prevents inclusion in ChatGPT training and retrieval systems, eliminating visibility opportunities within ChatGPT and related products. Allowing GPTBot enables ChatGPT citations and potential traffic from ChatGPT response links.
Why it matters for AI visibility
GPTBot decisions directly affect visibility within ChatGPT, one of the largest AI platforms. Allowing GPTBot indexing enables citations in ChatGPT, capturing visibility with ChatGPT's massive user base. Blocking GPTBot eliminates this visibility but may align with content protection or intellectual property strategies. The decision should be informed by expected return from ChatGPT visibility relative to content protection concerns.
Related terms
AI Web Crawlers
AI Web Crawlers are automated systems deployed by LLM providers and AI companies to index and retrieve web content for LLM training, inference retrieval, or answer synthesis processes.
GEOAI Indexing
AI Indexing is the process by which LLM providers and AI systems crawl, parse, and store web content for use in LLM training, retrieval-augmented generation, or answer synthesis processes.
SEORobots.txt
Robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers and other bots which pages they can crawl and which to exclude. Using simple directives, it manages crawl budget allocation, prevents indexing of duplicate or low-value content, and protects sensitive areas, while helping publishers communicate with both search engine and AI crawlers.
GEOLLMs.txt
LLMs.txt is a proposed convention: a plain markdown file served at the root of a website that gives large language models a curated summary of the site's most important content. It helps AI systems find, understand, and use the right pages at answer time instead of parsing cluttered HTML.