Crawl Budget
Crawl Budget is the number of URLs a search engine crawler can visit on a website within a given time period, allocated based on site authority and server capacity. Limited crawl budget requires strategic resource allocation to ensure crawlers prioritize important pages and avoid wasting resources on duplicate content, errors, or low-value pages that should not be indexed.
Definition
Search engine crawlers allocate a crawl budget based on site authority and hosting capacity. High-authority sites can have larger budgets, crawling millions of pages regularly. Lower-authority sites have smaller budgets, maybe thousands of pages per day. Crawl budget is spent on every page request, including duplicate content, redirects, and error pages. Wasting crawl budget on pages that should not be indexed reduces the crawl rate for important content.
Optimizing crawl budget involves blocking unnecessary pages with robots.txt (duplicate content, parameters, admin pages), fixing redirect chains that waste crawl requests, removing noindex pages that still get crawled, implementing proper canonicalization to avoid crawling multiple versions, and ensuring important content is discoverable through quality internal linking. Monitoring crawl data in Google Search Console identifies patterns and optimization opportunities.
Why it matters for AI visibility
Like search engine crawlers, AI crawlers have resource constraints and prioritize which pages to crawl. If your website wastes crawl budget on duplicate content and low-value pages, AI crawlers discover less of your unique, valuable content. Optimizing crawl budget ensures AI crawlers efficiently discover and index your best content, increasing the likelihood that important pages are available for AI systems to cite.
Related terms
Robots.txt
Robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers and other bots which pages they can crawl and which to exclude. Using simple directives, it manages crawl budget allocation, prevents indexing of duplicate or low-value content, and protects sensitive areas, while helping publishers communicate with both search engine and AI crawlers.
SEOCrawling and Indexing
Crawling is the process of search engine and AI bots discovering web pages by following hyperlinks and sitemaps, while indexing is the process of storing, parsing, and analyzing page content so it can be retrieved and ranked in search results or cited in AI responses. Not all crawled content is indexed; pages may be excluded due to directives, quality signals, or duplication.
SEOCanonical URL
A Canonical URL is a specified preferred version of a web page when multiple URLs contain the same or similar content, communicated via a rel=canonical HTML tag. Canonical tags consolidate ranking authority on the preferred version, prevent duplicate content confusion, and guide both search engines and AI systems to the authoritative source when multiple versions exist.
SEOGoogle Search Console
Google Search Console is Google's official free platform for website owners to monitor a website's presence in search results, report crawl and indexation issues, submit sitemaps and URL changes, and analyze comprehensive search performance data including queries, impressions, and click metrics.