Crawl budget is the number of pages Google’s bot crawls and indexes on your site within a given time period. If a page is not crawled, it will not appear in search results.
Most sites do not need to worry about crawl budget. If your pages are indexed on the same day you publish them, you are fine. According to Google’s official documentation, crawl budget optimization is primarily relevant for large sites (1 million+ pages with weekly changes) or medium sites (10,000+ pages with daily changes).
That said, understanding how crawl budget works helps you avoid common mistakes that waste crawling resources, especially on sites that generate many URLs dynamically (e.g. faceted navigation, URL parameters).
Crawl Rate Limit
The crawl rate limit defines the maximum number of simultaneous connections Googlebot uses to crawl your site and the delay between requests. This limit exists to prevent the crawler from overloading your server and degrading the user experience.
The crawl rate can change over time and is influenced by two main factors:
- Crawl Health – If your site responds quickly at a certain time, the limitation will be smaller, meaning more simultaneous connections will be established for crawling.
- Crawl Budget Limit in Google Search Console – Website owners can limit or reduce the crawl frequency of Googlebot through Google Search Console.
Crawl Demand
Even if the crawl rate limit has not been reached, Googlebot will not crawl your site much if there is no demand for it. The two main factors affecting crawl demand are:
- Popularity – More popular URLs or domains on the Internet tend to be scanned more frequently by Google.
- Staleness – Google’s systems attempt to prevent situations where URLs are no longer relevant in the index.
Broader events, such as migrating to a new domain, can also trigger an increase in crawl demand as Google re-indexes the new URLs.
Combining the crawl rate limit with crawl demand gives you the crawl budget – the total number of URLs Googlebot can and wants to crawl on your site.
Factors Affecting Crawl Budget
Low-value URLs waste crawl budget that could be spent on pages with actual content. Here are the most common causes, ordered by impact:
- Faceted navigation – For example, filtering by color or price in digital stores can help users but create many combinations of URLs. These include Session Identifiers and various parameters in the URL.
- Duplicate content on your site.
- Soft 404 errors – Errors received when the server returns a 200 as an HTTP Response Code for non-existent pages instead of returning a 404 error. These pages can interfere with crawling, as these URLs will be scanned instead of URLs with unique content on your site.
- Low-value content and spam content.
- Infinite navigation – Infinite pagination, calendars that allow endless browsing between months and years, or any navigation that generates unlimited URLs will waste crawl budget.
- Redirects – Every time a page on your site makes a redirect, it uses a small part of the crawl budget. Limit the number of redirects to avoid wasting this budget.
Addressing these issues frees crawl budget for pages with actual value, resulting in more efficient crawling and faster indexing of important content.
Tip! You might want to prevent your search result pages from being indexed to save the crawl budget.
FAQs
Common questions about crawl budget:
Summary
Crawl budget determines how many pages Googlebot crawls on your site within a given period. It is shaped by the crawl rate limit (server health, Google’s resources) and crawl demand (page popularity, content freshness).
For most small to medium sites, crawl budget is not a concern. For larger sites, reducing low-value URLs, fixing errors, and maintaining clean internal linking are the most effective ways to ensure important pages get crawled and indexed.
