Crawl budget is the amount of crawling Google can and wants to do on a site, shaped by crawl capacity and crawl demand. It is a real technical SEO concern, but not a universal one. Google’s crawl-budget guidance positions dedicated crawl-budget work mainly for very large sites, rapidly changing medium-to-large sites, and sites where a substantial share of known URLs remain in a Discovered – currently not indexed state.
That distinction changes the job. The aim is not to make Googlebot request as many URLs as possible. It is to make sure a large or fast-changing site does not waste crawler activity on duplicate, obsolete, low-value, or effectively unlimited URL spaces while important URLs remain difficult to reach or refresh.
Updated 7 October 2026: This guide was revised to reflect Google’s current crawl-budget guidance and to clarify the different roles of robots.txt, noindex, canonicalisation, URL inventory, and crawl diagnostics.

First Decide Whether Crawl Budget Is the Problem
A delayed crawl does not automatically mean a site has a crawl-budget problem. One new page waiting to be crawled, one indexing exclusion, or a temporary fall in crawl requests is not enough evidence on its own.
Google’s crawl-budget guidance is deliberately aimed at scale. It gives rough examples of sites with more than one million unique pages that change about weekly, sites with more than 10,000 unique pages that change daily, and sites with a large proportion of URLs reported as Discovered – currently not indexed. Google also states that these figures are approximations, not hard thresholds.
For a smaller site whose important pages are normally crawled soon after publication, routine technical SEO work is usually the better priority: keep discovery paths clear, maintain accurate sitemaps, fix serving problems, and investigate specific indexing issues rather than starting a standalone crawl-budget project.
Signals That Justify a Deeper Review
Crawl-budget analysis becomes more useful when the pattern is site-wide or template-wide. Examples include large numbers of important URLs waiting for a first crawl, server logs dominated by low-value parameters or filters, repeated crawling of obsolete URLs, persistent host-availability problems, or a fast-changing catalogue where important pages are not being refreshed as expected.
Search Console can point to patterns, but it does not prove the cause by itself. A useful diagnosis combines Page indexing data, Crawl Stats, representative URL inspection, and server logs where available. The question is not simply, ‘How many times did Google crawl?’ It is, ‘Which URL groups received those requests, and did that pattern match the site’s priorities?’

What Google Means by Crawl Budget
Google describes crawl budget through two main elements: crawl capacity limit and crawl demand. The combination determines the set of URLs Google can and wants to crawl on a hostname. It is better understood as a changing resource decision than as a fixed daily allowance that resets at a particular time.
Crawl Capacity Depends on Serving Health
Crawl capacity is designed to let Google fetch content without overwhelming the server. If a site responds reliably and quickly, Google can use more crawl connections. If response times deteriorate or the server returns errors such as 5xx responses or rate-limiting signals, Google reduces crawling.
This is why server health belongs in a crawl-budget investigation. Improving response times, avoiding unnecessary redirect chains, supporting efficient caching where appropriate, and correcting persistent server errors can make crawling more efficient. General site speed and server-response improvements may help, but the diagnosis should still be based on crawl and infrastructure data rather than assuming every performance issue is a crawl-budget issue.
Crawl Demand Is a Different Constraint
More capacity does not guarantee more crawling. Google says Googlebot demand varies with factors including site size, update frequency, page quality, relevance, popularity, staleness, and the inventory of URLs Google knows about. If demand is low, Google may crawl less even when the server could comfortably handle more requests.
The URL inventory is the part site owners can often influence most directly. If a platform exposes many duplicate, removed, sorted, filtered, tracked, or otherwise unimportant URLs, Google may spend time requesting addresses that do not deserve independent search visibility.
There is another important limit: crawling is not the same as indexing. Google explicitly notes that not every crawled page will be indexed. A page still has to be evaluated, consolidated where necessary, and considered suitable for the index. Crawl-budget work therefore cannot compensate for weak page purpose, duplication, or other indexing problems.

Find Where Crawler Time Is Being Spent
The strongest crawl-budget improvements usually begin with URL inventory rather than a directive. Group URLs by how the site creates them, then compare those groups with crawler activity. On ecommerce, directory, publishing, and marketplace sites, the largest sources of unnecessary crawl space often come from combinations rather than individual pages.
Useful URL groups to review include:
- faceted navigation and sorting combinations;
- tracking, session, and other technical parameters exposed through internal links;
- internal search-result URLs;
- duplicate category, taxonomy, archive, or print-style pages;
- soft 404s and obsolete URLs that still return successful responses;
- redirect chains that require several requests to reach the final destination;
- calendar, pagination, or filter patterns that create very large or effectively unbounded crawl spaces.
Once those groups are visible, decide which URLs are genuinely useful search destinations. That is a content and architecture decision as much as a crawler decision. A filtered page with stable demand and a distinct purpose may deserve to remain indexable. A sort-order variant that shows the same products in a different sequence usually does not need the same treatment.
Keep Discovery Signals Consistent
Internal navigation should point to preferred clean URLs wherever practical. If links repeatedly expose tracking or session variants, crawlers can discover a much larger URL inventory than the site intended. A sound internal linking structure also gives important pages a clear route without relying on an arbitrary rule such as ‘every page must be within four clicks of the homepage’. Click depth is a diagnostic clue, not a published Google threshold.
XML sitemaps serve a different purpose. They help Google discover preferred URLs, but they do not force crawling or indexing and should not be treated as a priority queue. Keep them focused on the canonical URLs you want considered for Search, and maintain accurate update information where the site supplies it. MOCOBIN’s guide to creating and maintaining an XML sitemap covers that implementation in more detail.

Match the Control to the URL’s Purpose
One of the easiest ways to create a crawl problem is to treat robots.txt, noindex, canonical tags, and status codes as interchangeable. They are not. The correct control depends on what the URL is supposed to do for users and search engines.
| URL situation | Primary goal | Typical control | Important caution |
|---|---|---|---|
| Duplicate or near-duplicate URL that users may still need | Consolidate search signals around a preferred version | Canonicalisation plus consistent internal links and sitemap signals | A canonical is a signal, not an absolute command. |
| Public page that users can access but that should not appear in Search | Prevent search inclusion | noindex while allowing Google to crawl the directive | Blocking the URL in robots.txt can prevent Google from seeing noindex. |
| Low-value URL space that Google does not need to crawl at all | Reduce unnecessary crawling | robots.txt after confirming the pattern and its purpose | Do not expect freed crawl capacity to be automatically reassigned to specific priority pages. |
| Permanently removed URL with no replacement | Show that the resource is gone | 404 or 410 response | Do not keep obsolete URLs returning a soft 404 or a misleading 200 response. |
| Preferred indexable page | Make the intended URL easy to discover and process | Successful response, consistent canonical signals, internal links, and sitemap inclusion where appropriate | Sitemap inclusion or a self-referencing canonical does not guarantee indexing. |
For crawl efficiency specifically, Google’s robots.txt guidance and current crawl-budget guidance make an important distinction: a noindex page still has to be requested so Google can read and process the directive. If a large class of URLs truly should never be crawled, robots.txt crawl controls may be appropriate. If the real objective is to keep a public page out of Search, noindex is the relevant tool instead. Mixing the two without understanding that dependency can produce the opposite of the intended result.
Broad blocking is not a substitute for classifying URL patterns. Faceted navigation is the obvious example. Some filter combinations may deserve independent landing pages; others are duplicate or low-value crawl spaces. The decision belongs at the pattern level, not as a blanket rule applied to every URL containing a question mark.

Verify the Change With Crawl Data
A crawl-budget fix is not complete when a directive is deployed. Check whether crawler behaviour actually changed and whether important URLs became easier to reach or refresh.
Google Search Console’s Crawl Stats report provides a site-level view of total crawl requests, download size, average response time, host status, response codes, file types, crawl purpose, and Googlebot type. The example URLs are representative rather than a complete log, so use server logs when you need a fuller pattern-level view of requests to filters, parameters, product templates, or other URL groups.
Compare the period before and after a material change. Useful questions include:
- Are server errors or long response times falling?
- Are low-value URL patterns receiving fewer requests where that was the intended result?
- Are important new or frequently updated pages being reached reliably?
- Are removed URLs returning the correct status rather than remaining as soft 404s?
- Do internal links, canonical signals, and sitemap entries now point to the same preferred URLs?
- Has the proportion of important URLs waiting in Discovered – currently not indexed improved, or is the pattern unchanged?
Do not use total crawl requests as a success metric in isolation. A lower request count can be healthy if it reflects fewer duplicate or obsolete URLs. A higher count is not automatically better if the additional requests are still being spent on crawl traps.
The useful target is crawl efficiency, not a larger number. If Google can reach the site’s preferred URLs, the server remains healthy, and large low-value URL spaces are under control, there may be nothing else to optimise. More crawling for its own sake would be a rather expensive way to create a prettier chart.











