Crawl Budget Explained: Does It Affect Your Indexing?

Crawl budget is the number of URLs Google is willing to fetch from your site in a given window, and for most sites it does not affect indexing at all. It only becomes a real constraint on very large sites or sites stuffed with low-value URLs. If a specific page is stuck waiting, the URL Indexer tool can prompt Google to crawl it without you reorganizing your whole site. This guide explains what crawl budget is made of, when it matters, and what to actually do about it.

What is crawl budget made of?

Crawl budget is the combination of two things: how much Google is willing to crawl, and how much it wants to crawl. Google describes these as the crawl rate limit (or crawl capacity) and crawl demand. Both have to line up before a URL gets fetched.

• Crawl rate limit (crawl capacity). The maximum number of simultaneous connections and the delay between fetches that Google uses so it does not overload your server. Fast, reliable hosting raises this ceiling. Slow responses, timeouts, and 5xx errors lower it, because Google backs off to avoid hurting your site.
• Crawl demand. How much Google actually wants to crawl your URLs. Popular pages, pages that change often, and fresh content get more demand. Stale, thin, or duplicate URLs get less. If Google sees no reason to refetch a page, demand stays low even when capacity is available.

Crawl rate is the per-fetch pace within that capacity. You cannot raise demand by asking nicely; you raise it by publishing pages worth crawling and keeping your server fast. The two factors multiply: a fast server with low demand still gets crawled lightly, and high demand on a slow server is throttled by capacity.

Does crawl budget matter for your site?

For most small and mid-sized sites, crawl budget does not matter and you should not optimize for it. Google has said directly that crawl budget is not something most publishers need to worry about, and that sites under a few thousand URLs are almost always crawled efficiently. If you have a blog, a local business site, or a store with a few hundred or a few thousand pages, Google can fetch all of it without running out of budget.

How does crawl budget connect to "discovered, currently not indexed"?

Crawl budget is one of the main reasons a page shows up as "discovered, currently not indexed" in Search Console. That status means Google found the URL but has not fetched it yet, so it cannot be indexed. On a site with crawl budget pressure, newly discovered URLs sit in a queue while Google spends its limited fetches on higher-priority pages. The fix is partly about value and partly about freeing up budget, which we cover in detail in the causes and fixes for "discovered, currently not indexed".

If instead your pages are getting fetched but not kept, that is "crawled, currently not indexed", which is a content quality signal rather than a budget signal. Crawl budget explains why a page never gets fetched; it does not explain why a fetched page gets skipped. To understand the full path from discovery to indexing, see how Google indexing works.

How do you fix crawl budget problems?

Fix crawl budget by removing the URLs Google is wasting fetches on and making your important pages easy to reach. Work through these in order:

1. 1Prune low-value URLs. Identify thin tag archives, internal search results, near-duplicate filters, and stale pages. Consolidate, redirect, or remove them so Google stops spending fetches on pages you do not need indexed.
2. 2Block what should never be crawled. Use robots.txt to keep crawlers out of infinite parameter spaces, faceted navigation combinations, and admin paths. Remember robots.txt blocks crawling, not indexing, so use meta noindex on pages you want kept out of the index but still crawlable.
3. 3Fix server errors and speed. Reduce timeouts and 5xx responses, and improve response time. A faster, more reliable server raises your crawl capacity, so Google fetches more URLs per visit.
4. 4Strengthen internal linking. Link to important pages from your homepage, hubs, and related content using descriptive anchor text. Pages buried deep with few internal links read as unimportant and get crawled last.
5. 5Keep sitemaps clean and current. List only indexable, canonical URLs with accurate lastmod dates. A sitemap full of redirects, errors, or noindex URLs wastes the signals Google relies on to prioritize crawling.
6. 6Avoid long redirect chains and duplicate URLs. Each hop and each duplicate variant consumes a fetch. Settle on one canonical URL per page and link to it consistently.

These steps help your whole site, but they take time to show effects. When you have a specific batch of pages or backlinks you need crawled now, you can prompt Google directly instead of waiting for the queue to clear. That is where submitting URLs comes in, which pairs well with bulk indexing a list of URLs.

Can you control how fast Google crawls?

You cannot set a crawl rate the way you once could; Google retired the manual crawl rate control and now adjusts automatically based on your server's responses. You influence it indirectly: keep your server fast and stable to raise capacity, publish and update content worth fetching to raise demand, and remove junk URLs so the budget you have goes to pages that matter. You can also prompt individual URLs to be fetched sooner. URL Indexer sends standard indexing-request signals to Google so a page gets looked at, without needing Search Console access or ownership of the site. That works for your own pages and for third-party backlinks Search Console will never let you submit. Crawlers often visit within a few days, and confirmed indexing can take from a few days to a couple of weeks.