What Is Crawling in SEO? How Search Bots Read Your Site

TL;DR: An SEO crawl is the discovery and download step. Googlebot finds a URL, fetches its content, and may render its JavaScript. Crawling does not guarantee indexing or rankings, so identify whether the failure happened at crawling, rendering, indexing, or ranking before changing your site.

Most crawl problems are diagnosed from the wrong end. A page is missing from search, someone concludes that Google has not crawled it, and robots.txt gets edited. Yet Google may already have fetched the page and decided not to index it, selected another canonical, or found no reason to rank it for the query being checked.

Crawl, render, index, and rank are separate gates. Google officially places rendering inside the crawling stage, but separating them during diagnosis makes technical failures much easier to locate.

Gate	What Google does	Typical failure	Where to investigate
Crawl	Discovers and downloads the URL	robots.txt block, timeout, network failure, server 5xx	URL Inspection, Crawl Stats, server logs
Render	Runs JavaScript and builds the rendered page	Blocked JS or CSS, failed scripts, heavy client-side rendering	URL Inspection and rendered output
Index	Analyzes the page, chooses a canonical, and may store it	noindex, duplication, weak content, another canonical selected	URL Inspection and indexing state
Rank and serve	Selects and orders indexed pages for a query	Poor query match, insufficient quality or relevance	Search performance and page-level relevance

What crawling in SEO actually means

Google Search Central defines crawling plainly:

“Crawling: Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.”

In practical terms, crawling is discovery plus fetching. Google learns that a URL exists and requests its contents so the page can be processed later.

The program doing this work is Googlebot. Google describes it as “also known as a crawler, robot, bot, or spider.” Those names refer to the same general type of software: an automated program that requests pages and follows links to discover more URLs.

Googlebot does not inspect your site by clicking around visually like a person. It requests HTML and supporting resources. During the crawl phase, Google can also render the page using a recent version of Chrome and execute its JavaScript. This matters if your meaningful text, navigation, or internal links do not exist until scripts run.

One further detail is easy to miss: Googlebot is predominantly mobile. Google says, “For most sites Google Search primarily indexes the mobile version of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawler, and a minority using the desktop crawler.” If important content exists on desktop but is absent from the mobile version, successful desktop testing may give you false confidence.

I have spent time fixing the wrong gate before. A URL missing from search does not prove that discovery failed, and requesting another crawl will not repair duplicate or unhelpful content (a frustratingly common category error).

Crawl, render, index, and rank are different jobs

1. Crawl: discover and download

Googlebot first needs to know that a URL exists. Once discovered, it attempts to fetch the page’s raw HTML and required resources. A robots.txt restriction, DNS or network failure, timeout, or server error can stop the process here.

A successful fetch gets the URL through the first gate. Nothing more.

2. Render: execute the page

Google officially includes rendering within crawling, but it is worth testing separately (not a fourth official stage, to be precise). Google says that during crawling it renders pages and runs JavaScript using a recent version of Chrome, much like a normal browser.

For JavaScript applications, Google documents the working sequence as crawling, rendering, and indexing. Pages returning a successful response can enter a rendering queue, where headless Chromium executes their JavaScript. Google Search uses an evergreen version of Chromium rather than a permanently frozen browser version.

That creates an important diagnostic split. Googlebot may fetch an HTTP 200 page while still failing to see its final content. Blocked resources, script errors, delayed client-side content, or links that only appear after user interaction can leave the rendered result incomplete.

3. Index: analyze and store

Google defines indexing as analyzing a page’s text, images, and videos and storing information in the Google index. During this process, Google also evaluates duplicates and chooses which URL it considers canonical.

A crawled page is not automatically indexed. A noindex directive can exclude it. Google may also decline to index thin or substantially duplicate material, or it may consolidate the page under another canonical URL. Our guide to how Google indexing works covers this handoff in more detail.

4. Rank and serve: answer a query

Indexed means eligible to appear. It does not mean visible for every search or guaranteed a particular position. Google still has to decide that the page matches a query and determine where it belongs among other eligible results.

• A URL can be discovered but not successfully fetched.
• It can be fetched but rendered incorrectly.
• It can be rendered but excluded from the index.
• It can be indexed but fail to rank for the query you check.

Google gives site owners a useful reality check: “Google doesn’t guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.” Technical accessibility is necessary. It is not a promise of inclusion or visibility.

How Google discovers pages

Google does not begin with a complete directory of every URL on the web. Its documentation identifies three main discovery routes:

1. Revisiting known URLs. Google returns to pages it has previously fetched.
2. Following links. Google extracts a link from a known page and discovers its destination.
3. Reading sitemaps. Google processes lists of URLs submitted by site owners.

This is why internal linking is infrastructure, not decoration. A page with no internal links pointing to it is an orphan. It may exist in the CMS and appear in the XML sitemap while remaining disconnected from the routes Googlebot normally follows.

Cyrus Shepard’s study of roughly 23 million internal links across about 1,800 sites found a directional relationship between internal links and Google clicks: “URLs with 0-4 internal links received an average of 2 clicks from Google Search, while URLs with 40-44 internal links received 4 times that many.” This was correlational, not proof that adding links mechanically multiplies traffic.

The study also found that the effect reversed after roughly 45 to 50 internal links. More is not always better. The useful conclusion is that important pages need several relevant crawl paths, not that every page should receive dozens of arbitrary links.

From what we see across sites on SEOJuice, orphan pages and weak internal routes appear far more often than genuine crawl-capacity problems. The fix is usually to connect the page from relevant, already-indexed content and retain it in the sitemap. A coherent structure such as content silos for SEO gives crawlers repeatable routes between related pages.

I used to treat sitemap inclusion as stronger evidence than it is. It proves that a URL was supplied to Google, not that Google fetched or indexed it (I was overreading the signal). A sitemap aids discovery; it does not replace internal links or compel an outcome.

For a single new or updated URL, URL Inspection in Google Search Console also offers a Request Indexing action. That adds a request to Google’s processing queue. It is a nudge, not a command.

robots.txt, noindex, and nofollow do different things

robots.txt controls fetching

A robots.txt file tells compliant crawlers which URL paths they may access. Google says it is used mainly to prevent crawler requests from overloading a site.

It is not a dependable removal tool. Google’s robots.txt documentation states that “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites” and explicitly calls the file “not a mechanism for keeping a web page out of Google.” Google can know a blocked URL exists without being permitted to fetch its contents.

Inspect every broad rule before publishing. A production Disallow: / consists of very little text and can remove Googlebot’s access to the whole site. The robots.txt generator can help draft a file, but its output still needs to be tested against your important paths.

noindex controls index eligibility

Use noindex when Google may crawl a page but should not retain it in search results. Do not also disallow that URL in robots.txt. Google must fetch the page to read its noindex directive.

Applying both can produce the opposite of the intended certainty. The crawl restriction prevents Google from seeing the indexing instruction, so the blocked URL may remain known through external or internal links.

nofollow is a link-level hint

The nofollow attribute is not a page-level crawl prohibition. Google treats it as a hint, not a guaranteed block. It should not be used to hide a URL, protect private content, or ensure exclusion from the index.

Faceted navigation is where these clean definitions become less comfortable. There is no universally safe robots.txt pattern for every catalogue. Some filtered URLs satisfy distinct searches; others create effectively infinite combinations. Blocking first and mapping value later is risky (and yes, this is one area where I distrust one-line fixes).

You probably do not have a crawl-budget problem

Crawl budget is the set of URLs Google can and wants to crawl. Google describes two components: the crawl capacity limit, which considers parallel connections, response time, and server health; and crawl demand, which reflects factors such as site size, update frequency, quality, popularity, and staleness.

The term sounds important enough to concern every website owner. Google’s thresholds say otherwise.

“If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.”

Google directs its crawl-budget guidance primarily at sites with more than one million unique pages changing about weekly, or sites with more than 10,000 unique pages changing daily. For most smaller sites, maintaining the sitemap and checking indexing coverage is sufficient.

If you publish a modest number of URLs and Google fetches new pages promptly, stop trying to optimize crawl budget. Fix broken discovery paths, accidental directives, rendering failures, server errors, or content quality instead. For genuinely large or highly dynamic sites, our crawl budget optimization guide covers the more difficult cases.

How to see what Google is crawling

Start with URL Inspection

For one missing page, URL Inspection in Google Search Console is the quickest starting point. It can show the last crawl date, whether crawling was permitted, the indexing status, and Google’s selected canonical.

Read those fields as a sequence. If crawling was blocked, repair access. If Google crawled the URL but excluded it from the index, another crawl request is unlikely to solve the underlying problem.

Use Crawl Stats for site-wide patterns

The Crawl Stats report shows Google’s crawling history, including request counts, total download size, average response time, server responses, availability problems, file types, crawl purpose, and Googlebot type.

Google calls this an advanced report and says sites with fewer than 1,000 pages generally should not need it. That is reasonable. I still inspect it after migrations or abrupt site-wide changes, but it is not a useful daily scoreboard (more crawling is not automatically better).

Use server logs for exact requests

Server access logs record requests that actually reached your infrastructure. They can reveal which URLs Googlebot fetches, how frequently it visits each section, and whether parameters or junk URLs are receiving attention while valuable pages are rarely visited.

Googlebot user-agent strings contain Googlebot/2.1, but user-agent text can be imitated. For reliable identification, verify suspected Googlebot requests through reverse and forward DNS rather than trusting the string alone.

For most sites, genuine Googlebot activity should be fairly gentle. Google says, “For most sites, Googlebot shouldn’t access your site more than once every few seconds on average.”

Five SEO crawl failures worth fixing

1. Orphan and near-orphan pages

An orphan has no internal link route from the rest of the site. A near-orphan may have one obscure link from an archive or pagination page that Google rarely revisits. Add contextual links from relevant, indexed pages instead of relying solely on a sitemap.

This is usually the first structural problem I check. The URL exists, so the team assumes the site exposes it properly. Those are not the same thing.

2. Blocked JavaScript or CSS

Google renders pages with Chromium. If robots.txt blocks resources needed to reveal content or navigation, the HTML request can succeed while the rendered page remains incomplete.

Compare Google’s rendered output with what users see, particularly on client-rendered applications whose initial HTML contains little substantive content.

3. Infinite crawl traps

Filters, sorting parameters, calendars, search pages, and session IDs can generate enormous numbers of low-value URL combinations. This becomes a material crawling problem on large sites.

Common remedies include consolidating duplicates with canonicals, limiting crawlable parameter combinations, and blocking appropriate patterns. Map which combinations deserve indexing before restricting them; a blanket rule can remove valuable category pages along with the junk.

4. Slow or unreliable servers

Google adjusts crawl capacity in response to server health. Persistent timeouts, slow responses, and 5xx errors can make Google reduce crawling. Crawl Stats can expose aggregate response-time changes, while logs identify the templates and URLs causing them.

5. Accidental production blocks

Check for staging robots.txt rules deployed to production, site-wide noindex directives, blocked assets, and CMS settings intended to discourage search engines. They are mundane failures with site-wide consequences.

During our seojuice.io to seojuice.com migration in January 2026, we treated discovery, fetching, canonical selection, and indexing as separate checks. An old-host URL being recrawled did not prove that its new-host replacement had been indexed, and a correct redirect did not prove that Google had switched its canonical yet. That separation kept us from reducing the migration to a misleading “Google sees it” checkbox.

We are a two-person team, Lida and me, so diagnostic order matters. We cannot spend days polishing crawl-budget theories while an orphan page or production noindex is doing the actual damage.

If those recurring issues are difficult to monitor manually, SEOJuice has a free plan with no credit card required. Its automation can add relevant internal links and its site audit flags robots.txt blocks, noindex tags, and blocked resources. It can fix or surface specific on-site problems; it cannot force Google to crawl faster, index a page, or award rankings.

Frequently Asked Questions

What is crawling in SEO?

Crawling in SEO is the process through which search-engine bots discover and download pages. Googlebot finds URLs by revisiting known pages, following links, and reading sitemaps. Successful crawling does not guarantee that a page will be indexed or ranked.

What is the difference between crawling and indexing?

Crawling means Google fetches a page. Indexing means Google analyzes the page, evaluates its canonical, and may store information about it in the Google index. A page can be crawled but not indexed because of noindex, duplication, content quality, or canonical selection.

What is a web crawler, spider, or bot?

A web crawler is an automated program that visits URLs, downloads content, and follows links to find other pages. Google’s crawler is Googlebot. Crawler, spider, robot, and bot are different names for the same general type of software.

How does Google find and crawl my website?

Google revisits URLs it already knows, follows links from known pages to new destinations, and reads submitted XML sitemaps. Internal links provide direct discovery routes, while the sitemap supplies a useful list of URLs Google can consider crawling.

Does robots.txt stop Google from crawling or indexing?

robots.txt restricts crawling, not indexing. A blocked URL can still appear in Google if it is discovered through links. To keep a crawlable page out of the index, use noindex and do not simultaneously prevent Google from fetching the directive.

How can I see what Google is crawling?

Use URL Inspection for the status of one URL, Crawl Stats for site-wide request and server patterns, and server access logs for the exact URLs Googlebot requested. Together, those sources distinguish an actual crawl failure from a later rendering or indexing problem.