crawl, index, serve: the pipeline behind every ranking

web · 2026-10-13 · 5 min

google finds pages in three stages — crawl (fetch), index (analyze and maybe store), serve (match to queries). nothing is guaranteed at any stage. understanding the pipeline diagnoses every visibility problem you'll ever have: each failure has exactly one stage, and each stage has its own fix.

stage 1: discovery and crawl

google finds URLs via links from known pages and submitted sitemaps. new sites with zero inbound links are invisible until linked or submitted — links are discovery, not just votes. crawl budget (how much google fetches) matters only at huge scale (google: ~1M+ pages or 10k rapidly-changing); small sites should ignore the term entirely and keep sitemaps current instead. blocked crawling (robots.txt, noindex, downtime) fails here: why is my website not on google.

stage 2: index (the maybe-store)

fetched pages are analyzed: content, metadata, duplicates, quality signals. outcomes: indexed (eligible to show), or excluded with a reason in console (crawled-not-indexed, discovered-not-indexed, duplicate-canonical, soft-404...). each reason is a different fix — thin pages need completing, duplicates need canonicals (canonical urls), junk needs deleting. indexing is eligibility, not victory.

stage 3: serve (the ranking you see)

indexed pages compete per query on relevance, quality, trust, context. indexed-but-invisible means losing here, not broken earlier — a content problem wearing a technical costume. the full game: how google decides what to show first. submitting URLs and sitemaps aids discovery; neither forces indexing nor ranking — anyone selling guaranteed indexing sells misunderstanding.

FAQ

crawled but not indexed — why?

consult console's exact reason: usually thin/duplicate/low-value verdicts on the page. fix the page (complete, differentiate, or consolidate), request re-indexing, wait days to weeks.

how do I watch my own pipeline?

console: page indexing report (stages 1-2), performance report (stage 3 evidence). the manual: what is google search console.