sitemap and robots.txt: hints, not locks, five minutes

web · 2026-10-13 · 6 min

a sitemap is a file listing the pages you consider important — a hint helping search engines discover URLs, never a guarantee they'll crawl or index them. robots.txt beside it tells crawlers which areas they may access. neither is security, neither is ranking magic; both are five-minute basics done once.

sitemap: the list of destinations

XML file (usually /sitemap.xml) with your canonical URLs, ideally auto-generated by the CMS/platform (wordpress SEO plugins, shopify, wix: built in). google: sites under ~500 well-linked pages may not need one; new sites with few inbound links, large sites, and media-heavy sites benefit most. submit it in search console's sitemaps report (you give google the URL; you don't upload anything). update automatically (regenerate on publish) — stale sitemaps listing dead URLs teach crawlers to distrust yours.

robots.txt: access instructions, not a lock

/lives at /robots.txt: which crawlers may fetch which paths. uses: keep crawlers out of junk (admin, carts, filters, staging) to save attention. NOT uses: hiding pages (blocked URLs can still appear in results — use noindex or passwords for real exclusion), security (attackers read robots.txt as a treasure map of your hidden paths). staging sites: password-protect, don't robots-rely — leaks happen.

the five-minute setup

generate sitemap (plugin/platform, verify it lists canonical URLs only). submit in console, watch the discovered count. write robots.txt: allow the site, disallow junk (admin, search-internal, parameters), point to the sitemap. never disallow everything on the live site (the classic launch-day catastrophe — staging's disallow shipped to production). check both after every redesign. console manual: what is google search console. pipeline context: how search engines crawl and index.

FAQ

HTML sitemap too?

the human-facing page listing (different thing, same name): useful for visitors on large sites, irrelevant for crawlers. do it for humans when navigation needs help.

sitemap errors — panic?

no: console flags dead/excluded URLs routinely. fix patterns (wrong URLs generated), not singletons. a sitemap with warnings still works; a missing one on a big site just discovers slower.