What Is robots.txt, and What It Cannot Do
robots.txt is a plain text file at the root of your site telling crawlers which paths not to request. The Robots.txt Generator writes one, and the important thing is a distinction most explanations skip.
Crawling is not indexing
robots.txt says "do not fetch this". It does not say "do not list this".
So a blocked page can still appear in search results - typically with no description, because the crawler was never allowed to read it. If other sites link to that URL, Google knows it exists and may index the address on the strength of those links alone.
Which means robots.txt is the wrong tool for keeping a page out of search. For that you need a noindex instruction in the page's own meta robots tag or HTTP header - and crucially, the crawler has to be *allowed* to fetch the page to see it. Blocking in robots.txt and adding noindex at the same time is self-defeating: the noindex is never read.
Google also stopped supporting a noindex directive inside robots.txt in 2019. Any example you find using it is out of date.
What it is genuinely for
Crawl budget. Stopping crawlers wasting requests on faceted search URLs, infinite calendars, internal search results and printable duplicates - the kinds of paths that generate limitless variations.
Server load, where a crawler hitting an expensive endpoint repeatedly causes real problems.
Keeping private directories unrequested, with the enormous caveat below.
It is a public file and a request
Two things to be clear about. The file is readable by anyone at yoursite.com/robots.txt, so listing a secret directory in it advertises that directory. More than one site has had its admin path discovered this way.
And compliance is voluntary. Well-behaved crawlers obey it; a scraper that wants your content simply ignores it. robots.txt is not security. Anything that must not be accessed needs authentication.
Blocking CSS and JavaScript
A habit that survives from years ago and now does harm. Google renders pages, so blocking your stylesheets and scripts means it sees a broken layout and cannot assess mobile usability. Let it fetch your assets.
Common questions
Do I need a robots.txt at all?
No. Without one, everything is crawlable, which is usually what a small site wants. An empty or missing file is fine.
How do I keep a page out of Google?
A noindex meta tag on a crawlable page - the Robots Meta Tag Generator produces it - or a password.
Should the sitemap go in it?
Yes, a Sitemap line is a sensible addition. The XML Sitemap Generator builds the file it points at.
Can I set a crawl delay?
Some crawlers honour a Crawl-delay line; Google ignores it entirely and uses its own rate, which you can adjust in Search Console if a crawler is genuinely overloading the server.
Does one file cover subdomains?
No. Each subdomain needs its own robots.txt at its own root, because they are separate hosts as far as a crawler is concerned.
Need a robots.txt for your site?