Crawling and indexing are different things
This is the single most consequential misunderstanding about robots.txt, and it causes real damage because the mistake looks like a fix.
A Disallow line asks a crawler not to fetch a URL. That is all it does. It says nothing about whether the URL may appear in search results, and Google's own documentation is explicit that robots.txt "is not a mechanism for keeping a web page out of Google".
So blocking a page can leave you in the worst of both positions: the page still appears in results, but because the content was never read, it appears with no useful description. Google notes that "the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results".
The self-defeating combination
Here is the trap that catches people who half-know this.
Someone wants a page out of search, so they add a noindex tag to the page and a Disallow line to robots.txt, reasoning that two measures are safer than one.
They are not. The Disallow stops the crawler fetching the page, so the crawler never reads the noindex tag, so the instruction to de-index is never received. The two measures actively cancel each other.
If you want a page gone from search, the crawler has to be allowed in to see the noindex. Counter-intuitive, and it is how the protocol works.
What to use for what
Pick based on the outcome you actually want, not on what feels strongest.
| Goal | Use | Not |
|---|---|---|
| Keep a page out of search results | noindex meta tag, and allow crawling | robots.txt Disallow |
| Keep content genuinely private | Password protection or real access control | robots.txt, which is public |
| Stop crawlers wasting time on junk URLs | robots.txt Disallow | noindex, which still costs a fetch |
| Remove something urgently | Search Console removal, then noindex | Either one alone |
robots.txt is not privacy
Worth stating plainly because it is a live risk rather than a theoretical one.
The file is public and always at a predictable location. Anyone can read it. So listing a path you would rather nobody visited does the opposite of hiding it: it publishes a list of the URLs you consider sensitive.
Reading a site's robots.txt is a standard first step in any reconnaissance, precisely because people put admin paths, staging environments and backup directories in it.
If something must not be seen, it needs authentication. robots.txt is a request to well-behaved crawlers and nothing more.
Common questions
Will robots.txt remove my page from Google?+
No. Google states robots.txt is not a mechanism for keeping a page out of Google, and a disallowed page can still be indexed if other sites link to it, typically appearing with no description.
Should I use both noindex and Disallow?+
No, they cancel each other. The Disallow stops the crawler fetching the page, so it never reads the noindex tag. Allow crawling and use noindex alone.
Why does a blocked page still show in search?+
Because links from other sites tell Google the URL exists. Google can index the address and anchor text without ever reading the page, which is why blocked pages often appear without a description.
Can I use robots.txt to hide private content?+
No, and it makes things worse. The file is public at a predictable location, so listing a sensitive path advertises it. Use password protection or proper access control.
Sources