What each one actually controls
Three different questions, which is why using the wrong one produces confusing results.
| Tool | Controls | Lives in | Crawler must fetch the page? |
|---|---|---|---|
| robots.txt Disallow | Whether the URL is fetched | One file at the site root | No, that is the point |
| noindex | Whether the page appears in results | A meta tag or HTTP header on the page | Yes, or it never sees it |
| canonical | Which similar URL is authoritative | A link tag or HTTP header on the page | Yes |
The combinations that cancel out
Two mistakes account for most of the trouble, and both come from assuming more signals are safer.
Disallow plus noindex. The Disallow stops the crawler fetching the page, so it never reads the noindex, so the page can remain in results. Google's documentation notes a disallowed page can still be indexed if other sites link to it. To de-index, allow crawling.
Disallow plus canonical. Same mechanism. A canonical tag is on the page, so blocking the page means the canonical is never read and the duplicate is never consolidated.
The general rule: robots.txt is the only one of the three that works without the page being fetched. Everything else requires access.
Picking the right one
Work backwards from the outcome.
You do not want this page in search results: noindex, and allow crawling so it can be read.
This page is one of several near-identical URLs and you want one to win: canonical on the variants, pointing at the preferred URL. Do not block the variants.
You want crawlers to stop wasting effort on a large set of low-value URLs: robots.txt Disallow. Accept that some may still appear as bare URLs in results.
The content must be genuinely inaccessible: none of the three. Use authentication.
Where each one goes
A practical detail that trips people up: only robots.txt is a file, and it must be at the site root.
noindex and canonical live on the page, either as tags in the head or as HTTP headers. The header form is useful for non-HTML files such as PDFs, where there is no head to put a tag in.
This has a consequence for hosted platforms. If your platform does not let you edit page-level tags, you may not be able to set noindex, and reaching for robots.txt as a substitute will not achieve the same thing. That is a real constraint rather than something to work around.
Common questions
What is the difference between robots.txt and noindex?+
robots.txt asks a crawler not to fetch a URL. noindex asks a search engine not to show a page in results. Google states robots.txt is not a mechanism for keeping a page out of Google, so for that you need noindex.
Can I use Disallow and noindex together?+
No, they cancel out. The Disallow prevents the crawler fetching the page, so the noindex is never seen. Allow crawling and use noindex alone.
Should I block duplicate pages in robots.txt?+
No. Use a canonical tag on the variants pointing at the preferred URL. Blocking them means the canonical is never read and nothing gets consolidated.
How do I noindex a PDF?+
With the X-Robots-Tag HTTP header, since a PDF has no head section for a meta tag. Same directive, delivered differently.
Sources