The short version
robots.txt is a plain text file at your site root that asks crawlers which paths to avoid. It is a request, not an enforcement mechanism, and it does not remove pages from search results. The failures that matter are a blanket Disallow: /, HTML served in place of text, and a missing Sitemap directive.
What robots.txt actually controls
It controls crawling, not indexing, and confusing those two causes most of the trouble people have with it.
A Disallow line asks a crawler not to fetch a path. It does not ask search engines to remove that page from results, and Google has been explicit that a disallowed URL can still appear in search if other sites link to it, typically with no description because the content was never read.
So if your goal is to keep a page out of search results, robots.txt is the wrong tool. You want a noindex meta tag on the page, which requires that the crawler be allowed to fetch the page in order to see the tag. Blocking a page in robots.txt and adding noindex to it is self-defeating: the crawler never reads the instruction.
It is also purely advisory. Well-behaved crawlers honour it. Anything scraping your site maliciously will not, and a Disallow line is not a security measure. Never use it to hide something sensitive, because the file is public and reading it is the fastest way to discover the paths you would rather nobody visited.
The failures worth catching
Most robots.txt validators produce long lists of cosmetic warnings. Four problems genuinely cost you traffic.
| Problem | What it does | How urgent |
|---|---|---|
| Disallow: / for user-agent * | Asks every crawler to ignore the whole site | Critical if unintentional |
| HTML served as robots.txt | Every directive is ignored, and it looks fine in a browser | Fix now, it is invisible |
| No Sitemap directive | Crawlers have to guess where your sitemap is | Low, one line |
| Blocking a path that has noindex | The crawler never sees the noindex, so the page can stay in results | Worth fixing, commonly misunderstood |
Crawl-delay, and who ignores it
Crawl-delay asks a crawler to wait a number of seconds between requests. It is one of the most misunderstood lines in the file, for a simple reason: Google does not support it.
Google has stated plainly that it ignores the crawl-delay directive, and manages its own crawl rate instead. Bing and several other crawlers do honour it. So a Crawl-delay line has no effect on the crawler most people care about, while genuinely slowing the ones that respect it.
That matters if the value is high. A crawl-delay of 10 means a crawler that honours it fetches six pages a minute at most, which on a large site means new content takes a very long time to be discovered. If you set it years ago to reduce server load and your hosting has since improved, it is worth removing.
AI crawlers: a choice worth making deliberately
A newer section of most robots.txt files concerns AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and others.
Blocking them is a legitimate decision, and plenty of publishers have made it. But it has a consequence people sometimes do not connect: those assistants then cannot read your site to answer questions about your business. If someone asks an assistant who to hire in your area, a blocked site is harder to recommend.
There is no universally right answer. A publisher whose content is the product has a real interest in not having it ingested. A local business that wants to be recommended has close to the opposite interest. What matters is that the choice was made on purpose rather than copied from a template.
The tester above lists which AI crawlers a robots.txt blocks, so you can at least see what you have decided.
Questions about this tool
Will robots.txt remove a page from Google?+
No. It asks crawlers not to fetch a path. Google has said a disallowed URL can still appear in results if other pages link to it, usually without a description. To keep a page out of search, use a noindex meta tag and allow the crawler to fetch the page so it can see it.
Does Google respect Crawl-delay?+
No, Google ignores it and manages its own crawl rate. Bing and some other crawlers do honour it, so a high value slows discovery on those without affecting Google at all.
Do I need a robots.txt file?+
Not strictly. A missing file is treated as permission to crawl everything, which is fine for most sites. The main reason to have one is to declare your sitemap and to keep crawlers out of paths that waste their time.
Is robots.txt a security measure?+
No, and treating it as one is dangerous. The file is public, so listing a sensitive path in it advertises that path. Malicious scrapers ignore it entirely. Use real access control instead.
Should I block AI crawlers?+
It depends on your interests, and it is a real trade-off. Blocking them protects content from being ingested, but it also means those assistants cannot read your site to recommend your business. Decide deliberately rather than copying a template.