Do I Need a robots.txt File?

What happens with no file at all

Nothing bad. A crawler requesting /robots.txt and getting a 404 concludes there are no restrictions and proceeds to crawl normally.

So the absence of a robots.txt is not an error, is not a ranking factor, and does not need fixing urgently. Plenty of perfectly healthy sites do not have one.

This is worth saying because audit tools flag a missing robots.txt as an issue, which leads people to create one containing nothing useful, or worse, to copy a template containing directives they do not understand.

The two reasons that are actually worth it

First, declaring your sitemap. A Sitemap line means any crawler can find your sitemap without guessing, and unlike a Search Console submission it works for every crawler rather than only Google. One line, real benefit, no downside.

Second, keeping crawlers off paths that waste effort. Faceted search URLs, internal search results, endless calendar pages, print variants: these can consume a lot of crawling for no benefit. On a large site that matters. On a fifteen-page brochure site it does not.

Notice that neither reason is about keeping pages out of search results, because robots.txt does not do that.

• Worth it: declaring your sitemap in one line
• Worth it on larger sites: excluding faceted search, internal search, calendars
• Not what it is for: keeping pages out of search results
• Not what it is for: hiding anything, since the file is public

A minimal file that is genuinely fine

If you want one, keep it short. A file that allows everything and points at your sitemap covers the majority of sites completely:

User-agent: * then Allow: / then a blank line then Sitemap: followed by your full sitemap URL.

That is it. It does everything most sites need robots.txt to do, and it contains nothing that can go wrong later.

Resist copying a long template. Most circulating examples include directives for platforms you do not run, paths you do not have, and crawl-delay values that Google ignores anyway.

The risk of having one

This is the honest counterweight, and it is the reason not to add a file casually.

A robots.txt you do not understand can block things you needed crawled. The classic case is a Disallow on a directory that also holds your CSS and JavaScript, which prevents Google rendering your pages properly.

The other risk is inheritance. A file created during a rebuild, containing a blanket disallow appropriate to staging, goes live with the site. That single line is far more damaging than never having had a file.

So: no file is safe, a two-line file is safe, and a long inherited file you cannot explain is the risky option.

Common questions