The limit and what happens at it
Google's robots.txt specification states a maximum file size of 500 kibibytes, and that content after the maximum is ignored.
Note what that means: the file is not rejected. It is truncated. So the directives at the top continue to apply and everything past the cutoff silently does not, which is a worse failure mode than an outright error because nothing tells you.
If your file is anywhere near that size, the order of the directives suddenly matters a great deal, and anything you consider important should be near the top.
Why a file gets that large
500 KiB is an enormous amount of plain text, so hitting it always has a specific cause.
Almost always: listing individual URLs instead of using patterns. A site blocking ten thousand specific product URLs one line at a time will get there. The same intent expressed as a handful of wildcard patterns takes a few lines.
Occasionally: a plugin or generator appending rules automatically and never pruning them, so the file grows with every configuration change.
Rarely, and more legitimately: a very large site with genuinely complex crawl management. Even then, patterns usually collapse it.
Using patterns instead
Two wildcards do almost all the work, and knowing them removes the need for long lists.
An asterisk matches any sequence of characters, so a rule can cover an entire family of URLs in one line. A dollar sign anchors the match to the end of the URL, which is how you target file extensions.
So blocking every URL containing a particular query parameter, or every URL under a path, or every file of a given type, is one line each rather than one line per URL.
If you are maintaining a long list by hand, that is the signal to move to patterns regardless of the file size.
- Asterisk matches any sequence, so one line can cover a whole family of URLs
- Dollar sign anchors to the end, which is how to target file extensions
- One pattern replaces hundreds of individual Disallow lines
- Shorter files are also far easier for the next person to understand
Other limits worth knowing
The file must be at the root of the host. A robots.txt in a subdirectory is not read, which surprises people who put one in a folder expecting it to govern that folder.
Each subdomain needs its own. A robots.txt on the main domain does not govern a subdomain, so a blog or shop on its own subdomain is unaffected by the parent's file.
Protocol matters too: http and https are treated as separate origins for this purpose.
Most robots.txt confusion we see is one of these three rather than anything to do with size.
Common questions
What is the robots.txt file size limit?+
Google enforces 500 kibibytes and ignores content past that point. The file is truncated rather than rejected, so directives near the top still apply and later ones silently do not.
Why is my robots.txt so large?+
Almost always because it lists individual URLs instead of using wildcard patterns. Ten thousand specific Disallow lines can usually be expressed as a handful of patterns.
Does robots.txt have to be at the root?+
Yes. A robots.txt in a subdirectory is not read. It must be at the root of the host.
Does one robots.txt cover my subdomains?+
No. Each subdomain needs its own file, and http and https are treated as separate origins.
Sources