Why it is invisible
The URL returns 200. Uptime monitoring is satisfied. Open it in a browser and you see a page rather than an error.
Meanwhile every crawler fetching it receives a document it cannot parse. Whatever your file was supposed to do, it is not doing it.
The direction of the failure matters. Because a missing or unreadable robots.txt is treated as permission to crawl everything, this usually fails open rather than blocking your site. That is the better direction, but it means your Disallow lines and your Sitemap directive have quietly stopped working.
What causes it
A framework catch-all route answering before any static file is served. This is the most common cause on single-page applications and modern frameworks: everything not explicitly routed returns the app shell.
A custom 404 page returning a 200 status. The file does not exist, the site renders its not-found page, and the status code says success.
A security or CDN layer serving a challenge page to anything that does not look like a browser. Crawlers get the challenge; you get the file.
A redirect to a human-readable page, occasionally added by someone who thought robots.txt should explain itself.
Confirming it
Do not judge by eye in a browser, because a browser renders whatever it is handed.
View source. A valid robots.txt is plain text: lines like User-agent, Disallow, Allow and Sitemap. If you see a doctype or html tags, that is the problem.
Checking the content-type header is the other reliable test. It should be text/plain. Anything reporting text/html is the same failure even if the body happens to look like text.
Our robots.txt tester reports this case explicitly rather than treating it as a parse failure, because the fix is completely different.
Fixing and preventing it
Work out what is actually answering that path. If a framework catch-all is responsible, you need an explicit route for robots.txt ahead of it. Most frameworks have a convention for this, and Next.js will generate the file from a robots definition.
If it is a 404 page returning 200, that is a bug worth fixing beyond robots.txt, because it affects every missing URL on the site and confuses crawlers generally.
To prevent a recurrence, add /robots.txt to whatever monitoring you already run, and assert on the content-type or the body rather than only the status code. A check that only looks for 200 will pass on a broken robots.txt indefinitely, which is how these last for years.
Common questions
How do I know if my robots.txt is returning HTML?+
View source rather than trusting the rendering, and check the content-type header. It should be text/plain and contain only lines like User-agent and Disallow. A doctype or html tags mean it is HTML.
Does this block my site from being crawled?+
No, it usually fails in the safer direction. An unreadable robots.txt is treated as permission to crawl everything, so the effect is that your Disallow lines and Sitemap directive stop working rather than your site being blocked.
Why does it look fine in my browser?+
Because the server returns HTTP 200 with a real document, just the wrong kind. Browsers render whatever they receive and monitoring that checks only the status code reports it as healthy.
How do I stop it happening again?+
Monitor /robots.txt and assert on the content-type or body, not just the status code. A 200-only check will pass on a broken file forever.
Sources