Who honours it and who does not
Crawl-delay asks a crawler to wait a stated number of seconds between requests. It was never part of the original robots.txt standard, which is why support is inconsistent.
Google's own specification documentation states plainly that it does not support the crawl-delay field. Googlebot manages its own crawl rate based on what it judges your server can handle.
Bing has historically honoured it, as have several smaller crawlers. So the line is not useless, it is just aimed at the crawlers most people are not worried about.
The cost of a high value
This is why the line deserves attention rather than being harmlessly ignored.
A crawl-delay of 10 asks a crawler that respects it to fetch at most six pages a minute. On a site of a few thousand pages, a full crawl then takes many hours of continuous fetching, and crawlers do not fetch continuously.
The practical result is that new content takes far longer to be discovered by those crawlers, and on a large site some pages may effectively never be reached.
So a crawl-delay set years ago to protect a struggling shared host, on a site that has since moved to decent hosting, is now doing nothing but slowing your discovery on Bing.
| Crawl-delay | Pages per hour | Effect |
|---|---|---|
| Not set | Crawler decides | Normal, and what Google does regardless |
| 1 | 3,600 | Barely a constraint |
| 5 | 720 | Noticeable on a large site |
| 10 | 360 | New content is slow to be found |
| 30 | 120 | Effectively throttling yourself |
How to actually slow Google down
If Googlebot is genuinely straining your server, robots.txt is the wrong lever and crawl-delay will do nothing.
Google's guidance is to adjust the crawl rate through Search Console, which is the supported mechanism. It is intended for the case where crawling is causing real load problems rather than as a routine setting.
Before doing that, be reasonably sure it is Googlebot causing the load. Verify against your server logs and confirm the requests genuinely come from Google, since plenty of scrapers claim to be Googlebot in their user agent and none of them read robots.txt anyway.
In most cases the honest answer is that the hosting is undersized rather than the crawling being excessive.
Should you remove it
If you did not set it deliberately and cannot say why it is there, yes.
It has no effect on Google, so removing it costs nothing there. It removes a self-imposed limit on Bing and others, which is a small gain. And it eliminates a line that will confuse whoever reads the file next.
Keep it only if you have a specific known reason: a genuinely constrained server plus evidence that a respecting crawler was causing problems. That is a real situation, just an uncommon one.
Common questions
Does Googlebot respect crawl-delay?+
No. Google's robots.txt specification states it does not support the crawl-delay field, and Googlebot manages its own crawl rate instead.
Which crawlers do respect it?+
Bing has historically honoured it, along with several smaller crawlers. So the directive slows those without affecting Google at all.
Is crawl-delay bad for SEO?+
It has no effect on Google. A high value does slow discovery on crawlers that honour it, so an unnecessary crawl-delay is a small self-imposed handicap rather than a penalty.
How do I reduce Google's crawl rate?+
Through Search Console's crawl rate setting, which is the supported mechanism. It is meant for genuine server load problems, and it is worth confirming from your logs that the traffic really is Googlebot first.
Sources