How this line gets onto a live site
Almost never on purpose. Three routes account for the large majority.
A staging or development site is correctly blocked from crawlers, then promoted to production with the robots.txt unchanged. This is by far the most common, and it can go unnoticed for months because nothing on the site looks broken.
A platform setting. WordPress has a discourage-search-engines checkbox, Shopify and other hosts have equivalents, and someone toggles it during a build and forgets.
A migration where robots.txt was copied from the wrong environment along with everything else.
What it actually costs you
Over time, close to all of your organic visibility, though not instantly and not in the way people expect.
Existing indexed pages do not vanish the moment the file changes. They persist for a while, then decay as Google stops being able to refresh them. New pages are never discovered at all.
Worth remembering from the related question: because robots.txt does not control indexing, some URLs may linger in results with no description rather than disappearing cleanly. So a site can look partly present in search while being entirely uncrawlable, which delays anyone noticing.
Telling deliberate from accidental
There are legitimate reasons for a blanket disallow, so check before you change anything.
It is correct on a staging or preview environment, on an internal tool that has no business being in search, and occasionally on a site mid-rebuild where the owner wants nothing indexed until launch.
It is not correct on anything you want found. If the site has a marketing homepage, product pages, or a contact form, a blanket disallow is a bug.
The quickest check: does this domain have any business appearing in search results? If yes, and the line is there, fix it today.
- Legitimate: staging, preview and development environments
- Legitimate: internal tools and admin applications
- Legitimate: a site deliberately withheld until launch
- A bug: anything with public marketing or product pages
Fixing it
Remove the Disallow: / line, or replace the whole file with something permissive. A minimal file that allows everything and declares your sitemap is enough for most sites.
Then check the platform setting, because if a checkbox is generating the file, editing the file by hand will be overwritten on the next save. On WordPress that is Settings then Reading, and the discourage-search-engines option.
Then request a fetch in Search Console so Google picks up the new file rather than waiting for its own schedule, and expect recovery to take weeks rather than days as pages are rediscovered and recrawled.
Common questions
What does Disallow: / mean?+
It asks the matching crawler to fetch nothing at all on the site. Under user-agent * it applies to every well-behaved crawler, making the whole site uncrawlable.
Will my pages disappear from Google immediately?+
No. Already-indexed pages persist for a while then decay as Google can no longer refresh them, and new pages are never discovered. Some URLs may also linger without descriptions, since robots.txt does not control indexing.
How did this get onto my live site?+
Usually a staging environment promoted to production without changing robots.txt, or a platform setting such as WordPress's discourage-search-engines option being left on.
How long does recovery take?+
Weeks rather than days. Google has to fetch the new robots.txt, then rediscover and recrawl your pages. Requesting a fetch in Search Console speeds up the first step only.
Sources