The four methods and what each misses
Understanding the gaps matters more than the techniques, because it tells you when to use which.
| Method | Finds | Misses |
|---|---|---|
| Read the sitemaps | What the site declares matters | Pages not listed, and stale entries still listed |
| Crawl internal links | What is actually reachable | Orphan pages nothing links to |
| site: search | Roughly what is indexed | Unindexed pages, and it is only an estimate |
| Server or CMS listing | Everything that exists | Only available for your own site |
Why the sitemap is the fastest start
It is one or two HTTP requests and gives you the site's own statement of intent, which is genuinely informative regardless of completeness.
It is also the only method that works quickly at scale. Crawling a large site takes a long time; reading its sitemap index takes seconds.
The caveats are real though. A sitemap can be stale, so a listed URL may no longer exist. And a page absent from the sitemap may still be live and indexed. Treat the list as a claim rather than a census.
Combining two methods finds the interesting cases
The useful work is in the difference between methods, not in either list alone.
In the sitemap but not reachable by crawling: orphan pages. Your site asks search engines to index them while linking to them from nowhere.
Reachable by crawling but not in the sitemap: pages the sitemap is filtering out, often because of a plugin exclusion or a noindex tag applied more broadly than intended.
Indexed but in neither: usually old URLs still in the index after being removed, or parameter variants.
Any of those three gaps is worth investigating. A single list, however complete, tells you much less.
Common questions
What is the fastest way to list a site's pages?+
Read its XML sitemaps, which takes a couple of requests and returns the site's own statement of which pages matter. It is not complete, but nothing else is either.
Is a sitemap a complete list of pages?+
No. Pages can exist and be live without appearing in it, and a stale sitemap can list URLs that no longer exist. Treat it as a claim rather than a census.
How do I find orphan pages?+
Compare the sitemap list against what a crawl of your internal links finds. Anything in the sitemap but not reachable by crawling is an orphan.
Does a site: search show every page?+
No. It shows roughly what is indexed, as an estimate, and Google is explicit that the count is approximate. Unindexed pages will not appear at all.
Sources