A sitemap lists a website’s URLs to make them easier to discover. But listing pages doesn’t mean they’ll all be crawled or indexed by Google. Submitting a sitemap is still just a signal to search engines, not a guarantee. To get the most out of this file, you need to understand what it contains, what it leaves out, and how to compare it with other data to identify the pages that are actually accessible.
XML sitemap and HTML sitemap: two files, two purposes
Have you ever seen a web page that looks like a giant table of contents, with dozens of links organized by category? That’s an HTML sitemap. It’s designed for human visitors and helps them navigate when a menu isn’t enough.
An XML sitemap, on the other hand, is a technical file. It isn’t meant to be read by website visitors. It’s intended directly for crawlers such as Googlebot. Each URL is accompanied by information such as the date it was last modified (the lastmod field).
Here’s a real-world example. On a travel website, the 1001 Découvertes sitemap groups its pages in a structured file that search engines can crawl. This type of file helps crawlers find recently added or updated pages without having to explore the entire site link by link.
One technical point to remember: Google ignores the priority and changefreq fields in an XML sitemap. Filling them in won’t change how often a page is crawled. Only lastmod, when it reflects a genuine content update, can point crawlers to a recently updated page.

Compare your sitemap, internal links, and indexing report
Most guides stop at creating the file. The real work starts afterward, when you compare three data sources: the sitemap, the site’s internal linking structure, and the indexing report in Google Search Console.
Why comparing these sources makes all the difference
A URL can appear in your sitemap without being linked to from any other page on your site. Google discovers it, but doesn’t consider it a priority because no other page points to it. This is an orphan page that’s been flagged but poorly integrated.
Conversely, a page may be linked from your main menu but missing from the sitemap. Crawlers can still find it, but you lose the freshness signal that lastmod could provide.
The Search Console indexing report adds a third layer. It distinguishes between discovered pages (Google knows the URL), crawled pages (the crawler has read the content), and indexed pages (they appear in search results). These three statuses don’t always overlap.
How to identify discrepancies
A crawling tool such as Screaming Frog can extract all the URLs from your XML sitemap and compare them with those actually accessible through internal navigation. Here’s what this comparison can reveal:
- URLs in the sitemap that return a 404 error or a 301 redirect: they clutter the file and waste the crawler’s crawl budget.
- Strategic pages that are missing from the sitemap despite generating traffic: adding them strengthens the signal sent to Google.
- URLs in the sitemap that have a noindex directive: a contradiction that sends search engines a confusing message.
- Orphan pages with no internal links that appear only in the sitemap: they’re unlikely to remain indexed over time.
A reliable sitemap contains only accessible pages with no errors or conflicting directives. Fixing these inconsistencies often has a greater impact than adding new URLs.
Technical sitemap limits and organization for large websites
The Sitemap protocol imposes strict limits. Each XML file can contain no more than 50,000 URLs or 50 MB uncompressed. Beyond that, you need to split the URLs across multiple files and group them under a sitemap index.
For an e-commerce site with thousands of product pages, this organization isn’t a minor detail. A single overloaded file slows down crawling. Grouping URLs by category (products, blog posts, brand pages) also makes it easier to diagnose when a group of pages is no longer being indexed.
When an HTML sitemap complements an XML sitemap
For a smaller website, an HTML sitemap serves as a useful complement. It creates internal links to deep pages that aren’t highlighted in the main menu. Visitors who get lost in the site structure can use it as a shortcut.
On a larger website, an HTML sitemap quickly becomes difficult to manage. It’s better to focus on the XML sitemap and a well-planned internal linking structure, with contextual links placed in editorial content.

Check and maintain your sitemap over time
A sitemap that’s created once and never updated loses its value. Whenever a page is deleted, redirected, or set to noindex, the file should be updated to reflect the site’s actual state.
CMS platforms such as WordPress often generate sitemaps automatically. This convenience comes with a catch: auto-generated files sometimes include unnecessary pages (tag archives, pagination pages, drafts published by mistake). Checking the contents regularly helps prevent sending Google noisy signals.
After each sitemap update, submitting it through Google Search Console can prompt a faster crawl. The coverage report then shows the number of pages submitted, crawled, and indexed—three figures that tell the real story of your site from the search engine’s perspective.
A sitemap remains a communication tool between your website and crawlers. Its value depends entirely on how well it aligns with the site’s actual structure: internal links, page HTTP status codes, and indexing directives. A clean, up-to-date file that matches your internal navigation gives search engines the best conditions for crawling and indexing your content.




