Sitemaps that tell the truth: why your XML sitemap is probably lying to Google
What a sitemap actually does
An XML sitemap is a list of URLs you hand to search engines with a claim attached: these pages exist, here’s where they are, and here’s roughly when they last changed. That’s it. It’s not a ranking lever. It’s not a guarantee of indexing. It’s a discovery aid, and its only job is to make crawling more efficient for a site that a crawler can’t fully map on its own.
I run sitemaps across a portfolio of content sites, and for years I treated mine like decoration. Generate it, drop it in robots.txt, forget it. The mistake wasn’t having a sitemap. It was never checking whether the thing was still telling the truth six months after I built it.
A sitemap that lies doesn’t get you penalized. It just gets ignored. Once a crawler learns that your lastmod dates are fake or that half your listed URLs redirect somewhere else, it starts trusting the file less, and you’ve spent effort producing a document nobody reads carefully anymore.
The lastmod lie
The most common mistake I see, including in my own early sitemaps, is a lastmod value that updates every single time the sitemap regenerates, whether or not the page content changed. A lot of CMS plugins and static site generators do this by default: the build script stamps every URL with the current timestamp because that’s easier to code than checking if the page actually changed.
The problem is that lastmod is supposed to be a signal for when to bother recrawling a page. If every page in your sitemap says “modified today,” that signal carries zero information. A crawler reading a file where 4,000 URLs all show the same timestamp on every visit learns quickly that the field means nothing on your site, and from then on it’s noise it can safely disregard.
The fix is unglamorous: pull lastmod from something real, like the file’s actual modification time or a updated_at column in your database that only changes when content changes, not on every deploy. If you republish a page with a typo fix, that’s a legitimate update. If you just redeployed the whole site because you changed a footer link, that’s not a content update for every page in the sitemap, even if your build process touches every file.
Priority and changefreq are dead weight
The sitemap protocol has two fields, priority and changefreq, that let you tell a crawler how important a page is relative to others on your site and how often you expect it to change. I used both religiously for years, hand tuning priority values on category pages versus blog posts, because it felt like I was giving the crawler useful instructions.
Google has said publicly, more than once, that it doesn’t use these fields for crawling or ranking decisions. That’s not inside information, it’s stated plainly in their own documentation and has been repeated by their search advocates for a long time. Other search engines vary in how much attention they pay to it. I stopped hand tuning priority and changefreq a while back and didn’t notice any change in crawl behavior, which lines up with what’s publicly stated.
I still set changefreq honestly where it costs nothing, because some crawlers outside the big two do reportedly still glance at it. But I don’t spend time optimizing these fields anymore, and I wouldn’t recommend anyone else does either. If you’re choosing between fixing your lastmod accuracy and tuning priority values, fix lastmod. It’s the field that actually gets used.
Only put pages you want indexed, full stop
This is the rule I broke most often without realizing it. A sitemap should contain exactly the canonical, indexable pages you want a search engine to know about. Nothing else.
In practice, sitemaps drift. A page gets noindexed after a content audit but stays in the sitemap because the sitemap generator doesn’t check the noindex tag. A URL gets 301 redirected to a new location but the old URL is still sitting in the file. A page gets canonicalized to a different URL, so the version in the sitemap isn’t actually the one you want indexed. Paginated URLs, filtered category views, and internal search result pages sneak in because the generator just crawls the site structure without checking canonical tags first.
Every one of these is a small credibility hit. When a crawler requests a sitemap URL and gets a redirect, a noindex, or a canonical pointing elsewhere, that’s a mismatch between what you claimed and what’s actually there. A handful of these on a large site is normal and not something to panic over. A sitemap where a meaningful chunk of URLs don’t resolve the way the sitemap implies is a sitemap that’s stopped being useful as a discovery tool, because the crawler can’t tell your accurate entries from your stale ones without checking each one manually.
The fix is to generate the sitemap from the same logic that decides what’s canonical and indexable, not from a separate crawl of your URL structure. If a page has a noindex tag or a canonical tag pointing elsewhere, it should never make it into the file in the first place. That’s a build-time check, not a “clean it up quarterly” task.
Split sitemaps by section, not by vibes
Once a site gets past a few hundred URLs, one flat sitemap file stops being useful for diagnosis. I split mine by content type or section: one sitemap for blog posts, one for category pages, one for product or tool pages, wrapped in a sitemap index file that references each one.
The reason isn’t crawl efficiency, it’s diagnostic. Google Search Console reports coverage and indexing status per sitemap file. If your blog posts are sitting in their own sitemap and you notice a chunk of them aren’t getting indexed, you know exactly where to look. If everything’s dumped into one file, you’re stuck cross referencing a giant URL list against the coverage report by hand.
This matters more than it sounds like on paper. I’ve caught indexing problems specific to one content type, tag pages that were thin and templated in a way that discouraged indexing, purely because that section had its own sitemap and its numbers stood out from the rest.
The size limits are real, and they’re not arbitrary
The sitemap protocol caps each individual file at 50,000 URLs or 50MB uncompressed, whichever comes first. That’s a documented spec limit, not a suggestion. If your generator ignores it and dumps everything into one file past that ceiling, some crawlers will simply refuse to process the excess, and you won’t get an error, you’ll just get silent gaps.
Sites well under that limit don’t need to think about this. But if you’re running a large directory, marketplace, or archive site, this is where the sitemap index structure earns its keep: split into files under the cap, reference them from an index file, and each chunk stays processable.
Dynamic beats static, eventually
Early on I generated sitemaps as static files during a build step and left them alone between deploys. That works fine for a site that changes rarely. It falls apart for anything with a CMS where content gets added, edited, or unpublished outside of a formal deploy cycle.
If your sitemap only refreshes when you run a build, and your content changes independently of that build, you’ve reintroduced the lastmod lie from a different angle: the file goes stale between regenerations, listing pages that no longer exist or missing pages that just went live. A sitemap generated dynamically from the same database that serves the pages, refreshed on a schedule or on publish, stays accurate without anyone remembering to run a script.
What I actually check now
On a recurring basis I spot check that lastmod dates vary and correspond to real edits, that a sample of sitemap URLs return 200 status with no noindex tag and no canonical pointing elsewhere, and that the sitemap index in Search Console shows a discovered count reasonably close to the submitted count. None of this is exciting work. It’s closer to bookkeeping than strategy. But a sitemap that accurately reflects your site is worth more than a bigger, messier one that doesn’t, because the whole point of the file is to be a source a crawler can trust without double checking everything in it.
If you want a second pair of eyes on your sitemap setup, or on the technical SEO underneath it, that’s the kind of unglamorous audit work we do at The SEO Desk.