A sitemap is a list of pages you are telling Google are worth indexing. Most of the sitemaps we scan contain URLs that redirect, no longer exist or carry a tag telling Google not to index them.

Sitemaps break in a specific way: silently, and always in the same direction. Pages get deleted, renamed, merged into others or set to noindex during a tidy-up, and the sitemap keeps listing them. Nothing errors. Nobody notices, because nobody reads a sitemap for pleasure.

What is an XML sitemap for?

An XML sitemap is a file listing the URLs on your site that you want search engines to crawl and index. It does not make pages rank, and search engines can find pages without it. What it does is speed up discovery of new pages and give the search engine a clear statement of which URLs you consider canonical — which matters most on large sites and on sites where internal linking is thin.

That framing is worth holding on to, because it sets the standard the file has to meet. A sitemap is a claim: these are my real pages, and they are worth your time. Its value comes entirely from being true. A sitemap that is half wrong is not half as useful — it is a source the search engine has learnt to discount.

Why does a sitemap with broken URLs matter?

Because the sitemap is a statement of intent, and contradicting it wastes the crawl budget you were trying to direct. Every URL that redirects, 404s or is marked noindex is a page you told the search engine to fetch and then told it to ignore. On a small site the waste is trivial; on a large one it means the pages you actually care about are crawled less often.

Three contradictions turn up again and again, and each has a different cause:

  • Listed but unreachable. The URL 404s, times out or errors. The page was deleted or renamed and the sitemap was not regenerated — or it was, from a stale cache.
  • Listed but set to noindex. The most self-contradictory of the three. The sitemap asks for indexing; the page's own tag refuses it. Usually the result of a noindex added for a good reason, on a page nobody removed from the list.
  • Listed but never crawled. The entry is present and the page is fine, yet the search engine has not fetched it. Often a sign the URL is blocked in robots.txt, orphaned from internal linking, or buried in a sitemap the search engine has stopped trusting.

TrustCtrl reports each of these separately, with the URL and the actual response, because the fix differs: the first is a regeneration problem, the second is a decision nobody finished, the third is usually about access rather than the sitemap at all.

How often should a sitemap be updated?

It should update automatically whenever pages are added, removed or moved — most content management systems do this, and a sitemap maintained by hand will drift within weeks. The lastmod date should reflect when the page content genuinely changed, because search engines use it as a freshness signal and a file where every entry claims today is a signal they learn to ignore.

That last point deserves emphasis, because the temptation runs the other way. Setting lastmod to the current date on every page looks like a free freshness boost. It is the opposite: a file where nothing ever differentiates itself carries no information, and a pattern of artificial freshness is precisely what search engines discount. Honest dates are more useful than flattering ones.

What is the difference between a sitemap and robots.txt?

A sitemap says which pages you would like crawled; robots.txt says which paths a crawler may fetch at all. They can contradict each other, and when they do, the block wins — listing a URL in your sitemap that robots.txt disallows achieves nothing except a warning in Search Console. Robots.txt is also where you should point search engines at the sitemap, using a Sitemap line.

While you are in robots.txt for that line, it is worth reading the rest of it properly. The same file decides whether AI assistants can fetch your pages at all, and the rules are matched per crawler — which is how a site ends up perfectly visible in Google and absent from every AI answer. That is the subject of are you blocking AI crawlers.

What Search Console adds

Google Search Console reports the sitemaps it knows about, when each was last read, and the errors it found. Three states are worth watching, and none of them announces itself: a sitemap Google has never successfully read, one it has not read for a long time, and one it read with errors.

Connect Search Console to TrustCtrl and those states are reported alongside what our own crawler finds in the file itself. The combination is what makes the picture complete: our crawler knows whether each listed URL is genuinely reachable and indexable, and Search Console knows whether Google is reading the file at all. Either half on its own leaves an obvious question unanswered.

A ten-minute check

Open your sitemap — usually at /sitemap.xml, and named in robots.txt if it is somewhere else. Pick ten URLs at random and open them. Count how many redirect, 404 or carry a noindex tag. If the answer is more than zero, the file is drifting, and the ten you checked are a sample of a larger number.

Then make the check repeat, because this is a maintenance problem rather than a one-off fix. TrustCtrl checks the URLs in your sitemap on each crawl and reports the ones that are unreachable, blocked from indexing or never crawled, next to titles and descriptions, content crawlers cannot see, speed, accessibility and security settings. One list, in plain language, with the technical detail for whoever regenerates the file.