Your page looks complete. You opened it and read it. The crawler received something else entirely — a navigation bar, a footer, and a spinner where your content should be.

This is the single most expensive invisible fault we see, and it is entirely a product of how modern sites are built. A framework ships a nearly empty HTML document with a JavaScript bundle. The browser runs the bundle, calls an API, receives the product descriptions or article text, and writes them into the page. By the time a human looks, everything is there.

Whether a crawler sees the same thing depends on whether it did all of that work too. Some do. Many do not.

Why does my page look fine in the browser but not to crawlers?

Because your browser runs the JavaScript and the crawler may not. When you open the page, your browser downloads the shell, executes the scripts, fetches the data and assembles the finished page — and that finished page is the only version you ever see. The crawler may take only the first step. The gap between the two is invisible from the inside, which is why it survives for years.

It also survives review. The developer checks the page and it works. The marketing team checks the page and it works. The agency demonstrates it on a call and it works. Nobody in that chain ever looks at the document the server actually sent, because there is no everyday reason to.

Can Google read JavaScript content?

Google can render JavaScript, but it does so in a second pass that is separate from the initial crawl and subject to its own limits. Content that requires JavaScript is therefore indexed later than server-rendered content, and sometimes not at all when the rendering fails, times out or depends on a request that the crawler does not make. Text present in the HTML the server sends is indexed on the first visit, every time.

"Sometimes not at all" is doing a lot of work in that sentence. Rendering fails for ordinary reasons: a script blocked by a consent banner that the crawler never dismissed, an API call that requires a session, a timeout on a slow third-party dependency, a bundle that throws an error under a slightly different environment. Each failure is silent. The page stays in the index with whatever thin content survived.

Do AI crawlers execute JavaScript?

Generally no, or not reliably. Many AI retrieval crawlers read the HTML as it arrives from the server and do not run a full browser. A page whose main content is injected after load can therefore be perfectly indexed by Google and completely empty to the assistant that was about to cite it. This is one of the most common reasons a well-ranked page never appears in AI answers.

That combination — ranking fine, never cited — is worth recognising, because it looks like a content problem and is not. The page is relevant, the page is authoritative, the page is found. It simply arrives at the assistant as a shell. No amount of rewriting fixes it; the fix is in how the page is delivered. It sits alongside the other AI-visibility failures in how to get cited by AI search.

How do I check what a crawler actually sees on my page?

View the page source rather than the inspector — the source shows what the server sent, while the inspector shows the assembled result — and search it for a sentence you know is in your main content. If the sentence is missing, crawlers that do not run JavaScript will not see it either. TrustCtrl automates the comparison, fetching the server HTML and the rendered page on every crawl and flagging pages where the main content exists only in the rendered version.

Doing it by hand is a good way to convince yourself the problem is real. Doing it continuously is the only way to keep it fixed, because this regresses easily: a component gets refactored from server-rendered to client-rendered, a caching layer changes, a consent banner starts gating a script that was previously free to run. None of those changes look like SEO work, and none of them will be reviewed as such.

What to do about it

The direction of the fix is always the same — get the important text into the HTML the server sends — but the route depends on the stack:

  • Server-side rendering or static generation. The framework renders the page on the server and sends finished HTML. Every modern framework supports it; the question is whether it was switched on.
  • Prerendering for crawlers. A middle path where a rendering service serves finished HTML to bots. It works, but it needs care: serving crawlers something materially different from what visitors get is how accidental cloaking starts.
  • Move the critical text out of the client bundle. Often the pragmatic answer. Product descriptions, prices, headings and body copy in the HTML; the interactive parts — filters, carousels, configurators — left to JavaScript.
  • Check the tail, not just the homepage. Home and landing pages are usually server-rendered because someone cared about them. Product detail, category and article pages are where this hides, and they are the ones that carry the long-tail traffic.

The related trap: pages that say they are fine when they are not

While you are looking at what the server returns, there is a second failure worth checking, because it has the same shape. A soft 404 is a page that no longer exists but answers 200 OK with a "not found" message rather than a 404 status. To a visitor it reads as a dead end. To a search engine it reads as a real page with thin content, so it stays in the index and keeps being crawled.

We found this on our own site while testing TrustCtrl against it — along with a 403 being reported as a broken link when it was a working page behind access control, and a truncated list being mistaken for a total. All three were only visible with real data on a real site, which is a decent argument for scanning your own site with the same tool you sell.

TrustCtrl visits your pages in a real browser, up to the page limit set for the site, compares the server response with the rendered result, and reports the pages where your content only exists in one of them. Each finding comes with the exact evidence — the URL, the response, what was missing — in a Simple view for owners and a Technical view for whoever fixes it. It runs alongside sitemap accuracy, titles and descriptions, speed and accessibility, so one crawl answers the whole question.