SEO Tools·5 min read

How Googlebot Sees Your Page (and What It Misses)

Search results are built from what the crawler fetched, not from what your visitors saw. Seeing your page the way Googlebot does starts with one blunt exercise: request the URL without running any scripts and read the HTML that comes back. The Search Engine Spider Simulator does exactly that. It strips the markup down to the plain text a search engine would extract, and checks the meta robots tag, canonical and robots.txt rules on the way. If your product copy isn't in that text, it isn't competing for anything. Ten seconds with the tool tells you whether your most important words ever reached the crawler at all.

Open the free Search Engine Spider Simulator All insights

The page you built and the page the crawler fetched

Same URL, two very different documents.

A browser does a lot of work on your behalf. It downloads the HTML, runs the JavaScript, applies the CSS, waits for API calls, and assembles the finished page you actually look at. A crawler's first pass skips almost all of that. It grabs the raw server response and moves on, which means the version of your page that gets evaluated first is the one before any script has run.

This gap surprises people constantly. A marketing site rebuilt in React can look identical to the old version in every browser while sending crawlers a nearly empty shell. Nobody notices until rankings slide, because nothing visible changed. The only way to catch it early is to look at the raw fetch yourself.

How Googlebot reads a page, in two waves

Googlebot works in two waves. The first wave fetches your HTML and indexes whatever text is already in it. Pages that need JavaScript to show content get queued for a second wave, where a headless Chromium renders them. That render can happen quickly or after a delay, and it costs Google resources, so thin or low-priority pages may wait.

Other crawlers are far less generous. Bing renders some JavaScript, many smaller engines render none, and several AI crawlers read only the raw response. The simulator deliberately mimics that strictest audience. Press Fetch as crawler and it requests your URL from its own server, removes scripts, styles, SVGs and iframes, and reports the words of text that survive. That number is your floor, the content every bot gets regardless of rendering ability.

One detail to keep in mind: the request uses the tool's own user-agent, not a Googlebot one. A site that serves different HTML to Googlebot may show the simulator another version. The Googlebot or Bingbot menu decides which robots.txt group and bot-specific meta robots tag are checked, not who the request claims to be.

A worked example: 1,400 words on screen, 58 in the HTML

Numbers from a client-rendered pricing page.

Take a SaaS pricing page built as a single-page app. In the browser it shows three plan cards, a comparison table, and an FAQ, roughly 1,400 words of visible copy. Run it through the simulator and the stat tiles tell a different story. Input: https://yourapp.io/pricing. Output: 212.4 KB HTML, 58 words of text. Because that is under 100 words, the tool also warns that the content is probably added by JavaScript. The Plain text tab reads something like: Pricing | YourApp Loading Sign in Get started Terms Privacy.

Every plan name, price, and feature row lived in JavaScript. After the team moved to server-side rendering, the same URL returned 96.1 KB HTML and 1,362 words of text, and the FAQ became readable to every crawler, not just the ones that render. Nothing about the design changed. Only the raw response did.

Content hidden from crawlers: where your text goes missing

Not every gap is as dramatic as an empty SPA shell. Content usually leaks away a piece at a time, and the Plain text, Headings and Links tabs make each leak obvious once you know what to look for.

JavaScript rendering explains most of it, but not all. Text baked into images has no HTML representation at all. Content loaded only after a scroll event or a click never appears in the initial response. Third-party embeds, reviews widgets, and comment systems inject their text from another domain, so it belongs to their response, not yours.

JS-injected copy: headlines and body text assembled client-side arrive as script tags, not words.
Lazy-loaded sections: anything fetched on scroll is absent from the first response.
Text in images: a hero banner saying Save 40 percent contributes zero words of text.
Third-party widgets: reviews and comments render from someone else's servers.
Script-driven navigation: links that only work on click, with no real a href, are missing from the Links tab.

Mistakes that starve crawlers of your strongest copy

Most indexing problems trace back to a handful of habits, and each one is visible in a raw fetch if you look.

Testing only in a browser. Right-clicking View Source is the honest check; the rendered DOM in DevTools is the flattering one.
Assuming Google's render pass is guaranteed. It runs when resources allow, often after a delay, and it does nothing for the bots that never render.
Shipping identical boilerplate on every page. When nav, footer, and cookie text outweigh unique copy, pages start looking like duplicates.
Blocking JS or CSS files in robots.txt. If the render wave can't fetch your bundles, even wave two sees the empty shell.
Ignoring HTML bloat. A 2 MB response full of inline SVG and framework markup slows crawling and buries your actual text.

Getting honest answers out of a spider simulator

Habits that make the tool earn its keep.

First, test your money pages, not just the homepage. Homepages are usually fine; the template pages generating your organic traffic, like product detail or article layouts, are where client-side rendering quietly eats content. One URL per template is enough.

Second, watch the ratio between KB HTML and words of text. A page shipping 300 KB of markup for 90 words of text is mostly framework overhead, and that ratio worsens as sites age. Third, rerun after every major frontend deploy. A single dependency upgrade can flip a component from server-rendered to client-rendered without anyone noticing, and the word count catches it the same day instead of a quarter later.

Fourth, read the Overview tab before you celebrate a good word count. A page can have perfect text and still stay out of search because of a noindex in its meta robots tag, a canonical that points to another URL, or a Disallow line in robots.txt. The tool flags the first two and shows whether robots.txt allows the bot you picked, with the matching rule and its line number. It also reads the HTTP status and any X-Robots-Tag header, because a noindex sent as a header is invisible in the HTML and is a common reason a page vanishes for no visible reason.

When the Search Engine Spider Simulator is the right tool, and when it isn't

The simulator answers one question: what text and markup does a bot receive from this URL? For neighboring questions, reach for the neighboring tools. To test many URLs against a robots.txt file, or check AI crawlers as well as Googlebot and Bingbot, use the Robots.txt Tester and Validator. If the fetched HTML looks fine but users land on errors, run the Broken Link Checker across the site to find dead destinations. And when the words are present but titles and descriptions look wrong in search results, the Meta Tag Analyzer inspects those tags in more depth than the Overview tab. Fetch first, then follow whichever number looks wrong.