How to use
Enter the page URL
Paste the full address, pick a crawler such as Googlebot, Bingbot or GPTBot in the menu, then press Fetch as crawler. Our server requests the page once, without running any scripts.
Check the Overview tab
Read the status code, X-Robots-Tag, title, meta tags, canonical, hreflang and schema types, and whether robots.txt allows the chosen bot. Warnings flag noindex, nofollow and a canonical elsewhere.
Compare text, headings and links
Open Plain text, Headings, Links and Images to see what exists in the HTML, and Raw HTML for the source. Anything you see in your browser but not here is added by JavaScript.
Why Search Engine Spider Simulator
- Shows status code, X-Robots-Tag, meta robots, canonical and robots.txt status in one view.
- Lists the headings and internal and external links in the raw HTML.
- Reports the text to HTML ratio and warns when a page has under 100 words.
- Copies the plain text or the full HTML source.
What Googlebot sees vs what users see
A browser downloads the HTML, runs the page's JavaScript and shows you the finished result. Googlebot works in two steps. It first fetches the raw HTML and reads the text and links in it. Rendering, where Google runs the JavaScript in a headless Chrome, happens later, after the page has waited in a render queue. Many other crawlers, including several AI crawlers, stop at the first step.
Take a React site built as a single-page app. In the browser the product page shows a heading, three paragraphs and a price. The raw HTML holds little more than a div with the id root and a script tag. Run it through the simulator and you see a word count close to zero, no H1 and hardly any links, which is exactly what a crawler that does not render will index.
- Server-side rendering or static generation, available in Next.js, Nuxt and similar frameworks, puts the content in the first response.
- Keep the title, meta description, canonical and main heading in the HTML, not injected by scripts.
- Use real a href links for navigation so crawlers can follow them without clicking.
Reading the Overview tab
Each row answers one question a crawler asks about the page. A problem in any of them can keep a page out of results even when the content is good.
| Field | What it tells you | What to look for |
|---|---|---|
| Title | The text search engines usually show as the headline | Present, unique, and roughly 50 to 60 characters |
| Meta description | A summary that may appear as the snippet | Present and specific to the page |
| Meta robots | Page-level instructions such as noindex or nofollow | No noindex on pages you want in search |
| Canonical | The URL you want indexed for this content | Pointing to this page, unless it is a duplicate |
| Language | The lang attribute on the html element | Matching the language of the content |
| robots.txt | Whether the chosen bot may crawl the URL | Allowed, unless you mean to block it |
| HTTP status | The server's answer and any redirect | 200 on pages you want indexed |
| X-Robots-Tag | Indexing rules sent as a header | No noindex on pages you want in search |
| Structured data | Schema types found in JSON-LD or microdata | The types you expect, such as Article or Product |