My Tool Studio
SEOURL-first

Robots.txt Tester and Validator

This robots.txt tester checks whether a URL is allowed or blocked for the crawler you pick, and shows the exact rule that decided it. Load a site's live robots.txt or paste a draft, list the URLs or paths you care about, then tick Googlebot, Bingbot, GPTBot or any of the other bots. Each result shows Allowed or Blocked and the rule that decided it, highlighted in the file view. The same screen works as a robots.txt validator, listing syntax problems such as rules placed before any User-agent line, and it can test the scripts, CSS and images a page loads. SEOs reach for it when Search Console reports a page as blocked by robots.txt and nobody can see why. Paste that URL, pick Googlebot, and the line responsible lights up.

Always freeNo sign upRuns in your browser

Full URLs or paths like /blog/post?page=2. Matching is case sensitive and includes the query string.

User-agents
Search engines
AI crawlers

Load a site's robots.txt or paste one to see results. Press Try sample to see how it works.

How to use

01

Load or paste the file

Choose Fetch from a site, enter a domain and press Load. Or choose Paste robots.txt to test a draft. A fetched file appears in an editable box, so you can try a fix before changing the live file.

02

Enter URLs and pick bots

Type one full URL or path per line, up to 100. Tick the crawlers to test, such as Googlebot, Bingbot or GPTBot, or type another user-agent token like AhrefsBot in the box below.

03

Read the verdicts

Each cell shows Allowed or Blocked with the matching rule. Select a cell to highlight that line in the file, check the AI crawler row and the syntax list, and download the results as CSV.

Why Robots.txt Tester and Validator

How Google picks the winning rule

First, the crawler finds its group: the User-agent line that names it most specifically, or the * group if nothing names it. Only the Allow and Disallow rules in that group count. Next, every rule whose path matches the start of the URL path, query string included, is a candidate. The candidate with the longest path wins. If an Allow rule and a Disallow rule are the same length, Allow wins. If no rule matches, the URL is allowed.

These examples come straight from that logic and match the results this tester shows.

Rules in the groupURL testedResult
Allow: /p ; Disallow: //pageAllowed. /p is longer than /
Allow: /folder ; Disallow: /folder/folder/pageAllowed. Same length, so Allow wins
Allow: /page ; Disallow: /*.htm/page.htmBlocked. /*.htm is the longer rule
Allow: /$ ; Disallow: //Allowed. /$ matches only the homepage
Allow: /$ ; Disallow: //page.htmBlocked. Only Disallow: / matches
Disallow: /admin/ ; Allow: /admin/help//admin/help/faqAllowed. /admin/help/ is longer

Robots.txt directive and wildcard reference

A robots.txt file is plain text with one directive per line, written as a name, a colon and a value. Names are not case sensitive; paths are. Lines starting with # are comments. Google reads only the first 500 KiB of the file.

DirectiveExampleWhat it does
User-agentUser-agent: GooglebotStarts a group for that crawler. * covers crawlers without a group of their own.
DisallowDisallow: /admin/Blocks /admin/ and everything under it, such as /admin/users.
AllowAllow: /admin/help/Reopens a path inside a blocked folder, when it is the longer match.
Empty DisallowDisallow:Blocks nothing, the same as allowing everything.
* wildcardDisallow: /*?sort=Any path followed by ?sort=, such as /shoes?sort=price.
$ end anchorDisallow: /*.pdf$URLs ending in .pdf. /file.pdf?v=2 does not match.
SitemapSitemap: https://example.com/sitemap.xmlPoints crawlers to your sitemap. Needs a full URL and applies to every group.
Crawl-delayCrawl-delay: 10Seconds between requests for Bingbot. Googlebot ignores it.

Common questions

Why does Search Console say 'Blocked by robots.txt' when my rules look fine?
Usually a rule matches more than you expect. Disallow: /p also blocks /products and /pricing, because rules match from the start of the path. Another common cause is a Googlebot group, which makes Google ignore everything you wrote under User-agent: *. Paste the blocked URL here, tick Googlebot and the tool names the exact line.
Does Disallow in robots.txt stop a page from being indexed?
No. Disallow stops crawling, not indexing. Google can still index a blocked URL without its content if other pages link to it. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header, because Google cannot see a noindex on a page it is not allowed to fetch.
How do * and $ wildcards work in robots.txt rules?
The * matches any run of characters, including none, so Disallow: /*.pdf blocks every URL with .pdf anywhere in it. The $ marks the end of the URL, so Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?download=1. Google and Bing support both, while some smaller crawlers treat them as plain characters.
Are robots.txt paths case sensitive?
Yes. Disallow: /Admin/ does not block /admin/, and this tester matches case exactly as Google does. Directive names such as Disallow and user-agent names such as Googlebot are not case sensitive.
How do I block GPTBot and other AI crawlers but keep Googlebot?
Add a group for the AI crawlers with Disallow: /, for example User-agent: GPTBot on one line and Disallow: / on the next, and leave your User-agent: * group as it is. Googlebot never reads the GPTBot group, so it keeps its access. Paste the file here and tick both GPTBot and Googlebot to confirm one is blocked and the other allowed. Blocking Google-Extended stops AI training use of your content without affecting Google Search.
What happens if robots.txt returns a 404 or a 5xx error?
A 404 or any other 4xx response except 429 means there are no rules, so crawlers may fetch everything. A 5xx or 429 response is treated differently: Google reads it as a temporary block on the whole site and holds back crawling until the file loads again. When you load a site here, the tool tells you which case you hit.
How long does Google take to pick up robots.txt changes?
Google generally caches robots.txt for up to 24 hours, so a change can take a day to apply. If you need it sooner, the robots.txt report in Search Console lets you request a recrawl of the file. Test the new version here first so the fix is right the first time.
Is this a replacement for the old Search Console robots.txt Tester?
It covers the same job. Google retired that tester and replaced it with the robots.txt report, which lists the files Google fetched and their errors but does not test URLs. This tool fetches any public robots.txt, or takes a pasted one, and tests URLs for Google and other crawlers. It follows Google's documented rules but is not Google's own code, so treat the URL Inspection tool in Search Console as the final word.
Which group does a crawler use when robots.txt has several User-agent lines?
A crawler obeys only the group that names it most specifically and ignores all others. Googlebot uses a User-agent: Googlebot group when there is one, and falls back to User-agent: * when there is not. Googlebot-Image falls back to the Googlebot group first. Several groups naming the same crawler are merged into one.
Can the robots.txt tester check whether my CSS and JavaScript are blocked?
Yes. Under Check a page's resources, enter a page URL and press Check resources. The tool fetches the page, lists every script, stylesheet, image and iframe it loads, and tests each one for the first crawler you ticked. Files on other hosts, such as a CDN, are tested against that host's own robots.txt. Blocked CSS or JavaScript can stop Google rendering the page the way visitors see it.

More SEO tools

View all

Meta Tag Generator

Build meta, OG and X tags

SEO
URL-first

Meta Tag Analyzer

Audit meta tags for one or 20 URLs

SEO
URL-first

Robots.txt Generator

Build a robots.txt with presets

SEO

XML Sitemap Generator

Build an XML sitemap with images

SEO