My Tool Studio
SEO

Robots.txt Generator

This robots.txt generator builds a valid robots.txt file from crawler groups you set up yourself. Start from a preset such as WordPress, WooCommerce, Shopify-style, Joomla, Drupal or Magento, or load a site's live file to edit it. Each group names one or more user-agents and holds its own Allow and Disallow rules and an optional crawl delay. Three switches block AI training crawlers, AI search and answer bots, or SEO tool crawlers as separate lists, so you can keep one and drop another. Add every sitemap you have, then copy, download or send the draft to the tester. A typical case: a shop launches with its cart and search pages open to bots. Pick the WooCommerce preset, block AI training crawlers only, test a few product URLs and upload.

Always freeNo sign upRuns in your browser

Replaces the rules in the User-agent: * group.

Crawler groups
Group 1

One or more tokens, comma separated. * means every other crawler.

Bot lists
AI training crawlers

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent

AI search and answer bots

OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User

SEO tool crawlers

AhrefsBot, SemrushBot, MJ12bot, DotBot, BLEXBot

Blocking training bots keeps your pages out of model training. Blocking search and answer bots can also keep you out of AI answers and citations.

Full URLs. Add each sitemap or sitemap index you have.

Written as # lines. Crawlers ignore them.

Generated robots.txt (5 lines, 110 bytes)
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

How to use

01

Pick a preset or load a file

Choose a preset from the menu to fill the User-agent: * group, or type a domain under Or edit a live robots.txt and press Load to start from the current file.

02

Adjust groups, rules and bot lists

Edit paths, switch a rule between Allow and Disallow, reorder rules with the arrows, or add a group for a crawler such as Bingbot. Set each bot list to Allow or Block and add your sitemap URLs.

03

Copy, download or test

Read any warnings above the output, then press Copy or Download and upload robots.txt to your domain root. Test this file opens the draft in the Robots.txt Tester first.

Why Robots.txt Generator

Example robots.txt files for common sites

Most sites need only a few lines. These are starting points; replace the paths with your own and test the result before you upload it. In every example, rules in the User-agent: * group apply to any crawler that does not have its own group.

Keep two things in mind. A crawler that finds its own group, such as GPTBot, ignores the * group completely. And a robots.txt file is a request, not a lock, so never rely on it to hide private pages.

SiteLines to use
Open site with a sitemapUser-agent: * ; Allow: / ; Sitemap: https://example.com/sitemap.xml
Online shopUser-agent: * ; Disallow: /cart/ ; Disallow: /checkout/ ; Disallow: /search ; Disallow: /*?sort=
WordPress blogUser-agent: * ; Disallow: /wp-admin/ ; Allow: /wp-admin/admin-ajax.php
Block AI training, keep searchUser-agent: GPTBot ; User-agent: CCBot ; User-agent: Google-Extended ; Disallow: /
Staging siteUser-agent: * ; Disallow: /

AI crawlers: training bots vs search and answer bots

AI companies run more than one crawler, and they do different jobs. Training crawlers collect pages to build future models. Search and answer bots fetch pages to cite them in AI search results, or when a user asks an assistant to read a link. Blocking the first group keeps your content out of training; blocking the second can also remove you from AI answers.

The generator keeps the two lists apart for that reason. Names change over time, so check each company's crawler documentation now and then.

CompanyTraining crawlerSearch or user-triggered bot
OpenAIGPTBotOAI-SearchBot, ChatGPT-User
AnthropicClaudeBotClaude-SearchBot, Claude-User
GoogleGoogle-Extended (a control token, not a separate crawler)Googlebot (regular Search)
PerplexityNone listedPerplexityBot, Perplexity-User
AppleApplebot-ExtendedApplebot
Common CrawlCCBotNone

Common questions

Does the Robots.txt Generator block AI crawlers like GPTBot?
Yes. Set AI training crawlers to Block and the file gets one group naming GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent with Disallow: /. AI search and answer bots such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot have their own switch, so you can stay visible in AI answers while opting out of training. Bots that ignore robots.txt are not stopped by any file.
Where does my finished robots.txt file have to live?
At the root of the host and nowhere else. Crawlers request /robots.txt on each hostname, so every subdomain needs its own copy. Upload the file so it loads at https://yourdomain.com/robots.txt.
How do I give Googlebot different rules from other crawlers in the generator?
Press Add a group for another crawler, type Googlebot in the User-agent box and add its rules. Googlebot then follows only that group and ignores User-agent: *, so copy any shared rules into it as well. You can list several crawlers in one group by separating them with commas.
When should I use an Allow rule in robots.txt?
Use Allow to reopen something inside a blocked folder. Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php is the classic case. For Google the longest matching rule wins, and Allow wins a tie, so the more specific Allow line takes effect.
What does the Crawl-delay field in the robots.txt generator do?
It adds a Crawl-delay line with the number of seconds you enter, inside that group only. Bingbot reads it as a pause between requests, while Googlebot ignores the directive completely. Leave it empty unless a specific bot is overloading your server.
Can I edit an existing robots.txt with this generator?
Yes. Type a domain under Or edit a live robots.txt and press Load. The tool reads the file, rebuilds its groups, rules, crawl delays and sitemap lines in the editor, and you change what you need. Comments in the original file are not carried over.
Will Disallow rules from this generator remove pages from Google?
Not by themselves. Disallow stops crawling, not indexing, so a blocked URL can still appear in results if other sites link to it. To remove a page from Google, allow crawling and add a noindex meta tag to the page instead.
How do I test a robots.txt file before I upload it?
Click Test this file. The draft opens in the Robots.txt Tester in paste mode, where you enter URLs, pick crawlers such as Googlebot or GPTBot, and see whether each URL is allowed or blocked. The draft is passed through your browser's local storage and is not uploaded anywhere.
Which robots.txt preset should I pick for WordPress?
Choose WordPress for a blog or brochure site. It blocks /wp-admin/, allows /wp-admin/admin-ajax.php that themes and plugins call, and keeps internal search results out. Choose WooCommerce for a shop; it adds the cart, checkout, account pages and sorted or filtered product lists. Check the paths against your own site before you upload.

More SEO tools

View all

Meta Tag Generator

Build meta, OG and X tags

SEO
URL-first

Meta Tag Analyzer

Audit meta tags for one or 20 URLs

SEO
URL-first

Robots.txt Tester and Validator

Test URLs against any robots.txt

SEO
URL-first

XML Sitemap Generator

Build an XML sitemap with images

SEO