A robots.txt file tells crawlers which parts of your site they may request. The Robots.txt Generator writes one from a preset and a few fields, and updates the file as you type, so you see every line before you copy it, download it or send it to the tester. There is no syntax to memorise.
Presets, groups and rules.
Start from a preset: Allow everything, Block everything, WordPress, WooCommerce, a Shopify-style store, Blogger, Joomla, Drupal, Magento or OpenCart. Each fills the User-agent: * group with the usual rules for that platform. To change an existing file instead, type a domain under Or edit a live robots.txt and press Load, and its groups, rules and sitemaps are rebuilt in the editor.
Each rule is an Allow or a Disallow for a path such as /cart/ or /*?sort=, and the arrows reorder them. Press Add a group for another crawler to give a bot such as Bingbot its own rules and its own Crawl-delay. Bingbot reads a crawl delay and Googlebot ignores it, so leave it empty unless a bot is overloading your server.
Add every sitemap URL you have, one per line. A relative path is flagged and left out. Warnings above the output also point out conflicting Allow and Disallow rules on the same path and a site-wide block.
Three switches, three lists.
Set AI training crawlers to Block and the file gets one group naming GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent, followed by Disallow: /. AI search and answer bots, such as OAI-SearchBot, ChatGPT-User and PerplexityBot, have their own switch, so you can opt out of training and still appear in AI answers. A third switch covers SEO tool crawlers such as AhrefsBot and SemrushBot.
Bots that ignore robots.txt are not stopped by any file. The switches cover crawlers that follow the rules.
The step people skip.
Click Test this file and the draft opens in the Robots.txt Tester and Validator in paste mode. Enter real URLs, pick crawlers such as Googlebot or GPTBot, and check that each is allowed or blocked as you meant. The draft is passed through your browser's local storage, not uploaded.
One exact location.
Upload the file so it loads at https://yourdomain.com/robots.txt. Crawlers only look at the root of each host, so a file in a subfolder is ignored and every subdomain needs its own copy.
On WordPress, uploading a real file replaces the virtual one WordPress serves, or you can paste the text into your SEO plugin's robots.txt editor. The WordPress preset already blocks /wp-admin/ and adds Allow: /wp-admin/admin-ajax.php, which themes and plugins call. Check any preset's paths against your own site before you upload.
A common misunderstanding.
Disallow stops crawling, not indexing. A blocked URL can still appear in Google if other sites link to it. To keep a page out of results, allow crawling and add a noindex meta tag to the page instead.
The same applies to the Block everything preset, which writes User-agent: * and Disallow: /. Use it for staging or test sites, and protect those with a password as well.
Try it now
The tool is one click away. No sign up, no upload, no payment.
Open Robots.txt Generator