Free Robots.txt Generator and Tester
Create a clean robots.txt file with CMS presets, sitemap lines and AI crawler rules, then test any URL against it using Google's matching logic.
Robots.txt Generator and Tester: Create a clean robots.txt file with CMS presets, sitemap lines and AI crawler rules, then test any URL against it using Google's matching logic. It is free, needs no sign-up and runs in your browser. How to use it: 1) pick a starting preset and add your website URL; 2) edit the disallow and allow paths and choose rules for each crawler; 3) test a URL, then copy or download your robots.txt file. Choose a preset for WordPress, Shopify, Magento or a custom site, then edit the paths. The tool writes a group for all crawlers, adds separate groups for any bots you block or fully allow, and lists your sitemap URLs at the end.
How the robots.txt generator works
Choose a preset for WordPress, Shopify, Magento or a custom site, then edit the paths. The tool writes a group for all crawlers, adds separate groups for any bots you block or fully allow, and lists your sitemap URLs at the end.
How the tester decides
The tester follows the rules Google documents for robots.txt:
- A crawler uses the most specific user-agent group that matches its name, otherwise the * group.
- Within that group, the longest matching path rule wins.
- If an Allow and a Disallow rule are the same length, Allow wins.
- * matches any characters and $ marks the end of the URL.
For example, with Allow: /p and Disallow: /, the URL /page is allowed because /p is the longer match.
AI crawlers
You can block or allow GPTBot, OAI-SearchBot, Google-Extended, CCBot, PerplexityBot, ClaudeBot and Applebot-Extended one by one. Blocking training bots keeps your content out of future models, but blocking search bots can reduce how often AI search tools cite your business.
Limitations
Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use a noindex tag for pages you want removed. The tester does not decode percent-encoded characters, and Google ignores Crawl-delay.
About this tool
The Robots.txt Generator and Tester is built and maintained by the Hayyat Networks team, a digital marketing, SEO and web development agency working with businesses in New York and Lahore since 2016. It is part of our free SEO Tools collection.
- Last reviewed
- CostFree, no sign-up
- Where it runsRuns in your browser
- Checking resultsThe method and a worked example are shown above, so you can verify every result.
- PrivacyIt runs entirely in your browser: what you type stays on your device and is never sent to us.
Spotted a problem or have an idea? Tell us and we will review it.
Link to this tool
Free to share in your blog, course notes or resource list. Copy the link below:
<a href="https://hayyatnetworks.com/free-tools/seo-tools/robots-txt-generator/">Robots.txt Generator and Tester</a> by Hayyat NetworksRobots.txt Generator FAQs
Where do I put my robots.txt file?
It must sit at the root of your domain, for example https://www.example.com/robots.txt. Each subdomain needs its own file. On WordPress you can upload it by FTP or edit it with an SEO plugin. Shopify generates the file for you, so edit the robots.txt.liquid theme template instead.
Does robots.txt stop a page from being indexed?
No. It only stops crawling. If other pages link to a blocked URL, Google can still index the address without its content. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header to that page.
Should I block AI crawlers like GPTBot?
It depends on your goals. Blocking training crawlers keeps your content out of future AI models. Blocking AI search crawlers can reduce your visibility and citations in AI answers, which matters more and more for local and service businesses. Many sites block training bots but allow search bots.
Does Google respect crawl-delay?
No. Google ignores the Crawl-delay directive and adjusts its crawl rate automatically based on how your server responds. Bing and Yandex do respect it. If Googlebot is overloading your server, fix server capacity or temporarily return 503 or 429 status codes.
Want our experts to implement these improvements?
Get professional support from Hayyat Networks. Request a free consultation.