Free Robots.txt Generator Tool
Create a valid robots.txt file to control how search engines crawl your website. Manage crawl budget, block private areas, prevent unwanted indexing, and improve technical SEO with standards-compliant robots.txt directives.
New to robots.txt? Start with our Robots.txt Guide for step-by-step guidance.
Build Your robots.txt File
Configure your crawl directives below. The preview updates automatically.
Robots.txt Preview
Updates automatically as you type
Example robots.txt files:
Start With a Template
Choose a preset for your CMS or website type
Learn More About Robots.txt
Deepen your understanding of robots.txt and how it affects search engine crawling and indexing:
Robots.txt Guide
Complete guide to creating and configuring robots.txt files.
Robots.txt Checklist
Step-by-step checklist for proper robots.txt implementation.
XML Sitemap Guide
Learn how sitemaps work alongside robots.txt for better discovery.
Crawl Budget Guide
Optimize how Google allocates crawling resources to your site.
Crawlability & Indexing Checklist
Ensure search engines can discover and index your pages.
Indexability Checker
Check if Google can crawl and index your pages.
Google Not Indexing Pages
Common causes and fixes when Google won't index your content.
How Search Engines Work
Understand crawling, indexing, and ranking to optimize effectively.
After Creating Your Robots.txt
Robots.txt generation is one step in the technical SEO process. After creating your file, complete these steps to maximize your search visibility.
Use this Robots.txt Generator to create a properly configured file.
Verify your rules with the Robots.txt Tester to ensure they work as expected.
Create a fresh XML sitemap to help search engines discover your pages. Sitemap guide →
Use the Indexability Checker and Crawlability Checklist.
Complete a technical website audit to find all SEO issues.
Use the Broken Link Checker to find and fix dead links on your site.
See how your pages appear in search results with the SERP Preview Tool.
Submit your sitemap URL in Google Search Console for faster discovery. Fix indexing issues →
What Is a Robots.txt File?
A robots.txt file tells search engines which pages and sections of your website can or cannot be crawled. It's a critical part of technical SEO that helps manage crawl budget and control search engine bots like Googlebot, Bingbot, and others.
Website owners use robots.txt rules to block private pages, prevent duplicate content indexing, and guide search engine bots toward important content. Proper crawl directives can significantly improve how search engines understand your site structure.
A properly configured robots.txt file can improve crawl efficiency, reduce unnecessary indexing of admin pages, login areas, and duplicate content, and help search engines focus their crawl budget on your most valuable pages.
Example robots.txt File
User-agent: *
Disallow: /admin/
Disallow: /login/
Allow: /
Sitemap: https://example.com/sitemap.xmlThis example blocks all bots from crawling /admin/ and /login/ folders, allows everything else, and points to the XML sitemap.
Robots.txt vs Meta Robots Tags
Many website owners confuse robots.txt with meta robots tags. While both control search engine behavior, they serve different purposes:
| Feature | Robots.txt | Meta Robots |
|---|---|---|
| Controls Crawling | ||
| Controls Indexing | ||
| Scope | File-level (entire site) | Page-level (individual pages) |
| When Applied | Before crawl | After crawl, before indexing |
| Google Support |
noindex meta tags to prevent pages from appearing in search results.
Can Robots.txt Prevent Pages From Being Indexed?
No. This is one of the most common SEO misconceptions.
Robots.txt blocks crawling, not indexing. If search engines find links to pages you've blocked in robots.txt, they may still index those pages without crawling them. This often results in:
- Pages appearing in search results with incomplete content
- Missing meta descriptions and titles
- Confusing user experiences from search results
noindex meta tags or X-Robots-Tag headers to prevent indexing, not robots.txt.
Learn about indexing issues →
noindex meta tag to prevent indexing.
Check your meta tags →
Common Robots.txt Rules for Popular CMS
| Path | Platform | Why Block |
|---|---|---|
/wp-admin/ |
WordPress | Admin login area, not needed for indexing |
/wp-includes/ |
WordPress | Core files, not needed for indexing |
/cart/ |
E-commerce | Dynamic cart pages, duplicate content risk |
/checkout/ |
E-commerce | Checkout pages, not useful in search results |
/search/ |
All sites | Search results create duplicate content |
/tag/ |
Blogs | Tag pages often create thin content |
/author/ |
Blogs | Author archives may not be useful for users |
/account/ |
All sites | User account pages, not needed in search results |
These are general recommendations. Always test your robots.txt rules before deploying to production. Review our Robots.txt Checklist before publishing.
Robots.txt and AI Search Crawlers
AI-powered search crawlers are becoming increasingly common. Just like traditional search engines, they check robots.txt files before crawling. Here's how common AI crawlers interact with robots.txt:
| Crawler | User Agent | Purpose | Robots.txt Support |
|---|---|---|---|
| Googlebot | Googlebot |
Search indexing | |
| Bingbot | Bingbot |
Search indexing | |
| GPTBot | GPTBot |
AI training (OpenAI) | |
| ClaudeBot | ClaudeBot |
AI training (Anthropic) | |
| PerplexityBot | PerplexityBot |
AI search (Perplexity) | |
| AppleBot | Applebot |
Search indexing (Apple) |
You can control AI crawlers just like traditional search engines by adding specific user agents to your robots.txt file. For example:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /This blocks OpenAI's GPTBot and Anthropic's ClaudeBot from crawling your entire site.
Robots.txt for ChatGPT, Claude and Perplexity
As AI crawlers become more prevalent, many website owners are asking how to control access. Here are the specific user agents:
GPTBot— Used by OpenAI for ChatGPT and AI trainingClaudeBot— Used by Anthropic for Claude AIPerplexityBot— Used by Perplexity AI for search
Use the same Disallow and Allow directives with these user agents to control AI crawler access to your content.
Search Engine User Agents Quick Reference
| Bot | User Agent |
|---|---|
Googlebot |
|
| Bing | Bingbot |
| OpenAI | GPTBot |
| Anthropic | ClaudeBot |
| Perplexity | PerplexityBot |
| Apple | Applebot |
| Yandex | YandexBot |
AI Search Visibility Best Practices
- Use Organization schema to help AI systems understand your brand. Generate schema →
- Implement FAQ schema for featured snippets and AI responses
- Submit your XML sitemap to help AI crawlers discover content. Generate sitemap →
- Use canonical tags to prevent duplicate content issues
- Consider AI crawler access as part of your content strategy
AI crawlers typically respect robots.txt directives, but always check their documentation for the latest guidance.
Robots.txt Best Practices
- Keep your robots.txt file small (under 500KB). Robots.txt checklist →
- Test your rules with a robots.txt tester tool
- Don't block CSS, JS, or image files unless necessary
- Use wildcards (*) carefully — they affect all bots
- Always include your sitemap location
- Regularly audit your crawl directives. Technical SEO checklist →
Common Robots.txt Mistakes
- Blocking Googlebot from your entire site
- Forgetting to test changes before deploying
- Blocking resources that affect page rendering
- Using disallow when noindex would work better. Learn about indexing →
- Not including a sitemap directive. Generate sitemap →
Frequently Asked Questions About Robots.txt
Everything you need to know about robots.txt files and search engine crawling
What does a robots.txt file do?
A robots.txt file tells search engine bots which pages or sections of your website they can or cannot crawl. It helps manage crawl budget and prevent unwanted content from being indexed. Robots.txt guide →
Can robots.txt block Google from indexing my page?
Robots.txt prevents crawling, not indexing. If other pages link to a blocked page, Google might still index it without seeing the content. Use noindex meta tags to prevent indexing. Learn about indexing issues →
Should I block my WordPress admin folder?
Yes, blocking /wp-admin/ is recommended to prevent crawlers from wasting crawl budget on login pages and admin areas that don't need indexing.
Where should I place my robots.txt file?
The robots.txt file must be placed in your website's root directory (e.g., example.com/robots.txt). Search engines automatically look for it there. Robots.txt checklist →
Does robots.txt affect SEO?
Yes, improperly configured robots.txt can block important pages from being crawled, hurting rankings. Correctly configured robots.txt improves crawl efficiency and technical SEO. Technical SEO checklist →
What's the difference between Allow and Disallow?
Disallow blocks crawling of specific paths. Allow permits crawling, which is useful for subdirectories within a disallowed parent folder. Allow overrides Disallow.
Should I block AI crawlers in robots.txt?
It depends on your goals. Most AI crawlers like GPTBot, ClaudeBot, and PerplexityBot respect robots.txt directives, allowing website owners to control access. Consider your content strategy when deciding. Robots.txt guide →
Related SEO Tools
Complete your technical SEO toolkit
After generating your robots.txt file, test it with our Robots.txt Tester or run a full SEO audit to identify other technical issues.