Free robots.txt Generator
Build a robots.txt file, decide which AI crawlers can read your site, and test any path before you upload it.
- Presets for GPTBot, ClaudeBot and 9 more
- Test a path before you upload
- Free, runs in your browser
Set each crawler (11)
Your robots.txt
User-agent: * Allow: /
Googlebot is allowed on /
Upload the file to the root of your site, so it opens at yoursite.com/robots.txt.
Blocking AI crawlers keeps you out of AI answers. Want to be in them instead? See AI search optimization
What robots.txt Can and Cannot Do
Robots.txt is a plain text file at the root of your site that tells crawlers which paths they may fetch. Well-behaved crawlers follow it; it is not a lock.
- User-agent
- Names the crawler a group of rules is for. * means every crawler without a group of its own. A crawler with its own group ignores the * rules entirely. Google: robots.txt
- Disallow and Allow
- Paths the crawler may not or may fetch. The longest matching rule wins, and Allow wins a tie, so you can open one file inside a blocked folder.
- Sitemap
- The full address of your XML sitemap, so crawlers find every page you want indexed.
- It does not remove pages from Google
- A blocked page can still appear in results if other sites link to it. To keep a page out of search, allow crawling and use a noindex tag instead. Google: robots meta tag
- Training bots and search bots are different
- GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot collect data for model training. OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot fetch pages to answer questions. You can block one kind and allow the other. OpenAI: crawlers
How to Create and Upload robots.txt
Five minutes, start to finish.
Choose the default
Allow the site for almost every business. Block only for staging or private sites.
List what to keep out
Checkout, cart, account, internal search and admin paths. Do not block CSS or JavaScript files; Google needs them to render the page.
Decide on AI crawlers
Allowing AI search bots lets ChatGPT, Claude and Perplexity read and cite your pages. Blocking training bots keeps your content out of future model training.
Test and upload
Test a few paths with the tester, download the file and upload it to your site's root folder. Check it opens at yoursite.com/robots.txt.
When You Want It Done for You
Let the right crawlers in, then give them something worth quoting.
AI search optimization
Get visible in AI search in under 90 days, or we work free until you get there.
OpenSEO services
The 90-Day Lead Engine: rank higher in 90 days or less, or we work free until you do.
OpenRobots.txt Questions
What is a robots.txt generator?
Where do I put robots.txt?
Should I block AI crawlers like GPTBot?
Does blocking Google-Extended remove me from AI Overviews?
Can robots.txt hide a page from Google?
Is an empty robots.txt OK?
Does Google follow Crawl-delay?
See Who AI Recommends Instead of You
Clean code helps engines read you. Our free AI audit tests your buyers' real questions across four engines, and it comes back in about a week.
- Your citation baseline across four engines
- Every question where a competitor is named and you are not
- Your real click loss to AI answers, in numbers
- An honest answer on whether you need us at all