Fix tool · Free
AI Crawler robots.txt Builder
Decide which search engines, AI assistants and AI training crawlers can read your site, see in plain English what that means, and export a clean robots.txt.
robots.txt
How to use the result
Put your robots.txt live in three steps
Download and compare
Save the file as robots.txt. If you already have one, import it first so you keep your existing rules, then review the Plain English tab.
Replace the file at your root
Upload it so it opens at yourdomain.com/robots.txt. On WordPress, Shopify or Wix, use the built-in robots.txt editor or your SEO plugin instead. Each subdomain needs its own copy.
Verify what crawlers see
Run the AI Visibility Scanner to confirm each AI crawler can reach you, then add an llms.txt file so assistants describe you accurately.
Questions
Frequently asked
Should I block AI crawlers in robots.txt?
It depends on what you want. If you want ChatGPT, Claude and Perplexity to find and recommend you, allow the AI search and assistant crawlers. If you do not want your content used to train models, block the training crawlers such as GPTBot, ClaudeBot and Google-Extended. The Block AI training preset does exactly that while keeping you visible.
What is the difference between GPTBot and OAI-SearchBot?
Both belong to OpenAI. GPTBot collects content that may be used to train OpenAI models. OAI-SearchBot finds pages to show and cite in ChatGPT search. You can block GPTBot and allow OAI-SearchBot, so ChatGPT can still link to you without training on your pages. Anthropic and Google offer a similar split.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended is a control token, not a separate crawler. It only tells Google whether your content may be used to train and ground Gemini models. Googlebot still crawls and indexes your site for Search, including AI Overviews, as long as Googlebot is allowed.
Do all crawlers obey robots.txt?
Reputable crawlers from Google, Microsoft, OpenAI, Anthropic, Apple and Perplexity say they follow robots.txt. It is a public request, not a lock, so some bots ignore it, and fetchers that act for a person, such as ChatGPT-User and Perplexity-User, may treat it differently. To truly protect private content, put it behind a login. Never list secret paths in robots.txt, because anyone can read the file.
Where do I put robots.txt?
It must sit at the root of your domain, for example https://yourdomain.com/robots.txt, and each subdomain needs its own file. Add a Sitemap line so crawlers find every page. If you do not have a sitemap yet, build one with our Sitemap Generator.
How long until changes take effect?
Most crawlers cache robots.txt for up to a day, so expect changes to apply within 24 hours. Blocking a training crawler stops future collection only. It does not remove content that was already gathered. Run the AI Visibility Scanner afterwards to confirm what each crawler can see.