robots.txt generator
One small file that tells crawlers where they may go. Easy to write, and easy to accidentally block your whole site with.
Your rules
Start from a preset, then adjust. The file builds itself on the right.
Your robots.txt
What robots.txt is, and is not
It is a plain text file at yoursite.com/robots.txt that asks crawlers not to visit
certain paths. Well-behaved crawlers obey it. Badly behaved ones read it as a list of interesting
places to look.
It is not security. Anything genuinely private needs a password. Putting
Disallow: /secret-admin/ in a public file is an advertisement.
The mistake that removes sites from Google
Those two lines block your entire site. They belong on a staging server, and they get copied to production more often than anyone likes to admit. It is one of the most common causes of "we relaunched and lost all our traffic".
Disallow does not mean deindex
This trips up a lot of people. If a blocked page has links pointing at it, Google can still list the URL in results — it just cannot see the content, so it shows a bare link with no description.
To actually keep a page out of search, let the crawler in and use a noindex meta tag.
Blocking it in robots.txt prevents Google from seeing the noindex tag, which does the opposite
of what you wanted.
The AI crawler question
You can block GPTBot, ClaudeBot, Google-Extended and the rest. Whether you should is a real decision, not an obvious one:
- Block if your content is your product and being trained on gives you nothing.
- Allow if you want to appear in AI answers that cite sources. That is becoming genuine referral traffic, and you cannot be cited if you are not readable.
Note these are different bots for different jobs. Google-Extended controls AI training
only — it does not affect your normal Google Search ranking at all.
Keep it boring
For most small sites the ideal robots.txt is three lines: allow everything, and point at your sitemap. Complexity here creates risk without much reward.
Common questions
Where does the file go?
The root of your domain, so it loads at yoursite.com/robots.txt. It will not work in a subfolder, and each subdomain needs its own.
Will blocking a page remove it from Google?
No. It stops Google reading the page, but the URL can still appear if other pages link to it. To remove a page properly, allow crawling and add a noindex meta tag.
Does Crawl-delay work?
Google ignores it. Bing and Yandex respect it. If Google is hitting your server too hard, set the crawl rate in Search Console instead.
Should I block AI crawlers?
It depends on what you want. Blocking protects your content from training use; allowing keeps you eligible to be cited in AI answers, which sends real visitors. There is no single right answer.
Do I even need a robots.txt?
Not strictly — no file means everything is allowed. Having one with a sitemap line is still worth the two minutes.
Want this checked on your real account?
A tool can only work with the numbers you give it. If you want someone to look at what is actually happening in your ad account, the first conversation is free.
Book a free consultation