A robots.txt file sits at the root of your domain and tells crawlers which parts of the site they may request. It is a crawl directive, not an indexing one, so a blocked URL can still show up in search results if other pages link to it. Use it to keep bots away from admin paths, faceted search, internal endpoints and duplicate parameter URLs that waste crawl budget, and use a noindex meta tag when a page must actually stay out of the index.
This generator writes the file for you. Pick whether robots are allowed or refused by default, add the directories you want kept out, then override individual crawlers where you need to. The per-bot list covers Googlebot, Googlebot-Image, Bingbot, Slurp, DuckDuckBot, Baiduspider, YandexBot, Twitterbot, AhrefsBot and SemrushBot. Sitemap lines and an optional crawl-delay are appended in the right places, and the output is ready to copy or download.
How to use the Robots.txt Generator
- Choose the default policy for all robots: allow everything, or disallow everything while you build the site.
- List the directories and files you want kept out, one path per line, starting each with a forward slash.
- Set any per-bot overrides, add your sitemap URL, and pick a crawl-delay if a specific crawler is hitting the server too hard.
- Click Generate, then Copy or Download and upload the file to the root of your domain so it answers at /robots.txt.
Frequently asked questions
Where does the robots.txt file have to go?
It has to sit at the root of the host and answer at https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain and each protocol needs its own file, so blog.example.com is not covered by the file on example.com.
Does Disallow stop a page from being indexed?
No. Disallow stops the crawl request, not the indexing. If other sites link to a blocked URL, Google can still list it without a snippet. To keep a page out of the index, allow the crawl and serve a noindex robots meta tag or X-Robots-Tag header.
Do all crawlers respect Crawl-delay?
No. Google ignores the directive entirely and asks you to set the crawl rate in Search Console instead. Bing, Yandex and several smaller crawlers do read it. Adding it is harmless, but it will not slow Googlebot down on its own.
Can I block bad bots with robots.txt?
Only the ones that choose to obey it. Robots.txt is a voluntary standard, so scrapers and malicious crawlers simply ignore it. Blocking those needs a server rule, a firewall or rate limiting. Naming a bot here works for well behaved commercial crawlers like AhrefsBot or SemrushBot.