Use robots.txt when your aim is to control crawling. Use noindex when a public page should stay out of Google Search. The distinction matters: blocking a page in robots.txt can stop Google from reading its noindex instruction.
Start by writing down the outcome you want for one specific URL. A search-results page, a private customer document and a useful product guide need different treatment. The examples below help you choose a control before using the Robots.txt Generator.
Choose the control that matches the problem
- Reduce crawling of a URL area: consider a robots.txt rule, after checking what else uses those paths.
- Keep a public HTML page out of Google Search: use noindex while allowing Google to crawl the page.
- Protect private information: require authentication or another real access restriction. Search directives do not protect a document from visitors.
A robots.txt restriction can leave a URL visible in search when Google discovers links to it elsewhere. That is why a crawl block is unsuitable as a reliable removal instruction. Google explains the purpose and limits of robots.txt, including why it is not a way to secure private material.
When noindex is the appropriate choice
Imagine a public event confirmation page that visitors need after registration, but that has no useful purpose as a search result. If it contains no private details and should remain publicly accessible, a noindex instruction may suit that goal.
For an HTML page, this instruction belongs in the document head:
<meta name="robots" content="noindex">A CMS or SEO plugin may provide a page-level setting that generates the tag. Check the delivered page source after saving; an editor checkbox alone does not confirm what visitors and crawlers receive.
Google must be able to fetch the page to see this instruction. Do not simultaneously block that URL in robots.txt and expect the new noindex tag to be processed. Google also supports an HTTP response header for noindex, which is useful for resources such as PDFs. Adding a Noindex: line to robots.txt is not a supported substitute. See Google’s noindex implementation guide.
If the page is already indexed, allow time for another crawl and processing. Repeatedly changing between incompatible controls makes troubleshooting harder.
Generate a small robots.txt example
Suppose a demonstration site has chosen to limit crawling beneath /search/. This example is about crawl control; it does not promise that search-result URLs will disappear from Google.
- Open the Robots.txt Generator.
- Leave Default policy for all robots set to Allow all robots to crawl everything.
- Leave Crawl-delay at No crawl-delay.
- Enter
/search/in Restricted directories and files. - For this demonstration, enter
https://example.com/sitemap.xmlin Sitemap URL. - Leave the individual bot settings at Same as default, then select Generate robots.txt.
Ignoring the generated comment lines, the output is:
User-agent: *
Disallow: /search/
Sitemap: https://example.com/sitemap.xmlReplace the demonstration sitemap address with your actual sitemap before using a file. The generator writes the text you request; it does not check your sitemap, upload a file or test the rules against your live site.
Review the scope before putting rules live
In this example, /search/ matches paths beginning with that exact prefix, such as /search/shoes. It does not match /search without the trailing slash or a homepage query such as /?s=shoes. Inspect your actual URLs instead of assuming every search feature uses the same path.
Paths are case-sensitive. Google does not support the crawl-delay field, even though the generator offers it for crawlers that may support it. Existing bot-specific groups can also change which rules apply. Google’s robots.txt specification documents matching and group selection.
Keep a copy of the existing robots.txt and compare it with your proposal. Preserve required rules instead of replacing a working file with a minimal example. A file containing Disallow: / under the applicable group blocks crawling across the site, so review that setting especially carefully.
Check what the website actually serves
- For robots.txt, open the file at the root of the intended host, for example
https://example.com/robots.txt, in a private browser window. - Confirm it returns your intended plain-text rules, rather than a login screen or HTML error page. Use UTF-8 when saving the file.
- For noindex, inspect the affected page source or response headers and confirm that crawling is allowed.
- Use the relevant Search Console reports to investigate what Google has fetched; a local generator preview cannot establish that.
Google’s file creation and testing guide covers placement and testing. Once an important page is accessible and eligible for indexing, review its search presentation with these practical title-tag examples. A title improvement cannot resolve a crawl block or a noindex instruction.