How to Create Robots.txt Rules for a Subdomain

Adding a restriction to the main website does not automatically give a subdomain the same crawling policy. Before writing a robots.txt rule, identify the exact host serving the URL. A file edited in the wrong place can leave the intended area unrestricted while changing an unrelated site.

This guide uses a fictional catalog at https://catalog.example.com/. Its owner wants cooperative crawlers to skip the catalog filter pages. The main website and its www version are separate checks.

Map each URL to its own robots.txt location

Google scopes robots.txt rules to the host, protocol and port where the file is served. Its robots.txt specification explicitly distinguishes a main domain from its subdomains. Start with a small inventory:

The repeated path does not make these the same URL. Nor does hosting the three names on one server automatically share their policy. Your server configuration might deliberately serve the same file for several hosts, but check each public response to establish that.

Keep the scheme in your record as well: an HTTP address is a separate scope from HTTPS. If the URL uses a nonstandard port, include that port when locating its robots.txt. Do not silently replace the observed address with a preferred homepage address.

Draft the restriction for the catalog host

Open the Robots.txt Generator and create a small comparison file:

1. Select Allow all robots to crawl everything under the default policy.
2. Leave the per-bot selectors at Same as default, with No crawl-delay.
3. Leave the sitemap field empty for this exercise. Enter /filter/ under Restricted directories and files.
4. Select Generate robots.txt.

Ignoring the dated comment lines, the generated directives are:

User-agent: *
Disallow: /filter/

For the intended catalog scope, these directives belong in the response at https://catalog.example.com/robots.txt. The path is relative to that host. It does not contain the subdomain name and does not select a destination server.

This is a demonstration draft, not a complete replacement for an existing policy. Preserve the current file and review its required groups and paths before merging an approved change.

Do not mistake URL cleanup for host selection

Replace the restricted-path input with https://catalog.example.com/filter/ and generate again. The current tool strips the scheme and host and produces the same Disallow: /filter/ directive.

That convenience does not bind the output to the catalog. If you put this output at https://example.com/robots.txt, its restriction concerns the main host instead. The catalog address entered in the field is no longer present in the generated rule.

Write the intended robots.txt destination beside your saved draft. The generator does not discover an existing file, upload to a host, or test crawler behavior. A correct-looking path alone cannot confirm correct placement.

Publish through the system serving that host

Use the catalog hosting or CMS configuration that actually produces its root robots.txt response. A physical text file and a CMS-generated response can have different editing routes. Check which one your server serves before changing a template or uploading a file.

Google’s file creation guide covers root placement and public testing. A file saved under /filter/robots.txt is not the catalog-wide robots.txt location. Save plain text in UTF-8. RFC 9309 defines the lowercase top-level /robots.txt endpoint and explains that these rules are not access authorization.

Use authentication for private information. For the separate goal of keeping a public page out of search results, review robots.txt versus noindex; a crawl restriction does not guarantee removal from search.

Verify the intended host and its neighbors

1. Open the catalog robots.txt URL in a private browser window. Check the requested URL, any redirect destination, response status and actual rule text.
2. Revisit the main and www robots.txt locations from your inventory. Confirm that deploying the catalog change did not accidentally alter their intended policies.
3. Review which crawler group applies to the target URL. Check the exact filter path against that group, rather than reading an unrelated group.
4. For a property you manage, inspect the accessible file through the Search Console robots.txt report. A local generator result is not evidence that Google fetched it.

If a robots.txt request redirects, investigate the response chain before drawing conclusions: seeing a different final hostname does not by itself identify the scope of the original request. Keep the requested URL and destination together in your review notes.