Small SEO Tools

Why a Googlebot Group Can Override Disallow All

A Disallow: / line under User-agent: * does not necessarily describe the rules Googlebot will use. A separate Googlebot group can give it a different policy. Read the applicable group before treating a default-policy dropdown as proof that every crawler is restricted.

Reproduce a default block with one exception

This controlled example compares two drafts. It is not a recommendation to block an existing public website. Open the Robots.txt Generator and select these settings:

  1. Set Default policy for all robots to Disallow all robots from everything.
  2. Leave every per-bot choice at Same as default.
  3. Choose No crawl-delay and leave the sitemap field empty.
  4. Enter /preview/ in Restricted directories and files, then select Generate robots.txt.

Ignoring the generated comment lines, this first draft contains:

User-agent: *
Disallow: /

Now change only the Googlebot per-bot choice to Allowed and generate again. The directives become:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /preview/

The current generator copies the restricted paths into an Allowed bot group. Its separate Allow: / line gives that group a different starting policy from the wildcard block.

Select the crawler group before matching a path

Google selects its most specific matching user-agent group. It does not merge a Googlebot group with the wildcard group. See Google’s group-selection reference.

For the second draft alone, compare these outcomes. ExampleBot represents a cooperative crawler with no matching named group in this file.

Crawler Path Applicable group Crawl permission
Googlebot /catalog/item Googlebot Allowed
Googlebot /preview/note Googlebot Disallowed
ExampleBot /catalog/item * Disallowed

Inside the Googlebot group, /preview/ is a more specific path match than /, so that restriction still applies. The unnamed crawler falls back to the wildcard group. RFC 9309 defines group selection and path matching.

Moving the wildcard block below the Googlebot block does not change Google’s group selection. Fix the intended rules rather than relying on file order.

Remove the unintended exception from the draft

If the intended example policy is a default crawl block with no Googlebot exception, change Googlebot back to Same as default and generate again. The separate Googlebot group disappears, leaving the first two-line draft.

Recheck both test paths. In that isolated draft, Googlebot now uses the wildcard block for each. If the exception was intentional, keep it and document that Googlebot may crawl the catalog path while the preview path remains restricted.

The generator creates text from your choices. It does not fetch your existing file, contact Googlebot or establish what a crawler has already fetched.

Compare the complete policy before deployment

Save the existing file and write down the intended outcome for each important crawler and sample path. Review all named groups in that complete file before merging a change. The minimal drafts above omit any unrelated production requirements.

After an approved edit, fetch the public robots.txt response and compare it with the reviewed version. Keep the generated draft and the served file together. For a separate path-scope problem, the exact-path matching guide explains how an end marker changes neighboring URL matches.

These are cooperative crawl instructions. RFC 9309 explicitly separates them from access authorization, so use actual access controls for private content.