A robots.txt rule for /draft-guide can also match /draft-guidelines. If you intended to limit crawling of one address, that shared beginning makes the rule broader than it looks. Add an end anchor only after checking the exact URLs you want covered.
This exercise uses a fictional site whose owner has chosen to restrict crawling of https://example.com/draft-guide. The intended restriction covers that address without a trailing slash or query string.
Compare a prefix with an exact ending
Start with this broad rule.
User-agent: *
Disallow: /draft-guidePath matching begins at the start of the path. With this rule, additional characters after draft-guide can still match. Appending $ requires the match to end there. RFC 9309 defines that end marker in its special-character rules.
User-agent: *
Disallow: /draft-guide$Use the following comparison to review the intended scope. It assumes the shown rule is the only restriction in the group that applies to the crawler. A match means that this rule restricts crawling of the example URL.
| Path and query | /draft-guide | /draft-guide$ |
|---|---|---|
/draft-guide |
Matches | Matches |
/draft-guide/ |
Matches | No match |
/draft-guide/notes |
Matches | No match |
/draft-guidelines |
Matches | No match |
/draft-guide?view=print |
Matches | No match |
/Draft-guide |
No match | No match |
Account for the query string and trailing slash
The final dollar sign also excludes the query-string version in the table. A rule ending at draft-guide cannot consume the additional ?view=print text. Google documents this behavior alongside its case-sensitive path examples. See Google’s URL matching reference.
Copy the actual addresses from your site before deciding on the rule. If your intended URL ends with a slash, the corresponding exact rule is /draft-guide/$. If the crawl-control decision includes several query variants, list and review those separately. Removing the end anchor broadens the rule to all the matching prefixes again.
Generate the narrow draft
- Open the Robots.txt Generator. For this isolated example, select Allow all robots to crawl everything as the default policy.
- Leave individual crawlers at Same as default, select No crawl-delay and leave the sitemap field empty.
- Enter
/draft-guide$in Restricted directories and files, then select Generate robots.txt.
Ignoring the generated comments, the output matches the second two-line example above. The tool retains the end marker. It creates rule text from your entries, so use your URL comparison to assess the requested scope.
Check the complete policy before saving
Keep a copy of the existing robots.txt and merge the reviewed change into the appropriate group. Check which group applies to the crawler and read its other allow and disallow rules. Another matching rule can affect the outcome.
After deployment, fetch the robots.txt served by the intended host and compare its text with your approved draft. Keep the target and neighboring URLs from the table in your review record.
For a page you want excluded from search results, choose the indexing control deliberately. Google explains that a crawl restriction can leave a URL visible in Search. See the robots.txt introduction and our guide to robots.txt and noindex.