How to Avoid Blocking CSS and JavaScript in Robots.txt

A page can open normally in your browser while its robots.txt rules prevent a crawler from requesting the stylesheet or script it needs. Start with the actual resource URLs, then review the rules that match them. A folder name such as /assets/ does not tell you whether its files are safe to block.

Find the files that the page needs

Choose one public page with missing content or an unexpected rendered layout. In browser developer tools, use the Network panel to identify its CSS and JavaScript requests. Record their complete URLs, including the host. Note which files provide the layout, navigation or main content rather than assuming every script serves the same purpose.

Google advises keeping resources crawlable when their absence would make a page harder to understand. Its robots.txt introduction explains this distinction. This does not mean every tracking request must be available for rendering.

Trace a broad folder rule to its effect

Consider a fictional shop at https://example.com/. Its public catalog uses /assets/site.css for layout and /assets/catalog.js to display product details. The owner has separately approved excluding its internal results under /search/. A draft contains:

User-agent: *
Disallow: /assets/
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

For a crawler following this group, /assets/ also matches the two required files. Removing that mistaken line leaves the separate search restriction intact:

User-agent: *
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

These examples describe only that fictional setup. They are not a complete replacement policy for your site. A sitemap line does not cancel a matching crawl restriction; see our guide to creating an XML sitemap for the separate discovery task.

Check the relevant host and crawler group

According to Google’s robots.txt specification, rules apply to the host, protocol and port serving that file. A script on a separate CDN needs its own host checked. Google also selects the most specific matching user-agent group; a Googlebot group is not combined with the wildcard group. Removing a line under User-agent: * will not fix a restriction in the applicable Googlebot group.

Reproduce the change in the generator

Use the Robots.txt Generator to compare two drafts without changing your server:

1. For this fictional example, select Allow all robots to crawl everything as the default policy. Leave every per-bot selector at Same as default and choose No crawl-delay.
2. Enter https://example.com/sitemap.xml in the sitemap field. In restricted directories, enter /assets/ and /search/, one per line.
3. Click Generate robots.txt. Compare the directives with the first example; the tool also adds dated comment lines.
4. Remove only /assets/ from the input, then click Generate robots.txt again. Confirm that Disallow: /search/ and the sitemap remain.

The generator writes text from your inputs. It does not fetch existing robots.txt files, identify required resources, test crawler access or upload anything. Its default-policy choice is part of this controlled example, not advice to open every path on a production site. Preserve and review all applicable existing rules before making a live change.

Verify the published rules and rendered content

Save the current file before an approved edit. After deployment, check the actual robots.txt response rather than relying on a local download. Confirm that the required resource URLs also return the expected CSS or JavaScript instead of a login page, error or firewall challenge.

For a property you manage, use Search Console URL Inspection to test the affected page. Review its rendered content, loaded resources and JavaScript errors. Google’s JavaScript troubleshooting guide describes these checks. Compare the main text and links with what a visitor should receive; a successful resource request alone does not prove the page renders correctly.

Keep access, indexing and security separate

Robots.txt controls cooperative crawling, not file permissions. It does not protect private files, and a blocked page URL can still appear in search results. For the different goal of excluding a page from indexing, read robots.txt versus noindex. Fixing an accidental resource restriction removes one possible obstacle; it does not guarantee indexing or a ranking change.