What Is Robots.txt?
Robots.txt is a plain-text file that lives at the root of your domain (https://yourdomain.com/robots.txt). It follows the Robots Exclusion Protocol and contains instructions specifying which bots (user-agents) are allowed or disallowed from crawling which URL paths.
The file uses a straightforward syntax. User-agent specifies which crawler the rule applies to (* means all crawlers). Disallow specifies a path the specified crawler should not access. Allow overrides a Disallow for a more specific path. Sitemap entries at the bottom of the file point crawlers to your XML sitemap.
An important distinction: robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in Google's index if its URL is discovered through external backlinks. To prevent indexation, you need a noindex directive — but the page must be crawlable for Google to read that directive.
- Plain-text file at domain.com/robots.txt following the Robots Exclusion Protocol
- Controls crawler access to URL paths — does not prevent indexation directly
- A blocked-but-linked URL can still appear in the index as a URL-only result without content
How Robots.txt Affects SEO
Correctly configured robots.txt conserves crawl budget by directing Googlebot away from low-value URLs: admin pages, staging content, duplicate filter/parameter URLs, session IDs, and internal search results pages. This concentrates crawl budget on your important, indexable content.
Upload your Screaming Frog CSV — free, instant, nothing stored.
See your internal link equity in 30 seconds →Incorrectly configured robots.txt is catastrophic. Blocking Googlebot from your /products/ path during a staging test that gets deployed to production can de-index your entire product catalog within days. This is one of the most commonly reported SEO disasters.
Robots.txt also interacts with your internal link graph in a subtle way: if you link internally to URLs that are blocked by robots.txt, those links pass no PageRank (Google cannot crawl the destination to count the link from the other side) and waste crawl budget on requests that return no usable content.
- Proper robots.txt conserves crawl budget by excluding admin, staging, and parameter URLs
- Accidental Disallow on important paths is a critical risk — can de-index content within days
- Internal links to robots.txt-blocked URLs waste crawl budget and pass no PageRank
How to Audit Robots.txt
Fetch and read your robots.txt file at yourdomain.com/robots.txt. Verify that no Disallow rules block paths you want Googlebot to crawl. The most dangerous patterns are Disallow: / (blocks everything), overly broad path blocks, and rules targeting Googlebot specifically that differ from your general rules.
Google Search Console provides a robots.txt tester that shows which version of the file Google has cached and allows you to test specific URLs against the current rules. Use this tool to confirm that all important URL patterns are accessible.
Screaming Frog respects robots.txt by default and marks blocked URLs separately. Run a crawl and check the Response Codes report for URLs returning blocked-by-robots status. Cross-reference with your internal links to find which internal links point to these blocked URLs.
- Read robots.txt directly at domain.com/robots.txt and audit all Disallow rules manually
- Use GSC robots.txt tester to verify specific URL patterns and check Google's cached version
- Screaming Frog marks robots.txt-blocked URLs — cross-reference with inlinks to find wasted internal links
How to Configure Robots.txt Correctly
A minimal, correct robots.txt for most sites should allow all crawlers access to all paths and simply include a Sitemap directive pointing to your XML sitemap. Only add Disallow rules for paths that genuinely should not be crawled — never as a blanket block.
Common legitimate Disallow candidates: /wp-admin/, /cart/, /checkout/, /search?q=, /?s=, /feed/, /author/, and other non-content paths that do not add indexation value. Blocking these conserves crawl budget without sacrificing visibility.
Never use robots.txt to block pages you simply do not want to rank. Robots.txt blocks crawling — if the URL is already indexed, it will remain indexed. If you need to remove a page from the index, use a noindex directive on a crawlable page, or request removal via Google Search Console.
- Default to open access — only Disallow paths with no legitimate indexation value
- Common valid Disallow paths: admin, cart, checkout, internal search, author archives
- Never use robots.txt to delist already-indexed pages — use noindex or GSC URL removal instead
Upload your Screaming Frog CSV — free, instant, nothing stored.
Find your site's weakest links before Google does →Frequently Asked Questions
- What is robots.txt in SEO?
- Robots.txt is a plain-text file at domain.com/robots.txt that tells search engine crawlers which URL paths they are allowed or not allowed to access. It controls crawl access, not indexation — a blocked URL can still appear in the index if discovered through backlinks.
- Does robots.txt block indexing?
- No. Robots.txt blocks crawling only. If a URL blocked by robots.txt is linked to from external sites, Google may index it as a URL-only result without any content (since it cannot crawl the page to see a noindex directive). To block indexation, use a noindex directive on a crawlable page.
- What happens if robots.txt blocks Googlebot from my site?
- If Disallow: / is set for Googlebot (or *), Google will stop crawling all pages. Already-indexed pages remain indexed until their next scheduled recrawl, at which point Google will attempt to recrawl and, being blocked, will eventually remove them from the index.
- Should I disallow my admin pages in robots.txt?
- Yes. Disallowing /wp-admin/, /admin/, and other backend paths is best practice. These pages have no indexation value, and blocking them conserves crawl budget for your content pages. Ensure that assets used by the frontend (CSS, JS) are not inadvertently blocked.
Find Internal Links Wasted on Blocked URLs
Upload your Screaming Frog CSV to LinkJuice and identify every internal link pointing to a robots.txt-blocked URL — links that pass no PageRank and waste crawl budget.
Upload crawl & start🔒 Runs in your browser. Your data never leaves your machine. No email needed.