On this page
Definition
Robots.txt is a plain text file placed at the root of a website that gives instructions to search engine crawlers about which areas of the site they may or may not crawl. It can help manage crawler access, protect low-value sections from being crawled and point search engines towards the XML sitemap.
Robots.txt affects crawling, but it does not reliably remove a page from search results. For that, other methods such as noindex directives or proper removal processes may be required.
Key Takeaways
- Robots.txt tells compliant search engine crawlers which parts of a website they can crawl, but it does not guarantee that blocked URLs will disappear from search results.
- A small robots.txt mistake can prevent Google from crawling important pages, CSS files, JavaScript files, images or staging areas, which may affect indexing and search visibility.
- Robots.txt should be reviewed during website launches, migrations, SEO audits, staging setup and major site structure changes.
Quick explanation
What Does Robots.txt Do?
How it works
How Search Engines Use Robots.txt
The process is simple in principle, but the SEO outcome depends on how the file is configured and how the rest of the website is structured.
-
A crawler visits the website
When a compliant crawler arrives at a domain, it usually requests the robots.txt file before crawling other URLs. This helps it understand any crawl limits set by the website owner.
-
The crawler reads the matching rules
The file can include user-agent rules, disallow rules, allow rules and sitemap references. A user-agent rule identifies which crawler the instructions apply to.
-
The crawler decides what it may crawl
If a URL is disallowed, a compliant crawler should avoid crawling it. If a URL is allowed, the crawler may still choose when and how often to crawl it based on search engine systems, site quality, internal links and server response.
Simple Robots.txt Example
This example allows most of the site to be crawled, blocks a private admin path and points search engines to an XML sitemap. The exact rules should always match the website structure.
User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://www.example.com/sitemap.xmlExample only. Do not copy robots.txt rules without checking your own URL structure, CMS behaviour and SEO requirements.
Important Robots.txt Terms
Robots.txt uses a small set of terms, but misunderstanding them can lead to crawl and indexing problems.
- User-agent
- The crawler the rule applies to. For example, Googlebot is Google's web crawler. An asterisk means the rule applies to all compliant crawlers.
- Disallow
- A directive that tells a crawler not to crawl a specific path or URL pattern. It controls crawling, not necessarily indexing.
- Sitemap
- A reference to the website's XML sitemap, which helps search engines discover important URLs that should be considered for crawling and indexing.
SEO impact
How Robots.txt Can Affect SEO
Robots.txt Is Not a Security Tool
Robots.txt vs Noindex
Robots.txt and noindex are often confused, but they solve different SEO problems. Choosing the wrong one can create unexpected search results.
| Factor | Robots.txt | Noindex |
|---|---|---|
| Main purpose | Controls whether compliant crawlers may access a URL or path. | Tells search engines not to include a page in their index. |
| Access required | Search engines can read the rule before visiting the blocked URL. | Search engines usually need to crawl the page to see the noindex directive. |
| Common risk | A blocked page may still be indexed if Google discovers the URL elsewhere but cannot crawl the content. | If the page is blocked by robots.txt, Google may not be able to see the noindex directive. |
Common Robots.txt Mistakes
Many robots.txt issues come from small configuration errors made during development, migration or urgent site changes.
Blocking the whole site after launch
Do this instead
Staging sites are often blocked from crawlers, which is sensible. The mistake happens when those rules are copied to the live site. Always check robots.txt before and after launch.
Using robots.txt to remove indexed pages
Do this instead
If a page is already indexed, blocking it may stop Google from seeing removal signals. Depending on the case, a noindex directive, redirect, canonical tag or removal request may be more suitable.
Blocking resources needed for rendering
Do this instead
Search engines need to understand how a page renders. Blocking important CSS, JavaScript or image assets can make the page harder to evaluate.
Robots.txt SEO Troubleshooting
If rankings, indexing or crawling change after a launch or site update, robots.txt should be one of the first technical checks.
Important pages are not being crawled
Likely cause
A disallow rule may be blocking a folder, URL pattern or resource path that contains public pages.
Solution
Review the robots.txt file, test affected URLs in Google Search Console and confirm that important pages are crawlable.
Pages appear in Google with limited snippets
Likely cause
Google may know the URL exists through links, but the page content may be blocked from crawling.
Solution
Decide whether the URL should be indexed. If it should rank, allow crawling. If it should not appear, use the correct removal or noindex approach.
A staging or test site appears in search
Likely cause
Robots.txt may not have been enough, or the staging environment may have been publicly accessible without proper controls.
Solution
Use authentication or access controls for staging environments. Robots.txt can help guide crawlers, but it should not be the only protection.
Robots.txt Review Checklist
Use this checklist when launching a website, migrating platforms, reviewing technical SEO or diagnosing sudden crawl changes.
-
Confirm the file is accessible at the root of the domain
Check that the file loads at /robots.txt and returns the expected content. A missing file is not always a problem, but an incorrect file can be.
-
Test important page types
Check service pages, product pages, blog posts, location pages, category pages, images and essential resources. Make sure the rules match the pages you want search engines to crawl.
-
Review the file after launch or migration
Do not assume staging rules have been removed. Robots.txt should be checked during final testing, after DNS changes and again once the live site is confirmed.
Business context
When Should Robots.txt Be Reviewed?
Benefits and Limits of Robots.txt
What It Can Help With
- It can guide compliant crawlers away from low-value or duplicate sections of a website.
- It can help search engines discover the XML sitemap when the sitemap reference is included.
- It can support crawl management on larger or more complex websites.
What It Cannot Do
- It does not guarantee that a URL will be removed from Google's index.
- It does not secure private files, admin areas or staging environments.
- It does not replace technical SEO work such as redirects, canonical tags, internal linking, page speed and content quality.
Our approach
How We Approach Robots.txt in Technical SEO
Robots.txt FAQs
These answers cover common questions about robots.txt, crawling and SEO.
Can robots.txt stop a page from appearing in Google?
Should every website have a robots.txt file?
Can I edit robots.txt myself?
Need Help Checking Your Website's Crawlability?
If your website is not being crawled or indexed as expected, we can review the technical foundation, including robots.txt, sitemaps, redirects, indexing signals and site structure. Start a conversation and we will help you work out the next sensible step.