What Is Robots Txt? Rules, Examples & SEO
What is robots txt in SEO? Learn how it controls crawling, what it cannot do, key examples, and how to audit it at scale.
What is robots txt? It is a plain text file at a site’s root that tells crawlers which paths they may or may not crawl. It controls crawling, not indexing or security. Search engines can still index blocked URLs from other signals, and private content still needs authentication.
Robots.txt is part of the Robots Exclusion Protocol. The formal protocol is defined in RFC 9309, which specifies the file location, supported groups, and matching rules. That matters because many articles treat robots.txt as folklore when it is now a documented standard.
What is robots txt in SEO#
In SEO, robots.txt helps manage crawl access. It can reduce wasted crawl activity on low-value URLs, protect staging sections from accidental crawling during development, and point crawlers to your XML sitemap.
It does not guarantee that a page stays out of search results. Google can still index a URL it discovers elsewhere, even if crawling is blocked, which is why robots.txt is different from noindex and from authentication. If you want to audit this quickly for one page or many, YepAPI’s On-Page API is the direct way to fetch live page-level SEO signals in code.
How robots.txt works#
A crawler requests https://example.com/robots.txt before crawling other paths on that host. The file must live at the site root for that host. A robots.txt file in a subfolder does not control the whole site.
The parser reads groups of rules by user-agent. The standard supports User-agent, Disallow, and Allow, while sitemap lines are widely used in practice. Google’s robots testing workflow lives in Search Console, which is the practical place to validate whether a rule blocks Googlebot as intended.
Here is the basic flow:
- The crawler fetches
/robots.txtfrom the host root. - It finds the matching
User-agentgroup. - It compares the requested path against
AllowandDisallowrules. - It crawls allowed URLs and skips blocked ones.
- It may still index blocked URLs if it found them through links, sitemaps, or other references.
That last step is the one most teams miss. Crawling and indexing are related, but they are not the same process.
Robots txt example by task#
Task-based examples are easier to implement than syntax dumps.
- Allow everything
User-agent: *
Disallow:- Block an admin area
User-agent: *
Disallow: /admin/- Block all crawling during development
User-agent: *
Disallow: /- Block a folder but allow one file inside it
User-agent: *
Disallow: /private/
Allow: /private/public-file.pdf- Declare your sitemap
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlThe third pattern is the classic technical SEO best practices mistake when it ships to production by accident. It is useful on staging. It is disastrous on a live site.
Robots txt disallow all: when to use it#
Disallow: / tells a crawler not to crawl any path on that host for the matching user-agent. Use it only in tightly controlled cases such as staging, temporary prelaunch environments, or short maintenance windows.
Do not use it as a security control. Anyone can still request those URLs directly if the server allows it. If the content is private, protect it with authentication or other access control.
Google robots txt: what Google supports in practice#
Google follows robots.txt, but only within the scope of crawling. It does not treat robots.txt as a noindex feature for long-term exclusion. For that, use a crawlable page with noindex, or apply an X-Robots-Tag header where appropriate.
The standard in RFC 9309 defines path matching and group handling. Google’s implementation also supports common pattern usage such as wildcard matching and end-of-line matching in its own parsing behavior, which is why testing matters more than guessing. When you need to inspect how a page itself is configured, YepAPI’s On-Page Content API helps you pull meta robots and other on-page signals programmatically.
Robots.txt is not indexing control or security#
This is the biggest implementation mistake.
Use this comparison to choose the right control:
| Need | Right tool | Why |
|---|---|---|
| Stop crawling a path | robots.txt | It tells crawlers not to fetch matching URLs. |
| Keep a page out of search results | meta robots or X-Robots-Tag | These control indexing when crawlers can access the content. |
| Protect private content | Authentication or authorization | Access control stops users and bots at the server. |
| Tell crawlers where sitemaps are | robots.txt sitemap line | It is a discovery hint, not an indexing command. |
Schema.org documents the noindex value for robots directives in its metadata vocabulary at schema.org. For site owners, the practical rule is simple: if a page must not appear in search, robots.txt alone is not enough.
Common robots.txt failure modes#
These are the issues worth checking first:
1. The file is in the wrong place#
Robots.txt belongs at the host root, such as https://example.com/robots.txt. A file at /files/robots.txt does not govern the host.
2. Production still has a sitewide block#
A leftover Disallow: / can wipe out crawling for the whole site. This often appears after migrations, redesigns, or launch-day DNS changes. If a migration also changed redirects, check your redirect setup against 301 redirect vs 302.
3. Important CSS or JS is blocked#
Blocked rendering resources can make pages harder for crawlers to interpret. Google’s rendering guidance on JavaScript SEO basics explains why fetchable assets matter.
4. Teams use robots.txt to hide sensitive URLs#
Robots.txt exposes path patterns in public. It does not secure them. If the server answers the request, the resource may still be reachable.
5. Rules are correct, but the wrong expectation follows#
A blocked URL can still be indexed without content if search engines discover the URL elsewhere. That is why robots.txt and noindex solve different problems.
How to find, create, and test a robots.txt file#
To open a robots.txt file, go to the site root and append /robots.txt. If the file does not exist, create a plain text file named robots.txt, upload it to the root of the host, and test the exact paths you care about.
A good workflow is:
- List the folders or URL patterns you want crawled or blocked.
- Write the smallest possible rule set.
- Add the sitemap line if you have an XML sitemap.
- Test key URLs in Search Console.
- Recheck after deploys, redirects, and CMS changes.
If you are also reviewing larger site health issues, YepAPI’s Instant On-Page Audit API can support automated checks alongside redirects and canonical tags, and YepAPI’s On-Page Lighthouse API helps inspect rendering-related issues that robots misconfigurations sometimes hide.
Robots txt checker: how to audit and monitor at scale#
Manual checks are fine for one site. They break down fast across many domains, subdomains, or client environments.
A practical robots txt checker workflow in code looks like this:
- Fetch each host’s
/robots.txt. - Confirm the file returns a valid response from the root.
- Parse user-agent groups and path rules.
- Flag dangerous patterns such as
Disallow: /on production. - Compare blocked paths against important templates, assets, and sitemap locations.
- Re-run on a schedule and alert on changes.
YepAPI’s On-Page API is the scale answer when you need automated technical checks rather than one-off browser reviews. For wider site inspection, YepAPI’s Web Scraping API can also help fetch public technical files and pages programmatically as part of an audit pipeline.
Why robots.txt matters for SEO#
Robots.txt matters because crawl access affects discovery efficiency. On large sites, a cleaner crawl path can help search engines spend more time on pages that matter.
It also matters because errors are high impact. A single bad rule can block key templates, assets, or an entire production site. Robots.txt is small, but it has outsized operational risk.
Use YepAPI to monitor robots rules in code#
When you need more than a spot check, use YepAPI’s On-Page API to automate technical audits across URLs and environments. Pair it with YepAPI’s On-Page Content API for robots-related page directives and YepAPI’s Web Scraping API for broader fetch-based validation.
That is the practical difference between knowing what robots.txt is and being able to monitor it reliably in production.
Pull robots-related technical checks via YepAPI’s On-Page API — $5 free credit, no card.
FAQ#
Is robots.txt legal?#
Yes. Robots.txt is a public technical standard for communicating crawl preferences to automated agents. Using one is normal and legal on your own site. The legal question usually concerns whether third parties choose to respect it, because robots.txt is a request about crawling behavior, not a legal access-control mechanism or contract by itself.
What does "blocked" by robots.txt mean?#
Blocked by robots.txt means a crawler was instructed not to fetch a URL that matches a disallow rule for its user-agent. It does not automatically mean the page cannot be indexed, and it does not mean users cannot open the page. It only describes crawl access, not indexing status or security.
How to open a robots.txt file?#
Open a browser and visit the site’s root domain followed by /robots.txt, such as https://example.com/robots.txt. If the file exists, you will see plain text rules. If it does not exist, the server may return a 404 or another response. The file must live at the host root to control that host.
How do I remove robots.txt from my website?#
Delete the file from the host root or replace blocking rules with the access you want crawlers to have. After removal, crawlers will no longer receive those robots instructions for that host. Check Search Console and key URLs after deployment, because old assumptions about blocked paths, assets, or staging settings can linger in your workflow.
Topics