Robots.txt Generator Free Tool
About Robots.txt Generator
What the Robots.txt Generator Does
A robots.txt file is a plain-text file placed at the root of a domain (for example, example.com/robots.txt) that tells search engine crawlers and other automated bots which parts of a site they are allowed to request. This Robots.txt Generator builds that file for you: you enter which user-agents (crawlers) you want to target, which folders or URL patterns should be blocked or allowed, and optionally your sitemap location and crawl-delay preference, and the tool assembles a syntactically correct robots.txt you can copy or download and drop into your site's root directory.
The point of generating it rather than writing it by hand is mostly about avoiding small mistakes that are easy to make and hard to notice: a missing colon, a Disallow rule under the wrong User-agent block, a trailing slash that changes what a rule matches, or a directive search engines simply don't support. Robots.txt syntax is short, but it's also unforgiving — a single misplaced rule can accidentally block your entire site from Google, and because crawlers just quietly stop visiting rather than throwing an error, that kind of mistake can go unnoticed for weeks.
How Robots.txt Files Actually Work
Robots.txt follows the Robots Exclusion Protocol, a convention that's been in informal use since 1994 and was formalized as RFC 9309 in 2022. It is not a security mechanism and it is not enforced by any authority — it's a set of instructions that well-behaved crawlers voluntarily read and respect. Major search engines (Google, Bing, Yandex, DuckDuckGo's crawler) follow it. Some bots, particularly scrapers and less scrupulous SEO tools, ignore it entirely. That distinction matters: robots.txt controls crawling, not access, and it should never be treated as a way to hide sensitive content.
A robots.txt file is made up of one or more groups, each starting with a User-agent line that names which crawler the following rules apply to, followed by Disallow and/or Allow lines that define paths. The file is read top to bottom, and for any given URL, the crawler applies the most specific matching rule it finds within the group that matches its own user-agent (or the wildcard * group if no specific one exists). Here's the core directive set that any generator, including this one, is working with:
- User-agent — names the crawler a rule group applies to, e.g.
Googlebot,Bingbot, or*for all crawlers. - Disallow — a path prefix the crawler should not request, e.g.
/wp-admin/or/checkout/. - Allow — a path that should be crawlable even if it falls inside a broader Disallow rule (mainly used by Google and Bing).
- Sitemap — an absolute URL pointing to your XML sitemap; this can be placed anywhere in the file and isn't tied to a specific user-agent block.
- Crawl-delay — a suggested number of seconds between requests; respected by Bing and Yandex, but ignored by Google (Google recommends adjusting crawl rate in Search Console instead).
Wildcards are supported by the major engines but not part of the original spec: * matches any sequence of characters within a path, and $ anchors a rule to the end of a URL. So Disallow: /*.pdf$ blocks any URL ending in .pdf, while Disallow: /*.pdf (without the anchor) would also match a URL like /report.pdf/download because it only checks for the substring, not the ending.
How to Use This Robots.txt Generator
The workflow is meant to take a couple of minutes even if you've never hand-written a robots.txt file before:
- Choose your target user-agents. Start with
*(all crawlers) unless you have a specific reason to write separate rules for Googlebot, Bingbot, or another named bot. - List the paths to disallow. Typical candidates: admin panels (
/wp-admin/,/admin/), internal search result pages (/?s=), cart and checkout flows on non-indexable stores, staging or duplicate parameter URLs, and any folder holding non-public assets. - Add Allow overrides if needed. If you disallow a broad folder but need one file or subpath inside it crawlable (a common WordPress example is allowing
/wp-admin/admin-ajax.phpwhile disallowing the rest of/wp-admin/), add that as a separate Allow rule. - Paste in your sitemap URL. Use the full absolute URL, e.g.
https://example.com/sitemap.xml, not a relative path. - Set crawl-delay only if you have a real reason to. Most sites don't need this; it's mainly useful for smaller servers being hit hard by aggressive or lesser-known crawlers.
- Generate, review, and download. Read the output line by line before publishing — the tool assembles valid syntax, but only you know which folders on your specific site are actually meant to be private versus just organizationally separate.
- Upload it to your domain root. The file must live at
https://yourdomain.com/robots.txtexactly — not in a subfolder, not with a different filename, and only one per domain (subdomains each need their own).
After uploading, it's worth testing the live file rather than assuming the upload worked. Fetch the URL directly in a browser to confirm it serves as plain text with a 200 status, and if you use Google Search Console, check the robots.txt status under Settings, or run individual URLs through the URL Inspection tool to see whether Google reports them as blocked or crawlable.
Why Robots.txt Matters for SEO
Search engines allocate a finite amount of attention to any given site — generally described as crawl budget — and how that budget gets spent affects how quickly new or updated pages get discovered and re-indexed. On a small brochure site this rarely matters. On a large e-commerce catalog, a site with faceted navigation generating thousands of filter-parameter URLs, or a publisher with deep archives, an unmanaged crawl can mean bots spending most of their visit on low-value duplicate URLs instead of your newest content.
Robots.txt is one of the tools for shaping that behavior, alongside canonical tags, noindex meta tags, and internal linking structure. It's important to be precise about what it does and doesn't accomplish, because this is where a lot of confusion happens:
- Blocking a URL in robots.txt stops crawling of that URL going forward — it does not remove an already-indexed URL from search results. A blocked URL that was indexed before can still appear in results, sometimes with no snippet, because Google can't recrawl it to confirm it should stay out.
- To actually remove a page from the index, you need a
noindexmeta tag or header on the page itself, and critically, that page must NOT be blocked in robots.txt, because a blocked page can never be crawled to see the noindex tag in the first place. - Robots.txt also has zero effect on whether a page can be linked to and followed via a rel="nofollow" or similar; those are separate, page-level controls.
Where robots.txt genuinely helps SEO is in keeping crawlers away from infinite spaces (calendar widgets, endless filter combinations, session-ID URLs), private administrative areas, staging environments if they're accidentally public, and duplicate-generating parameters, so crawl resources and index space go toward pages you actually want ranked.
Common Use Cases
| Scenario | Typical directive | Why |
|---|---|---|
| Block admin/back-end areas | Disallow: /wp-admin/ or /admin/ |
No SEO value, keeps crawl traffic off login-protected areas |
| Block internal search results | Disallow: /?s= or /search/ |
Prevents indexing of thin, near-duplicate query pages |
| Point crawlers to the sitemap | Sitemap: https://example.com/sitemap.xml |
Speeds up discovery of new/updated URLs |
| Protect a staging subdomain | Disallow: / for that subdomain only |
Stops pre-launch or test content from getting indexed |
| Exclude filtered/faceted URLs | Disallow: /*?color= |
Avoids near-infinite duplicate parameter combinations |
| Slow down an aggressive bot | Crawl-delay: 10 under that bot's User-agent |
Reduces server load from a specific crawler (Bing/Yandex honor this; Google doesn't) |
Robots.txt vs. Meta Robots vs. X-Robots-Tag
These three mechanisms get confused constantly because they all sound like they do the same thing. They don't, and mixing them up is one of the more common ways sites accidentally deindex pages they wanted to keep or leave pages indexed they wanted gone.
- robots.txt — a site-wide file that controls whether a crawler is allowed to request a URL at all. It cannot remove an already-indexed page.
- Meta robots tag — an HTML tag (
<meta name="robots" content="noindex">) placed in the<head>of an individual page, telling crawlers not to index that specific page after they've crawled it. Requires the page to be crawlable, so it must not be blocked in robots.txt. - X-Robots-Tag — the same instruction as the meta tag, but sent as an HTTP response header instead of HTML markup. Useful for non-HTML files like PDFs or images where you can't insert a meta tag.
If your goal is to hide a page from search results entirely, the meta robots tag or X-Robots-Tag is the correct tool, and the page needs to remain crawlable for that instruction to be seen. If your goal is just to keep crawlers from wasting time on a section of the site (without caring whether old URLs linger in the index), robots.txt is the right layer.
Best Practices When Building a Robots.txt File
- Keep it as short as you reasonably can — a sprawling file with dozens of overlapping rules is harder to audit and more prone to contradictions.
- Never disallow
/on a live production site unless you genuinely intend to block the entire domain from all crawlers. - Double-check that CSS and JavaScript files aren't blocked. Google renders pages to evaluate them, and blocking your theme's assets can make Google see a broken, unstyled version of your pages.
- Always include an absolute sitemap URL rather than assuming crawlers will find it on their own.
- Use the
Allowdirective sparingly and only when you need to carve out an exception inside a broader Disallow rule — it isn't needed for paths that aren't already blocked. - Remember robots.txt is case-sensitive and applies per-subdomain and per-protocol;
http://andhttps://versions, orwwwand non-wwwhosts, each need their own file if they're treated as separate origins. - Re-check the file after a site migration, replatform, or CMS change — default robots.txt files generated by staging setups (like a blanket
Disallow: /) sometimes get carried over to production by accident.
Limitations You Should Know About
Robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers from major search engines respect it, but nothing stops a scraper, a malicious bot, or a poorly coded tool from ignoring the file completely and crawling disallowed paths anyway. If you have content that must not be accessed by any automated system, robots.txt is the wrong layer — you need actual authentication, IP restrictions, or a noindex/password-protected setup instead.
It's also worth noting that robots.txt directives are advisory even among cooperative crawlers in a few specific ways: Crawl-delay isn't honored by Google at all, some engines interpret wildcard patterns slightly differently, and a robots.txt file that's unreachable (server error, timeout) can cause a crawler to either pause crawling that domain or, depending on the engine and how long the outage lasts, proceed as if no restrictions exist. Keeping the file small, valid, and reliably served avoids that ambiguity entirely.
Finally, because the file is public by default — anyone can view yoursite.com/robots.txt — listing sensitive folder names in it can act as a signpost for where private material lives, even though the crawlers you're trying to keep out will respect the block. If a path truly needs to stay hidden from people as well as bots, don't rely on robots.txt to keep it obscure.
Common Mistakes to Avoid
- Blocking the entire site by accident. A leftover
Disallow: /from a staging environment is the single most common robots.txt disaster, and it can sit unnoticed until organic traffic quietly drops. - Using robots.txt to try to remove indexed pages. As covered above, this doesn't work and can backfire by preventing Google from ever seeing a noindex tag.
- Blocking CSS/JS directories that are needed for the page to render correctly during Google's rendering pass.
- Putting rules under the wrong User-agent block, so a rule meant for all crawlers only applies to one named bot.
- Forgetting the trailing slash on folder paths, which can make a rule match more or less than intended.
- Assuming subdomains inherit the main domain's robots.txt. They don't — each subdomain needs its own file at its own root.
Frequently Asked Questions
Where do I put the robots.txt file after generating it?
It must be uploaded to the root of your domain, accessible at exactly https://yourdomain.com/robots.txt. Search engines only look for it at that specific location — a copy inside a subfolder like /pages/robots.txt will be ignored.
Will blocking a page in robots.txt remove it from Google search results?
No. Robots.txt only stops future crawling of a URL; it doesn't remove pages already in the index. To deindex a page, use a noindex meta tag or X-Robots-Tag on the page itself, and make sure that page is not blocked in robots.txt, since a blocked page can't be crawled to see the noindex instruction.
Do I need a separate robots.txt for my subdomain or my blog on a different path?
Subdomains (like blog.example.com) need their own robots.txt file, since crawlers treat them as separate hosts. A blog living on a path of the main domain (like example.com/blog/) is covered by the domain's single root robots.txt file.
Can I use robots.txt to hide private or confidential content?
No. Robots.txt is a voluntary convention that only well-behaved crawlers respect, and the file itself is publicly viewable, so it can even hint at where sensitive paths are. Genuinely private content needs authentication or server-level access controls, not a crawl directive.
Why does my robots.txt file say Disallow but the page still shows up in Google?
This usually happens when a page was indexed before it was blocked, or when other sites link to the URL. Google can still list a blocked URL (typically without a description, since it can't crawl the content) if it has other signals suggesting the page exists and is relevant. Removing it fully requires a crawlable noindex tag rather than a robots.txt block.
Does Crawl-delay actually slow down Googlebot?
No. Google has stated it doesn't support the Crawl-delay directive and ignores it; crawl rate for Google is instead managed through settings in Google Search Console. Bing and Yandex do honor Crawl-delay, so it's still useful if those crawlers are putting load on your server.
What happens if my site has no robots.txt file at all?
Crawlers treat a missing robots.txt as permission to crawl everything — there are no restrictions in place. A robots.txt file is only necessary when you actually want to limit or guide crawler behavior; a site with nothing to hide from crawlers can technically operate without one, though including at least a Sitemap directive is still good practice.