Robots.txt Generator Free Tool
About Robots.txt Generator
What This Tool Actually Does
The Robots.txt Generator on this page is a small browser form that builds a robots.txt file for your website. You pick which search engine crawlers may enter your site, choose a crawl delay, type in a sitemap address, and list a few directories you want kept out of search indexes. When you click the create button, the tool assembles the file line by line and shows it in a read-only text box. A second button downloads the same text as a file named robots.txt. The form has been sitting on this page for about nine years. It works entirely in your browser, which means nothing you type is sent to this site's server. The JavaScript that builds the file runs locally, on your machine, the same way a calculator adds numbers without phoning home. For a tool that handles website paths and sitemap URLs, that local behaviour matters, since you might be typing details you do not want logged anywhere.
The output begins with a comment line that credits a different company.
That line reads # robots.txt generated by atozseotools.com. It is a leftover from the original vendor who wrote this form years ago. This page cannot remove that line for you, and you cannot delete it from inside the tool. You can, however, delete it later with any text editor before you upload the file to your server.
This generator creates a robots.txt file in seconds. If you need a basic file that tells every crawler to stay out of one or two directories, you can produce it in under a minute. If your needs are more complicated, the tool will show its limits quickly, and you will want to edit the result by hand. Understanding exactly what each control emits, and in what order, is the difference between a file that works and a file that quietly fails.
How to Use This Tool
- Open the tool page. Find the input box above this article, where the form labelled Robots.txt Generator sits ready.
- Choose the default rule. Use the "Default - All Robots are" select to pick Allowed or Refused. This choice controls the
User-agent: *block that applies to any crawler you do not name individually. - Set your named crawlers. Work through the "Search Robots" list of fifteen crawlers and set each one to Same as Default, Allowed, or Refused. Leave most on Same as Default unless you have a specific reason to single one out.
- Fill the optional fields. Pick a Crawl-Delay value if you want one, type a full Sitemap URL if you have one, and enter up to six Restricted Directories. Clear the pre-filled
/cgi-bin/entry if your site does not use that folder. - Create the file. Click "Create Robots.txt" to see the output in the text area, or click "Create and Save as Robots.txt" to download it directly. A browser confirm box will ask if you are sure.
- Edit and upload. Copy the output into a text editor, remove the vendor comment line if you wish, adjust anything the tool cannot express, save the file as
robots.txt, and upload it to your site's root directory.
What Each Control Emits
The form has four kinds of controls, and each one maps to a specific part of the output file.
The "Default - All Robots are" select writes the User-agent: * block. The "Search Robots" list writes one block per crawler you set to Allowed or Refused. The Crawl-Delay select writes a Crawl-delay: line. The Sitemap field writes a Sitemap: line at the end. The Restricted Directories fields write Disallow: lines.
Two live tests were run for this page on September 4, 2026. The first test used default settings with nothing changed. The second test set Google to Refused, changed the default to Refused, chose a crawl delay of ten seconds, added a sitemap, typed /admin/ as a second directory, and set Naver to Allowed.
| Test scenario | Exact output lines |
|---|---|
| Defaults, no changes | # robots.txt generated by atozseotools.com · User-agent: * · Disallow: · Disallow: /cgi-bin/ |
Google refused, default refused, delay 10, sitemap set, /admin/ added, Naver allowed |
User-agent: Googlebot · Disallow: / · User-agent: naverbot · Disallow: · User-agent: yeti · Disallow: · User-agent: * · Disallow: / · Crawl-delay: 10 · Disallow: /cgi-bin/ · Disallow: /admin/ · Sitemap: https://example.com/sitemap.xml |
The second test yields multiple results simultaneously. Googlebot gets a full block with a slash after Disallow:, which refuses everything. Naverbot gets an empty Disallow:, which allows everything. Yeti also gets an empty disallow, even though you never touched the Yeti row in the form. The crawl delay and both directory lines appear after the star block. The sitemap line comes last.
The vendor comment line appears only in the first test output. In the second test the comment is still there, just omitted from the table for space. Every generated file, without exception, starts with that comment line crediting atozseotools.com.
The tool's output is deterministic. The same settings always produce the same file, in the same order, with the same spacing. There is no randomness and no hidden logic.
The Ordering Consequence You Must Understand
The order in which this tool writes lines is fixed, and that order has a real consequence for how search engines read your file. The generator writes named crawler blocks first, one after another. Then it writes the User-agent: * block. Then, after that star block, it appends the Crawl-delay line and any directory Disallow lines. The Sitemap line goes last.
Under the robots.txt standard, a directive applies to the user-agent group that precedes it. That means the crawl delay and your restricted directories bind only to the * group. They never apply to a crawler you listed by name, because those named blocks were written earlier in the file and have already ended.
Crawlers can misinterpret the directive.
Suppose you set Google to Allowed and then type /admin/ into a directory field. The output will contain a Googlebot block with an empty Disallow:, followed later by a star block with Disallow: /admin/. Googlebot matches its own named block first and sees no restrictions. The /admin/ line belongs to the star group, which Googlebot never reaches. Your admin directory stays visible to Google.
The same trap catches the crawl delay. If you set Googlebot to Allowed and choose a crawl delay of twenty seconds, the delay line lands in the star group. Googlebot ignores it because Googlebot has its own block earlier in the file. The delay applies only to unnamed crawlers that fall through to the star rule.
This is the single most important fact about this tool. The live test with Google refused and directories set shows the pattern clearly. Googlebot's block comes first with a full refusal. The directories still land after the star block, where they only matter for crawlers that did not match an earlier named block.
| What you set | Where the output places it | Who it actually affects |
|---|---|---|
| Crawl-Delay value | After User-agent: * block |
Only crawlers matching the star group |
| Restricted Directory | After User-agent: * block |
Only crawlers matching the star group |
| Named crawler set to Allowed | Its own block, before the star | That crawler only, with no directory limits |
| Named crawler set to Refused | Its own block, before the star | That crawler only, fully blocked |
If you want a directory restriction to apply to a named crawler, this tool cannot do it. You would need to hand-edit the file so that the Disallow: /admin/ line sits inside the Googlebot block, or you would need to rely on the star group alone and leave Googlebot on Same as Default.
The tool assumes a simple mental model where every crawler follows the same rules. Real robots.txt files often need per-crawler paths, and this generator does not support that shape.
The Crawler List Is From 2017
The fifteen crawlers in the "Search Robots" list were chosen years ago, and the web has moved on. Several names in that list belong to services that no longer exist or that have changed their crawler tokens entirely. Using the tool today means working with a snapshot of the search engine world from roughly nine years ago.
MSNBot is retired.
Microsoft retired the MSNBot token years ago and now crawls with bingbot. Slurp was the Yahoo crawler, but Yahoo's search results are now powered by Bing, so Slurp rarely appears in server logs anymore. Teoma was the Ask.com crawler, and Ask has essentially left the general search market. Gigabot belonged to Gigablast, a small engine that shut down. Robozilla was the crawler for the DMOZ directory, which closed in March of that same year. Nutch is an open source crawler framework, not a single commercial product. Yahoo MM, yahoo-mmcrawler, and yahoo-blogs are all Yahoo properties that no longer crawl independently.
The list also has some names that still matter. Googlebot and googlebot-image are current. Google Mobile is now folded into the main Googlebot, which uses a smartphone user agent by default. Baidu still crawls with baiduspider. Naver still uses naverbot and yeti. The ia_archiver token belongs to the Internet Archive, which still runs its Wayback Machine crawler.
| Crawler token in the tool | Status in 2026 | What to use instead |
|---|---|---|
| MSNBot | Retired | bingbot |
| Slurp | Retired, Yahoo uses Bing | bingbot |
| Teoma | Retired | None, Ask is marginal |
| Gigabot | Retired | None |
| Robozilla | Retired, DMOZ closed 2017 | None |
| Nutch | Framework, not a single crawler | Depends on the operator |
| ia_archiver | Active | ia_archiver |
| baiduspider | Active | baiduspider |
| naverbot and yeti | Active | naverbot, yeti |
Several crawlers that matter a great deal in the current year are missing from the list. Bingbot is absent, even though Bing is the second largest search engine. DuckDuckBot, the crawler for DuckDuckGo, is absent. Applebot, which crawls for Siri and Spotlight suggestions, is absent. YandexBot, the Russian search crawler, is absent.
The list has no AI crawlers. GPTBot from OpenAI, ClaudeBot from Anthropic, and CCBot from Common Crawl all appeared after this form was written. These crawlers now generate a large share of bot traffic on many websites, and site owners frequently want to control them. This tool cannot help with that, since none of those tokens appear in the list.
The tool covers Google and a handful of legacy names. If you want to block or allow Bing, DuckDuckGo, Apple, Yandex, or any AI crawler, you must add those blocks by hand after the tool generates its output.
Crawl-Delay and Google's Position
The Crawl-Delay select offers values of five, ten, twenty, sixty, and one hundred twenty seconds, plus a default of no delay. The tool writes whatever you choose as a Crawl-delay: line in the output. No crawler reliably obeys that line.
Google does not.
Google announced several years ago that it would retire support for several unsupported rules in robots.txt, and crawl delay was among them. Googlebot has not honoured Crawl-delay: for years. If you want to slow Googlebot down, the supported method is to adjust the crawl rate in Google Search Console, not to put a delay line in your robots file.
Bing does honour crawl delay. The bingbot documentation states that Bing supports the Crawl-delay: directive, with a minimum recommended value of one second. Yandex also supports it. You need to know which crawlers read the line. The tool's old description treated Crawl-delay as something Google respects. That claim was already outdated when the page was written, and it is clearly wrong now. The live test for this rewrite did not include a crawl delay test against a live crawler, because no browser test can measure that. The facts come from the public statements each search engine has made about its crawler.
| Search engine | Honours Crawl-delay? | Where to control crawl rate |
|---|---|---|
| No, retired years ago | Google Search Console settings | |
| Bing | Yes | robots.txt Crawl-delay line |
| Yandex | Yes | robots.txt Crawl-delay line |
| Baidu | Yes | robots.txt Crawl-delay line |
If your site is being hammered by Googlebot, a crawl delay line will not help. You need to log into Google Search Console, find the crawl rate setting, and lower it there. If your concern is Bing or Yandex, the delay line generated by this tool can make a difference.
The Crawl-Delay control applies to certain crawlers. It is a directive that some crawlers honour and others ignore, and the tool does not tell you which is which.
The Pre-Filled cgi-bin Line
The first Restricted Directories field comes pre-filled with /cgi-bin/. That value was sensible in the late 1990s and early 2000s, when many websites ran CGI scripts in a folder by that name. Those scripts often had security issues, and blocking search engines from indexing them was a common practice.
Very few modern websites use a /cgi-bin/ directory. If your site runs on a content management system, a static site generator, or a modern application framework, you almost certainly have no such folder. Leaving the pre-filled value in place tells crawlers to avoid a path that does not exist, which is harmless but pointless.
You may forget the field is filled. The default output, with no other changes, includes Disallow: /cgi-bin/ after the star block. If you download that file and upload it without looking, your robots.txt will contain a rule you never consciously chose.
You can fix this by adjusting the file's first lines.
Clear that first directory field before you click the create button if you do not use a cgi-bin folder. The field accepts any path relative to your site root, and it must end with a slash. You can type /admin/, /private/, /tmp/, or any other directory name you want to hide. The tool does not validate your paths. You can type a path without a leading slash, and the tool will happily emit Disallow: admin/ instead of Disallow: /admin/. That malformed rule will not match anything, because robots.txt paths are interpreted relative to the host root. You need to check your own input.
Correcting the Old Description of This Tool
The previous version of this page made claims about the tool that do not match what the code actually does. This section corrects those claims so you do not rely on features that do not exist.
The old copy said you enter which user-agents you want to target and which folders or URL patterns should be blocked or allowed.
The real tool has a fixed list of fifteen crawlers, not a free-form entry. It has six directory fields, not an arbitrary number. It has no Allow control at all. There is no way to type a custom user-agent token such as bingbot or GPTBot, and there is no way to express an allow rule. The old copy said the tool assembles a syntactically correct robots.txt file. The output is usually parseable, and the ordering binds your directories to the star group. If you believe your named crawlers are being restricted by the directory lines, the file is not doing what you think. The old copy also did not mention the vendor comment line at the top, which credits a different company and cannot be removed within the tool.
The old FAQ treated Crawl-delay as something Google respects. Google retired that rule years ago. The delay line still appears in the output, and Bing and Yandex still honour it, but Googlebot ignores it entirely.
| Old claim on this page | What the tool actually does |
|---|---|
| Enter any user-agents you want | Fixed list of fifteen crawlers from years ago |
| Block or allow folders and URL patterns | Six directory fields, no Allow lines, no patterns |
| Assembles a syntactically correct file | Directories bind only to the star group |
| Crawl-delay respected by Google | Google retired the rule years ago |
The old description was written when the tool was newer and the search landscape was different. The code underneath has not changed since then. What changed is the web, and this rewrite exists to bring the description in line with reality.
Limitations of This Generator
Every tool has boundaries, and this one has several that come straight from the code. Knowing them before you start saves you from producing a file that fails silently.
There are no Allow lines.
The robots.txt standard supports both Allow: and Disallow:, and many sites need to allow a specific path inside a disallowed directory. This tool only emits Disallow: lines. If you need an allow rule, you must add it by hand.
There are no wildcards or dollar-sign patterns. The modern robots.txt standard, defined by Google and now followed by other engines, supports * as a wildcard and $ as an end-of-string anchor. This tool emits only literal paths. You cannot express a rule like Disallow: /*.pdf$ through the form. There are no per-crawler directories. As described above, every directory you type lands in the star group. You cannot tell Googlebot to avoid /admin/ while letting Bing see it. The tool has no way to associate a path with a named crawler.
There are no comment lines of your own. The only comment in the output is the fixed vendor line. You cannot add explanatory text inside the tool. You can, of course, add comments later in a text editor.
There is no validation of the sitemap URL or your directory paths. You can type not a url in the sitemap field and the tool will emit Sitemap: not a url. You can omit the leading slash on a directory and the tool will not warn you. The output is only as correct as your typing.
There is no host directive. The Host: directive was proposed years ago for Yandex and some other crawlers, but this tool never supported it. Most search engines now ignore it anyway.
There is no way to upload an existing robots.txt file to edit it. The tool starts from a blank state every time. If you have a current file with custom rules, you cannot load it into the form. You can only rebuild from scratch or edit the generated output by hand.
There is no syntax check. The tool does not parse its own output to confirm it would be accepted by a crawler. You are responsible for reviewing the file before you upload it.
| Limitation | What it means for you |
|---|---|
| No Allow lines | Cannot permit a path inside a blocked directory |
| No wildcards | Cannot express pattern rules with * or $ |
| No per-crawler paths | All directories go to the star group |
| No custom comments | Only the fixed vendor comment appears |
| No validation | Bad URLs and paths pass through silently |
| No host directive | Cannot emit a Host line |
| No file upload | Cannot edit an existing robots.txt in the form |
| No syntax check | The tool never reviews its own output |
The rate limit situation is simple. This generator makes no request to this site at all, since everything runs in your browser. The site-wide throttle that applies to page loads, roughly fifteen requests within a single second from one visitor address, does not apply to the tool's operation. You can click the create button as many times as you like.
How to Hand-Edit the Generated File
Because the tool has real limits, most production robots.txt files will need at least a small amount of hand editing. Robots.txt is plain text, and any text editor works. The process takes a few minutes and requires no special software.
Start by removing the vendor comment line if you do not want it. The line # robots.txt generated by atozseotools.com sits at the top of every output. Delete it and replace it with your own comment, such as # robots.txt for example.com, last updated September 2026. Comments are ignored by crawlers, so this is purely cosmetic, but a clean file is easier to maintain. Next, decide whether you want named crawler blocks at all. If your site has no reason to treat Google differently from Bing, you can delete every named block and keep only the star group with your directory lines. That produces a shorter file that is easier to read. If you do want named blocks, move your directory lines into each block that should carry them.
For example, if you want both Googlebot and the star group to avoid /admin/, the generated file will have Disallow: /admin/ only in the star group. You need to copy that line into the Googlebot block as well. The result shows your directories grouped under the star entry.
User-agent: Googlebot
Disallow: /admin/
User-agent: *
Disallow: /admin/Add any missing crawlers you care about. Bingbot, DuckDuckBot, Applebot, YandexBot, GPTBot, ClaudeBot, and CCBot are all absent from the tool. Write blocks for them by hand. For each one, decide whether you want to allow everything, block everything, or block specific paths.
Check your sitemap URL. The tool does not validate it, so confirm it starts with https:// and points to a real file. A broken sitemap line does not break your robots.txt, but it wastes a crawl opportunity.
Review your directory paths.
Each one must start with a slash and end with a slash if it names a directory. A path like admin without slashes will not match the directory /admin/. The robots.txt standard matches paths as prefixes, so Disallow: /admin would block /admin, /admin/, and /administrator. Decide which behaviour you actually want.
Save the file with the exact name robots.txt, all lowercase. Upload it to the root of your domain, so it is reachable at https://yourdomain.com/robots.txt. Then fetch that URL in a browser to confirm the file serves correctly. Google Search Console has a robots.txt tester that can validate your file against Google's parser.
Testing Your Robots.txt After Upload
Once your file is live, you should verify it behaves as intended.
The cheapest test is to open https://yourdomain.com/robots.txt in a browser and read the raw text. Confirm the file starts with your comment or user-agent lines. Some servers return a 404 page for missing files, and uploading to the wrong directory produces exactly that symptom. Google Search Console offers a robots.txt report that shows Google's parsed view of your file. It will list any syntax errors and show which rules apply to Googlebot. This is the authoritative check for Google's behaviour, since Google maintains its own parser and publishes the standard it follows.
For a quick check of other crawlers, you can look at your server logs. After a few days, search for requests from bingbot, DuckDuckBot, or other crawlers you wrote rules for. If a crawler you blocked still appears in your logs, your rule may be malformed or the crawler may not support that directive.
The live test for this rewrite confirmed the tool's output but did not test any live crawler responses. No browser-based test can do that. The only way to know how a crawler treats your file is to observe that crawler's behaviour over time.
Related Tools
If you need an XML sitemap to pair with your robots.txt, the XML Sitemap Generator on this site produces a sitemap file that you can reference in the Sitemap line of your robots.txt.
When you move your site or change its URL structure, the Htaccess Redirect Generator helps you write Apache redirect rules that preserve your search rankings.
To check what Googlebot actually sees when it fetches a page on your site, the Google Crawler Simulator shows you the rendered content from a crawler's perspective.
Frequently Asked Questions
Does this tool send my data to the server?
No.
The generator runs entirely in your browser as JavaScript. Nothing you type in the form is transmitted to this site or any other server. The only network request involved is the page load itself, which is subject to the site-wide throttle of roughly fifteen requests per second from one address.
Why does my generated file start with a comment about atozseotools.com?
That comment line is a leftover from the original vendor who wrote this form. The tool has carried it forward for about eight years, and it cannot be removed from inside the interface. You can delete it with any text editor after generating the file and before uploading it to your server.
Why do my restricted directories not apply to Googlebot?
The tool writes named crawler blocks first, then the star block, then your directory lines. Since the directory lines come after the star block, they belong to the star group only. Googlebot, with its own block earlier in the file, never sees those directory rules. You must hand-edit the file to place directory lines inside named crawler blocks.
Does Google respect the Crawl-delay line?
Google does not. Google announced years ago that it would no longer support several unsupported rules, and crawl delay was one of them. Bing and Yandex do honour the directive. To control Googlebot's crawl rate, use the crawl rate setting in Google Search Console instead.
Why are Bingbot and GPTBot missing from the crawler list?
The list was assembled about nine years ago, before Bing adopted the bingbot token as its primary crawler and long before AI crawlers like GPTBot existed. The list reflects the search engine world of that era. You can add any missing crawler by editing the generated file by hand.
Can I use this tool to block specific files or URL patterns?
Not directly. The tool only emits literal directory paths in Disallow: lines. It has no support for Allow rules, wildcard characters, or dollar-sign anchors. If you need pattern-based rules, you must write them yourself in a text editor after generating the file.