Robots.txt Generator

Create the small text file that tells search engine crawlers which parts of your site they may visit – with warnings for common mistakes.

  • Files stay on your device
  • Free, no sign-up
  • Instant
Quick guide · 4 steps

How to use the Robots.txt Generator

When to use the Robots.txt Generator

New websites

Create a correct robots.txt file on day one, with a link to your sitemap.

WordPress sites

Keep crawlers out of the admin area while still allowing admin-ajax.php, which many themes need.

Reducing crawl waste

Stop bots from crawling endless filter, search and cart URLs on online stores.

Managing AI crawlers

Add rules for AI training crawlers if you prefer they do not collect your content.

Features and benefits

  • Allow and Disallow rules with path validation
  • One-click WordPress defaults
  • Optional rules for common AI training crawlers
  • Sitemap lines and warnings for risky settings

Example

Input
Disallow: /cart/
Sitemap: https://www.example.com/sitemap.xml
Output
User-agent: *
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

How the Robots.txt Generator works

robots.txt is a plain text file of groups. Each group starts with a User-agent line naming a crawler (or * for all), followed by Allow and Disallow rules for URL paths. ConvertNest builds these groups from your choices, checks that every path starts with a slash, adds Sitemap lines at the end and warns you about settings that could block your whole site.

A real-life example

An online clothing store notices in Search Console that Google is crawling thousands of URLs like /shop?colour=red&size=m&sort=price. Using the generator, the owner adds Disallow: /*?sort= and Disallow: /*?colour= for all crawlers, keeps product and category pages open, and adds the sitemap URL. Within weeks, crawling focuses on real product pages instead of endless filter combinations.

Good to know

robots.txt must live at the root of your domain, for example https://www.example.com/robots.txt. Crawlers look only there.

Disallow stops well-behaved crawlers from fetching a page, but it does not remove the page from search results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

robots.txt is public and voluntary. Never use it to hide private areas – protect those with passwords.

Tips for the best results

  1. Upload the file to the root of your domain so it lives at https://yourdomain.com/robots.txt.
  2. Test important URLs in Google Search Console’s robots.txt report after uploading.
  3. Always include your sitemap URL.
  4. Keep the file short and simple – complicated rules are easy to get wrong.

Common mistakes to avoid

Disallow: / on a live site

This blocks crawlers from the whole site. It is a common leftover from a staging server.

Blocking CSS and JavaScript

Google needs these files to render pages. Do not disallow /wp-content/ or asset folders.

Using robots.txt to hide private pages

The file is public and the pages can still be indexed from links. Use passwords or noindex instead.

Robots.txt Generator – frequently asked questions

Where do I upload robots.txt?

To the top folder of your website (the same folder as your homepage), so it is available at yourdomain.com/robots.txt.

Will Disallow remove a page from Google?

Not reliably. Use a noindex meta tag on a crawlable page to remove it from results.

Does Google follow Crawl-delay?

No. Googlebot ignores Crawl-delay. Some other crawlers, such as Bingbot, respect it.

Does blocking AI crawlers work?

The listed crawlers publicly state that they honour robots.txt. Others may not, and new crawlers appear over time.

Does every website need robots.txt?

It is not required, but it is useful for pointing crawlers to your sitemap and avoiding wasted crawling.

Is robots.txt case-sensitive?

Paths are case-sensitive: /Private/ and /private/ are different. Directive names are not.