When to use the Robots.txt Generator
New websites
Create a correct robots.txt file on day one, with a link to your sitemap.
WordPress sites
Keep crawlers out of the admin area while still allowing admin-ajax.php, which many themes need.
Reducing crawl waste
Stop bots from crawling endless filter, search and cart URLs on online stores.
Managing AI crawlers
Add rules for AI training crawlers if you prefer they do not collect your content.
Features and benefits
- Allow and Disallow rules with path validation
- One-click WordPress defaults
- Optional rules for common AI training crawlers
- Sitemap lines and warnings for risky settings
Example
Disallow: /cart/ Sitemap: https://www.example.com/sitemap.xml
User-agent: * Disallow: /cart/ Sitemap: https://www.example.com/sitemap.xml
How the Robots.txt Generator works
robots.txt is a plain text file of groups. Each group starts with a User-agent line naming a crawler (or * for all), followed by Allow and Disallow rules for URL paths. ConvertNest builds these groups from your choices, checks that every path starts with a slash, adds Sitemap lines at the end and warns you about settings that could block your whole site.
A real-life example
An online clothing store notices in Search Console that Google is crawling thousands of URLs like /shop?colour=red&size=m&sort=price. Using the generator, the owner adds Disallow: /*?sort= and Disallow: /*?colour= for all crawlers, keeps product and category pages open, and adds the sitemap URL. Within weeks, crawling focuses on real product pages instead of endless filter combinations.
Good to know
robots.txt must live at the root of your domain, for example https://www.example.com/robots.txt. Crawlers look only there.
Disallow stops well-behaved crawlers from fetching a page, but it does not remove the page from search results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
robots.txt is public and voluntary. Never use it to hide private areas – protect those with passwords.
Tips for the best results
- Upload the file to the root of your domain so it lives at https://yourdomain.com/robots.txt.
- Test important URLs in Google Search Console’s robots.txt report after uploading.
- Always include your sitemap URL.
- Keep the file short and simple – complicated rules are easy to get wrong.
Common mistakes to avoid
Disallow: / on a live site
This blocks crawlers from the whole site. It is a common leftover from a staging server.
Blocking CSS and JavaScript
Google needs these files to render pages. Do not disallow /wp-content/ or asset folders.
Using robots.txt to hide private pages
The file is public and the pages can still be indexed from links. Use passwords or noindex instead.
Robots.txt Generator – frequently asked questions
Where do I upload robots.txt?
To the top folder of your website (the same folder as your homepage), so it is available at yourdomain.com/robots.txt.
Will Disallow remove a page from Google?
Not reliably. Use a noindex meta tag on a crawlable page to remove it from results.
Does Google follow Crawl-delay?
No. Googlebot ignores Crawl-delay. Some other crawlers, such as Bingbot, respect it.
Does blocking AI crawlers work?
The listed crawlers publicly state that they honour robots.txt. Others may not, and new crawlers appear over time.
Does every website need robots.txt?
It is not required, but it is useful for pointing crawlers to your sitemap and avoiding wasted crawling.
Is robots.txt case-sensitive?
Paths are case-sensitive: /Private/ and /private/ are different. Directive names are not.