In this article
What robots.txt does
Robots.txt sits at the root of your site – yoursite.com/robots.txt – and tells crawlers which paths they may visit. Reputable crawlers such as Googlebot and Bingbot follow it. It is a set of requests, not security: bad bots can ignore it, and anyone can read it.
The basic syntax
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xml
User-agentnames the crawler;*means all.Disallowasks crawlers not to visit a path.Allowmakes an exception inside a disallowed folder.Sitemaptells crawlers where your sitemap is.
Blocking is not the same as hiding
A disallowed page can still appear in Google (without a description) if other sites link to it, because Google knows the URL exists but cannot read it. To keep a page out of search results, allow crawling and add a noindex robots meta tag instead – the Meta Tag Generator creates it. For private content, use a password.
Mistakes that hurt sites
| Mistake | Effect |
|---|---|
Disallow: / left over from a staging site | Blocks the entire site from search |
| Blocking CSS and JavaScript folders | Google cannot render pages properly |
| Blocking pages that have noindex | Google never sees the noindex |
| Wrong capitalisation in paths | Paths are case-sensitive; the rule does not apply |
Blocking AI training crawlers
Some AI companies run crawlers that collect web content for training models, and many respect robots.txt. Examples include GPTBot (OpenAI), Google-Extended (Google’s AI training control, separate from Googlebot search), CCBot (Common Crawl), ClaudeBot (Anthropic), PerplexityBot, Bytespider and Applebot-Extended. Blocking them does not affect normal Google or Bing search rankings. The generator can add rules for these with one click if you prefer not to have your content used this way.
How to create your file
- Open the Robots.txt Generator.
- Start from the WordPress defaults if you use WordPress.
- Add Disallow and Allow paths – the tool checks that each starts with “/”.
- Optionally block AI training crawlers, and add your sitemap URL.
- Read the warnings, then download
robots.txtand upload it to your site’s root folder.
Test it
Open yoursite.com/robots.txt in a browser to make sure it loads. Google Search Console shows how Google reads your robots.txt and reports pages blocked by it. Check it again after any site migration – leftover staging rules are a classic cause of traffic disappearing overnight.
WordPress note
WordPress creates a virtual robots.txt if no file exists, and SEO plugins can edit it. Settings → Reading → “Discourage search engines from indexing this site” must be unticked on a live site. Pair your robots.txt with a sitemap from the XML Sitemap Generator.