Website Tools

Robots.txt Explained – What to Block, What Not to Block

Robots.txt is a small text file with big consequences. One wrong line can hide a whole site from Google. Here is how it works and what a safe file looks like.

In this article
  1. What robots.txt does
  2. The basic syntax
  3. Blocking is not the same as hiding
  4. Mistakes that hurt sites
  5. Blocking AI training crawlers
  6. How to create your file
  7. Test it
  8. WordPress note

What robots.txt does

Robots.txt sits at the root of your site – yoursite.com/robots.txt – and tells crawlers which paths they may visit. Reputable crawlers such as Googlebot and Bingbot follow it. It is a set of requests, not security: bad bots can ignore it, and anyone can read it.

Robots.txt GeneratorBuild a valid robots.txt with allow and disallow rules, sitemap links, WordPress defaults and an option to block AI training crawlers. Download ready.
Open the free tool

The basic syntax

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml
  • User-agent names the crawler; * means all.
  • Disallow asks crawlers not to visit a path.
  • Allow makes an exception inside a disallowed folder.
  • Sitemap tells crawlers where your sitemap is.

Blocking is not the same as hiding

A disallowed page can still appear in Google (without a description) if other sites link to it, because Google knows the URL exists but cannot read it. To keep a page out of search results, allow crawling and add a noindex robots meta tag instead – the Meta Tag Generator creates it. For private content, use a password.

Mistakes that hurt sites

MistakeEffect
Disallow: / left over from a staging siteBlocks the entire site from search
Blocking CSS and JavaScript foldersGoogle cannot render pages properly
Blocking pages that have noindexGoogle never sees the noindex
Wrong capitalisation in pathsPaths are case-sensitive; the rule does not apply

Blocking AI training crawlers

Some AI companies run crawlers that collect web content for training models, and many respect robots.txt. Examples include GPTBot (OpenAI), Google-Extended (Google’s AI training control, separate from Googlebot search), CCBot (Common Crawl), ClaudeBot (Anthropic), PerplexityBot, Bytespider and Applebot-Extended. Blocking them does not affect normal Google or Bing search rankings. The generator can add rules for these with one click if you prefer not to have your content used this way.

How to create your file

  1. Open the Robots.txt Generator.
  2. Start from the WordPress defaults if you use WordPress.
  3. Add Disallow and Allow paths – the tool checks that each starts with “/”.
  4. Optionally block AI training crawlers, and add your sitemap URL.
  5. Read the warnings, then download robots.txt and upload it to your site’s root folder.

Test it

Open yoursite.com/robots.txt in a browser to make sure it loads. Google Search Console shows how Google reads your robots.txt and reports pages blocked by it. Check it again after any site migration – leftover staging rules are a classic cause of traffic disappearing overnight.

WordPress note

WordPress creates a virtual robots.txt if no file exists, and SEO plugins can edit it. Settings → Reading → “Discourage search engines from indexing this site” must be unticked on a live site. Pair your robots.txt with a sitemap from the XML Sitemap Generator.

Robots.txt GeneratorBuild a valid robots.txt with allow and disallow rules, sitemap links, WordPress defaults and an option to block AI training crawlers. Download ready.
Open the free tool
Keep reading

Related articles

All articles
Web

Do You Need an XML Sitemap? How to Make and Submit One

A sitemap lists the pages you want search engines to find. It will not lift your rankings on its own, but it helps new and large sites get discovered faster.

3 min read
Web

The Meta Tags That Still Matter for SEO – and the Ones That Don’t

Old SEO advice lists dozens of meta tags. Only a handful affect how your pages appear in search today. Here is what to set, and what to skip.

3 min read
Web

The Anatomy of a URL – Every Part Explained

Web addresses look like random strings, but every part has a job. Understanding them helps with SEO, analytics, debugging – and spotting scam links.

3 min read