Robots.txt Generator

Your robots.txt file is the first thing search engine crawlers read when they visit your site. It tells them which folders and pages they may crawl and which they should leave alone, such as admin areas, internal search results and duplicate print pages. A single typo can accidentally block your whole site from Google, so guessing the syntax is risky. Our free robots.txt generator lets you build the file visually: set a default rule, add per-crawler rules for Googlebot, Bingbot and others, protect specific paths, add a crawl delay where supported, and include your sitemap address.

Quick Answer: Set your default allow or disallow rule, add folders you want to protect, optionally add rules for specific bots, paste your sitemap URL, and the generator outputs a valid robots.txt file you can copy.

Free Online Tool

Robots.txt Generator

Build a valid robots.txt file with a visual rule builder.








How to Use the Robots.txt Generator

  1. Choose the default behaviour for all crawlers: allow everything, or block everything (useful only for staging sites).
  2. Add the paths you want to protect, one per line, such as /wp-admin/ or /checkout/.
  3. Optionally add specific rules for individual bots like Googlebot or Bingbot using the bot fields.
  4. Enter your sitemap URL so crawlers can find all your important pages quickly.
  5. Click Generate, copy the file, and save it as robots.txt in the root folder of your domain.

Key Features

  • Visual rule builder that produces syntactically correct robots.txt every time.
  • Per-bot rules for major crawlers including Googlebot, Bingbot and social preview bots.
  • Multiple disallow paths plus an allow override for exceptions inside blocked folders.
  • Sitemap line included automatically when you provide your sitemap URL.
  • Sensible WordPress defaults available with one click, covering admin and login pages.

Best Practices and Tips

  • Always allow CSS, JavaScript and image files; blocking them stops Google rendering your pages properly.
  • Test your finished file in Google Search Console’s robots.txt tester before and after uploading it.
  • Remember robots.txt is a crawling instruction, not security: never list sensitive URLs you would not want discovered.
  • Keep the file small and tidy; group rules by user-agent and remove entries for bots that no longer visit.

Common Mistakes to Avoid

  • Leaving ‘Disallow: /’ in place after launching a site, which blocks the entire domain from search engines.
  • Trying to hide private pages with robots.txt; blocked URLs can still appear in search results without descriptions.
  • Adding a crawl-delay for Googlebot, which ignores that directive entirely; use Search Console crawl settings instead.

Frequently Asked Questions

Where does the robots.txt file belong?

It must sit in the root of your domain, so it is reachable at yourdomain.com/robots.txt. A file placed in a subfolder is ignored by crawlers, and each subdomain needs its own file.

What is the difference between Disallow and noindex?

Disallow stops crawlers from visiting a page, but the URL can still be indexed if other sites link to it. A noindex tag stops indexing but requires the page to be crawlable so the tag can be seen. Use each for its own job.

Can robots.txt block AI crawlers?

Yes, you can add rules for AI user-agents such as GPTBot or ClaudeBot to disallow training crawlers. Whether each bot honours the rules depends on the operator, but the major AI crawlers state that they respect robots.txt.

How do I let Google into one folder inside a blocked directory?

Add an Allow rule for the specific sub-path. When rules conflict, crawlers generally follow the most specific matching rule, so Allow: /folder/keep/ inside a blocked /folder/ works for the major engines.

Does every site need a robots.txt file?

A missing file simply means everything may be crawled, which is fine for tiny sites. As soon as you have admin areas, staging copies or crawl-wasting pages, a deliberate robots.txt becomes worthwhile.

Related Tools