Skip to content
GivenTool

Generate and test a robots.txt file

Build robots.txt with presets (including blocking AI crawlers), validation and a URL tester.

Loading tool…

robots.txt is the plain-text file at the root of a site that tells crawlers which paths they may fetch. It is simple, and simple mistakes in it are costly: one stray "Disallow: /" can drop a whole site out of search, while a missing rule leaves duplicate filtered pages crawled forever. This generator builds the file from groups (one or more user-agents plus Allow and Disallow rules), adds Sitemap lines, checks it as you edit and lets you test any URL against it.

Presets cover the common cases: allow everything; a staging site that blocks everything; WordPress, which blocks /wp-admin/ but keeps admin-ajax.php reachable; an online shop that keeps crawlers out of cart, checkout, account and search pages; and two AI presets. "Block AI training crawlers" keeps search engines allowed but asks the crawlers that gather data for AI models (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent) to stay out. The stricter preset also blocks AI search and assistant fetchers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot), which can stop your pages being cited in AI answers, so choose it deliberately.

The tester applies the rules the way RFC 9309 (the Robots Exclusion Protocol) defines them: the group that names the crawler applies, otherwise the * group; within it, the longest matching path wins, Allow wins a tie, * matches any characters and $ anchors the end. Validation flags paths that do not start with /, sitemaps that are not absolute URLs, blocked CSS and JavaScript, crawlers named twice and Crawl-delay, which Google ignores. Remember that robots.txt is a request, not access control: reputable crawlers obey it, others may not, and a blocked page can still be indexed if other sites link to it. Use noindex or authentication for pages that must stay private.

How to use it

  1. Pick a preset and click Apply, or import your existing robots.txt.
  2. Edit the groups: user-agents, Allow and Disallow paths, and Sitemap URLs.
  3. Fix anything listed under Checks and test important URLs with the tester.
  4. Download robots.txt and upload it to the root of your domain.

Frequently asked questions

Where must robots.txt be placed?

At the root of the host: https://example.com/robots.txt. Crawlers do not look in subfolders, and each subdomain needs its own file.

Does blocking a page remove it from Google?

Not reliably. Disallow stops crawling, but a URL can still be indexed from links elsewhere, without a description. To remove a page, allow crawling and use a noindex meta tag or header.

Will blocking AI crawlers stop all AI use of my site?

Only for crawlers that honour robots.txt and use the tokens you list. It does not remove content already collected, and it cannot stop crawlers that ignore the file.

What does an empty Disallow mean?

"Disallow:" with nothing after it allows everything for that group. The generator writes it for groups without rules so the file stays valid.

Does Google support Crawl-delay?

No. Google ignores Crawl-delay (it adjusts its crawl rate automatically); Bing reads it. Other crawlers vary.

Related tools