DeskTray

Robots.txt Generator

Generate a correct robots.txt for your website in seconds.

Runs in your browser. Nothing is uploaded Free, no sign-up Works on mobile
Paths to block (one per line)For example admin pages, cart, internal search results
Sitemap URL(s)
Crawl delay in seconds (optional)Google ignores this; Bing and others respect it
Generated code
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout
Disallow: /search?

Sitemap: https://example.com/sitemap.xml

Upload robots.txt to the root of your site (example.com/robots.txt). robots.txt is not a security feature: blocked pages can still be visited and may be indexed if linked. Use noindex or passwords for private pages.

How to use Robots.txt Generator

  1. Choose whether crawling is allowed and list paths to block.
  2. Add your sitemap URL.
  3. Optionally block AI crawlers, then download robots.txt and upload it to your site’s root.

Frequently asked questions

Does robots.txt hide pages from Google?

It stops crawling, not indexing. A blocked page can still appear in search if other sites link to it. Use a noindex tag to keep a page out of results.

Can I block ChatGPT and other AI bots?

Yes. Tick the AI crawler option to add Disallow rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and others. Well-behaved bots respect it.

Related tools

All Web & SEO →