Premium tools, free for everyone. Most run right in your browser.

OpenMultiURL

Robots.txt Generator

The Robots.txt Generator helps website owners create crawler rules, add a sitemap address and download a ready-to-publish file. It runs in your browser, but robots.txt guides crawlers rather than securing pages.

  • Free, no subscriptions
  • Runs in your browser
  • Checked by AK Shine
See how it works before you use your own.

Start each path with a slash. Use * for any text and $ for "ends with", for example /*.pdf$

Training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot. AI search: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User.

Bing and Yandex follow it. Google ignores it.

What is the Robots.txt Generator?

The Robots.txt Generator creates a plain-text file that tells cooperating crawlers which parts of your website they may visit. Choose a default rule, enter paths and optional exceptions, and the file updates as you type. Website owners, developers and SEO specialists can also add a sitemap address, block named crawlers or choose an AI crawler preset. Paths without a leading slash get one, full page addresses become paths, and duplicate entries are removed. The tool is free with no subscriptions. Nothing is uploaded, and it works offline once loaded.

Who it is for: Website owners, bloggers, developers, SEO specialists and small business owners can use it to prepare crawler rules before publishing them.

How to use the Robots.txt Generator

  1. Under “Default rule for all crawlers”, choose “Allow everything (most sites)” unless you intend to request a site-wide crawl block.
  2. Enter paths in “Folders or pages to block, one per line”, then add any paths to keep crawlable under “Exceptions to allow inside blocked folders, one per line (optional)”.
  3. Choose an “AI crawlers” setting and, if needed, enter crawler names in “Block these crawlers completely, one per line (optional)”.
  4. Add your full sitemap URL under “Sitemap address (optional)”; use “Crawl delay in seconds (optional)” only for crawlers that support it.
  5. Review “Your robots.txt”, tap “Copy” or “Download robots.txt”, then publish the file at your site's root, such as https://example.com/robots.txt.

Why use it

  • See rules as you typeYour robots.txt updates immediately as you change paths, exceptions or crawler settings.
  • Clean up path entriesThe generator adds missing leading slashes, extracts paths from full addresses and removes duplicate entries.
  • Choose AI crawler rulesBlock the listed AI training crawlers while keeping AI search, or block both listed groups.
  • Keep generation localCreate and copy your file in your browser without uploading your entries.

At a glance

What you put in
Default crawl rule, blocked paths, allowed exceptions, crawler names, optional sitemap URL and crawl delay
What you get
Plain-text robots.txt with Copy and Download robots.txt
Limits
Generates only; does not upload files, test existing rules or prevent indexing
Privacy
Runs in your browser. Nothing you enter is uploaded or stored.
Price
Free for everyone. No subscriptions, no payments, no watermark.
Works offline
Yes, once the page has loaded
Last checked
by AK Shine

Worked example

Input
Default rule: Allow everything (most sites)
Folders or pages to block: /drafts/
Exceptions to allow inside blocked folders: /drafts/public/
AI crawlers: Same rules as other crawlers
Leave custom crawlers, sitemap and crawl delay empty.
Result
User-agent: *
Disallow: /drafts/
Allow: /drafts/public/

These two path rules request no crawling under /drafts/, except for the more specific /drafts/public/ path.

When to use something else

  • For private or staging content, use password protection or server access controls; even “Block everything” is only a request to crawlers.
  • To remove a page from search results, use an appropriate indexing method such as noindex while allowing crawling; robots.txt alone does not remove indexed URLs.
  • For testing an existing file or checking whether a particular crawler can fetch a URL, use a robots.txt tester or search engine webmaster tools; this generator does not test or publish files.

Questions people ask

What should I put in robots.txt for a new website?

For a public site, start with “Default rule for all crawlers” set to “Allow everything (most sites)”; with no blocked paths, the output has an empty Disallow: line. Add your sitemap under “Sitemap address (optional)”, and avoid blocking CSS or JavaScript needed to render your pages.

How do I block a folder but allow a page inside it?

Enter /drafts/ under “Folders or pages to block, one per line”, then /drafts/public/ under “Exceptions to allow inside blocked folders, one per line (optional)”. The more specific path takes precedence for supporting crawlers, and paths also support * and $ patterns.

Where do I upload robots.txt, including for WordPress?

Publish it at the site's top-level address, such as https://example.com/robots.txt, not inside a content folder. Each subdomain needs its own file, and WordPress users should check whether their host or SEO plugin manages it before replacing it. This tool provides Copy and Download robots.txt, but does not upload it.

Does blocking a page remove it from Google or hide private information?

No, a blocked URL can still appear in Google if other pages link to it, and robots.txt is public. For search exclusion, allow crawling so Google can read a noindex directive; for confidential content, use password protection.

Which bots does the AI crawlers setting block?

“Block AI training crawlers, keep AI search” lists GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent and Amazonbot. “Block all AI crawlers” also lists OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User. These are named presets, not a guarantee against every AI crawler.

Can I block a specific crawler completely?

Yes, put its user-agent name in “Block these crawlers completely, one per line (optional)”. The generator places blocked crawler names in a group with Disallow: /, but compliance still depends on the crawler.

Does Google support the Crawl delay setting?

Google ignores Crawl-delay, so “Crawl delay in seconds (optional)” will not slow Googlebot. Bing and Yandex support it, though crawler-specific behavior can differ.

Can this tool check whether my existing robots.txt blocks a URL?

No, it generates text from the controls and does not fetch or test an existing file. After publishing, check important URLs with a robots.txt tester and the relevant search engine's webmaster tools.

Next, you might need