Premium tools, free for everyone. Most run right in your browser.

OpenMultiURL

SEO

How to Create a Robots.txt File That Works Safely

How to Create a Robots.txt File That Works Safely shown on a coloured title card

Disallow: / is a small line with a big effect: it asks matching crawlers to avoid your whole site. That's the first thing I want you to recognise when learning how to create a robots.txt file. I don't start by collecting folders to block. I start by checking what must stay crawlable, then add only the rules I can explain. Here, I'll help you build a small file, publish it in the right place, and check the live result without confusing crawling with search removal.

What does robots.txt do, and do you need one?

I think of robots.txt as a public set of crawl instructions. A crawler reads it to find out which matching paths it should avoid fetching. It isn't a list of pages that must disappear from search. That difference matters before you write anything, because choosing the wrong control can leave you with a blocked page that's still listed in Google.

I separate crawling, indexing, and access

I choose the control that matches the job, rather than asking robots.txt to do everything.
My goalControl I would considerLimit I keep in mind
Discourage crawling of selected pathsRobots.txt Disallow rulesA blocked URL can still appear in search
Keep a public page out of Google SearchAn appropriate noindex directiveGoogle must crawl the page to read it
Protect private contentAuthentication or server-side access controlsRobots.txt doesn't restrict visitor access
Point crawlers to a sitemapA Sitemap field with a full URLThe sitemap field is optional

I wouldn't create a long file just because a template suggests one. If your public site has nothing that needs crawl restrictions, a minimal file can be enough, and not every site needs a robots.txt file. Google's robots.txt introduction explains the limits. My rule is simple: every restriction needs a reason tied to your actual site, not somebody else's folder names.

And I never put confidential locations in this file as a security measure. Anyone can read it. Listing a sensitive folder can reveal its address without protecting its contents, so I'd fix access protection first.

What should you check before writing crawler rules?

Before I draft rules, I open the site's existing /robots.txt address. There may already be a file, or the platform may generate its contents without a physical file in the hosting folders. I save the current text before changing it. Otherwise, I could replace useful instructions without knowing what they were there for.

  • I confirm the exact site address that needs the rules.
    • I check HTTPS versus HTTP and www versus the hostname without www.
    • I treat other subdomains and different ports separately.
  • I identify what currently serves robots.txt.
    • I check the hosting setup, CMS, or platform settings before uploading another file.
    • I keep a copy of the current response so I can restore it.
  • I write down the purpose of each proposed restriction.
    • I include an example path that should be blocked.
    • I include an important page or resource that must stay crawlable.

If the address is confusing, I use URL Parser to separate its parts before deciding which host I'm editing. Rules apply only to the same host, protocol, and port. A file served on your main domain doesn't automatically control your blog subdomain.

Tip from me: I keep a short reason beside my working notes for every blocked path. If I can't explain why it should be blocked, I leave it out until I've checked.

How do I create a robots.txt file with Robots.txt Generator?

I built Robots.txt Generator and tried this workflow with it as part of testing my tools on a budget Android phone and a laptop. I use the generator to prepare the text, but I keep the decisions separate: a generator can't know which folders your site needs crawled. I also don't treat generating a file as publishing or validating the live website.

  1. Open Robots.txt Generator and keep your checked list of paths beside it. I start with general crawler rules unless I have a reason to target a particular bot.
  2. Prepare the broad user-agent group. I check that the User-agent field contains an asterisk, which represents the general crawler match. I don't add restrictions just to make the file look complete.
  3. Add only the disallowed paths you've reviewed. If I don't need restrictions, I keep an empty Disallow: line in my minimal draft.
  4. Include your full sitemap URL if you have a reachable sitemap. I leave it out rather than guessing an address.
  5. Review the generated text line by line. I check especially for Disallow: /, incorrect path case, and restrictions covering required page resources.
  6. Save the final output as plain UTF-8 text named robots.txt. I then publish it through the system that controls the site's live file.

I start with an allow-all draft

For my minimal starting text, I type User-agent: followed by a space and an asterisk. On the next line, I type Disallow: with nothing after the colon. The empty value means this group blocks nothing. It isn't the same as adding a slash. I save it as robots.txt, not robots.txt.txt, and avoid a word processor that could add document formatting. The server should deliver the file with the text/plain media type.

How do User-agent, Disallow, and Allow rules work?

I read a rules group from its User-agent line downward. That line identifies the crawler the group targets; an asterisk is the general match. Disallow then gives paths that crawler should avoid. Allow can make a more specific exception. I keep each directive on its own line so the structure stays clear when I review the file later.

These are the rule differences I check before publishing.
Example line or fieldMeaning for the matching crawlerMy caution
User-agent field with an asterisk valueGeneral crawler groupSpecific bot groups need separate review
Disallow:No path blocked by this lineKeep the value genuinely empty
Disallow: /Block crawling across the siteDangerous on a public site that needs crawling
Disallow: /search/Block paths beginning with /search/Check the actual URL structure
Allow: /search/help/Permit a more specific matching pathCheck crawler support and overlapping rules
Sitemap: https://example.com/sitemap.xmlDeclare a sitemap locationReplace the example with your real URL

I check matching, not just spelling

Google's path matching is case-sensitive, so I don't assume /Search/ and /search/ mean the same thing. Google supports the asterisk as a wildcard and $ as an end marker, and the more specific matching rule takes precedence. I start with simple path rules because I find them easier to check than a pattern covering several URL shapes.

For a specific page, I might use Disallow: /draft.html$ when testing Google's behaviour. Without the end marker, a path rule can match more than the exact address I had in mind. And if I add a bot-specific group, I review its complete instructions. Bing warns that general directives may need repeating there; I don't assume every group inherits the general rules.

What is a safe robots.txt example with a sitemap?

I prefer a small example whose purpose you can see immediately. Suppose your site really has an internal search area under /search/, and you've decided not to have that area crawled. My draft would ask matching crawlers to avoid that path and point them to a sitemap. It isn't a template for every website.

  1. I start the first line with User-agent: followed by a space and an asterisk.
  2. I put Disallow: /search/ on the next line.
  3. I add Sitemap: https://example.com/sitemap.xml on its own line, replacing the example address with my real sitemap URL.

I confirm that /search/ is the path I mean. I don't add /admin/, /cart/, or /account/ merely because those names appear in other tutorials. Your site could use different paths, or a path could contain something important.

I treat the sitemap line as useful, not required

Google and Bing support Sitemap with a fully qualified sitemap URL. I use the full HTTPS address and open it before publishing. Google also allows multiple Sitemap fields. But leaving the field out doesn't make the rest of robots.txt invalid, and including it doesn't override a blocking rule. I still check that intended pages aren't disallowed elsewhere.

Tip from me: I test a blocked example and an allowed neighbour together. For /search/, I'd compare an actual internal-search URL with an important public page, rather than deciding the rule is safe just because the text looks right.

Where do I put robots.txt, including on a CMS?

I publish the file at the service's top-level /robots.txt path. For an HTTPS site at example.com, that's https://example.com/robots.txt. Uploading it inside a content folder won't give you site-wide control. The Robots Exclusion Protocol standard specifies that top-level location, UTF-8 encoding, and the text/plain media type.

  • For a site managed through hosting files, I place robots.txt where the host serves the website's top-level files, then check the public URL.
  • For a CMS-generated file, I change the source that actually controls its output, rather than assuming an uploaded copy will win.
  • For WordPress, Shopify, or another managed platform, I confirm the current editing method in that platform's setup before making changes.
  • For another subdomain, protocol, or port, I check its own robots.txt response instead of assuming the main site's rules apply.

I won't give you a universal CMS button path here because the editing method depends on the platform and configuration. The reliable check is the served result: after saving, does the exact public robots.txt address show your intended text? If it doesn't, I investigate which plugin, template, hosting layer, or generated response is controlling it.

How do I test whether my robots.txt file works?

I split testing into delivery and rule matching. A perfectly written file is useless if the wrong host serves it, and a file that loads successfully can still block the wrong pages. Google's creation instructions recommend checking that the uploaded file is publicly accessible and can be parsed.

  1. Open the exact live /robots.txt URL. I check that I see the intended plain text, not a login form, an error page, or the site's homepage.
  2. Check response behaviour. I use Redirect Checker to inspect redirects and response status, then investigate any unexpected destination or server error.
  3. Choose representative URLs. I include the homepage, an important content page, necessary CSS or JavaScript, and a URL each restriction should block.
  4. Evaluate the exact crawler and path. Where a crawler-specific testing feature is available, I use it; otherwise, I compare the matching group and rules against official guidance.
  5. Reopen the live file after the final edit. I also confirm that important pages and resources remain accessible, rather than testing only the file itself.

I use Redirect Checker for the response check, not as proof that robots.txt syntax or crawler matching is correct. Those are different jobs. I also keep a short record of which URLs I checked and what each should do, so a later change can be compared against the same expectations.

And I allow for caching. Google generally caches robots.txt for up to 24 hours and may use an older copy longer if it can't refresh the file. Seeing the new text in my browser confirms the live response, but it doesn't prove every crawler has fetched that version.

Which robots.txt mistakes should I avoid?

I watch for a slash that blocks everything

The first mistake I look for is Disallow: / in a group that covers the crawler I want visiting. A launch or migration is a good time to review it. I don't just delete unfamiliar rules at random, though. I compare them with the site's current purpose and restore a known-good version if the published change was wrong.

  • I don't block required CSS, JavaScript, or images blindly. Search engines may need those files to understand rendered pages.
  • I don't combine a crawl block with a noindex directive and assume Google can read the blocked page. It needs access to see that instruction.
  • I don't add crawl-delay expecting Google to follow it. Google doesn't support that field.
  • I don't assume a rule is safe because the file passes a syntax check. Valid syntax can still express the wrong instruction.

I also keep the file short. Google processes only the first 500 KiB and ignores content beyond that limit. The protocol requires crawlers to support a parsing limit of at least 500 KiB, not an identical maximum for every crawler. I wouldn't solve a large rule problem by endlessly appending paths without reviewing the structure.

What if robots.txt is missing or stops working?

I check the actual response before blaming a rule. Google treats most 4xx responses for robots.txt as though the file doesn't exist. A 5xx server error is different: crawling can pause or rely on an older fetched version, depending on the circumstances. A missing file and a broken server aren't interchangeable problems.

My troubleshooting order is location, response, live content, then matching rules. I confirm the hostname and protocol, inspect any redirect or error, and compare the served text with my saved draft. Only then do I work through the intended crawler's group and the exact affected path. That order keeps me from editing rules when the wrong file is being served.

After a CMS change or migration, I repeat the live checks even if nobody intended to change robots.txt. I also review restrictions when paths change. An old rule may no longer do its job, while a broader rule might now cover a new public section. I keep the previous working copy so I can reverse a bad edit without reconstructing it.

Questions people ask

How do I allow all search engines to crawl my site?

I use a general group with User-agent: followed by a space and an asterisk, then an empty Disallow: field on the next line. That group blocks no paths. I then check for bot-specific groups or other restrictions that could change the result. Allowing crawling doesn't guarantee a page will be indexed.

How do I block a folder or a specific URL with robots.txt?

I add a Disallow path under the appropriate User-agent group. For example, Disallow: /search/ matches that path prefix. For an exact ending in Google's supported syntax, I can use the $ marker. I test both the intended blocked URL and nearby URLs that should remain crawlable.

Can I use robots.txt to keep a page out of Google search results?

I wouldn't rely on it for that job. A disallowed URL can still appear in Google if discovered elsewhere. For a public page that shouldn't appear in search, I consider an appropriate noindex directive and leave it crawlable so Google can read it. For private content, I use authentication.

How do I add my sitemap to robots.txt?

I add Sitemap: followed by the full sitemap URL, such as https://example.com/sitemap.xml, replacing the example address with my own. I open that URL to confirm it's reachable. The field is optional, and Google allows multiple Sitemap fields. It doesn't cancel out restrictions affecting the pages listed inside.

How do I edit robots.txt in WordPress or Shopify?

I first find out what generates the current live file and use the editing method supported by that setup. I don't assume a hosting upload or a generic menu path applies to every installation. After saving, I reopen the public /robots.txt address and compare its response with my intended changes.

Can robots.txt block CSS and JavaScript files?

Yes, path restrictions can cover those files, but I avoid blocking resources needed to render important pages. I include representative CSS and JavaScript URLs in my checks, especially after broad folder restrictions. A page being allowed doesn't help enough if the resources needed to understand it are blocked.

What to do next

I would start with the smallest file that matches your needs, not a long template. Open Robots.txt Generator, prepare your general crawler group, and add only restrictions you've checked against real URLs. Include your full sitemap address if you have one. Then publish the plain UTF-8 file at the correct /robots.txt location, inspect the live response, and test important pages alongside the paths you meant to block. Keep the previous version before making another edit. And if your real goal is privacy or search removal, choose authentication or an appropriate indexing control instead of adding more Disallow lines.

Sources I checked

Tools from this guide

Keep reading