Premium tools, free for everyone. Most run right in your browser.

OpenMultiURL

SEO

How to Create an XML Sitemap Search Engines Can Crawl

How to Create an XML Sitemap Search Engines Can Crawl, shown on a coloured title card

A sitemap can load in your browser and still list the wrong pages. That's the problem I want you to catch before you press Submit. When I explain how to create an XML sitemap, I start with the URLs, not the XML. You need a clean list of pages you actually want in search, a file search engines can read, and a way to check what happened after submission. I'll walk you through those jobs separately, because finishing one doesn't finish the others.

What does an XML sitemap actually do?

I think of an XML sitemap as a suggested reading list for search engines. It lists page addresses you want them to discover and consider for crawling. It isn't a command, a ranking boost, or proof that those pages belong in search results. Google's Build and Submit a Sitemap documentation makes that limit clear: a sitemap is a hint.

I keep discovery, crawling and indexing separate

  1. Discovery means a search engine learns that a URL exists.
  2. Crawling means it requests that URL and reads the response.
  3. Indexing means it decides to include the page in its search index.

That distinction changes how I troubleshoot. If Google can read the sitemap but a page isn't indexed, rewriting the XML might fix nothing. I would check that page's access rules, canonical choice and indexing directives instead. And I wouldn't promise you that submitting the file will make every listed page appear in search.

I also keep ordinary navigation and internal links in place. A sitemap doesn't replace links that help readers move around your site. An HTML sitemap is a browsable page for people; an XML sitemap is a structured file for machines. You can use both, but they serve different jobs.

How do I find my existing sitemap URL?

Before I create another file, I check whether the website already serves a sitemap. A CMS, plugin or publishing system may already generate it. Keeping that working source is usually better than uploading a second file that becomes stale the next time you publish or remove a page.

  • Open your site's robots.txt and look for a Sitemap: line containing a full sitemap address.
  • Check your CMS or SEO plugin settings for its sitemap feature and published URL.
  • Try the common root location, such as https://example.com/sitemap.xml, without assuming every site uses that filename.
  • Check existing sitemap submissions in your search engine webmaster accounts.

For WordPress, I check the setup before adding anything

Many WordPress setups already provide a sitemap through the platform or an SEO plugin. I wouldn't assume that your filename or settings match someone else's tutorial. Find the actual address in your installation, open it, and see whether it lists pages directly or points to other sitemap files.

An existing file still needs a review. I check the domain, the page types it includes and whether its entries reflect the current site. Finding a sitemap answers where it lives. It doesn't answer whether its contents are right.

Which XML sitemap creation method should I choose?

My preference is to generate the sitemap from the same information that controls publishing. That makes it easier to keep approved URLs and real update dates together. Google recommends using a CMS-generated sitemap where available. Manual creation can suit a site with fewer than a few dozen URLs, but it becomes a maintenance job as the site changes.

I choose a method by looking at how the site publishes and changes.
MethodWhen I'd choose itWhat I'd check
CMS or pluginThe site already has a maintained sitemap featureIncluded page types, exclusions and canonical URLs
Crawler-based generatorI need to collect publicly discoverable URLsMissing orphan pages and unwanted discovered URLs
Automatic generation from site dataThe site has a changing or large page inventoryPublishing rules, update dates and file limits
Manual XMLThe site has few pages and rarely changesXML syntax and updates after every relevant change

A crawler's output isn't my final page inventory

A crawler-based XML sitemap generator can only work with what it discovers. It may miss orphan pages, dynamically generated pages or content behind authentication. I compare its output against my intended canonical URL list rather than treating the crawl as a complete inventory. Access-restricted content also needs an access decision, not automatic inclusion.

Tip from me: I choose the method I can keep correct after publication. A small automatic sitemap with sensible rules beats a carefully edited manual file that nobody updates.

What URLs should I include in an XML sitemap?

I include the preferred, live pages I want search engines to consider for indexing. The word preferred matters. If several addresses show the same content, listing all of them doesn't make the sitemap more complete. I want the list to agree with the site's canonical URLs, not compete with them.

  • Keep URLs that are public, fetchable and intended for indexing.
    • Use the full canonical address, including the correct protocol and hostname.
    • Include published pages you actually want represented in search.
  • Remove URLs that send conflicting signals.
    • Leave out redirected URLs, error pages and duplicate versions.
    • Leave out pages blocked by robots.txt or marked noindex.
  • Review parameter URLs before deciding what belongs.
    • Remove tracking-only variations when the clean address is canonical.
    • Don't remove parameters blindly if they identify genuinely different content.

For a messy starting list, I use Tracking Parameter Remover to clean tracking variations, then URL Deduplicator to remove repeated addresses. Neither step proves that a URL is canonical or indexable. I still have to compare the cleaned list with the website's actual publishing rules.

I also keep the intended site version consistent. If the canonical page uses HTTPS and a particular hostname, that's the address I list. Google attempts to crawl URLs as written. A sitemap full of old addresses that redirect is a list I would repair, not submit unchanged.

How do I build a valid sitemap.xml file?

If I make the file manually, I use a plain-text editor and save it as UTF-8 XML. I don't use a word processor that might add document formatting. The file needs a proper XML structure, not just a list of links renamed sitemap.xml. Here's how I read the basic parts without mixing them up with optional hints.

I check these parts in a basic XML sitemap.
PartWhat it doesMy check
XML declarationIdentifies the XML document and its encodingThe file is saved using UTF-8
urlsetContains the page entries and declares the sitemap namespaceThe sitemap namespace is http://www.sitemaps.org/schemas/sitemap/0.9
urlGroups the information for one pageEach intended page has its own entry
locContains the page's absolute URLThe address is complete, canonical and correctly escaped
lastmodOptionally records a meaningful page updateThe value is accurate, or I leave it out

I use absolute URLs and proper XML escaping

For example, I use https://example.com/about/ rather than /about/. Characters with a special meaning in XML need XML escaping. That's separate from URL encoding, so I don't run a whole sitemap through a URL encoder and expect valid XML. I use an XML-aware generator or validator to check the finished document.

I add lastmod only when I know the significant update date. A routine copyright-date change doesn't justify a fresh value for every page. Google uses lastmod when it's consistently accurate, but ignores priority and changefreq. So I don't spend time adjusting those fields for Google or invent dates to make old pages look new.

How do I check sitemap URLs with Open Multiple URLs?

I built Open Multiple URLs and tried this review workflow with it. I test my tools on a budget Android phone and a laptop, so I prefer manageable batches over opening the whole inventory at once. Open Multiple URLs helps me open pages for inspection. It doesn't generate XML, crawl a website or automatically audit indexing rules.

  1. Prepare the candidate page addresses as a plain list, with one full URL on each line. Don't paste the whole XML document.
  2. Open Open Multiple URLs and paste a manageable selection into its URL input.
  3. Use the opening control to launch the selected pages. If your browser blocks the tabs, review its pop-up permission for the tool and try again.
  4. Check whether each page opens at the intended address and shows the expected content. Note obvious errors, login screens and unexpected destinations.
  5. Use separate technical checks for response status, canonical tags, robots rules and noindex directives. Correct the source list, then regenerate the sitemap.

If an address changes while loading, I investigate it with Redirect Checker. A page looking normal doesn't prove the original URL returned a successful response. And a visual spot check doesn't establish that every URL in a large sitemap is correct.

Tip from me: I group my checks by page type, such as articles or products. That helps me look for repeated template mistakes without burying my phone or laptop in tabs.

How do I split large sitemaps and create an index?

I watch both limits, not just the URL count. Google's limit for a single sitemap is 50,000 URLs or 50 MB uncompressed, whichever comes first. A file can hit the size limit before the URL limit. Compressing an oversized file doesn't remove that uncompressed-size requirement.

I use a sitemap index to point to the separate files

When the inventory needs splitting, I generate multiple valid sitemap files and create a sitemap index that lists their full addresses. The index points to sitemap files, not individual content pages. I publish those files first, make sure each can be fetched, then submit the index address as the main entry point.

  1. Divide the canonical URL inventory into files that stay within both limits.
  2. Generate and validate every child sitemap, including its listed page URLs.
  3. Generate the sitemap index using the published addresses of those child files.
  4. Publish the set together and check that the index doesn't reference missing files.

I may separate products, categories and articles because that makes review easier. That grouping is an organisational choice, not a ranking trick. I also wouldn't assume a basic URL sitemap supplies all the metadata needed for multilingual, image, video or news use cases. Those extensions have additional requirements to check separately.

How do I publish and submit my XML sitemap?

I publish the sitemap at the site root when practical, for example https://example.com/sitemap.xml. Google recommends that location. Without Search Console submission, a sitemap's scope is limited to descendants of the directory where it's hosted. Root placement avoids an easy scope mistake when the file lists pages from across the website.

  1. Open the published sitemap address and confirm it serves the XML file, not a login screen, error page or ordinary HTML page.
  2. Open the matching website property in Google Search Console and select Sitemaps.
  3. Enter the sitemap address or path as requested by the form, check the resulting location, and press Submit.
  4. Add a discovery line to the site's root robots.txt using the full address, such as Sitemap: https://example.com/sitemap.xml.
  5. If you also want to submit to Bing, use the sitemap submission feature in Bing Webmaster Tools.

I use Robots.txt Generator when preparing that file, but I review existing crawl rules before replacing anything. Adding a Sitemap: line doesn't cancel a Disallow rule or override noindex. The file location and the listed pages still need to be accessible under the rules you intend.

Bing supports sitemap discovery through robots.txt as well as submission in its own tools. I keep that separate from Google's interface. And I don't recommend Google's old sitemap ping URL: Google deprecated that endpoint, which returns a 404. Search Console and robots.txt are supported discovery methods.

What sitemap mistakes should I check after submission?

I don't stop at a successful submission message. I check whether the file was fetched and processed, then compare the reported contents with what I intended to publish. File accessibility, XML validity and page eligibility are separate checks. Passing one doesn't prove the others passed.

I start with the symptom, then check the part most likely to explain it.
ProblemWhat I'd checkWhat I'd change
Sitemap can't be fetchedPublished address, server response and access restrictionsFix the location or access problem before resubmitting
XML can't be processedStructure, encoding and escapingCorrect the generator or manual XML
Unexpected URLs appearCMS settings and canonical inventoryFix inclusion rules and regenerate
Expected URLs are missingPublishing rules and crawler discovery gapsCompare against the intended page inventory
Listed pages aren't indexedPage directives, canonical choice and crawl accessInvestigate the pages rather than promising an XML fix

The mistakes I most want you to avoid are stale redirects, blocked pages, duplicate versions and made-up modification dates. Google's XML sitemap best practices warn against listing disallowed or nonexistent pages. I treat those entries as source-data problems, so they don't return during the next automatic build.

I update the sitemap when its inventory changes: pages are published, removed, redirected or made non-indexable. Meaningful content changes can also justify accurate lastmod updates. I don't refresh every date just because the sitemap was regenerated. My final check is simple: does this file still describe the pages I actually want search engines to find?

Questions people ask

Can I create an XML sitemap manually?

Yes. I'd consider manual XML for a small site with fewer than a few dozen URLs and infrequent changes. You still need valid structure, UTF-8 encoding, full canonical URLs and proper XML escaping. I would validate the file before publishing and make sitemap maintenance part of every relevant page change.

Does WordPress automatically create a sitemap?

Many WordPress setups provide a sitemap through the platform or an SEO plugin. I check the actual installation before adding anything, because the active feature, settings and address can vary. Look in your CMS settings and robots.txt, then inspect the published output instead of assuming the default list is right.

Where is my sitemap.xml file located?

I first check the root address, such as https://example.com/sitemap.xml, but that filename isn't universal. Your robots.txt may declare the actual location, and your CMS or SEO plugin may show it in settings. The address may also lead to an index that points to several separate sitemap files.

Should a sitemap include pages marked noindex?

No. I leave out pages marked noindex because they aren't pages I want included in search results. A sitemap doesn't override that directive. I also exclude blocked URLs, redirects, errors and duplicate versions, then check that the remaining entries match the site's preferred canonical addresses.

How many URLs can an XML sitemap contain?

Google's limit is 50,000 URLs or 50 MB uncompressed per sitemap, whichever comes first. I check both limits because URL count alone isn't enough. For a larger inventory, I generate separate sitemaps and a sitemap index containing their full addresses, then verify every referenced file is accessible.

Does submitting a sitemap guarantee that Google will index my pages?

No. I treat submission as a discovery hint, not an indexing promise. Google may read the file without crawling or indexing every listed URL. If a page stays unindexed, I check its access rules, indexing directives and canonical choice rather than assuming another sitemap submission will solve it.

What to do next

I'd start by finding your existing sitemap and checking whether its URLs match the pages you actually want in search. If you need a new file, choose a method you can maintain, generate valid UTF-8 XML, and publish it at the root when practical. Before submitting, use Open Multiple URLs to inspect a manageable sample, then check status codes and indexing rules separately. Submit the finished address in Search Console, add it to robots.txt, and review processing results. Keep the source list accurate as your site changes. That's a useful sitemap, without pretending it can guarantee indexing.

Sources I checked

Tools from this guide

Keep reading