How to Add Your Sitemap to Robots.txt

What is the sitemap directive in robots.txt?

The sitemap directive is a line in your robots.txt file that tells supporting crawlers where to find a sitemap for your website.

Sitemap: https://www.example.com/sitemap.xml

The part after Sitemap: must be the complete address of the sitemap, including https:// and the correct hostname. It can point to an individual sitemap or to a sitemap index that links to several smaller sitemap files.

The directive does not allow or block crawling. Your User-agent, Allow and Disallow rules continue to control crawler access separately.

Why add a sitemap to robots.txt?

A robots.txt file sits at a predictable address at the root of a site. Crawlers commonly request it before exploring other URLs, which makes it a useful place to advertise your sitemap.

• Consistent discovery: supporting crawlers can find the sitemap without a separate manual submission.
• One shared reference: your team has a clear record of which sitemap or sitemap index belongs to the site.
• Support for several crawlers: the declaration is not limited to one search-engine account.
• Cross-host sitemaps: the directive can establish the connection when a sitemap is hosted on another hostname under your control.

Adding the line does not guarantee that every sitemap URL will be crawled or indexed. A sitemap is a discovery hint. Each listed page still needs to be accessible, indexable, useful and technically sound.

It is also sensible to submit the sitemap in Google Search Console. Robots.txt helps discovery across supporting crawlers, while Search Console provides Google-specific processing information and error reporting.

Before you edit robots.txt

Find the correct sitemap and make sure it is worth declaring. Many content-management systems create one automatically, often at an address such as:

https://www.example.com/sitemap.xml
https://www.example.com/sitemap_index.xml

Open the URL in a browser and check that it loads successfully. A sitemap index should link to the individual sitemap files beneath it. An individual XML sitemap should list full page URLs.

Review the contents as well as the file itself. A useful sitemap should contain the canonical, indexable pages you want search engines to consider. Avoid filling it with redirects, error pages, duplicate URL variations or pages carrying a noindex directive.

Your robots.txt file should be available at the root of the relevant host:

https://www.example.com/robots.txt

If you use a hosted website platform, the sitemap and robots.txt file may be generated automatically. Check the platform’s search or SEO settings before editing files directly, because a manual change may be overwritten.

How to add your sitemap to robots.txt

Use the exact live hostname and add the sitemap declaration on its own line. The five checks below help you make the change at the right source and confirm what crawlers can reach.

1. 01
   Open the live robots.txt file.

   Visit /robots.txt on the exact hostname you are checking. Confirm that you have the correct site, protocol and subdomain before changing its source file or CMS setting.

2. 02
   Copy the sitemap’s full URL.

   Open the sitemap and copy its complete address. Keep the exact protocol, hostname, path and filename.

3. 03
   Add a Sitemap line.

   Add the directive on its own line. It is independent of a particular User-agent group, so it does not need to sit beneath a specific crawler rule. A clear convention is to place sitemap declarations together at the end of the file.

4. 04
   Publish the updated file.

   Save the change through the system that controls robots.txt. Then reopen the public /robots.txt URL to make sure the new line is present and has not been changed by a cache, plugin or deployment process.

5. 05
   Test both addresses.

   Open the sitemap URL directly from the published robots.txt file. Confirm that robots.txt and the sitemap each return successfully and that the link uses the intended live hostname.

Sitemap: https://www.example.com/sitemap.xml

Examples for common sitemap setups

One XML sitemap

For a smaller site with one sitemap, declare its full URL once:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The Allow: / line is optional because crawling is allowed by default when no rule blocks it. It is included here only to make the example easy to read.

A sitemap index

Larger sites often use a sitemap index that links to separate page, post, product or image sitemaps. Declare the index rather than repeating every child file:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap_index.xml

Several independent sitemaps

You can include more than one sitemap directive. Give each file its own line:

User-agent: *
Allow: /

Sitemap: https://www.example.com/pages-sitemap.xml
Sitemap: https://www.example.com/products-sitemap.xml
Sitemap: https://www.example.com/images-sitemap.xml

If those files are already listed inside a sitemap index, declaring the index alone is normally simpler.

How to check the finished setup

Do not stop after saving the source file. Verify what crawlers can reach on the public site.

• Open the live robots.txt URL and find every Sitemap: line.
• Follow each declared URL and confirm that it returns a successful response.
• Check that the sitemap belongs to the correct host and contains full canonical URLs.
• Run another Clearcrawl crawl and review the robots.txt and sitemap findings.
• Submit the sitemap or sitemap index in Google Search Console if you want Google-specific status and processing errors.

If you change the sitemap URL later, update robots.txt at the same time. Remove stale declarations once the old file is no longer part of the live setup. If the sitemap still cannot be fetched, check the common sitemap errors and confirm which pages are eligible for search.

Common mistakes

Frequently asked questions