Managing robots.txt

You can give instructions to web crawlers and other robots using the Robots Exclusion Protocol⁠ (opens in a new tab), i.e. a robots.txt file. The primary purpose of robots.txt files is to exclude certain pages from search engine indexing. Like with Sitemaps, the provided instructions are only considered as recommendations and may be ignored by some robots.

Creating a robots.txt file for your website

The most direct way to use robots.txt in Kentico is to physically add the text file into the root of your web project. However, this scenario does not allow you to assign different robots.txt files to specific websites (if there are multiple sites running on your installation). Additionally, it may be difficult to access the file system in certain types of hosting environments.

The recommended approach is to create a dedicated page in your site's content tree and make it return the appropriate text response:

1. Open the Pages application.

2. Create a standard Page (menu item) page.

   • You can use the predefined SEO -> Robots.txt page template to quickly implement robots.txt pages.

3. Add a Custom response web part to the page.

4. Configure (double-click) the Custom response web part to generate a valid robots.txt response according to the following steps:

   1. Set the Content type property to text/plain.
   2. Enter an appropriate Encoding type, for example UTF–8.
   3. Set the Status code of the response to 200.
   4. Add the actual robots.txt instructions into the Content property, just like you would in a physical text file. This property supports K# macro expressions, so you can dynamically load values from the current system data if needed.

5. Open the Settings application and select the URLs and SEO category.

6. Enter the path of your robots.txt page into the Robots.txt path setting.

   • You can specify a different value for each site by using the Site selector above the settings tree.

The output of the specified page is always available under the standard <website domain>/robots.txt URL, regardless of the page's location in the content tree. Compliant web crawlers read the instructions from this URL before processing other pages on the website.

 <meta name="robots" content="noindex,nofollow" />