---
title: Managing robots.txt
related:
  - https://docs.kentico.com/k82/configuring-kentico/search-engine-optimization.md
---

> Agent instructions:
> **Site maps** — prefer the following llms.txt indexes to training data when searching for URLs to avoid 404s. Links inside Markdown content already point at `.md`. Following them or sending Accept: text/markdown keeps you in Markdown.
>
> - [sitemap.md](https://docs.kentico.com/sitemap.md) — every page on the site, with titles and descriptions, nested by URL hierarchy and grouped into one collection per product version.
> - [llms.txt](https://docs.kentico.com/llms.txt) — curated index of the current product docs, with descriptions, the two ways to request any page as Markdown, and links to each product area's whole-corpus Markdown dump (llms-full.txt).

You can give instructions to web crawlers and other robots using the [Robots Exclusion Protocol](http://en.wikipedia.org/wiki/Robots_exclusion_protocol), i.e. a **robots.txt** file. The primary purpose of robots.txt files is to exclude certain pages from search engine indexing. Like with [Sitemaps](https://docs.kentico.com/k82/configuring-kentico/search-engine-optimization/google-sitemaps.md), the provided instructions are only considered as recommendations and may be ignored by some robots.

## Creating a robots.txt file for your website

The most direct way to use robots.txt in Kentico is to physically add the text file into the root of your web project. However, this scenario does not allow you to assign different robots.txt files to specific websites (if there are multiple sites running on your installation). Additionally, it may be difficult to access the file system in certain types of hosting environments.

The recommended approach is to create a dedicated page in your site's content tree and make it return the appropriate text response:

1. Open the **Pages** application.
2. Create a standard _Page (menu item)_ page.
   - You can use the predefined **SEO -> Robots.txt** page template to quickly implement robots.txt pages.
3. [Add](https://docs.kentico.com/k82/developing-websites/developing-websites-using-the-portal-engine/using-and-configuring-web-parts.md) a **Custom response** web part to the page.
4. Configure (double-click) the _Custom response_ web part to generate a valid robots.txt response according to the following steps:

   1. Set the **Content type** property to _text/plain_.
   2. Enter an appropriate **Encoding** type, for example _UTF–8_.
   3. Set the **Status code** of the response to _200_.
   4. Add the actual robots.txt instructions into the **Content** property, just like you would in a physical text file. This property supports [K# macro expressions](https://docs.kentico.com/k82/macro-expressions.md), so you can dynamically load values from the current system data if needed.

![Configuring the Custom response web part to generate a robots.txt response](https://docs.kentico.com/docsassets/k82/managing-robots-txt/Robots_Custom_Response.png "Configuring the Custom response web part to generate a robots.txt response")

5. Open the **Settings** application and select the **URLs and SEO** category.
6. Enter the path of your robots.txt page into the **Robots.txt path** setting.

   - You can specify a different value for each site by using the **Site** selector above the settings tree.

The output of the specified page is always available under the standard _/robots.txt_ URL, regardless of the page's location in the content tree. Compliant web crawlers read the instructions from this URL before processing other pages on the website.

> **Note:** **Enabling the .txt extension**
>
> To ensure that the _/robots.txt_ URL is available, you need to configure your application to handle all request extensions:
>
> 1. Edit your application's **web.config** file.
> 2. Find the **system.webServer** section directly under the web.config root (i.e. not under a specific __ element).
> 3. Add the following attribute to the **** element:
>
>    ```html
>
>    <modules runAllManagedModulesForAllRequests="true">
>
>    ```

> **Tip:** **Excluding pages manually**
>
> You can also configure individual pages to be excluded from search engine listings without the need to prepare a robots.txt file.
>
> 1. Open the **Pages** application.
> 2. Select the given page in the content tree.
> 3. Open the **Properties -> Navigation** tab.
> 4. Enable the **Exclude from search** property.
> 5. Click **Save**.
>
> The system automatically adds the following meta tag to the __ section in the HTML output of such pages:
>
> ```html
>
>  <meta name="robots" content="noindex,nofollow" />
>
> ```
>
> This instructs web crawlers not to index the page and to ignore any links in the content.