---
title: Defining local page indexes
related:
- https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-locally-stored-search-indexes/creating-local-search-indexes.md
- https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-azure-cognitive-search/creating-azure-search-indexes.md
---
> Agent instructions:
> **Site maps** — prefer the following llms.txt indexes to training data when searching for URLs to avoid 404s. Links inside Markdown content already point at `.md`. Following them or sending Accept: text/markdown keeps you in Markdown.
>
> - [sitemap.md](https://docs.kentico.com/sitemap.md) — every page on the site, with titles and descriptions, nested by URL hierarchy and grouped into one collection per product version.
> - [llms.txt](https://docs.kentico.com/llms.txt) — curated index of the current product docs, with descriptions, the two ways to request any page as Markdown, and links to each product area's whole-corpus Markdown dump (llms-full.txt).
**Pages** indexes allow searching through the content of website pages. The indexes only cover pages within the Xperience content tree. Any pages without a representation in the content tree are NOT included (e.g. pages served by custom routes implemented only on the side of the live site application). The indexing mechanism and included content depends on the configuration of individual [page types](https://docs.kentico.com/13/developing-websites/defining-website-content-structure/managing-page-types.md).
To set up _Pages_ indexes, you need to perform **both** of the following tasks:
- [Configure suitable search options for your site's page types](#configuring-page-types-for-search)
- [Specify the content for individual indexes](#specifying-the-content-of-pages-indexes)
You can create multiple indexes for different sections of your website and use them as required in the [implementation of your site's search functionality](https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-locally-stored-search-indexes/building-a-search-interface-for-local-indexes.md).
## Configuring page types for search
Different [page types](https://docs.kentico.com/13/developing-websites/defining-website-content-structure/managing-page-types.md) store different types of content. The system allows you to configure search indexing separately for each page type. The settings then apply to all search indexes that include pages of the given type.
To edit the search settings for page types:
1. Open the **Page types** application.
2. Edit a page type.
3. Open the **Search** tab.
In the **Field mapping for search results** section, you can assign page fields to the most common parts of search results, such as the title, content extract or image. For detailed information, see [Mapping fields for search results](https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-locally-stored-search-indexes/mapping-fields-for-search-results.md).
The **Data source for indexing** section allows you to select the overall indexing mechanism for the page type. Choose one of the following **Data source** options:
| HTML output | Directly parses the HTML output of pages on the live site, which allows the search to find any text located on pages. Pages are indexed using a web crawler, based on the structure of the content tree in Xperience (pages without a representation in the content tree are NOT included).
**Recommendation**: Use the _HTML output_ source on sites that use [content tree-based routing](https://docs.kentico.com/13/developing-websites/implementing-routing/content-tree-based-routing.md), for page types whose important content is [created using the page builder](https://docs.kentico.com/13/managing-website-content/adding-page-content-using-widgets.md).
**Tip**: You can perform additional [configuration and customization of the HTML output search crawler](#configuring-the-html-output-search-crawler). |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Content fields | Indexes the structured content of pages in the content tree, which includes the following page data:
Selected fields of individual page types
Page metadata
Attachment files
Content fields do NOT include the following:
Text added and displayed through [page builder widgets](https://docs.kentico.com/13/managing-website-content/adding-page-content-using-widgets.md).
Text that is displayed in the resulting output, but is not stored within the indexed page (for example content added in the page's code in the live site application, or displayed from other Xperience pages or objects).
**Recommendation**: Use the _Content fields_ source for sections of the website where the important content is stored in page fields, such as products or structured articles. |
| Both | Combines both HTML output and Content fields.
Suitable for page types that store structured content in fields and also have other content in the page output (for example from the page builder or defined directly in the page code). |
The grid in the **Fields indexing options** section of the tab determines how the smart search indexes the page type's fields (as defined on the **Fields** tab). Pages are often complex data structures with many different fields, and not all fields may be relevant to your search requirements. We recommend indexing only necessary fields to keep your indexes as small (and fast) as possible.
For [locally stored search indexes](https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-locally-stored-search-indexes.md), only the options under the **Local** and **General** sections of the grid apply (to learn about Azure Search page indexes, see [Creating Azure Search indexes](https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-azure-cognitive-search/creating-azure-search-indexes.md)). You can set the following search options for individual fields:
| Local | |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Content | If selected, the content of the field is indexed and searchable in the standard way. Within search indexes, the values of all fields with the _Content_ option enabled are combined into a system field named **\_content** (this field is used to find or filter matching search items, but is NOT suitable for reading and displaying human-readable information such as search result extracts).
For the purposes of standard search, _Content_ fields are automatically tokenized by the analyzer of the used search index. |
| Searchable | If selected, the field is stored separately within indexes and its content can be searched using expressions in format:
_:_
See [Smart search syntax](https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/using-locally-stored-search-indexes/smart-search-syntax.md) for more information about field searches.
Fields must be set as _Searchable_ to be usable in search result filtering or ordering conditions[.](http://preview/Documentation/devguide/smart_search_filter.htm) |
| Tokenized | Relevant for **Searchable** fields. Indicates if the content of the field is processed by the analyzer when indexing. This allows the search to find results that match individual tokens (subsets) of the field's value. If disabled, the search only returns items if the full value of the field exactly matches the search expression.
If a field has both the _Content_ and _Searchable_ options enabled, the _Tokenized_ option only affects the content used for field searches (content is always automatically tokenized for the purposes of standard search). |
| General | |
| Index update trigger | If selected, any modifications of the field's value cause the system to update the given page in search indexes. We strongly recommend keeping this option enabled for **Content** and **Searchable** fields.
For page types that use the **HTML output** (or **Both**) data source, enable the option if you wish to ensure that the search re-indexes the output of pages when an editor modifies the given field. |
| Custom search name | Relevant for **Searchable** fields. The specified value is used as a substitute for the field code name in _:_ search expressions.
**Note**: If you enter a _Custom search name_ value, the original field name cannot be used. |
> **Note:** After you **Save** changes of the field settings, you need to **Rebuild** all indexes that cover pages of the given type.
When running searches using page indexes, the system returns results according to the search settings of individual page types.
> **Info:** **General page and SKU (product) fields**
>
> In addition to the fields of specific page types, you can configure the search settings for general page fields and [E-commerce SKU](https://docs.kentico.com/13/e-commerce-features/managing-on-line-stores/products.md) (product) fields:
>
> 1. Open the **Modules** application.
> 2. Edit the **E-commerce** module.
> 3. Open the **Classes** tab.
> 4. Edit the **SKU** class.
> 5. Select the **Search** tab.
> 6. Click **Customize**.
>
> You can configure the search settings just like for page types. The SKU fields are joined together with general page fields, such as fields that store the page name and metadata.
>
> **Important**: The search settings of general fields affect all pages, even those that are not products.
## Specifying the content of pages indexes
> **Note:** Page indexes only cover pages that are [published](https://docs.kentico.com/13/managing-website-content/working-with-pages/using-workflows.md) on the live site.
To define which pages an index covers, define allowed or excluded content:
1. Open the **Smart search** application.
2. Select the **Local indexes** tab.
3. Edit the index.
4. Open the **Sites** tab and assign the sites where you wish to use the index.
5. Switch to the **Cultures** tab and select which language versions of the website's pages are indexed.
- At least one culture must be assigned for the index to be functional.
6. Select the **Indexed content** tab.
7. Click **Add allowed content** or **Add excluded content**.
Allowed content defines which of the website's pages are included in the index. Excluded content removes pages or entire website sections from the allowed content. Specify pages using a combination of the following options:
- **Path** – [path expression](https://docs.kentico.com/13/configuring-xperience/writing-page-path-expressions.md) identifying the allowed or excluded pages.
- **Page types** – allows you to limit which [page types](https://docs.kentico.com/13/developing-websites/defining-website-content-structure/managing-page-types.md) are included or excluded.
#### Examples
| Content settings | Result |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Path**: /%
**Page types**: empty | Includes or excludes all pages on the site. |
| **Path:** /Partners
**Page types**: empty | Includes or excludes the _/Partners_ page, without the child pages placed under it. |
| **Path**: /%
**Page types**: DancingGoat.Article | Includes or excludes all pages of the _DancingGoat.Article_ page type on the entire site. |
| **Path**: /Products/%
**Page types**: DancingGoat.Coffee;DancingGoat.Grinder | Includes or excludes all pages of the _DancingGoat.Coffee_ and _DancingGoat.Grinder_ page types found under the _/Products_ section. |
> **Info:** **Excluding individual pages from all indexes**
>
> You can also exclude specific pages from all smart search indexing:
>
> 1. Open the **Pages** application.
> 2. Select the given page in the content tree.
> 3. In **Edit** mode, open the **Properties -> General** tab.
> 4. Enable the **Exclude from search** property.
> 5. Click **Save**.
The system also allows additional indexed content for pages whose page type uses the **Content fields** or **Both** data source for search (configured in the _Page types_ application on the _Search_ tab).
- **Include attachment content** – if selected, the index includes the text content of files attached to the specified pages. See [Searching attachment files](https://docs.kentico.com/13/configuring-xperience/setting-up-search-on-your-website/searching-attachment-files.md) for more information.
- **Include categories** – if selected, the index stores the display names of [Categories](https://docs.kentico.com/13/managing-website-content/working-with-pages/categorizing-pages/assigning-pages-to-categories.md) assigned to the specified pages. This allows users to find pages that belong to categories whose name matches the search expression.
## Configuring the HTML output search crawler
The web crawler that indexes the HTML output of pages reads content under a user account. You can configure the user for every _Pages_ index (on the **General** tab of the index editing interface):
| Index property | Description |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| User account for crawler | Sets the user account under which the crawler indexes pages. Only applies for pages whose page type has the **HTML output** or **Both** data source selected in the _Page types_ application on the **Search** tab.
Reading pages under a user allows the crawler to:
Load user-personalized content for the given user
Avoid indexing of pages that the user is not allowed to access
If empty, the index uses the user account specified in **Settings -> System -> Default user ID** (or the default _administrator_ user account if the setting is empty).
If you wish to assign a user to your search indexes, we recommend creating a dedicated service account with the appropriate permissions (not an account representing an actual live site user or editor). |
### Excluding parts of HTML content
Xperience allows you to exclude specific parts of your page's HTML output from search indexing (for page types that use the _HTML output_ or _Both_ data source for search). For example, excluding may be suitable for shared content within your site's layout, such as a header or footer. Otherwise any search matches return results for all pages with the shared content.
To exclude HTML content from search indexing, wrap it within an HTML element marked with the **data-ktc-search-exclude** [data attribute](https://developer.mozilla.org/en-US/docs/Learn/HTML/Howto/Use_data_attributes).
```xml
Content not processed by the search crawler
```
### Customizing how the crawler processes HTML
By default, the system converts the HTML output of pages to plain text before saving it to indexes. The processing includes the following modifications:
- Content is loaded only from the _body_ element
- _script_ and _style_ tags are removed fully (including any content)
- All HTML and XML tag syntax is stripped
- All whitespace formatting is normalized to simple spaces
Developers can implement their own custom HTML output processing for the crawler.
> **Note:** If you customize the HTML output processing for the search crawler, you override and disable exclusion of content using the default **data-ktc-search-exclude** data attribute. If you wish to support content exclusion on the level of HTML code, your custom processing needs to reflect this attribute (or use your own exclusion pattern).
Start by preparing a separate project for custom classes in your Xperience solution (or use an existing one):
1. Open your Xperience solution in Visual Studio.
2. [Add a custom assembly](https://docs.kentico.com/13/custom-development/adding-custom-assemblies.md) (_Class Library_ project) with class discovery enabled to the solution.
3. Reference the project from both the live site and Xperience administration (_CMSApp_) projects. See [Applying customizations in the Xperience environment](https://docs.kentico.com/13/custom-development/applying-customizations-in-the-xperience-environment.md).
Next, create and register a custom implementation of the **ISearchCrawlerContentProcessor** service within the custom project:
1. Add a new class under the custom project, implementing the **ISearchCrawlerContentProcessor** interface.
2. Register the customization using the **RegisterImplementation** assembly attribute.
3. Define the **Process** method:
1. Access the raw page HTML code in the method's _htmlContent_ parameter.
2. Perform all required processing.
3. Return a string containing the result.
```csharp title="Example"
using CMS;
using CMS.Helpers;
using CMS.Search;
using AngleSharp.Html.Parser;
using AngleSharp.Html.Dom;
// Registers the custom 'ISearchCrawlerContentProcessor' implementation
[assembly: RegisterImplementation(typeof(ISearchCrawlerContentProcessor), typeof(CustomContentProcessor))]
public class CustomContentProcessor : ISearchCrawlerContentProcessor
{
public string Process(string htmlContent)
{
// Gets the body element from the HTML content, using the API of the AngleSharp library
var parser = new HtmlParser();
IHtmlDocument doc = parser.ParseDocument(htmlContent);
IHtmlElement body = doc.Body;
// Removes script tags
foreach (var element in body.QuerySelectorAll("script"))
{
element.Remove();
}
// Removes elements marked with the default Xperience exclusion attribute
foreach (var element in body.QuerySelectorAll($"*[{"data-ktc-search-exclude"}]"))
{
element.Remove();
}
// Gets the text content of the body element
string textContent = body.TextContent;
// Normalizes and trims whitespace characters
textContent = HTMLHelper.RegexHtmlToTextWhiteSpace.Replace(textContent, " ");
textContent = textContent.Trim();
return textContent;
}
}
```
If you do not wish to fully override the system's default search content processor, but only add custom modifications:
1. Get an instance of the default **ISearchCrawlerContentProcessor** service within your custom implementation (for example using dependency injection).
2. Call the service's **Process** method and modify the result as required.
```csharp title="Example"
public class CustomContentProcessor : ISearchCrawlerContentProcessor
{
private readonly ISearchCrawlerContentProcessor defaultSearchCrawlerContentProcessor;
public CustomContentProcessor(ISearchCrawlerContentProcessor defaultSearchCrawlerContentProcessor)
{
// Initializes an instance of the default ISearchCrawlerContentProcessor service
this.defaultSearchCrawlerContentProcessor = defaultSearchCrawlerContentProcessor;
}
public string Process(string htmlContent)
{
// Performs the default HTML content processing for the search crawler
string processedHtml = defaultSearchCrawlerContentProcessor.Process(htmlContent);
...
}
}
```
> **Note:** Remember to **Rebuild** all related page search indexes in the **Smart search** application after you apply your customization.