Defining local page indexes

Pages indexes allow searching through the content of website pages. The indexes only cover pages within the Xperience content tree. Any pages without a representation in the content tree are NOT included (e.g. pages served by custom routes implemented only on the side of the live site application). The indexing mechanism and included content depends on the configuration of individual page types.

To set up Pages indexes, you need to perform both of the following tasks:

• Configure suitable search options for your site’s page types
• Specify the content for individual indexes

You can create multiple indexes for different sections of your website and use them as required in the implementation of your site’s search functionality.

Configuring page types for search

Different page types store different types of content. The system allows you to configure search indexing separately for each page type. The settings then apply to all search indexes that include pages of the given type.

To edit the search settings for page types:

1. Open the Page types application.
2. Edit a page type.
3. Open the Search tab.

In the Field mapping for search results section, you can assign page fields to the most common parts of search results, such as the title, content extract or image. For detailed information, see Mapping fields for search results.

The Data source for indexing section allows you to select the overall indexing mechanism for the page type. Choose one of the following Data source options:

The grid in the Fields indexing options section of the tab determines how the smart search indexes the page type’s fields (as defined on the Fields tab). Pages are often complex data structures with many different fields, and not all fields may be relevant to your search requirements. We recommend indexing only necessary fields to keep your indexes as small (and fast) as possible.

For locally stored search indexes, only the options under the Local and General sections of the grid apply (to learn about Azure Search page indexes, see Creating Azure Search indexes). You can set the following search options for individual fields:

When running searches using page indexes, the system returns results according to the search settings of individual page types.

Specifying the content of pages indexes

To define which pages an index covers, define allowed or excluded content:

1. Open the Smart search application.
2. Select the Local indexes tab.
3. Edit the index.
4. Open the Sites tab and assign the sites where you wish to use the index.
5. Switch to the Cultures tab and select which language versions of the website’s pages are indexed.
   • At least one culture must be assigned for the index to be functional.
6. Select the Indexed content tab.
7. Click Add allowed content or Add excluded content.

Allowed content defines which of the website’s pages are included in the index. Excluded content removes pages or entire website sections from the allowed content. Specify pages using a combination of the following options:

• Path – path expression identifying the allowed or excluded pages.
• Page types – allows you to limit which page types are included or excluded.

Examples

The system also allows additional indexed content for pages whose page type uses the Content fields or Both data source for search (configured in the Page types application on the Search tab).

• Include attachment content – if selected, the index includes the text content of files attached to the specified pages. See Searching attachment files for more information.
• Include categories – if selected, the index stores the display names of Categories assigned to the specified pages. This allows users to find pages that belong to categories whose name matches the search expression.

Configuring the HTML output search crawler

The web crawler that indexes the HTML output of pages reads content under a user account. You can configure the user for every Pages index (on the General tab of the index editing interface):

Excluding parts of HTML content

Xperience allows you to exclude specific parts of your page’s HTML output from search indexing (for page types that use the HTML output or Both data source for search). For example, excluding may be suitable for shared content within your site’s layout, such as a header or footer. Otherwise any search matches return results for all pages with the shared content.

To exclude HTML content from search indexing, wrap it within an HTML element marked with the data-ktc-search-exclude data attribute.

<div data-ktc-search-exclude>Content not processed by the search crawler</div>

Customizing how the crawler processes HTML

By default, the system converts the HTML output of pages to plain text before saving it to indexes. The processing includes the following modifications:

• Content is loaded only from the body element
• script and style tags are removed fully (including any content)
• All HTML and XML tag syntax is stripped
• All whitespace formatting is normalized to simple spaces

Developers can implement their own custom HTML output processing for the crawler.

Start by preparing a separate project for custom classes in your Xperience solution (or use an existing one):

1. Open your Xperience solution in Visual Studio.
2. Add a custom assembly (Class Library project) with class discovery enabled to the solution.
3. Reference the project from both the live site and Xperience administration (CMSApp) projects. See Applying customizations in the Xperience environment.

Next, create and register a custom implementation of the ISearchCrawlerContentProcessor service within the custom project:

1. Add a new class under the custom project, implementing the ISearchCrawlerContentProcessor interface.
2. Register the customization using the RegisterImplementation assembly attribute.
3. Define the Process method:
   1. Access the raw page HTML code in the method’s htmlContent parameter.
   2. Perform all required processing.
   3. Return a string containing the result.

using CMS;
using CMS.Helpers;
using CMS.Search;

using AngleSharp.Html.Parser;
using AngleSharp.Html.Dom;

// Registers the custom 'ISearchCrawlerContentProcessor' implementation
[assembly: RegisterImplementation(typeof(ISearchCrawlerContentProcessor), typeof(CustomContentProcessor))]

public class CustomContentProcessor : ISearchCrawlerContentProcessor
{
    public string Process(string htmlContent)
    {
        // Gets the body element from the HTML content, using the API of the AngleSharp library
        var parser = new HtmlParser();
        IHtmlDocument doc = parser.ParseDocument(htmlContent);
        IHtmlElement body = doc.Body;

        // Removes script tags
        foreach (var element in body.QuerySelectorAll("script"))
        {
            element.Remove();
        }

        // Removes elements marked with the default Xperience exclusion attribute
        foreach (var element in body.QuerySelectorAll($"*[{"data-ktc-search-exclude"}]"))
        {
            element.Remove();
        }

        // Gets the text content of the body element
        string textContent = body.TextContent;

        // Normalizes and trims whitespace characters
        textContent = HTMLHelper.RegexHtmlToTextWhiteSpace.Replace(textContent, " ");
        textContent = textContent.Trim();

        return textContent;
    }
}

If you do not wish to fully override the system’s default search content processor, but only add custom modifications:

1. Get an instance of the default ISearchCrawlerContentProcessor service within your custom implementation (for example using dependency injection).
2. Call the service’s Process method and modify the result as required.

public class CustomContentProcessor : ISearchCrawlerContentProcessor
{
    private readonly ISearchCrawlerContentProcessor defaultSearchCrawlerContentProcessor;

    public CustomContentProcessor(ISearchCrawlerContentProcessor defaultSearchCrawlerContentProcessor)
    {
        // Initializes an instance of the default ISearchCrawlerContentProcessor service
        this.defaultSearchCrawlerContentProcessor = defaultSearchCrawlerContentProcessor;
    }

    public string Process(string htmlContent)
    {
        // Performs the default HTML content processing for the search crawler
        string processedHtml = defaultSearchCrawlerContentProcessor.Process(htmlContent);

        ...
    }
}