Creating A/B tests for pages

Kentico EMS required

Features described on this page require the Kentico EMS license.

The sections below describe the steps you need to take to successfully set up an A/B test on your website.

To set up an A/B test on your website, you need to do the following:

  1. Make sure your website logs all the necessary conversions
  2. Define an A/B test
  3. Add page variants to the test

Defining A/B tests

To start optimizing a page on your website via A/B testing, create and configure an A/B test:

  1. Open the Pages application and select the appropriate page from the content tree.

  2. Switch to the Analytics -> A/B tests tab.

    • Here you can find a list of all A/B tests assigned to the currently selected page and manage them as required.
  3. Click New test and fill in the A/B test’s properties.

    View properties

    Property

    Description

    Display name

    The name of the A/B test displayed in the administration interface.

    Description

    Can be used to enter a text description for the test in order to give information about its purpose, goals, etc.

    Start

    Set the starting date and time for the test. You can use the Calendar button () to select the exact date and time.

    End

    Set the date and time at which the test finishes. You can use the Calendar button () to select the exact date and time.

    Displayed conversion goals

    Select the conversion goals that you want to view in the test’s overview. Leave empty if you want to view all conversion goals.

    Tested culture

    Select which culture versions of the page should be included in the test.

    Visitor targeting

    Specify a macro condition that filters the tested visitors based on its criteria.

    Included traffic

    Set the percentage of the page visitors which will be included in the test.

    Creating a new AB test

  4. Click Save.

The system creates the test and assigns it to the page. The page is now the test’s Original variant.

Delayed start of A/B tests due to output caching

When scheduling an A/B test to start at a later time, the actual test may be visible on the live site with a delay due to output caching. Administrators should adjust the duration of output caching if necessary.

Managing all A/B tests from a single location

You can manage all tests assigned to different pages from Pages (application) -> AB Tests.Here you can manage all A/B tests defined on the current website and their page variants. The only difference is when you are creating new tests from this location. You need to explicitly specify the Original page property.

Adding page variants

Each A/B test that you create contains an Original variant. Original variant is the variant that you created when defining an A/B test. Each A/B test also needs to contains other variants that you can compare using the test.

Since each variant is represented by a separate page in the website’s content tree. You can utilize all page configuration and website design options provided by Kentico.

Usually, all the page variants of an A/B test use the same basic concept, but have one key difference that sets them apart from the original page, such as:

  • The position of an important page element.
  • A different color scheme intended to highlight a part of the page.
  • Alternative text content of headings or descriptions that could potentially be more interesting for visitors.

These are only basic examples of ideas that can be tested by individual variants. You can implement any other type of modifications according to your specific requirements.

It is recommended that you create test variants directly from the Original page. If the pages that you want to make into an A/B test variants already exist, perform the following steps:

  1. Creating A/B test page variants from existing pages
  2. Excluding A/B test variants from navigation and site map
  3. Excluding A/B test variants from site search

Creating test variants directly

You can save time when defining page variants by using an action that combines the two steps described above.

  1. Select the A/B test’s original page in the Pages application.

  2. Click New () in the menu above the content tree.

  3. Instead of selecting a page type, choose the A/B test page variant option below the list.

    You can also do the same by right‑clicking the page in the content tree and selecting New -> A/B test page variant in the context menu.

  4. Fill in the following options in the dialog:

    • Page name - set the name of the page representing the new page variant.
    • Save under location - set the path of the page under which the page variant will be placed. The Select button may be used to choose a parent page directly.
    • Assign to A/B test - select the A/B test to which the variant will be added.
    • Show in navigation, Show in site map, Exclude from search - these options may be used to directly set the corresponding properties of the new page. The default values hide the page variant from all standard website navigation.
  5. Click Save to confirm the entered data.

The system creates a copy of the selected page and automatically adds it as a page variant of the specified A/B test. The copy uses an ad-hoc page template. Now you can implement the required modifications that will distinguish the variant from the original page.

Creating page variants from existing pages

If you have already have modified versions of the original page and want to add them to an A/B test as variants:

  1. Select the original page.

  2. Go to Analytics -> A/B tests.

  3. Edit () the test and switch to its Variants tab.

  4. Click New variant and enter the following properties:

    • Display name - the name of the variant displayed in the administration interface.
    • Code name - sets a code name that serves as an identifier for the page variant. It is also used as the value of the A/B testing cookie stored for visitors assigned to this variant.
    • Variant path - must contain the path of the associated page. When users assigned to the variant access the original page of the given A/B test, they will see the page specified here instead. Use the Select button to choose an existing page from the website’s content tree.
  5. Click Save.

Repeat this process for all pages that you want to include as possible page options for the A/B test. The original page is automatically added as a variant when the A/B test is created, and highlighted in the list by a yellow background.

Important

When editing a running A/B test, it is necessary to always keep the variant Variant paths up‑to‑date and ensure that the target pages actually exist in the content tree. Otherwise visitors may encounter a Page not found error if the path of their variant is not valid.

To preserve conversion statistics, variants remain in the system even if their associated page is deleted.

It is recommended not to make any modifications to a running test.

It is recommended exclude page variants from being shown in navigation, site map and site search. This is done to prevent users from encountering multiple pages with nearly identical content. The system automatically displays the appropriate variant to users when they view the test’s original page, so the standard navigation is not necessary.

To exclude variants from navigation and site map

  1. Select the variant in the content tree.
  2. Switch to the Properties -> Navigation tab.
  3. Turn the Show in navigation check-box off to exclude the variant from being displayed by navigation controls and web parts (i.e. in the website’s menus).
  4. Turn the Show in sitemap check-box off to exclude the variant from the Google Sitemap and from being displayed by the Site map web part.
  5. Turn the Exclude from search check-box on.
  6. Save the changes.

Now, users can’t find the variant in the site navigation, site map. The variant also doesn’t get returned in searches users perform on the site.