Creating MVT tests

The sections below describe the steps you need to take to set up a multivariate test on your website.

Defining MVT tests

To start optimizing a page on your website via multivariate testing, create and configure an MVT test:

1. Open the Pages application.

2. Select a page in the page tree.

3. Switch to the Analytics -> MVT tests tab.

   • Here you can find a list of all MVT tests assigned to the current page and manage them as required.

4. Click New MVT test and fill in the following properties:

   Property	Description
   Display name	The name of the MVT test displayed in the administration interface.
   Code name	Sets a code name that serves as an identifier for the test. It is also used in the name of the browser cookie used to store which of the test’s variant combinations was assigned to a visitor.
   Description	Can be used to enter a text description for the test in order to give information about its purpose, goals etc.
   Culture	Used to select which culture version(s) of the page should be included in the test.
   Enabled	This check box may be selected to manually start or stop the test.
   Target number of conversions	Sets the number of conversion hits that must be logged to complete the test. Once this number is reached, the MVT test automatically stops working and switches to the Finished status. Leaving this property empty or setting it to 0 means that there will be no conversion hit limit for the test.
   Target conversion type	If the total option is selected, then the Target number of conversions will be compared with the total amount of conversion hits logged for all of the test’s combinations. If any combination is selected, then the test will be concluded when the specified number is reached by one combination (the one with the most conversion hits).
   Start/End	Sets the time interval during which the test should occur. You can use the Calendar button () to select the exact date and time.

   

5. Click Save.

The system creates the MVT test and assigns it to the page.

Creating testing variants

Once a page contains an MVT test, you can start creating the content options that you wish to evaluate. This is done by defining variants for the elements that make up the content of the page. You can use the following objects as variables:

• Web parts – these variants are separate instances of the original web part. Each variant’s properties may be configured differently and you can specify an alternative Web part layout.
• Web part zones – zone variants may contain any type or number of child web parts as required. You can also set the basic properties of the zone differently. When you add a new variant to a zone, the content of the original is automatically copied, so you do not have to rebuild the zone from scratch if you only need to make small modifications. Creating variants for individual web parts inside zone variants is not supported.
• Editor widgets – each variant is a widget of the same type as the original. You can set different values for the properties of each variant.

To add an MVT variant to a web part or zone:

1. In the Pages application, open the selected page on the Design tab.

2. Right-click the header of the given object (or click the  icon), hover over the MVT variants option and select Add web part variant.

   

3. Fill in the following properties for the variant:

   • Display name – the name used in lists of MVT variants in the administration interface.
   • Description – can be used to enter a text description of the variant.
   • Enabled – indicates if the variant should be considered as a possible testing option. If you disable an MVT variant, all testing combinations that include this variant will also be disabled.

4. Click Save & Close.

   • The standard configuration dialog opens – the variant is simply another instance of the original object.
   • By default, the variant has the same values in its properties as are set for the original, change them as required.

5. Click Save & Close once you have configured the variant’s properties to generate the content that you wish to test.

Each object for which testing variants are created has a slider in its header.

This slider can be used to switch between individual variants as needed (including the original). The content specified by the current selection on the page’s variant sliders is displayed in the Pages application on the Design and Page tabs, and in the Preview mode. You can also switch between different versions of the page’s content by using the combination panel. If you wish to view the content of variants while cycling through the slider on the Design tab, make sure that you have the Web part content checkbox selected in the page header.

The buttons on the right of the slider allow you to:

• Add new variants
• Remove the currently selected variant
• Open a list of all variants of the given object

At any time, you can Configure () the properties of the variant currently chosen on the slider.

When configuring variants, you can access the properties that affect the content on the General tab. The settings related to the MVT testing functionality of the specific variant are available on the additional Variant properties tab.

In the case of editor widgets, you can handle multivariate testing variants using a similar approach. The only difference is that you edit the widget variants on the Page tab of the Pages application. The slider and actions for variant management are located on the pop‑up menu of individual widgets.

You can access a list of all multivariate testing variants defined on a given page, by selecting it from the content tree of the Pages application and going to Analytics -> MVT variants. The variants of all three object types are included here, and they can be removed or edited as necessary.

Setting up testing combinations

Individual testing scenarios are represented by combinations of the MVT variants created on the page. By default, all possible combinations of the page’s content are included in multivariate tests, but you may perform additional configuration to fine‑tune your test as required.

Combinations can be managed for pages that have an MVT test defined via a panel located at the top of the page editing interface in the Pages application. This panel is available on both the Page and Design tabs, and in the Preview mode.

You can choose any of the available combinations through the Combination selector. Doing so causes the page to display the content defined by the variants that make up the given combination. The selection made through the combination panel is linked with the positions set on the MVT sliders of variable objects on the page. If you switch to a different variant through an object’s slider, the current combination also changes accordingly and vice versa.

Pages with several variants for multiple objects have a large number of possible combinations. For this reason, it is recommended to carefully choose which combinations should be included in the page’s MVT test. You can include or exclude a combination by selecting the Enabled check box located on the panel next to the selector. There is a close relationship between the status of a combination and its variants. If you disable a variant through its properties, all combinations that include the variant also become disabled. If a disabled combination is enabled at a later point, the system automatically enables all of its variants.

A good way to evaluate potential combinations is to check them in the Preview mode of the Pages application, and disable those where the variants clash visually, have conflicting logic or are otherwise incompatible. We recommend leaving the Default page combination enabled, so you can compare any improvements with the baseline statistics of the original page.

Another best practice is to set a descriptive custom name for each active combination,so that you can easily differentiate between combinations in the editing interface and identify them in the test’s result reports. To do this, simply select a combination through the panel, enter the appropriate text into the Custom name field and click the Change button. The default names assigned to new combinations are composed of a number indicating their order, followed by a list of the display names of all variants that are included in the given combination.

Once the page’s variants and combinations are all configured as needed, you can enable your MVT test on the Analytics -> MVT tests tab of the page. Making further modifications on the page while an MVT test is running is not recommended, since this may affect the accuracy and relevance of the measured statistics.

The Set as result action should only be used after the conclusion of the test, once you have analyzed the logged conversion data and identified the combination that provides the best results. The action allows you to easily set the winning combination as the permanent content of the page. Simply select the given combination via the combination panel, then click the button and confirm the action. This replaces the original web parts, zones and widgets with the variants included in the currently selected combination and removes all other MVT variants from the page.

Configuring conversions

You need to use conversions to track the behavior and actions of the website’s visitors in order to get MVT testing results. Conversions work on a site‑wide level and are not assigned to MVT tests in any way. You only need to define conversion objects and link them with the actions that you wish to log for the test.

There are many ways to specify that an action should be logged as a conversion. For example, you can have the system log a conversion hit whenever users view a specific page:

1. Select the given page from the page tree of the Pages application.
2. Open Analytics -> Settings tab.
3. Use the Track conversion name property to either create a conversion via the New button, or Select an existing one.
4. Click Save.

When an action designated as a conversion is performed anywhere on the website, the system checks if the given user has passed through a page with a running MVT test (according to the presence of a cookie). If this is the case, the conversion hit is logged for the content combination that was assigned to the user.

Once you enable the MVT test and it starts running on the live site, you can monitor the conversion statistics for individual combinations using predefined reports.