A/B testing website pages

A/B testing is an on‑line marketing technique used to optimize the pages of a website according to the reactions of visitors. This is achieved by creating one or more modified versions of a given page and running them all at the same time during a designated testing period. The system then divides traffic between individual page versions and tracks how the changes affect the conversions that the users make. Technique where you create a single modified variant that you test against an original variant is also called split A/B testing.

Determining which variant has the most conversions

In Kentico, the basic objects that provide this functionality are A/B tests, which you can create for specific website pages. The pages that differ from the original page and that you want to measure against the original are called variants. Kentico supports the following forms of A/B testing:

• Content A/B tests - allow you to test different variants of content pages against one another.
• E-mail A/B tests - allow you to test different variants of newsletters against one another.

Content A/B tests

Each content A/B test variant is linked to a document in the content tree of the website. You can design and configure each variant using the standard web development processes. The Original page for which the test was created is included as one of the test variants.

For a practical scenario, imagine that you have an e‑commerce page on your website where visitors can purchase products and you wish to modify it to be more user friendly in order to increase the amount of sales. By utilizing A/B testing, you can create multiple versions of this page with any type of modifications such as a slightly altered layout, graphical design or copy text. The alternative pages will then serve as variants of the A/B test. While the test is running, it automatically displays different page variants to different visitors and keeps track of all product purchases as conversions. When the test completes, after a certain amount of time, you can evaluate which variant encouraged more users to make purchases on the website and use it as the main page.

With A/B testing, results are tracked for entire page variants, which means that the measured statistics reflect the combined effect of all changes made to each variant. If you wish to monitor how individual page modifications affect the behaviour visitors, use Multivariate testing, which is another website optimization feature provided by Kentico.

How A/B testing works

When a visitor navigates to a page that has a running A/B test defined, one of the page variants configured for the given test will be displayed to them. The variant is chosen randomly for every user. With a large enough visit sample size, each page variant should receive roughly the same amount of traffic during the course of the test.

A persistent cookie is stored in the visitor’s browser. The cookie is used to identify which variant was assigned by the given A/B test. The name of the cookie uses the following format:

CMSAB<A/B test code name>

The cookie saves the code name of the assigned page variant as its value. This cookie expires either within 30 days after the last visit on the tested page, or on the date when the test is configured to end.

Any conversions performed on the website by users who have passed through an A/B test page will be logged under the assigned page variant, which is taken from the value of the A/B testing cookie. The logging of conversion hits is provided by the Web analytics application. In addition to monitoring conversions, the cookie also ensures that returning visitors are always shown the page variant that was previously assigned to them, which helps avoid confusion by maintaining a consistent appearance of the tested page.

It is possible to run multiple A/B tests concurrently for different pages on the same website. If you want to run multiple A/B tests on the same page, each test needs to be running on a different culture version of the page. Conversions will be logged for all tests defined for pages visited by a given user, according to the cookies present in the browser.