---
title: A/B testing marketing emails
related:
  - https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/email-marketing.md
  - https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/email-marketing/tracking-marketing-emails.md
---

> Agent instructions:
> **Site maps** — prefer the following llms.txt indexes to training data when searching for URLs to avoid 404s. Links inside Markdown content already point at `.md`. Following them or sending Accept: text/markdown keeps you in Markdown.
>
> - [sitemap.md](https://docs.kentico.com/sitemap.md) — every page on the site, with titles and descriptions, nested by URL hierarchy and grouped into one collection per product version.
> - [llms.txt](https://docs.kentico.com/llms.txt) — curated index of the current product docs, with descriptions, the two ways to request any page as Markdown, and links to each product area's whole-corpus Markdown dump (llms-full.txt).

> **Info:** **Enterprise license required**
>
> Features described on this page require the **Kentico Xperience Enterprise** license.

When preparing [emails for marketing purposes](https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/email-marketing.md), it can be difficult to predict how recipients will react to important elements, such as the email subject or the primary call to action. You may also want to find out which option out of several alternatives will have the most positive effect. Optimizing your marketing emails via A/B testing provides a possible solution to these problems.

A/B testing allows you to create multiple different versions of each email called _variants_. You then mail out the variants to a _test group_ of recipients, typically a portion of the marketing email's full mailing list. The best version of the email can be identified based on the tracking statistics measured for the test group, and then sent to the remainder of the recipients. The winning variant can either be selected automatically by the system according to specified criteria or manually after evaluating the test results.

A/B testing works best for marketing emails that have a large number of recipients. With too small a testing group, the results may be heavily affected by random factors and will not be statistically significant.

> **Info:** **A/B testing emails sent via marketing automation**
>
> A/B testing works in a slightly different way for marketing emails that are sent by [marketing automation processes](https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/marketing-automation.md) (emails that have the **Use in Marketing automation** property enabled).
>
> Such emails do not have an exact list of recipients. Instead, the mail-out is an ongoing process, where the email is sent to individual contacts when they reach a _Send marketing email_ step in an automation process.
>
> For emails with a running A/B test, the system randomly selects and sends one of the variants. Over the course of the test's duration, each variant should be sent to roughly the same amount of contacts going through the process. Once a winner is selected, marketing automation always sends the winning variant.

## Prerequisites

On-line marketing must be enabled for your website:

1. Open the **Settings** application.
2. Navigate to **On-line marketing**.
3. Select **Enable on-line marketing**.
4. Click **Save**.

A/B testing variants of marketing emails are evaluated based on the actions performed by recipients, so both types of [Email tracking](https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/email-marketing/tracking-marketing-emails.md) need to be enabled for tested emails:

1. Open the **Email marketing** application on the **Email feeds** tab.
2. **Edit** () an email campaign or a newsletter.
3. Switch to the **Configuration** tab.
4. Make sure **Track opened emails** and **Track clicked links** are enabled.
5. Click **Save**.

## Creating A/B tests for marketing emails

You can define A/B tests in the **Email marketing** application, either directly when [composing new marketing emails](https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/email-marketing/composing-marketing-emails.md), or when editing existing marketing emails on their **Email builder** tab:

1. Create a new email or edit an existing one.
2. Switch to the **A/B testing** tab of the **Email builder**.
3. Click **Create another variant**.

   ![Switching to the email builder A/B testing tab](https://docs.kentico.com/docsassets/13/a-b-testing-marketing-emails/EmailBuilderABtab.png "Switching to the email builder A/B testing tab")
4. Enter a **Name** for the variant.

   ![Creating a new variant](https://docs.kentico.com/docsassets/13/a-b-testing-marketing-emails/NewABVariant.png "Creating a new variant")
5. Click **Create**.

   - The content of the original variant is used as a starting point that you can modify as required. The variant uses the same template as the source email and retains the widget zone content of the original.

   ![Viewing created email variants](https://docs.kentico.com/docsassets/13/a-b-testing-marketing-emails/EmailBuilderVariantListing.png "Viewing created email variants")

Now that you have defined an A/B testing variant for the email, _Email builder_ displays the _A/B testing variant_ drop-down list. You can use the drop-down list to switch between individual variants, including the original email. The _A/B testing variant_ drop-down list shows the name of the currently selected variant. You can manage the A/B test variants using the following actions:

- **Create another variant** – creates another variant of the original email. You can add any number of variants.
- () **Remove variant** – deletes the currently selected variant.

You can modify the settings and content of variants just like when editing standard emails. Each variant may have a different subject, email template, editable region content etc. This allows you to test any variables that you need.

## Sending A/B test marketing emails

Once you define all of the email's variants, you need to configure how the test is sent out and evaluated.

### Standard emails

For marketing emails that you send directly from the _Email marketing_ application, configure the A/B test mail-out:

1. Edit the email in the **Email marketing** application and open the **Send** tab.
2. Define the size of the recipient test group by using the slider in the upper part of the page.

   - By moving the slider's pointer, you can increase or decrease the number of recipients that will receive the variants of the marketing email during the testing phase.

   > **Info:** 
   >
   > - The slider automatically balances the test group so that each variant is sent to the same amount of recipients.
   > - The overall test group size is always a multiple of the total number of variants created for the email.
   > - The remaining recipients who are not part of the test group will receive the variant that achieves the best results (i.e. the winner) after the testing process is complete.

   ![](https://docs.kentico.com/docsassets/13/a-b-testing-marketing-emails/SettingUpVariantSend.png)

   > **Tip:** **Using a full test group**
   >
   > You can set up a scenario where the test group includes 100% of all recipients. In this case, the A/B test simply provides a way to evenly distribute different versions of the marketing email among the recipients and the selection of the winner is done for statistical purposes only.
3. In the **Schedule of the mail-out** section below the slider, specify when to send out individual email variants to the recipients from the corresponding portion of the test group.
   - To schedule the mail‑out, enter the required date and time into the field below the list (you can use the   **Calendar** selector or the **Now** link) and click **OK**.
   - You can either do this for _(all )_ variants or only for those selected in the list.
   - If the mail-out time is the same for multiple variants, the system sends them in approximately 1-minute intervals between individual variants.
4. In the **Method of selecting the winning variant** section, select how the A/B test chooses the winning variant:
   - **Manually** – the A/B test does not select the winner automatically. Instead, the author of the email (or other [authorized users](https://docs.kentico.com/13/on-line-marketing-features/configuring-and-customizing-your-on-line-marketing-features/configuring-on-line-marketing-permissions.md)) can monitor the results of the test and choose the winning variant manually at any time.
   - **Automatically based on unique opens** – the system automatically selects the variant with the highest number of opened emails as the winner. This type of testing focuses on optimizing the first impression of the email feed, i.e. the subject of the emails and the sender name or address, not the actual content.
   - **Automatically based on unique clicks** – the system automatically chooses the winner according to the amount of link clicks measured for each variant. Each link placed in the email's content is only counted once per recipient, even when clicked multiple times. This option is recommended if the primary goal of your email feed is to encourage recipients to follow the links provided in the email.
5. When using an automatic selection option, set the duration of the testing period through the **Select the winning variant after** settings.
   - This allows you to specify how long the system waits after the last variant is sent out.
   - When the testing period is over, the test selects the winner and sends it to the remaining recipients.
6. Click **Send**.
   - If you only wish to save the configuration of the A/B test without actually starting the mail-out, click **Save** instead.

### Marketing automation emails

For marketing emails that are sent by [marketing automation processes](https://docs.kentico.com/13/on-line-marketing-features/managing-your-on-line-marketing-features/marketing-automation.md) (emails that have the **Use in Marketing automation** property enabled), you only need to configure the selection of the A/B test winner:

1. Edit the email in the **Email marketing** application (or directly from the **Marketing automation** application when designing a process with a _Send marketing email_ step).
2. Open the **A/B testing** tab.
3. Select how the A/B test chooses the winning variant:
   - **Manually** – the A/B test does not select the winner automatically. Instead, the author of the email (or other [authorized users](https://docs.kentico.com/13/on-line-marketing-features/configuring-and-customizing-your-on-line-marketing-features/configuring-on-line-marketing-permissions.md)) needs to monitor the results of the test and choose the winning variant manually at any time.
   - **Automatically based on unique opens** – the system automatically selects the variant with the highest number of opened emails as the winner. This type of testing focuses on optimizing the first impression of the email feed, i.e. the subject of the emails and the sender name or address, not the actual content.
   - **Automatically based on unique clicks** – the system automatically chooses the winner according to the amount of link clicks measured for each variant. Each link placed in the email's content is only counted once per recipient, even when clicked multiple times. This option is recommended if the primary goal of your email feed is to encourage recipients to follow the links provided in the email.
4. When using an automatic selection option, set the duration of the testing period through the **Select the winning variant after** settings.
5. Click **Save**.

The A/B testing period starts when a marketing automation process sends the email for the first time. While the test is running, the system randomly selects and sends one of the A/B test variants for each contact going through the process. Over the course of the test's duration, each variant should be sent to roughly the same amount of contacts.

After a winner is selected (either manually or automatically after the testing period ends), marketing automation always sends the email's winning variant.

## Evaluating A/B tests

The testing phase begins after the system sends out the first variant.

You can view the email tracking data measured for individual variants on the email's **Send** tab (or **A/B testing** tab for marketing automation emails), under the **Test results** section. The current tracking statistics are shown for each variant, specifically, the number of _Sent_ and _Delivered_ emails, the number of _Unique opens_ and the number of _Unique clicks_ performed by recipients. By clicking on the _Unique opens_ and _Unique clicks_ numbers, you can open a dialog with the details of the corresponding statistic for the given variant. Together with the absolute numbers, also the _Delivery rate,_ _Unique open rate_, and _Unique click rate_ are shown as percentage values.

> **Note:** **Apple Mail impact on open rate reliability**
>
> On September 20, 2021, Apple launched their Mail Privacy Protection (MPP) feature. Xperience cannot reliably track the open rate for recipients who use the **Apple Mail** app with the MPP feature enabled. This also includes users who use other email services, with their inbox linked to Apple Mail.
>
> Depending on the percentage of recipients with Apple Mail clients, Apple MPP may significantly impact the reliability of the open rate statistics. To mitigate this, we strongly recommend selecting your A/B test winner based on link clicks.

You can change the winner selection criteria at any point while the testing is still in progress. For standard emails, the test group slider is now locked. However, you can reschedule the sending of variants that have not been mailed yet using the selector and date-time field below the list.

Click **Select as winner**  to manually choose a winner (even when using automatic selection).

> **Info:** **Special cases with tied results**
>
> If a draw occurs at the end of the testing phase (i.e. the top value in the tested statistic is achieved by multiple email variants), the test postpones the selection of the winner and evaluates the variants again after one hour.
>
> In certain situations, you may need to choose the winner manually even when using automatic selection. For example, if you are testing the number of opened emails and all recipients in the test group open the received email.

Once the test is concluded and the winner is decided, the given variant is highlighted by a green background. You cannot perform any actions with the test except for viewing the statistics of the variants.

![](https://docs.kentico.com/docsassets/13/a-b-testing-marketing-emails/viewing_ab_test_result.png)

For standard marketing emails, selecting a winner opens a confirmation dialog where you can schedule when the winning email variant is sent to the remaining recipients who were not included in the test group. If you specify a date in the future, you will still have the option of choosing a different winner during the interval before the mail-out.

You can monitor the overall basic statistics of A/B tested emails directly on the **Send** (or **A/B testing**) tab of the email. If you want to see more detailed statistics, switch to the **Reports -> Overview** tab, where you can find statistics for all variants and the remainder together. When viewing the link click data on the **Reports -> Clicks** tab, you may use the additional **Variants** filter to display values for specific variants. The opened email data on the **Reports -> Opens** then also enables you to display all variants together. The statistics of the winning variant include both the corresponding portion of the test group and the remainder of the recipients who received the email after the completion of the testing phase.