A/B testing marketing emails

When preparing an email feed for marketing purposes, it can be difficult to predict how recipients will react to important elements, such as the email subject or the primary call to action. You may also want to find out which option out of several alternatives will have the most positive effect. Optimizing your marketing emails via A/B testing provides a possible solution to these problems.

A/B testing allows you to create multiple different versions of each email called variants. You then mail out the variants to a test group of recipients, typically a portion of the marketing email’s full mailing list. The best version of the email can be identified based on the tracking statistics measured for the test group, and then sent to the remainder of the recipients. The winning variant can be either selected automatically by the system according to specified criteria or manually after evaluating the test results.

A/B testing works best for marketing emails that have a large number of recipients. With too small a testing group, the results may be heavily affected by random factors and will not be statistically significant.

Prerequisites

On-line marketing must be enabled for your website:

1. Open the Settings application.
2. Navigate to On-line marketing.
3. Select Enable on-line marketing.
4. Click Save.

A/B testing variants of marketing emails are evaluated based on the actions performed by recipients, so both types of Email tracking need to be enabled for tested emails:

1. Open the Email marketing application on the Email feeds tab.
2. Edit () an email campaign or a newsletter.
3. Switch to the Configuration tab.
4. Make sure Track opened emails and Track clicked links are enabled.
5. Click Save.

Creating A/B tests for marketing emails

You can define A/B tests in the Email marketing application, either directly when writing new marketing emails, or when editing existing marketing emails on their Content tab.

1. Click Create A/B test to add the first testing variant for the email.
   

2. Type a Name for the variant.

   • The name will be used to identify the email variant while working with the A/B test.

3. Choose one of the following options to determine the initial content of the variant:

   • Create empty variant – select this option to create the variant from scratch. The variant will use the main template set for the email feed on the Configuration tab and the content of its editable regions will be empty.
   • Copy content from another variant – if selected, the content of an existing variant (or the original email) will be used as a starting point that you can modify as required. Choose the source from the list of emails. The variant will use the same template as the source email and all editable region content will also be copied. This option makes it easy to create variants for testing small changes, such as a different email subject or text headline.

4. Click Save & Close.

Now that you have defined an A/B testing variant for the email, the content editing page shows a slider at the top. You can use the slider to switch between individual variants, including the original email. The slider shows the name of the currently selected variant. You can manage the A/B test variants using the following buttons:

• Add variant – creates another variant of the email. You can add any number of variants.

• Remove variant – deletes the variant currently selected through the slider.

• Edit properties – allows you to change the name of the currently selected variant. If required, you may also rename the original email.

  

You can modify the settings and content of variants just like when editing standard emails. Each variant may have a different subject, email template, editable region content etc. This allows you to test any variables that you need.

Sending A/B test marketing emails

Once you define all of the email’s variants, you need to configure how the test is sent out and evaluated. You can do this on the Send tab of the email that you created.

1. Define the size of the recipient test group using the slider in the upper part of the page.

   • By moving the slider’s pointer, you can increase or decrease the number of recipients that will receive the variants of the marketing email during the testing phase.

- The slider automatically balances the test group so that each variant is sent to the same amount of recipients.
- The overall test group size is always a multiple of the total number of variants created for the email.
- The remaining recipients who are not part of the test group will receive the variant that achieves the best results (i.e. the winner) after the testing process is complete.

2. In the Schedule mail-out section below the slider, specify when to send out individual email variants to the recipients from the corresponding portion of the test group.
   • To schedule the mail‑out, enter the required date and time into the field below the list (you can use the  Calendar selector or the Now link) and click OK.
   • You can either do this for (all ) variants or only for those selected in the list.
   • If the mail-out time is the same for multiple variants, the system sends them in approximately 1-minute intervals between individual variants.
3. In the Winner selection section, select how the A/B test chooses the winning variant:
   • ****Manually**** – the A/B test does not select the winner automatically. Instead, the author of the email (or other authorized users) can monitor the results of the test and choose the winning variant manually at any time.
   • Automatically based on unique opens – the system automatically selects the variant with the highest number of opened emails as the winner. This type of testing focuses on optimizing the first impression of the email feed, i.e. the subject of the emails and the sender name or address, not the actual content.
   • Automatically based on unique clicks – the system automatically chooses the winner according to the amount of link clicks measured for each variant. Each link placed in the email’s content is only counted once per recipient, even when clicked multiple times. This option is recommended if the primary goal of your email feed is to encourage recipients to follow the links provided in the email.
4. When using an automatic selection option, set the duration of the testing period through the Select a winner after settings.
   • This allows you to specify how long the system waits after the last variant is sent out.
   • When the testing period is over, the test selects the winner and sends it to the remaining recipients.
5. Click Send.
   • If you only wish to save the configuration of the A/B test without actually starting the mail-out, click Save instead.

Evaluating A/B tests

The testing phase begins after the system sends out the first variant. If you need to make any changes to the configuration of the A/B test or the content of its variants, you can edit the email.

On the Content tab, you may modify the variants that have not been mailed yet, but the slider actions are now disabled. It is no longer possible to add, remove or rename variants.

If you switch to the Send tab, the test group slider is now locked. However, you can view the email tracking data measured for individual variants in the Test results section. The current tracking statistics are shown for each variant, specifically, the number of Unique opens and the number of Unique clicks performed by recipients. By clicking on these numbers, you can open a dialog with the details of the corresponding statistic for the given variant. It is also possible to reschedule the sending of variants that have not been mailed yet using the selector and date-time field below the list.

You can change the winner selection criteria at any point while the testing is still in progress.

Click Select as winner to manually choose a winner (even when using automatic selection). This opens a confirmation dialog where you can schedule when the winning email variant is sent to the remaining recipients. If you specify a date in the future, you will still have the option of choosing a different winner during the interval before the mail-out.

Once the test is concluded and the winner is decided, the given variant is highlighted by a green background. At this point, the system mails out the winning email to the remaining recipients who were not included in the test group. You cannot perform any actions with the test except for viewing the statistics of the variants.

You can monitor the overall basic statistics of A/B tested emails right on the Send tab of the email. If you want to see more detailed statistics, switch to the Reports -> Overview tab, where you can find statistics for all variants and the remainder together. When viewing the click-through data on the Reports -> Clicks tab, you may use the additional Variants filter to display values for specific variants. The opened email data on the Reports -> Opens then also enables you to display all variants together. The statistics of the winning variant include both the corresponding portion of the test group and the remainder of the recipients who received the email after the completion of the testing phase.