How to create an A/B test?

An A/B test is a way to split your message deliverability into testing groups that will help you decide which version performs better. We have a dedicated section just for A/B tests from which you can create, manage, and send them. We will present your variants and the winning message in an intuitive way. You also create them using our state-of-the-art editor, which makes creating beautiful messages a breeze.

How to create an A/B test

In order to create an A/B test:

  1. Go to Menu>> Email marketing>> A/B tests.
  2. Click Create A/B test.
  3. Provide your A/B test name – it will only be visible to you.
  4. Select from and reply-to addresses.
  5. Add your Subject lines variants. You can test up to 5 different subject lines to check which one gets you the most opens. For example, you can try personalizing your subject line, adding emojis, or including an offer.
    ab test set up.
  6. Select your recipients by clicking Add recipients.
  7. Design your message by clicking Design message. For more on using the new editor visit this section.
  8. Click Configure A/B test to set up how you want the test to run. You can use the slider to decide on the size of the testing group. You can test it with up to 50% of your recipients. Then select if you want us to send the message automatically to the rest of the group or if you’d rather do it manually. Next, decide what should be the winning factor (open rate or click rate) and after what time do you want the test to end. We will show you a quick summary of your test group. Once you’re done, click Save.
    Configure your AB test.
  9. Click tracking will be enabled by default to see which contacts clicked your links. This will help you estimate engagement.
  10. Select if you want to send clicks coming from the message as a source to Google Analytics.
  11. Choose if you want to send the test immediately or at a scheduled time.
  12. You can also enable the perfect timing.
  13. Now you can either save the message as a draft or start the A/B test.
    start ab test.

What statuses can my message have?

  • Preparing – we are sending your message to the testing groups.
  • Scheduled – the message is scheduled for delivery.
  • Testing – we have sent the samples and are testing variants results.
  • Testing finished – the samples have been tested and the results are ready to choose the winner (if you’ve chosen to manually select the winner).
  • Sending winner – we are sending the winning message to the rest of the recipients.
  • Cancelled – the message sending has been cancelled by the user.
  • Completed – the test is completed and all the variants, as well as the winning message, have been sent and tested.
  • Draft – the message is a draft and is waiting for completion.

How do I choose the winning message and where can I see it?

If you selected to choose the winner manually, you can expand your A/B test and then choose the winner from the Actions menu (vertical ellipsis) by clicking Send as winner.

You will be able to see your winning message above the variants. There you will see how many messages were delivered after the test stopped and what was the open rate and click rate for it. You will also be able to check detailed statistics for this message.

Where can I see the statistics?

You can see your message statistics if you go to Menu>> Email marketing>> A/B tests. There you will be able to see the overview results for the A/B test (delivered messages, overall open rate, overall click rate). Once you expand the section, you’ll get the results divided into 2 categories:

  1. Winner results.
  2. Variant results.

Each of them will have delivered messages, open rate, and click rate statistics.
In the statistics for each variant, we will only show the opens and clicks generated during the duration of the test. For example, if the testing period was 2 days, we will only show data collected during these 2 days, not in the following days after the test ended.
Winner results will include delivered messages, open rate, and click rate both from the testing period and after sending the winner.

For more statistics on the winner, you can click View Statistics.

winner statistics.

Legacy editor

How to create an A/B test in the legacy editor?
Whenever you create a newsletter message in the legacy editor, you will find it at the bottom of the first page, in the Distribution settings section.

Distribution settings.

Turn on the A/B Test switch to enable it. Once enabled, the newsletter creator will be extended with additional steps where you define the type of test and its settings.

After you create the message in the editor, click Nest step. You will be redirected to the A/B test creation step where you can choose what you want to test in A/B test.

choose ab test.

  • For the subject, you can create up to 5 different subjects to test.
  • For from field, you can test up to 4 different addresses.
  • For content, you can choose or create up to 5 different messages.
  • For delivery time, you can select up to 5 different days of the week or times of the day.

After clicking Next step, you will be taken to the Choose recipients step where you can configure your test recipients and how the test and winner message will be delivered.

ab test configuration.

On the summary page, you will see which type of test you’ve chosen and a summary of your message configuration. Then you can send the test immediately or schedule it for later.