A/B testing
How to compare two versions of your email and keep the one that performs best
A/B testing, or an A/B test, is a technique that consists of creating two versions of the same element to measure which of the two performs better.
In digital marketing, it’s used for many things. These are some of the most common:
- Choosing the best design for a website.
- Comparing two versions of an app.
- Finding the best newsletter for your email sends.
- Finding out which landing page gets more conversions.
- Comparing how effective two calls to action are.
- Testing specific design elements, such as buttons or images.
- Checking which copy sells the same product better.
In short: in email marketing, an A/B test sends two versions of a campaign to a small part of your list and then sends the winner to everyone else. Almost every tool has it, but what you can test and on which plan varies. In Mailrelay you can test the subject line or the sender on every plan, including the free account, and choose whether the winner is the version with more opens or more clicks.
Run your first A/B test for free
Test two subject lines or two senders and let Mailrelay send the winning version to the rest of your list.
How does an A/B test work?
The method consists of preparing two versions of the same element before launching it. It works for both physical and digital products. Here we focus on an email marketing example: choosing the better of two subject lines so that a campaign gets the best results.
You can run several tests in a row to compare more versions, but usually each A/B test is used to decide between two final options.
Imagine an animal shelter that wants to raise funds from its members. It prepares two subject lines for the same newsletter:
- One with the recipient’s name and a question whose answer is obvious to its list: “Name, don’t you just love animals?”.
- Another, more direct one that aims to get the recipient to read the email: “Our animals’ lives depend on you”.
The content of the email is exactly the same. Only the subject line changes.
To run the test, you choose a percentage of the list, for example 20%. Half of it, 10%, receives the first subject line, and the other half receives the second.
After a while, for example 24 hours, you check the statistics and pick the winning version. That version is sent to the remaining 80% of the list.

Winner by opens or by clicks? For years, opens were used, but since 2021 Apple Mail preloads the images of many emails and counts them as opened even if nobody reads them. That’s why clicks are a more reliable metric for deciding.
Which tools offer A/B testing for subject lines and content?
Almost every email marketing tool has A/B testing. The difference lies in what you can test and which plan includes it. This is the data as of November 2026:
| Tool | What you can test | Which plans |
|---|---|---|
| Mailrelay | Subject line or sender. The winner is chosen by opens or by clicks | All, including the free plan |
| Brevo | Subject line or content | From the Standard plan (not on Free or Starter) |
| Mailchimp | Subject line, sender, content or send time (one variable per test). Multivariate testing on Premium | Paid plans (not on the free plan) |
| MailerLite | Subject line, sender, content or send time. Multivariate testing on paid plans | Also on the free plan |
| Mailjet | Subject line, sender and content, up to 10 versions | Premium |
| HubSpot | Subject line, sender and content | Marketing Hub Professional and Enterprise |
Before choosing a tool, check two things: whether A/B testing is included in the plan you’re going to use and whether it lets you pick the winner by clicks.
What to test in your emails
If you’ve never run an A/B test, start with the subject line. Some ideas:
- The subject line with or without the contact’s name. We explain how in our post on dynamic variables.
- A question versus a statement.
- A short subject line versus a long one.
- With or without a number.
- The sender with the company name versus a person’s name. You’ll find more ideas in our email address entry.
Test only one thing at a time. If you change the subject line and the sender at the same time, you won’t know which of the two made the difference.
For important sends, I always recommend A/B testing the subject line. A good subject line can change the results a lot, so don’t go by gut feeling: test. Use subject lines that are really different, for example a question versus a number, not two sentences that differ by just one word. And don’t look only at opens: although it may not seem like it, the subject line also influences clicks. Look at both metrics together.
Jose Argudo, CMO at Mailrelay
How to run an A/B test in Mailrelay
In Mailrelay, the test is created from a newsletter you already have ready. You choose what you want to test (the subject line or the sender), what percentage of the list receives the test and whether the winner is the version with more opens or more clicks. You can pick the winner yourself or let the system send it automatically to the rest of the list.
You’ll find the step-by-step guide with screenshots in how to A/B test your newsletter in Mailrelay.
Benefits of A/B testing
Including A/B tests in your marketing strategy has several benefits:
- It increases the profitability of your actions. A/B tests let you easily find out what really works with your audience.
- It lets you experiment. What worked a few months ago may not work today. That’s why it’s worth testing constantly and in a controlled way.
- It improves engagement. With small changes, A/B tests show you how your audience prefers you to talk to them: the tone, the content they’re interested in, etc.
- The details make the difference. Things that seem minor, such as the size or color of a button, can make a piece of copy or a call to action work or not. A/B testing brings out the value of those details.
Let your data choose the subject line
A/B testing of subject line and sender on every plan, with up to 80,000 emails a month for free.
Frequently asked questions about A/B testing
What percentage of the list should I use for the test?
20% of the list, split between the two versions, is a good starting point. If your list is small, increase the percentage so that each version reaches enough contacts and the difference isn’t down to chance.
How long should I wait before choosing the winning version?
Many tests wait a few hours or a day, as in the example. The longer you wait, the more data you’ll have, but the later the winning version will reach the rest of the list.
Should I choose the winner by opens or by clicks?
Opens are useful for comparing subject lines, but Apple Mail counts many emails that nobody has read as opened. Clicks are more reliable. In Mailrelay you can choose which of the two decides the winner.
Can you run an A/B test for free?
Yes, with some tools. In Mailrelay, A/B testing of subject line and sender is included in every plan, including the free account, which lets you send up to 80,000 emails a month to 20,000 contacts.