A/B tests: everything you can test to improve results

The A/B test is an exercise in humility for even the most experienced marketer. On one hand, testing means researching what works best, and on the other, questioning what has been done up to now. Questioning means knowing how to change and evolve, which is the very principle of marketing.

In this post we want to explain the theory, but even more so the practice of this fundamental email marketing activity.

An A/B test, also known as split testing, means submitting two or more different versions of the same message to a sample of recipients, analyzing the reaction to each version, and determining which one is most effective. The goal is to optimize individual aspects of the email with the aim of improving performance not just by following intuition or common sense, but based on actual audience response.

In a two-variant A/B test:

• The “neutral” version is called the control
• The variant containing the changed element to be tested is called the treatment

Now let’s take a look at the A/B test from the perspective that is most important to marketers. We have identified four fundamental KPIs, and we have turned them inside out to analyze the elements which would allow an A/B test to improve email performance.

Improve open rates

The performance of an email’s subject is influenced by subtle nuances, which must be carefully weighed. The elements on which to perform the tests are as follows:

Copy

Changing one or two words is enough to give a decisive boost to the open rate. Here is an example using an imaginary app as the sender for a job search:

Variants
A: Test Jobs available with a 50% discount
B: Test Jobs available to establish a career

Audience
Customers

Metrics
Open rate

Outcome
Subject B had a +13.3% open rate compared to subject A

Takeaway
Customers are more excited about the idea of ​​”establishing a career” than being enticed by a discount.

Approach: affirmative or interrogative?

As a first step to the winner’s podium, the subjects of the emails sent by former US President Barack Obama certainly took note of this:

However, what if the White House had opted instead for the interrogative version – Hey? – would the campaign have turned out the same way? We will never know, but all marketers know the effectiveness of an interrogative subject line. Creating A/B tests on this text is essential: for example, try comparing “Create your free profile today!” with “Do you want to create a free profile?”

Length

Many marketers recommend keeping the subject line between 28 and 38 characters; however, this should not be taken as an excuse not to test, as there are so many variables. We know that iPhones only show 35-38 characters in portrait mode (80 horizontally), and the number varies (often decreasing) based on the size of different screens. On desktops, on the other hand, it is possible to provide a more detailed description and be more convincing. That is why it is always important to perform A/ B tests to compare a more concise and a more conversational version.

Customization

From length, we can now move on to the degree of customization: try conducting an A/B test comparing a generic, common subject, with one that has a dynamic field that automatically enters the recipient’s name. You can also speak directly to the user, on a first name basis.

The second level of the subject: the preheader

The preheader is that short text that is displayed after the subject in the recipient’s inbox which gives the recipient an extra chance to get an idea of ​​the email’s content.

The preheader is not just a service element available to recipients, but a strategic element that works to convince users to open the email. Including and improving a preheader leads to an increase in open rates, CTRs, and conversions. For this reason, it is advisable to also experiment with this level of content.

The effectiveness of emojis

Small but showy, frivolous but attractive. Emojis are ambiguous elements that are increasingly used in brand communication.

In this realm as well, the advice is to test if open rates show an upsurge with emoji use. You can also compare different emojis to understand which one is most effective. One thing is certain: some are used more often, as evidenced by this study of the most used emojis in email subjects:

Source: MailChimp

Improving CTR

We are now at the deepest level of the funnel: the click rate, which is most likely the most important metric to marketers. Here most of the text will inevitably hinge on the call-to-action, that focal point of the email that can be likened to the “waiting room” of conversion.

Color

Using contrasting colors, in reference to the email color scheme, is one solution for distinguishing the action to be performed. As a word of caution, don’t overdo it: one wrong choice could ruin the whole communication, and push clicks away.

Also consider that it is always a good idea to reference the corporate brand identity colors within an email.

Size

The size of the button can make a difference when it comes to CTA. The button should be big enough to be obvious, but if excessively large, it could be deemed too aggressive, thereby hindering the user experience.

The display on mobile devices is also an element to consider when choosing a button size. Even in this case, testing is best for understanding what is most effective.

Wording

One example is the classic “Buy now“. Clear and concise, this CTA could, however, worry less accustomed readers by suggesting a “forced” purchase just after the click. In this case we recommend testing softer solutions, such as “Add to Cart” or “Continue with purchase”. This is just an example to help you understand how text is also important for driving clicks and conversions.

Special effects

Buttons can be highlighted with particular shapes. The variables could involve:

• The roundness or angularity of the corners
• Three-dimensional effects or a flat design
• Shadows or lack thereof

The number of CTAs

There are campaigns where goals lead to a dilemma, in the sense that they diverge and force the marketer to make a choice: focus on a single call to action or dare to add two? The A/B test will give you a more or less unequivocal verdict.

Improve sending times

Identifying the right time for email delivery works as a driving force for open and click rates. Well-established hours in the middle of the day are not necessarily the best. Depending on the type of business, the sector of activity, and the target audience, the best times can vary greatly: in general, B2B companies get better performance during “desk” business hours, while B2Cs get better performance at all other times.

The best thing is to perform an A/B test for each of your sending categories (newsletter or DEM) to find out:

• The best day of the week
• The best time slot

Improving response rate

This is more of a niche metric, as email marketing campaigns that target the recipient’s response are not as common. For many, however, such as sales people, this is a fundamental indicator. Here are some points on which to carry out the all important A/B tests.

Comprehensive or concise body copy

Can your emails be scanned and understood in 8 seconds? If so, great, otherwise you risk losing the attention of your recipients.

Take an email template you have sent in the past and cut its content. You can do it yourself or let an application such as Hemingway help you. Then take the two versions of the email and compare them with an A/B test.

Degree of customization

As already mentioned above for the subject, customization with dynamic fields is now a must in email marketing. Try testing this element in the body of the email as well, calling the recipient by their name, including their company name, or customizing more in depth by specifying the recipient’s business role.

Conversational formulas

This is an even larger, more complex field, and is even more difficult to measure.

Many marketers and sales consultants use templates with predefined text. Often a few small tweaks or the reorganization of content are all that is needed to get increased response rates.

Testing is part of the DNA of email marketing. It is a highly immediate tool which only requires a few steps to compare different versions of a campaign.

If you haven’t tried MailUp yet, we suggest requesting a free 30-day trial of the platform. In just a few clicks you can experience what works best for your campaign and improve performance.