Introducing Improved A/B Testing and Message Checkup

A/B testing is one of the most common practices in marketing. There’s a ton of literature on it, and MailUp created ebooks, blog posts, and other marketing material to stress how important it is to reaching better campaign metrics.

We have set up our A/B tests for a complete makeover, which included redesigning the user interface and rewriting both the frontend and the backend code, to provide you with the best possible experience.

We also added these relevant features:

• Test segment can now be anywhere between 5% and 100% of the recipients (previously it couldn’t be more than 30%).
• When selecting 100% as test segment, no winner message is created. Statistics for this tests are used for running multivariate testing.
• Winner messages can now be tagged during setup, for improved sorting and statistics.
• Messages that are already queued for sending can now be part of an A/B test.
• Pausing an A/B test is now as easy as clicking a button.

Setting up an A/B test

We created a smooth, single wizard to help you navigate the setup.

Step 1. Main settings for a new A/B test

Here is where you can enter the main settings to get started with A/B testing:

• Test name
• Variants, now including a full search and thumbnails
• Test segment slider
• Winning message section to determine criteria and tags

Step 2 & 3. Sender & Recipient Selection

These are the same steps in use before, but they have been integrated in the wizard, thus significantly improving the user experience.

Step 4. Queuing & Review

A progress bar tracks the two operations that occur behind the scene:

• Queuing all recipients
• Sorting test messages randomly for the two testing groups

As opposed to the previous version, this procedure has now been optimized to the point that it runs around 4 times faster – boom!

Once loading is complete, you get a nice, full preview of how the A/B test has been set up. If the winner is decided automatically, it will be scheduled here.

Listing A/B tests

New and improved, with advanced search, plus clear details and status for each test. The Actions available for each test are now fully coherent to its status.

Any action to edit or alter an A/B test is managed from here, where in the past some actions were (wrongly) available from the Send status section. Speaking of Send status, clicking Manage A/B test from a message involved in an A/B test now redirects to the Details page for that test.

Viewing test results

Results are now much cleaner and easier to read, including a preview for all messages. Aggregate reports for each A/B test are still available.

Depending on test status, it is possible to reschedule test & winner messages, and pause/resume the entire A/B test.

One of the main jobs that a professional Email Marketing platform needs to perform is to help you answer a simple, but key question: is the message ready to go? That includes ensuring that you avoid issues that may negatively affect deliverability, and recommending best practices that can help increase the campaign’s performance.

In MailUp, the Collaboration feature helps you decide whether the look and the content of the message are ready by sharing it with all key stakeholders.

The Message Checkup feature, instead, focuses on determining whether the message is healthy. To do so, the system runs a number of checks, and many of them have been there for quite some time. The problem was… the feature was not visible enough. Despite being an area where we created a lot of value for our users, many of you weren’t taking advantage of it.

Checkup box in message Overview

In the Overview page for a message, a new Message Checkup box pops up right under the message thumbnail. In case of issues, the relative severity is highlighted, and a click redirects to the Checkup page for more details.

New checkup page

All checks are now in a single page, ordered by severity, with extended explanation of what they mean and how to act.

In this case, a potential issue is related to the absence of an unsubscribe link: everything else looks nice and ready to go.

Checkup alerts

In case the message presents some issues, a modal window pops up when saving or sending that message. These alerts are user-defined, so they can be turned off (for a specific message or for any message).

Here are some other improvements you will find from today on.

• You can now create a copy of Subscription and Profile Updates forms created with the drag & drop editor
• Revisited list selector, now bigger and clearer, with instant search (available when account has 10+ lists)
• Email messages search now by default searches only in subject, notes and ID. Full-text search inside content is optional, so standard queries are much faster

That’s a wrap! As usual, feel free to leave a comment and let us know what you think of the improvements you’ll see in your platform. If you haven’t tried MailUp yet, it’s probably time to give it a go.