The Results Are In: A/B Testing HTML vs. Plain Text Emails

The HTML vs. plain text debate has been raging for years. And when we asked you, we got a lot of mixed replies. Internally, we’ve been discussing the pros and cons of the plain text email and whether it has a place at Litmus. So, I decided there was only one way to settle this debate: to test it.

The beady-eyed among you may already know that Litmus already sends some stylized plain text emails. Yes, Litmus sends plain text emails! Don’t fall off your chair just yet.

Plain text emails have found a place in our email program, primarily those emails you receive when you download an ebook or report. We’ve also leveraged ye olde plain text for some communication that we thought would benefit from a more personalized outreach. For example, we use them for research requests and even some Litmus Live email campaigns.

And candidly, the plain text email has a much shorter turnaround time, which can make them handy in a pinch.

But are they really effective? Let’s dive into what our testing showed us.

Stay in the email loop with Litmus Weekly, your weekly digest of the newest and best Litmus content. Plus, get the best email links from around the web.

Subscribe today →

Our testing ground

We currently run a webinar program for customers (Leveraging Litmus) as well as our educational webinar series. To support the program and drive registrations, we send an initial email announcing the event to our audience, targeting customers for Leveraging Litmus webinars and non-customers for the educational webinars. Both of these initial email announcements are sent to the two audiences on the same day, at the same time.

This was our testing ground. Why?

• We wanted to learn if there was a difference between how customers and non-customers engage with plain text emails.
• The measure of success is easy with these emails—did the recipient register for the webinar?

We found a winner… or did we?

Both audiences were split 50/50 randomly and served a plain text style version or our regular HTML version of the email.

Customer

Our customers engaged a little bit better with our plain text style email, indicated by a slightly higher open and click-through rate. So we dug a little deeper to understand if the elevated email engagement was significant. And it was! Out of all the customers we emailed, 60% converted from the plain text style version.

Non-Customer

However, the email engagement tells a different story for our non-customers. This audience opened the plain text style email at a higher rate, however, the click-through rate was marginally lower. And that’s reflected in the conversions—49% of those we emailed converted from the plain text style version.

But this was a flawed test. Here, we weren’t just testing the format but also the messaging. If you read both versions of the email, the language is totally different.

So we ran another set of A/B tests:

1. For customers, we wanted to learn if the plain text style format and personal messaging drove more webinar registrations, so we pitted a plain text email against another plain text email and adjusted the style of messaging.
2. For non-customers, to confirm the format of the email didn’t matter, we re-ran the same test.

Customer

Non-customer

Confirming our findings

63% of the customer segment who converted from the email received the plain text, personal approach email versus our typical messaging used in an HTML email.

And for non-customers, of those we sent the emails to, 49% registered for the webinar on receiving the plain text style email—the exact same result as our first test! This seemed to confirm our hypothesis that for this segment, the email format doesn’t matter.

What did we learn?

From the tests we ran, we learned a few valuable things:

• Email engagement isn’t always a good indicator of the success of an email campaign
• We need to communicate to our customer and non-customer segments in different ways in order to achieve the same goal
• Plain text style emails do have a place in our email program

These learnings are far from definitive. Subscriber behaviors and how inbox service providers display email is constantly changing. There’s so much more we could learn from testing plain text style emails:

• Does including a headshot of the sender impact success?
• Is who the email is “from” important?
• Are HTML emails more distracting than plain text style emails?
• Long or short copy—what works best?
• What about other types of email—product updates announcements or the Litmus newsletter?

Some final thoughts

We email marketers can be quick to dismiss plain text emails. They can seem ugly, have a reputation of coming across as disingenuous, and feel “wrong.”

But we should understand that they have a place in our email marketing programs because they can:

• Add variety to how you speak to your audience, rather than always sending a promotional looking email.
• Change up how your emails visually look in the inbox. Sometimes your audience can get a bit numb to the same thing over and over again.
• Allow your brand to leverage a whole new type of messaging or voice that can bring some real personality to the table.

All I know is that I have a lot more to learn about our audience.

Have you pitted HTML against plain text? What did you learn?