Holistic Email Metrics Matrix: Are You Seeing the Whole Picture?

Email analytics are powerful… and also dangerous. Used correctly, analytics can inform both wise strategic decisions and smart tactical improvements. Used incorrectly, analytics give you complete confidence to make bad decisions. That’s why looking at email metrics holistically is important.

Learn the four parts of the Holistic Email Metrics Matrix and what to track for each:

1. Direct business impact
2. Indirect business impact
3. Subscriber relationships
4. Inbox provider relationships

The danger of email analytics

But first, why is a holistic view of metrics important? Unfortunately, there are lots of ways to reach the wrong conclusions with analytics, including:

• Not reaching statistical significance and making other mistakes with your A/B tests
• Using the wrong metric to measure success, like unwisely using open rates to determine a winning subject line
• Not recognizing the limitations of your ability to accurately measure a particular metric, especially complex ones like email marketing ROI
• Being overly focused on campaign metrics like opens and clicks, and failing to look at performance over time and deeper engagement metrics
• Not recognizing the influence of your email marketing efforts on the performance of other channels like direct mail and social media
• Focusing too much on company-centric metrics like email marketing revenue and not enough on metrics that indicate how subscribers and inbox providers feel about your emails

Those last three are symptoms of not seeing the big picture when it comes to email marketing metrics. Our Holistic Email Metrics Matrix provides that perspective.

4 perspectives you must consider

To avoid the danger of incorrectly analyzing email performance, you must have an eye on the whole picture:

Our Holistic Email Metrics Matrix shows that the response to an email campaign ripples outward, affecting your:

• Direct business impact in terms of revenue and leads generated, for example
• Indirect business impact as part of omnichannel campaigns or by influencing the performance of other channels more generally
• Inbox provider relationships and how they perceive you as an email sender
• Subscriber relationships and how they perceive your company and emails

Email marketing campaigns can be felt individually in the short term—as well as cumulatively over the long term as part of your overall email program. For instance, a particular email could do serious damage to your sender reputation, but you might not see the damage manifest in terms of blocks or a blocklisting until you’ve sent several more subpar emails.

Let’s break down each quadrant of the Holistic Email Metrics Matrix and which short- and long-term metrics are in each…

1. Direct business impact

These metrics help brands measure how their email program and campaigns are directly affecting their sales, profits, web traffic, and other important business key performance indicators (KPIs).

Email campaign success metrics

In the short term, email metrics like conversion rates and revenue demonstrate the direct value of sending a campaign. While the goal of one email may be to generate registrations of a webinar (lead generation) and another may be to create sales (conversions), these are generally the top metrics for determining the success of an email campaign. For example:

• Duration of engagement
• Clicks and click-through rate
• Post-click behavior (e.g. add to cart)
• Conversions and conversion rate
• Email revenue and profit
• Lead generation

What about the open rate?

While still useful in some cases—like measuring interactive emails, awareness, or readership—you’ll want to start looking at email engagement in new ways. Apple’s Mail Privacy Protection will make the open rate increasingly difficult to rely on.

See more of the email picture Dig deeper than the data your email platform gives you, and better understand your audience and the performance of your email campaigns. Go beyond opens and clicks →

Email channel success metrics

In the long term, metrics like email program revenue show the overall success your email channel is having. For a more subscriber-centric view of revenue, look at revenue per subscriber and subscriber lifetime value. Here are more examples:

• Email program revenue & profit
• Email revenue per email
• Email revenue per subscriber
• Subscriber lifetime value
• Subscriber RFM (recency, frequency, monetary)
• Email marketing return on investment (ROI)
• Lead pipeline value, etc.

2. Indirect business impact

Subscribers don’t always follow the highly measurable golden path you want them to. Sometimes they see an email, don’t open it, but then go to your store or dial your call center and convert because of it. Other times, they open an email, then open their browser and type in your URL directly and convert.

Because of these and other common behaviors, marketers can significantly under-measure email marketing’s effectiveness if they’re not connecting the dots.

Email campaign’s omnichannel influence metrics

In the short term, looking at the activity your brand experiences during the 48 hours after an email is sent can give you a good idea of the omnichannel influence of a particular email campaign. For example:

• Web traffic and sales
• Mobile app activity
• Store traffic and sales
• Social media activity
• Direct mail effectiveness
• Call center volume

Keep in mind that your goals may be changing from email to email, and also that one of your goals might be to reduce activity in another channel, such as call volume to your call center.

Email program’s omni-channel influence metrics

In the long term, looking at the activity your brand experiences during the 48 hours after you send an email versus the days when you don’t send email can reveal the omnichannel influence of your email program in general. You’ll want to look at some of the same metrics plus a few others to find the indirect effect:

• Web traffic and sales
• Mobile app activity
• Store traffic and sales
• Social media activity
• Direct mail effectiveness
• Call center volume
• Customer lifetime value of subscribers vs. non-subscribers
• Customer RFM (recency, frequency, monetary) of subscribers vs. non-subscribers

Email metric to watch: Comparing the customer value of subscribers compared to non-subscribers is a great way of measuring some of the otherwise hard-to-measure effects of email campaigns on customer behavior.

For example, if subscribed customers spend $147 per year on average, and your non-subscribed customers only spend $100, that difference speaks to the effectiveness of your email program, regardless of the level of email engagement of those customers.

3. Subscriber relationships

Email marketing is a give-and-take channel. In order for brands to win with email, subscribers have to feel like they’re winning as well. Maintaining a healthy subscriber relationship shows that your emails are giving your subscribers value.

Subscriber reaction metrics

In the short term, email metrics like opens and clicks indicate positive subscriber reactions, while unsubscribes and complaints indicate negative subscriber reactions.

• Opens and open rate
• Clicks and click-through rate
• Click-to-open rate
• Unsubscribes and unsubscribe rate
• Spam complaints and complaint rate

Email list health metrics

In the long term, the reactions to those individual campaigns affect the size of your list and how engaged that list is, which are key signs of the overall health of your email marketing list. Watch for:

• Active list size
• List growth and churn rate
• Inactivity rate
• Length of inactivity
• Open reach (percentage of subscribers who opened at least one email in past X days)
• Click reach (percentage of subscribers who clicked at least one email in past X days)
• Tracking metrics by acquisition source

Email metrics to watch: Often, a good portion of the negative reaction to your emails will come from one or two acquisition sources—such as email list rental and co-registration, which are among the five most problematic subscriber acquisition sources. Tracking email performance by acquisition source can reveal which ones are causing outsized problems and are in need of revamping or being shut down.

Healthy email list. Healthy ROI. Subscribers fuel your overall email program success. Learn how to get more good subscribers. And more revenue. Get the guide →

4. Inbox provider relationships

In addition to pleasing subscribers, email marketers also have to please inbox providers like Gmail and Yahoo! Mail, each of which can have different expectations for senders. Inbox providers pay closer attention to how your subscribers react to your emails—not supposed spam trigger words—including tracking many actions that marketers can’t see, such as foldering and deletes without opens.

However, they also look at how you treat their platform.

For instance, if you generate a lot of hard bounces by trying to send email to accounts that don’t exist on an inbox provider’s platform, that tells them that you may be buying email addresses, guessing at email addresses, not removing addresses that previously bounced, or not doing enough to avoid typos and transcription errors. Similarly, if you’re hitting old inactive email accounts that the inbox provider has recycled into spam traps, then it tells them you’re not paying attention to list hygiene.

Email engagement metrics

In the short term, inbox providers look at behaviors that demonstrate that subscribers are engaged with your emails. Unfortunately, some of these behaviors—such as foldering, deletes without opens, and spam trap hits—are impossible for marketers to see.

Other behaviors, such as forwards and duration of engagement, can’t be seen without special analytics such as Litmus Email Analytics. Even spam complaints can be tricky, since Gmail doesn’t report them through a traditional feedback loop.

But here’s some of what you can look at:

• Opens and open rate
• Duration of engagement
• Deletes without opens
• Replies
• Forwards
• Social shares
• Foldering or archiving
• Soft and hard bounce rates
• Recycled spam trap hits
• Spam complaints and complaint rate

Deliverability metrics

In the long term, how subscribers of a particular inbox engage with your emails and whether you abuse their platform adds up, potentially leading to deliverability problems.

Inbox providers look at engagement, bounce rates, and spam complaints over a period of time (often 30 days). So if one campaign has a spike in negative metrics, that probably won’t cause deliverability issues. But if you have a string of campaigns like these, then you’re likely in trouble with one or more inbox providers.

Keep an eye on metrics like:

• Delivered rate
• Deliverability or inbox placement rate
• Blocks
• Blocklistings
• Tracking metrics by inbox provider

Inbox problems have different tolerances for certain behaviors, which is due in large part to the different demographics and attitudes of their user bases. So you might experience problems with, say, Gmail and not AOL. You’ll only know if you track email performance by inbox provider.

See the whole picture

Many email programs are focused almost entirely on the direct business results of their email efforts. That’s due in part to the fact that that’s how they’re measured by executives.

That’s unfortunate because in many organizations, up to half of email’s outcomes will be felt in other channels. Proving those effects and getting credit for them could have a massive ramification on email marketing budgets.

Only focusing on company-centric metrics exposes brands to growing subscriber inactivity and deliverability issues. To have a well-balanced, high-performing email program, brands have to have a broad, four-quadrant approach to email metrics.

Want more resources like this? Subscribe to Litmus emails to get your regular dose of email inspiration, tips & tricks, and the latest trends & insights—right in your inbox. Get on the list →

Originally published on July 6, 2018, by Chad. S. White. Last updated July 8, 2021.