Email A/B Testing done right

Email marketing professionals often assume that increasingly granular segmentation automatically leads to better results. Audiences are divided by region, demographic characteristics, or presumed interests, in the hope that greater relevance will lead to more opens, clicks, and ultimately more revenue. In practice, however, the reality is often quite different.

Why Segmentation Isn’t Always the Answer

A good example comes from an A/B test we conducted for a travel company. The idea was simple: We divided newsletter subscribers into those in northern and southern Germany and sent each group regional travel offers. The expectation was clear. Recipients who received offers from their own region should feel more engaged and click on the content more often.

At first glance, this approach sounded logical. The reality, however, was different. Both newsletter versions achieved nearly identical results to a non-segmented mailing. Neither the open rate nor the click-through rate improved significantly. Regional segmentation also had no measurable impact on revenue.

Instead, a completely different pattern emerged. Recipients who had actively engaged with newsletters in the past accounted for the majority of clicks in this test as well. Highly active users opened and clicked again. Users with low activity, on the other hand, remained largely inactive—regardless of how closely the content was tailored to their region.

We observed this behavior not only in this test. In other projects as well, it became clear time and again that a contact’s overall activity level has a significantly greater impact on a campaign’s success than many traditional segmentation criteria.

Why Over-Segmentation Is Often Overrated

Many companies and email marketing agencies invest a great deal of time in creating increasingly detailed target audience structures. This results in segments based on regions, company sizes, industries, interests, or other characteristics. The underlying assumption is that greater relevance automatically leads to better results.

In practice, however, such oversegmentation often provides little additional benefit. At the same time, the effort required to create campaigns increases significantly. Furthermore, the individual target groups become smaller, making A/B tests statistically less meaningful.

In our experience, many segmentation approaches are far more complicated than necessary. Companies often focus on theoretical relevance, whereas the actual behavioral data of their contacts would be far more meaningful.

Which Segmentation Criteria Are Truly Important

Across many projects, two criteria in particular have proven to be especially valuable.

The first criterion is a contact’s activity. Anyone who regularly opens newsletters, clicks on content, and interacts with the company is highly likely to continue doing so in the future. Past activity is therefore one of the best indicators of future behavior.

The second criterion is the most recently purchased product or product group. This characteristic is based on actual purchasing behavior and provides valuable information about a contact’s interests. For example, someone who most recently purchased a specific product category is often also interested in similar offers or complementary products.

Both criteria are based on real behavioral data and are therefore, in many cases, significantly more meaningful than geographic or demographic characteristics.

Setting Up Email A/B Tests Correctly

If you want to conduct email A/B tests successfully, you should focus on the factors that actually influence user behavior. These include, for example, subject lines, how offers are presented, calls to action, or send times.

It’s important to test only one variable at a time whenever possible. If multiple changes are made simultaneously, it becomes nearly impossible to determine later which adjustment was actually responsible for the result.

Equally important is having a sufficiently large test group. Target groups that are too small often result in differences arising purely by chance, making it impossible to draw reliable conclusions.

Furthermore, companies should not focus exclusively on open rates. Clicks, conversions, and ultimately revenue are far more meaningful metrics when it comes to evaluating the actual success of a campaign.

Conclusion: User Behavior Trumps Complex Segmentation

Conducting email A/B testing correctly does not mean building increasingly complex target audience structures. It is far more important to use the data that actually allows for inferences about future behavior.

Our experience shows that a contact’s activity and purchase history are significantly stronger influencing factors than many traditional segmentation approaches. Companies should therefore first ensure that they are making effective use of this data before investing time and resources in increasingly granular segmentation.

The most important insight is this: A user’s actual behavior is usually the best predictor of future interactions. Those who take this into account will achieve better long-term results with their email campaigns.