Testing email subject lines honestly

Subject line tests are popular because they are easy to run: send two versions, see which gets more opens, declare a winner. The trouble is that the easy version of the test often tells you very little. This article covers the mistakes that make results unreliable and a more careful way to run a test.

Why open rates mislead

An open is recorded when an image in the email loads. Some mail apps, Apple Mail Privacy Protection among them, can load images automatically, so an open may be counted when nobody read the message. Other readers block images, so real reads are not counted. Open rates are therefore a noisy measure, and the noise is not the same across audiences.

Better signals:

• Clicks on a link in the email.
• Replies, for emails that invite them.
• Conversions that follow the click, such as a booked call.

If you still use opens, use them to compare two versions sent to the same kind of list at the same time, and do not treat the figure as an absolute.

Test one thing

If version A is short and plain and version B is long, contains a number and uses a question, you cannot say which change mattered. Change one element:

• Length
• A question versus a statement
• Including the recipient's company or topic
• A specific detail versus a general claim

Write down the single difference before you send.

Write the hypothesis first

A useful test begins with a sentence: "A subject line that names the topic will get more clicks than one that teases." Then choose the measure that will decide it, before you see results. This stops you from picking whichever number favours your preference afterwards.

Split the list properly

1. Choose recipients at random into two groups. Do not split by alphabet, signup date or region, since each can differ in behaviour.
2. Send both versions at the same time. Different send times add another difference.
3. Make the groups the same size.
4. Keep every other part of the email identical, including the sender name and the preview text.

Most email platforms offer a built-in split test. Check how it chooses a winner and whether it sends the rest of the list the winning version automatically, since a small test group can pick a winner by chance.

Be realistic about small lists

With a small list, differences between versions are often just chance. If one version leads by a few clicks, a repeat of the test could easily reverse it. Some honest options:

• Run the same test across several sends and look at the overall direction.
• Test bigger differences, which are easier to detect than small ones.
• Accept that a small list may not support firm conclusions, and use the exercise to learn from replies instead.

Do not stop the moment one version looks ahead. Decide the sample before starting and read the result once.

Keep a record

Create a simple table with date, audience, hypothesis, the two subject lines, the measure, results, and what you will do next. After several tests, patterns in your own audience become visible, which is worth more than a generic list of tips.

Don't forget what follows the subject line

A subject line only gets someone to look. If the email itself does not deliver on its promise, a high click rate for a misleading line can reduce trust and increase unsubscribes. Check that:

• The subject accurately describes the content.
• The preview text adds something instead of repeating the subject.
• The sender name is recognisable.
• You are not using urgency that is not real.

A short protocol

1. Write the hypothesis and the deciding measure.
2. Change one element.
3. Split randomly and send together.
4. Judge on clicks or replies, not opens alone.
5. Record the result and repeat before drawing wide conclusions.

To plan tests across email, ads and landing pages with a team that measures results, measured campaign testing is among the services Gorilla Vibe packages into its subscription.

Written with AI assistance for the Gorilla Vibe marketing team.