Subject-line A/B tests

How a test works

A subject-line test sends your post to a share of your list first, split evenly between two to four subject lines. After a wait you choose, the subject line with the best open rate or click rate goes to everyone else.

For example, with 10,000 readers, a 20% test group and two subject lines: 1,000 readers get subject line A and 1,000 get subject line B straight away. Four hours later, B has the better open rate, so the other 8,000 readers get B.

Each reader gets the post once. The test group and the rest of your list never overlap, and readers who subscribe during the wait get the winner with everyone else.

Setting one up

1. Open the post and go to the Email step.
2. Under Test subject lines, turn the switch on.
3. Subject line A is the post's subject line, set above. Click Add subject line to add B, and up to C and D.
4. Choose:
   • Test group: the share of your list that gets the test, from 10% to 50%.
   • Pick the winner after: 1, 2, 4, 8 or 24 hours.
   • Winner by: open rate or click rate.
5. Publish or schedule the post as usual.

The settings lock once the post is published.

Choosing the wait and the metric

Most opens happen in the first few hours after a send. A short wait picks a winner sooner but on less data; a long one is more accurate but delays the post for most of your list. Four hours suits most newsletters. For time-sensitive posts, choose 1 hour or skip the test.

Apple Mail opens emails automatically to load them, which inflates open rates. If many of your readers use Apple Mail, click rate is the more reliable way to pick a winner.

While a test runs

The post shows Testing subject lines with the time the winner will be picked, in your post list and on the post. To stop the rest of the send, open the post and click Cancel the rest. Readers in the test group keep the email they got, and nobody else is emailed.

The results

The post's Performance page, under Email, shows a Subject-line test table: each subject line, how many readers got it, its open rate and click rate, and which one won. The rates are for the test group only; the cards above show the whole send.

If there's a tie, or nobody opened or clicked, subject line A wins.

When a test doesn't run

The post still sends, with subject line A to everyone, when:

• your plan doesn't include subject-line A/B tests,
• there is only one subject line, or
• the test group would have fewer than 50 readers per subject line.

The results table then says the test didn't run.