Email Marketing: 7 Tips For Split-Testing Emails
By Anna Johnson on July 29th, 2010Split-testing emails is a sure way to work out what components of an email campaign generate the best results. By continually testing, tracking and innovating you will continue to improve open rates, click-through rates (CTRs), landing page conversion rates and other important metrics.
But you need to set up your tests the right way! If you don’t, you’ll get meaningless or misleading results, that will ultimately undermine your email marketing efforts.
So… what should you do to ensure your tests generate meaningful results? Here are seven (7) tips. These 7 tips provide a great check-list to refer to before conducting a given email test:
Tip 1: Split-test or multivariate test?
Firstly, depending on how many email subscribers you have, you may want to conduct a simple split-test – where you compare just one variable at a time (e.g. email subject line) – or a multivariate test, where you test multiple variables (e.g. type of promotion, subject line, graphics, etc).
I recommend you stick to a simple split-test unless your testing tool (see below) is able to manage multivariate tests AND you have a sufficiently large list to generate statistically meaningful results across all variables.
A lot depends on how representative your sample data is (i.e. the extent to which the subscribers exposed to your test reflect all your other or other potential subscribers). As a rule of thumb, however, assume you’ll need at least 1,000 responses per testing variable.
In other words, if you’re unlikely to get at least 1,000 responses (and certainly if you’re likely to get fewer than 1,000 responses) from your test, just run a split-test. Otherwise you may not get enough data to draw conclusions from your test.
Tip 2: Choose the right goals and metrics
Before you even set up your test, it’s important to be clear about your goals and the metrics intended to reflect the accomplishment (or not) of those goals.
For example, whilst metrics such as your email open rate and CTR may be indicators of a successful email campaign, they may not be as important as the landing page conversion rate. You may well find that some campaigns generate relatively low open rates but high conversion rates, which lead to higher sales and profits.
Depending on your business and goals, nor are ‘rates’ necessarily as important as actual numbers (e.g. dollar figures). So map out what you want your email subscribers to do (your goals), the metrics that will measure the extent to which they do those things, and establish which of those metrics you intend to test.
Tip 3: Test the right variables
Okay, so you know what metrics will indicate success or failure… but what components of your email campaign will you actually vary / test in order to measure the degree of success or failure?
Here are some ideas of variables to test, and, as you’ll see, they’re not just about the content of your email. As recent articles in Kikabink News have shown, the major determinants of response rates are often outside the content of the actual emails themselves:
- Day of sending
- Time of sending
- Frequency of sending
- Type of promotion (e.g. time-limited offer, coupon offer, etc)
- Subject line
- HTML versus text email
- Email layout
- Images / graphics / fonts / colors
- Length of email
- Providing the full content in the email versus providing a link to further information
- Including specific items such as price, discount rate, etc
- Email copy
Also, don’t forget to test removing, not just adding, certain items to ascertain the impact.
Tip 4: Properly configure your testing tool
Whether you’re using Google’s Website Optimizer or a new tool such as Optimizely, take the time to configure it properly and follow all the steps required to ensure it’s working and is measuring the metrics you intend to test.
Tip 5: Segment your email list
Unless you have a relatively small, ‘homogenous’ list, it’s a good idea to segment your email list. You may segment your list on the basis of, say, demographic data (age group, gender, etc) or behavioral data such as how they arrived on your list (e.g. co-registration, referral, email blast, etc) or which sign-up form they used. By segmenting your list you’ll be able to identify which campaign elements work better or worse with certain types of subscribers.
Tip 6: Base conclusions on statistical relevance
Nothing is conclusive (or as close to conclusive as you’re likely to get) until you have statistically meaningful results. This means that your test results are likely to carry across in future campaigns (all things being equal). In other words, you can safely say that Headline #1 will always beat Headline #2.
Your testing tool should measure for, and let you know, when you have enough data to draw statistically relevant conclusions. If it doesn’t… get another tool! You really don’t want to be calculating this stuff in your head or fiddling around with a spreadsheet program.
Tip 7: Keep on testing… as long as it pays
As the list in Tip #3 suggests, there are a lot of things you can test. To this end, once you’ve identified a ‘winner’ in a given test… that’s not the end of the story. It’s the beginning of the next test!
By the same token, you don’t want to be testing for the sake of testing. If you’re unlikely to do anything with the results of your test, don’t bother to test. Truly.
Testing is only worthwhile if you will, in fact, make substantive changes based on the results of your test(s) AND where those substantive tests are likely to yield sufficient returns on your investment (ROI) (i.e. the time, effort, etc you put into testing). If, given your limited time and resources, you’re better off spending your time doing something other than testing, then you should do the thing that generates the higher returns.
July 29th, 2010 at 5:47 pm
Anna, great advice. I also noticed something else worth doing related to email marketing. Make sure the email leads to a sales page that uses honest sales tactics. Otherwise, the reader has a “Oh yuck!” reaction. I wrote an article about that which you can read by clicking my name.