How many names do I have to have to mail for a valid test? (DM Notebook).
Here's the basic answer in two parts.
1. For large lists, say over 50,000 names, no number of names that any marketer is likely to be willing to use as a test will yield a statistically valid result.
I once saw a statistics chart which allowed that, in order to have 95 percent confidence that a test result of one percent could be valid with 10 percent plus or minus on a remail, a test cell of 37,500 names was required.
But who in newsletters often tests 37,500 names? So, for a list of 100,000, most marketers test 5,000 names, or perhaps 10,000, and if that works, retest with 25,000 to 50,000. Only if that also works, roll-out to the full list.
2. For smaller lists, under 10,000 names, there is little statistical reason for ever testing. The results obtained will not be valid as a predictor. Suppose you have a list of 8,800. You can mail 4,400 of them as a test. But, if you get 18 orders, which is quite acceptable to you (0.4 percent), a remail of the remaining half could bring in 28 orders (whoopee!) or eight, a failure-and both results would be "valid" based on the 18 orders you got from your first "test."
"If you don't believe how much response rates to small quantities can vary," says Allie Ash of Newsletter Holdings LLC, "mail two cells of 5,000 the exact same offer and see how much the results may differ."
What to do?
What to do? Most experienced marketers recommend testing in a quantity large enough to get XX positive responses. The most common numbers for "XX" are 40 or 50. Using 40, that means if you need a one percent response, you can test 4,000.
If your response rate is more like 0.5 percent, you need 8,000 names for the test. I've heard of bold or cautious marketers whose XX number could be as low as 20 or as high as 100.
In any event, remembering point one above, based on the results of that test size, do not roll out with a remail to a quantity more than five or ten times the test cell size.
Marlene Jensen, veteran magazine and newsletter publisher (and a member of the NL/ NL editorial advisory board), says (even about smaller lists) to test a significant percentage of names and go with it.
"I've heard of this Rule of 40 Responses," she says, "but I believe it was developed by companies dealing with bigger lists and then never adjusted for smaller lists.
"If you have a list of 3,000 names you want to use, and your desired response rate is one percent, that means if you mail the entire list, you would expect 30 orders. Does anyone actually believe that mailing the entire list will not give you a projectable response for mailing a second time? Just because you couldn't get 40 positive responses?
"Likewise," Jensen continues, "suppose you test every other name on that 3,000 list. Does anyone believe that the response you got would not be projectable against the remaining names, even if you only got 15 responses? To me, as long as you are testing a healthy segment of the entire universe, your results should be projectable."
Jensen adds that she launched Ancillary Profits newsletter with a test of 500 names selected from a list of 5,000, which drew 23 orders, a 4.6 response. The subsequent mailing to all 5,000 names drew 4.5.
|Printer friendly Cite/link Email Feedback|
|Publication:||The Newsletter on Newsletters|
|Date:||Oct 3, 2002|
|Previous Article:||What a 90 percent renewal rate is telling you. (Management).|
|Next Article:||Newsletter & Electronic Publishers Association. (Conference).|