Hazards and hurdles of successful testing.

Even though "you have to test" is a useful answer to almost every newsletter marketing question, a lot of testing done by newsletter publishers is of doubtful value. Here's an outline of common testing problems:

* Don't test too many variables. With enough cells you can test price, package, letter, brochure and more all in one mailing. But the process is too large, complex and expensive for most of us. Most of us are better off testing only A vs. B.

* Don't test unimportant variables. Is it really worth testing $347 vs. $367? Testing $287 vs. $377 might yield some data that would make an important difference.

* Most b-to-b tests are too small. Without getting into the old arguments about testing in sufficient quantities to get 40 positive responses (insert any number from 20 or 30 to 100), almost all newsletter test quantities are too small to yield statistically valid information.

Many times to get "valid results" you'd need to test a quantity of 10,000 or 20,000 or more (depending on desired response rate), but if the whole list is 4,800 and you decide to test 2,400, the results are almost guaranteed not to be statistically valid.

(Note: This is "conventional" wisdom. NL/NL Editorial Advisory Board member Marlene Jensen emphatically disagrees and has numbers from her own experience to back up her view.)

* Don't rely on Net Profit Ratio (NPR) comparisons entirely. If you mailed 40,000 pieces of your control and two 5,000 cells as tests, the unit costs on the test segments are bound to be higher than the remainder of the mailing. It would be more realistic, for analysis, to assign those test cells a pro forma cost representing what it would cost to mail them in quantity.

* Why didn't the results of the rollout equal the test results? Yes, it is true that sometimes the list owner might have stiffed you by using hotline or multiple buyer names for the test and, sometimes, seasonality can explain significant differences in results. But, if your test quantities were too small (see above) test results might only be "reliable" plus or minus 30 percent.

If you selected three lists as winners from an eight-list test, the odds are great that those lists "happened to" come in at the top end of the probability curve the first time and will do less well on the remail.

List A could have tested 30 percent above the "norm" (huzzah!) but come in 30 percent below the norm (aarrghh!) on the remail. AND, STATISTICALLY, BOTH RESULTS ARE EQUALLY VALID.

If you want further evidence of the vagaries of results from small tests, try mailing two identical 5,000 cell pieces and check the results.

* Too often, results, when even well done, tend to be filed and forgotten. What sort of turnover have you seen in your marketing department in the past five years? Is the NTBU (Not Tested By Us) phenomenon alive and well at your place? Do the people who are there now have any real idea what you tested and learned in 1999? And are those 1999 results still valid?

* Underestimating the unpredictability of human nature. A famous story used to circulate in the 1980s that a publisher divided his mailing for a price testing into four different cells with, obviously, four different prices. The mailing house, however, sent all four variations to the same one-fourth of the list.

The second-highest price came in on top. That's not just testing one price against three others to four different audiences. One quarter of the prospects received all four offers and still opted for the second-highest price. As is often the case in testing, go figure.

For the mathematically inclined

"Can I Trust That Test?" is a marvelous, one-page article I'd be glad to fax anyone who asks for it by e-mailing [e-mail]. It's written by Bruce Kinsey and begins:

"You're launching a modestly priced new product to 100,000 names. You pulled an nth select of 8,000 names to see how it does. The test pulls in 160 hard orders--a commendable 2 percent response. Based on that, you'd get 2,000 orders on the rollout.

"How sure can you be of the test result?

"Easy! Find the standard error of proportion, using this simple formula."

Kinsey provides the algebraic formula with annotations.