What to test (and what not to test) in 2004.

With the general picture for DM results for newsletter publishers ranging from mediocre to dismal, 2004 is definitely a time to be cautious with your direct mail.

"Cautious" yes, but not to abandon it entirely. In most cases, it remains the lifeblood of new orders. In these times it's important to decide what and when to test and probably just as important, what not to test.

What to test

Test only really important things. For most publishers, that means new packages and price offers. Leave testing smaller elements to those few publishers who mail in the millions.

Give a test your best shot. Include the premium buckslip and/or brochure, guarantee certificate, etc. in a test package. Don't allow yourself the temptation of looking at so-so results and thinking, "Well, maybe if we added a brochure...." I've seldom heard of a package improving response by sexing it up with a few additional features.

You can go the other way. If you have a control package that is doing well, you can test cutting expenses by stripping out an element or two and seeing how the results hold.

Don't push too many chips to the center of the table. If your universe is 30,000 names, don't test a new package or price with an A/B split. Segregate 5,000 names for the test cell and mail the remaining 25,000 your dependable control.

It isn't chiseled in concrete that you have to test at least 5,000 names. Statistically, that is only an "indication" of future results--not a guarantee--so you could drop 4,000 or even 3,000. It's just increasing the gamble.

Time your testing right. "Half-life analysis" holds that on a Third Class mailing, you will have 50 percent of the total orders you will receive somewhere between 18 and 22 mail receipt days after the first return to your office (not the drop date).

If results look good then, and your printer and mailhouse are nimble, you can be in the mail with a roll-out about six weeks from the original drop. So time the test accordingly that far ahead of your best mail dates.

What not to test

Don't reinvent the wheel.

Don't test teaser copy. Use it. It almost always works. Write a half-dozen, test them internally on staff, spouse, any prospects who wander by, and use the one that seems to work.

Don't test 9X12 and other oversize formats. They can work for consumer titles, but for business newsletters they almost never pay off.

Don't test copy length. Long copy sells. Believe it.

Don't test First Class mail. Like oversize envelopes, the increased expense is almost never justified.

Don't test sample issues. This can be an almost theological debate among newsletter marketers, but as a general rule they don't work as well as a well-crafted sales letter. Could your newsletter be an exception? Perhaps a 5,000 test cell would say.

Don't test forced free trials. They're great for many publishers but hard to test. They are expensive ($5-$6 per name) and results take literally months because most marketers find a considerable percentage of the results don't arrive until at the end of the trial, or even after. If you haven't done FFTs, pick your very, very best list and try it.

Don't test business reply envelopes. They're mandatory for consumer mailings but arguable for business titles and not important enough to be worth testing.

Analyzing results

The thing to remember about analyzing test results is that it is almost always as much art as science. The quantities that we normally use to test are just not large enough. To have 95 percent confidence in a one-percent test result, plus or minue 10 percent, requires a sample size of 37,000--larger than many newsletter universes.

An MBA-type marketer I know uses standard deviations and algebraic formulae to determine results. He said, "Newsletter publishers like to think they have 'test results,' but more often what they have are anecdotes."

Sometimes, as an "expert" who has created DM packages for many titles, I feel like the guy who made the finest buggywhips on the market. But, as yet, results from other channels are not replacing direct mail as the backbone of newsletter and specialized information marketing--so most publishers are going to need to stay in there pitching in 2004.