How much is testing really worth?

Is newsletter testing overrated as a marketing tool? I was looking at results from four different tests I was involved in and was struck by how little "new" we actually learned for the time, trouble and expense involved.

Perhaps learning the conventional wisdom and going with your best judgment is just as effective, as well as easier and cheaper. To wit:

* First Class vs. Third. We split-tested 19,000 pieces with an A/B split. Both envelopes carried "Your Latest Issue Is Enclosed" copy and the 1C cell also contained "First Class Mail" on the carrier.

Conventional wisdom would have it that a First Class mailing usually brings in an increased response, but perhaps not enough to cover the additional cost of postage. In this case it was pretty much a wash: The 1C mailing did return 14 percent more orders, but the net profit ratio was one penny more than the 3C mailing.

Of course, other things being equal, more orders are always better.

* Personalized vs. non-personalized. This one did surprise us. We split tested 18,000 names A/B between "Dear Fred" and "Dear Subscriber." Conventional wisdom holds that personalization will add as much as 20 percent to cost, but will increase response at least that much.

This time the generic version outpulled the personalized one by 64 orders to 45 and by more than $1.00 in NPR. Plus, the test cells were large enough to yield "valid" results.

Now, in this instance the 18,000 names were all in-house names, subscribers to related titles who already knew and loved us. Perhaps that made a difference. In cases such as this, all you can do is schedule a retest ASAP to see if you can confirm if you have been wasting money on personalization.

* Long letter vs. short. As a copywriter, I'm wedded to the idea that "long copy sells," but we tested two mailings (32,000 total). One was a 4-pager and the other an "expanded" version with (we hoped) more reader-centered benefits copy.

Both were successful, but the long copy didn't add anything. In fact, the 4-pager drew a total of 106 orders and the 8-pager 100 orders (but with the additional costs, of course).

Perhaps long and very long letters work for some consumer markets, but 4 pages seems entirely adequate for business-to-business offers.

* Full year vs. six-month trial. Business newsletters don't often use short-term trials as introductory offers, but "experts" often recommend testing a shorter term offer to lower price resistance. We tested 23,000 names split between $297 (marked as "Best Deal") and $197 ("Trial").

In this case, neither offer was a great success, but the trial offer bombed. It drew only a bit more than 50 percent of the response of the full-year deal.

My thought was that $197 just wasn't enough of a difference from the full price to stimulate response. We might have done significantly better on the trial offer if we had cut the price and term to $97 and four months.

In selling a business newsletter your real problem is overcoming "organizational inertia." Getting a check of any size out of them is probably way more than half the battle. But once an organization writes one check, it is inclined to keep up the practice--which is a vote for the trial offer.

* Conclusion. I'd be the last to say, "Don't test." But about 40 years of collective industry experience in business newsletter marketing inclines me to conclude that a "classic package"--with envelope teaser, mailed business rate, a four-page sales letter, perhaps a premium buckslip, and an order device offering a full-year deal at the best price you can come up with--will work if anything is going to in selling to a particular market.

Then again, if I haven't confused you enough already, the ultimate test of whether testing is worth it or not is to test.

RELATED ARTICLE: The "Booby Prize"--an update

In the August 6th DM Notebook, I awarded a booby prize to Sports Illustrated for a gift subscription offer which, while offering me the opportunity to give a gift, required me to renew my own sub at the "regular price" of $83.44. I had recently renewed at $39.

Since then, however, I've received a regular renewal notice and at the rate of--guess what?--$39.

My first thought was, "Geeze, I guess they take one shot at seeing if some bozos will renew at $83 before beginning the regular renewal series."

But, giving them the benefit of the doubt, I expect the gift sub offer mailing was sent to the entire list that includes people who came on board from at least a dozen different offers.

Then, the regular renewal series begins, offering to renew me at the price I'd paid for the past two years.

Such are the risks of "mass mailing to one person." Keep track of how and at what price each of your subscribers came in and mail to them accordingly.--F.G.