How effective is your marketing? Here's how to make sure you're evaluating marketing effectiveness intelligently.

How do we know it's working?" That's a critically important marketing question that astute marketers know they must eventually answer. Fortunately, evaluating effectiveness is not difficult, especially when measurement is considered as the strategy is developed.

Before we can begin any discussion of evaluating effectiveness, however, we need to quickly review the integrated marketing model we discussed in previous columns. Remember, integrated marketing has three components:

* Brand marketing

* Direct marketing

* Customer relationship management.

For the purposes of evaluation, though, we need to focus on the first two components. Brand marketing has a single goal: to create awareness. Direct marketing also has a single goal: to generate response. Direct marketing means extending an offer: apply, attend, call, visit, or fill out. While evaluating the effectiveness of brand and direct marketing is not difficult, you need to recognize that they are evaluated differently. First, let's take a look at ...

EVALUATING BRAND MARKETING EFFECTIVENESS

Before you can evaluate your brand marketing strategy, you must know in whose mind you are trying to build awareness. In other words, you must have a clear sense of who your target audience is. Every marketing initiative, whether brand-building or response-generating, must be directed at a specific, definable audience.

To evaluate the effectiveness of brand marketing, marketers use a classic pre- and post-strategy. To begin, you conduct an initial study of brand awareness. With these baseline data in hand, you then undertake your image-building strategies (PR, media, special events, direct marketing, advertising, and promotion).

After your initial campaign has run its course, you then repeat the research to determine if your awareness has changed or improved. This is called "post-." If you do not conduct a pre- and a post- in tandem, then you simply cannot evaluate the effectiveness of your brand marketing, your awareness-building strategies.

Let's work through a short example. Suppose you decide that you want to be known for two things: academic quality, and job placement. Of course, you are interested in building this awareness with multiple audiences, but for this illustration, let's focus on adults. As part of your pre- research, you discover that adults identify the words "academic quality" and "job placement" with you 9 percent and 16 percent, respectively.

You then undertake a brand-building strategy that keys in on those two qualities. You stress them in ads, publications, speeches, hometown news releases, and special events. Twelve months later, you repeat the research (the post-) with the same instrument, and a sample from the same population. Looking at the adult data, you discover that this population now uses these two qualities to describe you 17 and 23 percent of the time. Based on this data, you can begin to say that your brand marketing campaign has been effective.

Yet, there's a twist. When parsing the data, you discover that while aggregate top-of-mind awareness has increased to 17 and 23 percent respectively, men account for most of this growth, and that your awareness along these two dimensions has actually declined among women. This finding indicates that while your campaign is generally successful, it is not specifically successful because women account for 62 percent of your enrollment. As a result of this insight, you conduct a series of focus groups with women, and make some changes to your campaign. This data becomes the new pre- research. You must then plan to repeat the study in the near term to see if you have gained traction with women.

EVALUATING DIRECT MARKETING EFFECTIVENESS

To measure the effectiveness of your direct marketing strategies, you must be able to tabulate responses to your different offers (how many called, how many sent in business reply cards, how many attended, etc.) To assist in this tabulation, marketers often include unique contact information (special phone numbers, mailing codes, URLs, P.O. box numbers, etc). This allows them to track the effectiveness of different media and offers to different populations.

Let's say you decide to run some radio spots announcing an open house for your adult ed program. Each ad (and even each ad running on different stations) might include a unique telephone number so you can determine how effective that ad was, based on how many adults called that unique number. An ad that ran in the mornings on the country station might have a telephone number that ends with 4321. The same ad in afternoon drive time on the easy listening station might have a telephone number that ends 5321. By keeping track of the numbers that were called, you are measuring particular response rates.

Still, just as there were a few twists with measuring brand marketing, there are a few twists with measuring direct marketing. For example, suppose you send an annual fund piece to 14,037 alumni. You receive responses from 1,720 alumni who contribute a total of $103,975. This indicates a 12.25 percent response rate with an average donation of $60.45.

Based on these data, was your campaign "successful"? Maybe. But, to answer the question about how you did this year, you need to take a took at how you did last year. When you do so, you discover that Last year, 1,373 alumni contributed. Your "hit" rate this year of 1,720 represents a significant percent increase. That's the good news. However, you also discover that the average gift fell by 11 percent from last year's average of $68.

One important evaluation tool is the ability to make meaningful comparisons between the results of the current campaign and similar campaigns from previous years or cycles. In the above example, I have no way of knowing if a $60.45 annual contribution is good, unless I can compare it to last year. Of course, within this larger alumni cohort, I would also want to know how that response and average gift size holds up with such key segments as:

* Year of graduation

* Gender

* Type of degree

* Major

Before we close, let's talk about the new direct marketing math.

UNDERSTANDING THE NEW DIRECT MARKETING MATH

One of the simplest (and most misleading) evaluations we can perform involves comparing the relative cost of one strategy with another. Those of us with a strong finance bent and no real understanding of marketing often use this approach as we struggle to find meaningful common denominators. However, rather than asking, "How much?" we need to ask, "What's the return?"

"How much?" -s a lazy evaluation. We believe intuitively, that a $2 search piece is twice as expensive as a $1 search piece. But what if the $2 search piece was four times more effective? As noted above, the key is keeping track of key response rates. For recruiting, this means not only inquiries, but also conversions to apps, deposits, and matriculants. Let me illustrate.

Above is a grid that contains a number of key variables including the cost to print and mail, search size, cost of the search package, percent who respond, number who respond, percent respondents who matriculate, number of matriculants, and final cost per matriculant.

As you can see, the cost of Package B is twice as much, per unit, than Package A ($3.80 vs. $1.90). Yet, because Package B generates twice the response rate, you don't need to produce and mail as many. It generates a higher response rate because it has been customized for each student based on their ZIP, academic interest, test score, and other variables. Because of this higher response rate, the actual number of students who respond to Package B is greater than A, even though more Package A brochures were sent. At this point, the CPI (or cost per inquiry) for each brochure is amazingly similar.

And here's where the numbers begin to diverge in favor of the package that was initially more expensive, Package B. More respondents go on to matriculate from Package B than Package A. As a result, the final CPM (cost per matriculant) is only $40 for Package B and nearly 30 percent more, $55, for Package A.

Someone who is trained to ask, "How much does it cost?" would instantly settle on Package A. Someone trained to ask, "What's the response?" would quickly see the sense behind the dollars spent on Package B. (Who's the better equipped evaluator here?)

A FEW SIMPLE RULES TO HELP YOU MEASURE EFFECTIVENESS

Let's close the discussion of evaluation by reiterating six rules:

1. Recognize that you cannot measure the effectiveness of brand marketing and direct marketing strategies in the same manner.

2. Understand that the classic protocol for evaluating your brand marketing strategies involves three steps: Pre-campaign research, launch the campaign, and post-campaign research. If you do not do the pre- and the post-, you cannot evaluate your brand of your brand campaign.

3. Include unique contact information on your direct media so you can calculate not just general response, but response to specific offers and media.

4. While aggregate data is helpful, don't be afraid to parse. Knowing that the number of alumni who give is up, white the number of women who gave was down, suggests that your campaign warrants further examination.

5. Collect and compare longitudinal data. One set of data is a snapshot; comparing new data with historical data shows trends.

6. Finally, remember that the best way to increase direct response is to precede your direct marketing efforts with a strong brand marketing strategy. U

        Cost to     Search     Cost for    % Who     # Who
        Print/       Size       Search    Respond   Respond
         Mail

Pkg A    $1.90      10,000     $19,000      8.2       820
Pkg B    $3.80      7,500      $28,500     17.2      1290

          CPI     % Respond/    Number      CPM
                   Matrics     Matrics

Pkg A   $23.17        42         344        $55
Pkg B   $22.09        55         710        $40

Bob Sovier is a senior VP of Stamats Communications (www.stamats.com). His new book, An Integrated Marketing Workbook: Planning Guide for Colleges and Universities is available through www.strategypublishing.com. For information on his free newsletter QuickTakes, e-mail [e-mail].